All posts by murrayc

About murrayc

Murray Cumming is a software developer who works with Linux, C++, and Java. He runs Openismus GmbH - a small consultancy that works with open source platforms.

OnlineGlom: Slightly Saner Logins

General, , , ,

The OnlineGlom demo does not require a login. However, the code does let you set up a server that requires a login, and I noticed that a successful login for one person became a login for everybody else. So after the first login, it was as if no login was required for anybody. Yes, really. Of course, this would not do.

So I fixed that, I think, learning some things about Java Servlet sessions along the way. This text is mostly for my own reference, and so that people can tell me how wrong I am, because I’d like to know about that.

In the server-side code

Java servlets already set a JSESSIONID cookie in the browser, but you shouldn’t try to use that cookie to maintain a login across browser sessions. Instead, I now set and get a custom Cookie, in the server code, using javax.servlet.http.Cookie. HttpSession.getId() conveniently provides a unique-enough session ID for me to use in the Cookie. This page about Cookies with GWT seems to suggest setting the cookie in the client-side JavaScript code, using com.google.gwt.user.client.Cookies, but that sounds rather wrong.

I now store the username and password (Yes, that’s not good, so keep reading), associated with the session ID, in a structure that’s associated with the ServletContext, via the javax.servlet.ServletContext.setAttribute() method. I get the ServletContext via the ServletConfig.getServletContext() method. I believe that this single instance is available to the entire web “app”, and it seems to work across my various servlets. For instance, if I login to view a regular page, the images servlet can also then provide images to show in the page. I’d really like to know if this is not the right thing to do.

However, it still stores your PostgreSQL username and password in memory, so it can use it again if you have the cookie from your last successful login. It does not store the password on disk, but that is still not good, because it could presumably still allow someone to steal all the passwords after a breakin, which would then endanger users who use the same password on other website. I cannot easily avoid this because it’s the PostgreSQL username and password that I’m using for login. PostgreSQL does store a hash rather than the plaintext password, but still requires the plaintext password to be supplied to it. I think I’ll have to generate PostgreSQL passwords and hide them behind a separate login username/password. Those passwords will still be stored in plaintext, but we won’t be storing the password entered by the user. I’d like to make this generic enough that I can use other authentication systems, such as Google’s for App Engine.

To avoid session hijacking, I made the cookie “secure”, meaning that it may only be provided via a secure protocol, such as HTTP. I believe this also means that client (javascript) code is not allowed to read it, so it can only be read by the server via HTTP(S). I did that with the javax.servlet.http.Cookie.setSecure() method, though I had to make a build change to make that available.

The login servlet now checks that it has been called via HTTPS, by using the ServletRequest.isSecure() method, and uses HTTPS when testing via mvn gwt:run. It refuses to do any authentication if HTTPS was not used, logging an error on the server.

In the client side code

I added a check for what protocol is used. If it’s not https then I warn that login cannot work. This does not add any security, but it’s a helpful hint.

Actually, the entire site must therefore be served via HTTPS, not just the login page, or we would violate the Same Origin Policy by mixing protocols, which the browser would rightfully complain about. At this point I noticed that most serious sites with logins now use HTTPS for their entire site. For instance, Google, Amazon, Facebook. This seems like a good simple rule, though I wonder if many projects don’t enforce it just to make debugging easier.

I also converted the popup login dialog into a proper login page, making sure that it takes the user to the desired page afterwards.

Glom: Experimental MySQL Support

General, , ,

Glom uses PostgreSQL and doesn’t try to offer the user a choice of anything else. That’s because it does what Glom needs, there’s no need to confound the user with an incomprehensible choice, and I’ve no wish to maintain multiple sets of code. It’s hard enough keeping up with changes in PostgreSQL, though Glom’s regression tests help.

However, I played around with adding MySQL support as a build-time alternative via the –enable-mysql configure option. The basic stuff now works both in the UI and in the regression tests. Those tests can now run each self-hosting database test with all 3 backends.

This is mostly just so I could learn about MySQL, so I can reimplement it in Java for OnlineGlom. That would let me use Google’s Cloud SQL, which is based on MySQL. The main work has been figuring out how to initialize a MySQL database store on disk and then start and stop MySQL instances. It’s even more funky than with PostgreSQL. I did need an addition to libgda to support non-standard MySQL port numbers but, as usual, Vivien Malerba fixed that for me quickly. There’s also the rather huge problem that AppArmor on Ubuntu prevents us from starting MySQL with anything but the standard database data, and we can’t expect the user to go editing AppArmor config files. At least with MySQL 5.6, it should be possible to start a MySQL instance without it having no password for a few seconds, as is necessary with MySQL 5.5. I need to start and stop custom instances so I can run tests automatically.

I’ve committed it to Glom’s git master, and its in the 1.23.3 release, just in case anyone wants to improve it. There are some TODO_MySQL comments in the tests where we expect something to fail with MySQL. For instance, I have not added support for editing the MySQL users and groups. And there are likely to be problems with keeping data when changing field types, which doesn’t seem to be tested thoroughly for any backend. libgda is also missing some support for binary field types, needed for Glom’s image fields.

Over the years, various people have complained about Glom not using MySQL. Here is your chance to actually work on that, with tests to show if your work is enough.

Openismus running jenkins for continuous integration

General

Over the last couple of weeks, I’ve been playing with a Jenkins installation at jenkins.openismus.com, building some of the Openismus projects. Here are some notes about my experience.

Installation

This runs on an Amazon EC2 instance. Initial installation was surprisingly simple and well documented, though it took me a while to figure out how to use Jenkins properly. I initially used the official Ubuntu 12.10 packages for Jenkins, but they are a little old so I had to switch to using the Debian/Ubuntu packages from jenkins.org to fix a bug with the copyArtifacts plugin. The two packages seem to be structured very differently, so I had to remove all the Ubuntu Jenkins packages before installing the jenkins.org packages, to avoid a conflict.

See also the Jenkins standard security setup instructions, though I had to use the “Jenkins own user database -> Allow users to sign up” feature first, to create a user which I could then enter in to the matrix grid. I then disabled the “Allow users to sign up” checkbox.

Although Jenkins can use slave servers, and probably should, I’m doing everything on one server for now, because I’m afraid of the Amazon EC2 costs getting out of control. Luckily we don’t need to run each build more than once or twice per day to get some benefit. Later I will probably try running EC2 spot instances for the builds. Maybe that won’t be too expensive.

Git-based projects with Jenkins

You’ll need to use the pluginManager page to install the Git plugin, so that there is something other than “None” listed under “Source Code Control” when creating a job. Of course, we have to “apt-get install git” too. We must also specify a git username and email address for the “git plugin” on the configure page, to avoid “Please tell me who you are” errors in the job when Jenkins tries to locally tag the checked out git repository. Neither the configure page or the pluginManager admin pages seem to be linked from anywhere, so I had to discover them via google searches.

Be careful to specify the master branch rather than leaving that blank, or I think Jenkins will try building arbitrary branches, and maybe all of them.

You can specify a “git clean -dfx” via the “Clean after checkout” option under the “advanced” section, and you probably should so you get a truly clean build each time.

You can use the “Poll SCM” Build trigger, with the cronjob syntax, to regularly check the git repository for changes. This is not ideal, but to do it properly you’d need to add a git hook to the git repository to request a build from your jenkins server whenever there is a git commit.

Multiple branches

You can specify more than one git branch, to make Jenkins try building more than just one, but it’s hard see what branch was built when looking at the results.

Simple maven builds

For maven-based Java projects such as OnlineGlom, Jenkins is very straightforward, because maven typically downloads all the dependencies without expecting anything to be installed already, and “maven package” typically does the whole build.

Autotools builds, or similar

For autotools-based projects, you’ll need to make sure that you’ve “apt-get install”ed the project’s dependencies.

Then you must specify the configure and make (or qmake) steps in a build step.

Of course, many real-world projects will need newer versions of their dependencies. For instance, we build maliit-plugins, which depends on maliit-framework, which we develop in sync. For this, I:

  • Tell the maliit-framework job’s build to “make install” into a local directory, via the “–prefix=” configure option.
  • Use an “archive the artifacts” post-build action to store everything in that directory.
  • Use a “copy artifacts from another project” build step in the maliit-plugins job.
  • Export several variables so the build system has access to the dependency in the local prefix. For instance, (maliit uses hateful awful qtmake instead of autotools, but you’d need something similar for autotools): 
    export MALIIT_FW_PREFIX=$WORKSPACE/build_install
export C_INCLUDE_PATH=$MALIIT_FW_PREFIX/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=$MALIIT_FW_PREFIX/include:$CPLUS_INCLUDE_PATH
export PKG_CONFIG_PATH=$MALIIT_FW_PREFIX/lib/pkgconfig:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=$MALIIT_FW_PREFIX/lib:$LD_LIBRARY_PATH
export XDG_DATA_DIRS=$MALIIT_FW_PREFIX/share:$XDG_DATA_DIRS
export XDG_CONFIG_DIRS=$MALIIT_FW_PREFIX/etc/xdg:$XDG_CONFIG_DIRS
export QMAKEFEATURES=$MALIIT_FW_PREFIX/share/qt4/mkspecs/features/:$QMAKEFEATURES

You can use the “” build trigger to make Jenkins try a build whenever a dependency is built.

I have not tried this with multiple built dependencies. I imagine it could get awkard. It feels like Jenkins needs a plugin for autotools to make this simpler.

Multiple configurations

You can create “multiple configuration” jobs to try multiple ways of building your project. For instance, you might provide different sets of options to your configure script. But I couldn’t use this feature due to the spaces that it puts in the build paths. So I created separate top-level jobs for each configuration. Other people seem to do the same, maybe for the same reason.

Email notifcation

I’ve tried using Amazon’s Simple Email Service to send notification emails about build failures, but I don’t have that working yet. I’ll update this if I do.

Programming For Kids

General, , , ,

I tried a couple of applications with my four-year-old son: Pictomir and MIT’s Scratch.

Neither seem particularly well maintained and neither are suitable for young children without lots of supervision. Pictomir is easier to get started with, but not easy enough. Scratch is probably more interesting to older children, though they’ll partly be learning about how software is still so often arbitrarily annoying, and I’d prefer that they were introduced via a better example. I’m very tempted to write something better.

Pictomir

pictomir
Pictomir at first start up.

Pictomir leads the child through a series of levels, telling an R2D2 robot (Don’t tell George Lucas) to move around some isomorphic squares to paint some tiles. At the beginning the program exists and the child just needs to click the play button. In subsequent stages the child has to build part of the program himself, and then all of the program.

The available commands, at least at the start, are icons for left-turn, forwards, right-turn and paint at the top of the screen. These may be be dragged into the available boxes in the program at the right of the screen, though you can instead put them there by clicking to select and then clicking on the empty box in the program. However, the available commands are far away from the program where you must place them, so the child has to spend lots of time moving the mouse pointer back and forth across a wasteland of empty space.

I found an official Ubuntu package in the developer’s PPA. It was last updated to Ubuntu 10.10 (Maverick), but you can install it manually on later versions. All of its UI is in Russian, though you can ignore most of it because the UI is icon-based. When I built the newer code from source a few months ago, I think the UI was translated.

I guess that Pictomir was developed for a specific screen size and generally older PCs. It does not scale the playing area up, so the child has to deal with a tiny R2D2 on a tiny grid in the middle of wide empty space.

pictomir has a bare svn repository, though I can’t see how clean its commits are, and I don’t know if it’s still used. There are no commits since last year. My svn says that the svn format is too old to check it out, though I’ve checked it on on previous Ubuntu installations.

It’s website it quite awful (and only in Russian). I only discovered Pictomir thanks to commenters on Google+.

Scratch

The Scratch UI has several problems that make life hard for the child:

scratch
Scratch, after creating and running a small program.
  • Scratch requires the child to cope with tiny targets (See Fitt’s Law)
  • Scratch demands the use of drag and drop. This is frustrating for normal users, let alone children, and really hard with some laptop touchpads or trackpoints.
  • Update: Scratch requires the child to understand the difference between left-click and right-click. Right-click brings up a help context menu, which just causes confusion.
  • Update: The tiny commands have tiny text edit boxes. Children have a high chance of clicking it instead of clicking the command block itself. This problem could be partly solved by accepting drags on the edit box.
  • Scratch requires the child to double-click (see below).
  • The commands require the children to read, but a subset of the commands could instead be respresented by icons. The words are particularly hard for beginner readers because they are written in such a tiny font.
  • To get started with scratch, you need to figure out that you need to double-click on the first program command to actually start your program. There is no simple run button. You can cause your program to be run by clicking the green Flag button, but that’s something you need to put in your program.
  • When you try to move a command around in the program, it moves the command and all subsequent commands, as a group. So moving just one command means moving stuff around and then moving some of the stuff back to where you started.
  • Your program moves the Scratch cat sprite around the canvas, but that canvas is tiny, and the sprite can move off screen. The only way I found to get it back was to double-click the set-x-to and set-y-to program commands. I can figure that out, but beginners will not.
  • When you’ve drawn all over your canvas, the only way to wipe it is to go to the Pen set of commands and double-click the Clear command.
  • The sprite moves far too fast for the child to understand the relationship between the commands and the motion, without someone explaining it. If the command says move-10-steps then I’d expect the sprite to visibly move 10 steps. Advanced users might prefer it to be faster and smoother, but the default should make things obvious.

There are some other strange things in the UI which are probably a side-effect of its eccentric (Smalltalk) implementation.

  • Scratch has a menu (well, an icon-menu) for changing its language, instead of just using the user’s regular language.
  • None of the UI elements (menus, tabs, buttons, file chooser dialog, etc) match other standard applications on the system.

I think Scratch could be really popular with younger kids if it ran on Android with a touch interface and with less text, maybe as a simpler version. But the developers don’t seem to have any interest in that – they seem to be working on a Flash-based version of Scratch for the web, which is unlikely to work well with touchscreens, even if they can get Flash to work on Android and iPhone (they won’t). And I have no interest in hacking on Smalltalk. There seem to be various programming-language-based forks of at least the underlying engine, but none seem to be successful.

 

Online Glom: Image fields

General, ,

I found some work in one of my old branches and cleaned it up, so now OnlineGlom supports image fields too.

Online Glom with an image field

As usual, it was far more work than seemed necessary. GWT’s Image widget is not much more than a wrapper around the HTML <img> tag, so I had to create a separate service, with the same authentication system, to serve image data and invent a URL syntax to refer to the images from the database. It is certainly easier with GTK+ code on the desktop, even when delivering the image data asynchronously. This feels like something that a web progamming system should take care of, even if this is what happens behind the scenes. I wonder if any do.

Next, I want to make sure that OnlineGlom can handle tables whose primary keys are not numeric, because we’ve been hard-coding that in a few places. Then I hope I can start the big job of supporting data editing.

gtkmm maintainership

General,

I have not had enough time recently for my maintainership of gtkmm, glibmm, libxml++ and libsigc++. And I’m unlikely to find time soon. Family and (income earning) work have taken priority. That’s why glibmm and gtkmm didn’t have .0 releases until a couple of weeks after the GNOME .0 release, and why there was no real attempt to follow the GNOME release schedule this time. I’ve hardly touched cairomm for years.

It’s not a lot of work but I’ve had to find a few days every few months to keep glibmm and gtkmm up to date with API additions in glib and GTK+, to keep roughly in sync with them. There has been rather a lot of new API recently. See the list of new API in gtkmm 3.6, glibmm 2.34 , gtkmm 3.4, and glibmm 2.32. Krzesimir’s ongoing attempt to use the introspection information (rather than .defs files) would help with this but not hugely because we’d still need to take care that the API is right for C++. Some initial .hg/ccg creator script would probably be most helpful.

I have no time for the more difficult bugs that turn up occasionally. Luckily Kjell Ahlstedt, José Alburquerque and others arrive regularly in bugzilla with great fixes for hard problems. I trust Kjell and José to commit directly, but they deserve some decent patch review. Unfortunately I often keep them waiting far too long.

I need to make some change. As a start, I might stop receiving bug email for glibmm and gtkmm, and stop worrying about adding new API. I guess it will take at least one missed GNOME release cycle for some other people to take over my responsibilities.

Openismus Revived

General, ,

Here’s an update since my last status post in June.

Things have improved for Openismus even more though we are not complacent. Several proposals, including the ones I mentioned in that last blog post, have resulted in customer contracts. So we are now busy working on the Maliit input method system (virtual keyboard), Wayland, Rygel UPnP/DLNA and Evolution Data Server (EDS). We are even thinking of hiring another developer if we can find someone who is just right.

We are now established in the habit of creating proposals for customers, revising them, and shifting into implementation. We take our customer proposals seriously, making sure that the developers are the main authors and making sure that they don’t leave questions unanswered. If there’s something we might help your company with then we’d like the chance to convince you too.

Michael Hasselmann is now the main person negotiating new work for us and keeping some of that work on schedule. He has been very successful – a surprise to himself but not to us. We are calling him a Sales Engineer but that doesn’t really do his dedication justice.

Michael can travel much more than I could for the last few years. Right now he’s at the Automotive Linux Summit in the UK and tomorrow he will be at the X Developers Conference in Nürnberg with Jan Arne Petersen (where our interest is mostly in Wayland and input methods). On Monday and Tuesday he’ll visit me in the Munich office.

We have achieved this thanks, of course, to the hard work of our whole team at Openismus. They have fought hard so we can all keep doing worthwhile work that we enjoy. I am proud of them and glad to be part of this.

Generating SPDX files with licensecheck

General,

This week I had to provide an SPDX file to a customer. SPDX seems to be a way to describe the licensing of software components, to help with open source compliance. Here is an official example (though it is probably not up to date with the current SPDX specification).

However, there are no open source tools to create or edit SPDX files, so I created a little openismus-spdx-generator.py Python script that uses Debian’s licensecheck utility to scan a project and then outputs a skeleton SPDX file in RDF format. It is a quick hack with no real error checking and I have barely read the SPDX specification, so please do improve it.

My first impression is that SPDX is rather unwieldy. The RDF (XML) format is verbose and seems to focus on being a snapshot of software via checksums of all its source files, rather than specifying a particular version or revision of the software as a whole. I don’t see any attempt to list dependencies and their licenses. More strangely, it looks like the .XLS (Microsoft Excel) spreadsheet format is the preferred format, which sets off my corporate-drones-doing-painfully-silly-things-that-they-believe-are-normal alarm bells.

There are official Java-based SPDX tools to convert between the various SPDX formats, and maybe to validate SPDX files. You’ll need to build them with the ruby-based buildr build system. Then you are left with some .jar files that have to be run via “java -jar target/whatever.jar the-spdx-file” after setting JAVA_HOME correctly.  Java programs are hard enough to package on Linux distros, but I’m sure that buildr makes it even less likely.

Anyway, the tools crash for me on the provided example files. The git repository has no branches or tags, so it’s hard to know which version is supposed to work and I don’t have confidence that the specification, example files, and tools are in sync with each other at any particular time.

Most of the SPDX file contents will be the result of a scan anyway, so rather than demanding that source code is supplied to me with an SPDX file, I’d generally prefer that the software just had proper COPYING files and source code headers. That seems like an easier requirement to comply with.

It’s all a bit linuxfoundationy.