Tag Archives: Glom

www.glom.org

Brussels and Prague

General, ,

I’ve been travelling more than usual in May and I’m not finished.

I spent a couple of days in Brussels at Thomas Vander Stichele‘s place, getting a crash course in the Flumotion streaming media server because I’ll be writing Flumotion‘s user documentation.

I’m writing quite a lot of documentation these days, and enjoying it as a holiday from writing code. It’s stress free in comparison. I’d like to make Openismus known for creating documentation along with our development and QA work. There’s always a need for it.

After Brussels I spent a few days with Liam’s grandparents in Karlsruhe before heading to Prague for FOSSCamp. Being away from him for so long was not easy.

Things I read on the train to Prague:

  • The GStreamer Application Development Manual: This is a truly useful document. I now feel like I understand how to use gstreamer. That has allowed me to make sense of the surprisingly mature gstreamer C++ bindings and should help my understanding of Flumotion. I now want to read the GStreamer Plugin Writer’s Guide too.
  • The GNOME Documentation Style Guide: This was better than expected, written by true strunkandwhiteistas who show real experience of writing technical documentation. I didn’t learn much new but was glad to see the advice presented for others. Many points are repeated, but that’s maybe to allow each audience to get the whole message without reading the other sections. Still, the text is sometimes long winded, suggesting a rogue extra author.

FOSSCamp was not quite the dull unstructured talk-fest that I feared, just because of the quality of the attendees, each of whom had something fascinating to explain. It was indeed mostly just talk, with little chance of any resulting action, but it was at least interesting talk.

I quickly introduced Glom to a small group of people who seemed positive and led a larger discussion about updates of stable upstream releases in stable versions of distros, mostly focusing on Ubuntu because only the Ubuntu people seemed to have opinions. Maybe the other distros’ processes are not so easily influenced. I think we already have a result, which I can hopefully mention soon.

I stayed an extra day to meet André Klapper, who is attending the Ubuntu Developer Summit, so we could talk about his bugmaster work for Openismus. We fight entropy. I attended the Ubuntu Mobile sessions in the morning before taking the train home to Munich, but it was impenetrable to anyone not already involved. But that’s UDS – it’s for people already working on stuff.

Both FOSSCamp and the overlap with UDS allowed me to meet many of my favourite people and I am guilty of enjoying their company when I should have been meeting more new people instead.

On Saturday we fly to Scotland (North Berwick) for two weeks so Liam can meet his other grandfather and aunt. I’ve tried to plan the pain out of flying with a five-month old baby for the first time, but it’s sure to be a challenge.

PostgreSQL has no Bugzilla

General

PostgreSQL is an open source database. I guess that they like databases and understand when they can be useful.

But PostgreSQL has no bug tracker. Patches must be sent to mailing lists. Bugs are sent to a mailing list, apparently via a web-form, and some bugs are listed on a page, apparently written by a person.

Want to see what bugs are open and what’s happening to them? Can’t. Want to see what’s the current status of your patch? Can’t.

Discovering Unused Code?

General,

Glom has been through a few refactorings. I’m sure that some of its functions are never called by anything, and I’d like to remove those to clean up the code a little.

Is there some tool that can tell me what’s not used in a C/C++ application? gcov can apparently tell me what code is used at runtime during specific tests, but running the application shouldn’t be necessary. I guess that some combination of nm and strip could give me a list, but that’s not ideal.

Glom: Calendar Portal

General,

Over the last couple of days I added a new “layout item” to Glom’s detail view. It works like the existing Related Records portal, but presents the related records via a calendar. It uses Mathias Hasselmann’s recent gtk_calendar_set_detail_func() addition in GTK+’s svn trunk.

Obviously this isn’t a perfect example. Any school administrator would be annoyed at having to add new lessons every week even if they are the same as the week before. Schedules are notoriously difficult to handle in generic database systems, but maybe there’s a neat way to even make this example useful by adding just one or two new features.

Glom\'s Lesson Planner Example

Big Tables in Glom: Leaks and libgda

General, ,

While experimenting with Glom versus a big musicbrainz database, valgrind’s memcheck helped me to find lots of small memory leaks in Glom, libgda, and libepc, and one huge GValue leak in libgda. Viewing a table of 600,000 records initially took 2 minutes (or 2 hours in Glom 1.6) and 1000Mb of memory. It now takes 1 minute and 250Mb of memory, mostly all during the execution of the SQL query.

That’s still awful, but it looks like libgda-4.0 will do this in about 1 second with 35Mb of memory, no more than when using the raw libpq API. That seems acceptable and makes me think that Glom is going to be fine with large tables. I don’t know what’s wrong with libgda-3.0. Maybe it’s not really using a database cursor when asked.

That had worried me for a while so I’m glad to know it will be OK. But I’ll wait for libgda-4.0’s API to settle down before switching Glom to it, to avoid disrupting the other developers working on Glom.

I also think that the 250Mb is being leaked every time I switch from Details to List view in Glom, but valgrind’s leak check seems to hit an endless loop, or is otherwise overloaded by this challenge, even when using less rows, so I’m not sure what the cause might be. If the leak is still there with libgda-4.0 then it should be easier to investigate.

There are still some other smaller leaks reported by valgrind, but I don’t believe most of them. For instance, it reports some leaks in std::string constructors even when the std::strings are temporary instances, not created by new.

By the way, here is my valgrind command for leak detection:

G_SLICE=always-malloc G_DEBUG=gc-friendly GLIBCPP_FORCE_NEW=1 GLIBCXX_FORCE_NEW=1 valgrind –tool=memcheck –leak-check=full –leak-resolution=high –num-callers=30 –suppressions=/somewhere/valgrind-python.supp –suppressions=/somewhere/gtk.suppression yourapp &> valgrind_output.txt

I found some of those environment variables, and Johan’s GTK+ valgrind suppressions file on the live.gnome.org valgrind page. And here is the official Python valgrind suppressions file.

Big Tables in Glom: GtkTreeView

General, ,

For the past week or so I’ve been improving how Glom shows big tables in its list view. I created a local copy of the musicbrainz database for testing. For instance, the “artist” table has almost 600,000 rows. Glom 1.6 is awful at this, but Glom in svn trunk is getting better. There are several problems which I’ll mention in different posts.

By the way, I made it slightly easier to use Glom on an existing database, but it will still act oddly on non-Glom field types, and it needs you to edit an XML file. I’m now ready to add a use-an-existing-database feature to the GUI and make it handle other field types, and maybe other database constraints, if anybody wants to pay Openismus to do that.

GtkTreeView calls gtk_tree_model_iter_next() on every row

The first problem is that GtkTreeView always calls gtk_tree_model_iter_next() on every row in your model, even if it has millions of rows. Even if you have a custom GtkTreeModel and even if you are using fixed-height mode. Surprisingly that happens quite quickly, but it’s really not sensible.

And this makes it even less possible to attach extra data via pointers in the user_data fields in your GtkTreeIter, because you can’t free that data until the model is destroyed, because you have no other way of knowing when nobody else could be using that GtkTreeIter. GtkTreeIter is not reference-counted – it’s copied by value. Even tinymail (otherwise very efficient) must have a huge array in memory so that the GtkTreeIter’s index into that array stays valid.

So GtkTreeView just isn’t made for millions of records. I don’t want to implement a new GtkTreeView, but if someone does then I hope we have something in GTK+ that we can all use. A generic Iterator type in Glib (Philip Van Hoof’s page) might be a small start on that.

Glom 1.6 bug fixes

General, , ,

We didn’t do a new major version of Glom recently, though we usually follow the GNOME schedule, because some of the new features are not quite ready, such as the drag-and-drop layout, and the new printout designer.

But this does mean that we have a chance to get a bug-fixed version of Glom’s stable version into Ubuntu’s stable version, for the first time. I’ve had some success with that, thanks to Chris Brotherton. It can still take a long time (10 days last time, though it felt like longer), and the longer it takes the more chance there is that a change in some other package can be a reason to wait. But I have a feeling that things are getting easier.

Of course, the more usable Glom is in Ubuntu, the more bug reports we can get. Many thanks to Jani Monoses and Pavel Mlčoch for keeping me busy, particularly for discovering the problems with changed behaviour in PostgreSQL 8.3, which we discovered late because PostgreSQL 8.3 only just started working for some people recently. These bugfixes mean yet more tarball releases for Ubuntu to package, and there’s always the risk that they will stop taking new versions as they really hit their freeze. Hopefully, they’ll manage to do just one more update because Jani found some more critical bugs.

After Ubuntu Hardy is released then our upstream bug fixes will probably never get into Ubuntu Hardy unless we can claim that they are security problems, as is still the problem with Glom in Ubuntu Gutsy. At FOSSCamp in Prague in May I’m apparently meant to be leading some discussion about how to get stable updates into stable distros. I have some ideas about who a distro (and its users) should trust, and about what is currently just providing a false sense of security at the cost of user experience. I don’t know what to expect from FOSSCamp so it might not have any effect. I didn’t see the point in flying to the last FOSSCamp in Boston for one day just for some seemingly aimless meeting and talking, but Prague is not far away by train, so it’s worth a look.

Bug-fixed Glom 1.6 in Ubuntu 7.10 “Gutsy”

General,

I’m regularly annoyed that no stable version of Ubuntu has a bug-fixed version of Glom. For instance, Ubuntu Gutsy has Glom 1.6.0 and the ubuntu-updates policy does not allow it to have Glom 1.6.6. Unfortunately Glom 1.6.0 and some dependencies have bugs making it mostly unusable, such as a failure when changing field types, and not shutting PostgreSQL down properly.

The irony is that Glom is totally open source / free software and is using the recommended deployment method (shipping with the distro itself), but this doesn’t allow us to get bug fixes to users. I understand that non security bug-fixes for non-essential software is not a priority for Ubuntu but it’s frustrating for users. I suspect that they bend the rules for essential stuff like OpenOffice. I find myself wishing for a security problem to fix in Glom just so I can get the non-security bug fixes released along with the security fix.

However, Ubuntu recently created the PPA (Personal Package Archive) system, so Daniel has made some updated packages available in the Openismus PPA. Forcing people to add a repository will lose us many non-technical users but it’s the best we can do right now. I’ve updated the Glom download page.

Glom’s Initial Dialog

General,

Glom now lets you browse the local network to open a document already opened by someone else, so you can easily both use the same database system.

That adds a new button in a file chooser dialog (though it doesn’t use the file system or URIs). This has complicated our already odd start-up dialog. We have some thoughts about the initial Glom dialog in the wiki but, to summarize, this is my current best mock-up for what we should have. Apologies for the inept use of Glade and Inkscape:

glom_initial_dialog_mockup_from_inkscape.png

The treeview row buttons are to avoid having to select and then click a separate button. I think that’s easier. I generally hate start-up dialogs, so I’d avoid it if I could and I’d welcome suggestions. I do think it’s better than having a wizard and I do think something like this is needed to deal with the two common cases:

  • The first-time user starts Glom and has no idea what he could do.
  • The regular user starts Glom and wants to open the database that he uses every day. Unlike with a word processor, people will rarely use more than one or two Glom files.

Glom in Maemo Chinook extras

General, ,

After much struggling with branch merging and fighting with incompatible scratchbox rootstraps, an alpha version of Glom is now available for Maemo’s new Chinook SDK, in the extras repository. Thanks to Armin and Johannes.

glom_maemo_screenshot.png

Software developers might also be interested in the dependencies that we packaged, in addition to the existing gtkmm and hildon*mm libraries: libglademm, libpq (postgres client), libgda, libgdamm.

I think Glom on Maemo could be very useful for tasks such as

  • Restaurant order-taking
  • Stock taking in warehouses
  • Surveying of property
  • Note-taking for deliveries or checkpoints
  • Portable databases
  • Jack Bauer fighting crime

But so far this is purely a proof of concept. Some work would be needed to make it really useful. For instance,

  • The screenshot shows that there is a huge amount of wasted space. The spacing and padding needs to be reduced to 1 pixel everywhere, and the table title and mode can be removed. We use libglade and don’t want to have a separate .glade file for the two platforms, so maybe a new feature in Glade/libglade could help us.
  • Obviously it shouldn’t say it’s in developer mode. This client-only version can’t even do developer mode.
  • The Details/List tabs and navigation buttons need to go into the menu so they are not seen normally.
  • The treeviews needs to have column headers. Applications must do this explicitly when porting to Maemo because it GTK+ changes the GtkTreeView defaults – this is unnecessary and frustrates people.
  • Entry widgets need the hildon keyboard hints so appropriate on-screen keyboards are used for numeric or text fields, plus auto-capitalization where appropriate.
  • Use hildon widgets such as the time and date widgets (needs modifications to hildon to allow them to be empty (NULL), though these should probably be in GTK+ anyway).
  • Special layouts for the Maemo version. We need to let the developer specify a full layouts for the regular desktop version and a cut-down layout for the handheld version, so he doesn’t need to keep two separate copies of the .glom file.
  • We probably need Mathias’ height-for-width extended layout to make sure that we only have vertical scrollbars, though I think we can fit a lot into the horizontal space, at least in full-screen mode.
  • Keyboard navigation: Make sure that this works well by default and allow the developer to specify a tab sequence.
  • Dealing with loss and reconnection of the network, requesting new connections when necessary, using libconic.
  • Support Maemo’s offline mode and other states (suspend).
  • Avoid excessive network re-connections and duplicate SQL queries
  • Self-hosting of databases. Glom on maemo is currently client-only, though the desktop version has self-hosting by starting its own postgres instances. So at the moment you need a central database server and a constant network connection. It would be nice to allow the database to exist on the handheld itself. We’d probably want to port the client-only version to sqlite, though I suppose we should check postgres’s performance on the internet tablet before dismissing it. Our use of libgda should make it fairly easy to use sqlite if necessary, though some generated SQL will need attention.
  • A simple replication feature. People will often need to add or edit data while offline and then have everything reconciled when they get back to base. This is a complex issue, but I hope we can offer some simple strategies for this which don’t require much human intervention.

I hope Openismus can find funding for that work somewhere.

By the way, Armin is now working on the Windows port.

Update: Here is a new screenshot after making some of the easiest small changes:
Glom 1.6.6 on Maemo