I’ve done some work to document Glom’s Python API. Here is a temporary snapshot, though I will try to get it onto library.gnome.org for the future. I used Sphinx, which is now used for Python itself. I’m fairly happy with it, though it’s not everything I hoped for.

What is the Glom Python API?

Glom lets you use Python to write field calculations (so field values can be based on other field values) and buttons scripts (stuff that happens when you press a button). That Python can code use the record and ui objects that Glom provides.

That Glom API is implemented by the pyglom (actually pyglom_1_14 currently) Python module. We use boost::python, writing the docstring comments in our C++ code, like so.

Bad Tools. Bad Syntax

Because it’s just a Python module, with docstrings, we can use standard Python documentation tools. However, I am not impressed by those tools. I am not impressed by Python documentation in general, and I suspect that the tools and conventions are the problem.

I understand that many developers don’t see the need for Python documentation because the arguments and return types for Python functions could be of any type and behaviour at runtime, but I believe in a deterministic universe, I know that most Python functions actually behave in a predictable way, I think that should be documented, and nobody can seriously write an application if they are really meant to doubt the behavior of the APIs they use. So let’s assume that Python APIs should be documented properly.

In the end, I had pretty good results with Sphinx, particularly after writing lots of documentation in its reStructuredText format. The output from pydoc was quite awful, though that’s partly because Glom’s python module is created with boost::python, which has some docstring issues and limitations.

But the main problem is that I can’t find any convention for documenting the arguments, return types and possible exceptions of Python functions. Various things (PEP 257, PEP 287,  reStructuredText, Sphinx’s reStructuredText primer) have been written about docstrings, but none of them mention this fundamental issue. So, these things are typically not documented, or are documented in a freeform unreliable way.

For comparison, here’s an example with doxygen for C++, using the simple Javadoc-style syntax:

/** A brief description in the first sentence.
* A more detailed description. This can be multiple sentences.
* Parameters such as @a boo can be mentioned here too, and
* mentions of an other_method() or an OtherClass will show up
* as links in the HTML. The parameter types and return type
* will be linked too.
*
* @param boo The goo's foo.
* @param hoo The voodoo that you do.
* @result The current moo.
*
* @throws FooError
* @deprecated Use some_other_method() instead, which does the necessary foo and bar.
*/
Moo some_method(const Boo& foo, Hoo hoo);

gtk-doc uses very similar conventions, aiming to document the same aspects of APIs, for C.

doxygen will even automatically create links to classes and functions that you mention. sphinx’s reStructuredText requires awkward syntax such :class:`Record`, which also unnecessarily shows the module name in the resulting HTML.

Note also that Sphinx doesn’t default to using docstrings anyway. It seems to want us to write API documentation in separate files. I’m convinced that it’s a mistake – I’ve only known APIs to be properly documented when the documentation is directly in the code as comments, for instance via doxygen or gtk-doc. Luckily I could use docstrings via Sphinx’s autodoc module, though that doesn’t support all of reStructuredText, such as section headings.

I also like how doyxgen or gtk-doc documentation generally lists the functions at the top, with brief descriptions, allowing me to click on a function’s link to jump to the detailed description. Instead, with the spinx-generated HTML, we have to scroll through all the details of the whole class. And all the module’s classes are on one HTML page, obscuring things even more. Maybe there’s some way to fix these issues via Sphinx configuration.

Yesterday I released the stable Glom 1.14. It has some incremental improvements – most noticeably:

• Better alignment of widgets in the details view, still using simple automatic layout.
• Many small UI improvements by Daniel Borgmann after his initial review.
• Some extra formatting options, such as alternative colors for negative numbers, and custom horizontal alignment, though Glom still chooses good defaults depending on the field type. Choices may also be shown as radio buttons now.
• There is more Python API available from button scripts. They can now change field values, navigate to other tables, start new records, or print reports.It’s easy to add more API because we use boost::python.
  (I must figure out how to generate HTML documentation for our Python module’s API by adding documentation in our boost::python C++ code.)

I often edit Glom’s .glade files by hand instead of using Glade (though that should get better now that we have a GtkSourceView catalog for Glade, and that Openismus is paying Tristan to do a little cleanup in Glade). But I often make mistakes, so I added some tests. These run when doing “make check”, for instance during “make distcheck”, thanks to the autotools’ lovely built-in features.

Here are links to the two tests, in case it’s useful to anybody else. Maybe other people have other useful tests?

• Check the glade files for validity (a Glade DTD would make this even better if someone can finish it)
• Check instantiation: Check that all top-level windows and dialogs can be instantiated and check that none have visible=true.?
  This is C++, with libxml++ and gtkmm, but it’s simple.
  It would need slightly cleverer code to get secondary top-level objects such as GtkAdjustments and GtkTreeBuffers, because gtk_builder_add_objects_from_file() fails unless you specify secondary objects explicitly. But we avoid using these in Glom, because it’s easy to break the code at runtime when one is added to the .glade file, and it’s less of a problem now that Glade doesn’t create them when the default values would be acceptable. Another workaround is to put all windows in their own .glade file, but that makes it hard to work with them all together in Glade.

Finally Figured Out boost::python

After lots of experimentation and two previous failed attempts, Glom now uses boost::python, with very little use of the nasty Python C API remaining. This should make it easier to add Python API to easily access and set field/record/table/database details from the Python that’s used in calculated fields or button scripts.

Useful Things I Now Know about boost::python

boost::python is no fun to get started with, but it’s now far easier for me to use than the Python C API.

I’ve mentioned the awful boost::python documentation before. Here are some essential things that I figured out, which are not really documented. This is thanks to helpful people on the boost::python mailing list. Corrections welcome – there’s so much here that some of it must be wrong.

None of this is very obvious or pleasant. If anyone had their first real C++ experience with boost::python then I’d forgive them for being put off C++ for good. I love C++ so that would be unfortunate.

“Converting” between C and C++ object types

C++ to C: To get the underlying PyObject* from a boost::python::object (awful docs), when you need to use a C function:

PyObject* cobject = theobject.ptr();

To test for a boost::python::object with a null underlying PyObject*, do:

if(cppobject.ptr())

Do not do if(cppobject). That tests if the python object is actually a boolean that is PyTrue.

C to C++: To get a boost::python::object for a PyObject*, when you received one from a C function, but you then need to use the result in a C++ function, or just want the improved memory management:

• If the C function gave you a reference that you should later unreference:

  boost::python::object cppobject( (boost::python::handle<>(cobject)) );

  (You need those extra brackets, for “interesting” compiler reasons.)

• Or, if you need to take a reference.

  boost::python::object cppobject(boost::python::borrowed(cobject));

  (Yes. that’s horrible too. I see no reason to expose boost::python::handle in the API.)

• However, you’ll need to use allow_null too to avoid exceptions if the PyObject* might be null. Well, any pointer could be null, so say hello to:

  boost::python::object cppobject(  (boost::python::handle<>(boost::python::allow_null(cobject))) );

  or

  boost::python::object(boost::python::borrowed(boost::python::allow_null(cobject)));

  (Shoot me now. No, reducing it to b::p::whatever is not a significant improvement.)
  I understand that a null PyObject* may sometimes be an exceptional unexpected event, but forcing the use of a try/catch by default just for a null pointer check is annoying. Explicit functions such as wrap() and wrap_not_null() would be so much easier.
  See the equivalent for gtkmm (plus calling reference() when necessary with non-widgets).

Using boost::python with your own wrapped C++ classes.

• To get a boost::python::object for an instance of your C++ class that you’ve wrapped for Python with boost::python::class, just do:

  boost::python::object obj(new YourWrappedClass);

• To get a C++ instance of your wrapped class from a boost::python::object, use boost::python::extract (as also mentioned generically below):

  boost::python::extract<MyClass*> extractor(cppobject);
  if(extractor.check()) myobject = extractor;

Others

• To get a C++ value out of a boost::python::object do, for instance:

  boost::python::extract<std::string> extractor(cppobject);
  if(extractor.check()) mystring = extractor;

  You can do

  mystring = boost::python::extract<std::string>(cppobject)

  without the check() but that will throw an exception if the underlying type is not really what you expect.

• boost::python likes to throw exceptions. I think it only ever throws boost::python::error_already_set, though the (Python) error is often not already set when it’s thrown. When the error is set, you’ll need to use Python C API to discover what it is.
• To provide [] syntax in python for your wrapped class, you’ll need to know how the C API works. Add this voodoo to your boot::python::class declaration:

  .def("__getitem__", &MyClass::getitem)
  .def("__len__", &MyClass::len)

  Those methods can then have signatures like this:

  boost::python::object getitem(const boost::python::object& cppitem);
  long len() const;

• To use Python date or time values, you will need to use C Python functions. For instance:

  PyObject* cobject = cppobject.ptr();
  int day = PyDateTime_GET_DAY(cobject);
  int month = PyDateTime_GET_MONTH(cobject);
  int year = PyDateTime_GET_YEAR(cobject) );

Boost has no .pc files

boost is a complete pain as a dependency. I understand that they don’t want to freeze API or ABI, because it’s a place for gradually improving API, though I think they should just have regular stable/devel phases with parallel installs. But I can’t forgive how difficult it is to get the header and linker options to use boost libraries. There are some m4 macros out there but they are hacky and fragile, and don’t actually work for boost::python. It shouldn’t be hard to provide pkg-config .pc files, so you wouldn’t need to do any compilation or linker checks in configure at all. I hacked some m4 code together based on some existing stuff, but I couldn’t recommend it.

So distro packagers won’t enjoy this new dependency. Sorry.

We (Openismus) recently got our GtkToolPalette widget into GTK+’s git master, to be seen in recent GTK+ 2.19 (unstable) tarballs. We haven’t had much extra time for this kind of thing, so I’m glad it’s finally done after being worked on now and then by Mathias Hasselmann, Jan Arne Petersen and Johannes Schmid. Thanks to Matthias Clasen for valuable reviewing and cleanup.

We hope that this can replace the hand-coded tool palette widgets in Glade and Gimp, as well as making this easier for new applications. It replaces EggToolPalette in libegg, where we started the work. Please take a look and report any problems. I’m already using it in a Glom branch.

Here’s the GTK+ API reference for GtkToolPalette. There’s a GtkToolPalette example in gtk-demo.

And here’s a little introductory Gtk::ToolPalette chapter I wrote for the gtkmm (C++) book. Note that gtkmm’s API reference for this is mostly empty, but fixed already in git.