Tag Archives: libsigc++

Using GitHub Actions to check builds

General, , , , , , ,

GitHub Actions

A couple of weeks ago I spent some time adding GitHub Actions to some of my projects. I was lucky enough to get access to the beta, but it has now become available for everyone.

GitHub Actions lets you specify builds, tests, etc, to run on certain triggers, such as after every commit push. I’ve now used it in several projects, each using different build environments. It has worked very well, and has really improved the developer experience, and therefore improved the code, in a few of my projects. It has encouraged me to use pull requests even for changes to niche personal projects so I can benefit fully.

Actually, I’m not sure that Actions is the best name for the system. It really lets you describe Jobs, and each Job might use a few Actions. But maybe that’s just a matter of perspective.

Now that it’s public, the pricing suggests that most open source projects will have enough time for free, for which I’m grateful. Nevertheless, history shows that it wouldn’t be wise to tie your projects too closely to one proprietary system. For instance, that’s another reason not to duplicate your build rules in a GitHub Actions workflow file. You should have a simple build system that you can just call. I generally do that via a simple Makefile, as I mention below.

Here are the GitHub projects that I’ve added GitHub Actions to:

libsigc++

We now have GitHub Actions to build and run tests for libsigc++ with several versions of 3 C++ compilers: 3 versions of gcc, 5 versions of clang, and even 1 version of msvc (thanks to Stuart Dootson). We will also add an action to check our C++ code formatting via clang-format.  If you’d like every libsigc++ commit to work with some other compiler, please do try to make that happen via a pull request.

This will give us valuable confidence that our changes will work everywhere, and give us valuable clues when they don’t. It’s great to know that people can get this feedback automatically when they submit a PR, without waiting for a human.

I might be abusing GitHub’s generosity with so many builds jobs, but luckily there are not that many commits to this project. I wonder if GitHub hope to fund some of the GitHub Actions costs for open source projects via GitHub Sponsors.

angular-bigoquiz -client (The bigoquiz.com frontend)

I added GitHub Actions to build and run tests for this Angular project. I put the npm/ng commands behind a Makefile to keep the CI rules as simple as possible. The only thing worse than having undocumented or overly-complicated build steps, is repeating those incantations in multiple places.

I like how it uses just one Workflow definition, but runs it with several Node versions, thanks to the strategy.matix specifier, and the with.node-version specifier made available by the actions/setup-node action.

See the Workflow definitions.

go-bigoquiz -server (The bigoquiz.com backend)

I added GitHub Actions to build and run tests for this Go microservice. It only builds for go 1.12, but I think I could parameterise that like the node version for angular-bigoquiz-client.

Again, I put the go commands behind a Makefile to keep the CI rules as simple as possible.

I did some major refactoring of this code, adding tests along the way, and having CI made that fairly painless.

See the Workflow definitions.

Minor Imperfections

I have lots of experience using Atlassian’s competing Bitbucket Pipelines system. GitHub Actions feels very similar, though I still miss a few things:

Can’t trigger builds for individual commits

I often create PRs with many small commits while refactoring old code. 10 commits is not so unusual, and 30 has happened. Good test coverage makes this possible, and inevitably that test coverage shows a problem in one of the commits. When that happens in a BitBucket PR, with BitBucket Pipelines, I like doing a manual binary search, starting a new CI run for individual commits until I see where the problem started. (Actually, I wish that it would automatically do that binary search, like a git bisect.)

Unfortunately, I can’t see any way to start a job for an arbitrary commit, so I can only ever see the result for the latest commit in a PR.

No caching

Doing full builds of projects on every commit is simple but wasteful. Ideally, we could do reliable incremental builds, though that’s hard to get right. But some caching could save time and heat.

For instance, Bitbucket Pipelines lets me cache previous Docker build steps and even gives me a ccache cache to reuse C++ compilation units. Well, the ccache cache is not a real cache – it’s just a dump of files that’s re-generated every couple of weeks, and is stale in the meantime, but it’s better than nothing.

Providing CI build time to companies is obviously meant to be a profitable business. Unfortunately, it makes the incentives a bit confused. For instance, I don’t believe that Bitbucket are eager to help their customers spend less money on build time, so they don’t offer very capable caching. But competing services, such as GitHub Actions, if they get this right, could make them care about losing customers.

Brain, refactored

General, , , , , , ,

For a long time I was too busy to learn anything much new, while running a company and caring for two small children. But more recently, I had the time to catch up on some technical knowledge and I remembered how much I love pure learning.

This post seems to be a big list of what I’ve learned over the last couple of years. Actually, I meant to write it a year ago, so now it’s also a list of what I need to revise.

C++

I caught up with the C++ renaissance by reading Effective Modern C++. and watching heaps of CppCon, C++Now, and Meeting C++ conference videos, such as CppCon 2014: Herb Sutter’s Back to the Basics: Essentials of Modern C++ Style, (the one with almost always auto).

I exercised my growing knowledge of modern C++ by first using some modern C++ in gtkmm and Glom, and then doing a massive rewrite of libsigc++. I also explored some modern C++ generic programming techniques with tuples and dynamic programming.

Along the way I experimented, in Glom’s code, with some of the ideas from Sean Parent’s talks, such as the “no naked for loops” idea from his “C++ Seasoning” talk from Going Native 2013.

I read the old Boost Graph Library (BGL) book, to tie C++ and graph theory together in my head. I spent lots of time, off and on, trying to clean the BGL code up, and modernize it, but it’s a struggle and I’ve only managed to get some simple stuff accepted so far.

The BGL book is ancient but its ideas about C++ concepts are only just about to become mainstream, hopefully in C++20. I also read Alexander Stepanov’s  wonderful From Mathematics to Generic Programming book, which convinced me even more.

I tried to get a simple explicit operator bool check into clang-tidy itself, but that didn’t get far and I chose not to be a pain by being pushy about such an unimportant contribution.

I started to learn about some compiler theory by writing some actual code.

I listened to the complete backlog of CppCast podcast episodes and I now listen to every episode as soon as its published.

Java and Android

I revised and updated my Java skills by reading Effective Java Programming and Programming Android. I regularly listened to the Fragmented and Android Backstage podcasts, but not so much recently. I gained some real-world Android programming experience by creating the Android app for Galaxy Zoo after first trying to create an Android app for my Glom database system. The Galaxy Zoo app was quite popular, but sadly, it’s not working right now – I need to find time to adapt it to their changed  backend API.

Java and web

I spent some time with Java as a way to work on backend (distributed) systems, reading the JAX-RS book, the,Spring Microservices in action book, and the Spring in Action book.

I gained some more Java/GWT/AppEngine (and Resty-GWT) experience by creating the Big-O Quiz website. I  used bigoquiz.com to take organized notes about the computer science that I learned subsequently. It asks me questions to remind me. But see below about me rewriting bigoquiz.com with Go and Angular.

iOS / Objective-C

I learned iOS (iPhone/iPad) programming by watching all 18 lectures in Stanford’s “Developing iOS 7 Apps for iPhone and iPad” course and then creating the iOS app for Galaxy Zoo. I used Objective-C, but now I have a reason to learn Swift.

Like the Android app, this is currently broken, because the backend server now has a very different API.

Go

I learned Go via the The Go Programming Language book, taking extensive notes from the perspective of a C++ developer. To gain initial experience, I rewrote bigoquiz.com‘s backend with Go, instead of Java, and found the resulting code refreshingly simple.

I now use Go regularly at work for backend microservices and I’m quite content with it. For larger projects I’d still prefer C++, but not all projects need to be large.

Angular

I finally gave up on GWT, long after everyone else realised that it’s barely maintained. So I took the opportunity to rewrite the bigoquiz.com frontend in Angular, using Typescript. Learning it was easy with the excellent Angular tutorial.

I’ve since tried out React too.

Scala

I read the huge Programming in Scala book, because I use Scala on a large project at work. I’m fond of it, but I think its simplest ideas should just be in Java, and the cleverest stuff tends to make code hard to read.

Algorithms and Data Structures

I grew to love computer science by doing Coursera’s/Stanford’s “Algorithms: Design and Analysis” course, part 1 and part 2 by Tim Roughgarden. I then read Cracking The Coding Interview, The Algorithm Design Manual, Algorithms, by Robert Sedgewick, and the first half of the CLRS Introduction to Algorithms book. I worked through the first few lectures of Tim Roughgarden’s “A second course in Algorithms” to cover max-flow and min-cost matching algorithms.

I went through the material 2 or 3 times, from various perspectives, eventually making sure to actually implement each data structure and algorithm.

I also read Jon Bentley’s Programming Pearls book, for some practical advice.

I watched, and re-watched, far too many of Tushar Roy’s algorithm walkthrough videos.

Linux

I already have years of Linux development experience, but I knew I was missing out on some knowledge and habits. So I solidified my Linux knowledge by reading Linux System Programming. (And now, later, I think I should read it again, though I wish there was a newer edition.)

I’m still looking for a really good book about bash scripting and general command line cleverness.

Distributed Systems

I didn’t have practical experience with at-scale server-based systems but I became acquainted with the ideas by reading Google’s Site Reliability Engineering book.

I discovered it later, but Tim Berglund’s 4-hour Distributed Systems in One Lesson series of videos  is an excellent and engaging overview of the main themes. If it’s no longer available on YouTube but I think it’s so good that you should buy it.

Mikito Takada’s free Distributed Systems For Fun and Profit book is also an excellent introduction and overview.

I also listened to many old episodes of the Software Engineering Radio podcast and the Software Engineering Daily podcast, which often cover similar topics.

Performance

Even as a regular C and C++ coder, I realized how little I thought about the computer architecture that my code is running on. Various talks by Andrei Alexandrescu and Chandler Carruth really opened my eyes to these performance considerations that seem more relevant in at-scale server code than they were in typical desktop applications or even in embedded code.

For instance, Andrei Alexandrescu’s Code Dive 2015 “Writing Fast Code: part 1” and part 2 talks, as well as his “Optimization Tips – Mo’ Hustle Mo’ Problems” talk at CppCon 2014 and his “Writing Quick Code in C++, Quickly” talk from Going Native 2013.

For instance, Chandler Carruth’s “Understanding Compiler Optimization” talk from Meeting C++ 2015, his “Tuning C++: Benchmarks, and CPUs and Compilers! Oh My!” talk from CppCon 2015, his “Efficiency with Algorithms, Performance with Data Structures” talk from CppCon 2014, and his “Care and Feeding of C++’s Dragons” talk from Going Native 2013. Also his more recent “High Performance Code 201: Hybid Data Structures” from CppCon 2016.

Mike Acton’s famous “Data-Oriented Design and C++” talk, from CppCon 2014, really gets to the point about this too, from more of a C perspective.

Coding Habits

I finally took the time to learn Vim enough to use it as my daily editor, after reading Practical Vim. I also learned to use screen and Tux but I haven’t incorporated either into my daily habits.

Ironically I never found my Linux environment outside of the terminal to be as keyboard friendly as the old pre-X MacOS. So staying in the terminal helped me to keep my hands off the mouse again. I practiced with the Klavaro typing tutor to get some of my good habits back.

I worked through the first 25 tasks from Project Euler and most of the tasks from HackerRank. I spent two intensive months working through every task on InterviewBit (around 260 at the time).

I highly recommend InterviewBit – they have a vast set of coding problems, nicely organised, with helpful test cases and tips for when you get stuck, so you can actually improve. HackerRank is useful too, but it feels less directed and the need to parse stdin input for every task gets annoying.

Along the way, I took a note of each C++ typo or generic algorithm mistake that I made, and started counting how many times I made each mistake, so I could focus on breaking my worst habits.

Management

I read some books about project and people management too, revising agile processes with the Scrum and XP from the Trenches and Kanban and Scrum: Making the most of both free ebooks.

The Manager’s Path maps out the various roles that are popular at software companies today, with suggestions about regular actions that each might perform, but it doesn’t seem useful beyond establishing that common language, and I hope it becomes outdated.

I recently read Measure What Matters, and Radical Focus, about OKRs, and I’m very enthusiastic so far. I like the idea of openness and regularity in businesses, to allow focus and alignment, almost like well-functioning decentralized open source projects that I’ve known.

I also got interested in product management, but I haven’t taken it far yet. The Cracking the Product Manager Interview book was rather superficial. I hope to dig deeper into this topic in future.

Others

I revised my Design Patterns knowledge by reading Head First Design Patterns and I used it to build a Design Patterns quiz.

I learned the Lua programming language and the Moai game-development platform via the Programming in Lua book and the “Developing Mobile Games with Moai SDK” book. This was for an idea I’d had for a kids’ programming app that didn’t seem quite worth the effort in the end.

I learned a bit about Hadoop by reading the “definitive guide” book, but I never got around to using it.

C++17 in libsigc++ : invoke, apply, and constexpr if

General, , , ,

I’ve made a fairly minor 2.99.11 release of the unstable libsigc++, just to make use of C++17. Here are some brief descriptions of the C++17 features used in libsigc++ so far:

std::invoke()

std::invoke() invokes a callable object, such as a plain function pointer, pointer to a member method, or a functor, such as a sigc::slot, using one generic syntax. We provide the arguments, and possibly an object intance, in the std::invoke() call. For instance:

  • std::invoke(function, 1, 2);
  • std::invoke(method, object, 1, 2);
  • std::invoke(functor, 1, 2);

By using std::invoke in libsigc++ (and here), I hope that this makes the implementation more generic though it’s not immediately obvious how.

std::apply()

std::apply() is a little like std::invoke(), but lets us invoke a callable object with the arguments in a tuple. For instance:

  • std::apply(function, std::make_tuple(1, 2));
  • std::apply(method, std::make_tuple(object, 1, 2)); // I think.
  • std::apply(functor, std::make_tuple(1, 2));

This is useful for libsigc++ template code that uses tuples to concatenate or rearrange sets of variadic parameters. It lets us replace several slightly less generic helper functions that just called a  functor with a tuple unpacked as variadic arguments via a provided std::index_sequence. Receiving variadic parameters into a tuple is normal, so this complements that well, letting us use some of those parameters in calls to other functions.

constexpr if

I also used constexpr if to avoid the need for a helper template. Before C++17, templated code could sometimes need to use another helper template to perform different operations depending on the type, sometimes where some operations won’t even compile with the other type. The different operations would then be isolated in different template specializations of the helper template, and the main template could just have one generic call to that helper template.

But constexpr if lets you put it all in the main template. For instance, it let us remove the little with_type<> template, which only existed so we could call a functor for some types, but not for others.

I also updated the tuple-utils from my murrayc-tuple-utils project, including several  uses of constexpr (I also used the new C++17 nested namespace syntax when doing that).

libsigc++ has moved to GitHub

General, , ,

We have moved libsigc++ from git.gnome.org to github.com. This should make working on it even nicer.

I’ve also moved its website to GitHub Pages rather than Sourceforge, now using the simple Jekyll templating system. But I’ve made only minimal changes, so I’d be glad if someone submitted a pull request to make the website better and prettier.

We probably should have done this sooner, because libsigc++ is in no way GNOME-specific, but GNOME’s git has been very good to us over the years and it’s been hard to trust anything else after Sourceforge fell apart. However, with GNOME’s gradual move to gitlab.gnome.org (which gtkmm will happily use) we would have to change something anyway. I am a happy (paying) GitHub user and it feels like a natural home.

I’m still rather proud of my modern C++ rewrite of libsigc++ but I’m sure it could benefit from more use than just by gtkmm 4, which is itself not used much yet. Maybe this move to GitHub will help with that.

gtkmm 4 started

General, , , , , ,

We (the gtkmm developers) have started work on an ABI-breaking gtkmm-4.0, as well as an ABI-breaking glibmm, target GTK+ 4, and letting us clean up some cruft that has gathered over the years. These install in parallel with the existing gtkmm-3.0 and glibmm-2.4 APIs/ABIs.

A couple of days ago I released first versions of glibmm (2.51.1) and gtkmm (3.89.1), as well as accompanying pangomm and atkmm releases.

This also lets us use my rewrite of libsigc++ for libsigc++-3.0, bringing us more fully into the world of “modern C++”. We might use this opportunity to make other fundamental changes, so now is the time to make suggestions.

We did a parallel-installing ABI-breaking glibmm even though glib isn’t doing that. That’s because, now that GTK+ is forcing us to do it for gtkmm, this seems like as good a time as any to do it for glibmm too. It’s generally harder to maintain C++ ABIs than C ABIs, largely because C++ is just more complicated and its types are more exactly specified. For instance we’d never have been able to switch to libsigc++-3.0 without breaking ABI.

 

C++17: constexpr if

General, ,

Yesterday I played with constexpr if a little, using clang++ built from the latest LLVM sources. C++17 will not have many of the features I was looking forward to, but this is a really nice convenience.

It lets us clean up some of those little templates-plus-specializations that only exist to execute different code in a parent template depending on the template’s type, particularly when the specialization is on a bool or integer value. It won’t help if the template specializations need different member variables or need to provide different type aliases or typedefs.

For instance, libsigc++-3.0 still had a little with_type template with partial template specializations for true and false. Replacing it with constexpr if makes the intention clearer.

I made many similar simplifications by using constexpr if in my murrayc-tuple-utils, which is used by libsigc++-3.0 via copy/paste/rename.

In my murrayc-dp-algorithms project, some code became much clearer and more efficient with constexpr if, allowing me to remove some template specializations that only existed to satisfy the compiler.

libsigc++ 2.99.5: Even less code

General, , ,

I managed to rip out another chunk of slightly-incomprehensible code from libsigc++, and those improvements are in libsigc++-3.0’s 2.99.5 version.

Now there’s no common functor_base base class and I removed all those interdependent result_type typedefs that were scattered throughout the code. When I had unravelled the whole result_type chain, I could replace it with just a handful of decltype(auto) uses, realising that that’s what the whole mess was trying to achieve. I love decltype(auto).

Trying CMake Again

General, , , , , ,

I’ve recently been playing with CMake again, with much more success than last time. I still don’t love it, but:

I first tried it in the cmake branch of my little PrefixSuffix application because its build is rather simple.

I then finally allowed libsigc++ to have an additional CMake build system, at least in libsigc++-3.0, and Marcin Kolny got it done. This was possible now that libsigc++ doesn’t generate code from .m4 files. We wouldn’t have wanted to maintain two complicated build systems, but two simple build systems seems acceptable. I noticed many questions on StackExchange about actually using libsigc++ in CMake builds systems, so I added a hint about that to the documentation.

I also added a CMake build system to my Glom source code, to test something bigger, with many awkward dependencies such as Python, Boost, and PostgreSQL. It’s working well now, though I haven’t yet added all the tests to the build. This let me really try out CLion.

Minor Conclusions

I’m fairly pleased with the result. CMake doesn’t now feel massively more annoying than autotools, though it makes no attempt to do the dist and distcheck that I really need. I have now achieved a level of acceptance that means I wouldn’t mind much working on a project that uses CMake. I wouldn’t even need to hold my nose. The following complaints don’t seem to bother me quite so much any more:

I still find invoking CMake horribly obscure compared to ./configure. For instance

cmake -DCMAKE_INSTALL_PREFIX:PATH=/opt/something

seems ugly compared to

./configure --prefix=/opt/gnome

and I don’t see an equivalent for “./configure –help” to list all generic and project-specific build options. I also haven’t tried to replace mm-commons‘s wonderful  –enable-warnings=fatal to easily turn on compiler warnings as errors, encouraging my code to be free of warnings and deprecated API while not troubling people who just want to build the project with the least difficulties.

I still cannot understand how it’s normal in the CMake world to create these awful little Find*() scripts for each library that you might depend on, instead of just using pkg-config, though CMake’s pkg-config support seems to work well now. And hoping that your own library’s Find*() script ends up in CMake itself does not seem scalable.

It’s also a little odd that I cannot specify the link directories (-L flags) per target. link_directories() seems to apply to all targets.

CMake’s syntax still seems like a toy language. Its quoting (none/implicit/unknown) and separating (spaces, not commas) and use of whitespace (occasionally significant) sometimes makes m4 feel almost friendly. The non-case-sensitivity of function names (but case-sensitivity of variable names) just leads to arbitrarily inconsistent code and unnecessary style differences between projects.

I don’t like the CMake tradition of having separate CMakeList.txt files in sub-directories, because this can lead to duplication and makes it harder to see the whole build system at once. I prefer non-recursive autotools, so I tried to use just one CMakeList.txt for my CMake projects too, which was only a little awkward.

libsigc++: std::function-style syntax.

General, , , , ,

Yesterday I released version 2.99.2 of libsigc++-3.0. This changes the declaration syntax for sigc::slot and sigc::signal to use the same style as std::function, which was added in C++11. We don’t want to be arbitrarily and unnecessarily different.

We’ve also simplified sigc::mem_fun() to always take a reference, instead of either a reference or a pointer.

So, where you might do this for libsigc++-2.0:

sigc::slot<void, int, A> slot = sigc::mem_fun(this, &Thing::on_something);

signal<void, int, A> m_signal;

You would now do this for libsigc++-3.0.

sigc::slot<void(int, A)> slot = sigc::mem_fun(*this, &Thing::on_something);

signal<void(int, A)> m_signal;

Actually, as of version 2.9.1 of libsigc++-2.0, you can use both syntaxes with libsigc++-2.0, allowing you some time to adapt to the new API before one day moving to libsigc++-3.0.

libsigc++ 3.0: Very variadic

General, , , ,

I have just released libsigc++ 2.99.1, the first release of the libsigc++-3.0 API, which installs in parallel with the existing libsigc++-2.0 API. This is libsigc++ using much more modern C++, specifically C++14. The API itself is almost completely unchanged, but the implementation is now clearer and easier to contribute to. I’m rather proud of my work, but I’m quite sure that there’s still room for improvement.

Unfortunately, glibmm and gtkmm can’t use libsigc++-3.0 until they have their own parallel-installable versions. That’s not likely to happen until the mythical GTK+ 4 happens. Even though libsigc++ is mostly all templates in headers, its symbols do appear in the linker symbols for gtkmm method signatures, so changes to libsigc++ can break gtkmm ABI. I do have a sigc3 branch of glibmm in git just to show that it works.

With around 150 commits, I gradually refactored libsigc++ to use variadic templates with C++14 instead of generating code from nasty m4 files. Over the years, we have built up quite a large set of regression tests, run during “make check” and “make distcheck”, including various corner cases. Without these tests, the refactoring would have been much harder. With the tests, I could make small commits, knowing that each commit had not broken anything. When something did seem wrong, I could add a test and go back through the git history to find the problem, sometimes splitting commits into even smaller changes. I did a lot of that, rebasing several times, sometimes stopping and starting again after help from Jonathan Wakeley and Marcin Kolny. Those tests give me much more confidence in the end result than I could have if I had chosen to just reimplement the entire API from scratch.

The API is almost all .h files and according to wc, there are 24,145 lines of code in those files in libsigc++ 2.7.1 (after make), and  6,507 in libsigc++ 2.99.1. So there is now only 27% as much code.

This is possible because:

  • C++ variadic templates allow us to have one class or function where we previously had to generate multiple versions for 1 to 6 function parameters, sometimes with additional versions for const and non-const parameters or const and non-const (and volatile and non-volatile) member function pointers.
  • decltype(auto) lets us avoid lots of templated type traits just to correctly specify the correct type for methods.
  • The standard C++ type traits, such as std::conditional<>, std::result_of<>, std::is_base_of<>, std::remove_volatile<> and std::is_const<> let us write very generic code, sometimes replacing our own type traits. I added some more to libsigc++ to get compile-time type traits for member method pointers.
  • template aliases (like typedefs for templates) avoided the need for multiple functor classes deriving from a common base, even though I ended up not needing most of these aliases either.
  • I replaced our sigc::ref() and sigc::reference_wrapper() with std::ref() and std::reference_wrapper<>. Presumably these share a common history.
  • I removed some configure checks and ifdef-ed workarounds for older MSVC++ and Sun Forte compilers. Hopefully they aren’t necessary now, but we will see.

For some adaptors  I used the tuple utilities that I’ve been working on recently, for instance in sigc::bind(). These are copied into the libsigc++ source code, and I’d particularly welcome improvements to them in the form of patches or github pull requests.

I’m still not completely happy with all the overloads we have for sigc::mem_fun(), to take member functions that are non-const, const, non-volatile and/or volatile, but I have some things still to try. We might also remove several by not allowing both mem_fun(pointer, func) and mem_fun(reference, func).

Please do suggest ways to simplify the code yet further.