Over the last few months I have developed the habit of writing tests when fixing bugs in Glom. These run during “make check” or “make distcheck” so I can be sure that the bugs have not come back. (I know these tests should be smaller and even more numerous, but I can’t yet see a sensible way to break them up).
I had to do lots of build-system and code work to make this possible, because some code had to be moved out of the UI code, and many tests depend on a temporary local server instance. But now, it’s easy to add new tests and it feels so good.
When the tests succeed I have a feeling of confidence in the code, instead of a feeling of uncertainty. When they fail I feel glad that I had a chance to fix a regression before releasing a new version. I am surprised at how much more enjoyable this has made coding.
Preach the testing love, Brother Cumming!
Plus, when your tests reach a certain point, you feel freer to make larger changes without fear of regressions, because you can expect that any failures should have been caught by your tests. I certainly would be much more hesitant in fixing SpiderMonkey bugs if I weren’t backed up by thousands and thousands of regression tests.
How did you check that it worked before you had tests?
Steve, by using the UI and relying on testers of the unstable release versions. I had a lot of regressions.
I really want to test the UI too, but LDTP is very hard to get working.
If you can motivate people to manually run formalized UI tests, Ubuntu’s checkbox might be useful for you: It gives testers steps to perform with the UI and asks for the observed result. Simple but effective, if you find volunteers.
Unit tests are absolutely essential. Obviously for refactoring. More importantly for extending code someone else wrote: You never find all corner cases the original author had in mind documented. Last not least for testing your theory about what code you wrote. Shipping code that didn’t pass an extensive test suite is irresponsible, IMHO.
Full ack; I can’t live without regression tests any more, it feels like walking on a rope over a canyon without securing yourself :-) In most of my projects I have a relatively strict policy of always reproducing a bug or testing a new feature in a test first. This is not much harder to do than reproducing it ad-hoc, and will have an ever-lasting effect.
Wrt. UI tests: Last week I added some new GTK GUI tests for Apport (http://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/test/test_ui_gtk.py). This uses mybutton.clicked() and some idle_add()/timeout_add() hacks to simulate events; this is of course nothing fancy, but magnitudes easier to use than LDTP or mago, and for small things I find it quite useful. While it’s certainly some more effort in C++ than it is in Python, the technique should apply just as well. Maybe it helps you for some UI tests as well?