Shiny rails and testing

Man, Rails kicks ass.

After getting a hold of Ruby a few months ago, I've been looking deeply into Rails. There are a couple things at work that use it, and I'll probably wind up maintaining them if we don't replace them altogether.

There's little point in me talking about how great Rails is, since you can find Rails worship sites just by typing random letters into a google search, but I am going to emphasize the thing that really kicks unseemly amounts of ass -- the one thing that no one else has done nearly as well: testing.

Rails does first-class support for model (unit) testing and controller (functional) testing better than any other framework I know, including the one I support professionally. While unit tests have been around forever, there hasn't been a concerted push to get a comprehensive functional test strategy in Java. Part of the problem, of course, is that Java has more web frameworks than France has baguette shops, and several of them seem to make a point of being as different as possible from any other framework. Rails, being an all-in-one solution for your web needs, can offer integrated tests easily, but for the Java offerings you more or less have to reinvent the testing axle every time you reinvent the wheel.

Case in point: Our framework includes Struts 1.3.8, which for purposes of this discussion means we support it and give people a number to call if it breaks. We also ship a sample application that uses our framework, and I volunteered to maintain/update/rewrite it for the next release. The guy who did the first version is no longer with the company, and he never bothered to write unit tests for his models nor functional tests for his controllers. Now it's no indictment of the technology if the developer is too lazy to code properly, but after filling in the missing tests I got to Struts. I'm not willing to ship an app without all necessary test coverage, and controller routing and request processing falls within that box. (If you disagree with this, you are wrong.) So I poked around for some test help and came across StrutsTestCase, which seemed to be just what the doctor ordered. Unfortunately, it's apparently been abandoned and the latest release is three years old, not to mention it doesn't even compile under Java 5. So I've been working on something similar for our needs.

All told, the googling/studying/problem solving/coding that I'll have done by the time this is finished will add up to a decent chunk of effort. I remind those reading that Struts 1 is still the most widely used MVC Java web framework, so one would expect to find solid testing support for it. Well, you won't, and there isn't.

The comparison to Rails, where a test case for your controller can be generated with a simple script and the entire test written and done in minutes, reflects badly on the Java world. I can only conclude that Struts shops write their action mappings and then laboriously, manually test each one through a running browser. In addition to being inefficient, slow, and boring, it takes forever, which is another way of saying that it won't be done. This is the kind of thing the computer should be doing for you, people. This isn't a Java vs. Ruby issue, and it's not even an issue particular to Struts; there is a resistance to spending code effort to automate tests that can and should be automated.

We need to make managers aware of the need for automated functional testing so time for it gets built into schedules. We need to improve our testing tools so they'll be ready to handle new technologies. We must spend our time working on difficult problems, not fighting the machine.

Guess I'll start by writing a test library.

Getting religion: Testing

I've spent the past week at a Spring training session -- of which more later -- but the first thing I want to say has to do with testing.

We have maybe 20 people in here; most are on ancient Java tech, including 1.3 and JDBC, and Spring of course sounds compelling to them as it hits right where they need the most help. The subject of transactions came up, and naturally our instructors pushed hard on us that 1) they were necessary; 2) Spring made it easy to do them; 3) we should be doing it. Now I was prepared for 2 and 3, but I thought 1 was unnecessary. Of COURSE you do transactions when you're doing database development. All kinds of crap can go wrong if your operations aren't atomic by logical unit rather than by connection. Now while I wasn't surprised to hear people say transactions were a pain to implement in Java, I was shocked to hear them say that because of that, they didn't bother with them -- or even worse, said they were unnecessary. One guy claimed to know his database "well enough" that the kind of conflict described as arising out of a failed transaction "shouldn't be allowed to happen".

I had to bite my tongue to keep from laying into this guy.

So the instructor followed up with, "Well, how do you know you aren't having any problems with your code due to not using transactions?" "Well, we haven't seen any problems and our users haven't reported any." Um, genius, if your web site doesn't work for users for no apparent reason, they're not going to use it. They're not going to compose a detailed error report and help you track down the bug; that's not their job. The web is not an expanded testing department. It's not my mom's job to help you find the bugs in your site. If a user can't use your site, they'll just leave it and not come back.

Some rant-ish points:

• I don't care how well you think you know the database you're using; you're wrong. You don't know it completely; no one person does. No one person even wrote it, so it's ridiculous to say you understand it completely. Second, it's not just the database you need to worry about; it's also the JDBC driver, the VM, the native platform, the versions of each, and a hundred other factors. It is way bigger than you, and it's time to humble yourself to that; the days when a programmer understood the machine completely are long gone and aren't ever coming back.
• One day your database may change. I guarantee you won't understand it nearly as well as you do your current one, and that's incomplete at best.
• It is not okay to write some code, run it through a couple of use cases, fix the obvious errors, and declare it production-ready. Your job is not to show that your code might work given the right conditions; you must show that your code cannot break in the wrong conditions.
• Your test cases (the programmer's, I mean, not the QA department's) must test not only for correct results but for sensible and well-defined behavior when giving incorrect input. It doesn't just have to work; it also has to not break.
• Treat your users with respect. They aren't programmers, and they don't get paid to use your code. You get paid to give your users an easy, non-confusing experience. Don't try to lazy your way out of it by saying that bulletproof code isn't worth your time -- you have no other function but to produce that bulletproof code.

Remember this at all times:

This is computer science, not computer religion; we don't have faith that things work, we have proof that they do.

One of the bedrocks of scientific theory is the idea of falsification; a scientific theory must be able to disprove statements that are false. (This is why creationism and intelligent design aren't science, since they can't meet this standard.) In this case, a falsifiable statement would be something like: If this operation is interrupted by the user, the database update will fail. You then write a unit test that creates such an interruption and then check if the update failed (either an exception was thrown or the data violates some integrity constraint).

A unit test is really a small proof that a falsifiable statement is in fact false; when you assert something at the end of a test, you are basically claiming that the lines of your code are a list of statements that logically demonstrates the truth (or falsity) of your falsifiable statement. This is exactly the same thing as a geometric proof, and that's not a coincidence. A good unit test only exercises one part of your class, just as a good proof only tries to establish one fact.

You know that your code is ready when it has been proven to not fail. The trick is figuring out all the ways your code might fail, and that's impossible except for the most trivial programs. However, you should be able to predict the vast majority of failure scenarios and prove through unit and integration testing that they will not break your code. If you're not doing that, you're not practicing computer science.

It's time to get religion about computer science.