Shameless plug

I recently managed to maneuver Commons Email 1.1 through the release gauntlet, and since it's the first thing at Apache that I've taken all the way to the finish, I feel the need to announce it here. P.R. is best when it comes from completely compromised sources.

If you've ever wanted to do anything with email in Java, you owe it to yourselves to check it out and see if it helps. Bugs, feature requests, angry letters, and invitations to box socials are all welcome.

Open Source FTW!

Wow, I write one little blog and the heavy hitters take notice. First, thanks to all who read and commented. I'll try not to let success go to my head. Second, there are a lot of touchy political things going on in this discussion, and while I don't wish to offend anyone, being honest about my beliefs and my practices is bound to piss someone off. To that person I say "sorry, but I'm famous now". :)

Seriously, there are a couple of things I want to look at.

From Rod's blog, where he paraphrases my support company point into an incorrect metaphor:

Rod claims that dependable car repairs requires trained mechanics. This must come as a surprise to all the non-mechanics employed by every other garage in the world. Someone has to do the sales and run the business and fix the building and keep the website updated and find new customers, etc., etc. Mechanics don't get you any of these, and who's going to tell me they're not required for a successful garage?

As Rod surely knows, car mechanics almost never design or build the cars they are paid to repair. A very small group of designers produce almost all the automobile designs used in the world. The guy at your local Fix-It-Cheap Auto Service doesn't work for Toyota and probably never has, but (assuming he's a competent mechanic) you don't insist that he has to before you trust him to fix your car; he's read the manuals, studied the engine specs, read the latest bulletins and recall notices, and worked on dozens of cars very similar to yours. He fixes your car for a hopefully reasonable price and you're out and happy.

Rod, Juergen, and the other i21 engineers are like the car designers. They decide how Spring should be architected, and then they write code that implements the design they've settled on. While they certainly can support Spring, there's no reason some other engineer (me) can't study the same code and achieve the same level of expertise as far as a customer's support needs require. In my experience, a customer wants their immediate needs met when they open a support case. If we help them fix their problem, they're happy, and if we educate them on how to avoid similar problems in the future, they're very happy. Working with the community to find the best long-term solution that works the best for all Spring users is the second problem and one any good developer is happy to help with.

One thing i21 might look into is having a certified Spring technician program backed up by a rigorous curriculum that makes that really mean something. Toyota certifies mechanics on very strict tests. Cisco, Microsoft, Red Hat, etc., all have similar programs. The companies make money on course and training fees and crank out qualified engineers that spread the mindshare into every corner of every company they serve.

Rod speaks out against companies who, he says, try to make money off open source without fairly compensating the original developers of that code. SourceLabs is not such a company, as our active participation, including committer access, to several projects will show. By implying that these companies are parasites, he makes the fallacious implication that the original developers are always owed a cut of revenue derived from the code. If that's not what he means, it's hard for me to suss out any other interpretation of his statements on this issue. I strongly disagree with this opinion; a true open source project (the FSF term "free software" might even be applicable here) does not belong to any one person or entity, but rather to the community of its users. Rod seems to think that the purpose of Spring is to enrich Interface21; I believe the purpose of Spring is to make developers' lives easier. I make this claim because Spring is licensed under the Apache Software License, which explicitly allows free distribution and even (urk) proprietary forking. If Spring does not truly belong to all of us, then it is mislicensed. If Interface21 honestly feels that Spring belongs to them, they should change the license to reflect that instead of misleadingly claiming to be an open source company.

Finally, Rod claims that SourceLabs has not committed anything back to Spring. I could make the admittedly weak argument that I've personally sent two little tiny patches to the build file which were both accepted, but Rod has also said that valuable contributions to Spring include evangelizing and helping to broaden its community. We have been doing this since the company's inception. I've personally given one of my canned "Spring rocks" speeches to customers, and I've also encouraged other people I know in my professional sphere to use it. Such contributions are valuable not just because Rod says they are but also because they make life easier for everybody using the Java platform, whether they're our customers, i21's, or nobody's.

I'd like to close by offering my help in a professional capacity as a Spring enthusiast with time on his hands to Rod and Interface21. It is explicitly part of my job description to improve Spring however possible. Guys, where can I help? I'm happy to assist with anything you'd like assistance with: code, test cases, documentation, even giving speeches or talks at conferences and venues I visit. My offer is not contingent even on future employment at SourceLabs; I'm volunteering as an open source developer. Have Juergen (or whoever handles this sort of thing for you) drop me a line. I don't want committer access unless and until you've decided I earn it. I don't insist you adopt my patches. I don't insist on helping design the car. But I will help you however you'd like to make Spring better with dedicated time each and every working week.

So how about it? Would you like the help? Let me know.

Nonsense about Interface21

Rod Johnson, a man I respect and admire, has managed to step on one of my hot buttons. He's put up a new blog entry that reiterates his earlier complaint against other companies that offer support or services for Spring. While I agree with some of them, his brush is far too broad for the truth.

Before I start, full disclosure: I work for SourceLabs, a company that offers 24/7 enterprise-level support for Spring as part of a fully integrated and certified Java enterprise software stack. I'm proud of our products and how we've helped our customers.

Rod starts off with this:

OpenLogic needs to understand that the opening comment in Stormy's post that "Developers that work on open source software typically have day jobs that pay pretty well…so they work on open source software for free and write code during the day for big bucks" is largely wrong, understand where the open source software they hope to profit from comes from, partner appropriately, and set a price point that allows for genuine support. An alternative would be to stop claiming to provide enterprise support, and be clear that what is being offered is a kind of on-call development assistance, with no guarantee of being able to resolve critical issues.

Actually, Stormy's not largely wrong. Companies who sponsor open source by hiring its developers are neither a necessary nor sufficient condition for a successful project. The obvious example is Linux, which rapidly evolved into a stable and usable kernel long before the words Red Hat or SUSE entered the conversation. Back then, all Linux developers were far-flung, spare-time volunteers who didn't receive a dime for their work. When the first Linux companies decided to try offering support, there wasn't a "Linux21" company headed by Linus they could negotiate with, and Linus never claimed that he was the way, the truth and the life of Linux support.

Jesus may have said that no one comes to the Father without coming through me, but Linus explicitly disavowed control of the Linux community at a very early stage. He maintains his position through the universal respect of his judgment and that he has demonstrated that he is beholden to no one. Whether he succeeds or fails, he tries to do the best thing for Linux, not whatever company he's working for. Red Hat's Linux support is not somehow delegitimized for lacking an explicit Linus Stamp Of Approval, and neither is Canonical's, Novell's, SUSE's, etc. Plus, aren't those companies making money just fine, and isn't the competition between them good for all Linux users, individual and corporate? One might also mention XenSource, who never claimed to be the only legitimate source of Xen support, and yet recently sold for hundreds of millions of dollars. There are tons of consultants who are expert in Ant, Maven, Zimbra, JSF, etc., who may compete for contracts, but none of them can seriously claim to be the only legitimate option over the others.

Rod claims that dependable support requires committer access to the source code. This must come as a surprise to all the non-engineers employed by every other support company in the world. Someone has to do the sales and run the business and fix the servers and keep the website updated and find new customers, etc., etc. Committer access doesn't get you any of these, and who's going to tell me they're not required for a successful support company?

Good support means that you have a personal relationship with your customer. You know what they need, what other software they're using, what they're most concerned about, and you establish a bond of trust with them that you will be there for them. That goes way beyond giving them a pager number and saying, "Call me if it breaks".

Despite what Rod would have you believe, I (or anybody else) can offer top-quality Spring support thanks to the ASF license Spring uses. You don't need to be a Spring committer to fix bugs in it, and you don't need Interface21's permission or endorsement to use it. If there's a bug that impacts your production server, I can find it in the code myself, talk to the customer, and figure out what the best solution is: patch, workaround, client code change, etc. I can do that because I have access to the source code that matters -- the one the customer is using -- and because I understand, through careful and tedious preparation, what the customer needs.

Moving on, we next have this:

It may seem strange to you, but it's normal for individuals and companies to want a return on investing millions of dollars, passion, blood sweat and tears in something. Interface21 sustains and develops Spring and we do a good job. I think it's perfectly reasonable to expect that we can leverage this investment into an advantage in the support market.

Sure, Rod. You want to sell support for Spring, be my guest. Play up every advantage you can in the market, but it is a market. If you want to argue that control over committers makes you the best support option, go ahead, but as I said above, it's a pretty weak argument.

Authors and thousands of these developers contribute to the success of the project and community through their evangelism. That's an important way of contributing. OpenLogic does not; it merely aims to make money from projects that others have built, evangelized and conveniently maintain and enhance. Now that's a "sweat deal".

OpenLogic may not contribute to the projects they use, but at SourceLabs we do. Our credibility as an open source company depends on it. I'm a committer on the Apache Commons project. We have people in house who have committer access on Struts, Hibernate, and Maven, and we're also involved daily in Tomcat, Axis, and other projects. We have brought Spring into companies because we like working with it, thus increasing mindshare and support for Spring without anyone at Interface21 knowing anything about it or lifting one finger.

No one at Interface21 can seriously claim to be actively engaged with the other pieces of a full Java enterprise stack; Spring, for all its awesomeness, is a framework, not a stack. It doesn't provide everything an enterprise customer needs, and supporting only Spring is not going to cut it with companies who also need help with Hibernate or Struts. In fact, Rod has to say that if you do need help with the other parts of your Java solution, Interface21 can't help you since they don't control the source code of those projects.

It's true that aggregation in and of itself is not worth much. But neither is control of the code. Open source allows everyone to compete on a level ground in whatever arena seems valuable enough to conquer. I hope Interface21 isn't just afraid of a little competition from companies who understand that doing support right is much more complex than having an SVN account.

One final point for Rod: why did you open source Spring at all? If you're so convinced that no one else can offer credible support for it, why not just make it proprietary?

Shiny rails and testing

Man, Rails kicks ass.

After getting a hold of Ruby a few months ago, I've been looking deeply into Rails. There are a couple things at work that use it, and I'll probably wind up maintaining them if we don't replace them altogether.

There's little point in me talking about how great Rails is, since you can find Rails worship sites just by typing random letters into a google search, but I am going to emphasize the thing that really kicks unseemly amounts of ass -- the one thing that no one else has done nearly as well: testing.

Rails does first-class support for model (unit) testing and controller (functional) testing better than any other framework I know, including the one I support professionally. While unit tests have been around forever, there hasn't been a concerted push to get a comprehensive functional test strategy in Java. Part of the problem, of course, is that Java has more web frameworks than France has baguette shops, and several of them seem to make a point of being as different as possible from any other framework. Rails, being an all-in-one solution for your web needs, can offer integrated tests easily, but for the Java offerings you more or less have to reinvent the testing axle every time you reinvent the wheel.

Case in point: Our framework includes Struts 1.3.8, which for purposes of this discussion means we support it and give people a number to call if it breaks. We also ship a sample application that uses our framework, and I volunteered to maintain/update/rewrite it for the next release. The guy who did the first version is no longer with the company, and he never bothered to write unit tests for his models nor functional tests for his controllers. Now it's no indictment of the technology if the developer is too lazy to code properly, but after filling in the missing tests I got to Struts. I'm not willing to ship an app without all necessary test coverage, and controller routing and request processing falls within that box. (If you disagree with this, you are wrong.) So I poked around for some test help and came across StrutsTestCase, which seemed to be just what the doctor ordered. Unfortunately, it's apparently been abandoned and the latest release is three years old, not to mention it doesn't even compile under Java 5. So I've been working on something similar for our needs.

All told, the googling/studying/problem solving/coding that I'll have done by the time this is finished will add up to a decent chunk of effort. I remind those reading that Struts 1 is still the most widely used MVC Java web framework, so one would expect to find solid testing support for it. Well, you won't, and there isn't.

The comparison to Rails, where a test case for your controller can be generated with a simple script and the entire test written and done in minutes, reflects badly on the Java world. I can only conclude that Struts shops write their action mappings and then laboriously, manually test each one through a running browser. In addition to being inefficient, slow, and boring, it takes forever, which is another way of saying that it won't be done. This is the kind of thing the computer should be doing for you, people. This isn't a Java vs. Ruby issue, and it's not even an issue particular to Struts; there is a resistance to spending code effort to automate tests that can and should be automated.

We need to make managers aware of the need for automated functional testing so time for it gets built into schedules. We need to improve our testing tools so they'll be ready to handle new technologies. We must spend our time working on difficult problems, not fighting the machine.

Guess I'll start by writing a test library.

Getting religion: XML

XML is out of control.

The insidious little creature has ingratiated itself with the Java standards committees, which themselves were long ago bought and sold to the unholy behemoths that stuff the latest JEE turd down our throats. Under their tutelage, XML has become the configuration file format of the 21st century; we can't get away from it in Java development. The two major build systems use it. The entire JEE spec is predicated on it. Spring, which has always been billed as the solution to the mistakes of JEE, prides itself on its XML configuration. Hibernate, which despite its flaws is worlds better than JDBC access, requires us to do our ORM in glorious, repetitive XML.

This has got to stop.

XML is a markup language; that means it is supposed to contain human-understandable text and information about that text. It was designed to be flexible to ensure adapting to arbitrary formats is easy. Whichever side you're on of the OOXML/Office XML debate you're on, the fact that the document is represented in XML is a win for all developers.

The flexibility, which has been the key to XML's wild success, has also been contorted by eager Java beavers to twist it into a general-purpose configuration language. Now it's debatable whether an XML representation of, say, a properties file is any worse than a simple key/value listing, but I would argue that at least it's not worse. However, when you start mapping database table schemas to XML, inserting namespaces for different kinds of constructs, and attempting to integrate those configurations with other programs, you wind up in a nasty place quick. On top of that, some people have even begun to hack procedural logic into XML (see the antcontrib tasks and the JSP tag library). Suddenly your XML has become a crappy approximation of a programming language. At this point, why aren't you better off using a programming language? As the XML gets more and more hairy, the parser grows similarly hairy -- just so you can map your XML into Java. But why are you trying so hard to keep your configuration in XML anyway? So it can be portable? (What other app is going to use Spring's application-context.xml?) So other languages can read it? (Ruby doesn't need a Hibernate XML file for ActiveRecord and never will.) Even if you do think of a good answer to that question, is it worth the ugly creeping horror that is your configuration parser?

If you need a programming language, don't be afraid to use one.

All right, so maybe Hibernate can't read the database for the table metadata for some reason (though I'm still not convinced that it shouldn't at least try). What's wrong with using Java to describe the schema instead of XML? It's not like it's not going to wind up in Java anyway, and the extra level of indirection doesn't buy you anything. At least use Java (or Ruby, or something useful) to generate the XML if you insist on having it; at this point, reconfiguration becomes the same as refactoring, and a good Java IDE is immensely helpful with that.

(Thankfully, Spring has finally started work on a Java configuration option, and it is tasty.)

Bottom line, guys and gals: the right tool for the right job. XML is not always the right tool, so don't use it when it isn't.

Getting religion: Testing

I've spent the past week at a Spring training session -- of which more later -- but the first thing I want to say has to do with testing.

We have maybe 20 people in here; most are on ancient Java tech, including 1.3 and JDBC, and Spring of course sounds compelling to them as it hits right where they need the most help. The subject of transactions came up, and naturally our instructors pushed hard on us that 1) they were necessary; 2) Spring made it easy to do them; 3) we should be doing it. Now I was prepared for 2 and 3, but I thought 1 was unnecessary. Of COURSE you do transactions when you're doing database development. All kinds of crap can go wrong if your operations aren't atomic by logical unit rather than by connection. Now while I wasn't surprised to hear people say transactions were a pain to implement in Java, I was shocked to hear them say that because of that, they didn't bother with them -- or even worse, said they were unnecessary. One guy claimed to know his database "well enough" that the kind of conflict described as arising out of a failed transaction "shouldn't be allowed to happen".

I had to bite my tongue to keep from laying into this guy.

So the instructor followed up with, "Well, how do you know you aren't having any problems with your code due to not using transactions?" "Well, we haven't seen any problems and our users haven't reported any." Um, genius, if your web site doesn't work for users for no apparent reason, they're not going to use it. They're not going to compose a detailed error report and help you track down the bug; that's not their job. The web is not an expanded testing department. It's not my mom's job to help you find the bugs in your site. If a user can't use your site, they'll just leave it and not come back.

Some rant-ish points:

• I don't care how well you think you know the database you're using; you're wrong. You don't know it completely; no one person does. No one person even wrote it, so it's ridiculous to say you understand it completely. Second, it's not just the database you need to worry about; it's also the JDBC driver, the VM, the native platform, the versions of each, and a hundred other factors. It is way bigger than you, and it's time to humble yourself to that; the days when a programmer understood the machine completely are long gone and aren't ever coming back.
• One day your database may change. I guarantee you won't understand it nearly as well as you do your current one, and that's incomplete at best.
• It is not okay to write some code, run it through a couple of use cases, fix the obvious errors, and declare it production-ready. Your job is not to show that your code might work given the right conditions; you must show that your code cannot break in the wrong conditions.
• Your test cases (the programmer's, I mean, not the QA department's) must test not only for correct results but for sensible and well-defined behavior when giving incorrect input. It doesn't just have to work; it also has to not break.
• Treat your users with respect. They aren't programmers, and they don't get paid to use your code. You get paid to give your users an easy, non-confusing experience. Don't try to lazy your way out of it by saying that bulletproof code isn't worth your time -- you have no other function but to produce that bulletproof code.

Remember this at all times:

This is computer science, not computer religion; we don't have faith that things work, we have proof that they do.

One of the bedrocks of scientific theory is the idea of falsification; a scientific theory must be able to disprove statements that are false. (This is why creationism and intelligent design aren't science, since they can't meet this standard.) In this case, a falsifiable statement would be something like: If this operation is interrupted by the user, the database update will fail. You then write a unit test that creates such an interruption and then check if the update failed (either an exception was thrown or the data violates some integrity constraint).

A unit test is really a small proof that a falsifiable statement is in fact false; when you assert something at the end of a test, you are basically claiming that the lines of your code are a list of statements that logically demonstrates the truth (or falsity) of your falsifiable statement. This is exactly the same thing as a geometric proof, and that's not a coincidence. A good unit test only exercises one part of your class, just as a good proof only tries to establish one fact.

You know that your code is ready when it has been proven to not fail. The trick is figuring out all the ways your code might fail, and that's impossible except for the most trivial programs. However, you should be able to predict the vast majority of failure scenarios and prove through unit and integration testing that they will not break your code. If you're not doing that, you're not practicing computer science.

It's time to get religion about computer science.

So, you want to do Maven? Good luck.

Ick.

Maven is the Next Big Thing in Java build management. Its goals and philosophy are pretty far from Ant, which in all honesty is a step in the right direction. Ant scales poorly, uses some ridiculous syntax, and becomes harder to understand the more you try to do with it. For slapping together a simple .war or .jar, it works fine. For controlling a vast array of slightly differing builds in an intelligent and maintainable way... well, let's just not mention that.

Maven has caught on among the Movers and Shakers for one reason only: dependency management. Maven has "repositories" which store version information for "artifacts" (fancy name for jars). You specify what version of something you want, and it does the Right AND Intelligent Thing: downloads it to a well-defined and organized location, automatically sets up classpaths, and makes it available for other Maven builds. This: 1) flat-out rocks; 2) solves one of the biggest problems with making Ant scaleable; 3) makes version conflicts a manageable problem instead of a hair-pulling fit of frustration. "Dammit, I said commons-lang 2.1, not 2.2! Users couldn't download jars properly if it guaranteed them a second life as Jessica Alba's Barcalounger!" Such days are now but a painful memory.

Unfortunately, they just HAD to go and cover this juicy nugget of awesome with a ponderous load of wtf. The pom.xml file, which is to Maven as build.xml is to Ant, is stunningly complex. It's true that you can get a lot more mileage out of a lot less XML in the simplest case, but that assumes you've laid out everything in your project according to the Maven "standard directory layout" -- and don't ask, because you haven't. "No problem, I'll just specify the location of everything!" Sure, do that. But even if you can figure it out without any documentation (of which more later), your nice, cute, simple POM just doubled or tripled in size. Worse, it now looks slightly familiar... I know, it's that huge list of Ant properties that every medium-to-large project has which has nothing in common with the similar list from any other medium-to-large project!

So if I want to use Maven, my first option is to move everything relating to my project into an arbitrary directory structure, thus:

1. screwing up my working Ant scripts, thus requiring me to rewrite them;
2. screwing me on all the taskdefs I'm using that have no corresponding Maven plugin;
3. giving me yet another moving-crap-around-subversion headache;
4. randomly hosing other useful things like the build machine's triggers, my cool Ruby code coverage scripts I'm prototyping, or proprietary analyzers that the Manager insists we waste otherwise useful CPU time on;
5. hope that the next version of Maven doesn't change its mind about what the arbitrary directory structure should be (like v2 did with v1), or I'll have to repeat this entire process.
   

If that sounds unpalatable, no problem: make a nice big ugly POM telling Maven where everything is. How is this better than what I already have? I get the same finished builds out of this POM -- once I've got it debugged -- that I do with Ant. Now I have two ugly sets of build scripts to maintain instead of one.

This is progress?

Recently I migrated a little Java library project of mine to Maven 2 from Ant in order to learn how it worked. I had a simple 150-line Ant script that compiled, ran tests, created javadocs, checked versions, and threw the whole mess into a neatly packaged jar with a bowtie on top. I'm not that retarded, I like to think, but the conversion took around 8 hours, and we're talking about maybe 75 classes in two packages with one external dependency. Now I have a 200-line POM that does the same thing in a completely different way, and it's only that short because I bit the bullet and converted to The One True Project Organization. Why would I try this on a big important project on which screwups or delays might mean my job?

One last item on the rant parade: documentation. Maven 2's is pathetic, and Maven 1, which is well near abandoned only two years after a huge effort to convert people to it, is only slightly better. Tons of important docs on the live site are still "coming soon", and the plugin documentation is long on parameters and short on examples. I don't care how to use esoteric feature X of weird tool Y. I just want a drop-in for doing javadocs and creating shippable packages. This shouldn't be hard to do, and once you figure it out, it really isn't, but no way should I be forced to spend valuable time ferreting out the solution myself through trial-and-error. (No, mailing lists and bug reports are not the answer. I should NEVER have to Ask The Expert When Simple Things Break.) Please, Maven guys, think about the users here.

There is a bit of good news. Maven 2 comes with some Ant tasks that allow you to do Maven-ish things from Ant, including dependency resolution. My shiny new Ant script (after the Conversion) downloads one small jar of Maven tasks, feeds a few simple bits of info on package name and version number, and the whole shebang gets downloaded right then and there and built in one go without needing to restart the build. And if the user ever wants to try the Maven build, it will reuse the previously downloaded dependencies without a single tag of additional configuration. Freaking awesome. Groin-grabbingly transcendent. The Real Build Solution, when it comes, will work in all respects as this little combination does.

Mac Java considered harmful

For the past two months, I've been doing daily Java development on a PowerBook G4. It sucks. Why?

First of all, it's slow. "Well, duh, Java is always slow," squawks the ill-informed peanut gallery. No, as a matter of fact, it's not; runtime performance has gotten to the point where it's comparable with native code on almost every server platform, and I'm not hacking 3D game engines here.

What I mean is that Apple's implementation doesn't compare favorably with Sun's or IBM's. The latest released version of the SDK from Apple is still stuck on 1.5.0_06, which is getting pretty long in the tooth, and the beta download pushes it all the way to 1.5.0_07. The latest from Sun is _11. Also, 1.6 final has been out for over a month now while the Apple beta of 1.6 is months old with no word on updates.

Finally, there's the widely-reported Steve Jobs quote from just after the iPhone presentation (which -- you heard it here first -- will be a huge bomb):

Java’s not worth building in. Nobody uses Java anymore. It’s this big heavyweight ball and chain.

Gee, thanks, Steve. I guess I'll just go shut down those servers running my company's websites.

I know several people are saying he was only talking about the iPhone, but this is a pretty blanket condemnation when he could have said something like "Java doesn't make sense on the iPhone for us". I may have even accepted that prima facie, since client-side Java (outside of J2ME) has been more or less a bust. But combined with the apparent neglect of their Java implementation after extravagant promises on its performance on OS X, this sounds ominous.