Selling automated testing to management

I don't think there are that many developers out there these days that don't think that automated unit testing is a good idea. Most seem to agree that automated functional and integration testing are worthwhile as well, even if they're generally harder to achieve. Being able to be reasonably confident that the change you just made didn't disrupt far flung parts of the project just by executing a test suite is great. However it's not a tangible, quantifiable thing you can sell to management.

There are generally two cases where you need to sell the effort involved in building and maintaining these suites. The first (and easier) is when you already have the suites in place, but need to invest in maintaining them. This is an easier sell because usually management will have seen the benefit and/or the ongoing expense is fairly small, and justifying the ongoing effort isn't all that bad. I'll throw new development in here as well... writing the suites as you go usually isn't as bad as retrofitting suites onto an existing code base.

Which brings us to the second case. When you have a legacy product, adding automated test suites is usually time consuming at best, and technically challenging in most cases. Your legacy product probably wasn't built in a way that encourages automated testing. The interfaces may not be there, it's tough to run in a test sandbox, etc. If you want to get automated tests working, you're probably facing a significant investment in people terms, and could be facing some infrastructure expenditures as well.

The wrong way to pitch this is to go to management, roll out your proposal and say that you need to spend $500k over the next three months building something that provides purely qualitative benefit. Unfortunately most attempts at quantification are difficult. You can talk about improved quality, improved productivity, increased agility... but at the end of the day that $500k is staring them down and making everybody in the room uncomfortable.

A better way is to talk about it in terms that can be strictly quantified, and the easiest one I've found is around regression testing. Figuring out how much you spend each year on regression testing is usually pretty straightforward.

1. Write down how many times a year you execute a full regression test
2. Write down many people a single test cycle takes
3. Write down how long it takes
4. Figure out how much a tester's time is worth.

Doing the math hear should give you a dollar amount for your regression expenditure. The next step is simpler, all though less precise. Estimate how much you think you could cut the time you got for #3. From what I've seen this should be on the order of 50%-80%. This gives you a total savings realized from automated testing.

A quick example may help

1. We release twice a year, and execute on average two regression test cycles per release (one beta, one GA)
2. Each test requires four test engineers
3. The test cycle takes four weeks
4. Each tester costs the org $100k a year

this means we have a total cost of 4*4*(4/52)*100k which is about $123k. If I can cut my regression test time to 1 week, I save at least $92k per year. Keep the last two words in mind... that's per year. Forever.

Keep in mind this is a minimum. You still have all the intangible benefits. But $92k/year is something the business folks can plug into their project valuation tools and evaluate. When we ran these numbers for our project it became clear that dedicating people full time to build out automated test suites for existing projects made perfect financial as well as technical sense.

Dealing with Maven bloat and complexity

I recently came across an article questioning whether Maven is too complex and bloated. The short answer is it can be. If you aren't taking some steps up front to deal with some of Maven's quirks, the benefit it provides can quickly be outweighed by the administrative overhead and performance hit it's project management scheme imposes.

At my job we have a fairly sophisticated Maven project. It involves custom code generation plugins, cross-platform C++ compiles and deployments, assemblies, you name it. The number of engineers working on this project is probably around 80, and the number of Maven projects for this system is probably around 50.

It's not the most beautiful system in the world, but at the end of the day a lot of Maven's benefits are realized. Your average developer can check out an individual component from src, run "mvn test", and be reasonably assured
that things will compile, run, and execute unit tests, with minimal up front configuration of the build environment.

This is no small feat. It didn't start off easy. Maven is complicated. It's large. Documentation could no doubt be better. But at the end of the day, we're in a better place than we would have been with ant or make. Some of the steps we've taken to make things easier follow.

1. Create your own repository. If you want to be able to do repeatable builds, don't let your projects access any repositories outside your control. Poms get broken, sites go offline... this causes all sorts of chaos. We ended up with a single repository server, but multiple repositories. The salient ones are maven-releases, maven-snapshots, our-releases, and our-snapshots. We put the artifacts we needed into maven-releases and maven-snapshots manually. We added profiles that can be enabled to get to the public repositories, but these are never enabled on our automated build machines.
   
2. Use inheritance correctly. It's really tempting to use pom inheritance to capture project structure. This is what module tags are for. Pom inheritance is to allow you to apply similar configurations easily (like Java inheritance). It took us a long time to unwind this mess. We have a base-java pom that sets up all the reports we want to run for java, a base-model one to handle domain models, etc.
   
3. Fight the urge to tightly couple your large project. Maven leads you to fine grained componentization, which leads to a looser coupling of components in the build/release sense. There is a natural tendency to be uncomfortable with this (what's really going into my final build), but fight the urge to make it one giant system that gets built from the ground up. Executing releases is a nightmare with one giant system.
4. Use version ranges where possible. This makes dealing with #3 easier. The odds that a component needs that specific release of a component (especially if you're doing agile and releasing every 30 days) is pretty slim. Most just need the latest. Also get familiar with the dependency convergence report.
   
5. Make sure people understand what Maven goals are necessary for doing work. If lots of people are complaining about site generation taking too long, you have a clue that people don't get this. Your average developer should be running test and install, very rarely site.
6. Decide on a versioning scheme up front, and make sure you can execute on it. Nothing is more frustrating than realizing it takes two weeks to get all your poms revved to the next revision. The maven release plugin has been fairly unreliable, but it leads you into a set of best practices that work even if you're taking the steps manually.

After taking these steps, the nearly universal sentiment is that while Maven is indeed complicated, it's a step forward when compared to ant/make. At the end of the day, designing a build process for large systems is difficult and rarely gets the attention from the development org it deserves. Maven doesn't make it dirt simple, but it makes the overall management of the build system simpler.