FuzzyBlog

Escape from hackerdom or How to Implement Engineering Processes

i.e. How to Bring Web 2.0 Company Engineering Under Control

Yes Virginia this is another article in my Engineering Management 101 series of posts which can all be found under my Software Engineering tag.

Process is often to many engineers a dirty word. Sometimes its a very dirty or #*$&#(*$ word. However this is not necessarily the way process has to be. When implemented correctly process actually aids engineering increasing the ability of the organization to deliver quality product. Alas, that one word “correctly”, in the previous sentence makes all the difference. All too often process is handed down as an edit from above as if the hand of god had graced it upon the organization. Whenever this happens, process is almost always a failure — particularly in small, highly technical groups that are used to full control of their activities.

[ And, yes, given that web 2.0 companies are often started by a single “hackerish” individual that’s the type of “hackerdom” I’m referring to. ]

These types of groups, if not directly involved in the authoring of the processes, will tend to fight them tooth and nail. As in any type of successful organizational change, HOW you go about implementing things makes a world of difference.
So here are my Ten Tips towards Implementing Successful Engineering Processes:

1. Make Sure that the Precipitating Indicent for the Process would actually be Solved If There Was Process in Place. Generally the move towards process comes as a result of some sort of incident that makes senior management throw their hands up and say things like “Those (#$*#($ hackers; we need to bring order to our chaos”. The problem is that all too often the precipitating indicent wasn’t one that process might have solved.For example a network event happens like a partner’s ad server goes down intermittently causing unreliable site performance. Now any software engineer knows that when something happens to software that’s outside of your control to which a network connection is being established is devilishly tricky to debug. Unless you’ve specifically instrumented your code around that sort of thing, you’re just going to be scratching your head when the error occurs. And you’re going to be rapidly debugging just to diagnose the problem. When you run large production web sites these types of things do happen. And, when the partner has large server farms with some kind of dynamic server rotation based on DNS, it gets worse. When a large online portal first debuted their Add Feed to MyCustomPage buttons, we tried desperately to support it and it would work one time. Then we’d refresh the site and try and subscribe again and it would fail. What was happening, as best as we could diagnose from the outside, is that in the pool of servers that handled end user feed subscription there was one or two that were using an older version of code that couldn’t handle feed urls longer than (if I remember correctly) 128 characters. Given that we make long urls, this was clearly an issue to us. Now you could have all the process in the world and all the units tests and it still stands a very good chance of NOT catching that.And, engineers, particularly the really good ones, the ones you have to have to succeed in today’s Internet business are logical. When you try and justify a process change based on a precipating incident that simply isn’t valid and no amount of process would have solved, they immediately and dramatically discount the process model on its face. No matter whether or not the process model actually has valid elements, they discount it and dismiss it. This is sometimes referred to, or at least similar to, “Flipping the Bozo Bit“. Yes engineers do this both too easily and too frequently. Never the less if you want to work with engineers its important to understand.
2. Pick the Right Time to Implement Process.Here’s a secret about process that I probably shouldn’t say. If you want to make sure that you get process implemented, its almost as important to pick WHEN you implement the process as WHAT the process has in it. Let’s say that your engineering department has a new release under way. If you try and change processes before the release is finished, no matter how valid the process is, they will kick, scream and howl. They will fight the process model with every fiber of their being.Why? Well engineers purely hate the rules being changed on them. When an engineering group agrees to ship a product, they do so knowing that there are one or more constraints on them. An example of a constraint is the feature set. Engineers go into a release process knowing the feature set. Add more features to the release and, well, the rules have been changed. The engineers know all too well, whether explicitly or implicitly that new features have costs and that they may well cause the release date to be missed.But, if you wait until after the release is done, then the engineering group is often so damn tired that they’ll simply agree to anything just to have management stop annoying them. Once upon a time I was in a large, publicly traded software company. We had a release team of probably 35 people across our Cambridge, Boulder and Albany departments. We met our delivery date and the Chairman of the board walked through Engineering with a Shopping cart passing out bottles of champagne because we had actually shipped in Q4 thereby fulfilling his promise to Wall Street. And, promptly thereafter, new engineering processes were implemented. None of us were happy but we basically shut up and accepted it. It was the peace of exhaustion.
3. Avoid Holy Wars. There are exactly 2 holy wars in all of programming. One is the Editor war (VI versus Emacs versus an IDE) and the other is the Platform war (Windows versus Mac versus Unix). Any process document which mandates one of these has a high chance of failing. Any process document which mandates both of these has an even higher chance of failing.
4. Make sure that any process you want to implement is economically viable. Engineers tend to be logical (they can also be irrational of course; they’re human). If an engineering process that’s requested isn’t economically viable, well, they know its going to fail. And they simply won’t take it seriously. Lets say for example that someone gets the “Pair Programming” bug and they want to implement it. Well that’s great but if you have engineers that work from remote locations regularly, its not economically viable. Why? Well unless you’re willing to fly people to someone’s home office it ain’t going to work. Now lets say that your organization works with code in 3 different languages (C, Perl, PHP) and you want to implement a process by which all checked in code is reviewed by another person. Well unless your organization has substantial excess engineering capacity that process requirement is not valid. Multiple languages in the same organization generally means your staff are broken into specialists by language. So you might not even have the capacity which means you’re going to have to go out and hire another say Perl programmer just to review your existing Perl programmer’s code. And that’s not going to happen; its just management speak.
5. Don’t leave gaping logical inconsistencies in your process model. The classic mistake the virtually all web process models make lies at the database layer. Engineering process simply isn’t a matter of code checkins anymore. Often there’s significant process issues tied to how you implement changes at the data level. As an example I have regular jobs I kick off on our core data warehouse that can affect the categorization of 100K feeds or more at a time. Recently I changed the categorization of more than a million feeds in a single day. Any process model has to take into account row level versioning across multiple tables. And that dramatically impacts the process model. Where the heck do you check that into Subversion?
6. Identify the influencers and seek buy in. A long time ago I spent 3.5 years building, selling and installing high end ($250,000 + ) corporate knowledge management software. One of the key lessons from experience i learned is if you want new organizational processes to succeed then you find out who the influencers are and you go to them BEFORE you define a new process and you get their buy in as part of the definition step of the process. If you go around the key influencers then they’ll torpedo you every time. The way to succeed is to get the influencers to be part of writing the process spec. This is basic politics 101 / management 101.
7. Make sure its uniformly applied. Process models work when they’re rules in general not rules in specific. If you’re going to have a process model then it applies across the board; every programmer; every project from big to small.
8. Be Specific. Don’t leave a term in their like “Senior Software Engineering Staff”. If you’ve got someone in mind for, say, code reviews then use a name. Let people know who’s actually going to do what.
9. Make things such that process is required. Engineers always fight against new. things. Believe it or not we’re actually the most conservative people in the world. We know that most technology doesn’t work; heck we wrote the bugs that keep it from working. So if you want to get engineers to do something then make it so its actually required. Now let us say you have a sole engineer on a project and he’s not checking into subversion regularly. He owns all the code and he’s solely responsible for every single change. Well if you ask him why, he might reply “Look — I can use Subversion but its slower. I know every change I’m making and when I need to I make backups.” But, however, assign another engineer to that project permanently and I’ll bet you dollars to donuts that Subversion use won’t be a problem anymore. Why? Simple. At that point the use of Subversion is required.
10. Don’t ignore ongoing work in process. Directives like “Cease all coding until the requirements of this process model are met” in any business aren’t realistic. In an Internet business really aren’t realistic. Often times there are significant projects that have to be completed before a process model can be started. Its rare that engineering management knows everything that’s going on. We’ve seen this in case study after case study of factory floor workers and software engineers are new age factory workers.
11. Make sure it matches business constraints. When I was at Dataware, the company was recovering from an engineering manager who required an SEI (Software Engineering Institute) process model. Dataware had traditionally been a hacker driven culture which some of the most talented people I’ve ever been privileged to work with (Hi Jim Kearney; Hi Andrew Willemsen; Hi Pete Jenney; Hi Brian Giedt; Hi Ed Fischer). Anyway the company had newly gone public and management wanted to concurrently deliver innovative new products and have a dramatically more rigid process model. The problem is that the business constraint (innovation) didn’t match the process model (rigid).  This dichotomy basically crippled Dataware’s ability to innovate for a long time.  However it made lots and lots of pretty Gantt charts.
12. Beware blogged lists of steps. The advice you find on the web may only be worth what you pay for it.