Escape from hackerdom or How to Implement Engineering Processes

December 5, 2005 at 12:00 am | Posted in Software Engineering, Startup | Leave a comment

i.e. How to Bring Web 2.0 Company Engineering Under Control

Yes Virginia this is another article in my Engineering Management 101 series of posts which can all be found under my Software Engineering tag.

Process is often to many engineers a dirty word. Sometimes its a very dirty or #*$&#(*$ word. However this is not necessarily the way process has to be. When implemented correctly process actually aids engineering increasing the ability of the organization to deliver quality product. Alas, that one word “correctly”, in the previous sentence makes all the difference. All too often process is handed down as an edit from above as if the hand of god had graced it upon the organization. Whenever this happens, process is almost always a failure — particularly in small, highly technical groups that are used to full control of their activities.

[ And, yes, given that web 2.0 companies are often started by a single “hackerish” individual that’s the type of “hackerdom” I’m referring to. ]

These types of groups, if not directly involved in the authoring of the processes, will tend to fight them tooth and nail. As in any type of successful organizational change, HOW you go about implementing things makes a world of difference.
So here are my Ten Tips towards Implementing Successful Engineering Processes:

  1. Make Sure that the Precipitating Indicent for the Process would actually be Solved If There Was Process in Place. Generally the move towards process comes as a result of some sort of incident that makes senior management throw their hands up and say things like “Those (#$*#($ hackers; we need to bring order to our chaos”. The problem is that all too often the precipitating indicent wasn’t one that process might have solved.For example a network event happens like a partner’s ad server goes down intermittently causing unreliable site performance. Now any software engineer knows that when something happens to software that’s outside of your control to which a network connection is being established is devilishly tricky to debug. Unless you’ve specifically instrumented your code around that sort of thing, you’re just going to be scratching your head when the error occurs. And you’re going to be rapidly debugging just to diagnose the problem. When you run large production web sites these types of things do happen. And, when the partner has large server farms with some kind of dynamic server rotation based on DNS, it gets worse. When a large online portal first debuted their Add Feed to MyCustomPage buttons, we tried desperately to support it and it would work one time. Then we’d refresh the site and try and subscribe again and it would fail. What was happening, as best as we could diagnose from the outside, is that in the pool of servers that handled end user feed subscription there was one or two that were using an older version of code that couldn’t handle feed urls longer than (if I remember correctly) 128 characters. Given that we make long urls, this was clearly an issue to us. Now you could have all the process in the world and all the units tests and it still stands a very good chance of NOT catching that.And, engineers, particularly the really good ones, the ones you have to have to succeed in today’s Internet business are logical. When you try and justify a process change based on a precipating incident that simply isn’t valid and no amount of process would have solved, they immediately and dramatically discount the process model on its face. No matter whether or not the process model actually has valid elements, they discount it and dismiss it. This is sometimes referred to, or at least similar to, “Flipping the Bozo Bit“. Yes engineers do this both too easily and too frequently. Never the less if you want to work with engineers its important to understand.
  2. Pick the Right Time to Implement Process.Here’s a secret about process that I probably shouldn’t say. If you want to make sure that you get process implemented, its almost as important to pick WHEN you implement the process as WHAT the process has in it. Let’s say that your engineering department has a new release under way. If you try and change processes before the release is finished, no matter how valid the process is, they will kick, scream and howl. They will fight the process model with every fiber of their being.Why? Well engineers purely hate the rules being changed on them. When an engineering group agrees to ship a product, they do so knowing that there are one or more constraints on them. An example of a constraint is the feature set. Engineers go into a release process knowing the feature set. Add more features to the release and, well, the rules have been changed. The engineers know all too well, whether explicitly or implicitly that new features have costs and that they may well cause the release date to be missed.But, if you wait until after the release is done, then the engineering group is often so damn tired that they’ll simply agree to anything just to have management stop annoying them. Once upon a time I was in a large, publicly traded software company. We had a release team of probably 35 people across our Cambridge, Boulder and Albany departments. We met our delivery date and the Chairman of the board walked through Engineering with a Shopping cart passing out bottles of champagne because we had actually shipped in Q4 thereby fulfilling his promise to Wall Street. And, promptly thereafter, new engineering processes were implemented. None of us were happy but we basically shut up and accepted it. It was the peace of exhaustion.
  3. Avoid Holy Wars. There are exactly 2 holy wars in all of programming. One is the Editor war (VI versus Emacs versus an IDE) and the other is the Platform war (Windows versus Mac versus Unix). Any process document which mandates one of these has a high chance of failing. Any process document which mandates both of these has an even higher chance of failing.
  4. Make sure that any process you want to implement is economically viable. Engineers tend to be logical (they can also be irrational of course; they’re human). If an engineering process that’s requested isn’t economically viable, well, they know its going to fail. And they simply won’t take it seriously. Lets say for example that someone gets the “Pair Programming” bug and they want to implement it. Well that’s great but if you have engineers that work from remote locations regularly, its not economically viable. Why? Well unless you’re willing to fly people to someone’s home office it ain’t going to work. Now lets say that your organization works with code in 3 different languages (C, Perl, PHP) and you want to implement a process by which all checked in code is reviewed by another person. Well unless your organization has substantial excess engineering capacity that process requirement is not valid. Multiple languages in the same organization generally means your staff are broken into specialists by language. So you might not even have the capacity which means you’re going to have to go out and hire another say Perl programmer just to review your existing Perl programmer’s code. And that’s not going to happen; its just management speak.
  5. Don’t leave gaping logical inconsistencies in your process model. The classic mistake the virtually all web process models make lies at the database layer. Engineering process simply isn’t a matter of code checkins anymore. Often there’s significant process issues tied to how you implement changes at the data level. As an example I have regular jobs I kick off on our core data warehouse that can affect the categorization of 100K feeds or more at a time. Recently I changed the categorization of more than a million feeds in a single day. Any process model has to take into account row level versioning across multiple tables. And that dramatically impacts the process model. Where the heck do you check that into Subversion?
  6. Identify the influencers and seek buy in. A long time ago I spent 3.5 years building, selling and installing high end ($250,000 + ) corporate knowledge management software. One of the key lessons from experience i learned is if you want new organizational processes to succeed then you find out who the influencers are and you go to them BEFORE you define a new process and you get their buy in as part of the definition step of the process. If you go around the key influencers then they’ll torpedo you every time. The way to succeed is to get the influencers to be part of writing the process spec. This is basic politics 101 / management 101.
  7. Make sure its uniformly applied. Process models work when they’re rules in general not rules in specific. If you’re going to have a process model then it applies across the board; every programmer; every project from big to small.
  8. Be Specific. Don’t leave a term in their like “Senior Software Engineering Staff”. If you’ve got someone in mind for, say, code reviews then use a name. Let people know who’s actually going to do what.
  9. Make things such that process is required. Engineers always fight against new. things. Believe it or not we’re actually the most conservative people in the world. We know that most technology doesn’t work; heck we wrote the bugs that keep it from working. So if you want to get engineers to do something then make it so its actually required. Now let us say you have a sole engineer on a project and he’s not checking into subversion regularly. He owns all the code and he’s solely responsible for every single change. Well if you ask him why, he might reply “Look — I can use Subversion but its slower. I know every change I’m making and when I need to I make backups.” But, however, assign another engineer to that project permanently and I’ll bet you dollars to donuts that Subversion use won’t be a problem anymore. Why? Simple. At that point the use of Subversion is required.
  10. Don’t ignore ongoing work in process. Directives like “Cease all coding until the requirements of this process model are met” in any business aren’t realistic. In an Internet business really aren’t realistic. Often times there are significant projects that have to be completed before a process model can be started. Its rare that engineering management knows everything that’s going on. We’ve seen this in case study after case study of factory floor workers and software engineers are new age factory workers.
  11. Make sure it matches business constraints. When I was at Dataware, the company was recovering from an engineering manager who required an SEI (Software Engineering Institute) process model. Dataware had traditionally been a hacker driven culture which some of the most talented people I’ve ever been privileged to work with (Hi Jim Kearney; Hi Andrew Willemsen; Hi Pete Jenney; Hi Brian Giedt; Hi Ed Fischer). Anyway the company had newly gone public and management wanted to concurrently deliver innovative new products and have a dramatically more rigid process model. The problem is that the business constraint (innovation) didn’t match the process model (rigid).  This dichotomy basically crippled Dataware’s ability to innovate for a long time.  However it made lots and lots of pretty Gantt charts.
  12. Beware blogged lists of steps. The advice you find on the web may only be worth what you pay for it.

Python Link of the Day

December 3, 2005 at 10:05 am | Posted in python, Software Engineering | 6 Comments

Ok I have my 1st gui Python app up thanks to this. Its nothing more than a cross platform text box but just that is cool. Quite cool actually.

Now conceptually I really do like the idea that Python is brace i.e. { and } free. Having managed literally departments of programmers through different programming standards and the “indentation wars” I can see the desire to call “A Pox Upon their Houses; there shall be no braces”. Guido was absolutely brilliant when he made this decision and $guido_geek_fu++; as we would say in PHP syntax.

That said one of the more infuriating things I find about learning Python is that when you paste code in from the web, the indentation gets munged and then things don’t compile. Given that indentation is basically invisible and that any newbie will paste code in from things like O’Reilly’s Safari (i.e. an online version of the Python Cookbook), you can see the problem.

Am I the only person who’s ever noted this? I can’t be but I’ve never seen it discussed.  Niall I know you do Python as do you Simon (and, yes, I’m looking intently at Django).  Am I on crack here ?  Did I fall off the turnip truck somewhere in the basics of the learning curve?  Now bear in mind that I’ve spent the last several years steeped in brace centric languages and, at least for me, I find that the getting past the initial hump of a new language is the hard part.  After that its all the same stuff but that initial hump, the adaptation to a very new way, is the hard part.

Engineering Management 101 : BitRot and Why Engineers Hate Building Tools

December 2, 2005 at 1:19 pm | Posted in Software Engineering | 10 Comments

This is an article in a series that I’ll be writing. The next one may be “Why Engineers Don’t Always Do What They’re Told”.

Understanding BitRot

Among software engineers there’s a term BitRot. We don’t talk about it all that much and its not well understood but its this: Bits really do rot. You write some code. It pulls in some libraries. 3 months later you use it again and despite never having touched that code, it magically doesn’t work. Either one of the libraries changed or a Unix permission somewhere went from Foo to Bar or whatever. Even though bits are digitally perfect, they actually decay. Its an entropy thing. So I’m working on the new version of Feedster that will come out *censored* and I keep finding that what I’m doing is familiar. So I grep through the code base and, sure enough, I’ve done this before. And you what happened? The damn bits rotted on me. So when you manage engineers keep in mind that just because your people wrote code once upon a time, if that code isn’t in active use, it may well have rotted away.

Why Engineers Hate Building Tools

Now a very, very common case of BitRot is in the area of internally developed tools. You know what I mean — you have a site that does Bar so you need a little web form or executable that makes Foo easy. So you, generally some non geek, asks a developer to do it for you. Heck James did this to me last night. It happens all the time. Now the problem with tools and BitRot is that those tools are never industrial strength. They’re always, always last minute efforts that never get thought out well. They tend to break at the drop of a hat or with the slightest breeze. And what happens is the developers themselves never use them so we don’t see it when they break. We go and modify some core library which is called by it, change the library calls in the production version of the code and not in the tools.

So BitRot is one common problem with tools. Another one is that building great tools is actually really, really ***hard***. Now traditionally at Feedster I’ve done site level administration right at the raw SQL level. I mean its easy right to write this: “UPDATE foo SET BAR=1 WHERE qoo=BAH”;. Actually its not easy. But its trivial and I’m used to it. I spend 99% of my every day in Putty logged into up to 35 to 50 different sessions. Yes it is “King of Tabs” meets ssh for “Window Hell”. Since I always have access to the server canonically known as “dbwrite” it really hasn’t bothered me. But Feedster’s not just me anymore (technically it hasn’t been for a long time but as our traffic volume rises the # of support incidents rise correspondingly). And since I end up with a bunch of issues forwarded to me, my work load has been rising. So lately I’ve been on a “Tools Tear” to enable people like James (who is a sales guy and has no relationship to either support or QA but is just helpful as all get out) and Sheldon.

So you start implementing tools… And the complexity just ***grows***. An example is making minor edits to your site’s home page like adding the Feed of the Year contest button to the home page. So you figure:

  • sole text field w/a unique key
  • db save and load routines

That of course then leads to the need for custom styling or presentation attributes that you simply can’t predict so you to:

  • text field of unlimited size
  • contains html

Then you find out that you need to support edits to pages other than the home page so you write an html UI page which lets someone other than the programmer choose what page to edit. And you load it with the names of the unique keys that represent the pages to edit.

Then you find out that you have to support a staged environment. This requires a preview ui. It also requires a presentation attribute for “Hot Diggity Damn! This is Ready to Be Served”.

Then don’t even get me started on versioning… And yes I do know that changing text on the home page is both mission critical and non trivial but even the smallest tools have real world requirements. I’ve seen it recently.

And that’s not even taking account issues of login, permissions (not everyone should be able to edit the home page), security holes, etc.

Tools are complex. Engineers hate to write them, at least in a non resourced way, because we know whether by having thought through them in full or simply knowing intuitively “oh crap here I go again”.

Recommendation: Its not all lost. You can do tools right and here’s how:

  1. If you’re going to build tools then recognize up front that there are real costs to them. BitRot is real.
  2. Assign resources to tools and make building and maintenance of tools the way that new engineering hires are trained. If you’re a new hire then you gotta put your time in like anyone else.
  3. Make the engineers use their own tools! I’ve now stopped doing “UPDATE foo SET BAR=1 WHERE qoo=BAH” and been using my own tools and its made a huge difference.
  4. Have one landing page for your tools. If you’re a web company — and who isn’t — then integrate even crappy little tools into your Intranet’s main home page or they’ll never get used anyway.
  5. Metrics. And for the special GeekFu bonus round what you want to do is tie metrics into your tools. Why? Well its a lot easier to REFUSE requests for tools if you can say to people “Why Svringar you never used the tools I made last time”.

About Me

Although these days I’m known mostly as an engineer and I do operate as a CTO / Senior Engineer / Sole Contributor, in a previous existence called Dataware (Chief Technology Strategist; whatever that is) or Mascot Network (VP Eng), I ran large scale project teams.

Great Thoughts on Options in Software

November 27, 2005 at 11:53 am | Posted in Software Engineering | 13 Comments

Too Many Options! Matt has his usual wonderful thoughts, this time on options in software. Having once sold a piece of software with roughly this many options (shown on left). Matt’s right — most people won’t use the options and you can always expose a plugin api for it if you have to.

Good job Matt! This was a gutsy post to write — what Geek doesn’t want features? Having big options dialogs makes you look like you have tons of features. And you do, just not usable ones. Its all about the 80 / 20 rule and clearly you understand that. And, while Joel would argue that the 20 is always different, I don’t buy that. While there always are differences, there are also commonalities that, if you take the time to understand your users, can be leveraged to everyone’s advantage.

Microsoft Project – Any Improvements

November 15, 2005 at 11:06 am | Posted in Software Engineering | 15 Comments

I was wondering if anyone out there in BlogLand knows if Microsoft Project has had any decent improvements since Project 2000 ? I find myself using Project more and more to manage my own engineering, particularly the coding I have for our joint project with Mitsui and I’m > 5 years out of date.

The things I find most bad about Project 2000 are:

  • Its persistent and dasdardly tendency to eat its own data files. I’ve been nailed by this so many times in the past that it scares me. Still its the only project management tool I’ve got so I keep using it. In particular cutting and pasting things seems to cause corruption.
  • Saving to HTML. I find myself always having to select a Map when I save to HTML. Is there a new Feature like “Automatically Save to HTML using Template Foo?”
  • Do I have to update to a whole new version of Office to get a new version of Project? Right now I’m happily using my personal copy of Office 2000 I bought back in 1999. Feedster the company doesn’t even own a copy of Office on Windows (our machines other than myself and 2 admins are, I believe, all OSX or all Linux).  I’m dramatically more likely to upgrade to Project if it doesn’t require a new version of Office since I run projected exclusively on my old Thinkpad X20 laptop (only 384 megs of RAM; Windows 2000).

Robert ? Is there a blog by a Microsoft Project Product Manager that could address these questions?

Blog at
Entries and comments feeds.