From the Archives: Dreaming in Code by Scott Rosenberg

I have a whole pile of science-y book reviews on two of my older blogs, here and here. Both of those blogs have now been largely superseded by or merged into this one. So I'm going to be slowly moving the relevant reviews over here. I'll mostly be doing the posts one or two per weekend and I'll occasionally be merging two or more shorter reviews into one post here.

This one, of Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software, is from August 9, 2007.

Every organization relies on software these days. Big custom systems, shrink wrapped commercial software, all the various protocols and programs keeping the Net running. Big organizations, small organizations, tech companies of course, libraries in particular are relying on the fruits of software developers mental labours more and more. And with the rise of Web 2.0 in libraries and educational institutions, our reliance on our programmers will only get more pronounced. But how much do we really understand about the art of software development and the strange and wonderful habits of programmers, systems analysts and all the rest of the software bestiary?

Not much, it seems. And that's where this fascinating insider account a a high-profile open source software project comes in. Salon.com co-founder and author Scott Rosenberg spent three years as a fly on the way on Mitch Kapor's project to create the ultimate Personal Information Manager (PIM), Chandler. Kapor's project was highly idealistic from the very beginning; the idea was that he would use some of his software-boom fortune to finance a project to make every one's lives easier: a PIM that is flexible, sharable and open, able to handle calendaring, email, note taking and events. Unfortunately, the project was also cursed with design difficulties and numerous delays, with a schedule that stretched out from one year to two and three years and beyond (and not even implemented today). The book includes a colourful cast of both obscure and well-known software luminaries (like Andy Hertzfeld), and goes beyond merely recounting the ups and downs of Chandler but also offers a kind of history of attempts to organize and systematize software development. Name-checking such great software engineering writers as Frederick Brooks, Rosenberg talks about the whys and wherefores of structured programming, object orientation and others. Many chapters mix details of the vagaries of the Chandler project with relevant discussions of theoretical topics in software engineering (such as trying to create truly reusable software modules) with more philosophical musings on the art of software development. Most of all, Rosenberg places us firmly inside the workings of a programming project from hell, complete with gory details, tales from the historical trenches and a bit of that fantastic theoretical discussion on why software is so hard. (So, what's it really like being stuck in the programming project from hell? Trust me, I've been there and this is a pretty good example of the real thing.)

There are a couple of really good bits that really stood out for me in this book, bits that resonated with my own experiences managing and developing software. On page 54 he has a discussion of death march projects and the optimism/pessimism dichotomy that all programmers live with and obsess with every day. Having done a couple of death marches characterized by such extremes, it really resonated with me. On page 75, he begins a discussion various programming languages and the almost religious zeal most programmers have for their favourite ones - I was a big fan of Fortran as a young programmer. On page 274, Rosenberg has a telling comment about programmers' historical blindness, their inability to learn from their mistakes, to use the literature to learn from other's mistakes. I like the way he puts it: "It's tempting to recommend these [pioneering software engineering] NATO reports be required reading for all programmers and their managers. But, as Joel Spolsky says, most programmers don't read much about their own discipline. That leaves them trapped in infinite loops of self-ignorance." I like to think that as a librarian collecting the literature of software engineering, I can help in a small way to make programmers more aware of their past.

On a lighter note, I also like the joke that Rosenberg puts on page 275-276:

A Software Engineer, a Hardware Engineer, and a Departmental Manager were on their way to a meeting in Switzerland. They were driving down a steep mountain road when suddenly the brakes on their car failed. The car careened almost out of control down the road, bouncing off the crash barriers until it miraculously ground to a halt scraping along the mountainside. The car's occupants, shaken but unhurt, now had a problem: They were stuck halfway down a mountain in a car with no brakes. What were they to do?

"I know," said the Departmental Manager. "Let's have a meeting, propose a Vision, formulate a Mission Statement, define some Goals, and by a process of Continuous Improvement find a solution to the Critical Problems, and we can be on our way."

"No, no," said the Hardware Engineer. "that will take far too long, and, besides, that method has never worked before. I've got my Swiss Army knife with me, and in no time at all I can strip down the car's braking system, isolate the fault, fix it, and we can be on our way."

"Well," said the Software Engineer, "before we do anything, I think we should push the car back up the road and see if it happens again."

I'm going to use this joke when I do IL sessions for CS and Engineering grad and undergrad students, and maybe even to break the ice at a departmental meeting.

A great book, an insider view of software development, a real insight into how programmers think and work and how software projects grow and evolve, sometimes how they careen out of control. So, who would I recommend this book for? A number of different constituencies would find this book useful and entertaining.

• IT Managers would find this book very useful for its insights into the personalities of programmers as well as for its history of failed attempts to make a purely predictable engineering discipline out of programming.
• Programmers would find this book terrific, seeing a lot of their own eccentricities in the many stories. As well, programmers would get a lot of insights into their pointy-haired bosses attempts to turn them into engineers rather than the free-spirited hacker-artists they see themselves as.
• Families of the either of the two above groups will get valuable insight into the slightly deranged members of their families, their joys, obsessions and frustrations.
• People that support or employ software developers or managers, such as scitech librarians, HR people in tech firms, venture capitalists in software firms. They will hopefully come to understand how and why software projects are created and sometimes crash and burn. Not to mention how to mentor and encourage developers to take advantage of what is known to improve productivity. The other books and articles listed in the notes are also a treasure trove of further exploration and information. I hate it when books like this don't have a proper bibliography - it makes it a lot more trouble to sift through the notes later on for further reading.
• And really, anybody that uses software of any kind. And since basically everyone uses some sort of software these days, just about anyone would really appreciate this book. Understanding how the knowledge economy and the Internet boom is built from the ground up is certainly enlightening and important. You'll never see a bank machine, interact with a big company's insane internal systems procedures or even use a simple web application the same way. Understanding the challenges involved in getting these systems even close to right and the inevitability of their imperfections is an important revelation in the modern world.

Rosenberg, Scott. Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software. New York: Crown, 2007. 400pp. ISBN-13: 978-1400082476