Whether we are developers, project managers, business analysts, stakeholders or anyone else involved in the software development process, we all know that estimates for software development are always wrong: ALWAYS. I have never once been involved in a software project that met it’s estimates or even came close. Sometimes we get the job done earlier than estimated, but usually it runs over. You know it. I know it. We all know it. Even if that estimate seems reasonable at the time we do it, it always ends up being wrong. So why do we still bother to do them?

What would seem like the obvious answer is that we do them because the business team expects them. Businesses run on budgets and budgets are planned in advance. And because those on the business side of the aisle require that IT conform to their way of doing things, everything gets a budget. In many businesses, IT development doesn’t really have its own budget. Each department decides what projects they want done and when and the department bequeaths a tiny portion of their funds to pay for development of those projects.

assorted notepads

Photo by Patrick Perkins / Unsplash

The process generally works something like this:

  1. Business unit wants a new application created. They tell IT what they want created, usually describing a complex application in a single sentence or two describing X & Y. It should be completed in 4 weeks because it sounds so easy in their heads.
  2. The IT business analyst goes about documenting the actual requirements. They determine fairly early that instead of X, what is really wanted are A, B, C & D.
  3. The project manager takes this to the developers, who say that A, B and D will take 8 weeks to develop and that C is not possible in phase I as it will take an additional 12 weeks by itself.
  4. The PM takes this information back to the business unit. The business unit, feeling generous, says Ok, give me A, B, C & D, and in 6 weeks.
  5. The PM, wanting to impress the business unit in the hope of transfering out of IT, tells the developers they have 5 weeks to do everything, and their 5 weeks started 1 week ago.
  6. The developers go back to their basement cubicles and start sending resumes to recruiters.
  7. Three weeks into the project, the business unit adds E, F & G to the requirements, and still wants it all done in the original time-frame.
  8. After 12 weeks, the project is abandoned after only requirements H & I are completed because that’s “good enough” for phase I.

I may have over-generalized a little bit, but from the IT side that’s generally how it feels like it always works.

The reasons for the overruns are numerous and I won’t go into them here. But the real question is: If the product of the estimation process is always so completely wrong, why do we spend so much time even trying? Certainly, if there was no actual benefit, there would have been enough push-back that the process would have been largely abandoned over time. Here are the reasons that I believe we still go through the project estimate process.

It helps us find the real requirements

All too often (read “always”), what the project requester tells you they want is not what they actually want. Communication is a difficult thing and it’s often hard for someone to convey in words what they are actually picturing in their heads. They often leave out parts because in their mind everyone already knows those pieces. By having people of different skillsets, like BA’s and developers, walking through the requirements that are presented, they can ask questions about things that don’t make sense in their heads and eventually tease out of the requestor those details that are missing from the documentation.

It helps us prioritize

There are usually more projects being requested than capacity from the development staff. Going through the process helps us to understand what’s more important or less important than other projects. We can also spot where multiple projects might even be combined or good ways to break the proposed project up into smaller, more “developable” chunks. And yes that’s a real word. Don’t bother to go look it up.  Trust me. This prioritization goes a long way towards determining if we have enough developers and to help prevent the existing developers from getting too overloaded. It also helps us to find critical dependencies. I was once involved in a project where I spent two months of full time development on a project that was being created to be an add on to another application being created in a different part of the company, only to find out that other project had been pushed back a year or more. I still don’t know if that project ever did get done.

Project Plan

Project Plan

We enjoy creating

Most developers love planning out and creating something new. We love to spend time designing and creating some app or service in our heads that we may or may not actually create one day. Going through the estimation process stirs those creative juices in our brains as we plan out how we’re going to do this or that in the requirements list. Every new project is a chance to create a new masterpiece of code. We’re going to get it all right this time! Well, what we plan out in our heads is a masterpiece. The end product is often a horrific mess of code that terrifies and confuses us when we go back a year later and look at it again.

We love to experiment

Every new project is a chance for us to use some new library, language or feature. As we read through the requirements we might think “This is a perfect project for F#” or “We could use some machine learning here to figure out why customers always choose this product and not the other one”. It doesn’t matter how weak the argument is. If we can find an excuse to use something new, we’ll find a way to throw it in.

We love to complain

We secretly love finding new ways to complain about the business team. Every new project is a chance for us to complain that they “never let us do it the right way”. Oh, wait, maybe that’s just me….
***Note: If any of my business colleagues from current or previous jobs reads this, just know that I never said anything bad about you. You were always my favorite. Even you, Matt!  ;-)

It helps us mitigate risk AKA “Cover your ass”

All projects have some level of risk. By doing a detailed analysis of the project requirements we can spot high risk zones early on that might cause us trouble later. Are we storing Personally Identifiable Information (PII) that requires government compliance? Are we connecting to external vendors? Will this application mission critical or will it require high availability? And so forth. My manager at my second full time development job taught me this lesson clearly. Cover your ass, because no one else will do it.

This concept was clearly depicted at the recent Congressional hearing with Equifax’ former CEO. He tried to lay the blame for the whole incident on a single worker in IT instead of the real reasons, which are more likely inadequate funding and prioritization of proper security practices by the executives so they could take larger bonuses for themselves.  When things go to hell, IT is always the first one thrown under the bus, so to speak. So find those risks, estimate what it will take to mitigate that risk, and then triple it.

It helps us identify knowledge gaps

While it may sometimes appear otherwise, the geniuses that are developers do not actually know everything. I know! It’s difficult to grasp that concept, but it’s true. There are times that a project may require knowledge our infinite brains don’t already have. The sooner we figure that out and get studying that new technology, or bring in someone who already knows it, the better.

It can help uncover new business products or concepts

Sometimes talking through requirements can help inspire a brand new product. We may spot that million dollar idea for the company that we never even considered before. Or we may even just have an idea for how to make our current processes better so we can save the company millions of dollars and get that extra $100 bonus check ($34.27 after taxes). You just never know!

There are real reasons that we do estimations. However, providing an accurate, detailed time-frame for when a project will be completed is not one of them. It just won’t happen. Give it up. Projects will always finish sooner or, more likely, later than planned. Even after the 18th iteration of estimations 12 weeks into the project. That will never change, so stop worrying about it and focus on the productive things that DO come out of that process. In fact, lets just stop calling it an “Estimation” process altogether and come up with a better name for it. How about the “Project Enhancement Process”? Yes? No? Maybe?