Software Estimation - meaning and sources of error

Do you remember when did you make your first software estimate? I could not. But it must have come very naturally and unconsciously. Perhaps when I was examining my first assignment requirement, looking for room for an extra feature. Or when we divided work in my first group project. Or when I was asked "how long will it take?" during my internship. Nowadays, the projects I am estimating have gotten more complicated that I can no longer depend on my intuitive judgment for reliable estimates. It is an obvious need to upgrade techniques and mindset accordingly to meet the new situation. Reliable software estimate is not a black magic nor a fictional story, it is a skill that can be improved through retrospection, discipline and practice. In this post, I will briefly capture the meaning of accurate estimate to an organization and identify sources of estimation errors. (Slides are available)

Estimation in software development

Estimation is considered the building block of many activities in software development life circle. These include schedule planning (detailed schedule, complete work breakdown structure), budget planning (prioritize functionality, divide iterations) and resources planning. Accuracy of estimate affects these activities and in general the project ability to hit target. So when a consultant company like us is asked for an estimate, we are not asked for a tentative judgement that we can change our mind later on. We are asked for a commitment or a plan to meet a target.

Given that responsibility, developers' attitude towards estimation is still neglected. If I ask you how long would it take to finish your coming release, there is a huge change that I would get a single-point number answer. In fact a project outcomes follow a probability distribution (McConnell, 2006). A project might be done faster, or later. And the chance the project will be done in the middle of the distribution is most likely. Moreover, there is always a limit on how well a project can be done so the part on the left will be truncated.

What we have now is a representation of the probability distribution of project outcomes. The single-point "estimate" we had earlier is actually a target.

What is estimate used for in your company? If you don't really care how software development works, estimation is all about charging customer. Actually there is much thinking following that probability statement. Take estimate as a means of visualization, project planners will be able to find the gap between a project's business targets and its estimated schedule and cost.

A good estimate is an estimate that provides clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit targets - McConnell, 2006

As it is hard to get 100% accurate estimate and error is inevitable, what is better, overestimate or underestimate? In overall, the penalty for underestimation is more severe than that of overestimation. However

The focus of the estimate should be on accuracy, not conservatism. Once you move the focus away from accuracy, bias can creep in from different sources and the value of the estimate will be reduced - McConnell, 2006

Why your estimate is not accurate

The cone of uncertainty

The amount of uncertainty during software development process varies at different moments and typically narrows down over time. This implies that estimate made at the beginning of the project is less accurate that estimates made at later stages. At the beginning of a project, the targets haven't been fully conceptualized yet. Decisions at this level are broad and subject to changes in future. On the other hand, most decisions in development phase are significantly smaller and focus on implementation details. Do not expect the cone to narrow itself though. Unless decisions to remove uncertainty are made, the variation does not go away.

Organization structure that kills both productivity and predictability.

An environment that

• Makes room for multitasking to creep in
• Employs incompetent technical skills
• Apply incomplete/unskilled project management

Is a great condition to surpress our productivity. It suffers and goes down, rarely linearly but usually exponentially. The key is, the parameters of this exponential function are highly subject to human nature and context. In other words, they are a myth. Thank to this, there gone our predictability.

Unstable requirements

Everyone in the industry knows how destructive requirement changes can be. They prevent us from narrowing down the cone of uncertainty. They are also not well tracked and the project isn't re-estimated when it should be. Project control strategies are out of the scope of this post. But in term of estimation strategy, we can incorporate an allowance for requirement growth and changes (McConnell, 2006). 20%, 30% or 50% depends on your own judgement.

Omitted activities

Include time in your estimates for stated requirements, implied requirements and non-functional requirements. Nothing can be build for free and your estimates shouldn't imply that it can.

Unfounded optimism

Developers are hopelessly optimistic creatures. Yes, we are. That's a fact. It's a thing that we can't deny. This behavior has a close relationship with omitted activities. Our mind is not prepared for multi tasking. Instead of juggling things simultaneously, people have a tendency to focus on only one thing that they are most interested in, they know best or they perceive as being most important to the project and discard the rest. The moment you fail to conceive the project as a whole picture, subjectivity and bias creep in.

Estimate Influences

Because everything that helps you understand the nature of software development improve estimation accuracy.

Project size.

Project size implies the number of potential variations in a project. The bigger the project, the harder it is to be estimated. If building a 10-story building take 20 man-months, how long would it take to build a 100-story? Much more than 200 man-months. During the construction, many issues would emerge. Architecture to support the height, wind and natural disaster. Logistic so that materials are always ready and idle time is minimized. Power plan to keep the structure habitable. That's a perfect metaphor for software development. As the project gets bigger, there are more modules. These modules need to communicate with each other. The number of these communication paths is corresponding to the effort required for to complete the project. And it grows exponentially. This phenomena is widely known as diseconomy of scale.

One of the biggest problem I had related to this was that I failed to anticipate the rich interaction between components. The initial estimate was usually correct, but then missing pieces of how it interact with others kicked in and overflew the estimate.

Type of software

Different software types have different focus and difficulty. Developing an e-commerce website is generally easy and focuses on reusability to maximize profit. Developing a life-critical system is not only complicated on its own but also constrained by industrial or legal regulations. Although not all software development activities produce code, these differences are somehow reflected into LOC over time and demonstrate a strong different in velocity across different project types.

Personnel

Researchers have found a difference of 10-fold in productivity and quality between good and bad programmers with the same levels of experience and also between different teams in the same industries (McConnell, 2008). This degree of variation is not unique to the software industry but more a common behavior among occupations, including professional writing, sport and police work. The top 20 percent of the people produced about 50 percent of the output. On the other hand, 10 percent of the people contribute negatively to the output, i.e. can't fix the problems they created.

---

Steve McConnell. (2006). What Is an "Estimate"?. In: Software Estimation - Demystifying the black art. Redmond: Microsoft Press.

Steve McConnell. (2006). Value of Accurate Estimates. In: Software Estimation - Demystifying the black art. Redmond: Microsoft Press.

Steve McConnell. (2006). Estimate Influences. In: Software Estimation - Demystifying the black art. Redmond: Microsoft Press.

Steve McConnell. (2008). 10x Software Development. Available: http://blogs.construx.com/blogs/stevemcc/archive/2008/03/27/productivity-variations-among-software-developers-and-teams-the-origin-of-quot-10x-quot.aspx. Last accessed 26th Mar 2013.

[Lightning talk] Group by time interval in SQL

The other day I needed to write a query that grouped entries by time intervals (week, month, etc.)

I thought of PSQL's generate_series first as it is capable of creating series of numeric values with precise step. So my first attempt would be to select start and end dates of a series of one-week intervals, starting from the first case creation date.


It was not easy as it seems. I went through a hell of type casting. Then the easy step would be to group entries whose created date fails into these intervals.


In fact, I didn't even get to this step. I, in fact, went to the wrong direction, given the context of the feature I was implementing, it made more sense to group entries by calendar's week, not any random 7-day interval.


In order to achieve that, the most critical step was to be able to trace back the beginning of the week from any given date. Thank god, that was easy.


The price I had to pay for going the wrong way was high, but for this case, I guess I would eat it

3 Days of Lean and Kaizen

This week, from March 4th to 6th, the first Lean Mindset Workshop and Kaizen Camp Gathering in HCMC was held. The Agile Vietnam guys did an truly awesome job in bringing together top-notches around the globe the event. That included experts on Lean Software Development, Mary and Tom Poppendieck as well as Jim Benson and Tonianne Demaria Barry, founders of Personal Kanban.

Joining the event was both a last-minute decision and a bet to me. Three days before the event, Saigon Service Design Jam was held. It was also the first of its kind in Vietnam, and also lasted for three days. I couldn't afford the two events together. Selecting the Lean and Kaizen one was truly a leap of faith as I had stopped attending the events of Agile Vietnam a while back. Although I had to commuted 30km in Saigon's heat for the event, it was well chosen.

What was there?

The Lean Mindset Workshop occupied the whole March 4th.

Mary and Tom were an admirable 70-something couple who were spending their retired time travelling around the world giving Lean Mindset workshops and writing books. Honestly, I was overwhelmed by the amount of information. The usually-two-day workshop was compressed into one so that was a lot of talking and for a short period in the afternoon I was lost.

The workshop started by examining the death of companies that were driven under the name of productivity, developed securing manual and policy which ironically prevented innovations and finally failed to focus on its customer's values. The stability periods between economy crisis are getting shorter, the market is more fluctuating than ever and yet disruptions remain extremely hard to forecast. There is a rising need for organizations to structure themselves to be flexible against changes from time to time. The ideas of Lean Mindset are evolved around this concept.

We moved on to identify the seven flow disrupters in software development. Together in a group of 7-8, we discussed about the source, effects, and solution of each. The next step was to give our organization a health check based on the basic disciplines of a healthy organization. The list had a lot to do with quality control and as expected Cogini failed hard on testing disciplines (though we got most of the rest right).

Due to the time limit, we could only afford wrapping up Iteration, Kanban, and Continuous Delivery quickly. Kanban was still a pretty new concept in Vietnam software industry so it was important to stress that Kanban doesn't ask an organization to change anything that its currently does. Positioning itself as a supporting framework to visualize processes and policies, Kanban can be integrated smoothly with organization's current process. The three processes covered in the workshop are tools serving the purpose of making organizations more Lean and agile.

To end the day, we had an interesting talk about improving productivity

The legendary 10,000 hours

Balancing challenge and skill

The science of motivation

Kaizen Camp took over the last two days of the event. 

I found the way the workshop and the camp lined up and supported each other fascinating. If there had been only the workshop, most attendants would have came out like me, overwhelmed and confused by the massive amount of information. If there had been only the Kaizen Camp, there wouldn't have been enough topics to feed two days full of action. During the camp, we discussed work, life, and community. And under the spirit of kaizen, we also discussed improving all of them. The camp shared the same format with Barcamp and even though there were facilitators, the topics were community-centric and highly diverse. Fascinating!

The topics gave me the opportunities to challenge and practice what I was introduced to during the workshop. One of the reasons I stopped attending events of Agile Vietnam was that even though the sharing there was interesting, I found that each organization was unique in certain ways. How could I apply something that had been successful else where was always blur. Facilitating a group of motivated people isn't about teaching them or telling them what to do but more about asking triggering questions to get the participants to open up and speak out what is on their mind. And Mary, Tom, Jim and especially Toni made awesome facilitators. Many questions were answered specifically to Cogini context and the fog was just lifted. I ended up facilitating a session or two myself.

Nice people

I have attended a number of Barcamp, locally and internationally. Yet the attendants to this event managed to be the most diverse. I didn't met as many managers as I would like to, but there was a gentleman from TMA that really knew what he was talking. I met a girl from the North that seemed to have full of doubts in life yet decided to join a startup. I made friends with two engineers with tremendous experience from KMS, and a head of department of a game company. There was also a startup guy who seemed to be making similar mistakes as I did when getting to know Agile methodologies. And who could have known that I would run into a high-school friend there. Behind each attendant was a more sophisticated story. A startup with bureaucracy problems of a Forbes 500. A company that sent only programmers and no manager to the event. A sponsor that didn't mind that its name couldn't show up on media prints. Despite all of those differences, I was moved by their genuine desire to learn, share and make friends. Once again, my belief that engineers are nicest people on earth was strengthen. Thank you.

---

* Special thank to Huy Tran for allowing me to use his photos to illustrate for the post

Do unconventional interview questions work?

Whether today's ever-more-polymorphous interviews succeed in identifying better employees is an unanswered question. The use of peculiar questions and arbitrary tests may seem to go against one of the few rock-solid percepts of today's human resources profession. This holds that any method of selecting job candidates should be as closely related to the work as possible. Most HR people place the most faith in work sampling, where the candidate is asked to perform or simulate work similar to which he'd be doing if hired. Statistical studies of work sampling (a famous one was done by AT&T from 1956 to 1965) showed impressive predictive ability.

The usual justification for "creative thinking" riddles and personality assessments is that they test broad, general abilities, not tied to a specific set of skills. Whether they do that is hard to say. What's certain is that "pet" questions take on a talismanic quality for some interviewers. Just as athletes don't change their shirt during a wining streak, interviewers keep asking the same questions because of a few remembered instances where it supposedly "worked". That fact that many of the most admired, innovative companies use such interview questions seems to speak for itself ("You can't argue with success").

It is far from clear that either season holds water. The human resources profession is full of customary practices of no demonstrable value. The psychologies Daniel Kahneman tells the tale of a test once used by the Israeli military to identify candidates for officer training. A group of either recruits, stripped of insignia, was instructed to carry a telephone pole over a wall without letting it touch the wall or the ground. The point was to observe who tool charge (the "natural leaders") and who fell meekly into place behind them (the "followers"). "But the trouble was that, in fact, we could not tell", Kahneman said. "Every month or so we had a 'statistics day', during which we would get feedback from the officer-training school, indicating the accuracy of our ratings of candidates' potential. The story was always the same: our ability to predict performance at the school was negligible. But the next day, there would be another batch of candidates to be taken to the obstacle field, where we would face them with the wall and see their true natures revealed."

Similar tactics are alive and well throughout corporate America. In today's overheated job market, a common test is to seat a group of candidates for the same job around a conference table for a "group discussion". They know that only one will get the job. The discussion becomes a little reality show, with the recruiter quietly noting who takes charge. It's doubtful that it works any better than the Israeli army test did.

Proving that a hiring technique works - or that it doesn't work - is a complex exercise in statistics. Were once to demand that a hiring criterion be 100 percent reliable, employers would have to hand out jobs at random. There aren't any 100 percent reliable criteria - not work history, not grades, not anything. Hiring is always a game of chance. Many job seekers complain that some talented people do poorly on today's unconventional interview questions - ergo no one should use them in deciding whom to hire. This isn't a compelling argument for the reason given above. But psychological studies indicate that people are apt to view almost any criterion as "unfair" when it's used to decide who's hired or promoted. The sense of unfairness is greater when the criterion is unfamiliar. A traditional job interview is a conversation. The job offer or rejection comes days or weeks later, affording a certain emotional distance. Creative-thinking questions often bring the rejection right into the interview, right in your face. If you fail, you generally know you've failed. That feels worse than a rejection days later. Admittedly, this attitude may not make sense, but when have emotions ever had to make sense?

---

William Poundstone. (2012). Punked and Outweirded. In: Are you smart enough to work at Google?. New York: Little, Brown and Company. p48-50.

Beat 2013 like a boss

Annually, at this time of a year, we gather together looking back at the challenging year that just passed by and looking forward to a successful year ahead.

Changes are a crucial part of life. Hate it or love it, no one can deny that variation in life constantly face us head on, define us and make us who we are. And 2012 was such a milestone, full of changes. The world economy was still trying to keep its feet on the ground. Our belief was shaken. We have never been through that much up and down in our history.

Summarizing the full of event 2012 is a non-trivial task. I am afraid that the meaning of events that we have been through exceed the expression power of words and only the very people who had been there, done that can understand. In my best attempt, I would like to retrospect what happened to us and the difficulties that we faced and gear up for the coming year.


That web team was restructured into 3 functional teams was probably the most significant course of events. That was to reflect the fact that as the company grew up, we were on the edge of having to trade the dynamic nature for stability. The only way to keep us flexible yet manageable was to split up based on our field of expertise. Smaller teams allowed us to move faster, specialize in a few things that mattered and keep shipping.

If 2011 was a year of inception, of putting thing together, 2012 was the time we took bold moves to catch up with the world. We had chance to observe a spectacular raise of JavaScript (Backbone in particular) and Erlang in our work. We were also taking steps to master our tool kit. There had been more pull requests for Warp last year that ever. Finally good use of CSS framework and preprocessor enabled us to share the burden on designers' shoulders.

The fruit of that effort was reflected on the projects we took. In 2011, BnB was the only star project where multiple platforms were integrated for a world-class solution. In 2012, we were tackling complicated projects requiring multiple subsystems or possessing sophisticated business logic with confidence. Since Brendon left, we have been living without an official CTO, and the world didn't end yet.


Web continued to be the major revenue stream of the company. The number of technologies that we used, processes that we put into practice and projects that we shipped were steadily raising. However, in order to achieve the growth, we have made a number of decisions that eventually ironically prevented us from growing. Many of these decisions were probably right at the time, and on their own merit would not have caused us trouble. But in this case, the sum is much greater and more damaging than single pieces.

For example the decision to split up the team solved the problem of balancing flexibility and stability. At that time we overlooked the need of cross-team communication and cross-project overseeing. This reduced our ability to allocate resources to mission-critical projects. In general, lack of internal corporation slowed us down.

In order to retain our shipping pace, quality was sometimes traded for speed. The amount of technical debts stacked up, evolved into GigaDB and the likes, and gave us much headache.

And in order to keep the revenue stream that supported our working environment, we failed to allocate sufficient resource in PM. And even when people had the ball, the initiatives to take up PM works, every data about project and customer was jammed up in one single bottle neck.


However. Despite all those troubles and an unstable macro economy of 2012, we managed to grow so much horizontally and vertically. It doesn't matter which team you are in, developer, designer or administrator, applause yourself, you deserve that.


The end of lunar new year is drawing near and Tet can be felt in every breath. Yet in the last few days, a news has spread among us, shock us and demonstrate that event giants can collapse. To prepare for 2013 whose challenges are expected to be even more ferocious, we need to learn from last year, make up resolution for coming months and most importantly, better ourselves.

As a human being, try to fulfill your potential. Say "fuck you" to the status quo. Learn another language, explore another text editor, write a blog. Don't limit yourselves on what you can learn and want to do.

As a staff, we want to continue build up this great place with you. Lets put extra emphasize on code quality, make fewer GigaDB or Solar. Let us know you are growing and allow us to learn from your growth via TechTalk and HackDay. And finally better our ability to run business, from communication to estimation and even PM.

I wish you all a successful year ahead.

How do you write a job ad.

Wrapping up 2012 in preparation for Tet holiday brings back many cheerful memories. I had a chance to rewind things have gone by and numerous lessons the job has taught me, many in the hard way. I ran across this little job poster that we made back when there were 6 people in the team.

Cogini Vietnam first job ad 2011

The poster looks nice, but it fails all the current standard of us, mostly fails to envision the job and motivate potential applicants. But it was fun, crawling in the dark for a path. As we all grow up and get wiser, our approach becomes more methodical and less ad-hoc. Time goes by and there are a couple of things about job ad that I would like to share. To support my points, I crawled sample job ads from famous local and overseas tech firms.

Job ads

A well-written job ad helps a candidate screen him or herself in or out for the job. A job ad template typically consists of

• Main attractor
• Deliverables and activities
• Essential quality, preferences, and skills
• Contact information

Major attractor 

Major attractors define what will attract potential candidates. This is the most important pitch that sell candidates on the company. It is a common belief that this is the place to show off. I agree. Reading job ads, candidates are constantly asking themselves "What's in there for me?". Answer that one question right out of the gate will grab people's attention and keep them engaged. Skunkworks addresses itself as a Silicon Valley startup in the heart of Saigon and takes pride in the work environment it has to offer.

Our team is everything, so at Skunkworks.vn we pay close attention to the work environment:

• 100% product/innovation-focused startup company with experienced founders

• Flat organization - don't come to work for us if you want minions working for you!

• Open communication - we listen to our team

• Friendly, flexible, and fun culture 

But I also agree that bullets are a very ineffective way to deliver information intended to be recalled and acted upon. Images and stories are way better communication means to deliver the sales pitch.

Each company is a unique combination of its people, fields of expertise and culture. Any passionated employee can go rampant about the story of his company's genuineness. However it is well established that we can only hold small amounts of information in short-term, or "active", memory. Among all the possible number of key message points one can make, three (3) emerges as a magic number. Studies showed that the number of items we can easily is between three and four. In reality, number three is more common than four. Every great movie, novel or play has a three-act structure. Comedians know that three is funnier than two. The rule of three is one of the most powerful concepts in communication theories. 

Google and Twitter follow the rule of three closely. Each provides candidate three key messages about careers, with the third point prompts for action.

Google's Job Page

And Twitter's

But as programmers are spoiled these days, firms are going all their way to attract top-notches, there are more to say about the offer than just main attractors. Other benefits and perks are developing upon money (present and future), career growth, work/life balance and working environment (Have you ever turned into a green-eye monster just by looking at Google's office? I have). This is one of a few areas where bullet points are proven to be useful in attracting people attention. Bullet points are specialized in paralyzing readers with information. But the list is filled with benefits and perks, it leaves the candidate with an overall impression of "many good things".

I find Dropbox's job ad delightful

Deliverables and activities 

Deliverables and activities effectively describe the kind of work the hire is expected to perform. I usually construct this part with the same strategy used on main attractors, start with a short paragraph and follow up with bullet points to cover more details. A short paragraph describing the nature of the work is not something common among local tech firms' job ads. But as jobs are getting more diverged from each other, the need for a story to enable candidates to envision themselves in the work is receiving more care. Bullet points are hard to avoid in this section, but make sure that you have someone scan the list to make sure it reflects the actual job.

KMS' job description is pretty lengthy, but after all doesn't reveal much about the work

Dropbox once again scores with nice combination of story telling and bullet points

The more high level the position is, the more sophisticated the preface paragraph should be and the fewer bullet points. This is simply to reflect the fact that a senior position requires a completely different set of skills and responsibility from a entry-level job. At some point, the criteria of evaluation is no longer doing what one is told well and on-time, but the ability to take on unassigned work and responsibility, and deliver what one's firm values. Even though I am not fond of Rocket Internet as a business model and a company, I am convinced that its job ad for CTO in Vietnam is a well-crafted piece of work.

As our CTO you are a well-rounded strategic thinker responsible for establishing, executing and communicating the technical vision across all aspects of software development. You will be leading a top-flight technology team while overseeing research, development and quality management. Furthermore, you will be responsible for the establishment of an appropriate architecture and technology for our innovative projects. By embracing a culture of continual improvement, passion for technology and cross functional collaboration and communication, you will play an integral role in the company's strategic direction and growth. At Rocket, you will help us develop fast growing Internet companies and benefit from the broad expertise of our high-energy IT team. In return, you can expect an attractive salary and various opportunities to grow on, both, a personal and professional level.

Essential qualities, preferences and skills

They define the company's interest in the candidate professionally and personally. Many hiring managers get too deep into describing their ideal employees here that I often find this section of job ads unclear and unrealistic. This part is best done by giving details about how the candidate with perform activities and deliverables. E.g instead of writing "Must have negotiation skills", try "You will negotiate with suppliers and with the project team about milestones and tradeoffs". The second approach brings a breath of reality to the ad.

Skunkworks' ideal employee sounds pretty much a workplace super hero

• Open-minded and comfortable in an intense, high-growth startup

• Innovative and love to find new ways to resolve problems

• Able to adapt according to changing project schedules

• Self-motivated and can work independently

• Totally committed to quality

• Professional and responsible

• Able to communicate well in English (written and spoken)

In general, I try to avoid putting many soft skills here but the critical ones. This poses a similar problem as hypothetical questions in interview. Hypothetically, I am perfect.

Also unless I kept running into candidates with really bad cultural fits, I am not a big fan of explicitly points out desirable personal qualities. Again, many people have the tendency to idealize their personality. On the other hand, different personal qualities do not always need bad cultural fit (and diversity should always be embraced). Best personal qualities requirements I have read do not have their own section but are embedded into verbal descriptions in job ad, which in turn drive the way readers perceive the firm in the way it desires.

Similar to Dropbox's, Foursquare's job ad makes use of plenty verbal description to reflect the company cultural fits.

There isn't much to say about Contact Information so I am wrapping up my writing here. Hope you find it useful.

---

Note: By the time the post was finished, Skunkworks.vn was going through a restructuring process and its website therefore was inaccessible.

Code Refactoring P.1