Trust in Agile Environments

The author Frank Crane wrote “you may be deceived if you trust too much, but you will live in torment if you trust not enough.” It is not easy to let go of fear and doubt. Yet, unless you decide to live as a hermit crab, you must release some degree of fear and doubt if you are to function at all in modern society. If you are working as part of a software development team, you must have a great deal of trust in your fellow developers, testers, managers, business analysts, and other team members. Functioning well on any agile team requires a high level of trust.

Types of trust

It may help to define trust by describing what it isn’t. There are two basic types of mistrust; functional and dysfunctional. The latter is an extreme form of mistrust that can approach pathology – feelings of persecution, conspiracy theories, and excessive thoughts on the actions of others. Although distrust has traditionally been viewed as a bad thing, there can be some valuable benefits of mistrust; this is called functional mistrust. For example, you certainly wouldn’t loan your car or lend money to just anyone. We have all had experiences where we misjudged others, only later to be exploited. A certain level of functional distrust allows us to set boundaries, and also to prevent excessive group cohesion which may hinder good decision making.

However, a deficiency of trust can lead to many adverse effects: lack of cooperation, lower employee satisfaction, loss of commitment, and in its extreme form retribution and openly hostile behavior. It is when trust becomes this dysfunctional that we must break the cycle and analyze, understand and treat its root causes.

There are two basic kinds of mistrust: calculus based and identification based. Calculus based distrust is “confident negative expectations of another's conduct grounded in impersonal, arms-length transactions where the overall costs of maintaining trust are expected to outweigh the benefits of maintaining trust.” This type of mistrust is basically a cost-benefit analysis, a calculation of the return on investment of personal relationships. We see this all the time in our daily lives: we might trust the neighbor kid to mow our lawn, but we wouldn’t let them touch our computer. Establishing boundaries, setting expectations, crisply defining roles and keeping the lines of communication open will help to temper this type of mistrust.

Identification based distrust is “confident negative expectations of another's conduct grounded in perceived incompatibility of closely-held values, dissimilar or competing goals, and/or a negative emotional attachment.” In this type of relationship, we see little in common with one another. The other individual or group in such a relationship may even be seen as an adversary. This type of mistrust can be the most damaging as it is extremely personal. Although we would love to minimize our interaction with people who do not hold the same values as we do, we are often forced into situations where we must. On these occasions, using a third party as an intermediary can often help facilitate communication.

Signs of mistrust

Nosce te ipsum. Begin by examining yourself and your motives. Are you a mistrusting person? Here are some signs to look for:

• Constantly needing to explain and justify your own personal behaviors, viewpoints, and preferences, when others don't need to hear it
• Having a reputation as "unusually conservative," and being uncomfortable with risks and major change
• Chronically saying "No" to new situations or invitations
• Compulsively “mind-reading” others: assuming “I know what you’re really thinking, feeling, or wanting”
• Fear of the unknown

The second place to look for mistrust is in the environment. Are you in a place that has a low level of trust? Here are some signs to look for:

• Lack of enthusiasm following announcement of new initiatives by senior leaders
• Increasing complaints about policies and practices controlled by senior leaders
• Managers beginning to question decisions and actions of senior leaders
• Increased grumblings by groups of employees
• Morale issues
• Increased mention of senior leaders in exit surveys
• Active resistance to leader initiatives and change efforts
• Workplace gossip
• Nothing can get done without the boss’s approval, or conversely, nothing can get done unless employees go behind the boss’s back
• Performance reviews are not done on time
• Face to face communication takes second seat to instant messaging, email or text
• Too many meetings
• High turnover

How can you improve trust in your respective teams? Let’s start by examining how trust is broken before we look at how we can build it.

Breaking Trust

It is far easier to break trust than it is to build it. Trust is something we all value, but it is exceedingly fragile: it takes months and years to build, but only seconds to destroy. Here are some sure-fire ways to destroy trust in an organization:

• Withhold feedback and information – Don’t tell teams exactly what is expected of them. Don’t tell individuals exactly what is expected of them. Don’t give feedback on their performance. Make them question whether they are doing a good job. Don’t give them information they need to get the job done. Hoard information on the premise that people are not competent to understand it. All of this sends a strong message that people are untrustworthy.
• Dismiss evidence of the team’s capacity – Despite objective evidence of the team’s capability to deliver, impose requirements on them that don’t take it into account. Then, punish the team when they inevitably fail.
• Micromanage –Based on the assumption that people are basically incompetent, or less competent than you are, micro-manage their work to make sure they stay on track. Which, inevitably, they may not, because of the overhead of the micromanagement.
• Act and speak inconsistently.
• Seek personal rather than shared gain – when others come to believe that your own agenda is based on personal advantage, your motives immediately become suspect, and trust breaks down.
• Lie - Or perhaps worse, tell half-truths - no one trusts a liar.
• Be closed-minded – don’t actively listen to people. Assume you know better than they do, and that your ideas are always right. Based on this thinking, make all decisions yourself and don’t allow your subordinates to make them; or worse, reverse the decisions they do make. Don’t be open to new ideas and new ways of thinking about problems.

Building and Growing Trust

So how can we go about building trust in an organization? Here are some ideas:

• Establish and maintain integrity - Do what’s right regardless of the personal risk. Value and practice honesty always in your communication with others.
• Be consistent and predictable - Keep your commitments. Deliver what you say you will when you say you will. Meet your service level agreements. There should be no surprises at any milestone. Be respectful of other people’s time when conducting meetings.
• Consider all employees as equal partners – Americans have always been very independent. Our culture celebrates individual heroes. We are far more egalitarian and less autocratic than any other culture on the planet, and when autocratic systems are opposed on us, we tend to rebel. By treating everyone in a company as equal partners, all focused on the same mission and vision, you cannot help but engender a trust based environment.
• Consult with the others before you make commitments on their behalf – before signing others up for work, consult with them. Listen to the doubters; they will raise red flags and project risks you need to track.
• Communicate – Make sure the team exactly knows your needs. Communicate often. Be transparent. Be open about changes you are making. Actively seek feedback on everything. Better to over-communicate than communicate too little.
• Establish boundaries and test them – Have well defined (written) roles and responsibilities for everyone on the team, and actively communicate when tasks come close to the borders of those roles. Know exactly what is expected from every role and every discipline at every phase of a project.
• Take responsibility – Apologize if it’s your fault. Own up to your mistakes and work to improve.
• Be empathetic to those around you – Actively listen to others. People trust good listeners.
• Develop and demonstrate competencies – Always be continually improving in your chosen profession. You will more likely be trusted if you are seen as the “go-to” person in a feature area.

Trust in software development

In a traditional waterfall software development environment, we see the symptoms of the lack of trust: they are document heavy, senior management defines the designs and architecture as opposed to experienced software architects who know how to do the job better, and managers own the command and control as opposed to the teams building the software. For every task there is a methodology that is simple, neat, and wrong, and waterfall is one of those methodologies. Inexperienced managers will often ignore history and fall back to this less agile method because they believe all of the up-front work will save them, or because they don’t trust the teams to build what they actually want.

Agile teams are different. One of the basic precepts of an agile environment is trust and teamwork. Agile teams craft optimal solutions over a period of time, and the people who manage them need to create optimal environments in which they can function. Agile teams iteratively maximize value.

There are basically two different opinions on the subject of trust in an agile environment. The first states that you must initially trust the team to create the software in the first place. The second believes that trust is created out of the environment of the agile team; that trust is built by consistently and reliably delivering incremental value that is visible to stakeholders. Personally, I think that both have to happen: teams have to start somewhere, so they must be imbued with at least some level of trust in order to function. By then functioning well and iteratively delivering on commitments, trust is also iteratively built up.

Trust, But Verify?

We’ve all heard this saying. It was a signature phrase of Ronald Reagan when discussing relations with the Soviet Union. I think this axiom is oxymoronic: the Compact Oxford English Dictionary defines trust as acceptance of the truth of a statement without evidence or investigation. If you have the need to verify that something was done, then I submit that there wasn’t much trust in the first place. But that doesn’t mean we don’t check our work with appropriate QA measures. People make mistakes, and it’s QA’s job to protect our customers from those mistakes.

At the end of the day, trust is a choice. As such, it will always be accompanied by risk. A willingness to take risk with others makes a powerful statement to them. Lao Tzu said, “He who does not trust enough, will not be trusted.” Begin by trusting yourself, then others. We don’t really have much choice if we are to function at all, but the rewards will always outweigh the risk.

Agile Milestones

I’ve been engaged in a lively debate at work about the role of milestones in an agile environment.  How and when should they be used?  Is the definition of a milestone in an agile environment any different than in a traditional project?  From the outset, I want to go on record as saying milestones are important, but doing them incorrectly can lead us down a slippery slope away from agility.

We take the concept of a milestone from the numbered markers placed along roads, which provided reference points to travelers, and date back to the Roman Empire.  It is a convenient metaphor to apply this concept to a project management, but when applied to a software development effort, the metaphor sometimes tends to break down.  In the Project Management Book of Knowledge (PMBOK), a milestone is defined as “a significant point or event in the project” (p. 136).  This definition doesn’t really help us in the discussion, though, as further interpretation of the definition often leads to disagreement and confusion.  The PMBOK gives us a little more information, though, when it further describes one attribute of a milestone: whether it is mandatory (i.e. contractual) or optional.  I think we can use this key differentiator to help guide our discussion.

We must first realize that a traditional, waterfall view of projects, which have an “on time/ in budget/manage to the plan” approach, requires deterministic activities to be successful.  A great deal of modern project management still successfully uses this approach to these types of projects; construction is a good example of this.  We can see this in the approach to risk management.  Risk management in construction is fairly well understood – for example, actuarial models exist for most types of construction projects.  Plans for deterministic projects are generally static, the focus is on controlling cost, and the amount of time it will take to complete the project is, for the most part, predictable.

However, software development is a non-deterministic activity: it is a process of change, invention and design, not one of manufacturing.  It has “unknown unknowns” at the start of a project, some of which can never be fully understood until project completion.  Edison filled more than 40,000 pages of notes and tested over 1600 materials before he had a light bulb that would last just 40 hours.  Tesla had been trying to solve the puzzle of a rotating magnetic field for years.  One day while walking through a park in Budapest, reciting Faust, the solution presented itself to him in a blinding flash of insight.  Imagine asking Edison if his project to invent the light bulb were “on time”, or asking Tesla when he would complete the invention of the AC induction motor.  They would have probably scoffed at you.

Fortunately, software development is not completely non-deterministic as it is not purely invention.  We don’t have to re-invent databases, recursion or Unix; we stand on the shoulders of giants such as Jim Gray, Edsger Dijkstra and Dennis Ritchie for these and countless other advancements in computer science and engineering.  We can leverage this knowledge and re-use tools and design patterns when developing new software, and so not have to completely re-invent the wheel every time we develop a new system.  Good developers know a broad array of these design patterns, know how to apply the right tools and technologies to solve business problems, and more broadly align IT with business needs – one of the primary functions of the CIO (or in some organizations, the CTO).

However, software development projects remain at least partially non-deterministic, and they cannot be fully managed with traditional project management methods.  Success must often be defined as delivering as much value as possible in a reasonable timeframe.  Plans for non-deterministic projects are volatile, the focus must be on making tradeoffs between value and cost, and the amount of time it will take to complete the project is either unknown or only known within some degree of uncertainty.

Are milestones in software projects then useless?  I don’t think so.  To understand whether a milestone has value or not, as my friend Kenny Rubin said, we need to look at the economics.  Ask: what is the economic value of meeting the milestone?  What is the cost of being early or late?  It might be helpful to look at two different examples.

Example 1: Milestones with time-based economic impact

In the first example, imagine that a team needs to complete features in a software project for a demonstration at an upcoming annual convention.  If the software is late, the company will lose an opportunity to demonstrate new product features and corporate technology progress to a large audience.  Conversely, if the product is completed too early, then there is also value lost in terms of opportunity cost.  Figure 1 illustrates a simple value curve based on time.  One can of course argue the shape and slope of the curves based on a number of factors: a contract might specify that if a deliverable is late, penalties may be incurred; as well, having the software completed late may still add value (just not as much as if it were delivered on time), but the point is that maximum ROI is achieved by (just) meeting the milestone.



Figure 1: milestone based on economic value

Example 2: Milestones with no time-based economic impact

In the second example, let’s assume an operations team would like to have a tool developed to automate some part of the IT operational process.  Not having the tool is costing the company operational expenses in terms of manual processes that need to be run by hand every month.  In this example, the company is continuously paying the operational cost of the manual process, which for the sake of argument we can assume does not change over the short term.  Figure 2 illustrates this cost curve.  Given that there are probably many other projects queued up on which the team could work, and the team executes them based on the ROI of each project, as long as the development team is reliably producing value at a good, sustainable rate, what is the purpose of having milestones at all for this type of work?

Figure 2:  Milestone ROI with no time-based variability

There are many old-school arguments against this, but the most typical goes like this:  “How do we know if a team is meeting its commitments?”  This sounds good on the surface, but if we analyze it deeply we find an insidious belief: that meeting constraints is more important than delivering value.  The undercurrent of this belief still pervades a large part of the software industry, and agilists must constantly be on their guard against it, and work to educate others about it.  This is not easy; I have struggled to do this for over a decade.  I have not always been successful.

There is a simple reason this thinking is insidious: incentives work.  If the primary incentive for a software development team to meet delivery dates, then they will do their very best to make sure those dates are met.  But the upshot is that, in their quest to lower risk and do everything possible to meet dates, several things happen.  First, estimates become inflated.  What would have been estimated at 2 days now might take two weeks.  Let’s be clear: developers aren’t lying or being underhanded when they do this - they are simply reflecting the non-determinism of their craft back into their estimates in an attempt to meet the success criteria given to them.  Second, Parkinson’s Law starts to operate: “work expands so as to fill the time available for its completion.”  Focusing on constraints instead of value will cause developers who complete tasks early to continue to add features, bells and whistles until time runs out.  We call this gold plating.  Again, developers aren’t being underhanded when this happens.  Who wouldn’t want to say “not only did I get the work done on time, but I went the extra mile and added X!”

When we incent value instead of constraints, this does not happen.  Developers give the best estimates they can and strive to deliver as much value as possible in a given time frame.  The difficulty, of course, is measuring value.  Admittedly, it is much easier to look at a calendar and ask whether a team met a date than it is to do the really hard work of measuring how much value was delivered.  Personally, I think it’s worth it.  Business members of teams need to participate in the process, roll up their sleeves and do the hard work of calculating business ROI.

Command and Control in Agile Project Management

D-Day, or Operation Overlord, is one of the most studied military campaigns in history. There were a number of factors that led to the success of the Allied invasion, and one in particular that contains a lesson for those of us who participate in agile project management.

The Allies landed on five beaches in northern France on June 6, 1944, code named Utah, Omaha, Gold, Juno and Sword. At Utah, the 4th division landed in the wrong spot, several miles south of their planned landing. In command was General Theodore Roosevelt, who made the snap decision to get his troops off of the beach, press inland and try to connect with the 101st and 82nd airborne divisions rather than try to follow the initial plan and march north. At Omaha, General “Dutch” Cota and the 1st division were pinned down by the entrenched German 352nd infantry, whose position was missed by allied intelligence. Omaha was the scene of the bloodiest fighting on D-Day as a result. They went off-plan and called for naval destroyers to maneuver dangerously close to the beach to bombard the 352nd; this allowed allied soldiers to climb the cliffs and remove the machine-gun nests and 80mm guns which had held the 1st division back. At Gold, Juno and Sword beaches, forces under Field Marshal Montgomery met little resistance, and he decided to press these forces inland, taking several miles of territory on the first day of the invasion.

On the German side, confusion reigned. The 21st panzer division was stationed just south of Caen, and was commanded by Field Marshall Rundstedt. Although technically reinforced by the 12th SS panzer division located some 30 miles to the east, the 12th SS reported directly to Hitler, and refused requests from Rundstedt to help until receiving orders. At the time, Hitler was sleeping at Berchtesgaden, and did not wake up until almost noon. Frantic phone calls made by German field commanders for instructions were not answered until hours later. It was not until 4pm that Hitler gave his approval for the panzer divisions to participate and counterattack, but by then it was too late; the Allied forces had a firm foothold in France, and within a week their number of forces would triple.

General Eisenhower, the Supreme Allied Commander, gave us a powerful lesson in leadership and the role of command and control in an environment where there is rapid local change. He authorized all of his field commanders to make on-the-spot decisions based on local conditions without having to consult Supreme Allied Headquarters. In stark contrast, the inflexible command and control structure used by the German forces was a significant factor in their undoing; while the Allied commanders made local decisions and advanced, German forces were paralyzed by an inability to act.

Eisenhower was quoted as saying “Plans are nothing; planning is everything.” It is as true for business today as it was for the military then (and now). This lesson is observed in the training of modern American fighting forces: the most valuable trait - the one that is searched for and trained in military commanders above all others - is initiative. “Indecision is a virus that can run through an army and destroy its will to win or even to survive,” wrote Wendell Mayes. At higher levels, control is an illusion; strategy and coordination driven by mission and vision must govern. Trust is key to making this work: trust in local commanders and troops – or local managers and teams - to make decisions that incrementally move their effort toward the fulfillment of the mission.

Requirements? We don't need no stinkin' requirements

Sometimes, this is true. I’ve written hundreds of programs, sometimes very large ones that took me years to write, without a single line of a requirements document. We only really need requirements documents when one person needs to communicate to another what needs to be built, or to help organize our own work.

Requirements documents can be the bane of a software developers existence. Poorly written, confusing, ambiguous requirements have been the genesis of the demise of many a software project. More than once, I’ve had developers come to me and complain about poor requirements, or complete lack thereof. “How are we supposed to write this software without requirements?” comes the oft-heard mendicant whine. Let’s look at solutions to this dilemma.

Non-Agile approaches to the problem
1. Complain that the requirements stink. Complain that the requirements are missing major pieces, are poorly written, are missing functionality, don’t really address customer needs, and were basically written by idiotic product managers who just don’t understand the technical complexity of the software. Then, stumble along and write code that attempts to meet these requirements in a doomed attempt to make the product manager happy.
2. Give the requirements document back to the product manager with a detailed write-up of what is wrong. Create requirements templates for the product manager to use for future requirements documents. Install processes that strictly formalize the requirements process. Refuse to accept any requirements or specifications unless they are perfect. Teach those product managers not to mess with IT.

This approach often builds an adversarial relationship between development and product management, and usually in environments such as these you see similar adversarial relationships between development and QA, where everyone is so completely focused on adhering to the letter of the requirements and adopting process for process sake that the customer is forgotten.

Agile approaches to the problem
1. Remember three of the four agile principles: value individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, then
2. Go to the product manager. Ask them to clarify what they wanted. Spend 10-15 minutes drawing workflow diagrams and/or UI on a white board. Take a picture of the white board with your iPhone and send a copy to the product manager. Go write the code.

This approach, besides solving the problem more efficiently, yields some additional benefits. It builds a trust relationship between product management and development. Customers love being asked what they want, and not just handed what you think they want or meant. A mutual learning environment will blossom from this type of relationship, where product managers learn more about how software is built and can be more appreciative of problems when they occur, and developers get a deeper understanding of the customer. It’s win-win.

Ruthlessly applying the lean engineering axe
You might think that the previous example applies lean engineering, but we can go further. Try this scenario: sit down with the customer in front of your computer, and write the code as the customer elucidates what they want moment by moment. Have QA sit with you as you write, checking your unit tests as you go. Sound farfetched? It’s not. This model has been used successfully in agile environments for years. They are “a strong and direct expression of the Lean principles of pull, flow and value" (http://leansoftwareengineering.com/2009/04/07/feature-crews/).

The two approaches outlined above are somewhat idealistic, and reality usually falls somewhere between the two based on a number of factors, including organizational size and trust (more on trust later). We must always be careful not to fall into the trap of installing process for process’ sake, tools because we like new shiny toys, templates and documentation or any artifact because we think it will add customer value. Agile and lean are always trying to remove wasteful processes. Always be examining your customer value stream, and feel free to apply the lean axe when it’s needed.

Technical Debt

What is Technical Debt?

The term "technical debt" describes the obligation that a software development organization incurs when it chooses, knowingly or unknowingly, a design that saves time or money in the short term, but in the long term increases complexity and ends up being more costly. There are two basic kinds of technical debt; intentional and unintentional. The first occurs when an organization makes a conscious decision to optimize for the present rather than the future.

This is not always a bad thing. For strategic reasons, companies often incur technical debt when time to market is critical or to preserve startup capital, preferring to pay for the expense out of later (and hopefully greater) revenue than critical startup resources. Eventually, however, the debt must be paid.

At best, technical debt can be annoying, making maintenance difficult and slowing productivity. At its worst, it can make forward progress impossible and destroy productivity. Steve McConnell acknowledges that “if a company’s technical debt grows large enough, eventually the company will spend more on servicing its debt than it invests in increasing the value of its other assets. A common example is a legacy code base in which so much work goes into keeping a production system running (i.e., ‘servicing the debt’) that there is little time left over to add new capabilities to the system.” (McConnell, 2010) The industry estimates that it spends up 80% of its total cost on software maintenance (Pressman, 1992). We need to understand technical debt better, adopt in only when we have to, avoid it when we can, know it when we have it, and know what to do with it when we have it.

Signs of Debt

There are a number of ways you can tell if your software has developed significant technical debt. Here is a list of things to look for:

1. Large backlog of bugs
2. Fragile code
3. Unmaintainable/uncommented code and/or outdated comments
4. Security issues
5. Missing infrastructure
6. Lack of documentation and standards
7. Over-reliance on tribal knowledge for maintenance
8. Lack of understanding that you have technical debt and/or ignoring it

As the technical debt increases, and the size of the code base increases, it is important to note that the maintainability costs exponentially rise, and eventually reaches critical mass such that the code becomes unmaintainable. Every new line of code may be a potential new mine in a mine field; a ticking time bomb in a brittle code base where layers of technical debt have created ever increasing levels of risk when the source code needs to be touched. And the source code usually does need to be continually touched to meet continuing customer needs.

Fast, Cheap, Good: Pick Two

It’s been said that lowering quality lengthens development time. This is because the burden of the technical debt must eventually be paid off. If you’re a cowboy coder and you simply hack some code out, it may seem like you get the code out sooner, but the technical debt that must be paid off is much greater compared to the senior developer who creates a full set of unit tests, good designs, and code that is fully performant, secure, reliable, and maintainable from the start. Boehm (2000) discovered that as software proceeds through the life cycle it becomes progressively more expensive to change. Software does not get better over time; it gets worse - unless you actively take steps to fight the decay. In a word, software rots.

Unfortunately, many have interpreted this inflating cost curve to mean that you must produce perfect requirements, followed by perfect specifications, followed by perfect designs, etc. This is not what Boehm’s work suggests. Rather, the software base must be kept resilient and maintainable to keep it prepared for any further customer changes. Code bases begin to accrue technical debt when they are “moved into maintenance” and stop being refactored by the development team as new features are added and bugs are fixed. Refactoring is the process of restructuring an existing body of software without changing its external behavior for the purpose of improving readability, maintainability or to reduce complexity. Refactoring and unit testing are a primary means of fighting software rot.

Once you recognize that you have technical debt and that you have been accruing it, have a transparent discussion with your business owners about the costs of the technical debt. Chances are they may already be aware of it, at least indirectly, by recognizing the signs I’ve listed above. Make sure they understand what the technical debt of the code base is costing the business. They may have unrealistic expectations from software developers working on a legacy codebase. Other teams may be able to develop new features faster (or will be able to develop new features at all), and this may be frustrating to them. Part of this buy-in will be to develop some long-range thinking on the part of the business owners. This type of development is an investment that will yield dividends in the form of higher productivity and higher quality.

Paying off your technical debt

In order to pay your technical debt off you must first understand it. Where is your technical debt occurring, and how is it affecting your business? Do you have high maintainability costs? Look at your bug regression rates. Look at your backlog of bugs. Look at your code metrics. Do you have security issues? Do STRIDE-based security threat modeling on your application to understand potential security threats. Do you have adequate technical documentation and artifacts with respect to the product architecture? Do you have adequate unit tests in place before you start refactoring the code base? You need to list out your technical debt and make it durable by placing it in a permanent, visible location, like a team SharePoint site.

One thing developers love to do is to throw the old code out and start from scratch. I will agree with Joel Spolsky (2010) on this point and suggest that this is almost always a stupid idea. Legacy code may be hard to read, poorly architected, and full of bugs, but it’s been used, tested, and contains a great deal of real-world domain knowledge, bug-fixes and market leadership that will be discarded if you start from scratch. Better to iterate and refactor.

After understanding the nature of the technical debt you have, you need to build a business case with the entire team, including the product owners, on paying down the technical debt. Like most debt, you won’t be able to pay it all off at once, but a little at a time. After you have buy-in from all stakeholders, allocate a small portion of each sprint and pay off the most profitable debt items first. As long as you are paying down more debt than you are accruing, you will be incrementally improving quality and decreasing risk.

Working off technical debt can be a great way to boost team morale. If your Product Owners can be convinced to surface product backlog items dedicated to retiring technical debt, or even devoting entire sprints to the effort, measurable performance and quality improvements can be made which can be highly motivational for the entire team. These improvements will inevitably show up in improved customer satisfaction ratings as well.

References

Boehm, B. (2000). Software Cost Estimation with COCOMO II. New Jersey: Prentice Hall.

McConnell, S. (2010). 10x Software Development. Retrieved May 17, 2010 from http://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx.

Pressman, R., (1992). Software Engineering: A Practitioner's Approach. New York: McGraw-Hill.

Spolsky, J. (2010). Things you should never do, Part I. Retrieved May 17, 2010 from http://www.joelonsoftware.com/articles/fog0000000069.html.

Hansei and Kaizen in the Second Quadrant

Last Friday night, Ben, Pete and I were having drinks, and Ben mused that we needed to do a better job of training for our software applications. “Perhaps better training videos are the solution,” he mused, sipping from his coke. Ben was demonstrating Hansei. He often does this when we are out drinking, much to the annoyance of Pete and I, who were just trying to enjoy each other’s company and drink.

Addicted to the First Quadrant

I thought deeply about Ben’s statement. Often – too often, I believe – we operate in a hurry-up, get-it-done, meet-the-deadline, schedule-driven environment and don’t allocate enough time to do the things that might actually prevent us from have to spend time in that environment in the first place. Don’t get me wrong; it’s an addictive, adrenaline and caffeine fueled world, filled with excitement, and at the end of the day it leaves you feeling like you got a lot done. But all too often, getting caught up in the rush of fast-paced delivery leads to an environment of poor quality software, unhappy customers, broken relationships and a myriad of other problems that stem from poor planning, lost opportunities and lack of preventative measures. What can be done? What do these Japanese words mean anyway, and what is Dave getting at with quadrants?

The 14th Principle

Hansei is a Japanese word which means “relentless reflection.” It is a central idea in Japanese culture, and it is about acknowledging one’s own faults and always trying to correct them. Kaizen means “continuous improvement,” and practiced together they help to build a learning organization. This is the 14th of as many principles of management practiced at Toyota. We practice a little hansei and kaizen already in our software development today. We have a sprint retrospective where examine how we can improve our processes, and we have a built-in improvement step in the inspection process as well. I have seen this at work in the few months since it has been installed. All of the teams have already been getting incrementally better at estimating and scheduling their work, and measurably improving their quality. I want to acknowledge that improvement that the teams have made here. It’s good, but there is still a lot of room for progress.

The Second Quadrant

Stephen Covey, one of my favorite authors, classifies tasks in four quadrants:


The first quadrant contains those that are both urgent and important. Fighting fires and crisis management are examples of first quadrant activities. The second quadrant contains tasks that are important, but not urgent: planning and relationship building are examples of quadrant two activities. How much of your day is spent in these two quadrants? Do you balance them appropriately? (If you’re spending any time at all in quadrants three or four, you’re probably wasting your time).

One of the problems with operating in the first quadrant is that it’s addicting. It’s very satisfying to go to work day after day and solve a seemingly endless wave of problems. You look like an aggressive problem solver; someone willing to take on difficult situations and get the job done; a “can-do” person. Unfortunately, it is not the best way to deal with the situation, as it usually leads to the same thing happening over and over again (well, at least you have some job security). A better approach is to find and eliminate the root cause. This will drive first quadrant activities into the second quadrant, and bring some sanity to the workplace.

Fix it Once

Kiichiro Toyoda said that every defect is a treasure – but only if the root cause of the defect can be found and corrected across the company so that it does not occur again. The Toyota Company empowers every employee to completely shut the production line down when a defect is found in order to discover the root cause of the defect and prevent it from re-occurring. This is a very powerful paradigm, and in part has led to the highest quality automobiles in the world. Do you feel that way? When you see a problem, do you feel empowered to stop what you’re doing and make the changes necessary so that the problem never happens again? Why not? Wouldn’t that be adding real value to the company? Sun Tzu’s wit is not lost on us when he said, “When trouble is solved before it forms, who calls that clever?” Yet the best employees are those that do exactly that.

It will be interesting to observe Ben as he practices hansei and kaizen in the second quadrant and solves the training issue. I suggested he work with Dave Skender and work to install these pieces as potential deliverables within the MedAssurant Agile Process. These Work Products will need Owners, will belong to a Discipline, and be assigned to a phase of the Process.

More to the point, it will be interesting to watch each and every one on the team as you practice hansei and kaizen in the second quadrant and solve problems before they turn into fires. Franklin was right when he said “an ounce of prevention is worth a pound of cure.” We must, each and every one of us, practice hansei and kaizen in the time that we have, in the place that we are. That, or inevitably be consumed by the unquenchable fire of the first quadrant.

Why Traditional Software Planning Fails

Mike Cohn, founding member of the Agile Alliance, eloquently describes the main reasons that software projects have failed (Cohn, 2006). Here are a few of the main ones:

Lateness is passed down the schedule – traditional software planning processes focus on the dependencies between activities. As a result, best case is that everything finishes on time, but because of dependencies, a slip in any one activity is sure to cause a slip in the entire schedule. What is worse, such slippages are not simply additive. Lateness on one task may ripple over to many other dependent tasks, making all of them late, multiplying the lateness by a factor of the number of dependencies in the system.

We ignore uncertainty – traditional software planning fails because it assumes initial requirements can lead to a perfect and complete specification, and that all unknowns can become knowable at the start of a project. It fails to take into account that customers may change their minds, come up with new requirements, or change requirements during the course of a project. The old school assumes that manufacturing methods can be applied to software development; but studies by the Standish Group (1995, 1998) and others conclusively show these methods don’t work, no matter how rigorously they are applied. Usually we are implementing a new idea or design, which has unknowns and risks, but we need to achieve exacting quality. It’s common sense: we don’t have perfect information, so expecting the perfect estimate or the perfect schedule is silly (I am quoting Eric Brechner, author of “Hard Code,” and Development Manager of Microsoft Office). It is an ill plan that cannot be changed.

Estimations become commitments – when we give an estimate for the completion of any task, implicit in that estimate is a probability of completing that estimate in that time. Armour (2002) points out that every estimate is a probability; but you cannot make commitments to probabilities. You make commitments to dates. Before making a commitment to a date, software teams must be given the chance to use solid estimation techniques such as wide band Delphi estimation to uncover assumptions and risks associated with a project, and develop risk management plans to define how the teams will handle change when (not if) it occurs in their projects (PMI, 2004). These risk management plans must be approved by senior management; otherwise teams are just signing up for failure.

Plans and Planning

Planning is a critical step in any undertaking. We want the problem to be understood as well as conditions allow, and to uncover as many assumptions as early as possible. Why do estimates often fail us for engineering projects, particularly software development? They fail because software engineering is a design activity, not a manufacturing one. There is a common sense understanding of the difference between plans and planning:

A good plan, violently executed now, is better than a perfect plan next week. - Patton
In preparing for battle I have always found that plans are useless, but planning is indispensable. – Eisenhower

No battle plan survives contact with the enemy – Colin Powell

Did you notice the quotes above are from military men? Software teams are equally at war, but it is a mistake to think they are at war with customers, the operational staff, or executive management. They’re not. They (and the customers, operational staff and executive management) are together at war with uncertainty. Agile project planning is all about recognizing that uncertainty exists, minimizing that uncertainty as much as possible, and installing tools and processes that manage that uncertainty and risk within the framework of the business goals and objectives.

Doing it better

In environments that change very slowly, traditional project management methods focus on a full understanding of the product through careful analysis and planning. Before any significant investment is made, the uncertainty is therefore reduced to a level where risk can be comfortably managed.

However, when business and technical requirements change rapidly, often because the business is continually redefined, this full understanding is impossible to achieve. Planning is critical, yet we must be cautious about what we do with the results of our planning phase, and be careful not to fall into the trap of believing we can have perfect estimation and perfect scheduling at the beginning of a project. There will always be a lot of variability in project scope at the beginning of a project, and there will always be change throughout the life cycle of the product. Turing award winners Barry Boehm, Kristen Nygaard, Ole-Johan Dahl, and Peter Naur have all espoused and developed these agile development ideas. We ignore their research at our peril.

Software is not a unique industry in having high uncertainty at the beginning; what other industries do is constantly reiterate and re-plan. They don’t make a big up-front plan and treat it as inviolate. Estimation is a tough but important problem, and we need to be thoughtful about doing it better, as long as we don’t confuse valuable planning with a rigid and brittle plan, or estimates with commitments.

Armour, Phillip. (2002). Ten Unmyths of Project Estimation. Communications of the ACM 45, no. 11155-18.

Brechner, Eric. (2008). Hard Code. Redmond: Microsoft Press.

Cohn, Mike. (2006). Agile Estimating and Planning. Upper Saddle River, NJ: Pearson Education, Inc.

PMI. (2004). A Guide to the Project Management Body of Knowledge, 3rd Ed. Newtown Square: Project Management Institute.

Standish Group International. (1994). The Chaos Report. The Standish Group.