A Systems Thinking Alternative to Performance Reviews

I tried something new based on the concerns and research around annual performance reviews. I'm not qualified enough to make categorical claims about the typical annual performance review process, but I can say that by trying something different I enjoyed the performance review process more than any other year and I had some great conversations with my co-workers. Here is a quick report of some of the concerns, what I tried, and what I learned.

The concerns:
 
W. Edwards Deming is often quoted when expressing concerns over annual performance reviews. Deming's 85-15 rule implies that 85% of any employee's performance is based on the system they are working in rather than their own individual actions. If this is true, then any significant performance increases or decreases are not under the employee's control but rather those who control the system (i.e. management). This means that if employees are under-performing we should look to the system first and not the individual.

As a parent, coach and change agent, I have noticed one consistent trend with respect to individual improvements. If I try to push improvements on people then they will tend to resist the change. However, if I try to pull improvements from them (ask them what they want to learn/improve) I encounter much less resistance and more actual improvements. How might this influence the annual performance review process?

The experiment:

Note: At Protegra we have a flat structure so we don't receive feedback from managers but rather from the people we've worked with over the last year. As a company we've tried various approaches and are still experimenting to uncover better ways. The experiment I tried this year was based on the concerns above and also conversations with co-workers. In the end I came up with five questions to augment my involvement in our yearly process. After receiving permission from each person, I sat down with them and had a conversation that centered around the following questions:

a) What are you proud of? (Pull)
b) What do you want to learn or improve this year? (Pull)
c) What part of our team's system is preventing you from doing your job better? What should we improve or change? (Systems Thinking)
d) How is the company enabling or inhibiting you from achieving your best? (Systems Thinking)
e) What do you need from me? How can I help? (Pull)

The results:

I had fantastic conversations with several of my fellow employees on what they love about their job, what they are proud of, and the things they are doing to try and make their work environment better. I discovered some common interests and swapped ideas for personal improvement and growth. I learned more about what makes each person 'tick' and how I can support that. In the conversations I had with each person I left feeling inspired, listened to, and encouraged. I think we all felt a greater sense of ownership over our improvement plans for the year.

Just as importantly, I learned more about the value of Deming's 85-15 rule and its impact on performance. We had discussions about our system and the things we could (and did) change. In more than one case it became clear how powerful a simple systems change could be. Our conversations were slanted towards being a performance review of the system instead of a performance review of the individual.

This experiment was a lot of fun and I plan on doing it again. In fact, I have added an item to my backlog to extend this conversation to other people on the team throughout the year.

Some final thoughts:
After reading this some of you may be asking questions like "how does this help us address those individuals that actually have 'performance' problems?", or "how does this help us set salary?". I'm not sure it does in the way you might like. For the former, please don't wait until the annual review to deal with the issue (short feedback loops please!) and make sure to look at the system first. For the latter I'll leave that to other experts to discuss (see the TLCLabs blog below for one example). However, I'd guess that taking this approach nudges a group of people towards having a better system, being a more effective team, and becoming a more effective organization.

System Thinking References
- Demings 85/15 rule
- If Ackoff had given a Ted Talk
- It's not a behavioural problem its the system

Performance Review References
- Why performance reviews don't improve performance
- Why we don't do forced ranking and performance reviews - TLCLabs
- The Elephant in the Room: Appraisals and Compensation - Mary Poppendieck
- Get Rid of the Performance Review! (and some info on Performance Previews)

Subscribe to Winnipeg Agilist by Email

Blog Challenge!

Hello readers,

This is a quick post to let you know that a few friends of mine have started a blog challenge. The challenge (led by Dylan Smith) is to blog bi-weekly over the next six months with a winner take all purse. Be sure to check out their blogs as well:

Dylan Smith: http://www.geekswithblogs.net/optikal
Tyler Doerkson: http://blog.tylerdoerksen.com
David Alpert: http://www.spinthemoose.com
Dave White: http://www.agileramblings.com
Aaron Kowall: http://geekswithblogs.net/caffeinatedgeek/Default.aspx 

(P.S. No - this post doesn't count in the challenge)

Celebrate Failure? [Part 2]

Earlier this year I wrote about Agile's perspective on failure. In that blog I indicated that my brother-in-law (a psychologist) sent me some of the latest research on failure. In particular, there were two fascinating studies that helped me understand why failure is indeed a cause for celebration. In the first part of this series I summarized the results of a study looking at the effects of failure and success in the orbital launch industry. In this blog post I'll look at some interesting research that examines the role of attitude when failure occurs.

Your attitude towards failure (and your organization's attitude) does matter.

A group of researchers led by Joel Albert Kahn set out to discover what the effect of failure norms - or attitudes towards failure would have on closing gaps in performance. Is failure enough incentive for an organization to find ways to improve or does their attitude towards that failure matter?

In an exploratory study they surveyed teams within an automotive manufacturer to look for teams that had both strong and weak "failure acceptance norms". A team with weak failure acceptance norms would be characterized by defensive attitudes when failure occurs. A team with strong failure acceptance norms would be characterized by team members that have an acceptance of failure as normal.

Using survey data they identified the teams with the strongest and weakest failure acceptance norms and then observed those teams over a two year period. What they found is that attitude mattered - the teams with the strongest failure acceptance norms were the teams that closed performance gaps the most effectively.

To those familiar with Carol Dweck's book "Mindset: The new Psychology of Success" or who have seen Linda Rising talk about her research, this should come as no surprise. According to this research, people generally find that they have one of two attitudes.

The first group of people believe that each person is either smart, or not smart - they have a fixed mindset. In this group failure is a strong indicator that you are not smart - they become defensive when failure occurs and gravitate towards easier tasks that allow them to be successful. This group has weak failure acceptance norms.

The second group of people believe that if you work hard you can improve and overcome problems - they have an agile mindset. In this group failure isn't an indicator of intelligence but rather a challenge to try again and work a little harder. This group has strong failure acceptance norms.

Interesting - Carol's research on individuals matches that of Kahn's research on organizations and teams.

One further note about Carol's research. She found that it was relatively simple to move people from the first group to the second. Our words can be powerful. When failure occurs, celebrate it as a learning opportunity!

To end this series I have a suggestion: The next time your team fails, promote an agile mindset and buy them a cake to celebrate:

Credit: www.cakewrecks.com

Further Reading:
- Linda's presentation on this topic
- Linda's keynote from Much Ado About Agile
- Carol's website: http://mindsetonline.com/
- A short video of Carol describing the effects of praise on children

Subscribe to Winnipeg Agilist by Email

You can get the full research from your neighbourhood PhD student but here are a few more details on the studies mentioned above:

"Failure construction in organizations: Exploring the effects of failure norms - Kahn, Joel Albert, 1995, University of Michigan. School of Business Administration. 

Two exploratory studies were used in this research to develop an understanding of the effects of failure norms at an automotive manufacturer. Using the survey data, the two teams with the strongest and the three teams with the weakest failure acceptance norms were identified. These five teams were observed for two years and cause maps were collected from 31 members of the teams."

"Early models of organizations as rational systems assumed that a gap between what is expected of a performance and the performance as it actually occurs will stimulate a search for an alternative approach to a problem. An interpretive perspective, in which performance gaps are socially constructed, suggests that such a search is not the inevitable consequence of performance gaps. Instead, people interpret performance gaps defensively by retrospective rationalization and external attribution to avoid acknowledging failure. People will search for alternatives only when performance gaps are interpreted as failures and not when they are interpreted defensively. Whether or not people interpret performance gaps defensively is determined by norms that are communicated and enforced within an organization an that are considered binding within its teams. Thus, the central idea of this dissertation is that search is not an inevitable response to performance gaps as is often presumed. Instead, search is a response to failure only if people are encouraged to accept failure by strong acceptance norms. …it is the acceptance of failure (strong failure acceptance norms) that encourages the interpretation of performance gaps as failure."

Celebrate Failure? [Part 1]

Earlier this year I wrote about Agile's perspective on failure. In that blog I indicated that my brother-in-law (a psychologist) sent me some of the latest research on failure. In particular, there were two fascinating studies that helped me understand why failure is indeed a cause for celebration. This is the first in a two part series.

"We find that organizations learn more effectively from failures than successes" 

In a 2010 study (more details at the bottom of this post) on the effects of failure and success on organizational learning, a team of researchers found that failure was a crucial ingredient for longer term success.

In order to find suitable organizational data to support their research they searched for and found an ideal candidate - the Orbital Launch industry. This industry was ideal for the following factors:

• Every launch had a high incentive to succeed due to the high cost of failure.
• Placing objects in space is a relatively new activity so data is available for all launches ever attempted.
• Because it is a high profile industry the records were relatively easy to find.
• The sample data contained 4663 launch attempts, 443 failures, 36 organizations, and 9 countries. The data starts with the launch of Sputnik 1 on October 4, 1957 and ends March 2004.

(Note: The Orbital Launch industry is responsible for sending rockets into space to place a payload into orbit around the earth.)

As researchers their goal was to look at the causes of improved organizational performance. Did success drive improvements? What part did failure play in future success? Here are a few of their key findings:

1. Organizations learn more effectively from failures than from successes. Success causes organizations to be complacent in the belief that they have figured it all out. On the other hand, failure increases the desire to learn and challenge existing beliefs. Those organizations that fail end up being more successful in the end.
2. Success breeds complacency and overconfidence and reduces the incentive to learn. Organizations who regularly succeed may in fact suffer from this experience in the long term. 
3. Organizations learn more from large failures than from smaller ones. As part of a community that believes in failing (learning) fast, this one is the hardest for me. But at the least it supports the notion of celebrating failure when it occurs in the larger or the small.
4. Organizations should embrace failure so that they can learn from it. One response to failure is to punish those involved or hide the failure. Organizations that are open about their failures have an increased chance of learning and then improving because of it.

As many people like Luke have pointed out - failure isn't the goal. However, when it happens we can either run from it or celebrate it. The evidence from the Orbital Launch industry strongly suggests that celebrating failure is in order. For even more reason to celebrate, take a look at part 2.

Subscribe to Winnipeg Agilist by Email

You can get the full research from your neighbourhood grad student or PhD but here are a few more details on the studies mentioned above:

Failing to Learn? The Effects of Failure and Success on Organizational Learning in the Global Orbital Launch Vehicle Industry

Abstract: "It is unclear whether the common finding of improved organizational performance with increasing organizational experience is driven by learning from success, learning from failure, or some combination of the two. We disaggregate these types of experience and address their relative (and interactive) effects on organizational performance in the orbital launch vehicle industry. We find that organizations learn more effectively from failures than successes, that knowledge from failure depreciates more slowly than knowledge from success, and that prior stocks of experience and the magnitude of failure influence how effectively organizations can learn from various forms of experience"

Discussion and Conclusion: "These findings do not imply that organizations that fail in period t are more likely to succeed in period t + 1 than are organizations that succeed in period t. But they do imply that organizations that fail in period t improve their own likelihood of succeeding in period t + 1 (relative to their likelihood of success in period t) more than do organizations that succeed in period t. Our definition of learning is inherently self-focused - change in organizational performance as a result of prior experience. ... Consequently, the result presented here suggest that experience with failure allows organizations to improve their performance relative to their own previous baseline, but that experience with success does not generate similar levels of improvement. 

Indeed, this study not only yielded strong evidence that organizations learn by observing their own and others' failures, but also failed to uncover evidence of significant learning from observation of their own or others' successes. In the full models, coefficients estimating the effect of success experience on future performance are indistinguishable from zero. We don not interpret these results as evidence that organizations cannot learn from success to improve performance. But the fact that launch vehicle organizations (which face significant incentives to learn from success as well as failure) did not experience demonstrable learning from success suggests that organizational learning from success is far from an automatic process."

Implications for Practice: "Failure is often difficult for organization members to cope with. Because failures - and those that appear to be involved in them - are often stigmatized, organization members frequently refuse to acknowledge failure, refrain from communicating about it, or redefine it as success (March et al., 1991). Indeed, in Vaughan's (1996, 2005) analyses of the Challenger and Columbia disasters, she noted that the most significant organizational antecedent to both tragedies was the institutionalized practice of ignoring failures. 

Nonetheless, given failure's central role in organizational learning shown here, organizations that stigmatize failure may be depriving themselves of major opportunities for improvement. Consequently, the most significant implication of this study of practice is that organization leaders should neither ignore failures nor stigmatize those involved with them; rather, leaders should treat failures as invaluable learning opportunities, encouraging the open sharing of information about them. Indeed, this suggestion dovetails with existing evidence that members of organization that treat failure nonpunitively report more errors, but experience fewer serious failures, than member of organizations that seek to assign blame for failures (Edmondson, 1999; Weick, Sutcliffe, & Obstfeld, 1999)."

Build Quality In: Applicable in Football too

One of the lean software principles is "Build Quality In". In short, we try to build it right the first time in order to reduce or eliminate re-work and defects. I turns out that this same rule is applicable when training a running back not to fumble.

Here is a story in Yahoo about the Patriot's running back Green-Elllis who at the time of the article had an incredible streak - no fumbles in four seasons. His college coach said:

"Far too often, he has seen players fumble and then heard the coach afterward demand that his players protect the ball by not carrying it carelessly into traffic. This always struck him as backward. Why talk about ball security after a fumble? Why not before? What are the preventative measures? You’ve got to put running backs in a position where they can eliminate fumbles." 

To put this statement in the context of software development:

"Far too often, he has seen the number of failed tests rise and then heard the project manager and testers afterward demand that the software developers write better quality code. This always struck him as backward. Why talk about tests after development? Why not before? What are the preventative measures? You've got to put software developers in a position where they can eliminate defects."

Test first my friends, test first.