This may take a while...

I’ll tackle the first 3 in order, most likely. After that, I’ll see what looks the most interesting at the time.

What may be of interest to my readers is that, starting in January, Stanford is offering several other courses with the same online format. Here are the ones that I’ve seen so far:

• Software Engineering for SaaS
• Human-Computer Interface
• Natural Language Processing
• Game Theory
• Probabilistic Graphic Models
• The Lean Launchpad

I certainly won’t have the free time to take all of them (my commute is not that long), but I’ve signed up for all of them to at least have a look at the material. If they’re structured like the courses I’m currently taking, completing the “advanced” track of any of these (which means doing quizzes, exams, and programming assignments) gets you a certificate of completion from the instructor. Hopefully Stanford will start working with Mozilla, specifically their Open Badges initiative, and continue to push the boundaries of online education and credentialing.

Are you taking any of the current courses? Do any of next term’s look interesting to you? Leave a comment!

Mary started this part of the talk by showing “Drive”, a talk by Dan Pink (author of Drive) animated by RSA Animate. It’s a great clip, one that I’ve seen before but didn’t mind seeing again. In case you haven’t seent it, here you go.

The real take-away here is that money doesn’t serve as good motivation for tasks that require collaboration and cognitive skill. It may serve as some motivation, at least to do the minimum amount of work required to get paid, but you won’t necessarily get people going above and beyond. What are the assumptions that underpin the current model of financial compensation as motivation? Are financial incentives good for anything?

The traditional way of thinking is that financial incentives can motivate people to do better work, signal to the employees what the business thinks is important, and attract external talent to the company. Motivation has already been discussed here, but it’s worth mentioning that a bonus or raise to encourage better work (or as a reward for good work) is based on the assumption that job performance is under the control of the individual. This is not always – perhaps even not usually, according to systems thinking – the case. While it’s certainly important for a company to let its employees know what goals are important to the business, associating financial rewards with these goals opens the system up to gaming or micro-optimization, and sends the implicit message that, if people were compensated properly, almost any management or organizational problem could be solved. As for attracting top talent to an organization, the only trait you can be sure a high salary will attract is the love of a high salary. Like Dan Pink pointed out, hard-working, highly talented individuals are often more than happy to give their work away if they have the right motivation.

Mary goes on to enumerate even more assumptions behind financial incentives, taking time to challenge each one. The idea that money is an important motivator for people is challenged by research that shows most people claim that they are motivated by other factors, and that intrinsic motivation is the best motivator. As mentioned above, using financial rewards as an means to enhance performance is an instance of the fundamental attribution error, that is, that people are directly responsible for all of their own failings (as opposed to the environment that they are in). Individual rewards are predicated on the idea that individual performance can be reliably and unambiguously assessed, something that is incredibly difficult in a highly interdependent field like software development. An argument against group measurement and reward instead of individual – that is, that slackers will take advantage of hard workers – is challenged by evidence that shows very few people take advantage of situations like this, due to strong peer pressure to perform.

This is not to say that it is trivial to implement a bonus scheme and rewards plan that avoids all of the above problems, although Mary had some simple advice to get started – “Pay people enough that money isn’t an issue”. The key is to create what Arie de Geus calls a “living company”, where individuals know the company is invested and investing in them. Such a company’s goal is to perpetuate itself into the future, creating profits in the near or long term, as opposed to being focused soley on short term gains. As an example, see Amazon’s recent letter to shareholders, which emphasizes a long-term view and commitment to being around for a long time and solving interesting problems, possibly at the expense of short term profit margins. On the motivation side, it is critical to ensure your information workers are challenged sufficiently and improving their skills, as per Csikszentmihaly‘s idea of flow.

Keep people in the flow channel

This is an interesting topic for an Agile conference, as it’s something that really needs to start at an organization level, as opposed to at the team level (barring companies that have team-specific compensation). In that way I see it as more of a Lean topic – a distinction I tend to use is that Agile topics are often team based, while Lean tends to apply to the entire organization. A gross oversimplification to be sure, but one that helps focus (and hopefully not overly-narrow) my thinking. Either way, it was a great presentation and a provocative look at a one of the elephants in the room.

Further reading:

• Living Company, Arie de Geus
• The Human Equation: Building Profits by Putting People First, Jeffery Pfeffer
• Drive: The Surprising Truth About What Motivates Us, Dan Pink
• Flow: The Psychology of Optimal Experience, Mihaly Csikszentmihaly

After last year’s Agile Vancouver conference, I wrote about my to-read list, books that were either written by presenters or featured prominently in presentations. How did I do with getting through this list? Let’s see.

First up, books I didn’t read:

• Certain to Win: The Strategy of John Boyd, Applied to Business (Chet Richards)
• Teamwork Is an Individual Skill: Getting Your Work Done When Sharing Responsibility (Christopher Avery)
• The Fifth Discipline:The Art and Practice of the Learning Organization (Peter Senge)

No particular reason that I didn’t read these, they just haven’t bubbled to the top of my list. The Fifth Discipline is seemingly a fixture on my to-read list, so that should probably be the one (from these three) that I get to soonest.

And now, the books I read, with a short summary of my thoughts.

Fearless Change: Patterns for Introducing New Ideas

The meat of this book is written in the style of patterns, much like Design Patterns, or the original, A Pattern Language. It presents a framework for introducing change into organizations, and its’ smart writing and non-linear style had my flipping between sections for hours at a time – this is this style of book that would thrive as a Wiki. I can’t say that I’ve used any of the techniques knowingly since I read the book, but that’s primarily because I haven’t tried to drive any widespread organizational change in the last year. The next time I try, I’ll be laying out a game plan based on this material.

Agile Retrospectives: Making Good Teams Great

Focused on moving retrospectives away from dull conversations about what did and didn’t work, this book is a quick read. Unfortunately, I haven’t had much success in applying the suggestions in this book. I’m not sure if it’s because of team dynamics, or my inability to successfully implement the approach, but I find that whenever I try to make use of one of the activities it falls flat. Perhaps it’s just a habit thing – do it a few times and it will get easier. I haven’t given up yet, so I may yet post again on the subject.

Behind Closed Doors: Secrets of Great Management

Nothing really groundbreaking here, but it is a good starter for anyone looking to go into management, and a reference for those of us already in the role. Much of it I see as common sense, but that’s speaking as someone with 3 years experience managing a team where I know everyone and am quite familiar with the way the organization works. Side note – I had this in my backpack during this year’s conference, and happened to see Esther Derby and Johanna Rothman standing next to each other after Johanna’s keynote, so I now have a signed copy

Other Comments

So, I read 50% of the books – that’s not too terrible, given that I’ve read a ton of other books in the last year. One thing that I didn’t adopt was the SQR3 method, which is unfortunate – I find that I’m doing a lot of reading, but my recall and long-term understanding is not where it could be. Re-reading last year’s post (and thus rediscovering SQR3), and listening to the recent Manager Tools podcast on How to Read a Book has me committed to improving my reading style. Marking notes in the margin may be difficult with ebooks (I love my new Kindle), but that’s no excuse – either through the Kindle’s notation interface, or through hand-written notes in a notebook, I’ve resolved to improve my non-fiction reading.

Mary started by giving a history of performance appraisals, from their first documented use in 3rd century China, to Taylor’s scientific management theories in the early 1900′s, to their near ubiquity we see today. Given their prevalence, and the feeling one gets that nobody outside of the occasional HR organization enjoys either giving or getting performance appraisals, the assumptions behind these reviews merit examination. To help frame the discussion around these assumptions, Mary listed the 4 services that performance appraisals are assumed to provide: a motivation to improve performance, career / professional development guidance, a paper trail for corrective action, and a basis for pay and promotion. I’ll tackle the first three here, and save pay and promotion for the next post, Compensation.

The Bobs enjoy a good appraisal.

As a motivation to improve performance, yearly appraisals are prone to several problems. The value of feedback, either negative or positive, goes down in proportion to the cube of the amount of time between the event generating the feedback and the feedback itself*. If you are trying to effect change in an employee, the best time for feedback is “immediately”, followed by “as soon as you can get that person alone” for items that are better off discussed in a private setting. Additionally, combining positive and negative feedback from a large time frame into one review tends to mute the effect of the positive feedback, since people tend to focus more on the negative. You can think of these ratings as a game without a winner, where the best you can hope is maintain the status quo. If you were to give a high rating to a mediocre performer, it would not give them any motivation to change, so you’d still have a mediocre performer. If you were to give a mediocre rating to a high performer, the high performer (who probably knows she’s a high performer) will be disappointed and probably reconsider how hard she should be working.

To counter the assumption that appraisals are necessary for professional development, Mary touches on the body of research that covers how people across a wide variety of fields learn and master new skills. A concept I first heard about in Malcolm Gladwell’s book Outliers,  the steps to work towards mastery are as follows:

• Identify a specific skill that needs improvement
• Devise or learn from a teacher a focused exercise: designed to improve a skill
• Practice repeatedly
• Obtain immediate feedback, adjust accordingly
• Push the limits: expect repeated failures
• Practice regularly and intensely, maybe 3 hours a day

Now, most of us may find that list a bit daunting, but that is the price of mastery. That said, not everyone expects to master every skill they attempt to use – the closest to 3 hours a day that I spend on any one thing besides sleep is riding the bus, for example. The important thing to note is that in this list of elements of deliberate practice (having a teacher or coach, pushing your limits, obtaining immediate feedback, and dedication), appraisals don’t really cover any of them. How then can they be expected to promote meaningful growth?

The idea of creating an audit trail in case of punitive action or dismissal may appear to make sense in today’s litigious society. Indeed, Mary points out that formal performance reviews gained a lot of popularity following the 1960′s civil rights movement in the US, when such systems were set up to prevent untraceable systematic discriminatory dismissals. However, if this system is meant primarily for underachievers and others needing corrective action, why must all employees go through the process when the vast majority of them are in no danger of being let go?

In my next post I’ll write about the second half of Mary’s presentation, which challenges the use of appraisals as a basis for pay and promotion, and in doing so questions several other common aspects of the dominant compensation model and dives into the complicated area of motivation.

* Simmons’ Law. Yes, I just made that up, and I have no empirical data to prove it, but the disproportionate value of immediate feedback when compared to delayed feedback is a well-known phenomenon. At worst, I’m off by a coefficient.

Further reading:
- Get Rid of the Performance Review, Wall Street Journal
- Bringing Out the Best in People, Aubrey Daniels

Dave’s dialogue described an experience report where he and other Agile coaches were brought in to a large organization to assist with their Agile transformation, but the goal of the talk was not to educate the audience on how to do huge transitions. Rather, Dave presented a generic planning tool and thought process for leadership teams, called Agile Strategy Mapping, that is applicable for almost any type of strategic planning. Agile Strategy Mapping involves defining high-level strategic objectives in terms of critical success factors, defining necessary conditions for these critical success factors, and then creating a prioritized backlog of tasks required to realize these necessary conditions. Using Agile Strategy Mapping, Dave went on to show how a leadership team can implement organizational change and strategic goals in a focused, incremental way.

A simplified Agile Strategy Map: Goal in cyan, critical success factors in gray, necessary conditions in yellow. The white condition is a nice-to-have.

The term “leadership team” is an important one; Dave spent a healthy portion of the talk describing the fundamental differences between “working teams” and “leadership teams”. Unlike working teams, leadership teams are often not dedicated to a specific problem space. They are ultimately responsible for the work being done, and most of their influence comes through delegation. This is not to say that working teams do not have responsibility or are in any way inferior. The delineation of the two groups is not into a hierarchy of importance (although of course there is some role power involved), but rather is meant to serve as recognition that the two types of teams have different capabilities, requirements, and working styles. Dave’s main push is that, despite these differences, Agile Strategy Maps enable leadership teams to achieve goals in a more Agile fashion.

The concept behind the Agile Strategy Map isn’t rocket science, but the idea of breaking down large, over-arching strategy goals into bite-size pieces is a good one, and one I believe many organizations would benefit from. As is true for working teams, a prioritized backlog fosters collective ownership and shared goals. Even the act of creating the Agile Strategy Map can serve as a valuable focusing tool for leaders, helping turn a group of managers into a team. This team building and collective ownership can then serve to form the basis of a positive feedback loop, on the backbone of small definitive commitments, strong accountability, and a focus on customer value instead of small, local optimization.

Although I really enjoyed Dave’s presentation, and will be using Agile Strategy Mapping as the technique by which I will introduce ideas and recommendations from this conference to my management team, there were some notable shortcomings to the experience. At one point we broke into small groups (6-8 people) to try our own hand at Agile Strategy Mapping. I didn’t really enjoy my small group experience – the ideas generated for critical success factors were too generic and non-actionable (i.e. “mindset change” as a critical success factor), and the concept of necessary conditions seemed lost on most. I may have also been a victim of groupthink, in that the majority of people in my group were from a single company and thus presumably had some shared background and perhaps preconceptions about the planning process. Finally, there appeared to be a large number of individual contributors (as opposed to leaders / managers) in the audience. This is not a problem in and of itself, of course – I was certainly very interested in management and leadership when I was still a junior programmer. However, Dave’s presentation was squarely aimed at those in leadership roles – I feel some of the negative feedback he received may have been due to a disconnect between individuals who were expecting something targeted at grassroots efforts.

Further reading:

• Dave Sharrock’s slides on Leading Agile Change

I’m familiar with metrics, and have written about them before, but this is the first presentation I’ve seen on considering metrics about your code as it is. That is, instead of the usual metrics of product performance, or unit test coverage, or even team-based metrics such as velocity or throughput, the idea is to examine the evolution of your code over many revisions to see how it has progressed (or, quite often, regressed). This allows you to ask ask several interesting questions of your code base. As an example, for the following two cases, which one is easier?

- adding code to an existing method, or adding a new method
- adding a method to an existing class, or adding a new class

By looking at how code changes over time, we can give empirical answers to these questions, the assumption being that developers will choose to do the easy things. Histograms of method size vs. time and number of methods per class vs. time can answer that and other questions. Michael gave examples of three popular open source code bases – Clojure, JUnit, and Fitness – all of which seemed to honour some variant of the open-closed principal (that is, classes should be open to extension and closed to modification) since the vast majority of classes were not modified over time.

He went on to list many other metrics you could investigate from your source code’s data:

- method complexity vs. time, which allows you to see refactoring as dips in complexity
- commits vs. hours in the day, minute of each hour, second of each minute
- added complexity over time, added per hour of the day
- number of files touched per commit
- average lines per commit by month

Perhaps the most interesitng chart was a plot of the number of commits to a file vs. the complexity of that file, similar to the following:

Here it looks as if the code base is separated into four quadrants. In the upper right, you have files that have very high complexity that are also changed a lot. These are the big scary functions at the core of your product – the ones that have been around for a while, nobody really understands, and may have relatively few unit tests so refactoring is more difficult. In the upper left, you get code that was likely submitted already in a complex state, and only lightly modified thereafter; perhaps these methods were the result of weekend-long hacks, functionally complete but perhaps a little overly complex. The lower left contains well-factored, not frequently changed code – the utility or base classes of your product, for instance. Finally, the lower right quadrant contains simple code that is very frequently changed. As often as not this would be something like XML, configuration files, or other such data stored as code.

Although these metrics can be interesting, there some caveats. Variations in personal style, such as the average size of a commit, isolated vs. large changes, and the like can greatly skew the numbers. Comparing different codebases can also fall prey to variations in environment – differences in implementation language, nature of the problem being solved, and a variety of other problems.

Besides simply mining my own code base for the metrics Michael suggests, I see other opportunities with this line of thinking about code. Tying your revision control system to your defect tracking system could allow you to see trends of defects over time as they evolve in the code, and perhaps giving a threshold for refactoring – that is, if something has more than X complexity, it’s twice as likely to produce bugs. Predictions about which functions are prone to getting over-complex could be made using artificial intelligence and machine learning techniques. All of this could become an add-in to your continuous integration software – the list goes on.

Next post: Dave Sharrock on Leading Agile Change.

Further reading:

• Getting Empircal about Refactoring, Michael Feathers

Johanna Rothman

Although much work has been done and progress made in the 10 years since Agile got its name, Johanna argues that we’ve only done the easy work. Agile adoption is working well for small teams, in small- to medium-sized organizations, but it is much harder to find Agile success stories in larger organizations. Many such organizations have attempted an Agile transition only to fall back to waterfall or chaotic development methods after 2-3 years (or less).

Small Agile teams within larger organizations have not been too helpful either. These teams, perhaps established through the will of a talented manager or through an Agile pilot program, are often up against enormous organizational pressure. This results in the team building fences around itself, being very protective and defensive of its culture and practices. While seemingly well-intentioned, these fences lead to a local optimization problem, where the existing Agile team may hesitant to change their ways even for the good of the entire organization.

Johanna sees a fundamental disconnect between intent and action; while intent may have indeed crossed the chasm, action is falling behind. Part of this is due to the “breadth-first” adoption approach that is all too common. Instead of trying to gain deep understanding of the principles behind Agile, or examining individual Agile practices in the context of organizational problems, practices are often sprinkled around or introduced with little or no background. I know that I have had several exchanges similar to the following.

Developer: Yeah, we’ve been doing Agile for a while.
Chris: How’s that working for you? What do you do?
Developer: Well, we sit together when we code. QA and product management sit with us too.
Chris: Okay… have you seen any benefits to having an integrated QA team? Have your releases sped up?
Developer: Integrated? Well, they still do all the testing after dev’s done, so not much has changed. I guess we write more unit tests now. Oh, and our planning sessions are called sprints, and we stand up in meetings now.

A quick stand-up survey of the crowd confirmed that this experience is not uncommon. When a standing crowd was asked to sit down if you either don’t have a cross-functional team, don’t have a prioritized backlog, or don’t produce shippable product on a regular basis, about three-quarters of the room (myself included) took their seats. It seems that the marketing – that is, the idea and intent of Agile – has greatly outpaced the adoption (or retention) of practices and the mindest behind said practices.

The rest of the presentation was rooted in the idea of a “growth mindest” versus a “fixed mindset”, but Linda Rising gave a wonderful day 2 keynote on this very topic so I’ll defer most of my comments until then. One point that is worth mentioning is the idea that a successful and sustainable Agile adoption requires not just practices or ideas, but both personal and organizational cultural changes. Indeed, to quote Johanna’s slides: ”Agile encourages personal change – Agile requires cultural change.” Changes such as belief in power distance and comfort with uncertainty (Geert Hofstede‘s model as applied to organizations) are often overlooked by those transitioning to or already practicing Agile. To truly succeed in a venture such as organizaitonal change, a more holstic, cross-discipline view must be taken – a theme that continues to reappear over the course of the conference.

In the next post, I’ll cover Michael Feather’s talk on mining data from your version control system.

Suggested Reading:

• Mindset, Carol Dweck
• Cultures and Organizations: Software for the Mind, Geert Hofstede

Programmers have it easy

A lot of options

Let’s think about how a programmer progresses. Of course, there’s experience and time, the greatest of teachers. Even if you have a programmer solve a similar class of problems on a regular basis, if she’s keen she’ll come up with new and better ways of doing things. Add a piece of automation here, a new editor trick there, perhaps a refactor or a domain-specific language – there’s always some opportunity to learn something new.

Luckily, most programmers aren’t stuck solving the same problems over and over – they’re doing different things. This could mean a new programming language (or new techniques with the one you already use), a new platform, a new module or third-party API, or a new paradigm, such as incorporating more of a functional design style into otherwise object-oriented code. Programmers can also change their focus, for instance from user experience to scalability to embedded systems.

Additionally, professional programmers most often don’t have to do it alone. If you run into a database problem you don’t know how to fix, you can go look over the shoulder of the local SQL guru as they optimize your query. If you’re fortunate enough to be on a team where pair programming is the norm, you’ll be in almost constant learning mode. At a standup meeting, at an architecture design meeting, or at a security review, there’s a group of your peers (or superiors) solving technical problems.

All that said, the title of this section is a bit flippant – to really take advantage of these learning opportunities, a developer would ideally have the will to learn, the drive for self-improvement, and the hacker ethos of experimenting with programmable systems. It’s not that this learning comes for free, it’s just that it can be part of the job (or some on-the-side project, or an open-source effort, etc.).

What exactly would you say you do around here?

The pleasure's all on this side of the table, trust me.

So now, back to the land of management. What does a software development manager do, and how would one go about improving those skills?

Roughly speaking, you can break down what I do into four areas:

• People management. This includes conflict resolution, career development for my direct reports, and managing up.
• Project management. This involves task prioritization, risk assessment, project tracking, and scope management.
• Process / team management. This is broad, but in part consists of promoting technical and team best practices, and focusing on how the team does their work.
• Technical guidance. Although I’m not the most senior technical person on the team, I have substantial product experience and a higher-level context that proves useful during in-depth technical discussions.

Of course there is overlap. What is a team if not a group of people? Choosing continuous deployment could be process or project management, and encouraging pair programming is part team management and part technical guidance. The list goes on and on – I’ll stick with the above demarcations tfor now.

How do you grow?

There are several paths towards professional self-improvement, but I’ll lump them into three broad categories: Experience, Training, and Self-Study.

Experience in this case refers to professional experience, and covers most of what I talked about above with respect to programmer development. This is the knowledge that comes from trying new things, learning from mistakes (both yours and others’), and paying attention to what works in various situations, all as part of your day to day work life. There is always some sort of experiential learning to be had, should you choose to take advantage of it.

Training is another way to learn, and can be helpful in circumstances where there’s a specific skill set to be learned that is too broad to wait for experience to teach, too critical to be learned through trial an error, or too time-sensitive to be learned through self-study. School and certification courses are obvious forms of training, but I also count mentoring (external or within your company) and conferences under this heading.

Self-study is a broad term. and I use it here to refer to everything from reading blog posts and books, to podcasts and forums or user groups, to side projects and open source contributions. Self-study is the most accessible form of learning for most people in this profession, since it is plentiful and often free, but it is often the least taken advantage of.

Naturally, like the overlap in the management responsibilities above, the distinctions between these categories are not hard and fast. Would MIT Open Course Ware count as training, or self-study? Work experience often has aspects of self-study, even though it may not self-directed. A conference may be training, but what about an unconference?

So what are you doing about it?

Aspiring to greatness

Now that I’ve laid out the loose definitions for the types of learning I might undertake, as well as the areas in which this learning could be applied, I’ll get into what and how I’m trying to learn.

People management is an area where I think there is a limitless amount of experiential learning to be had, and it’s one I’ve been focusing on more recently.  Even if I was managing the same people for 10 years, they would change, I would change, and the project would change, so there would be new challenges and opportunities. That said, I have been pursuing self-study in this area (mainly through blogs, books and podcasts), and the upcoming Much Ado About Agile conference has several talks and tutorials on the subject of people.

Project management is something I do as part of my job, but I’m not really a project manager. I have experience working within my company’s project framework, but there’s a lot of inertia to overcome to change anything since it’s a framework the entire company uses. This is an area I’m not putting a lot of effort into improving on my own time, since it’s not a primary interest and there’s relatively little I can do about it in my current position, but I’m always on the lookout for things that I can improve during my day to day tasks.

Process and team management is where I’ve spent a the most time learning, through experience, training, and self-study. I’ve followed the Kanban community, posting questions and comments, and even helped review a great book on the subject. I’ve read books by Mary Poppendieck, Tom DeMarco, Esther Derby, and more, and my to-read list just keeps growing. I’ve attended and spoken at Agile meetups and conferences. I’ve tried to be an Agile coach for my team (which can sometimes be difficult). Although this is where I’ve focused most of my efforts, I’m starting to back off a bit here in favour of People- and Technical-centric learning, primarily to round out my management style.

Finally, learning about technical management. Although over the years I’ve done some self-study on this, reading books on design patterns, lean and agile development practices, and architecture, I haven’t been as rigorous about this as I could have been. Part of it has been that I haven’t been coding at all (maybe a few lines every few weeks), a situation which I’ve started to correct. Picking up new technologies, even if I’m not using them at work or for any serious projects, helps broaden my perspectives when faced with the challenges I do need to solve at work, providing me with a bigger technical toolkit. I’m also getting more involved in security reviews at work, and have recently had some work-sponsored training by a security expert, and I hope to get some further training on the technical track of the upcoming conference.  While I don’t know if I’ll ever go back to primarily being a developer, I think staying sharp technically is an important part of being a development manager.

Your turn

Thanks for making it this far! Do you agree or disagree with my assessment of the learning opportunities for programmers or managers? Have anything to add? Let me know in the comments.

Bill opened the talk with a few points about Ping’s philosophy towards metrics. He admitted that many metrics are not scientific measures and can be prone to inaccuracy, but they can still be useful to help ground a company that’s working in a rapidly changing environment. The use of metrics is simple – gather some metrics in as consistent a way as possible, monitor those metrics, and question change. In this case, change means for the better or for the worse. In this way, metrics can provide a good way to “Check” when implementing a Plan Do Check Act cycle of learning. Some of Ping’s other philosophies and practices reinforce the value of metrics, such as the idea of “Going Honest Early” – bringing up potentially problematic issues immediately – and the use of operation reviews, company-wide meetings in which the health of the engineering organization and its projects is discussed. These reviews, which I first read about in Agile Management for Software Engineering, provide valuable feedback to non-engineering facilities and are a practice that I’d love to see taken up in my office.

Next up was an Introduction of the classic “iron triangle” of project management, Quality, Time and Features. At Ping they fix time, committing to three month “trains”, or releases. If something is not ready in time due to quality issues, incomplete development, or any other reason, it’s not in – the rest of the completed features go ahead and ship. Lead time is established to be 3 months, with high confidence; in the last 5 years, there have only been 12 days in train delays. With time fixed, there is a tension between quality and features, with quality always winning. This is not to say that they gold plate – there is simply a zero defect release criteria. The concept of the “Iron Triangle” was elaborated into an “Iron Pentagon”, the additional two factors being resources and system.

Bill brought up several metrics that are used at Ping to observe many aspects of their development system. There were graphs of “utilization”, or how many team members were available to work on a project and how many were absent due to vacation, holidays, illness, etc. There was a slew of metrics related to quality, from the count of development-injected bugs to the average amount of time between finding and fixing a bug and the amount of time the test team spends on set up. Like all metrics, these require the gathering of data over time, and one of the metrics mentioned was a measure that I did have the data for, albeit not centralized – the number of various types of automated tests run by one of our products. A few hours of hacking data together, and a few minutes learning some jQuery and flot, and I get the following chart:

From here it’s easy to tell that 2008 was an awful year for unit tests, we greatly prefer unit tests to system tests, and that both have been growing over time. There are several more metrics that were mentioned that I’d like to take a look at, particularly the ones related to defects. I’ll post any graphs I make, and would love to hear stories from others who have sucessfully used interesting metrics to drive change in their development organization.

As an aside, I asked Bill after the talk about techniques used to loosely couple features such that they may be removed near project end without destabilizing others. He said that a variety of techniques were used, ranging from a preventative / planning approach – that is, simply not working of features at the same time that are likely to impact each other, making backout easy – to incorporating functionality “switches” in the product, which can be configured either at compile or run time to use either new or old code. These are techniques that I would love to start applying on my team, if there is ever a need to have strict adherence to a release schedule.