Geek Philosophy

Johanna Rothman‘s keynote to the Much Ado About Agile VI conference started with a simple question – has Agile crossed the chasm? While many people may believe that may be the case, Johanna is convinced that we are still in the early adopter phase.

Although much work has been done and progress made in the 10 years since Agile got its name, Johanna argues that we’ve only done the easy work. Agile adoption is working well for small teams, in small- to medium-sized organizations, but it is much harder to find Agile success stories in larger organizations. Many such organizations have attempted an Agile transition only to fall back to waterfall or chaotic development methods after 2-3 years (or less).

Small Agile teams within larger organizations have not been too helpful either. These teams, perhaps established through the will of a talented manager or through an Agile pilot program, are often up against enormous organizational pressure. This results in the team building fences around itself, being very protective and defensive of its culture and practices. While seemingly well-intentioned, these fences lead to a local optimization problem, where the existing Agile team may hesitant to change their ways even for the good of the entire organization.

Johanna sees a fundamental disconnect between intent and action; while intent may have indeed crossed the chasm, action is falling behind. Part of this is due to the “breadth-first” adoption approach that is all too common. Instead of trying to gain deep understanding of the principles behind Agile, or examining individual Agile practices in the context of organizational problems, practices are often sprinkled around or introduced with little or no background. I know that I have had several exchanges similar to the following.

Developer: Yeah, we’ve been doing Agile for a while.
Chris: How’s that working for you? What do you do?
Developer: Well, we sit together when we code. QA and product management sit with us too.
Chris: Okay… have you seen any benefits to having an integrated QA team? Have your releases sped up?
Developer: Integrated? Well, they still do all the testing after dev’s done, so not much has changed. I guess we write more unit tests now. Oh, and our planning sessions are called sprints, and we stand up in meetings now.

A quick stand-up survey of the crowd confirmed that this experience is not uncommon. When a standing crowd was asked to sit down if you either don’t have a cross-functional team, don’t have a prioritized backlog, or don’t produce shippable product on a regular basis, about three-quarters of the room (myself included) took their seats. It seems that the marketing – that is, the idea and intent of Agile – has greatly outpaced the adoption (or retention) of practices and the mindest behind said practices.

The rest of the presentation was rooted in the idea of a “growth mindest” versus a “fixed mindset”, but Linda Rising gave a wonderful day 2 keynote on this very topic so I’ll defer most of my comments until then. One point that is worth mentioning is the idea that a successful and sustainable Agile adoption requires not just practices or ideas, but both personal and organizational cultural changes. Indeed, to quote Johanna’s slides: ”Agile encourages personal change – Agile requires cultural change.” Changes such as belief in power distance and comfort with uncertainty (Geert Hofstede‘s model as applied to organizations) are often overlooked by those transitioning to or already practicing Agile. To truly succeed in a venture such as organizaitonal change, a more holstic, cross-discipline view must be taken – a theme that continues to reappear over the course of the conference.

In the next post, I’ll cover Michael Feather’s talk on mining data from your version control system.

Suggested Reading:

• Mindset, Carol Dweck
• Cultures and Organizations: Software for the Mind, Geert Hofstede

My team has recently adjusted some of its process to adapt to changes in our organization, and I thought that this was a good chance to get on a regular retrospective schedule. As a team, we are very good at communicating, and we are constantly reflecting on successes and failures. Because we address most issues in real-time, weekly (then bi-weekly, then monthly, then occasional) retrospectives became relatively quick, limited-value rehashes of the tired old “What Worked / What Didn’t Work / Actions required” formula. If we were going to start doing retrospectives more often, I wanted to make sure that we were getting something out of them.

Chatting with Esther Derby at the recent Agile Vancouver conference, and seeing her present, made me quite interested in her books. There’s been a copy of Agile Retrospectives sitting around the office that I had skimmed a few times over the years, but now that I was intent on improving my team’s retrospectives I sat down and read it cover to cover.  It’s a great book with many sample exercises that have proven effective for a variety of teams, but it’s probably the psychological and sociological theories and general frameworks for group learning that I found the most interesting.

Our first of the “new” retrospectives was on Tuesday of this week, and I think it could be considered a success based on the feedback I’ve received. It was great to see that, despite some relatively major changes in focus for us, the team is generally on the same page with respect to how were are working together. We came up with a few explicit actions, but my primary goal for this retrospective was to get a baseline of the team with respect to their thoughts on the current project, adherence to team values (which we defined), and general team satisfaction.

We’re definitely going to maintain our current practice of doing immediate retrospectives, root cause analysis, and other corrective techniques, as it’s working well for us. I think these new retrospectives will be a chance to look at things at a higher level, measure trends, and ensure that the team is still aligned with the company and with each other.

I’d like to start off this post by saying that I’ve never worked professionally in a company that uses business analysts (in the IT sense, #4 on the Wikipedia page) or UML. This is just speculation, and I’m tagging it as such.

I had lunch with a business analyst today (a friend of mine from university who I haven’t seen in a while), and it was really the first time I’ve had a chance to talk to a traditional IT business analyst and find out what they do. The role sounds like something that would be required – find out requirements from customers / stakeholders, and translate those requirements into formal specifications for developers. How else are the developers going to know what to write?

One way to improve this inherently error-prone process (hand-offs are hard!) is to use a formal specification mechanism such as UML. This gets rid of the ambiguity inherent in human language, and often provides a significant amount of design in the form of object hierarchies. This ensures that the developers are creating a system that is as close to the business analyst’s vision as possible. However, I would say that this is solving the right problem (disconnect between the customers and the developers) in the wrong way (even more up-front specification).

Of course, Agile has an answer to this, and this is how my team works – we have an on-site “customer”. Our product manager acts as a customer surrogate, talking to as many customers as possible (we have thousands) and understanding the market. Then, instead of writing a specification, he works in conjunction with the developers to evolve the ideas and product vision. There are no detailed diagrams, no lengthy descriptions of functionality – just a conversation, a continuous design and refinement of the ideas. Things are more open to change (and thus improvement), and can change more rapidly due to not needing to update extensive documentation.

I would say that in a way all members of my team are business analysts, as we are all trying to understand our customer needs and solve them through our work.

As previously mentioned, I’ve got a talk coming up for a UBC class. This is the required software engineering class in the UBC Computer Science program, and a class I vividly remember as being a waste of my time. Fortunately, the course content has changed quite a bit since I took it – after four months in this class, I came out with a 25 page specification package and no code written. It was one of my least favourite classes, and it convinced me that I didn’t care at all about process (how things have changed!).

I spoke at this class during the summer, but I plan to deliver quite a different talk this time. Part of this is time – this summer I had two hours, and two weeks from now I have half that. Time isn’t the only factor – the more fundamental difference is in the content. While the talks I’ve given to previous classes before have been rather broad overviews of Agile (and more recently Lean) concepts, this time I’m going more with what I know best. That means less theory, and more description of how my team successfully delivers software. I’ve talked to the class instructor about this and she also thinks it’s a good idea.

I’m still finalizing an outline, and I need to make sure I present this at a level that’s approachable to a class of students who likely have no real (professional) software development experience. I also hope to find time to polish my slides, and make sure I’m not just reading list after list of bullet points, but I’ll have to see what time permits.

Reading Scott Ambler’s recent post on Reworking the Agile Manifesto got me thinking about about Agility in the Enterprise, a subject I have no experience in but want to learn more about. Here’s a bit of background – I hope to blog more on this subject as my knowledge increases.

I’ve been involved with Agile software development in varying capacities (developer, coach, manager) for just over 5 years. As such, I feel like I have a pretty good grasp on the key concepts, what it takes to succeed, and pitfalls to watch out for. That said, all of my experience is with one company, and pretty much on one team, so my sample size is quite small. Additionally, my small team has been quite fortunate in that we’re the one of the only teams in my organization that doesn’t have critical dependencies on other teams. We have almost complete control of our end-to-end value stream, my developers feel close to customer issues, we have a cross-functional team with everyone that’s required to ship the software, and we have an on-site customer surrogate (our product manager, who talks directly to customers and other stakeholders frequently). We release software frequently, are quick to respond to customer needs, and are limited only by our headcount. It’s great – my team is being recognized as a standout in the organization, and sales is quite happy with the way we deliver software.

Why is this a problem? Well, my team represents less than 5% of our total worldwide engineering headcount. The other 95%? There’s a few of them that operate within similar parameters as my team (including another team in my local office), but the vast majority of the company is in a very different situation. They have larger teams with much larger code bases, where coordination is required between many geographically distributed groups, and they’re required to support a huge variety of platforms. Project scope is multiple times (or order of magnitudes) larger than projects I run in terms of effort. Additionally, most of the teams outside of my office have a very different team structure; development is segregated from product management, and both are segregated from QA. This part of the company is significantly older and more established than my team.

I believe in the agile way of thinking and think it has a lot to offer my company, and I would genuinely like other teams to see some of the benefits that my team has received from applying agile practices. That said, without having any direct experience at scaling agile practices, I feel a bit limited in both confidence and knowledge. My recent interest in organizational learning, combined with an opportunity I’ve been given to get more involved in the codification of some our processes, should help me move towards being able to apply my interests at a larger scale, but for now I’m very much in learning mode. I’m okay with that – I’m just looking forward to a time where I can help out other teams in other situations with specific advice instead of just paraphrasing the manifesto.

The first talk I attended in the afternoon of the second day of the Agile Vancouver conference was on a subject that I’m becoming increasingly interested in – team / organizational learning. Presented by Declan Whelan, “Learning is Key to Agile Success“, this was one of the talks that had me sold just based on the description. Peter Senge, social psychology, and the Theory of Constraints? I’m in.

Declan began the talk by stressing the need to create a safe environment for learning within a team or organization. Here, “safe” means “no (or few) negative consequences for certain amounts of failure”. An organization that does not tolerate failure will never push its boundaries and thus never grow and learn grow, as it will always be operating within its safe zone. He talked of the need to create a “safe to fail” environment (contrast with a “fail safe” environment), encouraging people to try new things and not fear personal repercussions from trying to go above and beyond.

Beginner’s mind was a topic in this talk, as it was in many others at the conference (I love it when Buddhist philosophy, another great interest of mine, overlaps with software!), and again Arlo Belshee’s concept of promiscuous pairing was discussed, this time in the context of learning as opposed to lining up with one’s natural cycles of alertness. There was also some discussion of the Satir change model, which came up when discussing the ways that groups (from families to teams to organizations) react when change happens. Peter Senge and Josh Seddon were invoked in the systems thinking context – you can’t change people, you need to try to change the system that they work within, then the change comes for free.

The Theory of Constrains was also mentioned, with the idea that learning is in fact the biggest bottleneck in information work. To illustrate, I’ll borrow an example from Christopher Avery. Think of the last project your team finished. Imagine the source code was completely lost, and business decided that yes, you need to redo that same project over again. No change in scope, just do the work over again. Most projects would see a dramatic reduction in effort the second time around, which makes sense – after all, you have all of your design decisions made, you’re familiar with the code, and you won’t make the same mistakes twice. You’ve learned, and this learning substantially speeds up the work. This is one of the reasons the Empire State Building was constructed at such an incredible rate (as discussed in Leading Lean Software) – the people involved had solved this problem before and learned.

Declan finished out the talk with several concrete examples of how to encourage learning, at a personal, team, and organizational level. Brown bags, open spaces, retrospectives, scheduled slack, practice fields, information radiators – there are many ways to increase the learning of your agile team. I’ve been toying with the idea of scheduled slack (for instance, having a task card on the board that amounts to ‘go learn something’), and I think it could be especially useful when combined with the idea of “practice fields” (areas of code where mistakes can be made with almost no consequence – personal projects, proof of concepts, technology demos, etc.).

The slides are online if you want to have a look, but like all good presenters there’s only so much you can get out of them .

Okay, on to day 2 of the Much Ado About Agile 2010 conference.

The first talk of the day on the Team Track (where I spent most of my time) was “How to Develop Your Innate Leadership Gift Daily: An Agile Approach to Growth”, presented by Christopher Avery. Today (day 3 of the conference) I attended a full day tutorial by Christopher on the same subject, so I’ll combine the discussion of this talk with my later comments on the full day session.

The next speaker was Joshua Kerievsky. Joshua is the person in the agile community that I’ve known of for the longest time, since he was the trainer that came to my company just over 5 years ago to move us from doing… well, whatever the hell we were doing, to his brand of XP. It worked wonders, and I’ve been a sold on the benefits of agility ever since, so I was excited to hear his talk on Sufficient Design.

The initial discussion was around what Joshua refers to as the “5.0″ syndrome. Throughout his consulting career, he’s frequently been called in around the 5.0 release to clean up a steaming pile of a program that’s been cobbled together as a result of the ”business guy” not listening to the requests of the “tech nerd” (developer, software architect, etc.). [side note: Joshua was hired to train us immediately after the laborious release of our 5.0 product.] The counterpoint to this is that, had the tech nerd taken the full software craftsmanship approach from the beginning of the project, development may have been so slow that the company may not have survived to release a 5.0.

Joshua structured the talk around 6 sliders / ranges of values, each of which should be taken into account when developing a system. These considerations were broken into two categories – those usually considered by the business guy, and those considered by the tech guy. These are:

• Value – low to high (Business)
• Demand – low to high (Business)
• Delivery – slow to fast (Business)
• Design – simple to complex (Tech)
• Debt – low to high (Tech)
• Development – assembled to crafted (Tech)

The important idea in this presentation is that each of these sliders should be positioned as appropriate to the project. The market will tell you what the business sliders should be set to, and the analysis of the problem should tell you where to set your technical sliders. If you’re really doing a one-off, such as an database migration script, it may be okay to incur technical debt. If you’re building a website for a non-profit who can’t afford to hire specialist maintenance programmers, you’re going to want to go with assembled components instead of custom crafted code.

This is Joshua’s idea of Sufficient Design, and he went over several real world examples of the idea in use. He also suggested an alteration to the advice given in The Pragmatic Programmer (and possibly elsewhere) that a developer should learn a new language every year. Instead, learn a new language every second year, and alternate with learning business concepts. This jives well with Esther Derby’s idea of increasing information overlap – having business people aware of how the work works and having programmers aware of how the business works can only serve to help the company.

I’ve spoken briefly about Esther Derby’s talk on “Becoming an Agile Manager” – it turns out all of the slides are online, and much of the content is discussed on her blog. For instance, here’s a post on expanding knowledge overlap.

The last talk I attended on the first day of the Agile Vancouver conference was put on by Linda Rising. If you haven’t seen Linda Rising give a presentation, you’re missing out. Her talks are a blend of psychology, chemistry, sociology, software development, and many other subjects. I loved her talk last year on “How We Deceive Ourselves” (I’d link to it but the Agile Vancouver website seems to have lost the videos from the past few years… UPDATE: The videos are here. Thanks @rbeier!), and was really looking forward to this talk.

Titled “Coffee, Tea, or Agile?”, Linda walked through the dawn of the industrial revolution, a critical time in human history which brought huge career moves and many elements changing at once. She focused on some of the developments that made it possible.

Clocks, which were not in widespread use before then, became essential as people needed to fit into schedules and shifts. Before the advent of clocks, people would generally sleep while the sun was down – an average of 10 hours a night, more in winter, less in summer. This is the way humans have been pretty much forever, which is concerning given that the average amount of sleep an adult gets nowadays is 7 hours – sleep is considered a waste of time!.

Caffeinated beverages played another critical role. The average European drank 3L of beer a day in the early 1800s – it was much cleaner than the water! The increasing popularity of plants that contained caffeine (tea leaves, coffee beans), that just so happened to release their caffeine in hot water, made boiled water (and thus perky, less drunk people) the norm. It also helped to break people out of their normal sleep cycles.

Linda then launched into a summary of caffeine’s prevelence in the workforce. It’s the most popular drug (she cites a study that says up to 80% of the world’s population consumes caffeine daily), and although it has been studied and considered relatively safe in does of around 300mg a day, (or two cups / 500ml of coffee), there’s a whole lot of people who drink way more than that each and every day. Have you seen the size of a Starbucks cup recently?

Next came the negative effects of caffeine, both directly attributable physical effects (decreased size of hippocampus, premature brain aging) and secondary effects (considerably less sleep). These effects were contrasted with what Linda called “the productivity myth” – namely, the idea that caffeine (and the extra work you do while buzzed) actually helps us be more productive. Although it may make us feel like we’re being productive (thanks, dopamine!), and being perky all the time helps us avoid napping (after all, no one naps but old people and babies, right?), study after study has shown that caffeine is no better than breaks! This is counter to soceity’s (and the software industry’s) notion of problem solving – the way to solve a hard problem is to down a ton of coffee, spend 13 hours straight in front of a computer, and emerge victorious. This equating of effort and productivity is starting to be addressed in some circles (such as lean’s belief that utilization is a poor measure), but the idea of “flow” (i.e. uninterrupted work for long periods of time) is largely taken as Truth.

Linda presented a number of interesting facts that she’s gleaned from studies:

• caffeine mildly improves performance of extroverts, but decreases the performance of introverts – not good given the programmer stereotype!
• human attention span / concentration is cyclical, not continous – see promiscuous pairing and the pomodoro technique
• naps are a learned technique, and finding our your cycle of sleep (and wakefulness) can accelerate learning and development
• in 1985, Jolt cola came out with *gasp* 72mg of caffeine in 12 oz. Now? You can get beef jerky with 150mg. Caffeine has become more pervasive and is used in greater amounts than ever

How does all of this relate to an Agile conference? Linda’s conclusion is that, since one of the main principles of Agile is to make it more about the people, why do indviduals (with organizational support!) revert to a command and control style of personal sleep / wakefulness management? Why, though most software developers know Brook’s Law, do we insist on tackling a difficult problem strictly by throwing more efffort at it? She advocates a re-evaluation and beginner’s mind approach to the subject of alertness, concentration, and productivity. If you dispel the productivity myth, work to find an individual’s learning and attention span cadence, and use proven methods of productivity enhancement (breaks, naps, regular context switches, exercise – treadmill desks!) instead of disproven ones (energy drinks, burning the midnight oil, “sleep is for the weak”), you can see real and tangible improvements in both worker productivity *and* happiness.

I was only a 4-cups-a-day person for a few months (while I worked in the games industry – never going back!), but I know of several people who were in Linda’s talk, myself included, who have vowed to get more sleep, cut down their caffeine intake, and even take the occasional cat nap.