Lean development is not the same as Agile

Agile project management techniques recognise that software project management is mostly about communication between the ‘client’ and ‘development team’. Lean techniques instead focus on determining as quickly as possible what adds value and what does not. My experience is that lean techniques recognise that value is added most quickly by increasing ‘communication’ with the end user. This communication is rarely explicit. Lean techniques emphasise implicit communication through observed usage, metrics and comparative split tests – real data quickly trumps opinion. Lean accelerates learning, minimising work in progress and the cost of learning what really adds value.

Agile improves on Waterfall

Agile project management techniques recognise that software project management is mostly about communication. Older techniques such as the Waterfall process spent too much time attempting to capture and formalize what the client actually wanted. Often vast amounts of time and money would be spent writing and agreeing abstract specifications which invariably turned out not to accurately reflect what the client envisaged. The creation of these abstract specifications did not identify and predict the technical difficulties the project would face and the magnitude of problems. Many of you will have seen the Tree Swing Cartoon about requirements.

Agile techniques address these issues by emphasising communication between the development team and their client throughout the project; accepting that it is very hard to determine exactly what a client wants without delivering designs and working software. Clients will change their minds as they see how in reality the things they envisaged actually work. Developers will discover that some features they thought would be simple to implement are not so simple. Both party’s understanding of the facts on the ground improve vastly as they make progress. They both accept that they need a dialogue. Both parties accept that it is sensible to make changes to the project “plan” as it progresses to completion and delivery.

Agile addresses the communication gap between developer and client. The dialogue leads to sensible decisions, better use of resources (productivity) and greater predictability. Agile techniques means software development increasingly does a better job of meeting the expectations of its clients. However, often there is another very important person who’s been left out of the conversation, the end-user (e.g. the customer on an e-commerce site). Agile is still relying on the client’s ability to determine what the end-user wants and what works.

The development team can help with this process (that’s product development team, including product management, UX and developers). UX people often use usability testing techniques to gather opinion and identify problems with designs. These are really useful and valuable, identifying some problems, encouraging best practices and avoiding opinion deadlock. Unfortunately they are highly artificial, the few users in their tests are often not actual customers, with an actual need, using their own time and money.

If used correctly usability testing techniques can smooth the edges of a design, making it flow better and making it easier to use. However, they are based on small sample sizes, often gathered from the same geographical source and demographic. They often miss the nuances of other demographics and of how people actually behave when it’s their own money they’re spending. For example, I’ve seen data that showed 6% of UK customers attempting to use just their house number and postcode to specify their address, on many sites that is insufficient. What are the chances that this 6%’s actual behaviour would be picked up in a artificial usability test – unfortunately it is not high.

There is a new kid on the block which aims to address this gap and bring the 3rd and most important party in on the conversation – the end user, in HD and in all their magnificent glory, warts and all. The new kid is Lean Development.

Lean

In October (2011) Eric Ries’ best selling book, “The Lean Startup: How Constant Innovation Creates Radically Successful Businesses” (which I thoroughly recommend by the way) was released to critical acclaim and quickly became a New York Times Bestseller. Eric’s book focuses more on startups and product development rather than specifically software development per-say. The main theme it shares with Lean Software Development is this gap between what the experts in a business think about a product or service and the reality of what the end-user/customer values and how they use it.

The lean approach borrows its name from Lean Manufacturing techniques such as the Toyota Production System. Lean manufacturing works on the basis that parts are pulled through the manufacturing process as they are needed, work in progress is kept to a minimum. This makes the manufacturing process highly reactive to customer demand and able to cope with day to day manufacturing problems whilst maximising throughput. If orders for a product increase, more of its parts are pulled through the manufacturing system. If there is a problem with manufacturing of a part upstream, then it temporarily stops pulling the parts it depends upon up through the manufacturing process.

In software development we’re not talking about manufacturing but product development, the process of determining what features a product or service should have, how it should work and making the feature available. Our goal is to build things that users value, that help them achieve their goals. For example, in ecommerce that’s finding a site, finding the product they want and successfully purchasing the product.

Lean development recognises that experts are making calculated guesses about what problems a user has, what they will value and how they use a feature. There is nothing wrong with this. We need to start somewhere and an expert’s guess is a very good place to start and with no information to the contrary it’s also a good next move. However, it is a bet, we are betting that the expert is correct. Lean development aims to reduce our exposure to this bet by reducing work in progress to a bare minimum.

Using a lean process, each new release of the software contains the bare minimum additions/changes needed to test the experts assumptions. We determine what the user thinks through observing how they behave, collecting data through analytics and analysing our own databases and software logs. This Minimum Viable Product (MVP) or feature (MVF) is the least work we can do, the least money we can spend to test the expert’s hypothesis, to learn more and plan our next step.

This leanness, this minimization of work in progress is key. The emphasis is build something quickly, try it, look at data and feedback, improve – quickly, repeat. Lean is real and fact based, it implicitly involves the end-user, it makes them a key part of the product development conversation. The user drives the development process, implicitly prioritizing and “pulling” the new features or tweaks that they want through the development process.

Agile processes have plenty of regular conversation with the client, including regular demonstration of progress. When using an agile process, releases of new software with new features to the end-user are months apart. In a lean process, the conversation involves the end-user, so time between releases is measured in days not months.

In a Lean process, real data quickly trumps opinion as assumptions are quickly put to the test, features evolved – refined, modified or even removed. Ideas for what to try next are picked up from scents in the data. These scents are not always purely quantitive, for example an increase in the number of times a question is sent to the support email address may provide a clue to a problem or opportunity.

Lean projects ARE typically explicit goals where success can be quantified and measured, for example the number of sales per day increases. Agile projects often have explicit goals but ones that are not explicitly quantified, instead Agile projects tend to be specific about how much time is available. Lean projects tend to focus on achieving quantifiable value for the end-user. Lean development recognises that delivering real tangible business value is more likely if you iterate quickly and respond to what the end-user is actually doing.

Lean development is not agile development – lean extends the product development conversation to the end-user and minimises the businesses exposure to expert bets.

Of course, the real world isn’t quite a black as white as I’ve just painted it. In reality the differences between what Fred calls Agile and what Wilma calls Agile can be huge. Some people’s “agile” processes will look more like the Waterfall process whilst others will share the attributes of a lean process to a greater or lesser extent. So please forgive the blunt contrast I’ve painted to illustrate my points. I hope you’ve enjoyed this post.

A/B testing small tweaks vs big changes

Found this question on Quora, “Should early stage startups be A B testing landing pages rather than isolated elements like headline copy?”. Here’s my answer…

As you’ve probably already found, given low traffic/data the tests may take a while to deliver any results. As either large or small changes could have significant impact, I wouldn’t advise testing big changes first, nor would I advise testing small ones first.

I’d start by trying to understand what problems there are:

  1. Think of the your goals for the site e.g. purchase, signup for service, signup for newsletter etc.
  2. Create some basic metrics for these goals e.g. number of signups, number of visitors, number of visitors who see signup page
  3. Create some “funnel” metrics from these e.g. % of visitors who see signup page, % of visitors who signup.

Once you’ve got these metrics, pause. Do any of these metrics surprise you? e.g. say the % of visitors who see the signup page is much much lower than you’d expected. It’s a good idea to get a few people to look at the metrics on their own and list what surprised them, then get together and compare.

Once you’ve some agreement on which metrics surprised you, you can prioritise which you’d like to improve. Take this list and brainstorm reasons for worse than expected metric / possible solutions. This is your test plan.

With testing it’s always worth bearing in mind the ROI of things. Take an incremental approach. If you have 5 suggestions for improving a metric but the cost (time) of doing them varies considerably then go for the lowest cost one first.

Leaner than agile: Better products more quickly and cheaply

This post advocates the use of a ‘lean’ software development process for web based products and services. I outline why those involved in product marketing and software development should consider using a lean approach in their organisation. Where I use the word ‘product’ you can happily substitute ‘service’ if it suits you better.

I caveat the above with “web based products and services” because the web offers fantastic opportunities to measure how the customer values products and features. Value can be measured in various ways, for example measuring usage and goal attainment, running split tests and getting feedback. One or more of these feedback mechanisms are essential to the lean approach.

So what do we mean by a ‘lean’ approach? The term ‘lean’ is borrowed from lean manufacturing processes (such as the Toyota Production System). Wikipedia defines Lean Manufacturing:

Lean manufacturing or lean production, often simply, “Lean,” is a production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination

For software development, my approach to lean is to:

Take the smallest possible step that can test an assumption or idea; move quickly in small increments and learn from each step we make, minimize work in progresses and quickly learn what our users value.

This enables us to quickly determine what adds value whilst wasting less development time on features which do not add value. This usually means minimize work in progress, releasing early and often, measure and test value to the user, then iterate – quickly. Here’s the Wikipedia entry for Lean Software Development for other’s take and perspective.

Values and processes

Before we go into this in depth, I’m not saying one software development process is universally better than another, every business is different and puts a different value on different things. The perfect process for a particular company can’t be found in a text book or blog post, to a certain extent it requires some trial and error to match the company’s values. A company’s values may change over time as a company grows. Use the right tool for the job at hand.

When I use the word ‘values’ I’m not talking about some airy fairy spiritual thing. Some business have customers that value stability and no change (or very slow change) in the product offering. Some services should be straightforward and just work – always, no exceptions, ever. Other businesses will make amazing gains from adapting their products quickly to delight their customers and gain business from their competitors. Startups usually fall into the later category (particularly early stage startups). For them there is immense value in their ability to quickly put new features and products out until they find the right markets and products/features for those markets.

For me one of the primary duties of the product development team is to maximise return on investment or put in less scary language their effectiveness. In the context of product development, I define effectiveness as:

Effectiveness = value added / cost of adding that value

Our aim is to increase the value of our product from our users’ perspective (hopefully also leading to more users) at a reasonable cost. You can think of cost as money or time or a combination of both – the gist is the same.

Adding value requires discovery of what adds value

To increase value, it is essential that we find which features users use and which features either help them or encourage them to complete goals. This is an investigative process, a search that involves lots of trial, measurement, refinement, retrial and so on. For a startup product development is a race, you want your product to be valued by users as quickly as possible. Learning from measurement, testing and feedback is *key* to the lean approach.

Discovering which features users really value or how features ought to work is rarely as simple as *just* asking them. Ask a sociologist, often people don’t do what they think and say they do. Over the years I’ve done a fair bit of split testing and testing by observing users complete test tasks. In both cases we’d frequently be surprised at which features users did and which they did not use. Do gather and encourage verbal/written feedback from your users but use this to inspire investigation, as a catalyst and not as sufficient evidence in itself.

The quick cycle of feedback then iterate again is essential here. With the non-lean approach you might get lucky, hit upon something and add a dramatic amount of value but then again you might not. The same can be said for the lean approach but the cost of discovering this is lower and you can learn and try again quickly. The non-lean approach will probably lead to increased value. However, the non-lean approach probably won’t maximise value or do so quickly and at minimal cost.

Minimize Work-in-progress and avoid queue bloat

Minimizing work-in-progress is important to improve effectiveness by reducing the amount of work which does not add value. For software, by work-in-progress I don’t mean just what the developers are working on at that particular moment, I also include anything they have recently finished which has not yet been released.

What happens when people have to wait for the software development team to get their new features developed and released? Let’s define ‘lag’ as the time from requesting some work to release of that work. Let’s consider how high lag leads to bloat and waste..

Often customers of software teams are like very hungry people waiting for their first meal for a day or two. Once they get served, they’re gonna eat. They’ll eat plenty more than they really need and become bloated. So, using the royal ‘I’… My development slot doesn’t come up another 6 weeks and even then it won’t get release for say another 6 weeks. Any new features or improvements I want won’t get released for another 12 weeks, so I’m going to stuff it with as much as I can. This probably means I’ll ask for a larger slot of development time than I really need, making others wait even longer, increasing the general hunger and leading to more bloat. The queue of work starts to grow with work that is not really necessary.

High lag puts people off trying alternatives. If you don’t try alternatives then you’re not searching for the best solution, not innovating. They’ll polish their new baby in the hope that it’ll succeed in what may be its one shot at success (at least for fairly long time).

Predictability vs Effectiveness

Organisations often put too much emphasis on predictability to the detriment of effectiveness (Value added / Cost of adding that value). They like the certainty that work will begin on features A, B, C, X, Y and Z in 6 weeks time and 6 weeks after that these features will be released. Hit those timescales with those features and nobody can have the finger of blame pointed at them. Except of course the people who are responsible for growing the business fast and before they run out of money, or get beat to it by a competitor.

It’s not just bloat, there’s another problem with 6 week plus release cycles. People become frustrated with the lack of progress and blame the busy development team. In their hunger, people start attempting to jump the queue. Jumping the queue leads to ill feeling and bad decisions. Some may argue that the solution is just to ban queue jumping, to staunchly refuse it. Resisting queue jumping is just treating a symptom rather than addressing a cause – a need. I think queue jumping is inevitable in a business that has customers and competitors and is measuring, testing, listening and responding to the world around them. Priorities change as we learn more about things, it’s a fact of life.

Organising Lean

So treating the cause rather than the symptom. If our approach is to learn from the customer by trialing features and ditching or iteratively improving them based on measurement, tests or feedback, then we’re learning. If we’re learning then our current priorities will change as we learn more. If our priorities change, then it’s important to be adaptive. The key to being adaptive is low lag, that is being responsive and minimising work in progress – a lean approach.

With lean we are searching for how to add value. It’s a search so by definition we don’t know everything up front before we begin. A common question about lean processes is what does a project look like and how is it organised. I like to define projects in terms of their goals rather than their steps. So for example, I might initially define a project as the goal, “Increase the percentage of forum users providing answers to other users’ questions”, then working with the team, we’d think about ways of doing this and how to measure and test them, all the time looking to keep the steps as small as possible.

You will likely have multiple projects going at the same time, depending upon the size of your team and how long it takes to collect sufficient data or other feedback. Switching between each project as things are learnt and next steps are determined. I’ve found the best way to organise this is as a list or work ‘pipeline’ with the following sections:

Backlog => (Pending => In progress => Ready) => Released

Some people call this a Kanban chart or board and use an actual board with post-its. Personally, I prefer a shared document or dedicated agile project tracking system such as Pivotal Tracker

New work requests are added to the backlog, which is kept in priority order. Like an agile process, before the next iteration begins, the decision makers should review the backlog and determine the highest priority items which will progress into the pending section at the start of the next iteration.

The sections enclosed in parentheses – Pending, In Progress and Ready are the work-in-progress. We minimize this by releasing early an often.

Summary

So in summary, the aim of lean software development is to improve effectiveness (return on investment), adding maximum value through measurement and feedback, and reducing or eliminating work on features which do not add value.

  • Always take the smallest possible step necessary to determine if we’re adding value

    Keep asking the question, is this really the minimum step in better understanding if this feature will be valued by users?

  • Learn. Measurement and feedback are king

    Get feedback, that is: usage data, split testing and/or testing with users.

  • Move quickly

    Minimise work in progress and release often.

  • Iterate and investigate. Innovation is a search, don’t be afraid

    The quicker you move the more you can try, the more you try the more likely you are to get a result. Remember you are measuring and getting feedback so you’ll know if you make things worse. As you’re moving quickly, you can quickly put things right quickly too.

  • Be responsive

    If you are not responsive, fear will take hold and bloat will gradually kill effectiveness (value/cost).

Page speed & conversion: Mozilla increase Firefox downloads 15%

I’ve previously written about the importance of page speed to optimizing conversion. Mozilla achieved a 15% increase in downloads of Firefox by speeding up their download page.

The guys at Mozilla noticed the page for downloading Firefox was slow. They worked out how to speed the page up. Split (A/B) tested the new fast page vs the old slow page and achieved a 15% conversion increase – in their case 15% increase in downloads of Firefox which could amount to an additional 60 million downloads per year.

Here’s their concise post on how they improved conversion through improved speed.

Site Conversion and Lipstick on a Pig

“You put lipstick on a pig, it’s still a pig” – Barak Obama

If I’m looking for a 40 inch LCD HD TV and your site only sells 28 inch Plasma you’re not going to make a sale. If you’ve spent money advertising your “wide choice of LCD HD TVs” then too bad :-p

When it comes to ecommerce website conversion the strongest lever probably isn’t how the site looks. If you don’t have the product people are looking for at an acceptable price then endlessly tinkering with how the site looks won’t make a jot of difference to conversion.

If you’re an ecommerce site then spend some time and effort determining what visitors to your site are currently looking for. They’re probably giving signals left right and center. What do they search for? What filters do they apply? What product sections do they click through to?

Thanks for reading,

Paul

P.S. my earlier, longer and slightly more serious post about 3 ways to improve conversion

3 Approaches to Improving Website Conversion

Looking to improve your site’s conversion? Many people focus a lot of effort and money on improving the design of their site but is that the only way of improving conversion? No, there are other approaches that you may not yet have considered. Whilst these are perhaps less obvious, in many cases using these could lead to considerable conversion improvement - often more significant than continuous improvement of a site’s design.

This post outlines 3 approaches to improving website conversion: 1. Improve site design (the usual focus of effort); 2. Improve site speed; 3. Demand / Supply match (for ecommerce sites this is often untapped and has much greater potential than 1. Future posts will investigate and elaborate each of these approaches.

1. Improve site design

This is the most obvious and most common approach to conversion improvement. I split site improvement into Aesthetic and Ergonomic improvements:

Aesthetic Improvements

Improving the visual attractiveness and appeal of the website in the hope that people will trust the ‘brand’ and aspire to own its products or use its services. Many many people focus the vast majority of their effort on aesthetics. Perhaps because often the people responsible for marketing have a background and experience in static print media, brochures, print adverts etc.

Ergonomic Improvements

Improving the usability of the site. Are the calls to action obvious, well labelled, placed in consistent and logical positions; does the site help try to prevent errors; is the flow between the various stages of a checkout process natural and logical, can the user undo previous actions and change previous personal information, quantities of product etc?

2. Improve site speed

A less obvious way to improve conversion is to improve the speed and responsiveness of your website. I say this is less obvious, perhaps it is obvious that faster is better but how many of us settle for a slightly sluggish site, where pages take 1 or 2 seconds to appear? What is less obvious, is that loading pages faster (certainly faster than 1s) can make a significant difference to conversion. Why?! Fast sites encourage learning by investigation. There is no wait “penalty” for clicking on links/buttons. The browsers’ Back button gives a quick ‘undo’ feature.

Not convinced!? Take a look at the effort Google are going to to speed up the web with Chrome, Speedy, Google DNS etc. Google even have a website dedicated to it, Lets Make the Web Faster. As Google’s Vice President of Search Products and User Experience Marissa Mayer says, “Users really respond to speed.”. Various studies and experiments by Amazon and Google show significant improvements in usage and conversion with speed.

Personally, the biggest step change in conversion I’ve ever seen occurred when we improved the page response time on a site by a factor of 4, taking pages from a slow typical response time of around 2s to just 0.5s. In that case it was pretty obvious we needed to do something about speed but the magnitude of the improvement in conversion was quite surprising – and very welcome!

3. Sell what your users are looking for

“Sell what your users are looking for” and its corollary, “Acquire users that are looking for what you sell”.

Sounds simple and obvious doesn’t it! But is it? What are your users looking for? I don’t mean what do you think they are looking for but what those actual users are actually looking for on your site. Do you know? Do you know what they’re looking for this month? Next month will you know what they are looking for?

I’ve seen this point arrogantly and confidently dismissed by some very experienced people with many years experience in busines but not online business. Why did they dismiss it? Well their years of experience in their industry sector and their sales figures showed that most customers wanted product X. They weren’t wrong about average trends across this industry sector but what they hadn’t considered is what the customers that their marketing was driving to the site where looking for might not match the industry averages. The mismatch between the demand their marketing created and the products they sold meant they had low conversion and hence poor return on their marketing spend at all but the lowest levels of spend. Their marketing was a very weak lever and as such it was hard to compete and hard to scale the business. Their best selling product was product X but if you stock product X rather than products Y and Z, that’s not surprising is it. As it happened about 35% of visitors where looking for product X but the other 65% where looking for products W, Y, A, B and Z which they didn’t offer. 65% of their visitors just were never going to convert into customers because they didn’t sell what these people were looking for.

Online business is not different to Offline business in some mysterious nefarious way, but with online you can relatively easily measure customer intent and desire and perhaps quickly respond to it. Business is competitive, if you don’t respond, sooner or later someone else will and they’ll beat you.

There are two approaches to solving this demand/supply mismatch: 1) change your marketing to attract more targeted traffic to your website, traffic that wants to buy product X; or 2) start selling the products your users are looking for, products W and Z as well as X. Before you can do either of these though, you need to know what user are after.

How do you know what your website’ users want? Look for onsite signals. Can you redesign your site so that users self-segment? Perhaps you can add filters or look at what people are searching for. Do you have this data? Have you ever analyzed it? Given this data, is your marketing spend proportioned to match?

For example, say your site sells televisions, what size televisions are they looking for? What % of users click on the “40 inch” filter? How does this compare to the products you offer? What if 35% of your users are looking for 40 inch televisions but only 1 of the 20 models you stock are 40 inch, 35% of your demand is for 40 inch televisions whilst just 5% of the range you offer matches. Even worse, what if you spend 40% of your money on adwords adverts for terms like, “40 inch lcd” or perhaps you have a banner advert on a home cinema lovers community site which mostly attracts the people looking for relatively large televisions.

Few businesses realise and make profitable use of such data, yet it is potentially available on many many websites, certainly all ecommerce sites.

I’ll follow up on each of these areas with future posts and update this post with links to the follow up posts. In the mean time, I hope you’ve found this both useful and thought provoking.

Tracking 404 and 500 errors in Google Analytics

Everyone would like to know if their users are seeing error pages due to broken links (404 not found) or server side errors (500 server error). Here’s how you can use Google Analytics and the relatively new Google Analytics Event Tracking API to track these errors.

There’s some brief advice on using this technique with WordPress, Rails or Apache at the bottom of this post.

Prerequisite – ga.js Script

Prerequisite, I’m assuming you’ve got Google Analytics set up for your site. If you did this some time ago, you might still be using the old urchin tracking script. If you are you’ll need to update to the newer ga.js script. If you know you are using ga.js, then feel free to move along to the next section.

If you are not sure, take a look at the source of one of your pages. If it looks something like the snippet below, saying urchinTracker() then you are using the old urchin tracking script and will need to upgrade:

Here are Google’s instructions for upgrading from the older urchin to the newer ga.js script.

If you are already using ga.js your pages should contain something similar to this:

Event Tracking Script

The event tracking script itself is very simple, a single line of javascript will log an event with Google Analytics:

The first parameter ’404′ is our new event category, the second parameter is the URL that has not been found, and the third parameter is the URL of the page that referred the visit.

This line should be added to your Google Analytics tracking script on your 404 Not Found error page, for example:

For 500 Server Errors it is very similar to the above but change the ’404′ to ’500′. Your tracking script becomes something like:

Viewing 404 and 500 Events in Google Analytics

Google Analytics Events are listed under Content. To see your new 404 and 500 errors, select Content / Event Tracking / Categories in the Google Analytics sidebar. You’ll see 404 and 500 listed as Categories once there have been 404 or 500 events. You can view details of these events by clicking on a particular category.

Here’s a screenshot of 404 events on one of my sites – I’ve deliberately created some 404s by pointing my browser at random-ish urls on my site’s domain:

Viewing 404 events in Google Analytics

Enjoy!
Paul

404 and 500 tracking on Apache

If you are using Apache and haven’t already specified a 404 error page you can do so by adding the following to either your .htaccess file or inside your site’s virtual host element:

ErrorDocument 404 /path/to/your/404.html

Replace /path/to/your/404.html with the path to your 404 error page. The 404 error page is just a plain old html page.

For 500 errors, you’ll need to specify a 500 error page in either your .htaccess file or inside your site’s virtual host element just like we did for 404 errors:

ErrorDocument 500 /path/to/your/500.html

404 and 500 tracking on Ruby on Rails

Ruby on Rails users will find you already have 404.html and 500.html files inside your public directory. You can add the GA tracking script there.

404 and 500 tracking on WordPress

WordPress users can add the following lines to their theme’s footer.php inside the Google Analytics tracking script:

So for example, your footer.php should look something like: