Ecommerce Resonance – Optimise Conversion by aligning Demand, UX and Supply

From Wikipedia, “.. resonance is the tendency of a system to oscillate with greater amplitude at some frequencies than at others.”

Have you ever heard someone say, “our website’s conversion is too low, we’re not ready to start spending money on PPC/Display advertising/etc..”. I’ve heard variations on this theme at a few companies. Often such statements receive firm unequivocal nods from around the table – conversion is low therefore the website isn’t good, it isn’t fit for purpose.

At a high level such statements seem self-evident. However, they miss a fundamental truth about conversion optimisation: Conversion Optimisation is not all about website UX. As I mentioned in my previous post on ecommerce:

Conversion is a measure of how well our site and supply meets the needs of the users that our marketing attracts.

I’ve highlighted the 3 components of conversion: site, supply and marketing. These 3 components need to be aligned so that they resonate – reinforcing each other not reducing each other. Think about these scenarios:

  1. If a user is looking for a red sports hatch, you’re probably not going to be able to sell them a caravan (demand vs supply)
  2. I want to book an apartment in Berlin on the 25th of next month but the accommodation website I just found through Google only has 4* hotels available (demand vs supply)
  3. I’m looking at a list of insurance products but it’s really hard to find which ones offer annual cover (site but also potentially supply and demand too)

Does your site sell what the user is looking for? Can you improve supply, aligning it so that it does? Can you improve demand to attract users who want what you are selling – so that it’s better aligned with supply?

Can you improve your site so that users tell you what they want? Consider changing your UX so that users can self segment, identifying to you what they want to buy. If we measure what users are looking for and share this around our organisations, we can create positive feedback loops:

  • supply should look to fulfil demand
  • demand should attract the right demand for the available supply
  • UX should help the user tell your organisation what they want

Here’s another slightly different scenario, this time it’s about how demand mix can affect conversion, specifically what I call “early vs ready”:

I am reading about things to do in Vancouver, I’m curious about hotels but I’m not ready to book a hotel today, it’s too early and I’m just planning (demand mix – early vs ready)

Your site may well have what the user is looking for but maybe it’s too early in the purchase cycle for the user to buy – they are “early” not “ready”. A marketing channel that attracts a lot of “early” but not “ready” will likely convert poorly. That’s fine so long as we understand that this poor conversion is a function of the marketing channel not the website.

The length of the purchase cycle, the transition from “early” to “ready” varies greatly depending on the product/service we’re selling. Having a marketing channel that is largely made up of users who are “early” but not “ready” is an opportunity but we’ll need to work hard on strategies to get the user to come back when they are ready to purchase.

Conversion is not just about site design.

Read my previous post: Ecommerce: Get the “right” products to the top

Ecommerce: Get the “right” products to the top

Whether it’s the results of a search on Google, Amazon, an ecommerce shop or a hotels website, users predominantly click on the top results in a list. Take a look at all these charts of clicks by position in Google Images. This is common sense – we expect the best results for us to be towards the top of the list. It’s imperative that we design our websites so that we get the “right” products to the top.

Which products are the “right” products? Well “right” is a personal and occasional thing. Here are some approaches that I’ve successfully used:

  1. Encourage use of filters. You are measuring how your design affects this – right? What % of users who view a list also use a filter?
  2. Encourage the user to self segment by product category. For example: hotels, apartments and guesthouses are pretty different beasts.
  3. Default order based on predicted revenue per impression – or put another way, users vote for products by buying them.
  4. Put unavailable or out-of-stock products at the bottom. For travel products, encouraging the use of travel dates can have a big impact

Encouraging users to self segment by product category is obviously a form of filtering. I highlight it separately because it’s potentially the most important filter and it can be used to give you a metric of demand by product category. So for example, on an accommodation website, by destination, you can see what percentage of your demand (users) are looking for apartments (vs hotels vs guesthouses etc). This metric can be used to improve supply. It may also be used to adjust demand – perhaps altering your advertising to get an appropriate level of demand to match your supply. To maximise conversion, we want demand, UX and supply aligned so that they resonate..

In many companies the default ordering of products is often a highly political topic. Sometimes it rivals homepage design for the political ire that it generates. A common concern is that this is simply putting the most expensive products at the top. That is incorrect and would indicate a poorly designed algorithm which does not maximise conversion or revenue. I believe this often stems from a fundamental misunderstanding of what conversion optimisation is about. Conversion Optimisation is not about increasing revenue by selling the more expensive products, it’s about maximising revenue by selling more of what users want. Conversion is a measure of how well our site and supply serves the needs of the users that our marketing attracts. Conversion is a measure of user satisfaction.

The lean-mindset – build simple solutions for the most likely use cases

Recently, a recruiter contacted me on LinkedIn wanting to discuss a CTO role at a well known internet travel company. The recruiter mentioned they wanted someone who had managed/directed 100+ people. I’d only directed a considerably smaller team – about 20 talented people. So it was obvious to me that I probably wasn’t the person they were looking for. Even though this wasn’t a good match, we politely continued the conversation. The recruiter was very surprised when I revealed that my team of about 20 had developed and maintained sites delivering several hundred million dollars of transactions every year and we had 10x growth over just a few years. Reflecting on this, I think what surprises people is that we achieved so much with relatively few people in pretty short space of time.

How were we so productive?! Well we were a startup, or at least had been a “startup” just a few years earlier. The startup mentality was still in most of our DNA at that point. Pragmatism ruled. My aim was to deliver real tangible value ($$) quickly. I recognised that I did not know with very high certainty what features or UI designs actually worked, actually delivered value ($$). I also recognised that neither did the UI experts, the marketing experts or the developers. How? Well we’d conducted quite a few AB tests and a good few user testing sessions, what they taught me is that whilst we experts have a good feel for what might be a problem and what the solution might be, we also get it wrong too often. Getting it wrong too often means at best you get slow progress.

We needed a process (definitely lower case ‘p’) that enabled us to do as little as possible, as quickly as possible, to put our ideas and assumptions to the test – running experiments on our websites to gather real insight and facts about what adds value. This was the motivation for adopting a lean and metric driven process. I’m not sure the terms ‘lean’ and ‘metric driven’ where in use circa 2006, if they were I certainly hadn’t heard of them.

Whilst struggling to cope with a myriad of new work requests, bugs, changes in priority, and react to the results of AB tests, I stumbled upon the idea of modelling tasks like a sales pipeline. As I understand it, sales often kept a list of potential sales ordered by likelihood. These ‘pipelines’ are often sectioned off into states like: ‘closed’, ‘in progress’, ‘contacted’, ‘dead’. Each non-”dead” prospect is assigned to someone. The pipeline is continuously updated. At any time someone can look at this single document and understand the state of sales work and who is working on which lead or prospect. New prospects can be added very quickly and easily. Updating the pipeline to reflect a prospect progressing through these states is trivial. Some ‘prospects’ result in a sale, some die, some lead to other things. Prospects and their status and priority are fluid.

It occurred to me that this approach handled uncertainty and changing priorities really well. Exactly the same attributes our development process needed to handle. I adopted a similar process for organising and communicating my teams tasks and deliverables. My ‘pipeline’ had sections marked: ‘done’, ‘in progress’, ‘upcoming’, ‘todo’.

It was important that we could quickly change priorities. Changing priorities meant I did not want lots of ‘in progress’ tasks. I decided to create a rule of thumb – nobody should have more than 1 task “in progress”. Like many constraints this was more difficult to adhere to than you may at first think and the constraint had knock on effects. The most important consequence is that tasks/deliverables need to be very small and self contained.

Having only very small tasks/deliverables is actually hard to do. It requires a uber-pragmatic mindset. You cannot get caught up building all encompassing solutions that cover every possible use case or outcome perfectly from the beginning. The important words here are “every possible”.

Technical people are trained in system thinking. Good technical people think of every possibility, every possible combination of inputs, states and outputs. This is fabulous but it means it’s very easy to get carried away, solving problems with little consideration of the probability that a problem occurs and whether the cost of a solution is really justified. (I’m a teccy. Don’t hate. This is just something to be aware of..)

I’d already bought heavily into the idea of using metrics to identify what worked and what did not, to “smell” opportunities for improvements. I quickly learnt
that to organise and schedule small tasks/deliverables very often meant building the simplest thing first and for the most likely use case. Making our solutions more sophisticated would be an iterative process. We’d put the first piece of work live and look at our metrics for data points that suggested other use case worth investing in. Until we had data points that suggested other use cases where were worth investing in, I’d “schedule” them as low priority – in our ‘todo’ section.

This is a very subtle but important point. How often have you sat in a meeting discussing solutions to edge cases? It can go on and on and on. Edge cases by their very nature are often difficult (expensive) to solve well. How often have you or your team invested good time (and hence money) solving those edge cases? With the magic of hindsight, would you have bothered if you’d know how little those edge cases mattered, how little monetary difference they made. How little value they added for your users?

Some of you have read that and are thinking something along the lines of, “but I do care, I want ALL our users to have a great experience” and “Nonsense, what would Steve Jobs make of that” etc. I don’t necessarily disagree with those sentiments but we’re talking about priorities. Priorities are not yes or no, they’re just about when (and only sometimes if). We’re also talking about product development on the web, where there is massive opportunity to collect data, learn and often quantify what’s really important.

Lean is a mind-set, perhaps more than it is a process. Do the simplest possible thing to address those edge cases and learn how often they happen. If they happen very infrequently then they’re not that important. Alternatively, if the data says they happen a lot, then that is the point you make the decision to invest in finding and delivering solutions.

Related Posts:

Lean development is not the same as Agile

Agile project management techniques recognise that software project management is mostly about communication between the ‘client’ and ‘development team’. Lean techniques instead focus on determining as quickly as possible what adds value and what does not. My experience is that lean techniques recognise that value is added most quickly by increasing ‘communication’ with the end user. This communication is rarely explicit. Lean techniques emphasise implicit communication through observed usage, metrics and comparative split tests – real data quickly trumps opinion. Lean accelerates learning, minimising work in progress and the cost of learning what really adds value.

Agile improves on Waterfall

Agile project management techniques recognise that software project management is mostly about communication. Older techniques such as the Waterfall process spent too much time attempting to capture and formalize what the client actually wanted. Often vast amounts of time and money would be spent writing and agreeing abstract specifications which invariably turned out not to accurately reflect what the client envisaged. The creation of these abstract specifications did not identify and predict the technical difficulties the project would face and the magnitude of problems. Many of you will have seen the Tree Swing Cartoon about requirements.

Agile techniques address these issues by emphasising communication between the development team and their client throughout the project; accepting that it is very hard to determine exactly what a client wants without delivering designs and working software. Clients will change their minds as they see how in reality the things they envisaged actually work. Developers will discover that some features they thought would be simple to implement are not so simple. Both party’s understanding of the facts on the ground improve vastly as they make progress. They both accept that they need a dialogue. Both parties accept that it is sensible to make changes to the project “plan” as it progresses to completion and delivery.

Agile addresses the communication gap between developer and client. The dialogue leads to sensible decisions, better use of resources (productivity) and greater predictability. Agile techniques means software development increasingly does a better job of meeting the expectations of its clients. However, often there is another very important person who’s been left out of the conversation, the end-user (e.g. the customer on an e-commerce site). Agile is still relying on the client’s ability to determine what the end-user wants and what works.

The development team can help with this process (that’s product development team, including product management, UX and developers). UX people often use usability testing techniques to gather opinion and identify problems with designs. These are really useful and valuable, identifying some problems, encouraging best practices and avoiding opinion deadlock. Unfortunately they are highly artificial, the few users in their tests are often not actual customers, with an actual need, using their own time and money.

If used correctly usability testing techniques can smooth the edges of a design, making it flow better and making it easier to use. However, they are based on small sample sizes, often gathered from the same geographical source and demographic. They often miss the nuances of other demographics and of how people actually behave when it’s their own money they’re spending. For example, I’ve seen data that showed 6% of UK customers attempting to use just their house number and postcode to specify their address, on many sites that is insufficient. What are the chances that this 6%’s actual behaviour would be picked up in a artificial usability test – unfortunately it is not high.

There is a new kid on the block which aims to address this gap and bring the 3rd and most important party in on the conversation – the end user, in HD and in all their magnificent glory, warts and all. The new kid is Lean Development.

Lean

In October (2011) Eric Ries’ best selling book, “The Lean Startup: How Constant Innovation Creates Radically Successful Businesses” (which I thoroughly recommend by the way) was released to critical acclaim and quickly became a New York Times Bestseller. Eric’s book focuses more on startups and product development rather than specifically software development per-say. The main theme it shares with Lean Software Development is this gap between what the experts in a business think about a product or service and the reality of what the end-user/customer values and how they use it.

The lean approach borrows its name from Lean Manufacturing techniques such as the Toyota Production System. Lean manufacturing works on the basis that parts are pulled through the manufacturing process as they are needed, work in progress is kept to a minimum. This makes the manufacturing process highly reactive to customer demand and able to cope with day to day manufacturing problems whilst maximising throughput. If orders for a product increase, more of its parts are pulled through the manufacturing system. If there is a problem with manufacturing of a part upstream, then it temporarily stops pulling the parts it depends upon up through the manufacturing process.

In software development we’re not talking about manufacturing but product development, the process of determining what features a product or service should have, how it should work and making the feature available. Our goal is to build things that users value, that help them achieve their goals. For example, in ecommerce that’s finding a site, finding the product they want and successfully purchasing the product.

Lean development recognises that experts are making calculated guesses about what problems a user has, what they will value and how they use a feature. There is nothing wrong with this. We need to start somewhere and an expert’s guess is a very good place to start and with no information to the contrary it’s also a good next move. However, it is a bet, we are betting that the expert is correct. Lean development aims to reduce our exposure to this bet by reducing work in progress to a bare minimum.

Using a lean process, each new release of the software contains the bare minimum additions/changes needed to test the experts assumptions. We determine what the user thinks through observing how they behave, collecting data through analytics and analysing our own databases and software logs. This Minimum Viable Product (MVP) or feature (MVF) is the least work we can do, the least money we can spend to test the expert’s hypothesis, to learn more and plan our next step.

This leanness, this minimization of work in progress is key. The emphasis is build something quickly, try it, look at data and feedback, improve – quickly, repeat. Lean is real and fact based, it implicitly involves the end-user, it makes them a key part of the product development conversation. The user drives the development process, implicitly prioritizing and “pulling” the new features or tweaks that they want through the development process.

Agile processes have plenty of regular conversation with the client, including regular demonstration of progress. When using an agile process, releases of new software with new features to the end-user are months apart. In a lean process, the conversation involves the end-user, so time between releases is measured in days not months.

In a Lean process, real data quickly trumps opinion as assumptions are quickly put to the test, features evolved – refined, modified or even removed. Ideas for what to try next are picked up from scents in the data. These scents are not always purely quantitive, for example an increase in the number of times a question is sent to the support email address may provide a clue to a problem or opportunity.

Lean projects ARE typically explicit goals where success can be quantified and measured, for example the number of sales per day increases. Agile projects often have explicit goals but ones that are not explicitly quantified, instead Agile projects tend to be specific about how much time is available. Lean projects tend to focus on achieving quantifiable value for the end-user. Lean development recognises that delivering real tangible business value is more likely if you iterate quickly and respond to what the end-user is actually doing.

Lean development is not agile development – lean extends the product development conversation to the end-user and minimises the businesses exposure to expert bets.

Of course, the real world isn’t quite a black as white as I’ve just painted it. In reality the differences between what Fred calls Agile and what Wilma calls Agile can be huge. Some people’s “agile” processes will look more like the Waterfall process whilst others will share the attributes of a lean process to a greater or lesser extent. So please forgive the blunt contrast I’ve painted to illustrate my points. I hope you’ve enjoyed this post.

Innovation, motivation & goal driven management

I believe that organisations that want to encourage innovation and improve employee morale should consider whether they can be more goal driven, focusing more on goals and less on tasks. All organisations are ultimately goal driven but often goals are turned into tasks too early, this immediately reduces staffs’ room to manoeuvre – to investigate, test, react, adapt and innovate to reach the goal.

Do you know that a set of features and associated tasks will delivery your goal? Do you know these features and tasks will deliver the best result using the time and resources available? Progress towards a goal is more important than completing a set of tasks. Allocate resources to the achievement of a goal and use those resources to explore problems and solutions, collect and use data to determine next steps.

Managers often prefer tasks because they predictably show a concrete output for a given input. Completion of tasks A and B delivers features X and Y. It can be demonstrate that task A has been completed and feature X delivered. The achievement of a goal, a measurable outcome, is often much harder to predict. Worse, metrics highlight sunk costs, where completion of a set of tasks does not deliver the desired goal.

This human tendency towards predictability over goal achievement is heightened by processes and structures that bet the house on the latest big idea. In a task orientated environment, a determined focus on goal achievement takes guts.

Management skill is stating the goal, posing the questions, defining and focusing on outcomes and the metrics around them. We need a process (lower case ‘p’) that helps us plan, agree, track, adapt and communicate progress towards a goal. Here’s my thoughts on such a process..

Start with the goal/s of a project and the timescale and resources available to achieve the goal/s. How will achievement of the goals be measured? These metrics should be outcomes not completion of tasks or features. For example, is the goal to increase conversion of a checkout process; to increase newsletter signups; to reduce customer service calls; to increase visibility of a product? All of these examples are goals and their achievement can be measured.

Brainstorm ideas for achieving the goals. Are there any prerequisites / stepping stones? Organise these as a tree you want to explore rather than a list of tasks that need completing. The goal is the root of the tree. The nodes are sub-goals, ideas you want to test and measure how successfully a feature may deliver a goal or contribute towards it. The tree gives you a mechanism to understand, track and communicate how the team/s are attempting to explore the problems, try solutions and ultimately how these contribute towards the goal.

Prioritise the nodes below the root, based on which you think are most likely to make progress towards your goal. Start work on the highest priority node. If you’ve multiple teams, teams can start on different siblings. Review results (goal metrics and sub-metrics), which sub-branches should you try next? Are there new sub-branches you want to add? Are they the next best thing to try? Based on your results, do you continue down those sub-branches or move to a sibling? The tree is dynamic, explore it, grow it, prune it based on progress towards your goal.

Goals and metrics for a project provides focus. They encourage conversation about whether the proposed features really achieve the stated aims. This encourages investigation, looking at data, proposing and conducting tests. Furthermore, it encourages conversations about the cost vs value of pieces of work and encourages everyone to bring to the table alternatives which may be more effective and/or lower cost.

Why not leave turning goals into tasks to the “doers”? Tasks (and expectations of their delivery) are a necessary part of successful management. It is important to know “what is going to be done” and “when it will be done”. However, I’d argue that tasks should be a proposed by the “doers” in response to being set a goal by the “leader”. The leader sets goals, puts them into context and explains how their achievement will be measured. The leader facilitates the process of turning goals into tasks and expectations about their delivery. The resulting tasks can be “hung off” each goal in the tree, abstracted away out of sight until they are needed, leaving the focus on goals.

The ultimate aggregator – Google, travel & mobile

In my post, Adwords, vertical search and aggregator ‘middle men’ I mentioned a simple search enhancement that could enable hoteliers to better compete with the aggregators (e.g. Expedia, Travelocity, Booking.com). When a user searches using the keyword “hotel” or “hotels” some simple filters such as room type or budget would be shown. Advertisers would be able to bid against keyword/s and filter value/s. This got me thinking about what else Google (or Microsoft) could do for the travel vertical and how the growth in mobile might precipitate this.

Google recently purchased ITA Software a provider of Flight information. This potentially enables Google to show enhance search results with lists of flights and prices for various carriers and routes. Google could piggy back Adwords adverts on the side of this; or they go further and offer advertisers the ability to advertise against specific carriers and routes. Could they do something similar for accommodation?

Google becomes an accommodation aggregator?

Google could themselves become an accommodation aggregator. Google Base or Google Places could be extended for hotels to take rates, rooms and availability information. The results of a hotel related search would be a list of hotels rather than the normal list of hotel related websites. The usual filters you see on an aggregator site would be available – including travel dates. Near each hotel result there’d be paid (Cost Per Click) links to (multiple) websites where a room in the hotel can be booked.

For users this seems like an improvement, cutting out a step and helping them compare prices from aggregators and perhaps the direct from the hotel itself. This is the most obvious move but unfortunately for the user this is perhaps also the the most unlikely.

Technically whilst this should be pretty straightforward, certainly for a company with Google’s resources, in reality it involves integrating lots of 3rd party systems and information. Room availability and pricing is ideally real-time information. Will 3rd party systems be fast enough for super speedy Google? Will they be reliable enough. Google sets the bar pretty high. What if the information becomes stale? The user finds a supplier of a particular hotel and room for £120, clicks through to discover it’s now £150 or worse no longer available. Still you could still make the case that this is better, certainly no worse for the user than the status quo.

The biggest problem is what would the aggregators make of this? Aggregators are very big spenders on Google Adwords and unlikely to voluntarily play ball on this. Getting the aggregators to put the effort into such technical integration is unlikely given they would probably stand to lose the most, at least in the short term

Currently, the name of the game is getting a big enough return per click to afford to appear in the key first few result that drive the vast majority of referral volume. The aggregators achieve this by aggregating lots of supply => lots of choice (see my previous post).

Hoteliers benefit from vertical search

Could Google circumvent the aggregators? Hoteliers could benefit hugely from such a vertical search. Their relevancy to any paid click would be tremendous, they’d be very likely to convert the click into a booking and achieve a good return on their click. Perhaps this is where Google Places is headed, adding real-time vertical specific information: rooms, rates and availability. Google could even become the preferred tool for independent hoteliers to manage their rates, availability and bookings. Perhaps Google wouldn’t want to be involved in the actual financial transaction, there are potentially numerous legal, regulatory and customer service problems (Google Checkout prohibits use for travel purchases). However, this wouldn’t stop Google offering a API through which a hoteliers website could instantly store booking information in Google’s systems or simply update their room inventory.

Holiday (vacation) lets

What about holiday (vacation) lets, villas and cottages? Google might see significant take up in this sector as it is more fragmented than hotels and technically simpler. Holiday properties are typically owned by private individuals rather than the “holiday companies” who market them. The holiday companies act as aggregators, offering advertising, calendar management and in some cases full booking service. Google could potentially offer some of these services: a simple advertising solution, calendar management and itinerary update. This is a ripe market, with plenty of opportunity for a technical innovator to come in and attack the cost base and effectiveness of the holiday companies’ traditional marketing and customer services models, both property owners and consumers would ultimately welcome this.

The mobile search problem and aggregator apps

Searching for a hotel via Google Search is painful on mobile. Often the sites listed are not designed for mobile, though no doubt in time most sites will have mobile versions. Personally I find clicking back and forth between Google Search and the sites listed far more clunky on mobile. I wonder if mobile users will adopt a preferred aggregator and stick with the aggregator’s mobile app. In time this could represent a very significant drop in Google’s ad revenue.

Google could avoid such a drift to aggregators’ mobile apps by more aggressively pursuing vertical searches on Google Mobile Search. Vertical search could reduce the steps necessary to find accommodation, pulling more of the steps into Google Search and enabling the user to enter a hotelier or an aggregator’s site at a later step in the selection and booking process.

In exchange for offering the user more relevant content, Google would be able to better understand what the user is looking for. Knowing more about the user’s goal and preferences opens up other opportunities. Having additional information about the user’s goals also enables Google to intelligently cross-link to relevant flights, cars, restaurants etc... More relevance leads to more clicks.

It’ll be interesting to watch and see, if and where Google Search, Places, Base and Adwords meet. Will aggregators’ mobile apps dent adwords revenue? Will mobile provide an impetus for vertical search? Interesting times.

A/B testing small tweaks vs big changes

Found this question on Quora, “Should early stage startups be A B testing landing pages rather than isolated elements like headline copy?”. Here’s my answer…

As you’ve probably already found, given low traffic/data the tests may take a while to deliver any results. As either large or small changes could have significant impact, I wouldn’t advise testing big changes first, nor would I advise testing small ones first.

I’d start by trying to understand what problems there are:

  1. Think of the your goals for the site e.g. purchase, signup for service, signup for newsletter etc.
  2. Create some basic metrics for these goals e.g. number of signups, number of visitors, number of visitors who see signup page
  3. Create some “funnel” metrics from these e.g. % of visitors who see signup page, % of visitors who signup.

Once you’ve got these metrics, pause. Do any of these metrics surprise you? e.g. say the % of visitors who see the signup page is much much lower than you’d expected. It’s a good idea to get a few people to look at the metrics on their own and list what surprised them, then get together and compare.

Once you’ve some agreement on which metrics surprised you, you can prioritise which you’d like to improve. Take this list and brainstorm reasons for worse than expected metric / possible solutions. This is your test plan.

With testing it’s always worth bearing in mind the ROI of things. Take an incremental approach. If you have 5 suggestions for improving a metric but the cost (time) of doing them varies considerably then go for the lowest cost one first.

Forget opinion and hearsay, use data to make decisions

Over the years I’ve worked with quite a number of companies, in quite a lot of different teams and projects. The older, wiser and more experienced I become, the more surprised I am at many many peoples’ preference for opinion or hearsay over data.

Have you ever spent days (or even weeks) making improvements to something that never gets used? Many people have unwittingly.

Web based businesses have so much data. Often a quick check of how often a page is viewed with Google Analytics can be massively enlightening.

Leaner than agile: Better products more quickly and cheaply

This post advocates the use of a ‘lean’ software development process for web based products and services. I outline why those involved in product marketing and software development should consider using a lean approach in their organisation. Where I use the word ‘product’ you can happily substitute ‘service’ if it suits you better.

I caveat the above with “web based products and services” because the web offers fantastic opportunities to measure how the customer values products and features. Value can be measured in various ways, for example measuring usage and goal attainment, running split tests and getting feedback. One or more of these feedback mechanisms are essential to the lean approach.

So what do we mean by a ‘lean’ approach? The term ‘lean’ is borrowed from lean manufacturing processes (such as the Toyota Production System). Wikipedia defines Lean Manufacturing:

Lean manufacturing or lean production, often simply, “Lean,” is a production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination

For software development, my approach to lean is to:

Take the smallest possible step that can test an assumption or idea; move quickly in small increments and learn from each step we make, minimize work in progresses and quickly learn what our users value.

This enables us to quickly determine what adds value whilst wasting less development time on features which do not add value. This usually means minimize work in progress, releasing early and often, measure and test value to the user, then iterate – quickly. Here’s the Wikipedia entry for Lean Software Development for other’s take and perspective.

Values and processes

Before we go into this in depth, I’m not saying one software development process is universally better than another, every business is different and puts a different value on different things. The perfect process for a particular company can’t be found in a text book or blog post, to a certain extent it requires some trial and error to match the company’s values. A company’s values may change over time as a company grows. Use the right tool for the job at hand.

When I use the word ‘values’ I’m not talking about some airy fairy spiritual thing. Some business have customers that value stability and no change (or very slow change) in the product offering. Some services should be straightforward and just work – always, no exceptions, ever. Other businesses will make amazing gains from adapting their products quickly to delight their customers and gain business from their competitors. Startups usually fall into the later category (particularly early stage startups). For them there is immense value in their ability to quickly put new features and products out until they find the right markets and products/features for those markets.

For me one of the primary duties of the product development team is to maximise return on investment or put in less scary language their effectiveness. In the context of product development, I define effectiveness as:

Effectiveness = value added / cost of adding that value

Our aim is to increase the value of our product from our users’ perspective (hopefully also leading to more users) at a reasonable cost. You can think of cost as money or time or a combination of both – the gist is the same.

Adding value requires discovery of what adds value

To increase value, it is essential that we find which features users use and which features either help them or encourage them to complete goals. This is an investigative process, a search that involves lots of trial, measurement, refinement, retrial and so on. For a startup product development is a race, you want your product to be valued by users as quickly as possible. Learning from measurement, testing and feedback is *key* to the lean approach.

Discovering which features users really value or how features ought to work is rarely as simple as *just* asking them. Ask a sociologist, often people don’t do what they think and say they do. Over the years I’ve done a fair bit of split testing and testing by observing users complete test tasks. In both cases we’d frequently be surprised at which features users did and which they did not use. Do gather and encourage verbal/written feedback from your users but use this to inspire investigation, as a catalyst and not as sufficient evidence in itself.

The quick cycle of feedback then iterate again is essential here. With the non-lean approach you might get lucky, hit upon something and add a dramatic amount of value but then again you might not. The same can be said for the lean approach but the cost of discovering this is lower and you can learn and try again quickly. The non-lean approach will probably lead to increased value. However, the non-lean approach probably won’t maximise value or do so quickly and at minimal cost.

Minimize Work-in-progress and avoid queue bloat

Minimizing work-in-progress is important to improve effectiveness by reducing the amount of work which does not add value. For software, by work-in-progress I don’t mean just what the developers are working on at that particular moment, I also include anything they have recently finished which has not yet been released.

What happens when people have to wait for the software development team to get their new features developed and released? Let’s define ‘lag’ as the time from requesting some work to release of that work. Let’s consider how high lag leads to bloat and waste..

Often customers of software teams are like very hungry people waiting for their first meal for a day or two. Once they get served, they’re gonna eat. They’ll eat plenty more than they really need and become bloated. So, using the royal ‘I’… My development slot doesn’t come up another 6 weeks and even then it won’t get release for say another 6 weeks. Any new features or improvements I want won’t get released for another 12 weeks, so I’m going to stuff it with as much as I can. This probably means I’ll ask for a larger slot of development time than I really need, making others wait even longer, increasing the general hunger and leading to more bloat. The queue of work starts to grow with work that is not really necessary.

High lag puts people off trying alternatives. If you don’t try alternatives then you’re not searching for the best solution, not innovating. They’ll polish their new baby in the hope that it’ll succeed in what may be its one shot at success (at least for fairly long time).

Predictability vs Effectiveness

Organisations often put too much emphasis on predictability to the detriment of effectiveness (Value added / Cost of adding that value). They like the certainty that work will begin on features A, B, C, X, Y and Z in 6 weeks time and 6 weeks after that these features will be released. Hit those timescales with those features and nobody can have the finger of blame pointed at them. Except of course the people who are responsible for growing the business fast and before they run out of money, or get beat to it by a competitor.

It’s not just bloat, there’s another problem with 6 week plus release cycles. People become frustrated with the lack of progress and blame the busy development team. In their hunger, people start attempting to jump the queue. Jumping the queue leads to ill feeling and bad decisions. Some may argue that the solution is just to ban queue jumping, to staunchly refuse it. Resisting queue jumping is just treating a symptom rather than addressing a cause – a need. I think queue jumping is inevitable in a business that has customers and competitors and is measuring, testing, listening and responding to the world around them. Priorities change as we learn more about things, it’s a fact of life.

Organising Lean

So treating the cause rather than the symptom. If our approach is to learn from the customer by trialing features and ditching or iteratively improving them based on measurement, tests or feedback, then we’re learning. If we’re learning then our current priorities will change as we learn more. If our priorities change, then it’s important to be adaptive. The key to being adaptive is low lag, that is being responsive and minimising work in progress – a lean approach.

With lean we are searching for how to add value. It’s a search so by definition we don’t know everything up front before we begin. A common question about lean processes is what does a project look like and how is it organised. I like to define projects in terms of their goals rather than their steps. So for example, I might initially define a project as the goal, “Increase the percentage of forum users providing answers to other users’ questions”, then working with the team, we’d think about ways of doing this and how to measure and test them, all the time looking to keep the steps as small as possible.

You will likely have multiple projects going at the same time, depending upon the size of your team and how long it takes to collect sufficient data or other feedback. Switching between each project as things are learnt and next steps are determined. I’ve found the best way to organise this is as a list or work ‘pipeline’ with the following sections:

Backlog => (Pending => In progress => Ready) => Released

Some people call this a Kanban chart or board and use an actual board with post-its. Personally, I prefer a shared document or dedicated agile project tracking system such as Pivotal Tracker

New work requests are added to the backlog, which is kept in priority order. Like an agile process, before the next iteration begins, the decision makers should review the backlog and determine the highest priority items which will progress into the pending section at the start of the next iteration.

The sections enclosed in parentheses – Pending, In Progress and Ready are the work-in-progress. We minimize this by releasing early an often.

Summary

So in summary, the aim of lean software development is to improve effectiveness (return on investment), adding maximum value through measurement and feedback, and reducing or eliminating work on features which do not add value.

  • Always take the smallest possible step necessary to determine if we’re adding value

    Keep asking the question, is this really the minimum step in better understanding if this feature will be valued by users?

  • Learn. Measurement and feedback are king

    Get feedback, that is: usage data, split testing and/or testing with users.

  • Move quickly

    Minimise work in progress and release often.

  • Iterate and investigate. Innovation is a search, don’t be afraid

    The quicker you move the more you can try, the more you try the more likely you are to get a result. Remember you are measuring and getting feedback so you’ll know if you make things worse. As you’re moving quickly, you can quickly put things right quickly too.

  • Be responsive

    If you are not responsive, fear will take hold and bloat will gradually kill effectiveness (value/cost).