The lean-mindset – build simple solutions for the most likely use cases

Recently, a recruiter contacted me on LinkedIn wanting to discuss a CTO role at a well known internet travel company. The recruiter mentioned they wanted someone who had managed/directed 100+ people. I’d only directed a considerably smaller team – about 20 talented people. So it was obvious to me that I probably wasn’t the person they were looking for. Even though this wasn’t a good match, we politely continued the conversation. The recruiter was very surprised when I revealed that my team of about 20 had developed and maintained sites delivering several hundred million dollars of transactions every year and we had 10x growth over just a few years. Reflecting on this, I think what surprises people is that we achieved so much with relatively few people in pretty short space of time.

How were we so productive?! Well we were a startup, or at least had been a “startup” just a few years earlier. The startup mentality was still in most of our DNA at that point. Pragmatism ruled. My aim was to deliver real tangible value ($$) quickly. I recognised that I did not know with very high certainty what features or UI designs actually worked, actually delivered value ($$). I also recognised that neither did the UI experts, the marketing experts or the developers. How? Well we’d conducted quite a few AB tests and a good few user testing sessions, what they taught me is that whilst we experts have a good feel for what might be a problem and what the solution might be, we also get it wrong too often. Getting it wrong too often means at best you get slow progress.

We needed a process (definitely lower case ‘p’) that enabled us to do as little as possible, as quickly as possible, to put our ideas and assumptions to the test – running experiments on our websites to gather real insight and facts about what adds value. This was the motivation for adopting a lean and metric driven process. I’m not sure the terms ‘lean’ and ‘metric driven’ where in use circa 2006, if they were I certainly hadn’t heard of them.

Whilst struggling to cope with a myriad of new work requests, bugs, changes in priority, and react to the results of AB tests, I stumbled upon the idea of modelling tasks like a sales pipeline. As I understand it, sales often kept a list of potential sales ordered by likelihood. These ‘pipelines’ are often sectioned off into states like: ‘closed’, ‘in progress’, ‘contacted’, ‘dead’. Each non-”dead” prospect is assigned to someone. The pipeline is continuously updated. At any time someone can look at this single document and understand the state of sales work and who is working on which lead or prospect. New prospects can be added very quickly and easily. Updating the pipeline to reflect a prospect progressing through these states is trivial. Some ‘prospects’ result in a sale, some die, some lead to other things. Prospects and their status and priority are fluid.

It occurred to me that this approach handled uncertainty and changing priorities really well. Exactly the same attributes our development process needed to handle. I adopted a similar process for organising and communicating my teams tasks and deliverables. My ‘pipeline’ had sections marked: ‘done’, ‘in progress’, ‘upcoming’, ‘todo’.

It was important that we could quickly change priorities. Changing priorities meant I did not want lots of ‘in progress’ tasks. I decided to create a rule of thumb – nobody should have more than 1 task “in progress”. Like many constraints this was more difficult to adhere to than you may at first think and the constraint had knock on effects. The most important consequence is that tasks/deliverables need to be very small and self contained.

Having only very small tasks/deliverables is actually hard to do. It requires a uber-pragmatic mindset. You cannot get caught up building all encompassing solutions that cover every possible use case or outcome perfectly from the beginning. The important words here are “every possible”.

Technical people are trained in system thinking. Good technical people think of every possibility, every possible combination of inputs, states and outputs. This is fabulous but it means it’s very easy to get carried away, solving problems with little consideration of the probability that a problem occurs and whether the cost of a solution is really justified. (I’m a teccy. Don’t hate. This is just something to be aware of..)

I’d already bought heavily into the idea of using metrics to identify what worked and what did not, to “smell” opportunities for improvements. I quickly learnt
that to organise and schedule small tasks/deliverables very often meant building the simplest thing first and for the most likely use case. Making our solutions more sophisticated would be an iterative process. We’d put the first piece of work live and look at our metrics for data points that suggested other use case worth investing in. Until we had data points that suggested other use cases where were worth investing in, I’d “schedule” them as low priority – in our ‘todo’ section.

This is a very subtle but important point. How often have you sat in a meeting discussing solutions to edge cases? It can go on and on and on. Edge cases by their very nature are often difficult (expensive) to solve well. How often have you or your team invested good time (and hence money) solving those edge cases? With the magic of hindsight, would you have bothered if you’d know how little those edge cases mattered, how little monetary difference they made. How little value they added for your users?

Some of you have read that and are thinking something along the lines of, “but I do care, I want ALL our users to have a great experience” and “Nonsense, what would Steve Jobs make of that” etc. I don’t necessarily disagree with those sentiments but we’re talking about priorities. Priorities are not yes or no, they’re just about when (and only sometimes if). We’re also talking about product development on the web, where there is massive opportunity to collect data, learn and often quantify what’s really important.

Lean is a mind-set, perhaps more than it is a process. Do the simplest possible thing to address those edge cases and learn how often they happen. If they happen very infrequently then they’re not that important. Alternatively, if the data says they happen a lot, then that is the point you make the decision to invest in finding and delivering solutions.

Related Posts:

Concurrency & Long running requests: Sinatra, EventMachine, fibers and Node

The problem – long running requests with blocking IO

Recently, I’ve been working on an alternative view onto Google Analytics data. This presents some challenges:

  1. Calls to the Google Analytics API vary but some can take over 10s.
  2. At this point I only have a single VPS instance which runs various ruby and wordpress instances and has limited memory and db connections

Problem 1, long running API requests (yes 10s is long) normally means that a process is tied up (blocking) waiting for a reply for 10s.

Problem 2, the single VPS instance has limited resources. Memory wise, I cannot have many processes running each using up a chunk of mememory.

1 + 2 = crappy performance, when there are multiple concurrent users the limited number of processes available to service the users requests may be tied up waiting for replies from Google.

The solution – get more concurrency from a process

Solution: get more concurrency from a limited number of processes. I can think of two solutions in the Ruby world:

A) Use JRuby which has native rather than cooperative threads
B) Use EventMachine and non-blocking async http requests.

Like node.js, eventmachine implements the reactor pattern – essentially requests cooperate by giving back control to the reactor when they themselves are blocking on something e.g. an http API call.

Solution A, use JRuby is reasonable but has some drawbacks. In theory a single JRuby process can concurrently handle 100s of concurrent requests, processing each in it’s own native thread. However, in practice we still need to be careful about non-reenterant code, particularly in gems and extensions we depend upon.

Solution B, initially appears more complex but is perhaps more predicatable. We’re explicitly recognising the blocking API calls issue and making it a 1st class issue for our system. Whilst I don’t like to make things more complex than they need be, often it’s better to recognise a fundemental issue and address it explictly.

How evented/async approach works

So how does B work:

  1. server receives a request from browser
  2. server “queues” the request in EventMachine’s reactor
  3. EventMachine’s reactor gets back control from whatever was running and pulls the next event from its reactor queue and executes it
  4. our app gets control from the reactor and starts processing the request
  5. our app makes a http request to Google Analytics API using an async library (em-http) and provides a callback to be executed when a reply is received; control returns to EventMachine’s reactor
  6. the EM reactor pulls the next event from its queue and executes it
  7. our earlier http API call (from 5) returns and the callback get’s queued on the EM reactor
  8. the currently executing event finishes, returning control to EM
  9. EM picks up the callback (from 7) which processes the results of the API query, builds a response and sends it to the browser. The HTTP request is finished
  10. So where’s the pain? Well async code with callbacks can quickly get messy, especially if there are callbacks nested within callbacks, nested within callbacks… and then an exceptional condition occurs. It is possible to structure code as sets of actions, where the callback code just links actions together, handling the flow of control between them. However, it’d be nicer if we could just write code that looks like normal sync server code. Ruby (1.9) Fibers and Sinatra-synchrony to the rescue…

    Fibers – explicit cooperation

    Ruby 1.9 has Fibers. Threads enable implicit concurrency, with some layer deciding when to switch control from one thread to another. Fibers enable explicit cooperative concurrency. A piece of code is wrapped in a fiber and executed. The fiber can yield, freezing its current state and returning control to its caller, at some later point the caller can resume the fiber, with the fiber executing from where it yielded control, with it’s pre-yield state intact.

    Here’s an example:

    Calling main( fiber ) gives the following output:


    Hello from main: 321
    Hello from the fiber: 123
    and we're back in main: 321
    And we're back in the fiber: 123
    back in main and finishing: 321
    

    Fibers, making async code look like sync

    I’ll just give a very brief overview. For more detail here’s a great article explaining how Fibers can be used to make async code look like sync code.

    Now we can just write async code that looks like sync code – no callbacks:

    So this is great, async code without the nested callback headache! I’m a fan of the lightweight ruby web framework Sinatra. This helpful and clever person has put Sinatra and EventMachine together (with a few other useful pieces) in he form of Sinatra::Synchrony.

    Node Fibers

    Ruby not your bag? Prefer javascript/coffeescript and node.js? I’m finding myself writing more and more javascript/coffeescript code in the browser – coffeescript + backbone.js + jasmine (for testing) is pretty good. I can certainly see the attraction of node.js, using the same language on the server as the client, particularly if it comes with the high-concurrency goodness of async events. Well there may be some good news in the form of Node-fibers – in particular take a look at the Fiber/future API.

Coffeescript and Jasmine for testing

Recently I’ve been writing a fair bit of browser-side Coffeescript + Backbone.js. Nice, but as the code base grows and gets more complex, the rubyist in me feels increasingly uncomfortable and less productive with the lack of TDD/BDD. What to do… Jasmine BDD for javascript seems to be a good answer.

Jasmine follows the same spec/BDD style as RSpec for Ruby and Kiwi for Objective-C (Kiwi is very good BTW). That is, set up your context using ‘describe’ + ‘before’, then make assertions on the post context state using ‘it/should’ blocks. Jasmine runs your test suite in the browser and seems to be blindingly fast.

For me, Jasmine tests also highlight one of the reasons I like Coffeescript more than plain old JS – less lines of code. BDD specs tend to have a lot of nesting as you build up contexts, so Jasmine written in JS tend to have a lot of nested functions. As these test functions tend to be pretty straightforward assertions, it’s seems a reasonably fair to compare coffeescript vs the generated js. Here’s one of my tests, notice that the coffeescript version is just 49 lines and pretty easy to read; the JS version is 78 lines and to my eye it’s not terrible but it is harder to read.

Coffeescript

Javascript

Developers, Vagrants, Puppets and Chefs

A friend recently pointed me at Vagrant. At first I wasn’t sure but after a play, I’m starting to thing this is a great idea.

Vagrant helps you create VMs for use as development environments. So for example, on my Mac I might run a Linux distro in a VM and use it to run the website + db that I’m developing. What’s more I can share that VM config with other developers working on the same project.

Vagrant works with Oracle’s free (and open source) Virtual Box virtualization software. If you’re a Mac user, you might be familiar with running Windows in VMWare Fusion or Parallels. This is the same sort of thing – though if you’re like me you’re more likely to run a Linux distro with Apache/Nginx, MySQL/Postgres etc..

Vagrant use VirtualBox to create and run virtual machines from your chosen distro and then provisions them with your development environment, webserver, app server, db etc using either Puppet or Chef. All this is done in a portable way and can be shared so that each of the developers on your team develop against exactly the same configuration.

Why is this a good idea? Here’s my top reasons..

  • consistent environment across the team – WoMM (“works on my machine”) is no longer an issue
  • bring in new developers more quickly with less frustration and teething problems
  • “hosed” environments can be dumped and new one created quickly and easily. Now you can ‘fork’ your environment not just your code.
  • dev environment can more closely match the server – even if you work on a MacBook. You could potentially use the same chef recipes or puppert scripts too.
  • It’s pretty easy to get up and running – why not run through the Getting Started with Vagrant guide and see what you think?

Here’s the Vagrant guys explanation of why this is a great idea.

Lean development is not the same as Agile

Agile project management techniques recognise that software project management is mostly about communication between the ‘client’ and ‘development team’. Lean techniques instead focus on determining as quickly as possible what adds value and what does not. My experience is that lean techniques recognise that value is added most quickly by increasing ‘communication’ with the end user. This communication is rarely explicit. Lean techniques emphasise implicit communication through observed usage, metrics and comparative split tests – real data quickly trumps opinion. Lean accelerates learning, minimising work in progress and the cost of learning what really adds value.

Agile improves on Waterfall

Agile project management techniques recognise that software project management is mostly about communication. Older techniques such as the Waterfall process spent too much time attempting to capture and formalize what the client actually wanted. Often vast amounts of time and money would be spent writing and agreeing abstract specifications which invariably turned out not to accurately reflect what the client envisaged. The creation of these abstract specifications did not identify and predict the technical difficulties the project would face and the magnitude of problems. Many of you will have seen the Tree Swing Cartoon about requirements.

Agile techniques address these issues by emphasising communication between the development team and their client throughout the project; accepting that it is very hard to determine exactly what a client wants without delivering designs and working software. Clients will change their minds as they see how in reality the things they envisaged actually work. Developers will discover that some features they thought would be simple to implement are not so simple. Both party’s understanding of the facts on the ground improve vastly as they make progress. They both accept that they need a dialogue. Both parties accept that it is sensible to make changes to the project “plan” as it progresses to completion and delivery.

Agile addresses the communication gap between developer and client. The dialogue leads to sensible decisions, better use of resources (productivity) and greater predictability. Agile techniques means software development increasingly does a better job of meeting the expectations of its clients. However, often there is another very important person who’s been left out of the conversation, the end-user (e.g. the customer on an e-commerce site). Agile is still relying on the client’s ability to determine what the end-user wants and what works.

The development team can help with this process (that’s product development team, including product management, UX and developers). UX people often use usability testing techniques to gather opinion and identify problems with designs. These are really useful and valuable, identifying some problems, encouraging best practices and avoiding opinion deadlock. Unfortunately they are highly artificial, the few users in their tests are often not actual customers, with an actual need, using their own time and money.

If used correctly usability testing techniques can smooth the edges of a design, making it flow better and making it easier to use. However, they are based on small sample sizes, often gathered from the same geographical source and demographic. They often miss the nuances of other demographics and of how people actually behave when it’s their own money they’re spending. For example, I’ve seen data that showed 6% of UK customers attempting to use just their house number and postcode to specify their address, on many sites that is insufficient. What are the chances that this 6%’s actual behaviour would be picked up in a artificial usability test – unfortunately it is not high.

There is a new kid on the block which aims to address this gap and bring the 3rd and most important party in on the conversation – the end user, in HD and in all their magnificent glory, warts and all. The new kid is Lean Development.

Lean

In October (2011) Eric Ries’ best selling book, “The Lean Startup: How Constant Innovation Creates Radically Successful Businesses” (which I thoroughly recommend by the way) was released to critical acclaim and quickly became a New York Times Bestseller. Eric’s book focuses more on startups and product development rather than specifically software development per-say. The main theme it shares with Lean Software Development is this gap between what the experts in a business think about a product or service and the reality of what the end-user/customer values and how they use it.

The lean approach borrows its name from Lean Manufacturing techniques such as the Toyota Production System. Lean manufacturing works on the basis that parts are pulled through the manufacturing process as they are needed, work in progress is kept to a minimum. This makes the manufacturing process highly reactive to customer demand and able to cope with day to day manufacturing problems whilst maximising throughput. If orders for a product increase, more of its parts are pulled through the manufacturing system. If there is a problem with manufacturing of a part upstream, then it temporarily stops pulling the parts it depends upon up through the manufacturing process.

In software development we’re not talking about manufacturing but product development, the process of determining what features a product or service should have, how it should work and making the feature available. Our goal is to build things that users value, that help them achieve their goals. For example, in ecommerce that’s finding a site, finding the product they want and successfully purchasing the product.

Lean development recognises that experts are making calculated guesses about what problems a user has, what they will value and how they use a feature. There is nothing wrong with this. We need to start somewhere and an expert’s guess is a very good place to start and with no information to the contrary it’s also a good next move. However, it is a bet, we are betting that the expert is correct. Lean development aims to reduce our exposure to this bet by reducing work in progress to a bare minimum.

Using a lean process, each new release of the software contains the bare minimum additions/changes needed to test the experts assumptions. We determine what the user thinks through observing how they behave, collecting data through analytics and analysing our own databases and software logs. This Minimum Viable Product (MVP) or feature (MVF) is the least work we can do, the least money we can spend to test the expert’s hypothesis, to learn more and plan our next step.

This leanness, this minimization of work in progress is key. The emphasis is build something quickly, try it, look at data and feedback, improve – quickly, repeat. Lean is real and fact based, it implicitly involves the end-user, it makes them a key part of the product development conversation. The user drives the development process, implicitly prioritizing and “pulling” the new features or tweaks that they want through the development process.

Agile processes have plenty of regular conversation with the client, including regular demonstration of progress. When using an agile process, releases of new software with new features to the end-user are months apart. In a lean process, the conversation involves the end-user, so time between releases is measured in days not months.

In a Lean process, real data quickly trumps opinion as assumptions are quickly put to the test, features evolved – refined, modified or even removed. Ideas for what to try next are picked up from scents in the data. These scents are not always purely quantitive, for example an increase in the number of times a question is sent to the support email address may provide a clue to a problem or opportunity.

Lean projects ARE typically explicit goals where success can be quantified and measured, for example the number of sales per day increases. Agile projects often have explicit goals but ones that are not explicitly quantified, instead Agile projects tend to be specific about how much time is available. Lean projects tend to focus on achieving quantifiable value for the end-user. Lean development recognises that delivering real tangible business value is more likely if you iterate quickly and respond to what the end-user is actually doing.

Lean development is not agile development – lean extends the product development conversation to the end-user and minimises the businesses exposure to expert bets.

Of course, the real world isn’t quite a black as white as I’ve just painted it. In reality the differences between what Fred calls Agile and what Wilma calls Agile can be huge. Some people’s “agile” processes will look more like the Waterfall process whilst others will share the attributes of a lean process to a greater or lesser extent. So please forgive the blunt contrast I’ve painted to illustrate my points. I hope you’ve enjoyed this post.

Innovation, motivation & goal driven management

I believe that organisations that want to encourage innovation and improve employee morale should consider whether they can be more goal driven, focusing more on goals and less on tasks. All organisations are ultimately goal driven but often goals are turned into tasks too early, this immediately reduces staffs’ room to manoeuvre – to investigate, test, react, adapt and innovate to reach the goal.

Do you know that a set of features and associated tasks will delivery your goal? Do you know these features and tasks will deliver the best result using the time and resources available? Progress towards a goal is more important than completing a set of tasks. Allocate resources to the achievement of a goal and use those resources to explore problems and solutions, collect and use data to determine next steps.

Managers often prefer tasks because they predictably show a concrete output for a given input. Completion of tasks A and B delivers features X and Y. It can be demonstrate that task A has been completed and feature X delivered. The achievement of a goal, a measurable outcome, is often much harder to predict. Worse, metrics highlight sunk costs, where completion of a set of tasks does not deliver the desired goal.

This human tendency towards predictability over goal achievement is heightened by processes and structures that bet the house on the latest big idea. In a task orientated environment, a determined focus on goal achievement takes guts.

Management skill is stating the goal, posing the questions, defining and focusing on outcomes and the metrics around them. We need a process (lower case ‘p’) that helps us plan, agree, track, adapt and communicate progress towards a goal. Here’s my thoughts on such a process..

Start with the goal/s of a project and the timescale and resources available to achieve the goal/s. How will achievement of the goals be measured? These metrics should be outcomes not completion of tasks or features. For example, is the goal to increase conversion of a checkout process; to increase newsletter signups; to reduce customer service calls; to increase visibility of a product? All of these examples are goals and their achievement can be measured.

Brainstorm ideas for achieving the goals. Are there any prerequisites / stepping stones? Organise these as a tree you want to explore rather than a list of tasks that need completing. The goal is the root of the tree. The nodes are sub-goals, ideas you want to test and measure how successfully a feature may deliver a goal or contribute towards it. The tree gives you a mechanism to understand, track and communicate how the team/s are attempting to explore the problems, try solutions and ultimately how these contribute towards the goal.

Prioritise the nodes below the root, based on which you think are most likely to make progress towards your goal. Start work on the highest priority node. If you’ve multiple teams, teams can start on different siblings. Review results (goal metrics and sub-metrics), which sub-branches should you try next? Are there new sub-branches you want to add? Are they the next best thing to try? Based on your results, do you continue down those sub-branches or move to a sibling? The tree is dynamic, explore it, grow it, prune it based on progress towards your goal.

Goals and metrics for a project provides focus. They encourage conversation about whether the proposed features really achieve the stated aims. This encourages investigation, looking at data, proposing and conducting tests. Furthermore, it encourages conversations about the cost vs value of pieces of work and encourages everyone to bring to the table alternatives which may be more effective and/or lower cost.

Why not leave turning goals into tasks to the “doers”? Tasks (and expectations of their delivery) are a necessary part of successful management. It is important to know “what is going to be done” and “when it will be done”. However, I’d argue that tasks should be a proposed by the “doers” in response to being set a goal by the “leader”. The leader sets goals, puts them into context and explains how their achievement will be measured. The leader facilitates the process of turning goals into tasks and expectations about their delivery. The resulting tasks can be “hung off” each goal in the tree, abstracted away out of sight until they are needed, leaving the focus on goals.

The ultimate aggregator – Google, travel & mobile

In my post, Adwords, vertical search and aggregator ‘middle men’ I mentioned a simple search enhancement that could enable hoteliers to better compete with the aggregators (e.g. Expedia, Travelocity, Booking.com). When a user searches using the keyword “hotel” or “hotels” some simple filters such as room type or budget would be shown. Advertisers would be able to bid against keyword/s and filter value/s. This got me thinking about what else Google (or Microsoft) could do for the travel vertical and how the growth in mobile might precipitate this.

Google recently purchased ITA Software a provider of Flight information. This potentially enables Google to show enhance search results with lists of flights and prices for various carriers and routes. Google could piggy back Adwords adverts on the side of this; or they go further and offer advertisers the ability to advertise against specific carriers and routes. Could they do something similar for accommodation?

Google becomes an accommodation aggregator?

Google could themselves become an accommodation aggregator. Google Base or Google Places could be extended for hotels to take rates, rooms and availability information. The results of a hotel related search would be a list of hotels rather than the normal list of hotel related websites. The usual filters you see on an aggregator site would be available – including travel dates. Near each hotel result there’d be paid (Cost Per Click) links to (multiple) websites where a room in the hotel can be booked.

For users this seems like an improvement, cutting out a step and helping them compare prices from aggregators and perhaps the direct from the hotel itself. This is the most obvious move but unfortunately for the user this is perhaps also the the most unlikely.

Technically whilst this should be pretty straightforward, certainly for a company with Google’s resources, in reality it involves integrating lots of 3rd party systems and information. Room availability and pricing is ideally real-time information. Will 3rd party systems be fast enough for super speedy Google? Will they be reliable enough. Google sets the bar pretty high. What if the information becomes stale? The user finds a supplier of a particular hotel and room for £120, clicks through to discover it’s now £150 or worse no longer available. Still you could still make the case that this is better, certainly no worse for the user than the status quo.

The biggest problem is what would the aggregators make of this? Aggregators are very big spenders on Google Adwords and unlikely to voluntarily play ball on this. Getting the aggregators to put the effort into such technical integration is unlikely given they would probably stand to lose the most, at least in the short term

Currently, the name of the game is getting a big enough return per click to afford to appear in the key first few result that drive the vast majority of referral volume. The aggregators achieve this by aggregating lots of supply => lots of choice (see my previous post).

Hoteliers benefit from vertical search

Could Google circumvent the aggregators? Hoteliers could benefit hugely from such a vertical search. Their relevancy to any paid click would be tremendous, they’d be very likely to convert the click into a booking and achieve a good return on their click. Perhaps this is where Google Places is headed, adding real-time vertical specific information: rooms, rates and availability. Google could even become the preferred tool for independent hoteliers to manage their rates, availability and bookings. Perhaps Google wouldn’t want to be involved in the actual financial transaction, there are potentially numerous legal, regulatory and customer service problems (Google Checkout prohibits use for travel purchases). However, this wouldn’t stop Google offering a API through which a hoteliers website could instantly store booking information in Google’s systems or simply update their room inventory.

Holiday (vacation) lets

What about holiday (vacation) lets, villas and cottages? Google might see significant take up in this sector as it is more fragmented than hotels and technically simpler. Holiday properties are typically owned by private individuals rather than the “holiday companies” who market them. The holiday companies act as aggregators, offering advertising, calendar management and in some cases full booking service. Google could potentially offer some of these services: a simple advertising solution, calendar management and itinerary update. This is a ripe market, with plenty of opportunity for a technical innovator to come in and attack the cost base and effectiveness of the holiday companies’ traditional marketing and customer services models, both property owners and consumers would ultimately welcome this.

The mobile search problem and aggregator apps

Searching for a hotel via Google Search is painful on mobile. Often the sites listed are not designed for mobile, though no doubt in time most sites will have mobile versions. Personally I find clicking back and forth between Google Search and the sites listed far more clunky on mobile. I wonder if mobile users will adopt a preferred aggregator and stick with the aggregator’s mobile app. In time this could represent a very significant drop in Google’s ad revenue.

Google could avoid such a drift to aggregators’ mobile apps by more aggressively pursuing vertical searches on Google Mobile Search. Vertical search could reduce the steps necessary to find accommodation, pulling more of the steps into Google Search and enabling the user to enter a hotelier or an aggregator’s site at a later step in the selection and booking process.

In exchange for offering the user more relevant content, Google would be able to better understand what the user is looking for. Knowing more about the user’s goal and preferences opens up other opportunities. Having additional information about the user’s goals also enables Google to intelligently cross-link to relevant flights, cars, restaurants etc... More relevance leads to more clicks.

It’ll be interesting to watch and see, if and where Google Search, Places, Base and Adwords meet. Will aggregators’ mobile apps dent adwords revenue? Will mobile provide an impetus for vertical search? Interesting times.

A/B testing small tweaks vs big changes

Found this question on Quora, “Should early stage startups be A B testing landing pages rather than isolated elements like headline copy?”. Here’s my answer…

As you’ve probably already found, given low traffic/data the tests may take a while to deliver any results. As either large or small changes could have significant impact, I wouldn’t advise testing big changes first, nor would I advise testing small ones first.

I’d start by trying to understand what problems there are:

  1. Think of the your goals for the site e.g. purchase, signup for service, signup for newsletter etc.
  2. Create some basic metrics for these goals e.g. number of signups, number of visitors, number of visitors who see signup page
  3. Create some “funnel” metrics from these e.g. % of visitors who see signup page, % of visitors who signup.

Once you’ve got these metrics, pause. Do any of these metrics surprise you? e.g. say the % of visitors who see the signup page is much much lower than you’d expected. It’s a good idea to get a few people to look at the metrics on their own and list what surprised them, then get together and compare.

Once you’ve some agreement on which metrics surprised you, you can prioritise which you’d like to improve. Take this list and brainstorm reasons for worse than expected metric / possible solutions. This is your test plan.

With testing it’s always worth bearing in mind the ROI of things. Take an incremental approach. If you have 5 suggestions for improving a metric but the cost (time) of doing them varies considerably then go for the lowest cost one first.

Forget opinion and hearsay, use data to make decisions

Over the years I’ve worked with quite a number of companies, in quite a lot of different teams and projects. The older, wiser and more experienced I become, the more surprised I am at many many peoples’ preference for opinion or hearsay over data.

Have you ever spent days (or even weeks) making improvements to something that never gets used? Many people have unwittingly.

Web based businesses have so much data. Often a quick check of how often a page is viewed with Google Analytics can be massively enlightening.

Adwords, vertical search & aggregator middle men

Many thought the internet would “cut out the middle man” by making it trivial for businesses and customers to find one another. Who’d need a travel agent when you could find a hotel’s website and book direct. The internet equivalent of the high street travel agent are the aggregators e.g. Expedia, Travelocity, Priceline and Booking.com. Unfortunately for the hoteliers and airlines that sell through these aggregators it hasn’t (quite) panned out like that…

Return per click and gross margin per sale

The hoteliers ought to achieve greater revenue per sale by advertising on Google themselves and selling direct to the customer. Remember the aggregators are middle men. The hoteliers and airlines pay the aggregator a percentage commission per sale. Hoteliers achieve greater gross margin per direct sale than the aggregators, they ought to be able to out-bid aggregators in Adwords – right! Wrong. The aggregators achieve a higher return per click than their suppliers despite lower gross margin per sale.

Travel aggregators beat their suppliers to dominate PPC marketing because they typically achieve a higher return per click than their suppliers. A higher return per click means aggregators can afford to spend more per click; their ads appear higher, achieve greater click-through rates and therefore achieve higher volumes. It’s pretty simple, aggregator’s achieve a higher return per click because they have greater range and availability of product. Let’s put that into context and see how it relates to PPC.

How is it possible that the aggregator achieves a higher return per click despite their smaller cut of a sale? Here’s the thing.. good old fashioned choice, that’s how. The customer is often considerably more likely to find what they are looking for on the aggregator’s site. This higher probability of a sale (conversion) is often many times higher than the suppliers. The aggregators’ greater choice and stock more than compensates for a smaller revenue per sale. Conversion dominates over revenue per sale.

The beauty of Google’s Adwords is that both aggregator site and an individual hotelier can reach a relevant audience. For hotels, the primary relevant parameter is location, so both aggregator and hotelier can advertise against searches for “hotels in brighton”, “brighton hotels” etc with relevant ad text and a link through to the relevant page on their websites.

Customer requirements and preferences

Here is a fictitious example. Let’s look at what 1000 people who clicked on our “brighton hotels” adverts are looking for. We look at which price filters these 1000 people use, this tells us their budget per night:

What customers are looking for

What the hotelier has

What the aggregators offer

If we overlay the hotelier’s distribution of available room prices over the distribution of prices customers are looking for, we see that there is a significant mismatch. The 30% of customers looking for a property under £70 are unlikely to book a room; likewise for the 10% looking for premium rooms over £150. That’s 40% of demand the hotelier is very unlikely to fill.

The Aggregator’s distribution of available room prices is a much better fit than the hoteliers. This example tells us nothing about whether the hotelier has sufficient quantity of £70 to £100 rooms to fulfil demand. In the case of the Aggregator it is also more likely they have sufficient rooms across their range of hotels to meet demand. Applying the same technique across over customer requirements and preferences magnifies the Aggregator’s advantage, they are much more likely to have a suitable room available in a hotel that meets the users needs and preferences.

To simplify and illustrate the point, let’s assume that both the hotelier’s website and the Aggregator’s site have identical conversion rates *when a suitable product is available* i.e. the speed, design, usability and error free operation are comparable. Given this demand price distribution above, the Aggregator is significantly more likely to have a *suitable product available*. The hotelier does not have rooms available and priced to meet 40% of demand. In this example the Aggregator may be twice as likely to have a suitably priced room available. If we include other customer requirements and preferences, room type, hotel facilities, hotel location etc, the Aggregator’s advantage over a single hotel compounds, the aggregator is likely to be at least 3 times as likely to make a sale as the hotelier.

Let’s assume the average price of a room booked is £100. For the sake of argument, let’s assume the nominal gross margin for the hotelier is 40%, that’s £40. On average our Aggregator gets 20% commission and after costs their gross margin for comparison is 15%, that’s £15. The return per click is:

Hotelier’s return = £100 * 40% * conversion_hotelier = £40 * conversion_hotelier
Aggregator’s return = £100 * 15% * (3 * conversion_hotelier) = £45 * conversion_hotelier

The 3x conversion advantage due to range of product better meeting customer demand is probably an understatement but illustrates the importance of having the products customers are looking for and its impact on conversion and hence on the CPC an advertiser can afford.

Adwords + vertical search = increased relevance

Adwords does a good job showing relevant adverts based on keywords. However, customers searching using Google’s normal text box search typically only specify a couple of their preferences. For example, “brighton hotels” says the customer wants #1 hotels and #2 hotels in Brighton. What if Google, for searches mentioning ‘hotel’, showed price and room type filters and extended Adwords to support CPC bidding against price ranges and room types as well as keywords?

Hoteliers could afford to spend more per click where they are confident the customer is looking for a type of room they can provide at the right price, their conversion and return per click would improve, enabling them to spend more per click for this smaller volume of more relevant searches.

What happens to aggregators in such a scenario? Aggregators would still be able to confidently advertise against these searches, their conversion would remain the same. However, hoteliers would be better able to compete on CPC, achieve higher positions and volumes.

Increasing advertiser diversity whilst improving customer relevancy would surely be a good thing for the both the customer and Google. Obviously such a vertical search could be taken further, but a model like this may be palatable for the aggregators – Google’s existing big Adwords spenders.

Related: The ultimate aggregator – Google, Travel, Vertical Search and Mobile