Ecommerce Resonance – Optimise Conversion by aligning Demand, UX and Supply

From Wikipedia, “.. resonance is the tendency of a system to oscillate with greater amplitude at some frequencies than at others.”

Have you ever heard someone say, “our website’s conversion is too low, we’re not ready to start spending money on PPC/Display advertising/etc..”. I’ve heard variations on this theme at a few companies. Often such statements receive firm unequivocal nods from around the table – conversion is low therefore the website isn’t good, it isn’t fit for purpose.

At a high level such statements seem self-evident. However, they miss a fundamental truth about conversion optimisation: Conversion Optimisation is not all about website UX. As I mentioned in my previous post on ecommerce:

Conversion is a measure of how well our site and supply meets the needs of the users that our marketing attracts.

I’ve highlighted the 3 components of conversion: site, supply and marketing. These 3 components need to be aligned so that they resonate – reinforcing each other not reducing each other. Think about these scenarios:

  1. If a user is looking for a red sports hatch, you’re probably not going to be able to sell them a caravan (demand vs supply)
  2. I want to book an apartment in Berlin on the 25th of next month but the accommodation website I just found through Google only has 4* hotels available (demand vs supply)
  3. I’m looking at a list of insurance products but it’s really hard to find which ones offer annual cover (site but also potentially supply and demand too)

Does your site sell what the user is looking for? Can you improve supply, aligning it so that it does? Can you improve demand to attract users who want what you are selling – so that it’s better aligned with supply?

Can you improve your site so that users tell you what they want? Consider changing your UX so that users can self segment, identifying to you what they want to buy. If we measure what users are looking for and share this around our organisations, we can create positive feedback loops:

  • supply should look to fulfil demand
  • demand should attract the right demand for the available supply
  • UX should help the user tell your organisation what they want

Here’s another slightly different scenario, this time it’s about how demand mix can affect conversion, specifically what I call “early vs ready”:

I am reading about things to do in Vancouver, I’m curious about hotels but I’m not ready to book a hotel today, it’s too early and I’m just planning (demand mix – early vs ready)

Your site may well have what the user is looking for but maybe it’s too early in the purchase cycle for the user to buy – they are “early” not “ready”. A marketing channel that attracts a lot of “early” but not “ready” will likely convert poorly. That’s fine so long as we understand that this poor conversion is a function of the marketing channel not the website.

The length of the purchase cycle, the transition from “early” to “ready” varies greatly depending on the product/service we’re selling. Having a marketing channel that is largely made up of users who are “early” but not “ready” is an opportunity but we’ll need to work hard on strategies to get the user to come back when they are ready to purchase.

Conversion is not just about site design.

Read my previous post: Ecommerce: Get the “right” products to the top

Ecommerce: Get the “right” products to the top

Whether it’s the results of a search on Google, Amazon, an ecommerce shop or a hotels website, users predominantly click on the top results in a list. Take a look at all these charts of clicks by position in Google Images. This is common sense – we expect the best results for us to be towards the top of the list. It’s imperative that we design our websites so that we get the “right” products to the top.

Which products are the “right” products? Well “right” is a personal and occasional thing. Here are some approaches that I’ve successfully used:

  1. Encourage use of filters. You are measuring how your design affects this – right? What % of users who view a list also use a filter?
  2. Encourage the user to self segment by product category. For example: hotels, apartments and guesthouses are pretty different beasts.
  3. Default order based on predicted revenue per impression – or put another way, users vote for products by buying them.
  4. Put unavailable or out-of-stock products at the bottom. For travel products, encouraging the use of travel dates can have a big impact

Encouraging users to self segment by product category is obviously a form of filtering. I highlight it separately because it’s potentially the most important filter and it can be used to give you a metric of demand by product category. So for example, on an accommodation website, by destination, you can see what percentage of your demand (users) are looking for apartments (vs hotels vs guesthouses etc). This metric can be used to improve supply. It may also be used to adjust demand – perhaps altering your advertising to get an appropriate level of demand to match your supply. To maximise conversion, we want demand, UX and supply aligned so that they resonate..

In many companies the default ordering of products is often a highly political topic. Sometimes it rivals homepage design for the political ire that it generates. A common concern is that this is simply putting the most expensive products at the top. That is incorrect and would indicate a poorly designed algorithm which does not maximise conversion or revenue. I believe this often stems from a fundamental misunderstanding of what conversion optimisation is about. Conversion Optimisation is not about increasing revenue by selling the more expensive products, it’s about maximising revenue by selling more of what users want. Conversion is a measure of how well our site and supply serves the needs of the users that our marketing attracts. Conversion is a measure of user satisfaction.

The lean-mindset – build simple solutions for the most likely use cases

Recently, a recruiter contacted me on LinkedIn wanting to discuss a CTO role at a well known internet travel company. The recruiter mentioned they wanted someone who had managed/directed 100+ people. I’d only directed a considerably smaller team – about 20 talented people. So it was obvious to me that I probably wasn’t the person they were looking for. Even though this wasn’t a good match, we politely continued the conversation. The recruiter was very surprised when I revealed that my team of about 20 had developed and maintained sites delivering several hundred million dollars of transactions every year and we had 10x growth over just a few years. Reflecting on this, I think what surprises people is that we achieved so much with relatively few people in pretty short space of time.

How were we so productive?! Well we were a startup, or at least had been a “startup” just a few years earlier. The startup mentality was still in most of our DNA at that point. Pragmatism ruled. My aim was to deliver real tangible value ($$) quickly. I recognised that I did not know with very high certainty what features or UI designs actually worked, actually delivered value ($$). I also recognised that neither did the UI experts, the marketing experts or the developers. How? Well we’d conducted quite a few AB tests and a good few user testing sessions, what they taught me is that whilst we experts have a good feel for what might be a problem and what the solution might be, we also get it wrong too often. Getting it wrong too often means at best you get slow progress.

We needed a process (definitely lower case ‘p’) that enabled us to do as little as possible, as quickly as possible, to put our ideas and assumptions to the test – running experiments on our websites to gather real insight and facts about what adds value. This was the motivation for adopting a lean and metric driven process. I’m not sure the terms ‘lean’ and ‘metric driven’ where in use circa 2006, if they were I certainly hadn’t heard of them.

Whilst struggling to cope with a myriad of new work requests, bugs, changes in priority, and react to the results of AB tests, I stumbled upon the idea of modelling tasks like a sales pipeline. As I understand it, sales often kept a list of potential sales ordered by likelihood. These ‘pipelines’ are often sectioned off into states like: ‘closed’, ‘in progress’, ‘contacted’, ‘dead’. Each non-”dead” prospect is assigned to someone. The pipeline is continuously updated. At any time someone can look at this single document and understand the state of sales work and who is working on which lead or prospect. New prospects can be added very quickly and easily. Updating the pipeline to reflect a prospect progressing through these states is trivial. Some ‘prospects’ result in a sale, some die, some lead to other things. Prospects and their status and priority are fluid.

It occurred to me that this approach handled uncertainty and changing priorities really well. Exactly the same attributes our development process needed to handle. I adopted a similar process for organising and communicating my teams tasks and deliverables. My ‘pipeline’ had sections marked: ‘done’, ‘in progress’, ‘upcoming’, ‘todo’.

It was important that we could quickly change priorities. Changing priorities meant I did not want lots of ‘in progress’ tasks. I decided to create a rule of thumb – nobody should have more than 1 task “in progress”. Like many constraints this was more difficult to adhere to than you may at first think and the constraint had knock on effects. The most important consequence is that tasks/deliverables need to be very small and self contained.

Having only very small tasks/deliverables is actually hard to do. It requires a uber-pragmatic mindset. You cannot get caught up building all encompassing solutions that cover every possible use case or outcome perfectly from the beginning. The important words here are “every possible”.

Technical people are trained in system thinking. Good technical people think of every possibility, every possible combination of inputs, states and outputs. This is fabulous but it means it’s very easy to get carried away, solving problems with little consideration of the probability that a problem occurs and whether the cost of a solution is really justified. (I’m a teccy. Don’t hate. This is just something to be aware of..)

I’d already bought heavily into the idea of using metrics to identify what worked and what did not, to “smell” opportunities for improvements. I quickly learnt
that to organise and schedule small tasks/deliverables very often meant building the simplest thing first and for the most likely use case. Making our solutions more sophisticated would be an iterative process. We’d put the first piece of work live and look at our metrics for data points that suggested other use case worth investing in. Until we had data points that suggested other use cases where were worth investing in, I’d “schedule” them as low priority – in our ‘todo’ section.

This is a very subtle but important point. How often have you sat in a meeting discussing solutions to edge cases? It can go on and on and on. Edge cases by their very nature are often difficult (expensive) to solve well. How often have you or your team invested good time (and hence money) solving those edge cases? With the magic of hindsight, would you have bothered if you’d know how little those edge cases mattered, how little monetary difference they made. How little value they added for your users?

Some of you have read that and are thinking something along the lines of, “but I do care, I want ALL our users to have a great experience” and “Nonsense, what would Steve Jobs make of that” etc. I don’t necessarily disagree with those sentiments but we’re talking about priorities. Priorities are not yes or no, they’re just about when (and only sometimes if). We’re also talking about product development on the web, where there is massive opportunity to collect data, learn and often quantify what’s really important.

Lean is a mind-set, perhaps more than it is a process. Do the simplest possible thing to address those edge cases and learn how often they happen. If they happen very infrequently then they’re not that important. Alternatively, if the data says they happen a lot, then that is the point you make the decision to invest in finding and delivering solutions.

Related Posts:

Concurrency & Long running requests: Sinatra, EventMachine, fibers and Node

The problem – long running requests with blocking IO

Recently, I’ve been working on an alternative view onto Google Analytics data. This presents some challenges:

  1. Calls to the Google Analytics API vary but some can take over 10s.
  2. At this point I only have a single VPS instance which runs various ruby and wordpress instances and has limited memory and db connections

Problem 1, long running API requests (yes 10s is long) normally means that a process is tied up (blocking) waiting for a reply for 10s.

Problem 2, the single VPS instance has limited resources. Memory wise, I cannot have many processes running each using up a chunk of mememory.

1 + 2 = crappy performance, when there are multiple concurrent users the limited number of processes available to service the users requests may be tied up waiting for replies from Google.

The solution – get more concurrency from a process

Solution: get more concurrency from a limited number of processes. I can think of two solutions in the Ruby world:

A) Use JRuby which has native rather than cooperative threads
B) Use EventMachine and non-blocking async http requests.

Like node.js, eventmachine implements the reactor pattern – essentially requests cooperate by giving back control to the reactor when they themselves are blocking on something e.g. an http API call.

Solution A, use JRuby is reasonable but has some drawbacks. In theory a single JRuby process can concurrently handle 100s of concurrent requests, processing each in it’s own native thread. However, in practice we still need to be careful about non-reenterant code, particularly in gems and extensions we depend upon.

Solution B, initially appears more complex but is perhaps more predicatable. We’re explicitly recognising the blocking API calls issue and making it a 1st class issue for our system. Whilst I don’t like to make things more complex than they need be, often it’s better to recognise a fundemental issue and address it explictly.

How evented/async approach works

So how does B work:

  1. server receives a request from browser
  2. server “queues” the request in EventMachine’s reactor
  3. EventMachine’s reactor gets back control from whatever was running and pulls the next event from its reactor queue and executes it
  4. our app gets control from the reactor and starts processing the request
  5. our app makes a http request to Google Analytics API using an async library (em-http) and provides a callback to be executed when a reply is received; control returns to EventMachine’s reactor
  6. the EM reactor pulls the next event from its queue and executes it
  7. our earlier http API call (from 5) returns and the callback get’s queued on the EM reactor
  8. the currently executing event finishes, returning control to EM
  9. EM picks up the callback (from 7) which processes the results of the API query, builds a response and sends it to the browser. The HTTP request is finished
  10. So where’s the pain? Well async code with callbacks can quickly get messy, especially if there are callbacks nested within callbacks, nested within callbacks… and then an exceptional condition occurs. It is possible to structure code as sets of actions, where the callback code just links actions together, handling the flow of control between them. However, it’d be nicer if we could just write code that looks like normal sync server code. Ruby (1.9) Fibers and Sinatra-synchrony to the rescue…

    Fibers – explicit cooperation

    Ruby 1.9 has Fibers. Threads enable implicit concurrency, with some layer deciding when to switch control from one thread to another. Fibers enable explicit cooperative concurrency. A piece of code is wrapped in a fiber and executed. The fiber can yield, freezing its current state and returning control to its caller, at some later point the caller can resume the fiber, with the fiber executing from where it yielded control, with it’s pre-yield state intact.

    Here’s an example:

    Calling main( fiber ) gives the following output:


    Hello from main: 321
    Hello from the fiber: 123
    and we're back in main: 321
    And we're back in the fiber: 123
    back in main and finishing: 321
    

    Fibers, making async code look like sync

    I’ll just give a very brief overview. For more detail here’s a great article explaining how Fibers can be used to make async code look like sync code.

    Now we can just write async code that looks like sync code – no callbacks:

    So this is great, async code without the nested callback headache! I’m a fan of the lightweight ruby web framework Sinatra. This helpful and clever person has put Sinatra and EventMachine together (with a few other useful pieces) in he form of Sinatra::Synchrony.

    Node Fibers

    Ruby not your bag? Prefer javascript/coffeescript and node.js? I’m finding myself writing more and more javascript/coffeescript code in the browser – coffeescript + backbone.js + jasmine (for testing) is pretty good. I can certainly see the attraction of node.js, using the same language on the server as the client, particularly if it comes with the high-concurrency goodness of async events. Well there may be some good news in the form of Node-fibers – in particular take a look at the Fiber/future API.

Coffeescript and Jasmine for testing

Recently I’ve been writing a fair bit of browser-side Coffeescript + Backbone.js. Nice, but as the code base grows and gets more complex, the rubyist in me feels increasingly uncomfortable and less productive with the lack of TDD/BDD. What to do… Jasmine BDD for javascript seems to be a good answer.

Jasmine follows the same spec/BDD style as RSpec for Ruby and Kiwi for Objective-C (Kiwi is very good BTW). That is, set up your context using ‘describe’ + ‘before’, then make assertions on the post context state using ‘it/should’ blocks. Jasmine runs your test suite in the browser and seems to be blindingly fast.

For me, Jasmine tests also highlight one of the reasons I like Coffeescript more than plain old JS – less lines of code. BDD specs tend to have a lot of nesting as you build up contexts, so Jasmine written in JS tend to have a lot of nested functions. As these test functions tend to be pretty straightforward assertions, it’s seems a reasonably fair to compare coffeescript vs the generated js. Here’s one of my tests, notice that the coffeescript version is just 49 lines and pretty easy to read; the JS version is 78 lines and to my eye it’s not terrible but it is harder to read.

Coffeescript

Javascript

Developers, Vagrants, Puppets and Chefs

A friend recently pointed me at Vagrant. At first I wasn’t sure but after a play, I’m starting to thing this is a great idea.

Vagrant helps you create VMs for use as development environments. So for example, on my Mac I might run a Linux distro in a VM and use it to run the website + db that I’m developing. What’s more I can share that VM config with other developers working on the same project.

Vagrant works with Oracle’s free (and open source) Virtual Box virtualization software. If you’re a Mac user, you might be familiar with running Windows in VMWare Fusion or Parallels. This is the same sort of thing – though if you’re like me you’re more likely to run a Linux distro with Apache/Nginx, MySQL/Postgres etc..

Vagrant use VirtualBox to create and run virtual machines from your chosen distro and then provisions them with your development environment, webserver, app server, db etc using either Puppet or Chef. All this is done in a portable way and can be shared so that each of the developers on your team develop against exactly the same configuration.

Why is this a good idea? Here’s my top reasons..

  • consistent environment across the team – WoMM (“works on my machine”) is no longer an issue
  • bring in new developers more quickly with less frustration and teething problems
  • “hosed” environments can be dumped and new one created quickly and easily. Now you can ‘fork’ your environment not just your code.
  • dev environment can more closely match the server – even if you work on a MacBook. You could potentially use the same chef recipes or puppert scripts too.
  • It’s pretty easy to get up and running – why not run through the Getting Started with Vagrant guide and see what you think?

Here’s the Vagrant guys explanation of why this is a great idea.

Lean development is not the same as Agile

Agile project management techniques recognise that software project management is mostly about communication between the ‘client’ and ‘development team’. Lean techniques instead focus on determining as quickly as possible what adds value and what does not. My experience is that lean techniques recognise that value is added most quickly by increasing ‘communication’ with the end user. This communication is rarely explicit. Lean techniques emphasise implicit communication through observed usage, metrics and comparative split tests – real data quickly trumps opinion. Lean accelerates learning, minimising work in progress and the cost of learning what really adds value.

Agile improves on Waterfall

Agile project management techniques recognise that software project management is mostly about communication. Older techniques such as the Waterfall process spent too much time attempting to capture and formalize what the client actually wanted. Often vast amounts of time and money would be spent writing and agreeing abstract specifications which invariably turned out not to accurately reflect what the client envisaged. The creation of these abstract specifications did not identify and predict the technical difficulties the project would face and the magnitude of problems. Many of you will have seen the Tree Swing Cartoon about requirements.

Agile techniques address these issues by emphasising communication between the development team and their client throughout the project; accepting that it is very hard to determine exactly what a client wants without delivering designs and working software. Clients will change their minds as they see how in reality the things they envisaged actually work. Developers will discover that some features they thought would be simple to implement are not so simple. Both party’s understanding of the facts on the ground improve vastly as they make progress. They both accept that they need a dialogue. Both parties accept that it is sensible to make changes to the project “plan” as it progresses to completion and delivery.

Agile addresses the communication gap between developer and client. The dialogue leads to sensible decisions, better use of resources (productivity) and greater predictability. Agile techniques means software development increasingly does a better job of meeting the expectations of its clients. However, often there is another very important person who’s been left out of the conversation, the end-user (e.g. the customer on an e-commerce site). Agile is still relying on the client’s ability to determine what the end-user wants and what works.

The development team can help with this process (that’s product development team, including product management, UX and developers). UX people often use usability testing techniques to gather opinion and identify problems with designs. These are really useful and valuable, identifying some problems, encouraging best practices and avoiding opinion deadlock. Unfortunately they are highly artificial, the few users in their tests are often not actual customers, with an actual need, using their own time and money.

If used correctly usability testing techniques can smooth the edges of a design, making it flow better and making it easier to use. However, they are based on small sample sizes, often gathered from the same geographical source and demographic. They often miss the nuances of other demographics and of how people actually behave when it’s their own money they’re spending. For example, I’ve seen data that showed 6% of UK customers attempting to use just their house number and postcode to specify their address, on many sites that is insufficient. What are the chances that this 6%’s actual behaviour would be picked up in a artificial usability test – unfortunately it is not high.

There is a new kid on the block which aims to address this gap and bring the 3rd and most important party in on the conversation – the end user, in HD and in all their magnificent glory, warts and all. The new kid is Lean Development.

Lean

In October (2011) Eric Ries’ best selling book, “The Lean Startup: How Constant Innovation Creates Radically Successful Businesses” (which I thoroughly recommend by the way) was released to critical acclaim and quickly became a New York Times Bestseller. Eric’s book focuses more on startups and product development rather than specifically software development per-say. The main theme it shares with Lean Software Development is this gap between what the experts in a business think about a product or service and the reality of what the end-user/customer values and how they use it.

The lean approach borrows its name from Lean Manufacturing techniques such as the Toyota Production System. Lean manufacturing works on the basis that parts are pulled through the manufacturing process as they are needed, work in progress is kept to a minimum. This makes the manufacturing process highly reactive to customer demand and able to cope with day to day manufacturing problems whilst maximising throughput. If orders for a product increase, more of its parts are pulled through the manufacturing system. If there is a problem with manufacturing of a part upstream, then it temporarily stops pulling the parts it depends upon up through the manufacturing process.

In software development we’re not talking about manufacturing but product development, the process of determining what features a product or service should have, how it should work and making the feature available. Our goal is to build things that users value, that help them achieve their goals. For example, in ecommerce that’s finding a site, finding the product they want and successfully purchasing the product.

Lean development recognises that experts are making calculated guesses about what problems a user has, what they will value and how they use a feature. There is nothing wrong with this. We need to start somewhere and an expert’s guess is a very good place to start and with no information to the contrary it’s also a good next move. However, it is a bet, we are betting that the expert is correct. Lean development aims to reduce our exposure to this bet by reducing work in progress to a bare minimum.

Using a lean process, each new release of the software contains the bare minimum additions/changes needed to test the experts assumptions. We determine what the user thinks through observing how they behave, collecting data through analytics and analysing our own databases and software logs. This Minimum Viable Product (MVP) or feature (MVF) is the least work we can do, the least money we can spend to test the expert’s hypothesis, to learn more and plan our next step.

This leanness, this minimization of work in progress is key. The emphasis is build something quickly, try it, look at data and feedback, improve – quickly, repeat. Lean is real and fact based, it implicitly involves the end-user, it makes them a key part of the product development conversation. The user drives the development process, implicitly prioritizing and “pulling” the new features or tweaks that they want through the development process.

Agile processes have plenty of regular conversation with the client, including regular demonstration of progress. When using an agile process, releases of new software with new features to the end-user are months apart. In a lean process, the conversation involves the end-user, so time between releases is measured in days not months.

In a Lean process, real data quickly trumps opinion as assumptions are quickly put to the test, features evolved – refined, modified or even removed. Ideas for what to try next are picked up from scents in the data. These scents are not always purely quantitive, for example an increase in the number of times a question is sent to the support email address may provide a clue to a problem or opportunity.

Lean projects ARE typically explicit goals where success can be quantified and measured, for example the number of sales per day increases. Agile projects often have explicit goals but ones that are not explicitly quantified, instead Agile projects tend to be specific about how much time is available. Lean projects tend to focus on achieving quantifiable value for the end-user. Lean development recognises that delivering real tangible business value is more likely if you iterate quickly and respond to what the end-user is actually doing.

Lean development is not agile development – lean extends the product development conversation to the end-user and minimises the businesses exposure to expert bets.

Of course, the real world isn’t quite a black as white as I’ve just painted it. In reality the differences between what Fred calls Agile and what Wilma calls Agile can be huge. Some people’s “agile” processes will look more like the Waterfall process whilst others will share the attributes of a lean process to a greater or lesser extent. So please forgive the blunt contrast I’ve painted to illustrate my points. I hope you’ve enjoyed this post.

Innovation, motivation & goal driven management

I believe that organisations that want to encourage innovation and improve employee morale should consider whether they can be more goal driven, focusing more on goals and less on tasks. All organisations are ultimately goal driven but often goals are turned into tasks too early, this immediately reduces staffs’ room to manoeuvre – to investigate, test, react, adapt and innovate to reach the goal.

Do you know that a set of features and associated tasks will delivery your goal? Do you know these features and tasks will deliver the best result using the time and resources available? Progress towards a goal is more important than completing a set of tasks. Allocate resources to the achievement of a goal and use those resources to explore problems and solutions, collect and use data to determine next steps.

Managers often prefer tasks because they predictably show a concrete output for a given input. Completion of tasks A and B delivers features X and Y. It can be demonstrate that task A has been completed and feature X delivered. The achievement of a goal, a measurable outcome, is often much harder to predict. Worse, metrics highlight sunk costs, where completion of a set of tasks does not deliver the desired goal.

This human tendency towards predictability over goal achievement is heightened by processes and structures that bet the house on the latest big idea. In a task orientated environment, a determined focus on goal achievement takes guts.

Management skill is stating the goal, posing the questions, defining and focusing on outcomes and the metrics around them. We need a process (lower case ‘p’) that helps us plan, agree, track, adapt and communicate progress towards a goal. Here’s my thoughts on such a process..

Start with the goal/s of a project and the timescale and resources available to achieve the goal/s. How will achievement of the goals be measured? These metrics should be outcomes not completion of tasks or features. For example, is the goal to increase conversion of a checkout process; to increase newsletter signups; to reduce customer service calls; to increase visibility of a product? All of these examples are goals and their achievement can be measured.

Brainstorm ideas for achieving the goals. Are there any prerequisites / stepping stones? Organise these as a tree you want to explore rather than a list of tasks that need completing. The goal is the root of the tree. The nodes are sub-goals, ideas you want to test and measure how successfully a feature may deliver a goal or contribute towards it. The tree gives you a mechanism to understand, track and communicate how the team/s are attempting to explore the problems, try solutions and ultimately how these contribute towards the goal.

Prioritise the nodes below the root, based on which you think are most likely to make progress towards your goal. Start work on the highest priority node. If you’ve multiple teams, teams can start on different siblings. Review results (goal metrics and sub-metrics), which sub-branches should you try next? Are there new sub-branches you want to add? Are they the next best thing to try? Based on your results, do you continue down those sub-branches or move to a sibling? The tree is dynamic, explore it, grow it, prune it based on progress towards your goal.

Goals and metrics for a project provides focus. They encourage conversation about whether the proposed features really achieve the stated aims. This encourages investigation, looking at data, proposing and conducting tests. Furthermore, it encourages conversations about the cost vs value of pieces of work and encourages everyone to bring to the table alternatives which may be more effective and/or lower cost.

Why not leave turning goals into tasks to the “doers”? Tasks (and expectations of their delivery) are a necessary part of successful management. It is important to know “what is going to be done” and “when it will be done”. However, I’d argue that tasks should be a proposed by the “doers” in response to being set a goal by the “leader”. The leader sets goals, puts them into context and explains how their achievement will be measured. The leader facilitates the process of turning goals into tasks and expectations about their delivery. The resulting tasks can be “hung off” each goal in the tree, abstracted away out of sight until they are needed, leaving the focus on goals.

The ultimate aggregator – Google, travel & mobile

In my post, Adwords, vertical search and aggregator ‘middle men’ I mentioned a simple search enhancement that could enable hoteliers to better compete with the aggregators (e.g. Expedia, Travelocity, Booking.com). When a user searches using the keyword “hotel” or “hotels” some simple filters such as room type or budget would be shown. Advertisers would be able to bid against keyword/s and filter value/s. This got me thinking about what else Google (or Microsoft) could do for the travel vertical and how the growth in mobile might precipitate this.

Google recently purchased ITA Software a provider of Flight information. This potentially enables Google to show enhance search results with lists of flights and prices for various carriers and routes. Google could piggy back Adwords adverts on the side of this; or they go further and offer advertisers the ability to advertise against specific carriers and routes. Could they do something similar for accommodation?

Google becomes an accommodation aggregator?

Google could themselves become an accommodation aggregator. Google Base or Google Places could be extended for hotels to take rates, rooms and availability information. The results of a hotel related search would be a list of hotels rather than the normal list of hotel related websites. The usual filters you see on an aggregator site would be available – including travel dates. Near each hotel result there’d be paid (Cost Per Click) links to (multiple) websites where a room in the hotel can be booked.

For users this seems like an improvement, cutting out a step and helping them compare prices from aggregators and perhaps the direct from the hotel itself. This is the most obvious move but unfortunately for the user this is perhaps also the the most unlikely.

Technically whilst this should be pretty straightforward, certainly for a company with Google’s resources, in reality it involves integrating lots of 3rd party systems and information. Room availability and pricing is ideally real-time information. Will 3rd party systems be fast enough for super speedy Google? Will they be reliable enough. Google sets the bar pretty high. What if the information becomes stale? The user finds a supplier of a particular hotel and room for £120, clicks through to discover it’s now £150 or worse no longer available. Still you could still make the case that this is better, certainly no worse for the user than the status quo.

The biggest problem is what would the aggregators make of this? Aggregators are very big spenders on Google Adwords and unlikely to voluntarily play ball on this. Getting the aggregators to put the effort into such technical integration is unlikely given they would probably stand to lose the most, at least in the short term

Currently, the name of the game is getting a big enough return per click to afford to appear in the key first few result that drive the vast majority of referral volume. The aggregators achieve this by aggregating lots of supply => lots of choice (see my previous post).

Hoteliers benefit from vertical search

Could Google circumvent the aggregators? Hoteliers could benefit hugely from such a vertical search. Their relevancy to any paid click would be tremendous, they’d be very likely to convert the click into a booking and achieve a good return on their click. Perhaps this is where Google Places is headed, adding real-time vertical specific information: rooms, rates and availability. Google could even become the preferred tool for independent hoteliers to manage their rates, availability and bookings. Perhaps Google wouldn’t want to be involved in the actual financial transaction, there are potentially numerous legal, regulatory and customer service problems (Google Checkout prohibits use for travel purchases). However, this wouldn’t stop Google offering a API through which a hoteliers website could instantly store booking information in Google’s systems or simply update their room inventory.

Holiday (vacation) lets

What about holiday (vacation) lets, villas and cottages? Google might see significant take up in this sector as it is more fragmented than hotels and technically simpler. Holiday properties are typically owned by private individuals rather than the “holiday companies” who market them. The holiday companies act as aggregators, offering advertising, calendar management and in some cases full booking service. Google could potentially offer some of these services: a simple advertising solution, calendar management and itinerary update. This is a ripe market, with plenty of opportunity for a technical innovator to come in and attack the cost base and effectiveness of the holiday companies’ traditional marketing and customer services models, both property owners and consumers would ultimately welcome this.

The mobile search problem and aggregator apps

Searching for a hotel via Google Search is painful on mobile. Often the sites listed are not designed for mobile, though no doubt in time most sites will have mobile versions. Personally I find clicking back and forth between Google Search and the sites listed far more clunky on mobile. I wonder if mobile users will adopt a preferred aggregator and stick with the aggregator’s mobile app. In time this could represent a very significant drop in Google’s ad revenue.

Google could avoid such a drift to aggregators’ mobile apps by more aggressively pursuing vertical searches on Google Mobile Search. Vertical search could reduce the steps necessary to find accommodation, pulling more of the steps into Google Search and enabling the user to enter a hotelier or an aggregator’s site at a later step in the selection and booking process.

In exchange for offering the user more relevant content, Google would be able to better understand what the user is looking for. Knowing more about the user’s goal and preferences opens up other opportunities. Having additional information about the user’s goals also enables Google to intelligently cross-link to relevant flights, cars, restaurants etc... More relevance leads to more clicks.

It’ll be interesting to watch and see, if and where Google Search, Places, Base and Adwords meet. Will aggregators’ mobile apps dent adwords revenue? Will mobile provide an impetus for vertical search? Interesting times.

A/B testing small tweaks vs big changes

Found this question on Quora, “Should early stage startups be A B testing landing pages rather than isolated elements like headline copy?”. Here’s my answer…

As you’ve probably already found, given low traffic/data the tests may take a while to deliver any results. As either large or small changes could have significant impact, I wouldn’t advise testing big changes first, nor would I advise testing small ones first.

I’d start by trying to understand what problems there are:

  1. Think of the your goals for the site e.g. purchase, signup for service, signup for newsletter etc.
  2. Create some basic metrics for these goals e.g. number of signups, number of visitors, number of visitors who see signup page
  3. Create some “funnel” metrics from these e.g. % of visitors who see signup page, % of visitors who signup.

Once you’ve got these metrics, pause. Do any of these metrics surprise you? e.g. say the % of visitors who see the signup page is much much lower than you’d expected. It’s a good idea to get a few people to look at the metrics on their own and list what surprised them, then get together and compare.

Once you’ve some agreement on which metrics surprised you, you can prioritise which you’d like to improve. Take this list and brainstorm reasons for worse than expected metric / possible solutions. This is your test plan.

With testing it’s always worth bearing in mind the ROI of things. Take an incremental approach. If you have 5 suggestions for improving a metric but the cost (time) of doing them varies considerably then go for the lowest cost one first.