Coffeescript and Jasmine for testing

Recently I’ve been writing a fair bit of browser-side Coffeescript + Backbone.js. Nice, but as the code base grows and gets more complex, the rubyist in me feels increasingly uncomfortable and less productive with the lack of TDD/BDD. What to do… Jasmine BDD for javascript seems to be a good answer.

Jasmine follows the same spec/BDD style as RSpec for Ruby and Kiwi for Objective-C (Kiwi is very good BTW). That is, set up your context using ‘describe’ + ‘before’, then make assertions on the post context state using ‘it/should’ blocks. Jasmine runs your test suite in the browser and seems to be blindingly fast.

For me, Jasmine tests also highlight one of the reasons I like Coffeescript more than plain old JS – less lines of code. BDD specs tend to have a lot of nesting as you build up contexts, so Jasmine written in JS tend to have a lot of nested functions. As these test functions tend to be pretty straightforward assertions, it’s seems a reasonably fair to compare coffeescript vs the generated js. Here’s one of my tests, notice that the coffeescript version is just 49 lines and pretty easy to read; the JS version is 78 lines and to my eye it’s not terrible but it is harder to read.



Developers, Vagrants, Puppets and Chefs

A friend recently pointed me at Vagrant. At first I wasn’t sure but after a play, I’m starting to thing this is a great idea.

Vagrant helps you create VMs for use as development environments. So for example, on my Mac I might run a Linux distro in a VM and use it to run the website + db that I’m developing. What’s more I can share that VM config with other developers working on the same project.

Vagrant works with Oracle’s free (and open source) Virtual Box virtualization software. If you’re a Mac user, you might be familiar with running Windows in VMWare Fusion or Parallels. This is the same sort of thing – though if you’re like me you’re more likely to run a Linux distro with Apache/Nginx, MySQL/Postgres etc..

Vagrant use VirtualBox to create and run virtual machines from your chosen distro and then provisions them with your development environment, webserver, app server, db etc using either Puppet or Chef. All this is done in a portable way and can be shared so that each of the developers on your team develop against exactly the same configuration.

Why is this a good idea? Here’s my top reasons..

  • consistent environment across the team – WoMM (“works on my machine”) is no longer an issue
  • bring in new developers more quickly with less frustration and teething problems
  • “hosed” environments can be dumped and new one created quickly and easily. Now you can ‘fork’ your environment not just your code.
  • dev environment can more closely match the server – even if you work on a MacBook. You could potentially use the same chef recipes or puppert scripts too.
  • It’s pretty easy to get up and running – why not run through the Getting Started with Vagrant guide and see what you think?

Here’s the Vagrant guys explanation of why this is a great idea.

A/B testing small tweaks vs big changes

Found this question on Quora, “Should early stage startups be A B testing landing pages rather than isolated elements like headline copy?”. Here’s my answer…

As you’ve probably already found, given low traffic/data the tests may take a while to deliver any results. As either large or small changes could have significant impact, I wouldn’t advise testing big changes first, nor would I advise testing small ones first.

I’d start by trying to understand what problems there are:

  1. Think of the your goals for the site e.g. purchase, signup for service, signup for newsletter etc.
  2. Create some basic metrics for these goals e.g. number of signups, number of visitors, number of visitors who see signup page
  3. Create some “funnel” metrics from these e.g. % of visitors who see signup page, % of visitors who signup.

Once you’ve got these metrics, pause. Do any of these metrics surprise you? e.g. say the % of visitors who see the signup page is much much lower than you’d expected. It’s a good idea to get a few people to look at the metrics on their own and list what surprised them, then get together and compare.

Once you’ve some agreement on which metrics surprised you, you can prioritise which you’d like to improve. Take this list and brainstorm reasons for worse than expected metric / possible solutions. This is your test plan.

With testing it’s always worth bearing in mind the ROI of things. Take an incremental approach. If you have 5 suggestions for improving a metric but the cost (time) of doing them varies considerably then go for the lowest cost one first.

Tip for email Marketeers – don’t keep changing your email address

Ever noticed how email clients like gmail don’t (by default) load images in emails? Most email marketing systems use an embedded image to track who opens their emails. To protect your privacy and thwart spammers discovering which email addresses are valid and active, gmail (and many others) don’t load images unless you tell them to.

You may have noticed a link for loading images, and a link for always loading images in emails from sender Y. I get quite a few marketing emails, ones I’m interested in from companies I’ve previously made purchases with. I’m continually amazed at how often I have to click the ‘always allow images from Y’ link because someone has yet again changed the email address the marketing email comes from.

So nice email marketing people, please stop changing your send address. Do this for me, so I don’t have to keep clicking on a link to see your image laden messages. Do this for yourself so you can better track who reads your emails. You do use stats of your email marketing funnel to improve your emails’ effectiveness don’t you?! There’s an idea for a future post.

8 ideas for understanding product / market fit using the web

Here are 8 ideas for understanding product / market fit using the web. Please add thoughts and other ideas as comments.

1. Run a news aggregator

Run a site which aggregates interesting news and articles related to your product area, presenting a headline / brief summary and then linking out to the original content. Categorise the articles and tag them. Keep a tally of which tags are used most often and observe clickthroughs, keeping a tally of clicks by tag.

A good source of content is twitter and it is relatively easy to do. Create a twitter list of people that post news and articles relevant to your area. Bingo!

This is more likely a qualitative aid than a hard quantitive technique, it would take quite some time and effort to make such analysis statistically significant and it is important to understand how the content you are posting may bias clickthroughs. However, if you do succeed in building an audience or community, that in itself is valuable.

2. Run a forum or question/answer site

An interesting example of this is Oracle Overflow. Whilst the aim may be to attract and help a community of Oracle users, this is potentially a gold mine of information on problems Oracle users encounter which in turn could spawn many ideas for products and tools. 

3. Survey, send out short questionnaires – perhaps with a prize draw

Ask customers or potential customers to fill out a questionnaire asking about their requirements and/or experiences. My advice would be to use a mixture of multiple choice and free text questions. Multiple choice questions have the advantage of being relatively quick to answer – many people don’t like writing. As multiple choice questions have a closed set of answers they are easier to process and analyse. However, sometimes you want to get a feel for problems people are having or their feelings towards your product so it can be useful to mix in some free text questions.

Achieving a good level of participation in your survey is very important. To encourage participation, keep your questionnaire relatively short and tell the user upfront how many questions and/or how much time it will likely take to complete. Consider giving them an incentive to complete your questionnaire. This doesn’t necessarily have to be something very big and expensive. I’ve successfully run a questionnaire offering a £50 SatNav as a prize and got over 500 replies. Obviously, the size of your response will be affected by the size of your audience.

Here’s an excellent post on “What my Survey Taught Me” .

4. Encourage questions and enquiries – make sure you follow up though

If you have the resources to answer questions then why not encourage visitors to your website to ask questions about your product / service? Along with their question make sure you automatically capture what product/content they are looking at and any options they may have already selected i.e. a hotel room in Amsterdam, April 25th. The product data and options are important because it may help you to categorize your data when you analyze it.

When reading through peoples’ questions, make sure you tag them into categories for later analysis – for example, if you sell radios you may find 30% of questions are asking whether the radio has a USB port for MP3 upload, if you’ve tagged these questions with for example ‘usb’ and ‘mp3′ this should be easier to identify and quantify.

Once you’ve collected a large set of questions you may be able to improve the product. If you can’t improve the product, perhaps you can improve sales by changing your sales pitch, or including an FAQ.

5. Gather reviews, use post purchase or enquiry questionnaires

Gather data and content about people’s satisfaction and experiences with you product and purchasing / enquiry processes. As before, make the questionnaires relatively short, use multiple-choice questions possibly with a few free text questions mixed in.

Design your questionnaire so you can categorize replies and if necessary take further action, for example: happy, unhappy, needs follow up. One way of doing this is to explicitly ask them if they are (a) very happy, (b) happy, or (c) unhappy.

If would like to use content from these questionnaires on your website, encourage advocacy by asking a question such as, “What would you say to a friend to encourage them to buy?”. You probably want to avoid using their answer to this question if they have earlier selected option (c) unhappy!

6. Design your site to self segment

Enable the user to self-segment by providing section or filter links.

For example, if you are selling televisions, you probably already provide filters to limit the list of available products by type, size and other features. Capture this information and use it to inform the selection of products you offer such that it better matches the demand your marketing is creating.

If your site is about operating an email server, you no doubt have sections about different aspects of operating the service, for example: backup, setup, spam, performance etc.. By getting people to self segment you can get a idea for where the biggest problems and pain points lie.

The following may be  a little controversial. If you use it, I’d recommend you use it with care. Consider providing links to products or content you don’t currently have but are considering. Don’t over do this though, people hate following links then being disappointed. Don’t do this is you cannot or do not intend to fulfill the need very soon. As a consolation, offer similar alternative products or related articles.

7. Look at funnel drop outs

Using Google Analytics or a similar tool, set up goal tracking e.g. product purchase, sign up for newsletter, post question, make enquiry. How to setup goals and funnels in Google Analytics.

Once you have goals and funnels set up and have a few weeks of data (depending upon how much traffic your site gets). Take a look at Google Analytics’ Goals > Funnel Visualisation. You’ll see a vertical list of funnels showing progress through each step. Take a good look at the right hand column, it shows where people who abandon the funnel go to. Any patterns in funnel abandonment will likely be useful for web design and conversion optimisation activities but they may also point to product opportunities to or problems.

For example, if you have a site which sells flights, you may have a banner or link to pre-book car parking at the airport. I wouldn’t advise adding such distractions to your checkout process, but if you already have them they may give clues as to related items your customers are looking for, or concerns they have with your product or process. In this example, perhaps flight and car parking could be offered as a single package/purchase.

8. AB test

This requires more expertise and effort but can be a very effective way of evolving a website to better suit customers needs. For those not familiar with AB tests the idea is that users are split into 2 groups, each group is shown different content, features or user interfaces. The test measures how likely each group is to complete a goal. If you are selling something the goal is usually a sale.

Do you have any techniques or tips for understanding how users use your sites or products? If so please share them in the comments.