A Lesson In A/B Testing

We’ve always felt strongly that we should share our lessons in business and technology with the world, and that includes both our successes and our failures. We’ve written about some great successes: how we’ve improved support response time, sped up applications, and improved reliability. Today I want to share an experience that wasn’t a success.

This is the story of how we made a change to the Basecamp.com site that ended up costing us millions of dollars, how we found our way back from that, and what we learned in the process

What happened?

This story starts back in February 2014 when we officially became Basecamp the company. This was a major change — a rebranding, the discontinuation of some products, the sale or spinoff of others, and more. As part of that process, we decided to redesign basecamp.com (our “marketing site”) to reflect that it was not only the home of Basecamp the product but also Basecamp the company.

The result was a fairly dramatic change, both in content and visual style. The redesign extended well beyond just the landing home page (you can browse the archived version before and after we became Basecamp), but the most noticeable change was to the main page.

[..]

These were certainly painful and expensive lessons to learn, and we’re fortunate that the fundamentals of our business are strong enough that this wasn’t anywhere near an existential crisis. We’ll be a better company as a result of having gone through this, and hopefully we won’t make the same or similar mistakes in the future.

We’ve been making minor site changes over the past month as we prepare for our redesign to launch next week, mostly our minor changes have been A/B tests to help us gauge what changes work and what changes do not, this helps as we plan out our redesign, but this post is a good reminder to test changes first as you never know what you might miss.

Source: https://signalvnoise.com/posts/3945-how-we-lost-and-found-millions-by-not-ab-testing