Split testing (also called A/B testing) allows for two versions of something to be tested between two distinct but equal groups of customers and the results compared to find the best way forward. If you've read anything on Lean Startup then you could be forgiven if you thought that split testing is only used for web based products. It's not. The split test can be used with two versions of a web site but can also be used with a physical product, sales campaigns, marketing campaigns, or most anything else that is related to changes that potentially adds value for the customer and grows the business. But what if your product is sold only to a couple hundred customers (a common occurrence with B2B)? Almost certainly traditional split testing with the product is not practical with so few niche customers. The problem then becomes, how can you tell if any product change is truly adding value for the customer and the business?
Niche Testing
You can do a thing I'll call 'Niche Testing'. Unlike split testing where you conduct simultaneous testing between two groups, niche testing is done before and during a product change with the key customers. Niche testing is conducted using the Build-Measure-Learn loop format and should work with as few as 30 customers. I picked 30 customers rather arbitrarily but is a number that is probably the high end of what the company will be capable of interviewing in a short period of time (3 people doing 2 interviews a day each in one week).
What I propose doing is conduct a customer satisfaction survey using your proven survey format to ascertain the current customer feelings toward the product. You will need to determine how you measure the customers' responses to the survey beforehand and identify specifically what you hope to learn from it. The survey would also contain questions relating to the area that the team is considering changing and improving. Don't use leading questions to promote your proposed change since this will almost certainly result in skewed responses; find out what your customers can't live without rather than what they can live with. The survey should be conducted face-to-face with the customer if at all possible so you can read their body language as they relate their experiences with the product and answer questions. The customer survey is your earliest MVP.
After the customer surveys are complete, you measure the response and consider your next step. There are four major categories of responses:
- the customers love the product as is and will continue paying support fees without any changes,
- the customers have no need for the changes you're considering,
- the customers have stopped using your product or are switching to a competitor's product, or
- the customers have a need for the type of changes you're considering.
In the first and second category of responses you need do nothing for now except cash the customers' annual support fee checks, right? Although this sounds nice and easy, it is most probably the recipe for business failure in the long run. Doing nothing means you're not innovating, not anticipating your customers' future needs, and not trying to grow your business. So what do you do if no one has the problem you envision to solve? The answer lies within the product team and the character of the customers' responses to your questions-the customers may have indicated a completely different problem than what you envisioned. The product team should weigh their hunches, insights, wisdom, intuition, and world experiences against customers not having or not knowing they have the problem you envision to solve. With either of these responses the team will probably need to pivot, find a different feature, unless you think the customers are unaware of the problem you want to solve.
In the third case you've lost your customer. This may be OK if, and only if, the customer's business has changed where they no longer have a need for your product. In all other circumstances you may have missed the customers' true needs or not innovating as aggressively as you should. With this response not only should the team pivot but they may need to reassess their product and customer segments.
In the forth case the team will 'perservere' and start building a demo, prototype or begin implementing the new features into the product. I would suggest a demo of the new features as a time and cost saving measure until you know, through innovative learning, that this feature will solve the customers' problem.
Throughout the Build-Measure-Learn loop you need to keep the focus on what your goals are. For B2B customers Retention, Referral, and Revenue metrics are king with maybe Retention being the most important. Each iteration through the Build-Measure-Learn loop adds more knowledge to what the customers want and need as long as you are measuring the right metrics.
Summary
Niche testing begins with conducting a survey with your key customers. From the survey results you then begin further and more detailed testing of your product innovation concepts, measuring the customers' responses and learning what changes to the product adds value for them. You may need to pivot to a different feature if customer responses are tepid. Continue through the Build-Measure-Learn loop until your learning goals are met.
No comments:
Post a Comment