What is Your Cost of Experimentation?

I have been taking my readers through a series of posts on innovation decision making built around the WRAP process from Chip and Dan Heath’s book Decisive.  The WRAP acronym allows you to remember their four decision anchors – Widen your options, Reality Test Your Assumptions, Attain distance before deciding and Prepare to be wrong.  If you need to catch up, you can find the first two posts here and here.

For this week’s post, I want to explore more deeply one aspect of the”R” in WRAP that we began exploring last week: reality testing your assumptions.  A great current example of this can be found in this recent piece on U.S. Banks and their Chief Innovation Officer, Dominic Venturo.

In this piece, Dominic discusses a current beta test U.S. Banks is running on voice biometrics as a way to replace lengthy and hard to type passwords for mobile devices.  You can imagine the tension here between usability and security when it comes to transaction banking on your phone.

Several items to note here:

1. Dominic stays away from the “shiny object” syndrome and keeps the focus on the customer’s pain point, which is the difficulty of entering long passwords.   He is keeping the spotlight tightly on the needs of his users.
2. They have used a strategically meaningful sample.  While not revealing details, you can tell they have done a great deal of research, and that the information they are tracking provides a nice cross section of users.  I’m willing to bet they have done their trial design homework, and have the splits and sample sizes well worked out.
3. They are using their vast employee pool as the beta population.  This is shrewd and extremely effective in developing cost-effective testing methods.  Employees bring an already vetted and reasonably diverse view to the table – all within the firewall of a confidential “family”  discussion of candid and authentic feedback.

Stepping back for a moment into the current world of innovation, we know active experimentation trumps analysis for speed of context and quality decision making.  And while this capability is one of the great hidden competitive advantages of an agile organization, a core issue for any firm that needs to perform active experimentation is the cost of forming and executing these meaningful experiments:

The cost of forming and executing viable experiments is a new competitive battleground.

Much earlier in my career, when I was leading design engineering for a global firm involved in custom private communications systems, one of the first things we did was purchase the capital to build the “engineering” system.  This meant that we purchased everything we needed to build a full scale rollout of the end-to-end system that employees would use to get real world data at scale.  We are talking about many square miles of coverage and hundreds of devices – an investment of several million dollars.

The upside of this was an unprecedented speed and quality of user feedback on incremental changes and updates that built some of the finest and most respected products in the world. The key thing to note here is that once the investment was made, the incremental cost of evaluating the changes was very low.  Payback started right away, as each major release of hardware and software could be immediately vetted inside our firewall to ensure a high-quality customer experience.  Being on the leading edge of technology application allowed substantial gross margin benefits to the P&L, and easily offset the cost of the system.

Pulling this together – there are several meaningful questions for you, the innovation leader:

1. What is your cost of experimentation?  Do you have a platform of talented individuals ready and willing to try things out?  Are they skillful in giving you feedback?

2. What platforms do you need to invest in now to allow you to get rapid and low cost feedback for products on your roadmaps?  Are you paying as much attention to information gathering and experimentation platforms as you are the product?

3. Are you making the best possible use of your employees’ intrinsic knowledge?  Sometimes you can find out amazing information from using a broad cross section of users that carry your badge.

I would love to hear how you are thinking about this very important new competency. Please drop me a line, or leave a comment below.