Build Experiments Based on Your Hypothesized Assumptions

by Saad Kamal, Product Manager for AWH

Image for post
Image for post

Why experiment

We have known and unknown assumptions and biases, so we can’t truly predict outcomes as well as we’d like to. What we can do, however, run experiments to discern how hot (or how cold) our known, predicted assumptions are in the real world with real people.

Preparing to experiment

We experiment because we want answers to our assumptions. We have assumptions because we’ve talked to our potential target audience around the problem space of what our product is trying to solve for.

Persona

You’ve come across personas before, and they come in all shapes and sizes. For those who are new to the wacky world of personas, in over-simplified terms — a persona is a top layer of understanding of a type of user/customer/audience. Think of it as a profile of a person who has the rolled-up characteristics of actual people who fit that classification. I think of my personas as actual people

  • Picture: I need a face to the person; it helps everyone on the team empathize with the people we build things for.
  • Demographics: Age, Location, Education, Income…
  • Background: A paragraph or two about the persona’s background against the product or problem space.
  • Hopes/Desires: Specifically, around the problem space and then around the ‘jobs’ that need to be done.
  • Painpoints/Frustrations: Again, around the problem space and then around the jobs that this persona needs to do in that problem space.
  • Frictions to Change: Most people have the same frictions — Time, Energy, Money, but not necessarily for the same reasons. The way I define friction is, ‘If my product achieves all their hopes and desires and alleviates all of their pain points and frustrations for this person — what still gets in the way for this person to adopt the product solution.”

Questions

From the aggregation of commonalities that compose your persona’s hopes, frustrations, and frictions, you have questions. The how’s and the why’s as to the reasoning behind you hearing what you heard. Let’s say, for example, you’re working on a product that hopes to solve the problem of budgeting for a vacation. Your interviews bring to light that a common pain point is that people have a hard time accounting for visa fees for foreign travel. You can ask, ‘How can Jim (persona) account for the ancillary fees for getting a visa in Ecuador based on his Philippine passport?’ And thus, the spark for the experiment begins.

Hypothesis

From the questions which spawn out of your persona’s desires, pain points and frictions, the next step is to come up with some hypothesis as to why those are on your list and if they can be explained. I put explained in italicizes as what you’re really doing is asking more questions, rather than explaining the phenomena. I’m partial to the “What if” hypothesis, you can read about that here from the last post. What if Jim knew how much his visa fee would cost him for his international trip before arriving at the airport?

Image for post
Image for post

Experiment

Now that we have an asked hypothesis, we can start setting up our experiments to feedback into our assumed hypothesis. This is where we can start thinking about implementation approaches that will alleviate Jim’s woes.

What to expect

Whether you set up A/B tests, multi-variance tests, in-person usability and experience tests or any other method will heavily depend on the resources you have on-hand and what you’re actually experimenting with in the first place.

  • Timebox your experiments. Experiments inherently have no definite end. Whether it’s time-based, or metric-based, it is up to you to be disciplined enough to define what ‘done’ means for the purposes of your experiments.
  • Experiments begat experiments. It’s likely, even probable, that your experiments don’t answer the initial questions you started with. That’s ok. What you’ll likely end up noticing is that you’ve opened up pandora’s box of question marks. Cool. You’re gaining a deeper understanding of the problem if that’s the case.
  • Just because the findings of a small experiment have made their way through to production, doesn’t mean that you can’t gain more insight from the broader audience. Keep learning from your implementations, keep asking your target audience and let that guide you to your next set of experiments.
  • The second you think you know the answer, challenge yourself to test those assumptions. Be your ego’s biggest critic. You may be right this one time, but you’re not going to be right all the time.
  • Value learning over knowing. This isn’t confined to the world of product experimentation, rather for the entire journey of your product’s life. Don’t assume, the people who use your product will surprise you in both ugly and beautiful ways.

Written by

An elite team of product creators and data problem solvers.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store