The Ethics of Product and Software Experimentation
June 30, 2020
In my short fifteen months as a Research Triangle area resident, it has lived up to its reputation as both a tech startup hub and a center of medical research. In fact, this reputation is why my fiancee and I ended up here. Rachael is pursuing her Master’s and Ph.D. in clinical psychology at UNC-Chapel Hill, and I work in product at Pendo. When we get our friends together, it’s like we’re speaking different languages. I don’t understand the intricacies of what they do day-to-day, nor do they understand what it means to be a product manager. They don’t speak in KPIs, OKRs, or ARR.
However, we do speak one common language: that of experimentation. As product managers, we form hypotheses. We design and conduct experiments. We collect data, form conclusions, and communicate the results. By design, we follow the same scientific method used by clinical researchers.
To the psych students’ surprise, product managers use the same vocabulary as they do to diagnose patient issues and run psychological studies at a massive scale. It’s just focused on user behavior instead.
Research and regulatory bodies
When Rachael and her colleagues propose a new scientific study, it must be approved by the university’s Institutional Review Board (IRB). This lengthy process exists to protect the welfare, rights, and safety of the participants, and guarantee that the study follows federally mandated requirements. The National Research Act of 1974 established guidelines for studies on human subjects as a reaction to infamous experiments like Milgram’s shock therapy studies and Zimbardo’s Stanford prison experiment. In the pharmaceutical and medical device industries, such studies must be approved by the FDA in much lengthier and more rigorous processes.
Software and … product managers?
In some regards, product managers are researchers as well. We interview customers, consult users about their problems, and form hypotheses about the best way to improve their lives. Like Rachael and her psychology colleagues (or any principal investigator on a large scale research project), we use experiments to study human behavior.
Given these similarities, should we start framing our jobs like scientific studies and using the same terms to describe our work? Are there enough commonalities to justify the regulation of running “experiments” on human subjects? What if the purpose is to help them do their jobs faster and more efficiently? Do PMs need their own version of the Hippocratic oath taken by doctors?
That said, the nature of our “experiments” are different. In B2B software, they’re typically run on a smaller scale with lower stakes. The primary objective is to validate some hypothesis. It can be resource-intensive and slow to design the ideal experiment with the proper controls and variables, so we do the best we can with what we have.
To clarify, I’m not trying to argue that the software industry should be subject to the same processes and regulations as medical studies. I am, however, trying to make the case that product managers have a responsibility to be intentional, mindful designers of experiences and “experiments.”
The responsible product manager
With the absence of a regulatory body to manage the ethics of software feature experimentation, the responsibility to ethically validate hypotheses falls on the product manager and product designers. As the principal investigators of these research questions, we have an obligation and responsibility to protect the subjects of our experiments.
Perhaps when we design and implement these solutions, we can make what we do a little more ethically aware. For example, we might ask ourselves the following questions:
- Are customers aware that they are the subjects of an experiment seeking validation of an idea?
- Is this experiment being communicated clearly and openly?
- Did the customers consent to have their product experiences change for their benefit?
At the rate we need to work to grow a business and establish a new market, the only protection of our customers’ rights, welfare, and safety lies in our courage to ask the right questions, think through the implications, reach out to customers outside of our expected user groups, and constantly improve. It is our collective responsibility to consider the implications of the products we put out into the world.
Wed Sep 21
A Beginner’s Guide to Unmoderated Testing
If you work in the world of product development, you are no doubt familiar with go-to user testing methods, like User Interviews. Yet not all methods are as widely used, even though diversifying your research method can enrich your data sets. That’s why in this post, we break down the basics of Unmoderated Testing.
Fri Sep 2
Connector Spotlight: Sandy Chahine
A passionate wanderer that firmly believes in the mantra of “trust the process,” Sandy Chahine, Design Researcher, doesn’t only work at the intersection of research and strategy at Connected. She is always eager to facilitate a conversation around important topics whether that be as one of the leaders of the Connected Women’s Network (CWN) or as a member of the Diversity, Equity, and Inclusion (DE&I) committee. Described as curious, passionate, and reflective by her colleagues, we know she’s always going a step further to learn, but also teach individuals around her. As someone who is into all things urbanism and public space, can you guess what her favourite product is?