We're often asked about the "ROI" of collaboration.
At times the question seems absurd, for reasons that I'll explain later. Most of the time, though, we're quite sympathetic to those asking the question: They seek reassurance that collaboration will produce an acceptable return on their investment.
In this version of "Luke's Notebook", I will start a conversation on how you can develop an ROI model suitable for your use of collaborative frameworks. Be forewarned, though, that I will challenging simplistic notions of ROI, and when and why you think you need to calculate it.
On paper, ROI could not be simpler. To calculate it, you simply take the gain of an investment, subtract the cost of the investment, and divide the total by the cost of the investment. Mathematically, it is:
ROI = (Gain - Cost) / Cost
In practice, ROI is hard to calculate because it is based on what you consider the gains and costs used in the calculation. Let's call these ROI Factors. Because ROI is often characterized as hard (direct), or soft (indirect), the factors themselves must also be characterized as hard or soft. Examples of hard ROI include such things as:
- Travel Costs: Using online frameworks dramatically reduces or completely eliminates the costs of travel (e.g., airfare, hotels, meals, and so forth). One of Conteneo's largest clients, an $8B annual revenue financial services company, easily justifies the cost of their Weave license by simply keeping track the number of plane flights eliminated by online forums.
- Time-to-Market: Using prioritization frameworks such as Buy a Feature to eliminate unnecessary features enables product teams to release products more quickly.
- Decision Time: One of the virtues of a collaborative framework is that it helps teams reach decisions faster. You can measure this time-saving as part of your hard ROI.
Soft, or indirect benefits, are based on less easily quantified, or second-order effects. Examples of soft ROI include such things as:
- Engagement: Companies who have asked their employees to participate in portfolio prioritization have reported significant improvements in their employees engagement in the resultant projects: instead of questioning the priorities of the company and delaying work, these employees feel eager to get to work because they were involved in the process.
- Customer Smiles: "This was fun, thanks". "That was a fun exercise". "It was hard. But lots of fun". These are actual copies from the chat logs of customers using online frameworks to help prioritize backlogs. Collaboration is enjoyable when supported through a framework. Put another way, when was the last time your customers thanked you for asking them to take a survey?
- Sharing Tacit Knowledge/Experience: Frameworks provide a natural way for the participants to transmit tacit knowledge. Within a company, this tacit knowledge can help employees learn from others why a certain proposal is not optimal. When customers are involved, we see them routinely sharing best best practices.
As you can guess, hard ROI is easier quantify and calculate than soft ROI.
CHOOSING THE RIGHT ROI FACTORS FOR ANALYSIS
Because frameworks are used to solve specific problems, it is important to select the ROI Factors that are relevant to the frameworks chosen and the surrounding context. Let's use the Ideation-Shaping-Prioritizing model common in Design Thinking frameworks to help select and analyze ROI factors.
|Some Frameworks that Fit
||Prune the Product Tree
- Buy a Feature
- 20/20 Vision
- Prune the Product Tree
Generic Factors, affecting games in all of these areas.
Let's start with Hard ROI factors that cover all three of these areas:
- Meeting Time: The time saved by using an in-person or online forum to reach a conclusion in a meeting faster than traditional brainstorming.
- Travel Costs: Using online frameworks to eliminate the costs of travel, including such things as airfare, hotels, meals, and so forth.
- Total Cost-Per-Participant: What is the total cost per participant? Note that Conteneo's Weave platform tends to be neutral when used for market research regarding recruitment and participant incentives. The really big advantage for Weave is that our subscription-based model does not force you to pay per-participant research fees.
Unique or specific ROI factors for these areas.
- Idea Quality: What is the quality of the ideas generated?
- Idea Quantity: How many ideas are generated?
- Development Costs:The Shaping process will typically motivate some initial prioritization. This can be captured as part of the cost savings of not doing implementing a feature or a project.
- Time-to-Market: Using prioritization frameworks like Buy a Feature to eliminate unnecessary features, thereby enabling a product team to release products more quickly.
- Development Costs: The cost saved by not implementing a feature or a project.
Generic Factors, affecting games in all of these areas.
Here are some Soft ROI factors that cover all three of these areas:
- Engagement: Companies who have asked their employees to participate in portfolio prioritization games have reported significant improvements in their employees engagement in the resultant projects: instead of questioning the priorities of the company and delaying work, these employees feel eager to get to work because they were asked for their opinions.
- Customer Smiles: "This was fun, thanks". "That was a fun exercise". "It was hard. But lots of fun". These are actual copies from the chat logs of customers using Weave. Collaborating with customers brings a smile. Put another way, when was the last time your customers thanked you for asking them to take a survey?
- Sharing Tacit Knowledge/Experience: Framework-based collaboration provides a natural way for the players to transmit tacit knowledge. Within a company, this tacit knowledge can help employees learn from others why a certain proposal is not optimal. When customers are involved, we see them routinely sharing best best practices ("You really don't want to buy the flibblebopper feature, because if you configure your system this way you can work-around the problem. What you need is the geeble, because it solves a problem for which there is no work-around").
BUT SOME OF THESE FACTORS ARE STILL SUBJECTIVE!
I know. ROI Factors such as "Idea Quality" is a completely subjective concept. It depends on how you define quality. When you're dealing with ROI factors, especially in areas like ideation, your best approach is to be completely candid with about what you and your company are willing to pay for an idea. This can be challenging, especially since no one can accurately predict a priori how many ideas will be generated in an ideation event or the quality of these generated ideas. But, it is helpful to try.
In one client project, we were working with a group of about 140 executives. The client felt that if they got 500 total and 50 "high quality" ideas from the 1/2 day session they would be happy. The definition of "high quality" was that these ideas would pass the first gate in their Stage-Gate New Product Development Process. This means that each executive would need to contribute 4-6 ideas and that each group of 8 executives would need produce between 3 and 4 high quality ideas. This seemed entirely reasonable to us, but... what would this cost? And would the costs be worth it?
To put numbers behind this, let's say that the fully loaded costs of an executive are $200,000. Let's further assume that executives work 50 weeks/year, or 250 days, or 2,000 hours, for an effective "executive rate" of $100/hr. The 1/2 meeting therefore cost them $100/hr/executive * 4 hr * 140 execs = $56,000. Let's add in travel costs, meeting room costs, facilitator costs, and say that the 1/2 day meeting cost an even $100,000. That means that this team was valuing an "idea" at $200 and a "high quality idea" at $2,000. Based on on their New Product Development metrics, this client felt that these numbers would be very good. (And yes, the math here is simple, but trust me... all ROI is simple math. If your ROI ends up looking like a
Collateralized Debt Obligation
, chances are pretty good that you don't have any ROI and you should avoid the investment).
When we finished the event, the client team claimed they had identified 560 ideas and 65 high quality ideas - about $1500/idea. They were very pleased with the "ROI" of the event. I'm not at liberty to disclose which of these ideas have materialized as actual products, but the result of this initial engagement is that this client continues to use frameworks like Innovation Games® in these meetings.
SOMETIMES HARD ROI ISN'T HARD TO CALCULATE
Product Managers and Product Owners in Agile Software Development teams use our frameworks to prioritize their features. In Agile terms, an "Epic" is a really big chunk of work - something that you'd probably include in a press release! Epics are typically decomposed into smaller, more digestible chunks of work called "stories". A story has the quality that it can be completed within a specified time box referred to as a "Sprint". To see why Agile Teams work so hard on prioritization, let's look at the hard costs associated with delivering a new epic into the market. This is an interactive spreadsheet, and I encourage you to enter your own values. As you can see, the point is that sometimes it can be quite easy to calculate the ROI of playing our games.
WHEN ROI IS A SUBSTITUTE FOR FEAR
There are times when a potential customers' insistence for "guaranteed ROI" borders on the absurd: It is actually quite rare that business can generate ROI from collaborative activities without some degree of work and/or changes to their existing process. And these kinds of ROI come with risk. Unfortunately, not all people want to acknowledge risk, especially when it comes to investments. To see what I mean, consider a supermarket that needs to choose between repairing a broken rooftop compressor or replacing it with a new one. The comparison can be done quite straightforwardly, and depending on the payback period of the new compressor the supermarket manager can generally make a clear choice. This clear choice is not so clear when it comes to the kind of ideation and collaboration infrastructure created by the games.
I suspect the real reason that most people attempt to demand "guaranteed ROI" is not because they need a precise and accurate ROI, or even a perfect guarantee. Instead, I think their question is motivated by fear, such as:
- Fear of the unknown: The concept of framework driven collaboration and serious games to solve business problems is relatively new. Early adopters may not need economic justification, but the early majority does.
- Instead of fear of the word game, you could put Fear of Chaos. A natural reaction to inviting and including more colleagues in any process is to assume that efficiency decreases and the whole process will take too long.
- Fear of the word games: After all, aren't games supposed to be fun? We're a serious business. We do serious stuff. We don't have time for games or fun.
- Fear of delayed ROI: Suppose you acquire an annual enterprise license for Innovation Games® online. What if you don't get the "ROI" in the first 6 months? Indeed, how will you know which game your company played produced the ROI, as any single game could provide the requisite ROI.
- Fear of looking bad: What happens if we try a game and our customers or employees don't like it? Will I look bad? Will my boss fire me? Better not try this until it is "guaranteed".
Note that I'm not backing away from the need of these people to calculate an estimated or potential ROI from the use of the games. Calculating a game-based ROI is definitely a way to reduce these fears. And these ROI calculations, when used as a precursor to playing the games, can help sharpen the focus on producing actionable results, which we consider a great use of ROI. However, ROI can't overcome the fear of changing the way you work or the fear of the unknown.
USING ROI TO COMPARE OPTIONS
There are times when ROI is used to compare different ways of achieving the same goal. In these cases, your first step is to determine the full set of costs associated with the project before considering the potential gain. In many situations, our clients find that the costs associated with using the games are the same or lower than costs of other techniques. But watch out! You'll find that these kinds of ROI analysis can't really be done as easily as you'd like, because choosing a different way to achieve the goal actually changes the goal you're trying to achieve.
For example, let's suppose that you're using the games to prioritize potential features for a new product and you're comparing traditional surveys with Buy a Feature online. Some costs that are the same, regardless of the tool:
- Developing the participant screener.
- Recruiting participants.
- Paying a research incentive.
Some costs that will be different:
||Buy a Feature
|Preparing the questionnaire.
||Preparing the features for the game.
||Most of our customers report that preparing the features for our games improves communication between product management, product marketing, and product development because features are described in terms of benefits and projected costs. Note also that you can't easily design a single questionnaire to get the same data that you get in a Buy a Feature game, because a Buy a Feature game produces three kinds of results: the priorities of the customers, the reasons behind these priorities, and critical requirements that shape features.
|Administering the questionnaire.
||Playing the game.
||Surveys don't include facilitators, so it is easy to assume that the costs of playing a game with a facilitator must be higher. However, make sure that you're comparing the right numbers. A single game played with 7 people is equivalent to 7 completed questionnaires.
||The costs associated with analyzing results varies with the design of the survey. For example, while surveys that don't include any free-form text input are easier/cheaper to analyze, they lack the ability to generate deep insights into the motivations behind survey choices. Our games, on the other hand, include rich chat logs that provide deep insights.
|People hate surveys.
||People love games.
||This is the kicker, isn't it? How do you calculate the ROI of customers who were happy to give you their feedback vs. customers who reluctantly took a survey? Since it has inherent value, you can put a number against it and include it in your calculations. But keep in mind that "taking a survey" needs to factor into the "cost" factors of ROI (since people don't like to do it) while "playing games" needs to factor into the "gain" of the method (since people like to do it).
We have had a mixed set of experiences with customers who calculate comparative ROIs. In some cases customers were expecting certain categories of costs to simply disappear (e.g., "What? You mean we still have to develop a screener? Uh, yeah, if you want to get the right people playing your games."). When these costs didn't disappear, they decided to go with their tried and true method. While this is disappointing, a far greater disappointment occurs when the ROI for the games is better than traditional approaches but the customer still decides to go with a traditional approach. This isn't a rejection of the ROI, but a fear of the method (see previous section).
Fortunately, most of the time a comparative ROI calculation opens up more meaningful conversations about the merits of games vs. traditional approaches, which inevitably leads to discussions of "Business Value" vs. ROI. There are a large number of actions that we take in business where the individual ROI of any one action is either unknown, impractical to calculate, or negative, and yet we continue to do them because the accumulation of these actions produces sustainable business value. For example, every month I wrote this blog and Ari Gaal publishes our newsletter. These are the actions of a company who is committed to the long term success of its customers and has a similar commitment to success of a new industry. However, I'd never ask Ari to calculate the ROI of single newsletter. It just doesn't make sense.
It is really quite exciting when customers change their conversation from ROI to business value, because this means that we're getting even closer to determining how games can help them accomplish their goals, and, typically results in customers using the games.
WHEN ROI IS ABSURD
Many years ago, I was hired by Qualcomm to help them create an ROI tool for their FleetAdvisor System. As I was developing this tool, one of the Qualcomm executives I interviewed said "I never believe the ROI shown in these ROI tools. They all demonstrate absurdly large ROI after the second year. Our customers aren't stupid, and telling them to expect an unrealistic ROI from our systems insults them." He was right, so we worked on several ways to present a realistic ROI.
This, then, is my own concern with calculating the ROI of Innovation Games. It can be absurd. We've seen teams playing Product Box generate the seed idea for a multi-million dollar revenue software product platform. What portion of the ROI should we claim when the idea is clearly shown to be an outcome from playing the game? While some companies might try claim the complete ROI, we can't, because we know that good ideas are not enough. Smart Product Development, Effective Marketing, Efficient Sales... the list goes on. All contribute to the total ROI.
We've seen globally distributed teams playing Buy a Feature online eliminate more than $10M of unnecessary projects from the project portfolio. Amazing, stupendous, and yes, a bit absurd, ROI. But if that helps you make the case to your management team, go ahead and use it.
What's the value of a new idea? What's the value of avoiding unnecessary work? Tell us your numbers for these and we'll show you how Weave can exceed your ROI requirements. Without being absurd.