In January, Wharton professor Christian Terwiesch gave his MBA final exam to ChatGPT. It passed with flying colors.
Now, he’s at it again with a new experiment to determine whether ChatGPT can come up with product ideas better and faster than his students. It can. And cheaper, too.
“I was really blown away by the quality of the results,” Terwiesch, a professor in the operations, information and decisions department, said in an interview with Wharton Business Daily. (Listen to the podcast.) “I had naively believed that creative work would be the last area in which we humans would be superior at solving problems … so we set up this horse race of man versus machine.”
The experiment is described in his latest white paper, which is co-authored with Wharton OID professor Karl Ulrich, former Wharton graduate student Lennart Meincke, and Cornell Tech professor Karan Girotra. Terwiesch and Ulrich also co-authored The Innovation Tournament Handbook; they wrote about the new experiment in an op-ed for The Wall Street Journal.
The scholars started with 200 ideas submitted by MBA students from a 2021 class. The class assignment required them to come up with products for the college student market that would retail for $50 or less. The timing of the class is noteworthy because the ideas predate the release of ChatGPT and other generative AI tools now available for free.
Next, the scholars fed the same assignment to ChatGPT-4, prompting it for 200 ideas. For the first set of 100, the program was given instructions with no context or examples. For the second set of 100, it was “seeded” with examples of good product ideas.
“I had naively believed that creative work would be the last area in which we humans would be superior at solving problems.”— Christian Terwiesch
Predictably, the chatbot made quick work of the task, generating 200 ideas in about 15 minutes. (A single human generates about five ideas in 15 minutes; the rate goes down for groups.) But the team didn’t stop there. They wanted to know whether the ideas were viable, so they conducted a blind survey with potential consumers to gauge their interest in buying the products.
The results were eye-opening. The average purchase probability of a ChatGPT product was 47%, compared with 40% for the human idea. The preference for “seeded” ideas was even higher at 49%.
Let AI Be Your Co-pilot
Terwiesch, who is also co-director of the Mack Institute for Innovation Management, said there are three big takeaways from the experiment, and the first is a “no-brainer.” Everybody should be using ChatGPT to help them generate ideas.
“It doesn’t matter whether they are entering the local business plan competition or working on the cure for cancer,” he said. “It’s cheap. It’s fast. It’s good. What’s not to be liked? Worst case is you reject all of the ideas and run with your own. But our research speaks strongly to the fact that your idea pool will get better.”
Second, he thinks generative AI can help people move faster through the bottleneck of innovation, from creating ideas to evaluating them. That latter stage is far more difficult and more critical for success, he said. A hundred ideas may yield only five good ones, and it takes human expertise to discern the winners.
“Third, I think we should not boil it down to man versus machine,” Terwiesch said. “We need to find ways in which the AI becomes your creative co-pilot. Together, you can become a more innovative team.”
__
Guest Author: Christian Terwiesch is the Department Chair and a Professor in Wharton’s Operations, Information and Decisions department and co-director of Penn’s Mack Institute for Innovation Management.
Seeking to build and grow your brand using the force of consumer insight, strategic foresight, creative disruption and technology prowess? Talk to us at +971 50 6254340 or engage@groupisd.com or visit www.groupisd.com/story
