We recently travelled to Unite Paris by Kameleoon for a day of talks and conversations with experimentation, product, UX, and digital teams from across the industry. The roundtable sessions were a real highlight, with smaller, open discussions where practitioners could share experiences and challenges. A big thank you to Kameleoon for such a brilliantly organised event!
Here’s a look at the talk Mark Pybus, CEO at Creative CX, gave on the studio stage:
Better problems, better experiments: a live ideation experiment
Mark took to the studio stage alongside Alexandre Chomienne, Digital Performance Manager at Courir, for a session on problem-first experimentation. Rather than just talk at the audience for half an hour, we actually ran a live experiment with them.
For some context, Courir is the number-one sneaker retailer in France. It has 330 stores and over 3,200 collaborators across Europe, sold 9 million pairs of sneakers in 2025, and generated €800 million in turnover, 18% of it digital. The challenge we set the audience was to improve Courir’s product detail pages and increase the conversion rate.
The audience was split into two groups. Group A was shown only a Courir product page and a high-level objective (e.g. “How do we improve the experience when some products are sold by marketplace sellers?”), then asked to come up with solutions. Group B was given the same page, plus a triangulated problem statement based on real customer research:
Shoppers can’t tell which sizes of the same Adidas shoe receive discounts and loyalty perks, so they find out at checkout and abandon their carts.
When we then compared the kinds of responses shared by the two groups, the difference was clear. The people who understood the problem came up with noticeably more focused solutions than those in the ‘generic-goal’ group, whose responses were much more scattered and unsystematic. It was a live demonstration of why understanding the problem matters, and what happens when you ideate blindly.
The problem with “spaghetti testing”
This is where “spaghetti testing” comes in: throwing lots of ideas at the wall and hoping something sticks. Teams that don’t have clear customer problems to go after have no choice but to test random ideas, and far too much time is wasted trying to solve problems that don’t exist.
To demonstrate this further, we gave the same two briefs as in the original live experiment to an AI model, Opus 5.5, to see how it fared. Without the problem statement, only 4% of its ideas fit the actual problem, and 63% were untargeted, generic ideas that could apply to almost any product page. With the problem statement, problem-solution fit rose to 95%, and the untargeted ideas dropped to 7%.
What problem-first looks like in practice
At Creative CX, we use ongoing opportunity discovery research to bring together stakeholder interviews, expert reviews, moderated usability testing, and data analysis. These findings inform quantified, triangulated problem statements, which then feed into focused ideation sessions and experimentation.
This gives teams a clear connection between customer problems, the ideas they develop and the tests they run. To learn more, read our blog post on problem-first experimentation.
As an example from a travel brand we work with, Ideas that addressed a problem statement from our research won 44% of the time, compared with 29% for fly-in, business, competitor-inspired or ad hoc ideas that weren’t tied to any known problem. They were also far less likely to end up conclusive.
The opportunity nobody thought was there
Discovery research also uncovers opportunities that nobody expected. Take the login step on a hotel client’s booking journey. Nobody thought there was an opportunity there, because the page followed all the best practices:
- Prominent CTAs
- A clear booking summary
- A secure checkout padlock
- Guest checkout available
- A low drop-off rate
On paper, there was nothing to fix. But during moderated usability testing, a participant hesitated at the “checkout as guest” button. In a hotel context, check-out typically means leaving the hotel and settling your bill. That’s not what someone booking a room is trying to do. The wording introduced confusion at a point where customers needed to move forward with their booking.
“If I don’t have an account
I can’t log in…and I can’t
checkout as a guest yet”
It was a small detail we wouldn’t have found without discovery research. We tested it, and the winning variation removed any mention of “checkout as a guest” site-wide!
Tackling a troubling confirmation bias
The live experiment showed what happens when teams start with a clear customer problem. But getting the problem-first approach right goes beyond the ideas we generate. Mark went on to explore how confirmation bias can creep into the research process, and why making experimentation accessible to more people across the business matters.
We see confirmation bias everywhere in digital, so much so that the Creative CX team have aptly dubbed it FILTH, or Forcing Insights & Learnings To Hypotheses.
It tends to play out like this:
- The team has an idea or hypothesis. Maybe they’re copying a competitor, or the idea just came to someone.
- They cherry-pick data insights and research learnings to back it up.
- They prioritise and push through their “data-driven” ideas.
The research becomes a way of validating the solution rather than understanding the problem. Problem-first experimentation flips this, starting with the customer and working out which problems are worth solving before deciding what the solution should be.
It feels especially relevant as AI and automation make experiments faster and easier to launch. When building and launching gets cheaper, the quality of the question matters even more.
Experimentation is too valuable for one team
The final message was that experimentation shouldn’t sit with a single team. Giving more people across the business the power to experiment leads to better ideas and greater velocity.
YNAP is a good example of why. Its central optimisation team, working apart from Product, eventually hit a limit on how much impact the current operating model could deliver.
The two teams were also working on the same parts of the journey, which led to clashes over releases and objectives, and meant most releases went live untested. Working with YNAP, we moved them to a Centre of Excellence model, where each Product Team could test its own ideas with support from a central team. Over two years, they went from 4 to 20+ A/B tests a month and tested 5.8x more ideas.
If you want to find out more about YNAP’s journey, read the full case study here.
More teams testing means more problems get solved, and that’s the point of starting with the customer.
Lessons to take away
- Problem-first for higher win rates and greater impact
- Don’t be FILTHy
- Enable every team to experiment, to scale your problem-solving
A huge thank you to Alexandre and Courir, and to everyone who joined us and got involved in the live experiment. Merci, Kameleoon; we’ll see you next time!
