AI may have changed how quickly organisations can create solutions, but it has not changed the importance of choosing the right problems to solve.
Why understanding the problem matters more than ever
Organisations often move too quickly from a perceived issue to a proposed solution, without spending enough time understanding the problem they are really trying to solve.
At CCX, we’ve spent years helping organisations avoid exactly that through a problem-first approach to experimentation. The reason is simple: the better you understand the problem you’re trying to solve, and how it impacts your business goals, the better your experimentation programme becomes. Read more about our problem-first approach here.
When more time is spent on research, diagnosis and understanding the customer journey, we consistently see higher win rates, fewer inconclusive results, and experiments that move metrics that matter.
The risk of solution-first thinking isn’t that individual experiments fail. It’s that businesses spend time and money improving the wrong things. Teams dream up solutions to perceived issues or vaguely understood journey challenges, often influenced by stakeholder opinions, specialist experience, or competitor activity.
Even when data is involved, it can be used to reinforce an existing hypothesis rather than to understand what’s really happening, a form of confirmation bias or what we like to call FILTH (Forcing Insights & Learnings To Hypotheses), which we’ve explored in more detail here.
That’s why I’ve found the current conversation around AI so interesting. We’re seeing exactly the same pattern emerge again. The difference is that this time, the technology itself is encouraging solution-first thinking.
Unlike previous waves of technology change, AI didn’t arrive through long procurement cycles and controlled rollouts. Tools like ChatGPT became available to almost anyone almost overnight, dramatically lowering the barrier to trying something new.
That accessibility quickly flowed into organisations. It democratised innovation and enabled people across every part of a business to experiment with new ways of working. That’s a real opportunity, but it also creates a risk: the technology is so easy to reach for that it can become the starting point.
Instead of asking “What problem are we trying to solve?”, many organisations are asking “Where can we use AI?”
Once AI becomes the starting point, prioritisation becomes harder to manage. The tools are accessible, and the pressure to move quickly is real. Before long, teams find themselves with a growing list of AI ideas: agents to build, workflows to automate, content to generate, journeys to personalise.
Some of those ideas may be valuable. But without a clear link to a meaningful problem, it’s hard to know which ones deserve investment or scaling. The question isn’t whether AI can be used; in most cases, it can. The better question is whether it’s solving a problem that matters enough to justify the time, cost and change required.
Once a potential AI use case is linked to a meaningful problem, the next question is not “Can we build it?” but “Can we demonstrate that it creates measurable value for us?” One of the biggest misconceptions about AI adoption is that success comes from building AI capabilities. In reality, success comes from proving they create value.
Every AI Idea Starts as a Hypothesis
Once you’ve identified a meaningful problem, the next challenge isn’t simply building an AI capability. It’s understanding whether that capability actually creates value.
Every proposed AI capability starts with an assumption: that an agent will reduce service handling time, a tool will help travellers rebook faster after disruption, or AI-generated content will improve conversion on a product page.
None of these is a fact. They are hypotheses waiting to be tested. The challenge isn’t simply deploying AI; it’s discovering whether a particular capability creates value, for whom, and under what circumstances.
Proving that means measuring it properly.
Measure Against the Alternative, Not Against the Past
This is the part most organisations get wrong. Many still measure AI with a simple before-and-after comparison: we introduced an AI assistant in June, and productivity improved by July. But was that because of AI, or because of seasonal demand, changing workloads, or something else entirely?
Experimentation answers this far more reliably. Rather than comparing today with last month, it compares different approaches under the same conditions. A proportion of customers, employees or processes continue as they are, while others experience one or more AI-driven alternatives.
Measuring those groups in parallel isolates the impact of each approach with far greater confidence, giving organisations confidence that any difference in performance is genuinely caused by the AI capability rather than changes happening elsewhere in the business.
This is also where experimentation goes beyond traditional A/B testing. Different prompts, agents, workflows or experiences can be compared simultaneously through A/B/n experiments, allowing organisations to learn not simply whether AI performs better than today’s approach, but which implementation delivers the greatest value, for which users, in which situations.
Importantly, experimentation isn’t just about deciding whether AI is better than today’s process. It helps organisations compare multiple AI approaches against each other, refine prompts, optimise workflows and understand which solution performs best for different customers, employees or contexts. The goal isn’t a simple yes-or-no decision. It’s continuous learning.
And once you can measure properly, you do not need to scale blindly.
Experiment to Learn
Experimentation doesn’t stop once you’ve decided AI is worth pursuing. In many ways, that’s where the most valuable learning begins.
Rather than asking whether an AI capability is simply “good” or “bad,” experimentation helps organisations discover which implementation works best. Different prompts, agents, workflows or customer experiences can be tested against one another. Different audiences can receive different approaches. New ideas can be refined and improved over time.
Progressive rollout is one way experimentation supports this process. Introducing an AI capability to 5%, 10%, or 25% of users allows organisations to learn with confidence before committing to a wider deployment. But the real value isn’t simply reducing risk. It’s creating an environment where AI capabilities can be continuously improved before—and even after—they are scaled.
AI has dramatically reduced the cost of creating potential solutions. Experimentation helps organisations discover which ones deserve to be scaled.
The Real Advantage
A problem-first mindset doesn’t guarantee an AI solution. Sometimes, better research reveals that customers simply can’t find the information they need. Sometimes, a confusing journey, an unnecessary approval step or poor content is the real issue. Starting with the problem means AI becomes one possible solution, not the default one.
That doesn’t make AI any less valuable. It is one of the most significant technologies we’ve seen in decades, and it deserves the enthusiasm.
But the organisations that create the greatest value won’t be those that build the most agents or deploy the latest models. They’ll be the ones that consistently identify the right problems, test multiple solutions, measure their impact and scale what delivers meaningful business value.
AI has dramatically reduced the cost of creating potential solutions. It hasn’t reduced the cost of choosing the wrong ones.
That’s why the organisations that succeed with AI won’t be those that build the most. They’ll be the ones that consistently understand better, experiment better and scale better.
Which is why understanding still comes first.
