Highlights From Summer Experimentation Elite 2025
This June, the Creative CX team travelled to Birmingham to attend the annual Summer Experimentation Elite conference.
Taking place over two days, there was a host of round tables and talks from well-known industry names, as well as newer names taking the stage for the first time.
The CRO Talks Roundtables
Day one opened with a new addition to the Ex-Elite format: the CRO Talks Round Table sessions. Moderated by Iqbal Ali, Abi Hough, Matt Beischel, Els Aerts, Bhavik Patel, Natasha Senior, and our own Chris Gibbins these discussions gave attendees the chance to explore key topics voted on in advance.
The opening round focused on one of the most commonly debated questions in experimentation: how do you find and define problems to focus on for experiments and experiment programmes?
The two other topics Creative CX’s Chris Gibbins ran on his table were 11:30 -12:30: Quick tests vs Strategic Programmes: How do we balance them? And 12:15 – 12:55: What do you do to ensure the robustness and accuracy of experiments?
TLC Summit Talks
Alongside the roundtables, day one featured the TLC Summit talks, including a session delivered by our strategy consultant, Sadie Minors. We’ve shared a full write-up of Sadie’s talk here.
With the roundtables and TLC Summit complete, the second day focused on a full line-up of speaker sessions covering a wide range of experimentation topics.
Ruben de Boer – The Experimentation Mindset: How We Excel Websites but Overlook Our Organisation
Ruben opened the day by looking back at when he was setting up a culture of experimentation within a company with limited experimentation experience.
By applying his knowledge of experimentation to his own colleagues, Ruben was able to pinpoint where to focus his efforts by treating his colleagues like website users; taking time to understand their needs, track engagement, test communication approaches, and reflect on what worked.
Ruben demonstrated the idea of mapping both the formal and informal stakeholder hierarchy, identifying who is for, against, and ambivalent to experimentation to form the relevant approach.
He concluded by saying that shifting mindsets is a long game, but with curiosity, collaboration, and persistence, you can turn experimentation into a shared, company-wide mindset.
Juliana Jackson – Rethinking Marketing Analytics in 2025: Aligning Insights, Intent, and Impact
Talk 2 was the fantastic Juliana Jackson, taking to the stage for the first time at Experimentation Elite.
Juliana’s talk was a fantastic combination of marketing and data, calling attention to the fact that the modern consumer journey is more fragmented and harder to track than ever before.
With rising expectations, shrinking attention spans, and increasingly complex touchpoints, brands face a shifting landscape where traditional marketing tools fall short, leading to disconnected data.
Juliana explained that to stay relevant, brands must rethink both their data strategies and their approach to experience design.
With the first mention of AI of the day, Juliana suggested that it could help surface deeper, cross-channel insights when combined with unstructured, contextual data.
Combining user intent with landing page design often drives better experiences, so it’s important to consider the offline journey that leads users to your site.
She concluded by noting that brands that lead with helpful, intent-driven content, and consider monitoring people-first metrics, will drive better customer engagement, and ultimately reap the benefits.
Desiree van der Horst – One Slider, Many Opinions and the 6V Framework That Got Us Through It.
Next up was Desiree van der Host, talking us through her journey with the Loop homepage and how she applied the 6V Framework to navigate contrasting opinions.
The 6V framework focuses on structuring your data and research based on Value, Versus, View, Validated, Verified, and Voice.
Desiree showed us her Figma board of research she conducted based on the proposal of a homepage carousel, which came from the CEOs and didn’t necessarily fit with expected best practice.
This combination of data and insights led the team to a successful test, with a number of successful iterations.
Desiree’s talk highlighted that best practices don’t necessarily apply universally, and showed the importance of experimentation in determining the best experience for your own audience.
She also deftly showed how easy it is to set up and follow the 6V framework, and how invaluable it is to combine multiple data sources, as well as your own learnings.
Vignesh Lokanathan – The Invisible Step: Are We Optimising For Those Who Need It Most?
Closing out the morning we had the wonderful Vignesh Lokanathan, winner of the TLC summit last year.
Vignesh shared a very personal story regarding his son and the visible, and invisible, barriers he faces both in the physical and digital worlds.
Many of the obstacles aren’t even ‘invisible’, per se, but things many people wouldn’t notice until they became barriers, such as a small step being unnoticeable until you are navigating with a wheelchair.
I thought this talk was very impactful, with Vignesh highlighting how we’re breaking the promise the internet was built on; to make information accessible to everyone, and we can fix it by considering moral, legal and commercial perspectives.
By ensuring your website is compliant with WCAG guidelines and making an effort to listen to your customers, particularly those with disabilities, you can begin improving the experience for everyone.
Vignesh highlighted some interesting examples of where accommodations for disabilities also benefit able users – such as high contrast pages being easier to read in sunlight.
Vignesh’s message was clear – we have the power to make a difference, all it takes is a shift from having an awareness of accessibility to actively looking for opportunities to improve the experience for everyone.
Annette Rowson – Primark’s Digital Transformation: Test-Driven Customer Journeys
The first talk after lunch was the wonderful Annette Rowson, taking us through her journey to set-up an experimentation programme at Primark.
Annette’s journey spanned the last 3 years, beginning with first understanding the existing digital and data landscape at Primark – identifying a lack of relevant tools, development resources, general awareness of experimentation and data-driven culture.
Annette spent the next year addressing each of these in turn, onboarding new tools, proving the value of experimentation via low-impact, easy wins, aligning with product squads to unlock additional resources and setting up a number of data-led initiatives.
However, though she managed to set up the programme she found this resulted in a large number of inconclusive tests, with bad practices creeping in.
She has therefore spent the last year working on better aligning teams and optimising the experimentation process itself, shifting away from easy wins to high impact testing.
One of Annette’s biggest recommendations is the importance of patience, as well as the methodical approach she took to understand the current barriers before addressing them.
Nils Stotz – Experimentation With AI: No Hype, Just a Supernova!
Next, Nils Stotz took to the stage with a very different take on AI. Talking through the 8 different steps of experimentation, he highlighted the different ways AI could be used for each.
Nils identified 23 different tasks AI could assist with across these 8 steps, including automated scenario building, pattern recognition and insight summarisation.
Fortunately, Nils also addressed the common concerns with using AI, namely issues of trust, accuracy, over-reliance and complexity, and how to best address them – urging companies to focus on compliance, embracing AI and running internal AI-based sessions to reduce worries.
Katie Dove – The Psychology of Personalisation
The penultimate talk of the day was hosted by the insightful Katie Dove.
Katie’s talk was an insightful look at personalisation, taking it beyond the stereotypical simplicity of adjusting experiences based on user data, and instead focusing on making visitors feel seen and understood.
I thought this talk complemented Juliana’s talk really nicely, once more emphasising the importance of understanding and connecting with your audience.
Katie showed the effectiveness of utilising call-backs and asking your visitor questions to make them feel known and form a connection with your brand.
She also showed how sometimes the experience can be too good, such as a plumber only taking a minute to fix a leak, and instead it can sometimes be better to show perceived value by building in fake loading times, or else “displaying your workings” to show your visitors how and why they are receiving a certain experience.
David Mannheim – You Are What You Measure
The final talk of the day was from the undeniable David Mannheim, complete with beat poetry, a chair and a satsuma.
If you have heard of David before, it’ll come as no surprise that his talk also hinged on personalisation, but specifically how to best measure the impact of this, and optimise your testing programme going forward.
Once David had finished his first poem, it was interesting to hear his thoughts concerning choosing the best KPIs for your programme, warning that “you manage what you measure,” so make sure you are measuring the right parameters.
This came with a number of examples of perverse incentives, driving unfavourable behaviour both by customers and companies.
Indeed, he quoted Goodhart’s Law – “when a measure becomes a target, it ceases to be a good measure” – to emphasise the importance of carefully considering targets.
Before finishing with a final poem, he instead implored us to start considering the journey a user is on, rather than optimising based on pages, serving content relevant to the user’s state of mind rather than their position on the website.
Conclusion
We thought this was the best Experimentation Elite yet, with a great variety of complementary talks and perspectives.
Topics ranged from stakeholder management and organisational culture to AI insights, accessibility, and advanced personalisation techniques.
One of our main highlights from the conference was the amount of actionable guidance for refining all of our experimentation practices.
We’re already looking forward to seeing everyone again in London this December!
