The Pandemic Impact – 3 factors to consider for your CRO programme

Sadie Neve

The arrival of the Coronavirus has caused the world to turn into quite a surreal place. 

Since Coronavirus first hurtled into our headlines, it has had a profound disruption on global economies, the environment, and to political decision making. But perhaps one of the most dramatic disruptions is the unprecedented shift to our normal routines and habits, with everyone adapting to a new way of living and reliance on technology being bigger than ever.

With more than a third of the global population in “lockdown”, people are increasingly turning to shopping online, meaning a business’s website is now at the forefront of the shopping experience.  Obviously, and for the best, all ‘non-essential’ businesses within the UK should have closed their physical places of work in turn for the health and safety of their employees. However for those businesses who are still going, and more importantly, are running an experimentation programme alongside their digital efforts, the question arises on whether testing should be paused in this current climate or continued?

We personally at CCX have been having this internal debate on whether to launch experiments with our clients during this time. Every client is different, so we don’t think a one size fits all approach is appropriate. In order to help each client make this important decision, we’ve been thinking about the following three factors:

Shift in user behaviour

It is safe to say that some industries have been more severely impacted by COVID-19 than others.

The first thing to recognise is whether there has been a shift in your business KPIs during this time. For example, if we take the Travel Industry, which has been massively impacted, a previous focus for them may have been the number of bookings and their average order value. Now it may turn to increasing the ability of users self-serving online and reducing the amount of calls to customer services.

It may be that your tests are associated with simpler success metrics, looking at sharpening up other areas of the site services, copy optimisation, or improving UX functionality in areas that often get neglected from the core conversion funnel. Somewhat like a spring clean for your website!

Confirming what your KPIs or desired goal is during COVD-19 would allow you to determine whether experimentation can be of value. It may be during this time, that the challenges you are trying to overcome outweigh the desired goals reached through experimentation.

Representative population

When running an optimisation program, representative samples and generalisability are two key factors that increase our confidence in the data.

Known threats to the generalisability of A/B testing data are typically categorised into three factors:

  • Time-related factors – variability of user behaviour based on time
  • Population change factors – variability of user interest either during the test or after it
  • Novelty/learning factors – variability in behaviour of returning and loyal users.

For online businesses during COVID-19, the main threat is population change, with the majority of the population now having a higher intent to purchase goods online and also a greater sensitivity for instant gratification, especially as we inevitably get more bored of Netflix or HouseParty.

It is therefore likely that for some businesses there is a surge in conversion rate. For example, the online food delivery segment is one sector that stands to gain, already being up 11.5% in revenue in the year to date versus the comparable period of last year.

Any tests that are running during this current climate are going to be impacted by population changes, meaning that they are not likely to perform in the same manner pre-COVID-19. This is mainly due to extraneous variables. Therefore, it is important to understand the magnitude of this before deciding whether to continue with your experimentation programme or not.

Internal confidence of data

Trusting the data that your experimentation produces is vital to having a strong culture of experimentation. It is likely that if you have achieved this throughout your business, it didn’t happen overnight. Therefore why risk it overnight by testing in COVID-19?

It is important to understand the internal view of running experiments and understand whether results and data would even be accepted.

One route you take may be having a more defined test scope that tests in a simpler, controlled environment. This could mean segmenting to users or a behaviour that seems unchanged from the pandemic to now. It will be even more important to gain internal agreement on test metrics beforehand and determine a clear acceptance criteria and expectations. This should help overcome any internal challenges that you may face.

Remember there is always the option of re-running experiments post-COVID-19 to validate your results uplift accuracy, resources permitting. 

Hopefully discussing the above factors internally will allow you to reach a decision on whether you should or shouldn’t be experimenting during this time.

Don’t kill off your CRO programme

We believe that this is the perfect time to see how adaptable your experimentation programme actually is, with the need to pivot quickly to a new customer base and mindset. With online sales as the lifeline for some business’s survival, your website needs to adapt even for the short-term to meet visitors’ needs. This may mean shifting your strategy to be more tactical as you focus on frictional and functional experiments such as quick fixes, message alerts, or campaign promotions. We believe there is a place for experimentation in some shape or form to support the business and respond to our new norm.

However, if experimenting during this time isn’t the right thing for your business then please don’t neglect your programme! In the long-term it will benefit you from putting your time and effort into front loading experiments, so when the time is right you can be prepared to capitalise on lost time. This may also be the time to focus efforts on research and training. Ensuring you have up-to-date research or running question-led research could help to gather more insights that could fuel your experimentation. There may also be some new industry trends or UX practises that you have missed out on! If you are relatively new to experimentation or have a low maturity within the business, this may also be the perfect time to run training sessions to ensure that everyone within the team has the same foundation of knowledge. If everyone understands the importance and premise around experimentation, this will help when overcoming internal challenges to restart your programme in the future.


Whatever your decision, we understand at CCX that it may be a really difficult time for some people and businesses at this time. Therefore if you have any questions around this blog or in general, please do not hesitate to reach out to us via our website or on LinkedIn.


Written by, Sadie Neve 

Sadie is the Conversion Rate Optimisation Manager at CreativeCX. She has an academic background in Psychology and a keen interest in uncovering user pain points and motivations through quantitative and qualitative research. 


Let's talk!


61 Queens Street,
London, England, EC4R 1AE

+44 (0)20 7682 2820