Why A/B tests flicker, and what it costs you

If you have ever run an A/B test on a storefront, you have probably seen it: the page loads, the original headline appears for a heartbeat, and then it snaps into the variant. Shoppers notice. The industry calls it flicker, and every testing tool on the market has been accused of it at some point.

Here is the uncomfortable truth: flicker is not a bug that good engineering simply removes. It is a tradeoff written into how browsers work, and every tool faces it. What separates tools is whether they are honest about the tradeoff and how much they charge you for each side of it.

Where the flash comes from

A browser's job is to show your page as fast as possible. A testing script's job is to change that page before the shopper sees it. Those two jobs race each other on every page view.

If the script runs after the browser paints, the original content shows first and then changes. That is the flicker. If the script runs before the browser paints, the shopper only ever sees the variant, but the page waited for the script, which means the script sat on the rendering path.

There is no third option. Anyone who promises zero flicker and zero rendering cost at the same time is hiding one of the two, usually behind an anti-flicker snippet that quietly hides your entire page while their script loads.

The hidden-page trick, and its bill

The common industry answer is exactly that: a snippet that sets your page to invisible, waits for the testing script, and reveals the page when the script is ready. It works, in the sense that nobody sees the original content. It also means your page is blank for as long as the script takes, on every view, for every visitor, including the ones who are not in any experiment.

The bill for that arrives in two places. Shoppers on slow connections stare at a blank page. And your performance metrics absorb the delay, which is why testing tools have a reputation for slowing stores down.

How to reason about it instead

The honest framing is per experiment, not per tool. Ask one question: does this experiment change something the shopper sees at first paint?

If yes, above the fold, the flash of the original is exactly what the experiment must not show, and paying a small, bounded cost on the rendering path is the right call. Bounded is the key word: there should be a hard deadline after which the page shows itself no matter what, so a slow network can never hold your store hostage.

If no, below the fold or on a later page of the journey, then the rendering path should not pay anything at all. Run the script fully async and let the page paint at full speed.

What we do about it

Proa's Runtime script makes this exact tradeoff explicit instead of hiding it. One mode applies changes before paint, with a hard engine-enforced deadline so the page always reveals itself. The other stays entirely off the rendering path. Switching is one attribute on the tag, and we will tell you which mode fits what you are testing, because the answer genuinely depends on it.

You can read how the whole mechanism works on the Runtime script page. And if a tool ever tells you flicker is simply solved, ask them what their anti-flicker snippet does while it waits.