Let’s address the room’s elephant — email pop-ups are TRICKY.

They can incentivize that oh-so-important first purchase and help you build your oh-so-important mailing list, but they are, by nature, pop-ups.

Get them wrong and your bounce rate will go sky high 🎈

So let’s talk testing, optimization and safeguarding your site from bad implementation.

Conversion optimization is both art and science. A part of the art is your strategic thought behind your plan. It’s imperative that you don’t test randomly, but take a structured, top-down approach that prioritizes tests based on a pop-up’s degree of intrusiveness.

The following is a sample playbook you can use to plan your email pop-up A/B tests in a way that tackles the biggest elephants first. I recommend you test specific variables, in this order, before moving on to more trivial design elements.

Order-of-operations, ranked by element conversion influence Presentation style > Trigger timing > Headline & microcopy > CTA style > Offer > Everything else

This is just one approach! It’s certainly fine to test “radical redesigns” head-to-head, but this is a systematic style of testing (and I still recommend nailing stages 1 and 2 before layout/design testing).

Start with a Minimum Viable Design

This means bare bones layout, copy, offer and form that follow design conventions you expect from most pop-ups. Choose one offer to use consistently from stages 1-4 to ensure you’re perfectly isolating your variables.

In general, less is more when it comes to copy and clutter. If you must, for legal or operations reasons, include privacy statements, terms and conditions, segmentation options, etc. include them in all your variants consistently until you get past stage 5. You may do 2 or more “rounds” of testing within each stage to ensure you have a very strong “Control” version to move to the next stage. Take your time. Optimization is a marathon, not a sprint!

Testing stage 1: Presentation style

Most pop-up servers enable you display your offers as box modals, full page takeovers, “floating box” (sliding left or right) and “floating bar” (popping bottom/center).

The objective for this first round is to determine which style works best with your audience.

You’ll set up a single A/B/c test with 3 variants*: 🧪 Control: Standard modal

🧪 Variant A: Floating box (right-aligned, to match typical eye tracking pattern)

🧪 Variant B: Floating bar

I recommend to avoid full page takeovers and focus on these less-intrusive patterns.

Make sure each variant has the same general design attributes, trigger timing (e.g. Immediately) and settings. Controlling these variables ensures you get a true head-to-head comparison between these options.

Make sure to measure bounce rate, time on site, revenue per visitor and conversion rate, not just % of form submits.

Next, you’ll follow up with an A/B round testing the winning pattern against no pop-up at all.

* Your pop-up builder may only support A/B and not A/B/x in a single round. If so, keep testing rounds until you find your conclusive winner. If your testing and pop-up tools can support it, include “no pop-up” as a variant in your initial testing round. Otherwise, use the A/B testing feature in your pop-up app for round 1 and your website testing tool for round 2).

So long as you feel confident your pop-up is not harming other core metrics, proceed to the next stages.

Testing stage 2: Trigger timing

Trigger timing is so important to get right because of it’s impact on user experience. You want to ensure you’re serving it right.

Typical settings enable you to set timing to:

⏰ Immediately

⏰ After X seconds on page

⏰ After X% scroll

⏰ After X page views

⏰ After X time on site

🧪🧪🧪 Depending on your site traffic, you may be able to test all these conditions head-to-head in an A/B/e test with 5 variants to get the true apple-to-apple data within a single time frame and context.

🧪🧪 If you have medium to low traffic, stick with an A/B testing Immediate vs 5 seconds, or A/B/c with Immediate, 5s and 3s.

Why just seconds on page, what about other trigger options?

Remember, pop-ups were designed not just for ecommerce, but also B2B SaaS and content sites to encourage subscriptions. Scroll depth, time on site and page view triggers are less relevant to ecommerce. It’s arguably worse to display pop-ups to users with such signals of engagement.

Entry pop-ups on ecommerce sites are so common, it’s best to stick with conventions. Show it quickly or not at all to minimize intrusion.

Remember “don’t make me think”? Trigger it before they’ve had time to think ;-)

Testing stage 3: Headline & microcopy

Copy is king, so this is the stage where you challenge your control headline (and associated sub-line copy). Make sure you’re still featuring the same offer at this step. You’re testing for tone, length, etc.

And ensure you’re using the same font/styling for each variant so you’re not introducing any rogue variables (unless your traffic is sufficient to multivariate each set of permutations together).

Testing stage 4: CTA button style and label

As critical as CTA buttons are, nobody will click them without a compelling offer — hence the Headline stage comes first. But once you’ve perfected your copy, you need to make sure your button is converting to its potential.

Here’s where it’s tempting to mix variables, e.g. Control is black button with “Sign up” label against a pink “Get my offer.” You’ll see a larger spread between variants in your test results, but won’t be able to isolate which variable was more effective.

So keep them streamlined (or use A/B/x testing, providing sufficient traffic), and don’t be afraid to run a few rounds of this stage.

Testing variables: Color, Shape, Size, Label copy

CTA inspo

Testing stage 5: Offer

Now that you’ve gotten style elements out of the way, you can confidently test offers head-to-head (keep your winning variables for previous tests in the new creative).

If you’re not at liberty to test sweeter deals or alternatives, move ahead to stage 6!

Offer ideas

% off

Obviously your % discount is a business decision — you can’t just test 10% against 30% without following through, and your margins dictate your wiggle room. If you’re testing discounts, make sure to measure not just clicks but revenue and profit. You may find more overall contribution value in a 10% offer versus a 20%.

Instead of testing 10% against 20%, consider something ending in a “lumpy” digit, such as 15%. We scan offers so fast, the ‘5’ may be more attention grabbing than a ‘0’ - especially with the % sign and ‘O’ in “Off” are visually similar characters.

Dollar discount

These feel more like a gift card equivalent to cash, and may be more compelling than a vague percentage (though % is often be the better value, it’s about perception).

Free gift

Giveaway contest

Be mindful that this tactic carries legal implications that may fall under gaming laws and can, depending on your jurisdiction, require certain term disclosures and other caveats. Check with Legal!

Shipping offers

Mystery offer

The lack of clarity here is less persuasive than a clear, quantifiable discount. But it’s a trending tactic that may work for you. The mystery aspect may also increase your welcome email open rate.

Testing stage 6: Everything else

Congrats! Now that you’ve battle tested your table stakes, you’re free to test more radical redesigns, including variables like:

🧪 Modal shape and size

🧪 Imagery (see this post for photography tips)

🧪 Privacy policy visible vs no

🧪 Font size, colors and styles

🧪 Form field shape and styling

🧪 Email hint text

🧪 With or without logo/branding

🧪 Light on dark vs dark-on-light

🧪 Layout - image left, right, top aligned, background image and no image, etc

😎 Check out the Email pop-up database for more ideas and tips

👉 Follow on LinkedIn for daily ideas 💡