A/B/n Testing

No Comply's A/B/n testing system enables you to run controlled experiments by showing different campaign variants to different segments of your audience. This helps optimize campaign performance through data-driven decisions.

Index

Overview

A/B/n testing distributes traffic across multiple campaign variants that share the same audience, tracking performance metrics so you can identify the most effective messaging, creative, or targeting approach.

Core Concepts

Understanding the distinction between audience limiting and variant weighting is essential for effective A/B/n testing.

Audience Limiting

Audience limiting controls what percentage of your total audience is eligible to see any variant of the campaign. This is configured using the audienceLimit property.

Variant Weighting

Variant weighting determines how traffic is distributed among competing campaign variants. This is configured using the variantWeight property and operates independently from audience limiting.

Benefits:

Precise control over test exposure and traffic distribution
Clear separation between reach and variant allocation
Flexible scaling for any test size or complexity

Campaign Selection Logic

When multiple campaigns match the same trigger conditions, the system uses a two-stage selection process:

Stage 1: Specificity-First Selection

Campaigns are grouped by the number of trigger conditions. Campaigns with more triggers automatically take priority—no A/B testing occurs between campaigns of different specificity.

Example:

Campaign A: utm_campaign=summer + utm_source=email (2 triggers)
Campaign B: utm_campaign=summer (1 trigger)
Result: Campaign A automatically wins

Stage 2: Proportional Weighting

When multiple campaigns have identical trigger specificity, A/B testing occurs. Traffic is distributed based on variantWeight values, and users maintain consistent variant assignments across sessions using strata values.

Example:

Campaign A: utm_campaign=summer (1 trigger, weight 40)
Campaign B: utm_campaign=summer (1 trigger, weight 60)
Result: A/B test with 40/60 distribution

Campaign Configuration

To set up an A/B/n test, create multiple campaigns that share the same trigger conditions but differ in their content, targeting, or goals.

Basic A/B Test Setup

Create two campaigns that compete for the same audience with identical trigger specificity:

Create Campaign A (Control)
- Set up your standard campaign with existing messaging or creative
- Configure identical trigger conditions (e.g., utm_campaign=summer-promo)
- Important: Ensure the same number of trigger conditions as Campaign B
- Set audience limit to your desired test exposure (e.g., 80%)
- Set variant weight to 50 for equal traffic distribution (default)
- Define your conversion goal (e.g., "Summer Promo Control Purchase")
Create Campaign B (Variant)
- Use the same trigger conditions as Campaign A
- Critical: Match the exact number of triggers to enable A/B testing
- Apply identical audience limit (80%) for fair comparison
- Set variant weight to 50 for equal traffic split (default)
- Create your alternative messaging, creative, or targeting approach
- Define a corresponding conversion goal (e.g., "Summer Promo Variant Purchase")

This setup ensures that 80% of your audience is eligible to see the test, with traffic split evenly (50/50) between the control and variant experiences. Both campaigns must have the same number of triggers or the system will automatically prioritize the more specific one instead of running an A/B test.

Multi-Variant Test Setup

For testing multiple variants simultaneously (A/B/n testing), create three or more competing campaigns with identical trigger specificity:

Create Control Campaign
- Set up your baseline campaign with current messaging
- Configure trigger conditions for your test scenario
- Set audience limit to your desired exposure (e.g., 90%)
- Set variant weight to 25 for equal four-way distribution
- Define control group conversion goal
Create Variant A Campaign
- Use identical trigger conditions as the control
- Ensure same number of trigger conditions for A/B/n testing
- Apply same audience limit (90%) for consistent exposure
- Set variant weight to 25 for equal traffic allocation
- Implement your first alternative approach
- Define variant-specific conversion goal
Create Variant B Campaign
- Match trigger conditions and audience limit with other variants
- Maintain equal trigger specificity across all variants
- Set variant weight to 25 to maintain equal distribution
- Test your second alternative messaging or creative
- Configure appropriate conversion tracking
Create Variant C Campaign
- Ensure consistent trigger conditions and audience settings
- Keep identical number of triggers for fair A/B/n testing
- Set variant weight to 25 to complete the four-way split
- Implement your third alternative approach
- Set up variant-specific goal tracking

This configuration exposes 90% of your audience to the test, with traffic distributed equally (25% each) across four different experiences. You can adjust variant weights to allocate more traffic to promising variants or reduce exposure to underperforming options. All campaigns must have the same trigger specificity to ensure A/B/n testing occurs rather than automatic priority selection.

Implementation

A/B/n tests work automatically once configured. No additional code changes are required beyond the standard No Comply implementation.

Standard Implementation

<!-- Load No Comply script -->
<script
  type="module"
  src="https://content.nocomp.ly/GROUP_ID/routes.js">
</script>

<!-- Add campaign component -->
<campaign-group></campaign-group>

User Assignment

Users are automatically assigned to test variants using persistent strata values (0-99), delivering consistent experiences, randomized distribution, and reliable results over time. Assignments persist across browser sessions via local storage, preventing users from switching between variants and contaminating results.

Test Requirements

Tests activate when:

Multiple campaigns share identical trigger conditions and equal specificity
User meets audience limiting criteria for at least one variant
Campaign scheduling constraints are satisfied
No previous persistent selection exists for the user

Important: If campaigns have different numbers of trigger conditions, the more specific campaign automatically wins without A/B testing.

Analytics & Tracking

No Comply automatically tracks comprehensive A/B/n test metrics through the platform's analytics pipeline.

Tracked Events

Test Participation: Variant assignments with competing campaign details
Campaign Impressions: Variant display events with campaign context
Campaign Evaluations: Campaign eligibility decisions and rationale
Goal Achievements: Conversions attributed to the active variant
Selection Rationale: Selection mode captured as strata, priority, or single_match

Key Metrics

Conversion rate and participation by variant
Audience limit utilization and selection distribution
Statistical significance indicators
Revenue attribution and user engagement patterns

Selection Rationale Tracking

Selection reasons are captured as:

strata: Variant selected through A/B weighting using the user strata value
priority: More specific campaign won outside of A/B testing
single_match: Only one campaign met the trigger conditions

Attribution Types

The system supports three attribution methods:

Last Touch: Goals attributed to most recently shown campaign
No Attribution: Goals not linked to campaign activity
Direct: Goals from direct campaign interaction

Best Practices

Follow these guidelines to ensure effective A/B/n testing and reliable results.

Test Design

Define clear, measurable goals before starting tests
Ensure sample sizes and test duration support statistical significance
Test one variable at a time for clearer insights
Consider trigger specificity when designing competing campaigns

Configuration Guidelines

Use identical trigger conditions across competing variants for A/B testing
Keep trigger counts aligned to avoid automatic priority selection
Set consistent audience limits for fair comparison
Choose meaningful goal names and document test hypotheses and expected outcomes
Use descriptive campaign names for clearer analytics

Traffic Allocation

Start with equal weighting (50/50) for initial A/B tests
Adjust weights based on performance data and risk tolerance
Reserve higher weights for promising variants in subsequent tests
Consider control groups for baseline comparison
Remember that weights default to 50 if not specified and can use any scale (e.g., 1, 2, 3 → 16.7%, 33.3%, 50%)

Monitoring & Analysis

Monitor performance regularly but avoid premature conclusions
Validate outcomes with statistical significance testing
Account for external factors that might influence results
Document findings and apply learnings to future campaigns

This approach to A/B/n testing delivers data-driven optimization while keeping your campaign strategy flexible.