This document aims to help Korean users learn Framer by addressing the shortage of Korean-language resources. We've translated content from the official blog and added practical information. We hope this offers some help to Framer users.

Introduction to Framer A/B Testing

Framer includes built-in A/B testing functionality. It is a tool designed to allow designers and marketers to set up and analyze tests in just a few minutes.
Refer to this guide for easily designing A/B tests and using the results to create a better website.

Advantages of Framer A/B Testing

• No-Code Setup: Create tests with just a few clicks.

• Fast Real-Time Data: Collect and share results instantly.

• No Performance Impact: No delay or layout shifts in page loading.

• Multilingual Page Functionality: Apply equally across various regional site versions.

When to Run A/B Tests

A/B tests are most effective on high-traffic pages. The more visitors, the quicker you can obtain significant results.
It's particularly suitable for testing new ideas on pages designed to drive conversions, such as those with clear conversion events like link clicks or form submissions.

Setting Up A/B Testing

1. Creating a New Test

A/B tests always start within a page, requiring one page and one conversion event to get started. Conversion events can include link clicks, form submissions, or specific page views.

1. Right-click on the page to be tested

2. Select “New A/B Test”

3. Completion is marked by the creation of a new page variant

You can create up to 5 variant pages per page and rename them for easy identification later when viewing results. Feel free to change the design, but it is crucial to use the same tracking ID across all variants to optimize for specific calls to action (CTAs).

Note: CMS pages are currently not supported, so please take this into account.

2. Configuring the Test

Once you've made your changes, it's time to configure the test.

1. Click the “Configure” button in the top banner

2. Verify the message prompting you to select conversion steps in the Analytics tab

3. Click “+” under “Steps” in the right panel

4. Select the type of event to track

Note: You can add multiple steps, but the last step is used to calculate conversion rates and measure results.

3. Starting the Test

After configuring, click the “Start” button and publish the site. If using a staging environment, note that you need to deploy the site. Framer will then start recording page views and events for the test, and you can review the results in the analysis at any time. If there are two variants, visitors are randomly assigned 50% to each.

4. Analyzing Results

Once there are enough visitors, results will start appearing immediately. However, it's best to wait for substantial data to accumulate to ensure reliable conclusions. Framer's result summary shows key metrics and the time left until a winner is declared.

In the result analysis screen, you can review the following key metrics:

• Conversion Rate (CR)
  This is the ratio of unique visitors who completed the conversion steps to total unique visitors, reflecting the percentage of converted visitors.
  
  

• Lift
  This represents the increase/decrease in conversion rate compared to the control group. If positive, it indicates that the variant is likely performing better than the control.
  
  

• Best (Probability of Being Best)
  The “Probability of Being Best” percentage indicates the likelihood of the variant being the best. If this value exceeds 90% and sufficient data is collected, Framer declares it as the winner.

Once enough data is collected, Framer will display graphs. These graphs allow you to compare each variant's performance over time. The bands around the results line represent confidence intervals; narrower bands indicate higher confidence in the results. If the lines on the graph are clearly separated, you can confidently determine the final outcome.

5. Concluding the A/B Test

When the test results are clear enough, review and decide. If there is a substantial difference in results, you can safely stop the test by clicking the “Stop Test” button.

After the test ends, you will be prompted to replace the control version with one of the other variants. If a particular variant significantly outperforms the control, select that variant as the new control. Once you republish the site, all visitors will see the top-performing page, and you may freely delete other variants.

Sometimes, even with sufficient data, there may not be a definitive best-performing page. In such cases, you can still replace the control, though such a replacement may not significantly affect the analysis.

Frequently Asked Questions (FAQ)

Q. Does Framer's A/B testing use cookies?
Framer's A/B testing operates without cookies. Each visitor is assigned an anonymous identifier based on their IP address and browser information, allowing them to view the same page throughout the day. This anonymity resets daily and cannot be used for tracking users across different days/devices.
The benefit of not relying on cookies increases reliability, though there's a limitation of tracking conversions only on a daily basis. While Framer uses a standardized advanced A/B testing tool, plans to add a cookie-based system for more long-term and flexible conversion tools in the future are in place.

Q. Can I modify variants during a test?
Modifications are discouraged as they can affect results. If changes are desired, it is recommended to end the current test and start a new one.

Q. What statistical method does Framer use to determine a winner?
Framer employs a Bayesian approach. This method dynamically calculates the probability of one variant outperforming another as data accumulates. Unlike conventional methods that rely on fixed sample sizes or p-values and emphasize frequency, the Bayesian model calculates a performance-based expectation (PBB). A variant is considered a winner if its PBB probability exceeds 90%.

Q. What is the minimum number of visitors needed to declare a variant as the winner?
To declare a winner, the PBB of a variant must demonstrate a minimum 90% probability of being the best result and meet the internal criteria for visitor count. This ensures that the results are reliable and not based on a small sample size.

Q. How long should a test run?
There is no set duration. However, with low traffic or similar performance across variants, it might be difficult to predict final results due to insufficient test data. Such results will be indicated as uncertain, at which point you can choose to stop the test or continue running it for more insights.

Pricing Policy for A/B Testing

The A/B testing and funnel features are included in the Scale plan's Advanced Analytics add-on. Please refer to the information below about the add-on, and detailed pricing can be found on Framer's Pricing page.

• 14-day free trial available

• Billing is based on events (page views, clicks, form submissions)

• Usage is calculated monthly and included in the site plan billing cycle

This article is based on a translation and adaptation of Framer's official blog post ‘How to run an A/B test on your Framer site’.