Understanding How Amazon Does A/B Testing

When you hear the word "testing," you might be thinking of your lab setting. Finding data in a controlled environment is helpful. But how does this apply to making discoveries on an eCommerce platform? This is where Amazon A/B testing and Amazon Split testing come in. Do you know how to do Amazon ab testing?

eCommerce is not a controlled environment, so can you effectively run tests on this platform? The short answer is yes.

The key to making an informed decision is by using data-driven insights. To gather factual data, you must test a hypothesis thoroughly and confirm its statistical significance.

When trying to increase your sales rank as an Amazon seller, testing is no different. In this case, we refer to Amazon A/B testing, otherwise known Amazon as split testing.

How Amazon does AB testing

1. What is Amazon A/B Testing? (Definition)

Amazon ab Testing

Amazon Split Testing) is the process of testing the before and after of your product. It often refers to changing your product title, description, or images to see what results in a better click-through rate (CTR) and conversion rate (number of sales). Basically the product's landing page.

 Amazon's A/B testing platform is known as Manage Your Experiments (MYE). MYE enables you to see two different versions of content and how they perform based on three different types listing in Amazon Seller Central:

  • Product images
  • Product titles
  • A+ Content

At this time, MYE does not allow variants based on product descriptions. However, not having access to MYE does not mean you do not have access to A/B testing. We will revisit that a bit later. You can also check out how to create optimized, A+ content in your Amazon product listings here.

Who is Amazon A/B Testing (MYE) Available To?

MYE is available to members of the Amazon Brand Registry. To sign up for the Amazon Brand Registry, check out our detailed guide to Amazon Brand Registry for everything you need to know to get set up.

Amazon does not extend MYE testing to wholesalers, drop shippers and those who do not own brands.

MYE testing is also only available to sellers in the U.S., excluding those in other countries. We will be exploring alternatives to MYE testing in later sections.

What Products are Eligible for MYE Split Testing on Amazon?

Amazon a/b testing

MYE is limited to high-traffic listings that receive several dozen orders (or more) on their product listing. ASINs (Amazon Standard Identification Number) without significant traffic will not qualify for this service.

You need to choose a product with decent traffic. That way, you can make enough sales to justify entering the market. You can choose which product to select for Amazon ab testing by reviewing your Amazon listing and analyzing which target audience to focus on first. If you need help to understand how to improve your product listing optimization, you can use a data-driven listing optimization tool.

If you wish to run A+ Content Tests, the product listing must have published A+ content to be eligible.

How Does it Differ from Polling (Surveys)?

When running split tests against surveys, the most prominent difference comes from the recency of exposure. People aren't likely to give accurate answers on something that happened minutes or hours ago.

Polling is handy in some ways, but some biases get in the way. A/B testing has a lower chance of these biases, as you are running on whether or not you made a sale.

The product listing isn't based on what people think they want; it is based on what works. So instead of placing your marketing strategy on something up in the air, you can select the title, content, or images that work. To understand more about Amazon ad campaign bidding, check out this article.

How Much Time Does A/B Testing Take?

The length of A/B testing varies heavily depending on how long you want it to last. The testing process from MYE in total can be anywhere from four to ten weeks.

With short testing periods, your data becomes less accurate. This is because of changing environmental factors within that short time frame.

An extensive picture analysis enables you to see how your product description changes without relying on slow-selling periods. More extended testing periods are almost always better.

You'll also want to ensure that you account for the time it will take to compile the data. Important bullet points include the following:

  • Click-Through Rate (CTR) - This tracks the number of people who click on your product title
  • Conversion Rate - This area tracks the number of people who decide to buy your product
  • Bounce Rate - The number of people who left your product page without buying

2. Why is Amazon A/B Testing Important?

Amazon ab testing

 Amazon A/B Testing is important because it enables you to understand better how the user experience relates to sales. It allows you to know how product details translate into sales. 

Without the proper use of split-testing tools, you are limiting your potential sales. This can impact things like user engagement, so conversion optimization is essential.

What is Conversion Optimization?

Conversion optimization is the creation of content meant to encourage a specific buyer behavior. This can include following you on social media, buying your product, or signing up for your email newsletter.

Through A/B testing, you give yourself the most excellent chance to optimize your listing for your ultimate goal: Sales. Each optimization process needs different steps, so this requires implementing multiple strategies.

For example, increasing your social media presence might increase sales. However, the direct result isn't going to be increasing the number of sales you get.

Of course, you need to be sure that any data you get has statistical significance.

Understanding Statistical Significance on Amazon

DataHawk specializes in data insights, but not all data is helpful. It needs to be statistically significant. 

Statistical significance enables us to understand how valuable the data is. Statistically significant data have causation, meaning users can determine its usefulness by re-creating the scenario.

To determine statistical significance, these are some good things to keep an eye out for:

  • Does the data gathered have enough people involved to determine that it is valid?
  • How much variation does your data have when confirming your theory?
  • How much of a threshold for error does the data have?

You'll find that Amazon is aware of this, which is why products with small traffic aren't allowed to participate. To get the most effectiveness from MYE, you need to set up a plan for ten weeks.

Ideally, you'll also want to be sure one of the two experiments is already a confirmed success. If your current listing title already gets some results, keep it.

If you confirm two listings with no experience against each other, you aren't testing against proven data. Instead, you are trying two random elements you have almost no experience working with.

If you don't believe in A/B Testing, here is a quick roundup of all the benefits of testing it:

  • You can use it to test practical product description formats you can apply across the board on your product listings.
  • You can use it to use popular descriptors that enable you to see what your customers value most.
  • You can see results over a long period in an active environment, making it far more effective than offering surveys.
  • Most A/B testing uses proven and essential data through conversions. Data gathering is simple: fewer conversions means you likely need to change something.

We'll provide a few direct examples in our later section devoted to best practices.

But knowing that A/B testing existed before MYE, how does MYE change things? Below, we will dig into how the new platform improves Amazon's testing abilities.

3. How Amazon Experiments Changes the A/B Testing Game

How Amazon does ab testing

Amazon Experiments is part of the implementation of the MYE platform. It changes the game by simplifying the testing process.

The Amazon Experiments dashboard enables you to see differences with great ease. Its main benefits include the following:

  • The ability to easily set a time limit for testing
  • A dashboard that will contain data to inform you of clear winners
  • A breakdown of stats to understand where each option excels

Are There Any Drawbacks to Amazon Experiments?

Those using Amazon's experience appreciate the simplistic approach, but there is an apparent weakness in its use: The lack of price testing.

Much of this comes from Amazon preferring its users to compete for the lowest price. If you happen to be the best seller, you might not have to worry about this. However, Amazon almost always favors lower pricing.

 It also doesn't enable you to change the detailed product descriptions, severely limiting your chance of comparing infographics or lifestyle images in your later content.

While Amazon does provide powerful tools through the MYE system, there are some ways around it. We will address how to get around MYE's limitations a bit later. Our next section will delve into how to use the Amazon Experiments Tool.

4. How To Use Amazon A/B Testing Tool [Amazon Experiments]

Using Manage Your Experiments (MYE), your process is done in four steps:

  1. Starting the experiment
  2. Choosing your product
  3. Choosing the variant (Title / A+ Content / Images)
  4. Selecting the type of content

Once you fill out all four sections, it may take Amazon anywhere from a few days to a few weeks (depending on how far out you scheduled the content). Typically, the unavailable times come from Amazon's current workload.

Step One: Starting an Experiment

The first step to starting your experiment is going to your MYE dashboard. You can do so from your Seller Central dashboard by clicking the "Create a New Experiment" dropdown.

Before starting an experiment, you'll want to brainstorm some ideas about what works. This experiment process applies to one ASIN at a time, so you will want to be picky.

If this is your first experiment, do so with one of your less critical ASINs. Unless your highest selling ASIN is struggling, there is no need to test something that currently works.

Step Two: Choose Your ASIN

Next, the screen will ask you to choose an ASIN. If you have a high number of products, you might want to have this information available ahead of now. This information prevents you from needing to search.

If you choose an ASIN with variants (ex., a shirt with multiple colors), the system will ask you to decide which variants should be part of your testing. 

At this point, you will receive notice if your ASIN lacks the required traffic to qualify. If you do not have enough traffic, you will need to grow your audience. If you hold the most outstanding share of your competitors, the overall ASIN might be too small to test.

Because A+ Content does not apply to a single ASIN, it will not ask you to choose variants. Instead, it will gather the traffic among all ASINs where you use this content.

If you do not see your ASIN on this list, it means your traffic is too low to qualify.

Step Three: Add Experiment Details

Your next step will enable you to choose the details of your experiment. For now, the details are broken into three sections:

  • Experiment Name - This section is the name of the experiment that you choose. Customers will not see this title.
  • Hypothesis - This section details your believed result. For example, your hypothesis could be that Title A will result in more sales than Title B.
  • Duration and Start Date - It enables you to select the run time for tests (four to ten weeks) and the start date of your test.

Finishing all three sections will enable you to move to the next step.

Can I Choose To End My Experiment Early If One is Winning?

Amazon enables its users to end experiments early if they believe there is a clear winner. However, Amazon heavily discourages its users from doing so, leading to an incomplete picture.

The longer your test phase is, the better data you have. More data is never a bad thing, as you have the choice of whether to exclude it.

Step Four: Choose Your Type of Experiment and Submit It

Your chosen experimental content should be in this section. As discussed earlier, you can change one of three options:

  • Product Titles - This is arguably the most critical part of your product listing. You can choose between two target keywords in your title to see what works better.
  • A+ Content - A+ Content enables Amazon Brand Owners to place enhanced content through image, video, and text placements. Because this requires a different approval process, you might save time by choosing already approved content.
  • Images - It enables you to choose between multiple photos. Be aware that primary images still need to follow Amazon's guidelines and have an entirely white background.

All three content options must follow respective guidelines. So no high-pressure sales requests, inappropriate dialogue, or offensive content.

Once submitted, your experiment will go through a validation process and reach approval. Once your experiment is approved, it will start during your pre-selected start date.

Can You Edit an Amazon Experiment After Submitting It?

If your content is approved or the experiment is in progress, you cannot edit the content. Otherwise, scheduled content that has yet to be approved can be edited, but expect the review schedule to be delayed.

If you wish to edit the duration of the experiment, you can do so at any time. If you cancel the experiment, you will need to make a new submission.

Once you cancel the experiment, it will revert to your original content. 

5. How To Do A/B Testing Without Qualifying for Amazon Experiments

Amazon Ab testing

If you aren't a member of the Amazon Brand Registry, you've got another option: manual testing. Manual testing involves you personally changing listing details to see results.

Manual testing requires more attention to detail but can be just as helpful as Amazon's MYE tool. If you don't know where to start, we will provide you with what you need to do.

Even if you have access to Amazon Experiments, this process can still be helpful. After all, Amazon does not associate MYE with price testing.

Step One: Choose One Variable For Testing

First, you'll want to make sure the process starts with incredible simplicity. This process involves choosing a single independent variable.

The independent variable is a single, changeable factor of your product listing. When you change it, you should notice a measurable change in performance.

Using MYE as an example, let's say that you decide to flip the location of the brand and a feature in your title. For instance, selling lotion that starts with the word "soothing" instead of having it three words deep.

Your other (more likely) test will be to change the price. If you gain a higher profit margin from increasing your price by $1, it can be tempting. However, losing the buy box for that change is not ideal.

What If I Want To Test Multiple Variables?

Multivariate testing involves using multiple changes to see what is effective. In this case, you might see something like this:

  • Test #1: A title change
  • Test #2: A title change and a detailed description change
  • Test #3: A title change and a price change
  • Test #4: A title, price, and description change

The idea is to include multiple individual changes to see what works. You can also test these in isolated formats to see if those personal changes have profound impacts.

Multivariant changes are significant when you want to test out a complete brand overhaul. However, you might find that gathering the data is complicated.

It is challenging to tell the exact location where this data comes from when multiple changes occur. If this is your first test, stick to single-variable A/B testing.

Step Two: Choosing Your Hypothesis (Goal)

Your hypothesis is an assumption you are making as a result of your change. Another word for a hypothesis is "goal," referring to the desired outcome.

Another name for your goal is the dependent variable. In scientific testing, the dependent variable changes depending upon what changes in the independent variable.

If you do not believe something will improve your business, don't do it. Unlike a laboratory, your business shouldn't be a testing facility to see what works. Your business is created to generate profit.

In this case, your goals should be SMART, which is an acronym we will explain below:

What are SMART Goals, and How Do They Help My Amazon Analysis?

SMART Goals simplify your analysis by applying five different requirements:

  • Specific - Specific goals are those with a narrow purpose. Your plan to "grow business" is excellent, but how will you do that with this test?
  • Measurable - Your goal needs to be associated with numbers. It would be best if you had percentages and dollar amounts to know your success.
  • Achievable - Your goal needs to be possible, meaning that making $1 million in quarter four on flip-flop sales is probably not going to happen.
  • Relevant - Your goal needs to be related to the topic. If your plan desires increased sales, it needs to be specific to your product from the change point.
  • Time-Bound - Your measurable period needs to be specific (one month, one week, a year, etc.). Remember that more extended testing periods have better results.

If your goals lack all of these features, you are going to be guessing. A 10% increase in sales over one month due to your changes is a SMART goal because it is doable and specific. Saying that you want to make $1 million eventually is not feasible without smaller goals.

Without dealing with the small picture, you cannot hope to accomplish the big picture.

Step Three: Create an A/B Testing Calendar

An A/B Testing Calendar enables you to determine how long this experiment will last. Just as always, more extended periods of testing are preferred, so stick with Amazon's suggestion of 10 weeks (two months).

Be sure you have specific and planned days for your changes. That means you should account for one month for situation A and another month for situation B. Plan a week where you can gather the data.

If you use tools like Datahawk, we help by storing historical data for you. So you don't have to worry about tracking when it's already done for you.

The calendar fulfills the "time-bound" part of your SMART goal. If you exceed your time with project A, you will likely want to account for extra time with project B.

What if My A/B Testing Calendar Runs Into Major Events (Prime Day / Holidays)?

If your testing calendar runs into significant events, your data is compromised. In this case, information is skewed, and favor will likely go towards the product that spends more time during the event.

During the holiday season and Amazon Prime Day, your focus should be elsewhere. Testing phases are best done during regular seasons.

What if I Sell Seasonal Products?

The rules above still apply if you sell snowboards, surfboards, winter coats, or other seasonal items. You will need to account that people are more likely to buy your items during popular seasons.

Using historical data, you can determine how your product has generally behaved around this period. This information can enable you to make educated decisions.

If you know your product sales tend to increase 10% during winter, account for that in your threshold for error. Otherwise, try and make your testing phases stick to periods without transitions.

Step Four: Establish Your Confidence Level (Error Threshold)

In this crash course on statistics, a confidence level is essential when running any test. It determines your willingness for data to go outside of your threshold.

Ideally, this threshold will be low, but luck isn't always on our side. This is why it is crucial to spread your confidence interval.

The confidence interval (level) is typically at least 95%. This means that your error could be within 5% of your initial testing and still be accurate.

In the case where you fall outside five percent, you need to reassess your hypothesis. This retesting needs to occur whether you are above or below your testing number.

For example, assume you initially sold 10 thousand units and ended up selling 15 thousand when your goal was to sell 14 thousand.

If you have a five percent error threshold, you need to be within 700 units of 14 thousand (0.05 * 14000). This error threshold determines whether the data is statistically significant. In this case, you'll likely want to run further testing because your significance allowance was 5% (700) when you had 1000 units.

If your tests were within 500 units, you could act upon this data.

Why Do I Need To Retest If I Outdo My Original Figures?

Why wouldn't you accept the best possible outcome for your business? The reality is this: You always need to scrutinize your data.

If you sell more units than you expect, you need to account for other potential variables. Anything can impact your potential sales:

  • Check to see if you have more positive customer feedback. A higher seller rank in your tracking data will cause you to have to re-evaluate data.
  • If a product you sell suddenly becomes trending, the number of people who want to buy your item will increase.
  • If you are comparing your items between seasonal changes, your data is likely inaccurate.
  • If you improve your customer service, shipping speed, or other aspects of your business, you need to take another look at testing.

When it comes to Amazon, you can't put other parts of your business on the backburner for testing. However, you cannot proceed without accounting for unusual variables.

Step Five: Act On The Results of Your Data

What you do with your data is entirely based on the results of your hypothesis. As stated earlier, your hypothesis (or goal) is a specific determination of your results.

Using our earlier example, we said our units should grow by 10%. Let's assume our units went from 10 thousand to 11 thousand. To find the percentage change, we take 10 thousand and divide it by 11 thousand to get this:

10 / 11 = 0.91 (rounded up)

We take this minus one and multiply it by 100 to get our percentage value:

1 - 0.91 = 0.09 percent * 100 = 9%

Knowing our confidence level is 5%, we can safely assume that this data is accurate. The data can be no more than 15% and no less than 5%. Anything beyond this is data that needs to be retested with different figures. 

This information tells us your hypothesis/goal has been met. In scientific terms, your hypothesis is true. If the theory is not accurate, you must test an alternative approach.

In some cases, you can mistake the data, typically in type one or type two errors.

Statistical Type One and Type Two Errors - How They Apply to A/B Testing

In statistical reasoning, you have two types of errors: one and two:

  • Error Type One: Error Type One is a false positive, meaning that there was no significant change or detectable change. Regardless you still believe that there is a link.
  • Error Type Two: Error Type Two is the acceptance that your hypothesis was false. This means you did not detect a change when there was one.

Committing to error type one in marketing decisions might result in a long-term loss due to the less effective change. Selecting type two is when you do not take advantage of a chance you missed.

In cases where there is an unrelated variable, you might run into an irrelevant Type Three Error. Type three errors are when you reject the hypothesis for the wrong reason.

For example, your numbers are likely to drop if you receive a series of negative reviews during the testing process. This change isn't related to a change in your title, images, or price. Instead, it comes from another source.

What Should I Change If I Need to Retest?

Depending on your results, you can test a nearby figure or an entirely different variable change.

If you had an increase outside of your confidence level, you would want to retest your change. You do not wish to lose out on the potential growth that change can cause, but you don't want to establish this as your marketing strategy yet.

If you had no visible change or a drop in conversions, you'd want to try another variable. You don't want to go with something that fails.

You'll want to account for outside variables (seasonal changes, customer feedback, customer service experiences, etc.) in both cases. External sources have a terrible habit of making your data questionable.

How Do I Make These Changes in a Controlled Environment?

All good lab testing is done in a controlled environment. But Amazon is not a laboratory, and your competitors won't sit around for your test to get done.

So how can you ensure your data is incorruptible?

You don't.

You can do everything in your power to ensure you account for wild variables. But eCommerce is not a controlled environment.

With any marketing campaign, it is essential to run multiple tests and gather from experience. The best form of data is from experience, which is why our company specializes in providing historical data.

We will revisit how you can use our software to learn from historical data, shortening your A/B testing times by providing consistent data flow. First, we will address how Amazon Web Services (AWS) handles this.

6. Using The Amazon API for A/B Testing

Amazon API for AB testing

The Amazon API (Application Programming Interface) enables developers to use data from Amazon and automate the information-gathering process.

Amazon recommends two tools when using the API: Amazon Personalize and Amazon SageMaker.

What is Amazon Personalize?

Amazon Personalize is a metrics-tracking system a developer might use to gather data. The process is similar to what we mentioned above (research, create a goal, run experiments, measure results).

The main difference is that Amazon Personalize does this all automatically. This means if you can use this API, the entire process above turns into gathering data.

Amazon Personalize also enables you to plug variants at will. This way, you can quickly test alternative hypothesis options using the same information.

It also enables you to track the p-value.

What is the P-Value, and How Is It Important in Statistical Analysis?

P-Value refers to the chance you will see your hypothesis coming true. Ideally, you would want your p-value to be small, as that involves less fluctuation.

To accurately gain a p-value, you must run multiple tests. Many statistical models can run those tests automatically. Amazon Personalize can help you get closer to that level.

The problem with Amazon Personalize is that it specializes in making customer recommendations. If your ultimate goal is to run price or title variation tests, this tool is less useful.

For other machine learning tools, there is Amazon SageMaker.

What is Amazon SageMaker?

Amazon uses Amazon SageMaker to manage Alexa, its voice-activated system. Machine Learning (ML) enables Amazon, Alexa, and some partners to stay on the cutting edge of business needs.

Amazon SageMaker is less applicable to new sellers. Instead, they are best for established brands with strong off-traffic segments.

Using SageMaker studio, your developer team could feasibly run multiple tests with many variations. Given that it is built on a machine learning platform, SageMaker can address variants and respond accordingly.

Data is sent directly to where they need to be through the Amazon API. SageMaker is also solid for testing by customer segments, which can be incredibly useful for larger companies.

7. Using DataHawk in Manual Split Testing

Amazon Manual split testing

DataHawk's demo software gives you a good idea of using this, so we will be referencing it often in this short guide. DataHawk uses powerful data sets to help you optimize your business.

Here is how to get started:

Step One: Gather Your Historical Data

Under the product dashboard, you'll see a date range you can select in the upper-right-hand corner. This date range enables you to go as far back as needed.

You can insert custom ranges that go back several years, but the default methods involve anywhere from the past week to the past twelve months. It's essential to select a date range to match your testing phase, which should ideally be done in a month.

In this way, DataHawk is constantly performing the "A" part of your A/B testing. It is continually tracking current factors in your product.

If you want to select a steady time from a few weeks back, you can do so. Just be careful not to go to times where the data is no longer relevant (when you had less seller feedback).

Step Two: Set a Calendar Date for Your Listing Change

Whether you change the price, product description, title, or images, it's essential to select a specific date range for your change. This date range should be the same length as your historical data, going back several months if possible.

If you don't select a specific date range, your data is not time-bound. Therefore, it is less valuable than someone who has a distinct and measurable goal.

You'll want to set notifications on buy box changes, rank changes, and new competitors entering the niche from this period. Keeping other aspects consistent is one way you can keep the data handy.

What If I Lose The Buy Box During The Testing Phase?

Recall our earlier suggestion of not using an ASIN that is exceptionally important. In this case, you'll want to consider the idea of changing things back.

However, buy box changes are not solely based on product descriptions and titles. You might see if other factors are affecting this (customer experience, bad reviews, etc.).

When testing, it is natural to want to jump the gun and return to comfort. However, the data you gather from A/B testing is essential in learning how your product works.

Step Three: Note The Change Explicitally In Your Historical Data

Because this is part of your testing phase, you must know when it starts and ends. This data is something beneficial, providing you with a written record of these changes.

The Reports & Automation section enables you to create reports under specific periods. You can save them in a folder and associate them to a document explaining the test by exporting from these times.

The document should confirm or deny your theory (hypothesis), ensuring future business partners do not make the same mistake. Testing changes to your product involves the confirmation of positive results and the elimination of adverse effects.

8. Best Practices for Amazon A/B Testing

Now that we have a few options to consider with A/B testing, we can go through the best practices. While the testing environment might change, the basics of testing typically do not.

Here is what you should keep an eye out for:

Best Practice #1: Target A Specific Audience

When creating A/B testing results, you must know your customer. Being a member of the Amazon Brand Registry gives you access to Amazon Brand Analytics, a treasure trove of data.

This data can enable you to base your changes on logic. For example, you might not market your product to 50-year-old women like you would sell to 20-year-old men.

It would be best if you made all business decisions with some logic. Typically, some of the research has been done for you. So before making changes, make sure you keep your audience's expectations in mind.

You can start by asking them about their most important priorities. Using data gathered from a survey may help you make the first step.

Best Practice #2: Make Your Testing Scalable

Scrolling back up, you might recall a section where we mentioned that changes on one product could apply to other products. Ideally, you'll want that to apply to almost all of your testing.

For example, testing a new format for your Amazon title where you put a product descriptor in front is one option. Face lotion starting with the word "soothing" might attract those who had a bad experience with face lotion in the past.

The example above is very straightforward, but it does state the point. If you can, make sure your testing can be applied to multiple products.

If it works, you can expand this into multiple tests that extend on a product-by-product basis. Eventually, you could grow to your entire product category.

Best Practice #3: Stick To One Variable

Unless you are a data scientist, chances are you'll find it challenging to test multivariance. Multivariance testing involves the application of multiple independent variables.

Because changing multiple things at once has the potential for numerous impacts makes it hard to find what affects the change. Adding a higher-quality image next to what you perceive to be a more engaging description adds confusion.

Always start with small changes as you will. While some more apparent changes don't require testing (like a higher quality picture), many changes do.

When adding multiple variables, it's best to leave those to third-party A/B testing tools. Those who specialize in this practice are better suited to working with it.

Best Practice #4: Stick To A Variable That Tracks Performance

Our early example, conversion rate optimization, refers to several different things. To convert means to convince someone to subscribe to your product or idea.

Conversion includes the following:

  • Making a sale
  • Having someone subscribe to your newsletter
  • Gaining a follower on social media
  • Having someone download an eBook from your email marketing campaign

While each of these is meant to make a sale eventually, they are all considered performance metrics. So when choosing a variable, make sure it applies to one of your Key Performance indicators (KPIs).

KPIs are the ultimate example of SMART goals, enabling the user to link activities directly to success. How you define success depends on the ultimate goal of your hypothesis.

Best Practice #5: Be Picky With Your Data

When it comes to finding data, any data will not do. Instead, you want data that works. In the case of split testing, this data must fall within your confidence levels.

Recall that a confidence level refers to the error allowance you have. If the data falls outside your error allowance, you need to either throw it out or test it again.

There are many reasons you should retest your data. Here are a few examples:

  • The number of sales you made during a period was too high (above your confidence level)
  • The number of sales you made was too low (below confidence level)
  • There was a freak incident that caused your product listing to drop on the Amazon ranking chart
  • You had some staff changes and logistical issues preventing you from performing adequately

When selecting your data, you should do it during a period of semi-normalcy. There will never be a period where your company is 100% not crazy, but there needs to be consistency between periods for data to be valid.

9. A/B Testing FAQs

Are Rules for Experimental Content Different From Standard Listings? 

No, the general guidelines for standard listings still apply when testing on experimental content. To follow Amazon's policies, avoid high-pressure sales content, offensive content, and content that infringes on copyright.

The rules for A+ Content also apply here, so review the Amazon Product Detail Page Rules for more information.

How Often Do You See Results from Manage Your Experiments?

MYE calculates results every week, but it sometimes takes up to two weeks if the data is slow. Results automatically show with updates, so be sure to check it regularly.

Can You Run Experiments With Multiple ASINs?

No. Amazon does not allow you to create one experiment under multiple products. If you receive an error, it usually comes from A+ Content testing.

Double-check that your selected page has the A+ Content. If it doesn't, be sure to add the content before proceeding.

10. Conclusion

So, is Amazon A/B testing worth it? The answer is yes. There is no question that this testing method is an excellent way to find out what works and what does not work on Amazon.

Even if you don't have access to Amazon Experiments, Amazon already has an excellent platform where you can run tests. Utilizing secondary products and ASINs, you won't have to sacrifice your hot items to see what works.

You can gain several benefits from A/B testing, but data is the most crucial part of this. Check out DataHawk's Amazon Analytics software to see how this data can be helpful for you. Utilizing our data-driven process for your Amazon business is one way to stay ahead of the competition.

Read also

September 17, 2021 Use Case
Measure Selling Performance Efficiency Post Ad Spend With DataHawk

Cracking correlations between your Ad Spends and Ad Sales and indirect...

February 10, 2021 Educational
Understanding Amazon API For Advertisers and Sellers 

The Amazon Product Advertising API is an exclusive tool, only availabl...

October 27, 2020 Educational
How to Reduce Wasteful Ad Spend on Amazon?

A carefully planned strategy for optimizing your Amazon ads can put yo...

Get the latest eCommerce and Amazon insights and trends delivered straight to your inbox


IE Notice
Please note that DataHawk no longer supports Internet Explorer.

We recommend upgrading to the latest Microsoft Edge, Google Chrome, or Firefox.