Understanding How Amazon Does A/B Testing

When you hear the word “testing,” you might be thinking of your lab setting. Finding data in a controlled environment is helpful. But how does this apply to making discoveries on an eCommerce platform? This is where Amazon A/B testing and Amazon Split testing come in. Do you know how to do Amazon ab testing?

eCommerce is not a controlled environment, so can you effectively run tests on this platform? The short answer is yes.

The key to making an informed decision is by using data-driven insights. To gather factual data, you must test a hypothesis thoroughly and confirm its statistical significance.

When trying to increase your sales rank as an Amazon seller, testing is no different. In this case, we refer to Amazon A/B testing, otherwise known Amazon as split testing.

Amazon Split Testing) is the process of testing the before and after of your product. It often refers to changing your product title, description, or images to see what results in a better click-through rate (CTR) and conversion rate (number of sales). Basically the product’s landing page.

Amazon’s A/B testing platform is known as Manage Your Experiments (MYE). MYE enables you to see two different versions of content and how they perform based on three different types listing in Amazon Seller Central:

• Product images
• Product titles
• A+ Content

At this time, MYE does not allow variants based on product descriptions. However, not having access to MYE does not mean you do not have access to A/B testing. We will revisit that a bit later. You can also check out how to create optimized, A+ content in your Amazon product listings here.

MYE is available to members of the Amazon Brand Registry. To sign up for the Amazon Brand Registry, check out our detailed guide to Amazon Brand Registry for everything you need to know to get set up.

Amazon does not extend MYE testing to wholesalers, drop shippers and those who do not own brands.

MYE testing is also only available to sellers in the U.S., excluding those in other countries. We will be exploring alternatives to MYE testing in later sections.

MYE is limited to high-traffic listings that receive several dozen orders (or more) on their product listing. ASINs (Amazon Standard Identification Number) without significant traffic will not qualify for this service.

You need to choose a product with decent traffic. That way, you can make enough sales to justify entering the market. You can choose which product to select for Amazon ab testing by reviewing your Amazon listing and analyzing which target audience to focus on first. If you need help to understand how to improve your product listing optimization, you can use a data-driven listing optimization tool.

If you wish to run A+ Content Tests, the product listing must have published A+ content to be eligible.

When running split tests against surveys, the most prominent difference comes from the recency of exposure. People aren’t likely to give accurate answers on something that happened minutes or hours ago.

Polling is handy in some ways, but some biases get in the way. A/B testing has a lower chance of these biases, as you are running on whether or not you made a sale.

The product listing isn’t based on what people think they want; it is based on what works. So instead of placing your marketing strategy on something up in the air, you can select the title, content, or images that work. To understand more about Amazon ad campaign bidding, check out this article.

Amazon A/B Testing is important because it enables you to understand better how the user experience relates to sales. It allows you to know how product details translate into sales.

Without the proper use of split-testing tools, you are limiting your potential sales. This can impact things like user engagement, so conversion optimization is essential.

Conversion optimization is the creation of content meant to encourage a specific buyer behavior. This can include following you on social media, buying your product, or signing up for your email newsletter.

Through A/B testing, you give yourself the most excellent chance to optimize your listing for your ultimate goal: Sales. Each optimization process needs different steps, so this requires implementing multiple strategies.

For example, increasing your social media presence might increase sales. However, the direct result isn’t going to be increasing the number of sales you get.

Of course, you need to be sure that any data you get has statistical significance.

Those using Amazon’s experience appreciate the simplistic approach, but there is an apparent weakness in its use: The lack of price testing.

Much of this comes from Amazon preferring its users to compete for the lowest price. If you happen to be the best seller, you might not have to worry about this. However, Amazon almost always favors lower pricing.

It also doesn’t enable you to change the detailed product descriptions, severely limiting your chance of comparing infographics or lifestyle images in your later content.

While Amazon does provide powerful tools through the MYE system, there are some ways around it. We will address how to get around MYE’s limitations a bit later. Our next section will delve into how to use the Amazon Experiments Tool.

Using Manage Your Experiments (MYE), your process is done in four steps:

1. Starting the experiment
2. Choosing your product
3. Choosing the variant (Title / A+ Content / Images)
4. Selecting the type of content

Once you fill out all four sections, it may take Amazon anywhere from a few days to a few weeks (depending on how far out you scheduled the content). Typically, the unavailable times come from Amazon’s current workload.

The first step to starting your experiment is going to your MYE dashboard. You can do so from your Seller Central dashboard by clicking the “Create a New Experiment” dropdown.

Before starting an experiment, you’ll want to brainstorm some ideas about what works. This experiment process applies to one ASIN at a time, so you will want to be picky.

If this is your first experiment, do so with one of your less critical ASINs. Unless your highest selling ASIN is struggling, there is no need to test something that currently works.

Next, the screen will ask you to choose an ASIN. If you have a high number of products, you might want to have this information available ahead of now. This information prevents you from needing to search.

If you choose an ASIN with variants (ex., a shirt with multiple colors), the system will ask you to decide which variants should be part of your testing.

At this point, you will receive notice if your ASIN lacks the required traffic to qualify. If you do not have enough traffic, you will need to grow your audience. If you hold the most outstanding share of your competitors, the overall ASIN might be too small to test.

Because A+ Content does not apply to a single ASIN, it will not ask you to choose variants. Instead, it will gather the traffic among all ASINs where you use this content.

If you do not see your ASIN on this list, it means your traffic is too low to qualify.

Your next step will enable you to choose the details of your experiment. For now, the details are broken into three sections:

• Experiment Name – This section is the name of the experiment that you choose. Customers will not see this title.
• Hypothesis – This section details your believed result. For example, your hypothesis could be that Title A will result in more sales than Title B.
• Duration and Start Date – It enables you to select the run time for tests (four to ten weeks) and the start date of your test.

Finishing all three sections will enable you to move to the next step.

Can I Choose To End My Experiment Early If One is Winning?

Amazon enables its users to end experiments early if they believe there is a clear winner. However, Amazon heavily discourages its users from doing so, leading to an incomplete picture.

The longer your test phase is, the better data you have. More data is never a bad thing, as you have the choice of whether to exclude it.

First, you’ll want to make sure the process starts with incredible simplicity. This process involves choosing a single independent variable.

The independent variable is a single, changeable factor of your product listing. When you change it, you should notice a measurable change in performance.

Using MYE as an example, let’s say that you decide to flip the location of the brand and a feature in your title. For instance, selling lotion that starts with the word “soothing” instead of having it three words deep.

Your other (more likely) test will be to change the price. If you gain a higher profit margin from increasing your price by $1, it can be tempting. However, losing the buy box for that change is not ideal.

What If I Want To Test Multiple Variables?

Multivariate testing involves using multiple changes to see what is effective. In this case, you might see something like this:

• Test #1: A title change
• Test #2: A title change and a detailed description change
• Test #3: A title change and a price change
• Test #4: A title, price, and description change

The idea is to include multiple individual changes to see what works. You can also test these in isolated formats to see if those personal changes have profound impacts.

Multivariant changes are significant when you want to test out a complete brand overhaul. However, you might find that gathering the data is complicated.

It is challenging to tell the exact location where this data comes from when multiple changes occur. If this is your first test, stick to single-variable A/B testing.

Your hypothesis is an assumption you are making as a result of your change. Another word for a hypothesis is “goal,” referring to the desired outcome.

Another name for your goal is the dependent variable. In scientific testing, the dependent variable changes depending upon what changes in the independent variable.

If you do not believe something will improve your business, don’t do it. Unlike a laboratory, your business shouldn’t be a testing facility to see what works. Your business is created to generate profit.

In this case, your goals should be SMART, which is an acronym we will explain below:

What are SMART Goals, and How Do They Help My Amazon Analysis?

SMART Goals simplify your analysis by applying five different requirements:

• Specific – Specific goals are those with a narrow purpose. Your plan to “grow business” is excellent, but how will you do that with this test?
• Measurable – Your goal needs to be associated with numbers. It would be best if you had percentages and dollar amounts to know your success.
• Achievable – Your goal needs to be possible, meaning that making $1 million in quarter four on flip-flop sales is probably not going to happen.
• Relevant – Your goal needs to be related to the topic. If your plan desires increased sales, it needs to be specific to your product from the change point.
• Time-Bound – Your measurable period needs to be specific (one month, one week, a year, etc.). Remember that more extended testing periods have better results.

If your goals lack all of these features, you are going to be guessing. A 10% increase in sales over one month due to your changes is a SMART goal because it is doable and specific. Saying that you want to make $1 million eventually is not feasible without smaller goals.

Without dealing with the small picture, you cannot hope to accomplish the big picture.

An A/B Testing Calendar enables you to determine how long this experiment will last. Just as always, more extended periods of testing are preferred, so stick with Amazon’s suggestion of 10 weeks (two months).

Be sure you have specific and planned days for your changes. That means you should account for one month for situation A and another month for situation B. Plan a week where you can gather the data.

If you use tools like Datahawk, we help by storing historical data for you. So you don’t have to worry about tracking when it’s already done for you.

The calendar fulfills the “time-bound” part of your SMART goal. If you exceed your time with project A, you will likely want to account for extra time with project B.

What if My A/B Testing Calendar Runs Into Major Events (Prime Day / Holidays)?

If your testing calendar runs into significant events, your data is compromised. In this case, information is skewed, and favor will likely go towards the product that spends more time during the event.

During the holiday season and Amazon Prime Day, your focus should be elsewhere. Testing phases are best done during regular seasons.

What if I Sell Seasonal Products?

The rules above still apply if you sell snowboards, surfboards, winter coats, or other seasonal items. You will need to account that people are more likely to buy your items during popular seasons.

Using historical data, you can determine how your product has generally behaved around this period. This information can enable you to make educated decisions.

If you know your product sales tend to increase 10% during winter, account for that in your threshold for error. Otherwise, try and make your testing phases stick to periods without transitions.

In this crash course on statistics, a confidence level is essential when running any test. It determines your willingness for data to go outside of your threshold.

Ideally, this threshold will be low, but luck isn’t always on our side. This is why it is crucial to spread your confidence interval.

The confidence interval (level) is typically at least 95%. This means that your error could be within 5% of your initial testing and still be accurate.

In the case where you fall outside five percent, you need to reassess your hypothesis. This retesting needs to occur whether you are above or below your testing number.

For example, assume you initially sold 10 thousand units and ended up selling 15 thousand when your goal was to sell 14 thousand.

If you have a five percent error threshold, you need to be within 700 units of 14 thousand (0.05 * 14000). This error threshold determines whether the data is statistically significant. In this case, you’ll likely want to run further testing because your significance allowance was 5% (700) when you had 1000 units.

If your tests were within 500 units, you could act upon this data.

Why Do I Need To Retest If I Outdo My Original Figures?

Why wouldn’t you accept the best possible outcome for your business? The reality is this: You always need to scrutinize your data.

If you sell more units than you expect, you need to account for other potential variables. Anything can impact your potential sales:

• Check to see if you have more positive customer feedback. A higher seller rank in your tracking data will cause you to have to re-evaluate data.
• If a product you sell suddenly becomes trending, the number of people who want to buy your item will increase.
• If you are comparing your items between seasonal changes, your data is likely inaccurate.
• If you improve your customer service, shipping speed, or other aspects of your business, you need to take another look at testing.

When it comes to Amazon, you can’t put other parts of your business on the backburner for testing. However, you cannot proceed without accounting for unusual variables.

Amazon Personalize is a metrics-tracking system a developer might use to gather data. The process is similar to what we mentioned above (research, create a goal, run experiments, measure results).

The main difference is that Amazon Personalize does this all automatically. This means if you can use this API, the entire process above turns into gathering data.

Amazon Personalize also enables you to plug variants at will. This way, you can quickly test alternative hypothesis options using the same information.

It also enables you to track the p-value.

What is the P-Value, and How Is It Important in Statistical Analysis?

P-Value refers to the chance you will see your hypothesis coming true. Ideally, you would want your p-value to be small, as that involves less fluctuation.

To accurately gain a p-value, you must run multiple tests. Many statistical models can run those tests automatically. Amazon Personalize can help you get closer to that level.

The problem with Amazon Personalize is that it specializes in making customer recommendations. If your ultimate goal is to run price or title variation tests, this tool is less useful.

For other machine learning tools, there is Amazon SageMaker.

Amazon uses Amazon SageMaker to manage Alexa, its voice-activated system. Machine Learning (ML) enables Amazon, Alexa, and some partners to stay on the cutting edge of business needs.

Amazon SageMaker is less applicable to new sellers. Instead, they are best for established brands with strong off-traffic segments.

Using SageMaker studio, your developer team could feasibly run multiple tests with many variations. Given that it is built on a machine learning platform, SageMaker can address variants and respond accordingly.

Data is sent directly to where they need to be through the Amazon API. SageMaker is also solid for testing by customer segments, which can be incredibly useful for larger companies.

DataHawk’s demo software gives you a good idea of using this, so we will be referencing it often in this short guide. DataHawk uses powerful data sets to help you optimize your business.

Here is how to get started:

Whether you change the price, product description, title, or images, it’s essential to select a specific date range for your change. This date range should be the same length as your historical data, going back several months if possible.

If you don’t select a specific date range, your data is not time-bound. Therefore, it is less valuable than someone who has a distinct and measurable goal.

You’ll want to set notifications on buy box changes, rank changes, and new competitors entering the niche from this period. Keeping other aspects consistent is one way you can keep the data handy.

What If I Lose The Buy Box During The Testing Phase?

Recall our earlier suggestion of not using an ASIN that is exceptionally important. In this case, you’ll want to consider the idea of changing things back.

However, buy box changes are not solely based on product descriptions and titles. You might see if other factors are affecting this (customer experience, bad reviews, etc.).

When testing, it is natural to want to jump the gun and return to comfort. However, the data you gather from A/B testing is essential in learning how your product works.

Because this is part of your testing phase, you must know when it starts and ends. This data is something beneficial, providing you with a written record of these changes.

The Reports & Automation section enables you to create reports under specific periods. You can save them in a folder and associate them to a document explaining the test by exporting from these times.

The document should confirm or deny your theory (hypothesis), ensuring future business partners do not make the same mistake. Testing changes to your product involves the confirmation of positive results and the elimination of adverse effects.

When creating A/B testing results, you must know your customer. Being a member of the Amazon Brand Registry gives you access to Amazon Brand Analytics, a treasure trove of data.

This data can enable you to base your changes on logic. For example, you might not market your product to 50-year-old women like you would sell to 20-year-old men.

It would be best if you made all business decisions with some logic. Typically, some of the research has been done for you. So before making changes, make sure you keep your audience’s expectations in mind.

You can start by asking them about their most important priorities. Using data gathered from a survey may help you make the first step.

Unless you are a data scientist, chances are you’ll find it challenging to test multivariance. Multivariance testing involves the application of multiple independent variables.

Because changing multiple things at once has the potential for numerous impacts makes it hard to find what affects the change. Adding a higher-quality image next to what you perceive to be a more engaging description adds confusion.

Always start with small changes as you will. While some more apparent changes don’t require testing (like a higher quality picture), many changes do.

When adding multiple variables, it’s best to leave those to third-party A/B testing tools. Those who specialize in this practice are better suited to working with it.

Our early example, conversion rate optimization, refers to several different things. To convert means to convince someone to subscribe to your product or idea.

Conversion includes the following:

• Making a sale
• Having someone subscribe to your newsletter
• Gaining a follower on social media
• Having someone download an eBook from your email marketing campaign

While each of these is meant to make a sale eventually, they are all considered performance metrics. So when choosing a variable, make sure it applies to one of your Key Performance indicators (KPIs).

KPIs are the ultimate example of SMART goals, enabling the user to link activities directly to success. How you define success depends on the ultimate goal of your hypothesis.

When it comes to finding data, any data will not do. Instead, you want data that works. In the case of split testing, this data must fall within your confidence levels.

Recall that a confidence level refers to the error allowance you have. If the data falls outside your error allowance, you need to either throw it out or test it again.

There are many reasons you should retest your data. Here are a few examples:

• The number of sales you made during a period was too high (above your confidence level)
• The number of sales you made was too low (below confidence level)
• There was a freak incident that caused your product listing to drop on the Amazon ranking chart
• You had some staff changes and logistical issues preventing you from performing adequately

When selecting your data, you should do it during a period of semi-normalcy. There will never be a period where your company is 100% not crazy, but there needs to be consistency between periods for data to be valid.

MYE calculates results every week, but it sometimes takes up to two weeks if the data is slow. Results automatically show with updates, so be sure to check it regularly.

So, is Amazon A/B testing worth it? The answer is yes. There is no question that this testing method is an excellent way to find out what works and what does not work on Amazon.

Even if you don’t have access to Amazon Experiments, Amazon already has an excellent platform where you can run tests. Utilizing secondary products and ASINs, you won’t have to sacrifice your hot items to see what works.

You can gain several benefits from A/B testing, but data is the most crucial part of this. Check out DataHawk’s Amazon Analytics software to see how this data can be helpful for you. Utilizing our data-driven process for your Amazon business is one way to stay ahead of the competition.