What can you learn from Netflix's A / B Testing process?

Have you ever wondered why Netflix offers such a great stream experience? Do you want to learn how they have perfected their homepage as well as redesigning the layout on the user interface with A / B Testing? If so, this article will help you answer these two questions in great detail.

I will start by sharing some information I obtained from the Designers + Geeks event that I attended last week at Yelp. Two great speakers, Anna Blaylock and Navin Iyengar (currently both product designers at Netflix) have come up with many interesting insights they have gleaned for many years implementing A / B Testing with participation. Join dozens of millions of Netflix members and provide many product-related examples to help listeners better understand their own designs.

Images from presentations

Experiment

I really enjoyed this first slide and they were smart to use an image from TV Show "Breaking Bad" to explain the concept (the main idea) of the experiment.

Images from presentations

Scientific method

Images from presentations

Hypothesis

In science, a hypothesis is an opinion or an explanation that you must then check its accuracy through research and experimentation. In design, a theory or speculation can also be called a hypothesis.

Images from presentations

The idea of ​​the basis of the hypothesis is that no results have been identified before. That is something that can be checked and therefore, these tests can be copied again.

The general concept behind A / B Testing is to create an experiment with the participation of a control group (Control Group, also called a control group) and one or more test groups (also called Cells). - groups will receive different therapies during implementation. Each member belongs to only one Cell in an experiment that has been identified with one of the other Cells designed as "Default Cell". This cell represents the Control Group - the group received the same experience as all other Netflix members who did not participate in the test (follow the sharing on Netflix Blog).

This is how A / B Testing has been done at Netflix: As soon as the test is conducted, they will track the measurements of importance. For example, it could be factors like stream time and retention rate. Once participants provide enough meaningful conclusions, they will turn to the effectiveness of each test and determine the winner of the various variations.

Images from presentations

Images from the presentation about the hypothesis process

Experiment

Each test is an activity of the testing process. Some companies like Netflix perform many tests to create user data. The important thing here is that it takes time and effort to organize the test accurately to ensure sufficient quantity and type of data, to serve the purpose of clarifying doubts about concerns. The more effective the better.

You may realize that the featured programs on the Netflix homepage seem to change every time you visit again. All of them are in complex Netflix tests to stimulate users to see their Show.

Home when I log in for the first time

Images from the presentation: For those who are not registered, the "House of Cards" page will be displayed in this form

Home when I log in for the second time

Home when I change user accounts

Home when I convert User type to children

Home when I'm not registered

The idea of ​​A / B Tesing is to show different content to different user groups, gather their reactions and use the results to build future strategies. According to an article posted on Netfix Blog by Gopal Krishnan:

If you do not "catch" a member's attention within 90 seconds, that member will likely have no interest and switch to another activity. Those failures can happen at certain times because we don't display the content they need or because we have provided the right content but do not give enough convincing reasons that at Why should they watch that show?

Netflix had done a test in 2013 to see if they could create some artwork to attract the audience's attention to the title and this is the result:

Images from Netflix Blog

A signal that appeared early showed that the members were sensitive to the change of artwork and another sign confirmed that there were better ways to help Netflix members find their favorite stories while spread. experience on this site.

After that, Netflix has created an artwork system that has aspect ratios, aspect ratio, different touch up, Title Treatment (movie logo), but has been positioned, but with background. They also replicated experiments on other TV shows to track the effects of related artworks. Below are a few examples:

Images from Netflix Blog: Two images are tagged to attract more users than the rest

Images from Netflix Blog: The final image is more efficient than the rest

What do I learn?

A / B Testing is the most reliable way to learn about user behavior. In the role of designers, we should think about our work through the "lens" of the experiment.

When and why should A / B Testing be done?

Once you have a design put into production, use A / B Testing to modify the design and focus on the two core measurements: retention rate and revenue. By changing A / B Testing throughout the product and tracking users over time, can you know whether the change will improve the retention rate or increase revenue? If so, make the changes the default. Thus, A / B Testing can be used to further improve your business indicators.

Are users looking for or doing something you want them to find or do? My experience is that normally, users don't always finish the job as quickly as they expect and sometimes, they don't even find a button placed on your page. The reason is so much: probably because the design is not really intuitive, the colors are too prominent, the "low tech" users, can't make decisions due to too many choices on one page and some other reasons .

Are the interfaces intuitive?

Sadly, when it comes to user behavior, visualization can be misjudged and the only way to prove it is to do A / B Testing. This is the best solution to confirm whether a UX design is more effective than another. For example, if you want to determine whether a change in design changes the subscription rate of users who clicked on Google ads, Netflix will create different test designs and test them. Many people think that the design is only aimed at images that are specific to winning but realize that the design is expressed both on the image and the price (price) will receive the highest conversion rate.

Discover the limits

The best ideas come from discovering many other ideas. At the office, our product team interacts with each other through many different projects. With the participation of many departments (from design to product managers, developers), we have unlocked the limits together. Some of the best ideas sometimes come from the developer or product manager after testing the samples.

Watch what people do, not what they say

It is important to remember when talking to users that they always talk about something but do it differently. I have carried out several steps in the process of user testing and have a perfect example to explain to you why I affirmed that.

I have a user who achieved high results in the contact list on the phone and asked him if he regularly categorized / filtered contacts. He said no because he didn't need to do so. However, when he discovered that the "dropdown" menu had been refreshed, he was amazed at the classification as well as the filters became more convenient than before and immediately asked when the key would be mode to bring this feature into operation.

Use data to assess the scale of opportunities

1. Always ask why.
2. Data can help you form ideas.
3. Check whether there are any A / B Testing procedures that conflict.

There is nothing interesting about being a UI and UX designer. However, understanding users makes this work interesting. There is no complete design. Because, there are always many ways to edit the design to give users the best experience possible. I always take the opportunity to make some changes, measure their reactions and coordinate with the Product Team to determine the next steps.