A/B testing has been around for a long time and gives you an amazing way to test new website content, processes, workflows, etc. At a very high level, you split your website's traffic between two deployments of code and measure the success of each version of the site based on your needs. A very simple concept that can be very difficult to achieve without special setup or 3rd party tools. Fortunately for anyone running their web applications using Azure's App Services (formerly Websites), there is an extremely easy way to achieve A/B testing, which we'll look at in this post.

Setup

To start, we'll need an App Service created in Azure and deploy our site to it. For this demo, I've created a new App Service and selected the S1 Standard plan as we need Deployment Slots in order to make this work. I've named the App Service "CTV-AB-TESTING" which gives us a URL of http://ctv-ab-testing.azurewebsites.net.

Creating App Service

Now that we have a production site for our "A" traffic, we'll need to create at least one Deployment Slot for our "B" traffic. I've created a new slot and named it "STAGING".

Creating deployment slots

Next, I've created an extremely basic website that simply shows the environment name on the page. Using Visual Studio we'll publish a version to the production site, and another version to the staging site. We can confirm the sites have been published by navigating to their respective URLS, http://ctv-ab-testing.azurewebsites.net for production, and http://ctv-ab-testing-staging.azurewebsites.net for staging.

Production Site

Staging Site

Now that we have both sites deployed and have verified they are up and running, we can now configure Traffic Routing to setup our A/B testing. Navigate back to the production App Service blade in Azure and ensure the "All Settings" blade is open. Under the "Routing" header you'll find the link for Traffic Routing.

Traffic Routing

On the Traffic Routing blade you may be surprised to see how simple the configuration is. All you need to do is select a Deployment Slot for the dropdown menu and provide a percentage of the production traffic that should be routed to it. It's that simple!

Traffic Routing

Make sure click "Save", and you're done. Your site is now configured for A/B testing!

Testing

Now that the easy part is done, you'll want to test it out to ensure it's working properly. This can be done in a number of ways from automated tools to simply hitting refresh in the browser. For this demo, I created a quick and simple console application that will hit the production site, count the number of times each version comes up, and calculates the percentage for me. The code is very simple:

class Program
 {
 static decimal prodCount = 0;
 static decimal stagCount = 0;

 static void Main(string[] args)
 {
 while (1 == 1)
 GetWebContent();
 }

 private static void GetWebContent()
 {
 HttpClient client = new HttpClient();
 var response = client.GetAsync(new Uri("http://ctv-ab-testing.azurewebsites.net"));
 var html = response.Result.Content.ReadAsStringAsync().Result;

 if (html.Contains("PRODUCTION"))
 prodCount++;
 else if (html.Contains("STAGING"))
 stagCount++;

 decimal percentage = 0;
 if (prodCount != 0 && stagCount != 0)
 percentage = (stagCount / prodCount) * 100;
 else
 percentage = 100;

 Console.Clear();
 Console.WriteLine("Prod hit {0} times, Staging hit {1} times, Percentage: {2}%", prodCount, stagCount, percentage.ToString("0.00"));
 }
 }

Running this app displays all the information I need to ensure the Traffic Routing is properly configured and the percentages are in line with my setup.

Console Application

As you can see during this run of the console app, the split percentage is well over the 20% I setup. During my testing I routinely saw higher percentages than what I setup. Nothing drastic, but keep this in mind as you use this feature. If you are targeting a certain percentage, you may want to bump your target down a little with the anticipation the final numbers may run a little over.

In a real world scenario, your testing efforts will be much more deep and involved. You'll want to capture metrics on user behavior (maybe using Application Insights), conversion rates, and more. However, for initial setup and a quick test to ensure routing is working properly, this does the trick.

Conclusion

In this post, we saw how you can very quickly and simply set up A/B testing on any website hosted on Azure App Services. A/B testing provides site owners an amazing way to test new features, layouts, and more. If A/B testing is important to your business and you've been thinking of making the jump to Azure, you may have one more reason to leap now.

About the Author:

brandon atkinson


Brandon Atkinson is an architect in our Richmond, VA office. He has been a software engineer for over 10 years and has been focused on Microsoft technologies including .NET, ASP.NET, Azure, SharePoint, and Universal Windows Platform.