Group Assignment in a Frontend vs. Backend A/B Test

Where do you store user groups? How do you make sure your test is actually able to be evaluated?

Growth is all about experimentation. Reforge has a program about it, companies have been launched around it, and leaders talk about it. How do you actually implement a successful experiment?

The questions already being discussed are how to:

• Decide which product feature should be behind a test

• Read the results of an A/B test

• Prioritize features based on data to have an educated hypothesis instead of finger-in-the-wind prioritization

Imagine the point where the feature is almost implemented, you have a hypothesis, and are at the cliff of actually implementing the A/B test. How do you:

• Understand what parts of the test are in frontend vs. backend?

• Store who’s part of each group in the test?

• Make sure you implement a test that will have useful results at the end?

Note: we’ll be talking about in-product tests where the user is logged in. Marketing tests are a separate can of worms which maybe we’ll open another day.

What is the difference between frontend and backend tests?

Justin of Technically uses a restaurant analogy quite nicely. The frontend of a restaurant is what you see when you walk in (tables, food, drink, service) while the backend is behind the two swinging doors (kitchen, staff, supply chain for ingredients, management).

Focusing on web applications, the frontend is what you see on your screen—the UI/UX. The backend powers this experience and usually consists of a set of APIs and databases. Similarly for experimentation, the frontend test elements are in the browser (UX or the cache), while the backend test elements are in the database.

▶ Definition: A frontend test relies purely on the user experience and user groups can be defined in the browser cache; a backend test requires business logic and/or user assignment to be stored in the database.

Test themselves aren’t necessarily only frontend or only backend—they can be both. Imagine a 2x2 grid of possibilities, overlaying where the feature is implemented (frontend vs. backend) with where user groups are assigned (frontend vs. backend).

The two components we need to distinguish between are:

• The actual feature. A frontend-only feature is a new design, UX page, button placement, etc. It really just has to do with what the user sees. A backend-only feature on the other hand exposes new functionality or new database values (that the end-user interprets as a new experience).

• Population assignment. Deciding who sees which version of the test means first defining the total population then randomizing the split criteria. In a frontend test populations are defined in the browser cache on the spot, while in a backend test users are grouped in the database and can be known in advance.

Deciding how the feature will be implemented is usually done on the engineering team, separately from deciding whether it’s being tested or not. While it’s not decided in the scope of the test, understanding the implementation dictates how to define user groups and measure the test itself. For the scope of this post, we will focus on population assignment and measurement.

Caveat: many features involve both backend and frontend changes, usually ones that expose a completely new set of functionality. Because these features involve the backend in some capacity, they require the same setup as a purely backend test.

Where and how to assign A/B test user groups

Assigning user groups in the frontend vs. backend has repercussions. Specifically:

• Frontend assignment means a user can reset their group and see both experiences (if browser cookies are cleared) while backend assignment stays fixed

• Frontend data is never as complete as backend data (ad-blockers and incognito mode, etc)

• Backend user groups can be assigned in advance and known, while frontend assignment happens on the fly

• Backend assignment is usually more work to implement

Backend features should go with backend assignment, otherwise some wonky things start to happen. Imagine a test of making a $70 product have $10 shipping or an $80 product have $0 shipping. From the user’s perspective, it would be extremely confusing to see one experience than the other and lower trust in the brand. Besides the strange UX this would cause, usually pricing is calculated on the backend and numbers are simply pulled into the frontend—making a frontend test implementation against engineering architecture in this case.

Frontend features have the choice: when launching something like a new summary dashboard page, users can either be assigned to view this feature as soon as a user is created in the backend or as soon as they load the first screen in the browser.

So how do you choose?

👉 Choose frontend user group assignment if optimizing for speed and simplicity.
How: fire an analytics event.

Frontend user group assignment can be implemented by firing an analytics event (Amplitude, Segment, etc) upon a page to denote the group. For example, consider the following properties:

{
    "user_id": "ABCD"
    "page_url": "www.mysite.com/home"
    "event_type": "A/B Test",
    "properties": {
        "test_group": 1
        "test_name": "Dashboard Test"
    }
}

They are illustrative to the test name, group assignment, who the user is, and what page the test assignment happened. Firing an event for assignment instead of relying on analytics from a specific page as part of the test means not assuming someone viewed an experience due to lack of information. It makes group assignment explicit and easier to QA.

👉 Choose backend user group assignment for complete data.
How: store user assignment in a database model.

Consider a table with the following columns:

• id

• created_timestamp

• user_id

• test_name

• test_group

This information gets us who is being assigned to what group as part of which test and when. It’s static and subscribes to the permanent and highly auditable nature of databases. We can assign users to a group immediately upon signup, no matter if they actually experience the test or not. This means we know populations ahead of time, and users are guaranteed to only see one experience. If implemented properly, backend test assignment yields the most complete data.

Group assignment influences how you analyze results

What is the most important data point in an A/B test? Group assignment.

If you don’t properly store or understand group assignment, statistical significance doesn’t mean squat. Know whether:

• Groups are assigned in advance or as soon as one experience is prevalent

• Groups are subject to ad-blockers or are thoroughly assigned

Knowing these specifics is required to correctly query the test population and segment cohorts not subject to random variables (like ad-blocker usage). Implementation details aren’t usually details—rather crucial pieces of information for proper analysis.

Thanks for reading! I talk all things marketing, growth, analytics, and tech so please reach out. I’d love to hear from you.