We are adding support for “Multiple-Weighted Variants” within Rules that allows developers to define more than one variant per Setting in a Rule.
Remote Config uses Rules to target specific user groups and deliver different settings for each group. Rules are linked to Settings, which map to variables in the game code to override their default values when the audience criteria for a Rule is met.
For existing Remote Config users, you can use this new functionality to run a randomized experiment with two or more variants, commonly known as “A|B testing”.
You can now create a Rule where, instead of a single value mapped to a Setting, you can create multiple variants of that value for a particular Setting. Each variant also has an associated weight that defines the allocation of that particular variant across the population of players that the Rule applies to. You can either define custom weights for each variant – with the only condition being that the sum of weights across all variants must equal 100% – or if weights are all left undefined, Remote Config will assume uniform, random distribution for the variants.
Known Limitations
Analysis and Reporting
Remote Config is limited to the handling of the assignment and delivery of settings.
The system today does not do any of the following:
- Automatically track a target metric
- Perform any statistical hypothesis testing to calculate sample size or assess the significance of sample data
- Determine “winners”
Remote Config does create a unique variantId that is appended to the assignment event when serving assignments to the player at runtime. In this way, you can use the assignment summary and aggregate reports to describe how often each variant was selected (see Assignment Reports for more details).
If your game also sends user behavior events to Unity or your own analytics endpoint, you can cross reference this assignment summary report with a user behavior events report using the assignmentId, and then perform your own statistical analysis to determine the results of your test.
Variant Stickiness
Remote Config currently cannot track specific user assignments at runtime, so variant stickiness cannot be guaranteed. As it stands today, if the request attributes are identical, Remote Config will always serve the same assignment to the requesting user - including variant selection. However, if those attributes change, or the Rule configuration changes, there is the risk of serving up a different variant the next time that user requests an assignment.
A|B Testing Example
Step 1: Initial configuration
Go through the process of configuring your project for Unity Remote Config by following the steps here: Configuring your Project for Unity Remote Config.
After creating an Environment and Config (either in the dashboard or Editor GUI), create a new Rule with type “variant”, then link your Settings to this Rule.
Step 2: Add new variants to the Rule
Add a new variant to the Rule using either the Editor package GUI or in the Remote Config Dashboard.
Editor GUI
Dashboard
When using the dashboard, click on the “Settings” tab to add variants and edit variant-specific settings.
At the start, each Setting listed in the Rule will include one value by default. You can add variants by clicking on “Add Variants”.
Make adjustments as needed to the variant weights and setting values, then save your changes by selecting “Save” on the dashboard or “Push” in the Editor GUI.
Step 3: Enable Rule and Play Game
Finally, make sure your new Rule is set to “Enabled” to see your variant assignments at runtime.
