A/B experiments: test other instructions with real customers
Compare your agent's current instructions (A) with another version (B) in real conversations and keep the one that works better.
Do shorter replies resolve more? Is it better to offer the appointment in the first message? With an A/B experiment you test it with real customers instead of guessing.
- Variant A is your agent as it is today.
- Variant B is the same agent (same engine level, tools, knowledge and schedule) with different instructions that you write.
Each new conversation the AI handles goes to A or to B and stays there until the end. Omnifox records how each one ended and tells you which one worked better.
Available from the Escala plan (and on Exclusivo). On other plans you will see "A/B experiments are available from the Escala plan" and the Upgrade plan button. Only people who can edit agents (owner and admins) see it.
How to create an experiment
- Open AI Agents, open your agent and go to the Instructions tab.
- Above the editor press A/B experiment (next to Use template, History and Optimize with AI). The agent must be saved.
- Press New experiment (or Create the first experiment).
- Fill in:
- Experiment name (required) and Description (optional). E.g. "Shorter replies".
- Conversation split: what share of new conversations goes to B. By default 50% / 50%.
- How success is measured: one of the three metrics (see below).
- Minimum sample per variant: how many conversations with a result each variant needs before deciding (100 by default).
- Variant B instructions: they start as a copy of the current ones; change only what you want to test.
- Press Save. The experiment is a Draft.
- In the list of experiments press Start.
Tip: change one thing only per experiment. If you change five at once, you will not know which one made the difference. If B is the same as A, it will not let you save.
The three metrics
| Metric | Success | Failure |
|---|---|---|
| Resolved without a person | The conversation ended (resolved, closed or 72 hours without messages) without going to a person. | The AI handed it to a person, it was assigned to someone on the team or a person replied. |
| Conversion (appointment or order) | An appointment was booked or an order (not cancelled) was created in the conversation. | It ended (closed or 72 hours without messages) with neither. |
| Satisfaction (survey) | The customer rated it well: 4 or 5 out of 5, 8 to 10 out of 10, or thumbs up. | They rated it badly. Conversations with no survey answer within 7 days do not count. |
To measure satisfaction, the survey must be on in Workspace → Satisfaction survey; if it is not, the editor warns you.
While it runs
- Only the new conversations the AI handles are split; open ones stay as they were.
- Each agent can have only one running experiment. To start another one, pause or finish the current one.
- Pause: every conversation uses A until you press Resume. A draft or paused experiment can be Edited.
- If the agent is off, the experiment receives no conversations.
- Delete (bin) removes the experiment and its results; the agent stays as it is.
The results
View results shows the days running, the conversations with a result in A and in B, the Probability of winning of each variant and the Comparison: successes, failures, Waiting (not ended yet), success rate and how far each one is from the minimum sample.
They update on their own every hour and every time you open the page. With the resolution metric, when the AI hands a conversation to a person the failure is recorded instantly.
How the winner is declared
- When both variants reach the minimum sample and one has more than a 95% probability of being better, it is declared the winner automatically: A won or B won.
- If with three times the minimum sample there is no clear difference, it ends with No winner.
- To decide earlier: Finish → Declare A the winner, Declare B the winner or Finish without a winner.
Use B's instructions
If B wins, press Use B's instructions in the agent and confirm with Yes, use B's. The agent replies with B's text from then on and the previous version stays in the instructions History, so you can go back whenever you want. If A wins, there is nothing to do.
If it doesn't work
- I don't see the A/B experiment button: check your plan (Escala or higher), that you can edit agents and that the agent is saved.
- It won't let me start: there is already a running experiment on that agent.
- No conversations come in: the agent must be active and connected to a channel; only new conversations count.
- Satisfaction has no data: turn on the workspace satisfaction survey.
All the details, with screenshots, are in chapter 12 of the AI Agents complete guide PDF manual and in the AI Agents complete guide.
Related articles
Related articles
-
Create your first AI agent
Learn what an AI agent does in Omnifox, how to give it instructions, connect it to a channel and test it in the sandbox before it talks to real customers.
-
How to write good instructions for your AI agent
Learn how to write clear instructions so your AI agent replies with the right tone, limits and actions.
-
AI Agents complete guide
AI Agents in depth, in 65 chapters: creating, instructions, tools, knowledge, behavior, tester, inbox, workflows, analytics and usage, with every setting explained.
-
Connect your AI agent to WhatsApp, Instagram, Messenger and web chat
An agent with no channels serves nobody. Connect it to your WhatsApp, Instagram, Messenger or website chat from the Connections section, then switch it on so it starts replying.
-
Start here: AI Agents in 8 minutes
AI Agents essentials in a short video: creating an agent, its instructions, tools, knowledge, channels, the tester, hours and handing over to a person.