Policy testing
Test a decision policy change before it decides anything: replay, shadow runs and A/B tests
Miji is an enterprise decision automation platform, and the Lab is how a policy team finds out what a change would do before a customer meets it. Replay one recorded case on a draft and compare it with the original. Batch-replay thousands of past cases and see every decision that would move. Shadow a challenger beside live traffic, where it decides but never acts. Keep scenario sets of cases that must keep passing. And when the evidence is in, A/B test the change on a capped share of real traffic, with guardrails that stop it on their own.
Every one of these reads what the original runs recorded, so nothing is sent, charged or written while you look — and the original record is never edited.
In short
- Replay a recorded run from any step on a draft, and see both runs side by side.
- Batch-replay a sampled population of past cases and get a matrix of decisions that would change.
- Shadow a challenger on a percentage of live runs; applicants only ever get the production decision.
- Scenario sets: named cases with expected outcomes, run before every change like tests.
- A/B tests with a capped split, automatic guardrails and a second approver to declare a winner.
Why policy changes need testing
A credit policy, a claims rule or a routing threshold is code that business people own. Lowering a score cut-off by twenty points, changing an LTV limit or adding an overlay changes real outcomes for real people, and the usual way to find out how many is to ship it and count. That is the change a credit committee, a compliance officer or an examiner will ask about later.
Champion-challenger testing answers the question first. The champion is the policy that runs today; the challenger is the proposed one. Run the challenger against the same cases — past ones, then live ones that it cannot act on, then a small share it can — and decide with the numbers in front of you.
Five ways to test a change, from one case to live traffic
| Tool | Question it answers | Acts on people? |
|---|---|---|
| Replay | What would this one case have done on the draft, and which step changed? | No — answered from the original run's record |
| Batch replay | Across thousands of past cases, how many decisions move, and in which direction? | No |
| Shadow run | What would the challenger decide for the cases arriving now? | No — it decides, it never acts |
| Scenario set | Do the cases that must keep passing still pass? | No |
| A/B test | How does the challenger perform on a capped share of real traffic? | Yes, with guardrails and a second approver |
Replay one case and compare it with the original
Pick a finished run, the step to start from, and the change to try: the versions it originally read, the current ones, or the saved draft. Miji runs it again as a new run. The original is only ever read, so the record of what happened the first time stays exactly as it was.
Every connector call, model request and lookup is answered from what the original run recorded. Emails are not sent, records are not written, and the comparison says so at the top. The result is the two runs side by side, step by step, with the first differing step named and why it changed in one sentence.
Captured from the running product; the figures are seeded sample data.
Batch replay: thousands of past cases against a proposed change
Choose the population: a workflow, an environment, a date range, a decision step, and a sample size with a seed so the count you preview is the count that runs. Choose the change: a saved scenario's rule and table edits, or the saved draft of the workflow. Then start the batch and watch it finish.
The results page is built for a credit committee. A matrix of original decision against new decision, what drove the changes (which rule version, which lookup row), and a list of every changed case that opens its own comparison. Personal fields are masked throughout, and the CSV export is the same masked view.
- The population count is the API's preview, keyed on the exact request that runs; the browser never counts.
- Each changed case links to its replay comparison.
- Export to CSV with personal data masked; there is no unmasked view.
Shadow runs: a challenger beside live traffic
A batch answers what a change would have done to the cases you picked. A shadow answers what it would do to the cases arriving now. You name the production deployment to watch, the challenger (a saved draft or a saved overlay), a sample percentage and an end date. As live runs finish, the shadow re-runs its share of them on the challenger.
Applicants only ever get the production decision. The challenger reads only what each live run recorded and makes no calls of its own, and the dashboard counts the side effects it would have caused and reports them as suppressed. What you watch is agreement by day, the would-change matrix, and the latency the challenger adds.
- Sampling is a pure function of the shadow and run ids, so the same run is always in or always out and the pick can be checked later.
- The challenger is frozen at start; editing the draft afterwards changes no shadow run.
- A shadow ends on its date, when you stop it, or the moment the champion is redeployed.
Scenario sets: the cases that must keep passing
A set is a list of named cases built on real past runs: the strong applicant, the score exactly at the cut, the same run with a lower income, the thin file. Each case says what it expects, either the branch a decision step must take or the named result the run must end with.
Run the set against the versions each case originally read, against today's versions, or against the saved draft. Every case reads pass or fail with the reason, and a failing case opens its comparison. Teams keep a set per policy and run it the way engineers run tests: before every publish.
A/B tests with guardrails
The one experiment that acts on people, so it is the one with the most controls. A test splits one production deployment's traffic with another deployment of the same workflow. Each applicant is assigned by a seeded hash on a key you choose, so a returning applicant always gets the same side, and the challenger's share is capped.
Guardrails compare the challenger's bad-result rate, error rate and manual-review share against the champion and stop the test automatically on a breach. Results show each outcome rate with a 95% range and flag what is significant. Declaring a winner records the result and ends the test; it never deploys anything, and it needs a second approver.
- One active test per champion; the challenger's share is a whole number up to 50%.
- Any doubt routes the run to the champion: an ended test, a redeployed champion, a challenger that cannot be read, an input with no subject.
- One person asks for a decision and a second person approves it; a stop sends all traffic back to the champion at once.
What makes this safe to run on regulated decisions
- Nothing leaves the building. Replays, batches and shadows answer every connector call, model request and lookup from what the original run recorded. Emails are not sent and records are not written; a step the original never recorded stops with its reason rather than going live.
- The record stays the record. A replay is a new run linked to its source, never an edit of it, so the audit trail of what happened the first time is untouched.
- Versions are named. Comparisons name the rule and lookup-table versions each side read, by content fingerprint.
- Personal data is masked. Batch results and their CSV export mask personal fields; there is no unmasked view.
- Two people for anything that acts. Declaring an A/B winner needs a second approver, and declaring one never deploys anything.
On the roadmap
Not available today. Listed so you can see where this is going — the capabilities above are the ones you can use now.
Draft, approve, publishBuilding
A required review step between writing a policy change and it deciding anything, so a tested change is also an approved one.
Bring a policy change you are weighing. We will batch-replay it on sample cases on the call.
Book a demoFrequently asked questions
How do you test a credit policy change before it goes live?
Run the proposed policy against cases whose outcome you already know. In Miji you can replay one recorded application on a draft, batch-replay thousands of past applications and see every decision that would change, shadow the draft beside live traffic without it acting, and keep scenario sets of cases that must keep passing. Only then A/B test it on a capped share of real traffic.
What is champion-challenger testing?
Running a proposed decision policy (the challenger) against the same cases as the current one (the champion) and comparing outcomes. Miji supports it at every stage: on past cases with batch replay, on live cases without acting through shadow runs, and on a capped share of real traffic with A/B tests that stop on their own when a guardrail is breached.
Does replaying a case call the credit bureau or send emails again?
No. A replay is answered from what the original run recorded: connector calls, model requests and lookups. Nothing is sent, charged or written, and a step the original never recorded stops with its reason instead of making a live call.
Can a shadow run affect an applicant?
No. Applicants only ever get the production decision. The challenger reads what each live run recorded and makes no calls of its own; the dashboard counts the side effects it would have caused and reports them as suppressed.
How do A/B tests on decision policies stay safe?
The challenger's share is capped at 50%, each subject is assigned by a seeded hash so a returning applicant always gets the same side, and guardrails on bad-result rate, error rate and manual-review share stop the test automatically. Any doubt routes to the champion, and declaring a winner needs a second approver.
Is personal data exposed in test results?
Batch results mask personal fields throughout, and the CSV export is the same masked view; there is no unmasked view.