The owner of an online coffee shop in Almaty is watching an analytics demo. The screen has revenue, a funnel and a button for starting a test. His question is simpler: where is paid order O42 for 15,000 ₸, and which page variant did the buyer see? The sales representative suggests paying for the subscription first and discussing integration later.
That order is a useful starting point for the choice. The shop and O42 come from sending a paid purchase from the server; the comparison figures that follow are illustrative. We will reproduce the purchase path using test records and compare the cost of maintaining two setups.
Write the requirement before meeting the vendor
The shop plans to compare two checkout variants. Eligible participants receive a variant before buying and keep it under the experiment's rules. The analysis needs outcomes for all assigned participants, including people who buy nothing. Payment records supply a numerator, but cannot reconstruct the conversion denominator.
A tool requirement could read: "For each participant and experiment, recover the assigned variant, exposure, subsequent orders and payments within the chosen window." Add a data owner and an export method. This gives the vendor a demonstration to deliver, beyond showing every feature in the product.
The illustration contains a T-shirt product card. The useful part is the connection it shows: page variant, order O42, payment and result. The coffee shop sells a different product, but the record links are tested in the same way.
Follow O42 from assignment to the report
An Order ID identifies an order. The participant needs a separate ID: a person can place several orders, while an assigned participant may place none. Keeping only O42 and a "B" label is insufficient for comparing groups.
In the test environment, the team follows this chain:
| Step | What must remain in the data |
|---|---|
| Participant receives a variant | Participant ID, experiment ID, variant and assignment time |
| Page variant is shown | Exposure record under the rule chosen in advance |
| Order O42 is created | Link to the participant and the right experiment |
| Payment is confirmed | O42, amount 15,000 ₸, currency KZT and payment time |
| Analytics receives the data | Receipt time and one counted payment |
| Refund is received | Separate transaction and a result updated under the metric's rule |
Assignment and exposure can happen at different times. Decide in the plan who enters the main analysis. Do not keep only buyers or only people who reached a convenient reporting step.
Send the payment record again to test delivery. O42 should still contribute 15,000 ₸, although the delivery log contains two attempts. Contact, deal and payment use different keys: the customer's contact ID cannot replace their purchase ID.
Label test records and separate them from production. During acceptance testing, they must not enter financial reports or become real advertising conversions. One successful order confirms this path; failures and other cases still need testing before launch.
Give each part of the setup a job
In a small project, a spreadsheet, CRM and a small handler may cover some of the work. More sources may call for a warehouse and a separate experiment system. Company size alone is little help without a list of jobs.
| Job | What to ask the vendor to show |
|---|---|
| Events and behaviour | Source records, a segment and the required unit of analysis |
| Sales | Order, confirmed payment and correction history |
| Variant allocation | Stable assignment and an exposure log |
| Exchange between systems | Retries, delays, failures and recovery |
| Messages | Unsubscribe, exclusion after an order and frequency limits |
| Reporting | Metric definitions and a reproducible calculation |
One product may cover several rows. Each row still needs an owner: an analytics vendor does not become responsible for the payment status in your CRM.
A feature flag can switch a variant on. A causal comparison also needs suitable assignment, an experiment unit, metrics and an analysis plan. Changing a page between Monday and Friday cannot separate the variant's effect from other changes during the week.
The GrowthBook experiment results documentation includes group-balance and other data-quality checks. In a demo, ask to see a warning on a deliberately broken export. This also establishes which checks work in the proposed configuration and which data sources must be connected. A platform name cannot fix missing payments.
Continue acceptance testing with failures
After the normal path, make the test receiver unavailable, delay a payment and change the order in which records arrive. The acceptance record needs the result, how the issue is detected and what the owner does.
Two timestamps matter for O42: when the customer paid and when the information reached the report. A delayed payment record must not quietly move the purchase into another period. Time zone and reporting window belong in the metric definition.
Test device changes within the identity model the product claims to support. If it cannot link an anonymous phone to a laptop, document that boundary in the result. Matching similar names just to make the table reconcile is not an acceptable repair.
Messages need a separate test: the person withdrew consent, already ordered or reached the frequency limit. The system should skip sending and record why. Current-state checks form part of acceptance for the whole setup.
Get data you can leave with
Ask for a small export containing participants, assignments, available events, orders and metric definitions. Another specialist should be able to reproduce the participant and payment counts. A screenshot of the final conversion rate cannot do that job.
Before buying, establish whether the fields and history you need are available, what permissions and API limits apply, and how exports work after the subscription ends. Take exact terms from the proposal and the chosen plan's documentation. Downloading a contact list does not solve experiment-history migration.
Session recordings also need acceptance testing: dummy content in protected fields is hidden, permissions work, and retention and deletion are configured. Do not enter real passwords or payment details for this test. Test the selected configuration, including new form fields after an update.
Put subscriptions and labour in one budget
In our comparison, the first setup requires 30,000 ₸ a month for services and 20 hours of maintenance. The second costs 90,000 ₸ and needs four hours. The team values an hour at 5,000 ₸ for this budget. These are project estimates, not tied to any service's pricing.
| Cost | First setup | Second setup |
|---|---|---|
| Monthly services | 30,000 ₸ | 90,000 ₸ |
| Maintenance | 20 × 5,000 = 100,000 ₸ | 4 × 5,000 = 20,000 ₸ |
| Monthly total | 130,000 ₸ | 110,000 ₸ |
| One-off implementation | 60,000 ₸ | 180,000 ₸ |
| Implementation and six months of operation | 840,000 ₸ | 840,000 ₸ |
The second setup costs 20,000 ₸ less each month but 120,000 ₸ more to implement. Under these assumptions, it recovers the difference in six months. A shorter project removes that argument for choosing it.
Both estimates need training, migration, storage, support and eventual exit costs. Establish what is already included to avoid counting the same work twice. When volume grows, recalculate the unit actually billed: events, contacts, messages or computation. Measure maintenance hours in the pilot instead of relying on a sales promise.
Exercise: choose after the pilot
Build a job table for your client and run an order through both setups. Keep evidence another specialist can use.
Suppose the pilot shows that the first setup needs only 12 hours a month. Its cost becomes 30,000 + 12 × 5,000 = 90,000 ₸. The second still costs 110,000 ₸ and more to implement. If both meet the same requirements, the old labour-saving calculation no longer justifies the second. Buying it needs another demonstrated reason.
If neither setup could link the participant to O42, postpone the dashboard choice until the data is repaired. A documented break and a costed repair help the client more than a new interface carrying the old mistake.
What to read while preparing the choice
Brian Balfour's principles, process and tactics help define the work before choosing a tool. Take your O42 scenario, export requirements and budget to the vendor meeting: they give you grounds to accept or reject the proposal.