The owner of an online coffee shop in Almaty forwards you a post called "10 retention hacks". It has a bean subscription with auto-billing that can only be cancelled by calling a manager, a discount that "expires in 24 hours", and an unsubscribe button that reads "No, I don't like fresh coffee". "Let's do it, retention will go up." Renewals may rise because some people struggle to leave. We will use the shop from the retention architecture example; the numbers are made up.
Why a trap isn't retention
Retention helps you understand whether people return to the product and get value from it. Hard cancellation adds people who couldn't get through on the phone to the renewal count. The online booking service for salons has a customer like this: the salon pays but gets no online bookings, creating a risk of delayed churn. A customer charged for unwanted coffee for the third month running is unlikely to recommend the shop to colleagues the way Aliya does.
The picture shows the same gap at a reporting service: 90 renewals versus 27 finished reports, a gap of more than threefold.
The other tricks in the post also distort choice. A discount that "expires" and returns the next day creates fake urgency. A "No, I don't like fresh coffee" button shames people for declining. On our reading of Kazakhstan's Law on Consumer Protection (as amended up to 20 August 2026), a pre-selected paid service looks like an imposed one, and the law lists an obligation to pay for imposed services among violations of consumer rights (Art. 8-1(2)(3)). This is not legal advice.
The shop leaves these tricks out of its test. Testing hard cancellation by renewal count would mix people who want to stay with people blocked by the interface. So the shop tests something useful instead: how many orders it gains by helping customers come back.
What to test and what to simply roll out
At the stage where the bag is running low, the shop has three steps: a postcard in the first parcel with the date the kilogram will run out, a reminder 21 days after delivery for those who agreed to messages, and Instagram guides tagged utm_campaign=reorder. All three are meant to lift the share of newcomers with a second order within 60 days from a quarter to a third, as set in the shop's goal.
A test is not free: while it runs, half the newcomers miss a step that may well work. So you test what could do harm or costs real money. A postcard costs next to nothing and uses up nobody's consent, so it goes in every parcel, with the launch date recorded and printing in the budget. Stories can't be split into groups: Instagram decides who sees them. A reminder, though, can be sent to one group and withheld from the other, and it can both add orders and do harm: someone messaged at the wrong moment replies "STOP" or unsubscribes. That is the step to test. When there are more ideas than time for tests, RICE helps put them in order.
The test plan
The flow specification schedules the reminder for day 21 after the first delivery. Customers whose first payment falls within recruitment and who agreed to messages enter the test, about 100 of the 150 newcomers each month. The group is assigned once, right after the first payment and consent, before the first message: an even ID gets the reminder, an odd ID does not. The group without a reminder is the holdout. In the send group, WhatsApp is used with consent to that channel, otherwise email with consent to email; a repeat order, complaint or withdrawal of consent is checked before sending. All other service is the same in both groups.
The hypothesis follows the template from landing page hypotheses:
If we remind newcomers 21 days after their first order is delivered that their beans are running out, more of them will place a second order within 60 days, because at two cups a day a kilogram lasts about a month and delivery takes 2-5 days. We'll check the share with a second paid order in both groups, the "STOP" replies and the unsubscribes.
- The unit is the customer. The group stays fixed until the end of the test. Before launch, find out how IDs are issued: parity must be independent of acquisition channel, date and branch. If it is not, the developer sets up random customer assignment and stores the assigned group. An ID split is convenient only when that condition holds.
- The main metric covers the whole assigned group. The share with a second paid and non-refunded order within 60 days of the first payment, among all consenting customers assigned to each group. A customer stays in the calculation even if the message never arrived, they didn't read it or they later unsubscribed. Comparing only readers mixes the reminder's effect with their readiness to buy.
- Guardrail metrics. "STOP" replies, email unsubscribes and complaints to the manager. Before launch you write down the threshold at which reminders will be paused and rewritten.
- Duration and readings. Set recruitment duration, the metric window and interim reading dates before launch. An early effect does not trigger stopping or extending the test.
The window matters in the tool too. In Amplitude, Return On counts a return on the specified day, and Return On or After on that day or later. A 60-day window can be built with Return On (Custom), but only the CRM knows which orders were paid and not refunded, so the share with a second order within 60 days is counted from order dates in the CRM.
How long until you get an answer
Open the calculators and the A/B test planner tab. For planning, use a 25% baseline, the rate among all the shop's newcomers; before launch, refine it using mature cohorts of customers who consented to messages. The calculation targets 33%, an 8 percentage point increase. Minimum detectable effect is entered relative to the baseline: 8 / 25 × 100 = 32%, so enter 32. Leave significance at 5%, power at 80% and two variants. At 100 participants per illustrative 30-day month, enter 3.33 people per day.
The planner rounds the sample up: 473 people per variant, 946 in total and 285 days of recruitment. Add 60 days after the last entrant: about 345 days, or 11.5 months altogether. The calculator's verdict is over eight weeks. At this volume, an 8-point difference will not yield a quick answer.
The calculator advises raising the effect or not running the test: right for a website page, where promotions, seasons and cookies all change over six months. Here groups are split by customer ID once, so the test can run longer, provided the decision date is written down in advance. Both groups go through seasons and promotions together, and changes to the flow are logged and kept separate from the original variant. Suppose the shop starts the test at the end of October. The plan fixes 285 days of recruitment and a final reading 345 days after launch. The planned 80% power requires 473 participants with mature windows in each group. The calendar does not ensure that sample: if traffic is lower or groups are uneven, the final report states the shortfall and uncertainty, without concluding that there is no effect. The observed result is no reason to extend the test.
The plan includes two interim readings without picking a winner. The first three months of recruitment yield about 150 people per group; their windows mature around month five from launch. Six months yield about 300 per group, read around month eight. Using the same normal approximation as the planner, power for a real 25% versus 33% difference is about 37% and 61%. Reducing the control share from 50% to 20% needs more people: under an equal-variance approximation, the multiplier is 0.25 / (0.8 × 0.2) = 1.5625. That means about 1,480 participants, nearly 15 months of recruitment plus 60 days of waiting.
The test has a price. If the reminder adds 8 percentage points, the holdout misses an average of 473 × 0.08 = 37.84, or about 38 second orders. At 6,000 ₸ contribution each, that is 473 × 0.08 × 6,000 = 227,040 ₸, roughly 227,000 ₸ over the test. This conditional estimate of forgone orders does not subtract messaging costs. In return, the shop gets an effect estimate for calculating the value of a new customer.
How to read the numbers while the test runs
Suppose six weeks in, the owner opens the CRM: 8 second orders among even IDs, 6 among odd ones. "It works, send it to everyone." Nobody's 60-day window has closed: the earliest participants have been in the test for 42 days. A two-order gap is also too fragile for a decision, as in the rule for choosing creatives. At this point, look at message delivery and guardrails, without picking a winner.
The planned readings in months five and eight describe only participants whose 60-day windows have closed, keeping their original groups. They show the range of effects compatible with the data, but leave the decision date unchanged. Looking for significance every week and stopping at the first "green" result creates more false wins.
Guardrails are watched weekly: if the threshold for "STOP", unsubscribes or complaints is reached, sending stops under the safety rule, without declaring a winner. Assigned groups are checked separately. If after two months they have 130 and 70 people instead of roughly equal counts, postpone effect analysis and look for duplicate IDs, an assignment error or a filter that removed people after the split. If a reminder went to an odd ID, the sending broke, not the split.
A subscription without a trap
Back to the subscription from the post. The idea itself is fine: a kilogram arrives every month without reminders, and the customer doesn't have to remember to order. The trap is in the exit. A subscription that can be cancelled, paused or skipped for one delivery in the same place and just as quickly as it was started is fine to offer and to test.
A random half of eligible new customers can be shown the subscription offer, and the two groups compared. The subscriber count won't do for this: some subscribers would have ordered anyway and now simply pay automatically. To use the same planner, the metric needs to be a share: for example, at least one further paid and non-refunded order within a preplanned 60 days of the first payment. Each customer counts once in their assigned group. Orders per customer can be a secondary metric, but this two-proportion planner does not calculate its sample size. Guardrails stay the same, with cancellations and "what is this charge" questions added. Set duration and sample size before launch using the baseline rate and expected effect for this particular offer.
Task: a retention test plan for your own business
Take a business with repeat purchases and one retention step that could both add purchases and do harm: a reminder, a discount mailing, a subscription.
If the planner says more than a year, the test has three ways out. You can test a step with a large expected effect, but only if you can explain where such an effect would come from. You can widen the intake: include everyone who reaches the right day after any order, newcomers and regulars alike, and then the main metric is written down again. Or you admit the test is out of reach, roll the step out and watch, noting the date in the log. An underpowered test with a winner picked from early numbers is worse than all three: it gives you confidence you don't have.
Hard cancellation, a discount with a repeatedly resetting timer and a pre-ticked paid checkbox stay out of this plan. The retention they produce is made of people who couldn't leave, and in a spreadsheet they look the same as people who want to stay.
Video: retention processes
A talk by Andrey Zakonov of Yandex on the Epic Growth channel. Compare its ideas, like those from any list of "hacks", on your own customers against a group with no change.