Skip to content
Back to Academia

From an article to a practical test

00.07Onboarding and navigation
On this page

The owner of an online coffee store compares five days before a tracking change with five days after: the report shows 90 orders instead of 60. "Sales are up by half. We can increase the budget." The CRM holds 100 paid orders in each period, the store's usual 20 a day, or about 140 a week. The change helped the store see sales it was already making. Increasing ad spend on the strength of that report would cost more without a sounder basis for the decision.

It is the same store as in the retention lesson: an order is 15,000 ₸, with 6,000 ₸ left after the goods and delivery; the reconciliation was invented for this lesson. All figures in this example are made up. The marketer's job starts with the question they wanted to answer when they opened the article about analytics setup.

The register holds 100 orders both before and after. Analytics counts 60, then 90: the upper line stays level, while the lower arrow shows the increase in recorded orders.
The register holds 100 orders both before and after. Analytics counts 60, then 90: the upper line stays level, while the lower arrow shows the increase in recorded orders.

Open an article with a task in mind

It is easy to spend an evening in Academia: read about a CRM, follow a link to events, then move on to advertising. By morning you have three ideas and no decision. A more useful starting point is a note from work: "The CRM has paid orders I cannot find in the channel report." The article now has a purpose: help you find and fix the cause of the gaps.

A task needs someone waiting for the result and a decision that depends on it. At the store, the owner sets the advertising budget, the marketer reconciles orders, and the developer changes event delivery. The marketer can complete the reconciliation; increasing sales depends on a different set of actions. Drawing that boundary helps everyone agree on what the contractor is being paid to do.

If you do not have a project yet, use the test site from the Start track. You can submit a form, trigger a form error and save screenshots. State in your notes that this is a test site. Its data can support a case study about setup, without pretending it is revenue from someone else's business.

Put the test into four rows

HADI stands for Hypothesis, Action, Data, Insights. It gives you an order for working through a question. The card below describes a technical check, so reproducible scenarios and an order reconciliation establish the result. Comparing customer behaviour requires an experiment plan.

StepWhat to record for the store
H, hypothesisSome paid orders go missing on their way to analytics; first find what the missing orders have in common
A, actionMatch orders by ID, reproduce the error and change event delivery where the error occurs
D, dataOrders marked paid in the CRM, events with the same IDs, missing orders, duplicates and amount mismatches
I, insightWhich scenarios are fixed, what remains unmatched and whether the report is ready for budget decisions

"Install a new tool" does not fill the first row yet. It says nothing about the cause. For example, if two orders are sent with the same ID, moving event delivery to a server will preserve the problem. First you need one order for which you can show where the ID changed or disappeared.

The card may change after you read the article. You suspected a delivery failure but found different time zones in the CRM and the report: late payments appeared on the next day. In that case, fixing the comparison period is more useful than implementing the technology in the headline.

Reconcile the same orders

The store's CRM contains created, paid, cancelled and refunded orders. For this check, the team takes paid orders without cancellations or refunds, using the same period, time zone and currency. It then looks up each payment in analytics by order ID. Matching totals can still hide different gaps and duplicates, so you need the list of matching IDs.

Use one row per order: ID, payment time, amount in the CRM, amount in analytics and the reason for any mismatch. Customer names and phone numbers are unnecessary for this reconciliation. Keep amount mismatches separate: an event may have arrived but reported only part of the order value.

Of 100 orders, 60 were found at first and 90 after the fix. If all matched IDs are unique, coverage rose from 60% to 90%, an increase of 30 percentage points. The relative increase is (90 - 60) / 60 = 50%. That figure belongs next to measurement coverage; the number of sales in the CRM is still 100.

For simplicity, every order in this reconciliation costs 15,000 ₸. The CRM therefore holds 1,500,000 ₸ in revenue in each period. Analytics first shows 900,000 ₸, then 1,350,000 ₸. The extra 450,000 ₸ appeared in the report; the store received no extra payments for that amount. The contribution from 100 orders also remains 600,000 ₸ before other costs.

Decide what finished means

For an event collection fix, the schedule depends on the scenarios and the check against production data. Before the work starts, the team lists payment methods and repeat delivery of the same event, assigns an owner for reconciliation, and agrees how to handle the remaining mismatches. After the fix, the marketer works through the list with the developer again.

Our 90 matches leave ten missing orders. Group them by cause: an order is still being processed, delivery failed, an ID differs, or the event was never included in the setup. Completion depends on the agreed acceptance criterion, not on the new rate looking better than the old one. If some orders remain outside analytics, describe that limitation alongside the report.

Repeat purchases run on a different schedule. The same store measures them within 60 days of the first payment. It sends a reminder 21 days after delivery and waits for participants' observation windows to close. A week can establish whether the messages are being sent correctly; the answer about 60-day repeat purchases cannot have arrived yet. The retention test includes the timing calculation.

Keep a record you can return to

A useful reconciliation note fits into a paragraph: "There are 100 paid orders in each period. Analytics contains 60 and 90, with no duplicates among them. The delivery error in the documented scenario is fixed. We are investigating the ten remaining missing orders by ID. CRM revenue did not change." Attach the table and the steps for reproducing the error.

The owner now knows how to use the report. The marketer continues investigating gaps, the developer gets reproducible cases, and the budget discussion uses spend, payments and contribution from the CRM. For the next article, the team takes the next unanswered question and keeps the link in the same card.

Exercise: a card for your article

Choose one Academia article that serves a task you are working on. Write the problem in the client's words, the decision they need to make and the four HADI rows. Name the data source, unit of measurement, schedule and completion criterion. If it is a test project, say so in the first row.

Then correct the owner's claim from the opening: "Sales are up by half. We can increase the budget." Your response needs both analytics revenue figures, the CRM figure and the next action.

Analytics showed 900,000 ₸ and now shows 1,350,000 ₸; the CRM holds 1,500,000 ₸ for each period. Sales did not grow; recording did. This data is not enough for a budget decision: the example provides neither spend nor a link between orders and channels. The next action on the card is to investigate the ten missing orders and reconcile confirmed payments with spend. Reading has produced a task another person can review and accept.

Need help applying this to your analytics? Discuss your project with Goodlabs →