A manager receives an attractive review digest: three main themes, percentages and an explanation for delivery delays. When a colleague opens the source texts, the customers have only mentioned late delivery. The model supplied the cause. A useful report has started to resemble a diagnosis even though the team has not investigated delivery operations.
A store needs a weekly digest of review themes, with verified observations passed to the relevant owners. AI proposes groups and a draft. A person checks the meaning, an ordinary calculation counts records, and the service owner decides what to investigate. All figures below are illustrative.
Define an output that can be accepted
The report should include the period, number of included reviews, themes, linked IDs, frequencies and brief observations. If a complaint's cause is absent, record that it needs investigation. Possible explanations can be listed separately as questions to test.
The process owner sets the boundary: this analyses texts from an agreed source, rather than studying every buyer. Someone who leaves a review already differs from someone who says nothing. A theme's frequency in this group does not establish its prevalence across the customer base.
Acceptance has checkable requirements. Each theme rests on source records. A quotation preserves their meaning. The count can be reproduced from IDs. An unsupported cause does not become a fact. Failure on one of these requirements cannot be offset by a high score for clear writing.
Organise the input first
Define the source, period boundaries, eligible record types and duplicate-removal rules. The same message appearing twice in an export should not double its frequency. Two different messages from one person require an agreed rule: are you counting messages, enquiries or buyers? In our example, the unit is an individual review with a stable ID.
Retain the original text, language and source connection in an approved working environment. Give the model only information needed for the task. Agree on access and data scope before starting; names, contact details and order information are not needed for every theme analysis.
Long reviews, empty records and messages with several themes must not disappear without explanation. If a record is excluded, retain the reason and count such cases. Otherwise a new tool might look more accurate simply because it stopped processing difficult texts.
Separate model, rule and human responsibilities
| Step | Who or what performs it | What remains to be checked |
|---|---|---|
| Selection and exact duplicate removal | Agreed rules | Period, source and exclusion count |
| Theme proposals | Model following instructions | Theme connections to source IDs |
| Counting | Formula or program using checked IDs | Uniqueness and denominator |
| Semantic acceptance | Responsible employee | Quotations, limitations and absence of new facts |
| Passing on observations | Review owner | Recipient and next action |
Automatic file retrieval can come later. First establish that the team can produce a useful, verifiable result. Moving data without human involvement will not fix unclear categories.
Allow several themes and an uncertainty state in the instructions. “The order arrived late, but the employee helped resolve it” contains delivery timing and support. A single “negative” label loses some meaning. “Late because there were not enough couriers” adds a cause absent from the source.
Count reviews rather than matching phrases
The checked set contains 20 unique reviews. Delivery appears in 12, packaging in eight and support in six. A review may belong to several themes; within each theme, its ID counts once.
| Theme | Unique reviews mentioning it | Share of 20 reviews |
|---|---|---|
| Delivery | 12 | 60% |
| Packaging | 8 | 40% |
| Support | 6 | 30% |
The shares total 130% because the categories overlap. Arbitrarily dividing by 26 mentions to force a 100% total would create a different measure. R01, for example, may belong to both delivery and support. It remains one record in the overall review count.
Theme frequency does not establish sentiment. Support can be praised or criticised. If that matters to the decision, add and evaluate a separate rule while retaining the source wording. Do not treat all six mentions as complaints without reading them.
Prepare an evaluation set
Collect varied inputs: short and long, clear and ambiguous, with several themes or insufficient information. The languages used need reviewers who understand them. Define acceptable answers for ambiguous cases separately: exact wording need not always match a reference.
OpenAI's evaluation guide recommends grounding evaluation in real tasks and comparing automated assessments with human judgement. For this process, a set of error types and expected properties helps compare new instructions with the previous version.
Separate tuning examples from evaluation inputs. If you change the request after every result on the same twenty texts, better performance there says little about the next week. After tuning, save the version and check new cases. Repeating some inputs helps expose inconsistent responses.
In the initial evaluation set, 18 of 20 reviews were tagged without errors. In two, the model added a cause for the complaint. The fully accepted share is 90% for this set alone. Before inclusion in the final table, all 20 records were reviewed and both errors corrected. The theme table above shows the result after that acceptance step.
Measure the path to release
Suppose a manual weekly digest takes three hours. The new process requires 30 minutes of input preparation, ten minutes to obtain the draft, an hour of review and twenty minutes of corrections. The total is two hours. This is an illustrative measurement for the same volume and accepted quality, including rejected attempts.
Across four monthly releases, manual work takes twelve hours and the new process eight. At an internal value of 5,000 ₸ an hour, that is 60,000 ₸ and 40,000 ₸ of capacity. The hourly value is chosen for the calculation and does not describe a market rate.
Suppose one monthly maintenance hour is needed in addition to the time for each release, adding 5,000 ₸ of capacity. The tool has a separate allowance of 10,000 ₸ a month, without reference to any product's tariff. The new process totals 40,000 + 5,000 + 10,000 = 55,000 ₸ under this valuation. The difference from the manual process is 5,000 ₸ a month.
Initial setup takes another three hours, valued at 15,000 ₸. The first month of the new process therefore totals 70,000 ₸ against the manual 60,000 ₸. The first month pays for setup; the saving comes from the recurring editions. Record actual payments separately from the value of time.
Assign an owner and a recovery path
The employee releasing the digest needs the source records, criteria and authority to stop it being sent. Keep the instruction version, tool, review time, errors and corrections in a log. A decline can then be connected to an input or process change.
If the file has not arrived, records are missing or nobody can check the meaning, the report does not become ready merely because it is due. Define the manual route, recipient notification and recovery owner. Repeated failures call for a review of scope and method.
Expansion starts with another checkable task. Automatically delivering a draft to an editor differs from sending conclusions to customers. Publishing or changing budgets needs permissions and acceptance conditions for that particular action. Successful review grouping has not tested them.
Exercise: what is known after the pilot?
Choose a weekly process of your own that you want AI to handle partly or entirely. Record the counting unit, source data, acceptance criteria, owner, total working time and additional cash expenses. Identify the step a person will retain and how they will check the result.
The team shows that 18 of 20 reviews were tagged without errors and proposes sending future reports without review. It also calls the table's 130% total an error and wants to divide every theme count by 26. How should these proposals be assessed, and when does the calculation cover initial setup?
Two results contained unsupported causes, so the 90% figure does not justify removing review. The process needs correction and evaluation on new inputs. Themes overlap: each share answers how many of the twenty reviews mention that theme. Dividing by 26 changes the measure's meaning. Keep the denominator and explain the overlap explicitly.
Under the illustrative valuation, the recurring new process costs 5,000 ₸ less per month than the manual one. Three months of that difference cover the initial 15,000 ₸ of capacity if volume, quality and effort stay the same. Both options total 180,000 ₸ across three months: manual work is 3 × 60,000, while the new process is 15,000 + 3 × 55,000. This is a threshold for comparing resources, not a promise of cash returning to the account or permission to remove the editor.
A completed process record lets a colleague reproduce the count and identify who accepts the release. Without source data or time for review, quality assessment is not yet supported. If labour and payments share one unexplained total, separate them before deciding the budget. Resolve these gaps before expanding the pilot.