The content team used to prepare a piece in an hour and spend another twenty minutes checking it. With AI, the draft appeared in ten minutes, but the editor spent fifty verifying it and fifteen correcting it. The rollout presentation said “Six times faster”. The manager asked whether that referred to the draft or to a piece ready for release.
At the same final quality, the complete cycle fell from 80 to 75 minutes. The saving was five minutes, or 6.25%. This is an illustrative process, without a claim about any particular model. The rollout decision depends on what happens between the first response and an accepted piece.
In a working measurement, record time by stage, including failed attempts that had to be rewritten or discarded.
Choose work with an accountable reviewer
“Introduce AI” has no defined endpoint. Preparing a product listing does: the editor receives the required format, checks specifications against the source and approves publication. Reviewing enquiries has a different outcome: themes connect to source records, counts can be reproduced, and unknown causes are not presented as facts.
Describe the input, output, frequency, doer and approver. Then identify errors that require the result to be returned. An attractive headline cannot compensate for a false price. A missing product limitation can make otherwise clear, concise copy unusable.
A limited task whose sources are available to the reviewer is useful for a first pilot. If the existing manual procedure keeps changing and nobody has agreed on quality, the comparison will be unreliable. Agree on the result before speeding up its production.
Map the current process
Record the journey from assignment to accepted version: collecting facts, preparing, checking, correcting and handing over. Track queueing separately from working time. Two elapsed hours might include ten minutes of work and an hour and fifty minutes of waiting. Reducing delivery time and assessing staff capacity require different measures.
Include ordinary materials and difficult cases: a missing condition, conflicting sources, several languages or an unsuitable style example. If the manual work covered hard texts and the AI got easy ones, you compared the texts, not the ways of working.
Keep the existing working method. It supports both comparison and recovery from failures. A process transition defines what changes and when the old mandatory work ends. Adding a generator without changing review can simply create another queue.
Separate tuning from evaluation
Use one set to refine the instructions: which facts to use, which format to return and how to mark missing information. Keep another set for evaluation. Editing the request after each response on the evaluation set means it can no longer be presented as an independent test.
Where possible, the assessor should not know how the text was produced. They use the same criteria, record material errors and measure the time needed to reach acceptance. For a generative task, repeat some inputs: one successful response does not establish consistency.
The NIST AI RMF links evaluation to the conditions of use and monitoring after deployment. In a working pilot, keep the tested inputs, instruction versions, execution method and observed limitations. When the model, sources or product rules change, repeat the affected checks.
Make review possible
“A person will check it” hides several requirements. That person needs competence, time, a current source and authority to return the work. Check a price against the active offer, reproduce a calculation and find a quotation in its original source. A link supplied by the model also needs checking.
A second model can provide another error-finding pass. Agreement between responses does not replace a source. If two systems repeat the wrong pilot duration, the customer still receives the wrong promise.
Separate permissions. Drafting an email, selecting recipients and sending it have different consequences. Evaluating a text-drafting pilot does not justify immediately allowing the system to change audiences or publish without an editor. The process owner defines who releases the result and what happens when there is uncertainty.
Russian and Kazakh versions need a reviewer who understands the language and the offer. One successful translation does not establish quality in another language. Return unfamiliar terms and ambiguous wording for investigation instead of accepting them for speed.
Calculate capacity and additional cash
Apply the 80 and 75 minutes to 120 accepted pieces a month. Assume the time includes all attempts, review and corrections, and final quality is the same. Manual work takes 9,600 minutes, or 160 hours. The AI process takes 9,000 minutes, or 150 hours. Ten hours are released.
For the capacity estimate, use an internal hourly value of 5,000 ₸. The manual process uses 800,000 ₸ worth of capacity; the new process uses 750,000 ₸. The 50,000 ₸ difference does not yet include maintenance or tool expenses.
The pilot budget separately allows 30,000 ₸ a month for the tool and other paid expenses. Maintenance takes five hours monthly: investigating errors, updating instructions and repeating checks. At the same hourly value, that is 25,000 ₸ of capacity.
| Monthly calculation | Manual process | AI process |
|---|---|---|
| Time to prepare the accepted volume | 160 hours | 150 hours |
| Value of that time | 800,000 ₸ | 750,000 ₸ |
| Additional maintenance | Included in the existing process | 25,000 ₸ |
| Additional paid expenses | 0 ₸ | 30,000 ₸ |
| Labour value plus additional expenses | 800,000 ₸ | 805,000 ₸ |
Under these assumptions, the new option costs 5,000 ₸ more using the stated valuation. One-off pilot preparation is not yet included and must be added separately. If maintenance is already within the measured 150 hours, do not add five hours again.
The table compares the value of resources used. With unchanged salaries, 50,000 ₸ does not return to the bank account. The team releases ten preparation hours, spends five on maintenance and pays an additional 30,000 ₸ for the tool. The manager must decide what useful work the remaining five hours can support and whether it justifies the new expense.
Check the data boundary
For each input, establish whether the task needs it and who may pass it to the chosen service. Storage, access and data-use conditions depend on the product, workspace and configuration. Before live operation, the person responsible for that environment checks them; another plan's marketing description is not sufficient evidence.
Initial examples can be prepared without customer data. Retain the language and task complexity, or the evaluation will be too easy. Removing a name alone may leave other identifiers, so the input contents need review too.
Keep a clear route back to manual work. If a source is unavailable or quality falls, staff should know where the task goes, what to tell its requester and how to record the failure. One successful draft cannot evaluate this part of the process.
Exercise: only drafting became faster
After an instruction change, drafting takes five minutes instead of ten. Review and corrections remain at 50 and 15 minutes. Assume new inputs confirm this timing and the previous quality. Recalculate the full cycle and the monthly example with 120 pieces, five maintenance hours and 30,000 ₸ of additional expenses.
The full cycle is 5 + 50 + 15 = 70 minutes. Against the manual 80 minutes, the saving is ten minutes, or 12.5%. Across 120 pieces, that releases twenty hours, valued at 100,000 ₸. After five maintenance hours and the additional 30,000 ₸, the calculated difference favours the new process by 45,000 ₸ before one-off setup. With fixed salaries, this still values capacity use; it is not a separate line of additional profit.
Prepare the same record for your pilot: accepted volume, quality, total labour, additional payments, maintenance and the work that will use the released hours. Expansion depends on those conditions and fresh evaluation, rather than the speed of the first response.