Measuring adoption after training.

Goal

Find out whether AI training changed how people work, and express the result as an ROI figure you can defend.

Who it is for

Owners, managers and HR or L&D leads in New Zealand organisations that have trained staff on ChatGPT, Claude, Copilot or Gemini.

Time

30 minutes.

You need

Any AI chat tool and the fictional survey results below. Later, your own admin usage report and ten timed tasks.

You will produce

A 30/60/90-day measurement plan and a first ROI estimate with every assumption written down.

Why measure adoption after training?

Because attendance is not adoption. Training only pays off if people use the tools on the tasks you trained them for, and check the output properly. Datacom’s 2026 State of AI Index, a vendor survey of senior leaders at New Zealand organisations with 100 or more staff, found 66% had provided AI skills training in the past year, yet lack of internal capability or skills was still the most cited barrier to scaling AI, at 20%. MBIE’s research on SME adoption reached a similar conclusion: confidence and capability are key factors in whether businesses adopt AI and expand its use.

What should you measure, and when?

Measure in three layers, each answering a different question, on a 30/60/90-day rhythm:

Business plans now include some admin reporting. As listed on 19 October 2026, ChatGPT Business offers “usage visibility and spend controls”, Microsoft’s Copilot Business plan lists “analytics to measure adoption, usage, and business impact”, and Claude’s Team and Enterprise plans list usage analytics. These show activity. None of them shows whether the work got faster or better.

Exercise: analyse a post-training survey

This is a fictional pulse survey from a 12-person team, three weeks after training. The target task was client renewal summaries, done by the eight account managers.

Illustrative survey results. Fictional insurance brokerage, Tauranga.
Target task: renewal summaries (account managers only).

id,role,used_last_week,used_on_target_task,uses_per_week,mins_saved_per_use,confidence_1to5,main_blocker
A1,account manager,yes,yes,8,10,4,none
A2,account manager,yes,yes,12,6,3,checking policy wording takes time
A3,account manager,no,no,0,0,2,not sure what is allowed with client data
A4,account manager,yes,no,3,5,3,only use it for emails
A5,account manager,yes,yes,10,12,5,none
A6,account manager,yes,yes,6,8,4,output too long
A7,account manager,no,no,0,0,1,no time to practise
A8,account manager,yes,yes,15,20,5,none
A9,claims,yes,no,4,5,3,claims system not connected
A10,claims,yes,no,2,4,2,not sure what is allowed with client data
A11,admin,yes,no,5,6,4,none
A12,admin,no,no,0,0,2,forgot login

1. Run the analysis. Paste the data and the worked example prompt into ChatGPT, Claude, Copilot or Gemini.
2. Check two numbers by hand. Recount the weekly active users and the target-task users yourself. If the tool got either wrong, tell it and ask it to redo the table.
3. Compare the two savings figures. Note how far the per-person total and the median-based estimate differ, and which respondent causes the gap.
4. Read the blockers as tasks. Turn each blocker group into one action with an owner, such as “send the client data rule to all staff”.
5. Plan the timed sample. Pick ten real renewal summaries for day 60 and decide who times them, with and without AI, including checking time.
6. Draft your 90-day plan. Copy the table above and fill in your own target tasks, sources and dates.

Worked example: the analysis prompt

Below are pulse survey results from a team three weeks after AI
training. The target task is renewal summaries, done by account
managers only.

Calculate and show your working:
1. Weekly active rate (used_last_week = yes) across all staff.
2. Share of account managers using AI on the target task.
3. Median minutes saved per use among target-task users.
4. Weekly hours saved on the target task two ways: (a) summing each
   person's uses x minutes, (b) total uses x the median.
5. Blockers grouped into themes, with counts.
6. Any responses that look inconsistent or need follow-up.
Do not estimate anything the data does not contain.

[paste survey data]

Example output (illustrative and abridged; your tool’s wording will differ):

1. Weekly active: 9 of 12 staff (75%).
2. Target task: 5 of 8 account managers (62.5%).
3. Median minutes saved per use: 10 (values 6, 8, 10, 12, 20).
4. (a) 620 minutes, about 10.3 hours a week.
   (b) 51 uses x 10 minutes = 510 minutes, about 8.5 hours a week.
   A8 alone accounts for 300 of the 620 minutes.
5. Blockers: unclear data rules (2), output quality or checking (2),
   tool does not fit the work (2), no time to practise (1), login (1).
6. Follow up A8: 20 minutes saved on each of 15 uses is possible
   but should be timed. A10 used the tool while unsure what client
   data is allowed: confirm the rule with them first.

Worked numbers: from survey to ROI

Self-reports are a starting point, not the answer. Suppose the day-60 timed sample shows renewal summaries actually save 7 minutes each once checking is included, not the 10 people reported. Using illustrative assumptions:

• Time saved: 51 uses a week x 7 minutes = 357 minutes, about 5.95 hours a week.
• Value: 5.95 hours x an assumed loaded staff cost of $70 (NZD) an hour = $416.50 a week; over 44 working weeks, $18,326.
• Costs: 12 licences at an assumed $50 a month = $7,200 a year, plus training time of 12 people x 4 hours x $70 = $3,360.
• First-year net benefit: $18,326 minus $10,560 = $7,766.

The self-reported total of 10.3 hours was about 1.7 times the timed figure. Our AI ROI calculator runs the same arithmetic with your numbers, and calculating time saved by AI covers timing methods in more depth.

Where this goes wrong

• Counting logins as success. Activity on easy tasks hides zero change on the work you trained for.
• Averages driven by one enthusiast. Use medians and look at the spread.
• Ignoring review time. If checking an AI draft takes longer than writing it, the saving is negative. Track error rates with a quality review checklist.
• Surveying once. A day-30 spike often fades by day 90 without follow-up. An internal AI champion keeps momentum.
• Treating blockers as attitude. Most are rules, access or fit, and they are fixable.

Check your work

• I recounted the active and target-task numbers by hand and they match.
• I can explain why the median-based estimate is lower than the per-person total.
• Every blocker has an action and an owner.
• I have chosen ten real tasks for a timed sample at day 60, including checking time.
• My ROI estimate lists every assumption and includes licences for unused seats.
• My 90-day plan names the decision we will make at the end.

Where to go next

Next in the pathway: using AI for market research. To build the survey itself, start from our AI adoption survey, and for judging a pilot rather than a rollout, see evaluating an AI pilot. Engineers measuring output quality over time should read monitoring AI quality. If adoption has stalled, our AI training for business teams is built around your own documents and target tasks.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.