A vendor post crossed our feed recently with an argument we agree with. A lawyer who checks an AI’s final output has not supervised it. Supervision means deciding in advance what the system is for, where it must stop, what it records, and when a human steps in.
That is right: sign off only the output, and you inherit decisions you never saw being made.
Then the same post described its product. It was an intake assistant for law firms, and the pitch was that it applies the firm’s criteria for who qualifies as a client, but does not decide whether anyone has a legal case.
Those two claims cannot both hold.
The screen is the decision
Screening someone out is a decision about whether they have a case. It is made earlier, and it is made more quietly, but it is the same decision.
Picture a caller who was dismissed 80 days ago. The Employment Relations Act 2000 gives an employee 90 days to raise most personal grievances with their employer. The criteria say “not a fit,” and no lawyer ever sees it. The caller rings off believing they have been told something about their case, and in a sense they have. Ten days later the window closes, and reopening it takes the employer’s consent or the Authority’s leave.
The most consequential legal call in the whole exchange was made by the part of the system that was meant to be doing admin.
A stop the tool controls is not a stop
The post said that when something falls outside the assistant’s role, the workflow stops. That works only if the system knows it is outside its role. The thing being supervised decides when supervision begins.
A stop point worth the name is one the tool does not control: a mention of dismissal, a deadline, or signs of distress sends the matter to a person, and so does every declined enquiry.
Review the rejections
The post also said there was no need to review every interaction. That much is fair, but it needs a replacement, not a blank. Full review is unrealistic. Logging and a sampled audit are not, and the sample should lean toward the rejections, because that is where the harm sits. An accepted enquiry reaches a lawyer, who can still catch a bad call. A rejected one reaches no one.
Employers have the same problem
Employers should read this as their problem too. Swap the intake assistant for an AI-assisted shortlist or a performance screen, and nothing in the argument changes. Defining the tool’s job does not remove the judgement if the job contains it.
Flag an employee, and the question becomes what a fair and reasonable employer could have done in all the circumstances (s 103A). Filter out a candidate, and the Human Rights Act 1993 looks at what happened to them, not at which step of the process did it. “The system applied the criteria” answers neither.
Nor does “a human checked the result.” A manager who signs off a shortlist sees the names on it, not the names that never reached it.
Where to look
For any AI tool that sorts people — callers, candidates, or employees — three questions do most of the work:
- Where in the process is the judgement made?
- What sends a case to a person, and does the tool control that trigger?
- What is logged, and who reads the rejections?
We have written before about where human judgement has to sit, and why it has to leave a trail (The Human Judgement Principle), and about why a tool you have not named is one you cannot oversee (You Cannot Oversee, Verify, or Defend a Tool You Have Not Named).
The question worth asking
So the useful question is not whether a human checks the output. It is where the judgement sits in the process, and whether a human is standing there when it is made.
- Employment Relations Act 2000, s 103A (test of justification) and s 114(1), (3), (4), and (7) (raising a personal grievance; the 90-day employee notification period)
- Human Rights Act 1993
Lex Praxis advises New Zealand employers on employment law and AI governance. This article is general commentary and does not constitute legal advice. For advice specific to your circumstances, contact us directly.