AI is already helping with work that affects people's rights, money, jobs, health and access to public services. It drafts legal letters, reconciles accounts, summarises evidence, scores applications and suggests what to do next. The rules written specifically for AI are still being developed, in South Africa and almost everywhere else. Waiting for a complete rulebook can feel like the cautious option. It is not, because the accountability for these decisions already exists. Professional duties, privacy law, contracts and the ordinary expectation that an organisation can explain what it did all apply today.
The most common answer to "how do we keep AI safe?" is a phrase: human in the loop. It sounds reassuring and means almost nothing on its own. It can describe a specialist who checks every source before signing. It can also describe a team leader who approves forty AI-prepared cases in an hour because the queue is long and nobody has ever been asked what they checked. Both are humans. Both are in the loop. Only one of them is a control.
This six-part series is about the design questions that apply whenever AI influences a consequential outcome, whether that outcome is a court submission, a credit decision, an insurance claim, a job application or a government benefit. It starts with the person who is supposed to be watching.
This series is general information about governing AI in organisations, not legal advice. Where it describes court judgments or legislation, it summarises them for a non-legal audience; read the primary sources and take advice from a qualified legal practitioner before relying on any interpretation for your organisation.
A near miss at Spekboom
Spekboom Finance is fictional, and its processes and figures are illustrative. Spekboom is a South African consumer lender and collections business, with its head office in Umhlanga and about 900 staff. It offers personal loans and vehicle finance, runs its own collections team, and has a small in-house legal team that prepares letters of demand and briefs external attorneys for debt recovery and repossession matters.
In mid-2026 the legal team started using an AI research assistant to find authorities faster. In September 2026, a paralegal used it to prepare a letter of demand for a disputed vehicle finance account. The letter was well written. It cited three judgments in support of Spekboom's position. Before it went out, the senior legal adviser checked each citation against a law report database, as she always did. One of the three judgments did not exist. The case name was plausible, the citation format was correct, and the summary of what it supposedly decided fitted the argument perfectly.
Nothing went out, and nobody was harmed. But the adviser asked an uncomfortable question at the next management meeting: what would have happened if she had been on leave? The paralegal had reviewed the letter. The paralegal was a human in the loop. The loop would not have caught it.
That near miss is the thread through this series. Spekboom has five AI uses in flight, from call summaries to a tool that helps decide hardship applications, and the same question applies to all of them.
What the courts have already said
South African courts have now dealt with non-existent authorities in legal work several times, and the pattern is instructive for anyone outside the legal profession too.
The clearest judgment is Mavundla v MEC: Department of Co-Operative Government and Traditional Affairs KwaZulu-Natal and Others (7940/2024P) [2025] ZAKZPHC 2, handed down in the KwaZulu-Natal Division in Pietermaritzburg in January 2025. In an application for leave to appeal, the supplementary notice of appeal cited nine authorities. The court records that only two of them could be found to exist, and one of those was cited incorrectly. The notice had been prepared by a candidate legal practitioner, who denied using ChatGPT. The court said the real source of the authorities remained unknown. It is important to be precise here: the court did not find that AI had been used.
What the court did do was test the tool itself. The judgment records that the judge asked ChatGPT about two of the non-existent cases, and that it wrongly confirmed they existed. The court said this illustrated the tool's unreliability as a source of legal research, and stated that "relying on AI technologies when doing legal research is irresponsible and downright unprofessional". It also observed that the most basic check by whoever signed and supervised the work would have discovered the problem. Leave to appeal was dismissed, the firm was ordered to pay certain additional costs itself, and the judgment was referred to the Legal Practice Council for its attention and further action. The court made no finding of misconduct against any individual; that is for the Council to investigate.
Mavundla was not the first or the last. The judgment itself discusses Parker v Forsyth NO and Others [2023] ZAGPRD 1, a Johannesburg Regional Court matter in which fictitious, ChatGPT-sourced authorities were given to the other side's attorneys and the court made a punitive costs order. In Northbound Processing (Pty) Ltd v South African Diamond and Precious Metals Regulator and Others, decided in the Gauteng Division in June 2025, heads of argument contained incorrect or non-existent citations produced with an AI research tool; the court accepted there was no deliberate attempt to mislead and referred the matter to the Legal Practice Council for investigation.
The same failure has reached national policy. In April 2026 the Minister of Communications and Digital Technologies withdrew the draft National AI Policy after its reference list was found to include fictitious, AI-generated citations, saying the lapse showed why vigilant human oversight over the use of AI is critical (SAnews).
None of these involved a lack of humans. Every document passed through qualified, intelligent people. What was missing was a check that actually tested the output against something outside the tool.
Presence is not control
So what separates a human who is present from a human who is in control? In our view, meaningful oversight needs a person who:
- has the competence to identify material errors in this kind of output;
- has access to the original evidence, not only the AI's summary of it;
- has enough time to do the review properly;
- is authorised to reject or change the output, not only to approve it;
- is not measured in a way that rewards automatic acceptance, such as cases cleared per hour with no quality measure;
- can explain the checks they performed; and
- accepts responsibility for the final decision.
Each of these is a design decision, not a personal virtue. If a reviewer sees only the AI's summary, they cannot find what the summary left out, however diligent they are. If the workflow allows thirty seconds per case, the review will take thirty seconds. If the screen offers "Approve" and nothing else, the reviewer cannot correct. If performance targets count throughput and nothing else, acceptance is what the organisation is paying for.
The seventh point matters most for accountability. A decision with a human name on it should be a decision that person is willing to defend. If the honest answer to "why did you approve this?" is "the system recommended it", the human was a signature, not a reviewer.
Spekboom's paralegal had the competence and the time. What the paralegal lacked was a defined check: nobody had said that every authority must be found in a primary source before a letter goes out. The adviser had that habit. The process did not.
Verification must be independent
The most tempting check is also the weakest: asking the same tool, or a similar one, whether its output is right. That is what made the court's own experiment in Mavundla so telling. A system that can generate a plausible non-existent case can equally generate a plausible confirmation of it. Asking the model is not verification. It is the same source, consulted twice.
For legal authorities, independent verification means:
- retrieving the reported judgment from an authoritative legal database or law report;
- checking the case number and the court;
- reading the relevant passage, not a summary of it; and
- deciding whether the passage actually supports the proposition it is cited for.
That last step catches a quieter failure than invention. A real case, correctly cited, can still be quoted for something it does not say.
The same principle applies outside law. A reconciliation is verified against bank statements and the ledger, not by asking the tool to recheck its sums. A summary of a customer call is verified against the recording or the notes, not against a second summary. A claim that a document contains a clause is verified by finding the clause. The general rule: verify against the evidence the output is about, using a source the AI did not produce.
The check has to leave a trace
An organisation cannot show that oversight happened if the only evidence is that a box was ticked. The flow we recommend for any AI output that influences a consequential decision is short:
- The AI proposes an output.
- A competent person checks it against the primary evidence.
- If there is a material issue, they reject it, correct it or escalate it.
- A person makes the decision.
- The checks performed and the decision are retained.
The fifth step is what turns a habit into a control. "Checked" on its own records nothing. "Three authorities verified in a law report database on 14 September; one not found and removed; letter approved by the senior legal adviser" records the work. Part 3 goes into what a full evidence record should contain.
What Spekboom changed
Spekboom did not ban the research assistant. It was saving the legal team real time on the first draft of research. Instead it changed three things.
Every AI-produced authority or figure is checked against a primary source by a named reviewer, and the check is recorded. For legal work, that means the case is found in a law report database and the relevant passage is read. For reconciliations, it means the figures are traced to bank statements and the ledger. The record says what was checked, by whom and when.
The reviewer's authority and time were made explicit. The paralegal prepares; the senior legal adviser or an external attorney reviews and signs, and has the authority to send the work back. The legal team's targets now include a quality measure, not only turnaround.
Training covered the failure, not only the tool. Every person using the assistant was shown the near miss and the published judgments, so that they understood the risk is not a typo. It is a confident, well-formatted invention that looks exactly like the real thing.
Common mistakes
- Treating a human's presence as the control. A person who cannot see the evidence, has no time, or cannot change the outcome is a signature, not a review.
- Asking the AI to check itself. The same source consulted twice is not verification, as the court's own test in Mavundla showed.
- Reviewing the summary instead of the evidence. The reviewer must be able to see what the summary left out.
- Measuring reviewers only on throughput. If acceptance is what gets counted, acceptance is what you will get.
- Recording "approved" without recording the check. Without a trace of what was verified, oversight cannot be shown after the fact.
- Assuming the risk is only in legal work. The same failure applies to reconciliations, summaries, assessments and policy documents.
Questions to ask this week
- Where in our organisation does AI output influence a decision about a person, money or a legal position?
- For each of those, who reviews the output, and what exactly do they check it against?
- Do our reviewers see the original evidence, or only the AI's version of it?
- Can each reviewer reject or change the output, and do they have the time to?
- How are reviewers measured, and does that measure reward acceptance?
- If a decision were challenged tomorrow, could we show what was checked, by whom and when?
What to take from this part
Presence is not control. Meaningful oversight needs competence, evidence, time, authority and an honest measure of the reviewer's work.
Verification has to be independent of the tool. Check against the primary evidence, using a source the AI did not produce.
The check has to leave a trace. Record what was verified, by whom and when.
The courts have already set the expectation. Generated research must be checked against authoritative sources, and the person who signs carries the responsibility.
Next in the series: AI may assist the decision; it must not inherit authority.
How CloudNala can help
We help organisations turn "human in the loop" into oversight that holds up: mapping where AI output influences consequential decisions, designing review steps with the right evidence, time and authority, building the verification checks and records into the workflow, and setting measures that reward quality rather than throughput. We are not a law firm, so we work alongside your legal advisers on anything that turns on legal interpretation.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za