How to deal with visual inspection AI being a black box
What can and cannot be managed about visual inspection AI — operational management under ISO/IEC 42001
“We comply with ISO 13485, the quality management standard for medical devices. If we use AI for visual inspection, how do we manage the AI model?” “We want AI for inspecting automotive parts, but can we document the grounds for the judgement?”
We get asked this. What the questions have in common is bafflement about where AI sits within an existing quality management system, how to run it, and what to keep as controlled documents.
ISO 13485 is a strict QMS standard covering the design and manufacture of medical devices, with high demands on traceability and change control. In the automotive industry there is a long culture of tying the grounds for an inspection judgement to a process FMEA and an inspection standard. Both are worlds built on making the grounds — why it was judged that way — explicit and documented. Bring in something often called a black box, and not knowing how to join up the management of it is a natural response.
As one answer, this article sets out what a person can manage in visual inspection AI, and how to manage it along the lines of ISO/IEC 42001. It follows on from our earlier column, AI standards and guidelines: a practical guide; this one is for people who have already adopted AI and are running it, and focuses on the substance of the management.
What this article covers
- The real difference between what can and cannot be managed about AI.
- The concrete items to manage in order to run visual inspection AI properly.
- How to connect AI management to ISO 13485 or an automotive QMS.
- Why having operational rules matters at least as much as building the model, when it comes to holding accuracy and improving it.
First: separate what can be managed from what cannot
The first thing that matters in running visual inspection AI properly is understanding exactly what a person can manage and what they cannot. Design management rules without that distinction and you get either expectations that do not match reality, or quality risk from a management item nobody noticed.
What cannot be managed (the uncertainty to accept)
The clearest example is guaranteeing inspection accuracy in advance for every product that will be made. Which anomalies will appear among the products flowing down the line, and how often, cannot in reality be known in advance. So managing to “always guarantee X% accuracy on any product” is impossible in principle.
Nor is it realistic to manage the parameters — the weights — inside the AI model one by one. A neural network has millions to hundreds of millions of parameters, and no person can understand or track each value as a meaning.
On which point, we sometimes hear: could explainable AI (XAI) not make the grounds for the AI’s judgement visible? XAI comes in stages, and it is worth being clear about them.
At the first stage, visualising with a heat map which region the AI attended to when judging is already practical, and is implemented in some visual inspection AI including ours. Being able to see where the anomaly was detected alongside the judgement genuinely helps in checking operation and in supporting an inspector’s decision.
What XAI is ultimately reaching for — explaining, in a form a person can verify and warrant, the logical grounds for why the AI judged that place anomalous — is advancing as research, but there is as yet no established track record or methodology for running it as a routine quality record in an industrial setting such as visual inspection. A heat map shows where it looked; explaining why it judged so means going into the internal structure of the model, and it is too early to treat that as something a person can manage and warrant.
XAI may well change how visual inspection AI is managed in future. But the answer for operational management today is not to lean on the AI explaining itself; it is to design and record the processes a person can manage. That is the position of this article.
What can be managed (the variables people deliberately control)
There are clear management items that people can define, record and improve. Put briefly, the heart of it is designing and managing the process and the criteria by which you confirm that the inspection gives the right result on a defined subject.
That is exactly what ISO/IEC 42001 asks for. Starting from the premise that AI is not perfect, build a management system that reliably manages what people can manage and keeps improving it. That is the right posture towards the uncertainty particular to AI.
The concrete items to manage in visual inspection AI
Below, the main things to manage in running visual inspection AI. Each corresponds to a requirement in ISO/IEC 42001: data governance, the risk management system, human oversight, keeping records.
1. Managing the training data
What the AI is trained on largely decides how accurate it is. Start by writing down the criteria for which images go into training: documented definitions of good and defective, and the conditions an image must meet (capture conditions, quality criteria).
Keeping training data and validation data clearly separate is the precondition for evaluating performance honestly. As the earlier column said, a report that confirms performance using only the training data cannot be relied on.
Versioning the data is essential too. Recording when and on which data a model was trained lets you trace the cause when accuracy changes. The flow for collecting and adding data when an unknown defect appears on the floor should also be settled in advance as a rule.
It also matters to name who labels the images and who checks and approves the labels. Leave it to one person’s habits and the quality criteria wobble when they move on.
2. Managing the training parameters
Training has parameters you set in the software: the number of epochs, the learning rate and so on. Unlike the internal weights, these are things a person deliberately sets and changes, and they can be managed.
The point is to keep a register of which parameter settings produced the model currently in production. Managing the history of changes alongside it lets you analyse, afterwards, what caused a change in performance.
What matters is not leaving the parameters open for anyone to change. Put an approval step in front of a change, and record why it was made and whether performance was checked afterwards.
3. Managing the decision threshold
AI inference comes out as a probability that an image is defective. The threshold sets the line above which it is judged NG.
Setting a threshold looks like an engineer’s tuning work, but it is really a quality policy decision: how many missed defects you will accept, and how much overkill. It should be set with the agreement of the floor, quality assurance and management, and the grounds documented.
Changing it also needs an approval step and a record. Change it informally as “just nudging a number” and the cause of a later performance problem cannot be traced.
4. Accepting a trained model, and managing retraining
When a newly trained model comes out, you need a rule for which metrics count as a pass, and who approves putting it into production. Set pass criteria using precision, recall, F-score and the like, and write down who checks what before approving.
Defining the trigger for retraining is an important item too. Settle the conditions in advance: regularly, monthly or quarterly; when a new kind of defect is confirmed on the floor; when routine verification detects a fall in accuracy.
The loop-back arrangement mentioned in the earlier column belongs here. A mechanism by which the AI trains itself further on the floor risks changes outside anyone’s control, so retraining should happen under control, and the flow of approval before anything reaches production should be kept.
5. How to verify that the inference is still sound
You need a way to check routinely and periodically that the AI is still working correctly. This is the practical form of the human oversight ISO/IEC 42001 asks for.
Concretely: prepare verification sample images whose results are known — master images of good and defective parts — and set a procedure for checking the inference at the start of a shift or at regular intervals. It is close to the role of a limit sample in manual inspection.
Document the criteria for assessing the check (for instance, that every known defective image is judged NG), and keep the verification records. Settle also what happens when a result falls outside the criteria: stopping production, considering retraining.
Items to add from a quality management point of view
Alongside the above, connecting to an ISO 9001-style QMS calls for these as well.
What to do when a nonconformity occurs
Define the corrective and preventive flow for when a defect the AI missed is found downstream or in the field. Record what happened, why it was missed, and what will be done, and feed it into retraining or a review of the threshold where needed.
Training operators, and managing permissions
Set out who may operate the AI system (permissions), what they must be taught (operating procedure, routine verification, what to do when something is wrong), and keep the training records. A system anyone can operate creates a black box of its own.
Change control (cameras, lighting, software updates)
Any change that could affect the AI’s judgement should require re-verification before and after. Replacing or moving a camera or a light, or updating the software, tends to be filed as a hardware matter, but these bear directly on the inference. A change-control rule prevents accuracy shifting without anyone noticing.
Against ISO 13485 and automotive QMS: an answer to the black-box problem
Having set out what can be managed, let us answer the objection at the top head on: AI is a black box, so it cannot be brought into an existing QMS and managed.
The short answer is that “AI is a black box, therefore it cannot be managed” is a misunderstanding. It is true that no person can interpret the weights inside a neural network one by one. But management is not a matter of explaining the inside of the model. The traceability ISO 13485 asks for, and the grounds the automotive industry asks of a process FMEA or an inspection standard, are in essence that you can trace in documents what the input conditions were, what process was followed, and against what criteria the judgement was made. All of which is achievable for AI by applying the management items set out here.
Point by point: the design and development change control ISO 13485 requires is met by the history of parameter changes and the approval flow after retraining. Verification and validation records are met by the acceptance criteria for a trained model and the records of routine verification. Traceability is secured by versioning the training data and keeping the inference logs. And documenting the grounds for a judgement, as the automotive industry asks, is answered by documenting the basis for the threshold, the definition of the verification samples, and the pass/fail criteria, as a set.
The point is this: you cannot show the internal grounds for why the AI judged that pixel NG, but you can document completely the grounds of the process — which data it was trained on, at which threshold, through what verification this model is being run. It is the second that the standards ask for, not the first. In manual inspection, too, nobody can document what went on in the inspector’s head; what can be managed is which limit sample was the reference, which trained person did the work, and by what procedure. Managing AI has the same structure.
So a company already running ISO 13485 or an automotive QMS does not need to rebuild its management framework from scratch. The realistic and correct approach is to add the AI-specific items set out here — the training-data register, the record of parameter changes, the document giving the basis for the threshold, the verification procedure — to the QMS documents you already have: the change-control procedure, the verification record forms, the training records.
In short: what it means to manage what can be managed
Making the model more accurate matters, of course. But however accurate the model, without operational rules for holding and improving that accuracy while it is in use, the accuracy degrades over time.
AI carries an inherent uncertainty. Recognising that, then finding what can be managed, writing it down, recording it and improving it: that is the substance of what ISO/IEC 42001 asks of an AI management system.
Look again at the items set out here and every one of them is a variable a person can judge, record and improve. Rather than leave it to the AI, the people using the AI design the framework of management themselves. That view is the root of keeping AI usable as a tool you can trust.
On your own floor: what is managed, by whom, and how? Starting from that question, we would suggest taking stock of the gaps.
If you are unsure about setting up operational management for visual inspection AI, or about meeting ISO/IEC 42001, do talk to us. We will design operational rules that fit your floor as it actually is.
