How I test whether this is safe to use

The system drafts. People approve.

Nothing I build writes into your controlled documents. Anything it produces reaches your quality management system the same way anything else does, through your own change process, with your own people's names on it. If a workflow only works by skipping that, I will not build it.

What it has to pass before anyone relies on it

Before a system goes anywhere near live work, we agree a test set in advance: 25 to 40 real questions from your site with source answers your people agree are correct, or 30 to 50 real record packs. Real ones, not examples I have written. Then it is measured against them, and you see the result rather than take it on trust.

The gates it has to clear:

• Every substantive answer cites its source and its revision. 100%, not most. An answer without a traceable source is a failure whether or not it happens to be right.
• At least 95% of claims are supported by the document they cite. Measured, not assumed.
• Zero unauthorised retrievals. The access tests are run deliberately: we ask it for things the user is not permitted to see and confirm it does not return them.
• Zero material wrong-version answers. This is the one that matters most and the one general-purpose tools fail. A system that answers correctly from a superseded procedure has done the most dangerous thing available to it.
• It abstains when the evidence is missing or contradictory, and routes the contradiction to whoever owns the document, rather than choosing between them.
• It survives a prompt injection test. Instructions hidden in a document do not become instructions to the system.
• The time saving is real and agreed in advance. Usually 30 to 40% off the time it takes to find current approved evidence. If it does not hit the number, it has not worked, whatever else it does well.

Bounded, read only, and in shadow first

The first scope is 1 site, 1 workflow, 1 named decision, a fixed set of source documents, and 3 to 5 named users. Least privilege access. No personal data in a first pilot.

Shadow mode means it runs alongside your existing process while people carry on doing the job the way they do it now, so the 2 can be compared before anything depends on it. There is a named human reviewer, a manual fallback that still works if the system is switched off tomorrow, and a log of what it was asked and what it answered.

There are written stop conditions agreed before we start. One access breach or one material wrong-version answer stops the pilot. So does a false indication that something is complete when it is not, if that reaches live work. So does the thing costing your people more time than it gives back, which is the failure nobody plans for and the one I have seen most often.

You get a system card, a data map, a risk log and the evaluation results. They are yours whether or not the pilot passes.

What I never hand to a machine

Permanently outside anything I build, on any site, at any price:

• Process safety decisions. Permits, isolation, emergency response.
• Set points, recipes, dosing and alarm response.
• Hazard assessment in place of a competent person.
• Deviation classification, root cause and CAPA approval.
• Product disposition and batch release.
• Competence sign-off.
• Which revision of a document is the authoritative one, and whether a change is safety significant.
• Legal interpretation.
• Employment, disciplinary and health decisions.

These stay with your supervisors, your quality people and your competent persons, where they already sit and where the regulator expects to find them.

The system can find the evidence, draft the summary, flag the gap and show its working. It does not decide.

Where your data goes

During an audit or an assessment: nowhere. I review documents on your site and put no file of yours into any AI tool. Demonstrations run on material I have written myself, so nothing of yours needs approving and nobody at your end signs anything.

If we go further than that, the terms are contractual rather than promises on a website:

• No client content is used to train any model.
• Inputs and outputs are deleted within 30 days.
• Sub-processors are listed by name and dated.
• Processing can run in a UK or EU region, or inside your own environment.
• No controlled document, worker record, batch record or investigation file ever enters a consumer or public AI account. Client material is processed only in an environment you have approved in writing.

You get a 1-page data map in plain English: what goes in, where it is processed, what is retained and for how long. Not an appendix, a page you can read.

Who stays responsible

You do, and so do the people who already hold the duty.

Where AI touches health and safety risk, the dutyholder is still the dutyholder. A document assistant is not a high-risk system under the EU AI Act, but the AI literacy duty applies to your people and I will tell you what it means for the workflow we build. Data protection needs a defined purpose, minimisation and a proper controller to processor contract, which is one reason first pilots avoid personal data entirely.

None of that is the pitch. Regulation sets the boundary. It is not the reason to do this.

I have 3 endings to a conversation and 1 of them is no

If the workflow you want cannot be released responsibly, I will say so and not sell it to you. If the real problem turns out to be your document library rather than anything I can build on top of it, I will tell you that, and it is usually cheaper. If a pilot hits a stop condition, it stops, and I do not argue for the technology.

I would rather lose the work than hand you something you cannot defend to an auditor.