7 Evidence Pack items an auditor or board will ask for

The AI Act is not a PDF policy. If an AI system is going into production, the board, auditor or process owner should see evidence: what exactly is running, who is accountable, which tests it passed, what is logged and how it can be stopped safely. We call that organised set an Evidence Pack.

An Evidence Pack is not a magic compliance certificate. It is a working package tied to a specific version of the model, prompt, data, code and configuration. It enables a decision based on results rather than a demonstration prepared around three convenient questions.

Open the free printable Evidence Pack template. You can complete its fields in the browser and save it as a PDF.

Table of contents

• 1. System inventory and owner card
• 2. Risk classification and organisational role
• 3. Golden set and acceptance criteria
• 4. Evaluation plan and results
• 5. Logs, versioning and monitoring
• 6. Human handoff and incident handling
• 7. Kill switch, rollback and the go/no-go decision

1. System inventory and owner card

The first item sounds simple: what exactly are we assessing? In practice, many organisations cannot answer without a chain of messages and meetings. A product name is not enough. The pack needs to identify:

• the use case and business process;
• user groups and people affected by the system;
• the business owner and technical owner;
• model, supplier, prompts, tools and integrations;
• data sources, owners, sensitivity classes and retention;
• the code, model, index and configuration version covered by the decision;
• the environment: PoC, test, limited production or full production.

An owner card is not a contact list “just in case”. The business owner accepts the impact of failure and the approved use boundary. The technical owner is responsible for the version, observability, response and reproducibility of an outcome. If neither role can be named, the system is not ready for an informed go-live decision.

Evidence in the pack: a current system card, data-flow diagram and an unambiguous owner for the production decision.

2. Risk classification and organisational role

The second question is not “do we use AI?” It is: which rule and responsibility apply to this particular system? At least four questions must be separated:

1. Could the use case involve a prohibited practice?
2. Is the system within a high-risk area under Annex I or III?
3. Do Article 50 transparency duties apply, for example to a chatbot, deepfake or AI-generated informative text?
4. Is the organisation acting as a provider, deployer, importer or distributor?

Scope cannot be inferred from the tool name. The same technology may be an ordinary editorial assistant in one process and part of a system affecting access to a service in another. The role, use case, data and consequence matter.

The Evidence Pack should contain a preliminary qualification, its rationale, open questions and the person who approved the next step. If classification is not clear, compliance or legal validation is needed. The template itself does not replace legal advice or a formal conformity assessment.

Evidence in the pack: a qualification table with rationale, scope and an owner for every open risk.

3. Golden set and acceptance criteria

A demo can look excellent because it shows selected cases. A golden set reverses that logic: representative questions, tasks, expected outcomes and edge cases are recorded first, and only then is the system assessed.

A useful golden set covers more than ordinary situations. It should include:

• easy, hard and ambiguous tasks;
• current, old, conflicting and incomplete data;
• cases where the system must refuse to answer or act;
• attempts to exceed permissions;
• scenarios critical to the customer, safety or compliance;
• paraphrases of the same instruction;
• cases that reveal differences between system versions.

Acceptance criteria are recorded before the final test. Otherwise, the team will fit the interpretation to the result. A threshold may cover correctness, grounding, refusal rate, prompt-injection resistance, latency, unit cost or uncontrolled actions.

Evidence in the pack: a versioned golden set, sampling rationale and thresholds with the name of the person who approved them.

4. Evaluation plan and results

A report saying “the tests passed” is not a result. The Evidence Pack should let another person understand what ran, against which version, with which threshold and exactly what failed.

A minimum evaluation plan for a production GenAI or RAG system usually covers:

• answer quality or task success;
• grounding and source-citation accuracy;
• hallucinations and behaviour when data is absent;
• prompt injection, data leakage and permission enforcement;
• agent tools and integration actions;
• fallback after a model, source or API failure;
• cost, latency and behaviour under load;
• regression after a model, prompt, data or code change.

Results must also state limitations. A test over 200 cases from one process in two languages does not establish quality for another department, an unknown data population or autonomous actions. An honest “not tested” is more useful than a green status with no boundary.

Evidence in the pack: machine-readable results and a readable summary with outcomes, thresholds, blockers and links to test artefacts.

5. Logs, versioning and monitoring

An AI system changes even when its interface looks the same. A supplier updates the model, the knowledge base receives new documents, the prompt changes, and users start asking different questions. Evidence from acceptance day therefore expires quickly.

The pack should state whether every material operation can be traced to:

• the model, prompt, code and index version;
• input, output, sources and tool calls;
• a decision, refusal, fallback or escalation;
• the user or role and enforced permissions;
• operation time, cost and status;
• log-access and retention rules;
• alerts, thresholds and the person responding to deviation.

Monitoring should not stop at API availability. Track quality trend, cost, fallback growth, human escalations, answers without sources and permission incidents. Every signal needs a threshold and a response. A dashboard without an owner is only a screen.

What numerical proof looks like

In the public mojApteczka case, we disclose 302,516 drug-interaction records in the knowledge base, 100% data extraction on the stated validation set, n=200, and USD 0.0006 cost per AI scan. Those numbers are evidence for one specified system and scope — not a guarantee of the same result in another implementation.

An Evidence Pack adds context to every metric: version, sample, method, threshold, date and owner. That makes the result challengeable, repeatable and comparable after a change.

6. Human handoff and incident handling

“Human in the loop” is not a control if nobody knows when a person takes over, who receives the alert and what they see after escalation. Handoff must be designed as an operational process.

The pack should record triggers such as low confidence, no reliable source, sensitive data, an irreversible action, a decision with legal or financial impact, conflicting data or an integration failure. Each trigger needs a receiving role, channel, response expectation and safe system behaviour while the decision is pending.

A good handoff passes context: request, sources, history, reason for escalation, warnings and actions already taken. After resolution, the outcome should feed the error register, golden set or remediation plan. Otherwise, the organisation pays for the same incident repeatedly.

Evidence in the pack: a trigger table, roles and SLA, a log or screenshot from a test escalation and the mechanism that closes the feedback loop.

7. Kill switch, rollback and the go/no-go decision

The final item answers the board’s most practical question: what will we do when the system stops behaving safely? “We will fix the prompt” is not enough.

A kill switch should stop a risky function or agent action without waiting for a full new release. Rollback should identify the last safe model, prompt, data and code version, recovery time and tests run after restoration. A fallback is also needed: manual service, sourced answers only, read-only mode or temporary shutdown.

The Evidence Pack then records one decision:

• GO — thresholds are met within the approved scope;
• CONDITIONAL GO — the system operates within limits, while open risks have owners and dates;
• NO-GO — a quality, security, compliance or operational-control blocker remains.

The decision includes a remediation plan: gap, priority, action, owner, due date and closure evidence. Sign-off is not decoration. It identifies who knowingly accepted the scope and risk.

Evidence in the pack: a tested stop scenario, rollback plan, approved decision and remediation actions.

Common mistakes

1. A pack without a version. Results do not identify the model, prompt and data they cover.
2. Thresholds written after testing. Criteria are fitted to the outcome.
3. Averages only. A good overall score hides a critical error in a small segment.
4. Monitoring without response. An alert exists but has no owner or procedure.
5. Handoff “to a human”. Nobody knows which human, how quickly or with what context.
6. Treating a template as a certificate. Empty fields are not evidence, and the pack does not replace legal assessment.

What to do now

First, open the Evidence Pack template and complete the system card and acceptance criteria. If you cannot identify the version, owner or go/no-go threshold, start with that gap before running more tests.

If you need independent golden-set design, evaluations, red teaming and a production decision, see AI Assurance & Governance. Semitora can assess a system built by your team or another supplier without taking over the whole project.

FAQ

Is an Evidence Pack a document required by the AI Act?

The AI Act does not impose one document with this name. An Evidence Pack is a practical way to organise technical and operational evidence. Specific duties depend on the organisation’s role, the use case and the system classification.

Does a completed template prove compliance?

No. Evidence comes from attached results, logs, versions, decisions and controls that actually operate. The template organises them; it does not replace legal advice or a formal conformity assessment.

Is the Evidence Pack created only before production?

The first complete version should precede the production decision. It then needs an update after a material change to the model, prompt, data, integrations, use scope or risk.

Who should approve go/no-go?

At least the business and technical owners. Depending on the system, security, privacy, compliance, legal counsel or a sector-process owner may need to join the decision.