Somewhere between a proof of concept and a regulated production system, someone will ask: why did the model do that, on that specific case, for that specific customer? If the honest answer is 'we're not sure, it's a language model,' the system isn't ready for the environments where accountability is required — healthcare, finance, anything touching a compliance framework. The good news is the fix isn't exotic; it's disciplined logging, decided before launch.
What auditors and regulators actually ask for
- Input provenance — exactly what data and context the model saw when it produced a given output, reconstructable after the fact.
- Decision rationale — for anything beyond simple generation, a record of what the system considered and why it chose the path it did, not just the final action.
- Human oversight evidence — where a person reviewed, approved, or could have overridden the system's action, and whether they did.
- Change history — which model version, prompt version, and configuration produced a given historical decision, so a six-month-old case can be explained against the system as it existed then, not as it exists today.
The three-layer log that satisfies most frameworks
- Request layer — who asked, what they asked, when, and under what permissions.
- Reasoning layer — what the model retrieved or considered, and (for agents) what tools it called with what parameters.
- Outcome layer — what changed as a result, and the human approval or override tied to it, if any.
Build the audit trail before the audit, not during it. Retrofitting explainability onto a system that was never designed to produce it is far more expensive than deciding the logging schema in the architecture phase.
Where this pays off beyond compliance
The same evidence trail that satisfies an external auditor is what your own team needs during an incident review, a customer dispute, or a routine 'why did it do that' question from leadership. Teams that treat auditability as a compliance tax build it late and grudgingly; teams that treat it as operational infrastructure find it's the reason they can debug a production issue in twenty minutes instead of two days.