AI Safety and Governance

Safety and governance covers everything that decides what an AI system is allowed to do, what it actually does, and how you prove either afterward. It spans technical controls such as input filtering and permission scoping, evaluation practices that measure behavior before and after release, and organizational obligations that arrive through regulation and customer contracts. The area exists because language models accept instructions from anywhere text can reach them, and because an agent holding tool access can act on those instructions.

For someone building, this is the part of the system that turns a working demo into something you can put in front of customers. For someone buying, it is the set of questions that separates a vendor who has thought about failure from one who has not. Most of the actual work is unglamorous: scoping permissions, recording actions, deciding who approves what, and checking that behavior has not quietly shifted since last month.

Start here

How the layers stack

Think of five layers, each catching what the one before it missed. Input safety inspects what arrives, including text the user never typed, which is why indirect prompt injection matters more than the direct kind. The model layer covers alignment and evaluation, the properties you can shape but never fully guarantee. Output safety and content moderation check what comes back. The action layer is where least privilege, approval gateways and tenant isolation live, and it is the only layer that reliably limits damage, because it constrains capability rather than words. The record layer, audit trail and explainability, is what lets you answer questions after the fact. Governance is the practice of deciding who owns each layer.

Where to start

Start with prompt injection, because it reframes everything else. Once you accept that any retrieved document, email or web page can carry instructions, you stop trying to solve safety with better wording and start designing around capability. Guardrails come next as the general shape of runtime checks, then the input and output halves as concrete places to put them. Least privilege and audit trail follow, since they are what you would want already in place during an incident. Leave evaluation and red teaming until you have something worth measuring, and read the regulatory terms last, once you know which data you actually touch and on whose behalf.

Reading the regulatory terms

Regulations in this area answer different questions, so it helps to sort them before reading. Data protection frameworks such as the European General Data Protection Regulation govern personal data: what may be collected, on what basis, and what a person can ask you to do with it. The EU AI Act organizes obligations around the risk of an AI system's use case and the role a party plays. Security attestations describe audited controls that an organization operates. Contract terms such as a data processing agreement and residency commitments define where data lives and who may handle it. None of these substitute for the others, and none of this is legal advice.

What people get wrong

The most common mistake is treating safety as a filtering problem. Filters help, but a system that can send email, move money or delete records needs its permission set narrowed first, with the filter as a second line. The second is confusing compliance artifacts with behavior: an attestation describes controls at an organization, not whether a given agent will refuse a bad instruction. The third is assuming a system that passed evaluation stays passed, when models, prompts and data all change, making drift monitoring an ongoing cost rather than a launch task. Finally, teams log conversations but not actions, which leaves the audit trail useless exactly when it is needed.

GDPR vs EU AI Act

One is a data protection framework centered on personal data, while the other places obligations according to the risk of an AI system's use case.

Every term in AI Safety and Governance

AI Governance: AI governance is the set of policies, roles, processes, and records an organization uses to decide which AI systems it builds or buys, under what conditions they operate, who is accountable for them, and how they are monitored after deployment. It converts abstract principles into decisions that can be evidenced.
AI Guardrails: AI guardrails are controls placed around a language model that constrain what reaches it and what it is permitted to produce or do. They run outside the model itself, using classifiers, rules, allow lists, schema validation, and policy checks at the input, retrieval, output, and tool layers. Guardrails complement training-time safety rather than replacing it.
AI Incident Response: AI incident response is the process for detecting, containing, investigating and remediating harmful failures of an AI system, such as harmful output reaching users, unauthorized actions taken by an agent, or exposure of data through model behavior. It adapts established security incident practice to failures that are probabilistic, hard to reproduce and often invisible in conventional error monitoring.
Alignment: Alignment is the research problem and engineering practice of making an AI system pursue the goals its developers and users actually intend, including unstated norms and constraints. It covers both specifying the right objective and ensuring the trained system genuinely internalizes it rather than learning a proxy that scores well during training.
Anonymization: Anonymization is the process of transforming data so that individuals can no longer be identified from it, by anyone, using any means reasonably likely to be used. Under the GDPR truly anonymized data falls outside data protection law entirely, which is why the standard is strict and why many datasets described as anonymized do not actually meet it.
Approval Gateway: An approval gateway is a control that pauses an automated action and requires an authorized person to confirm it before execution. Actions are classified by consequence, so routine reads proceed automatically while irreversible, costly, external, or cross-boundary operations wait for a decision. The gateway records who approved what, when, and on which exact parameters.
Audit Trail: An audit trail is a durable, append-only record of significant events in a system, capturing who or what acted, on which resource, at what time, with what parameters, and with what result. Unlike debug logs, it is retained on a defined schedule, protected from modification, and designed to be read by investigators, auditors, and regulators.
Bias: Bias in AI refers to systematic differences in a system's behavior or accuracy across groups, contexts, or inputs, particularly where those differences are unjustified and cause harm. It originates in training data, labeling choices, objective design, and deployment context rather than in any single component, and it is measured against a chosen fairness definition.
CCPA: The CCPA is California's consumer privacy law, significantly amended by a later ballot measure, granting residents rights to know, delete and correct personal information and to opt out of its sale or sharing. It applies to for profit businesses meeting statutory thresholds that handle California residents' personal information, regardless of where the business itself is located.
Confused Deputy Problem: The confused deputy problem is a security flaw in which a program holding legitimate privileges is tricked by a less privileged party into misusing those privileges on that party's behalf. The deputy is authorized and behaves as designed, but it acts on a request without carrying the requester's authority. The pattern was named in the computer security literature in 1988.
Content Moderation: Content moderation is the classification and handling of text, images, or other media against a defined policy covering categories such as violence, sexual content, harassment, self-harm, and illegal activity. In AI systems it is applied to user input, to generated output, or to both, with actions ranging from blocking to rewriting to escalating for human review.
Content Provenance: Content provenance is the practice of attaching verifiable records of origin and edit history to digital media, so a viewer can check where an asset came from and how it was modified. Open specifications, most prominently the work of the Coalition for Content Provenance and Authenticity, define cryptographically signed manifests that travel with a file or are recoverable from it.
Data Leakage Prevention: Data leakage prevention is the practice of stopping sensitive information from leaving a controlled environment through unintended channels. In AI systems the channels include prompts sent to external providers, model outputs delivered to the wrong recipient, over-permissioned retrieval, logs and telemetry, and training on data that was never cleared for that purpose.
Data Minimization: Data minimization is the principle that a system should collect and keep only the personal data genuinely needed for a stated purpose, and nothing beyond it. It appears as a legal requirement in privacy regimes such as the GDPR and as an engineering practice in security design, where a smaller data footprint limits both regulatory exposure and the damage any single breach can cause.
Data Processing Agreement: A data processing agreement is a contract between a controller and a processor that governs how personal data may be handled on the controller's behalf. Article 28 of the General Data Protection Regulation requires such a contract and specifies terms it must contain, including the subject matter, duration, nature and purpose of processing, and the categories of data involved.
Data Residency: Data residency refers to the geographic location where data is stored and processed. It is distinct from data sovereignty, which concerns which country's laws govern the data, and from data localization, which is a legal requirement to keep certain data within national borders. Buyers frequently ask about residency because location shapes both legal exposure and access risk.
Data Subject Access Request: A data subject access request is a formal request from an individual asking an organization to confirm whether it processes their personal data and to provide a copy along with details of that processing. Recognized under the GDPR and comparable laws, it carries statutory response deadlines and is one of the most common ways privacy compliance is tested in practice.
Deepfake: A deepfake is synthetic audio, image or video that convincingly depicts a real person saying or doing something they did not. The term covers a spectrum from consensual entertainment and dubbing to non consensual intimate imagery, political disinformation and fraud, and the harms and legal treatment differ sharply across that spectrum.
Differential Privacy: Differential privacy is a mathematical definition of privacy guaranteeing that the output of an analysis is nearly unchanged whether or not any single individual's record was included. It is achieved by adding calibrated random noise to computations, and the strength of the guarantee is expressed by a privacy budget parameter, conventionally written as epsilon, where smaller values mean stronger privacy.
Egress Control: Egress control is the restriction of outbound network connections from a system to an explicitly approved set of destinations. In AI deployments it is a primary defense against data exfiltration, because an agent that has been manipulated through injected instructions still cannot send data anywhere the network policy does not permit.
Encryption At Rest: Encryption at rest is the practice of storing data in encrypted form on disks, databases, object storage and backups so that the raw media cannot be read without the corresponding keys. It is a baseline control in nearly every security framework, and it defends primarily against theft of storage, misplaced backups and improperly decommissioned hardware.
Encryption In Transit: Encryption in transit is the protection of data while it moves across a network, normally using Transport Layer Security, so that traffic cannot be read or silently altered by anyone positioned between the endpoints. It is a baseline expectation for public internet traffic and, in modern architectures, increasingly for internal service to service traffic as well.
EU AI Act: The EU AI Act is the European Union's regulation on artificial intelligence, Regulation (EU) 2024/1689, which entered into force in August 2024. It classifies AI systems by risk and attaches obligations accordingly, from outright prohibition to transparency duties. It applies to providers and deployers placing systems on the European Union market regardless of where they are established.
Explainability: Explainability is the degree to which the reasons for an AI system's output can be presented in terms a person can understand and check. It matters where decisions affect people, where errors must be diagnosed, and where regulation or contract requires a rationale. For large neural networks it remains only partially achievable.
GDPR: The General Data Protection Regulation is European Union law governing the processing of personal data, applicable since May 2018. It requires a lawful basis for processing, sets principles such as purpose limitation and data minimization, grants individuals enforceable rights over their data, and applies to organizations outside Europe that target or monitor people within it.
Hallucination: A hallucination is output from a language model that is fluent and confident but factually wrong or unsupported by any source. It arises because such models predict likely continuations of text rather than retrieving verified facts, so plausibility and truth can diverge. Common forms include invented citations, fabricated figures, and imagined product features or API methods.
HIPAA: HIPAA is a United States federal law whose privacy, security and breach notification rules govern how protected health information is used and disclosed. It applies to covered entities such as health plans and most healthcare providers, and to business associates that handle protected health information on their behalf under a written agreement.
Indirect Prompt Injection: Indirect prompt injection is a variant in which the malicious text is planted in content the model later retrieves, such as a web page, document, email, calendar invite, or code comment, rather than typed by the user. The victim triggers the attack simply by asking the assistant to read or summarize that content. It was described in published research in 2023.
Input Safety: Input safety refers to checks applied to a message or document before it reaches the model. Typical checks cover injection patterns, prohibited request categories, personal data that should not enter context, oversized or malformed payloads, and rate or quota limits. Blocking at this stage prevents cost, exposure, and the generation of unsafe content in the first place.
ISO/IEC 27001: ISO/IEC 27001 is an international standard specifying requirements for an information security management system, a documented framework for identifying security risks and applying controls to treat them. Organizations can be certified against it by an accredited body following an audit, and certification is commonly requested in enterprise and international procurement.
Jailbreak: A jailbreak is a prompt or conversation designed to make a model produce content its safety training and usage policies are meant to refuse. Techniques generally work by reframing the request, for example as fiction, translation, research, or a hypothetical, so the harmful intent is less apparent to the model's learned refusal behavior.
Key Management: Key management is the set of practices governing how cryptographic keys are generated, stored, distributed, rotated, revoked and destroyed. Because encryption transfers the security of data onto the security of its keys, key management is the control that determines whether encryption provides real protection or only satisfies a checklist.
Least Privilege: Least privilege is the security principle that every component, credential, and process should hold only the permissions required for its specific task, and only for as long as the task runs. Articulated for computer systems in the 1970s, it limits the damage from any single compromise, bug, or manipulation, since a bounded identity can only cause bounded harm.
Model Drift: Model drift is the degradation of an AI system's performance over time as production conditions diverge from the conditions it was built for. It covers data drift, where input distributions shift, concept drift, where the relationship between input and correct answer changes, and version drift, where an underlying model or prompt is updated beneath a working system.
Model Evaluation: Model evaluation is the systematic measurement of a model or AI system against defined criteria using repeatable test sets. It spans capability benchmarks, task-specific accuracy on representative data, safety and refusal behavior, robustness to adversarial input, and regression testing after any change to a prompt, model, or retrieval configuration.
Model Provider Data Policy: A model provider data policy is the published set of terms describing how an AI vendor handles the content sent to its models, covering retention, training use, human review, subprocessors, geographic processing and deletion. It is the document a deploying organization must read before routing customer data through a model, because these terms vary widely and change over time.
NIST AI Risk Management Framework: The NIST AI Risk Management Framework is a voluntary framework published by the United States National Institute of Standards and Technology for identifying, measuring and managing risks from artificial intelligence systems. It is organized around four functions, Govern, Map, Measure and Manage, and is widely used as a common vocabulary for AI risk work rather than as a certifiable standard.
Output Safety: Output safety refers to checks applied to a model's response after generation and before it is displayed, sent, or executed. Typical checks cover prohibited content, leaked system instructions or credentials, personal data, unsupported factual claims, schema conformance, and unsafe tool arguments. It is the last automated point at which a bad response can be stopped.
Over-Refusal: Over-refusal is the failure mode in which an AI system declines a harmless request because it superficially resembles a harmful one. It is the mirror image of unsafe compliance, and because most safety interventions trade one against the other, systems are normally measured on both rates rather than on refusals avoided alone.
PII Detection: PII detection is the automated identification of personally identifiable information inside text, files, or database fields. Techniques combine pattern matching with checksums for structured identifiers, named entity recognition for names and addresses, and contextual classifiers for ambiguous cases. Detection is a prerequisite for redaction, access control, retention enforcement, and breach assessment.
PII Redaction: PII redaction is the removal or replacement of personal data in text before it is stored, logged, or sent onward. Methods include masking with placeholders, replacing values with reversible tokens, generalizing a value to a coarser range, and substituting realistic synthetic values. The method chosen determines whether the original can ever be recovered.
Prompt Injection: Prompt injection is an attack in which text supplied by an untrusted party is interpreted by a language model as instructions rather than as data. Because a model sees one undifferentiated context window, attacker text can override developer intent and trigger disclosure of hidden instructions or unauthorized tool actions. It is widely considered an unsolved class of vulnerability.
Pseudonymization: Pseudonymization is the processing of personal data so that it can no longer be attributed to a specific person without additional information kept separately and protected. The GDPR defines it explicitly and treats it as a security and risk reduction measure, but pseudonymized data remains personal data and stays fully within the scope of data protection law.
Purpose Limitation: Purpose limitation is the principle that personal data collected for one specified purpose may not be reused for an unrelated purpose without a fresh legal basis. It is a core requirement of the GDPR and similar frameworks, and it is the rule most often tested when data gathered to deliver a service is later proposed as training material or analytics input.
Red Teaming: Red teaming is structured adversarial testing in which people deliberately try to make a system behave harmfully, leak data, or bypass its controls. Unlike a fixed evaluation suite, it is open-ended and creative, aiming to discover failure modes nobody anticipated. Findings are then converted into permanent test cases so the same weakness cannot silently return.
Responsible Disclosure: Responsible disclosure is the practice by which a person who discovers a security vulnerability reports it privately to the affected organization and allows time for a fix before publishing details. Also called coordinated vulnerability disclosure, it depends on a published reporting channel, a stated timeline, and assurance that good faith researchers will not face legal action.
Right To Erasure: The right to erasure is a data protection right allowing an individual to request deletion of personal data an organization holds about them. Established in Article 17 of the GDPR and echoed in other regimes, it is qualified rather than absolute: several grounds must apply, and exemptions exist for legal obligations, defense of legal claims and certain public interest purposes.
Role Based Access Control: Role based access control is an authorization model in which permissions are attached to named roles, and users receive access by being assigned those roles rather than by receiving permissions individually. Changing what a role can do changes access for everyone holding it. The model is generic and standardized, independent of any particular vendor or product.
Safety Evaluation: Safety evaluation is the systematic testing of an AI system's behavior against defined harm categories, measuring how often it produces disallowed content, complies with manipulation attempts, or takes unsafe actions. It complements capability evaluation, which measures what a system can do, and it usually combines automated test suites with adversarial human probing.
Sandboxed Execution: Sandboxed execution is running code or tools produced or invoked by an AI system inside a strongly isolated environment with restricted access to the filesystem, network, host resources and other tenants' data. It exists because model generated code should be treated as untrusted input, regardless of whether the model that produced it is considered reliable.
Single Sign-On: Single sign-on is an authentication arrangement in which a user authenticates once with a central identity provider and then reaches multiple independent applications without entering credentials again. The applications trust assertions issued by that provider instead of storing passwords themselves. It is a generic model implemented through open protocols rather than a single product.
SOC 2: SOC 2 is an attestation report produced by an independent accounting firm assessing a service organization's controls against the Trust Services Criteria defined by the American Institute of Certified Public Accountants. Security is always in scope, with availability, processing integrity, confidentiality, and privacy optional. It is a report on controls, not a certification or a legal requirement.
Tenant Isolation: Tenant isolation is the separation of one customer's data, configuration, and workloads from every other customer's inside a shared system. It is enforced across storage, retrieval, caching, background jobs, and logs. Cross-tenant leakage, where one customer sees another's data, is generally treated as the most severe defect class in multi-tenant software.
Tool Poisoning: Tool poisoning is an attack in which the descriptions, schemas, or metadata that tell an AI agent how to use a tool are altered to contain hidden instructions. Because the agent reads these definitions as trusted configuration, malicious text placed there can influence its behavior without ever appearing in a user message. It is a supply chain variant of prompt injection.
Training Opt Out: A training opt out is a setting or contractual term under which a provider agrees not to use a customer's inputs and outputs to train or improve its models. Defaults differ substantially by provider, product tier and region, so whether content is used for training is a question to verify in current terms rather than assume.
Watermarking: Watermarking is the embedding of a detectable signal into AI generated content so that its machine origin can later be identified. For images and audio the signal is imperceptible modification of the media; for text it is usually a statistical bias applied during token selection. Robustness against editing and paraphrasing remains a significant and actively researched limitation.
Zero Data Retention: Zero data retention is a service configuration in which inputs and outputs are processed to fulfill a request and then not stored afterward. It is offered by some AI providers under specific plans or agreements and is frequently required in regulated procurement, though the exact scope of what is and is not retained varies considerably between vendors.
Zero Trust: Zero trust is a security model that removes implicit trust based on network location and requires every request to be authenticated, authorized, and continuously evaluated against policy. Being inside a corporate network confers no privilege by itself. Access decisions consider identity, device state, and context, and grant only the privilege needed for that specific request.

Back to the AI Glossary

AI Safety and Governance

Start here

How the layers stack

Where to start

Reading the regulatory terms

What people get wrong

Commonly confused

Prompt Injection vs Jailbreak

Input Safety vs Output Safety

Hallucination vs Model Drift

GDPR vs EU AI Act

Every term in AI Safety and Governance