Governance, Asset Inventory, and Risk Ownership
- Maintain a current inventory of AI agents, models, tools, plugins, vector stores, memory stores, data sources, data lineage documents, and external providers.
- Define trust boundaries across users, the agent runtime, the model provider, the orchestration layer, the tool execution layer, enterprise systems, and human approval interfaces.
- Assign clear control ownership for deployment approval, tool access scope, monitoring, incident response, and risk acceptance.
- Establish agent identities, acceptable use policies, and prohibited actions for agents with clear agent goals.
- Require periodic security and risk reviews before expanding agent capabilities.
- Align governance to enterprise security, privacy, legal, and third-party risk programs.
Secure AI Development, Change Management, and Continuous Assurance
- Integrate AI agents into the secure system development life cycle (SDLC), applying secure by design and privacy by design principles with controlled development, testing, and production release processes.
- Threat model each agent workflow, including user input, retrieval, memory, inference, tool calls, application programming interfaces (APIs), human approvals, output delivery, tenant boundaries, and privileged actions.
- Address AI-specific threats such as prompt injection, indirect prompt injection, data exfiltration, server-side request forgery (SSRF), memory poisoning, cross-user or cross-tenant leakage, excessive permissions, malicious tool output, model inversion, unsafe file ingestion, and compromised third parties.
- Separate reasoning, retrieval, and action execution into controlled stages, with least-privilege tool access, external policy enforcement, and explicit authorization for high-impact actions.
- Apply formal change management for prompts, models, policy logic, tool permissions, memory configuration, retrieval pipelines, orchestration logic, external providers, and action scope.
- Maintain version control, approval history, auditability, and rollback capability for prompts, policies, model versions, orchestration logic, evaluation datasets, configurations, and infrastructure as code (IaC).
- Perform conventional security testing, including code review, static application security testing (SAST), dynamic application security testing (DAST), software component analysis (SCA), API testing, secrets scanning, dependency scanning, IaC scanning, container scanning, and cloud configuration review.
- Conduct AI-specific adversarial testing for jailbreaks, prompt injection, data exfiltration, unsafe tool use, memory poisoning, cross-user leakage, SSRF, malicious document ingestion, guardrail bypass, and unauthorized actions.
- Maintain large language model (LLM)-specific regression tests for known attack patterns, policy bypasses, guardrail failures, sensitive data exposure, tool misuse, retrieval manipulation, and memory misuse.
- Validate privacy and confidentiality controls, including data minimization, sensitive data filtering, output limits, differential privacy or privacy-preserving techniques (where applicable), and prevention of unauthorized disclosure.
- Reassess risk and retest controls when models, prompts, tools, permissions, memory behavior, retrieval sources, providers, deployment environments, or the action scope change.
- Continuously monitor deployed agents for anomalous behavior, tool usage, policy violations, retrieval activity, memory access, guardrail failures, unsafe outputs, and emerging attack patterns.
- Conduct behavioral, performance, and governance drift detection to identify degradation in quality, bias, policy adherence, refusal behavior, tool-use behavior, and safety controls.
- Manage AI supply chain risk across models, datasets, plugins, APIs, libraries, containers, external providers, and infrastructure dependencies.
- Establish security release gates, rollback procedures, and incident response integration for AI-specific failures such as data leakage, prompt injection compromise, unsafe tool execution, or provider compromise.
Strong Identity, Authentication, and Authorization
- Use per-agent and per-workload identity; do not share accounts or long-lived tokens.
- Prefer workload identity and federated access over static credentials.
- Issue short-lived credentials with automatic rotation.
- Apply least-privilege controls to every tool, API, knowledge source, and data store the agent can access.
- Use role-based access control (RBAC)/attribute-based access control (ABAC) and “just-in-time” elevation for sensitive functions.
- Require multifactor authentication (MFA) for operators and administrators.
- Separate identities for human users and functions and the agent runtime, tool runner, and admin/operator.
- Enforce authorization checks not only for tools, but also for memory retrieval and context access.
Network Segmentation, Isolation, and Sandbox Execution
- Run tool execution, code interpretation, browser automation, and file parsing in sandboxed environments such as containers, virtual machines (VMs), or microVMs.
- Restrict sandboxes with read-only file systems (where feasible), prohibit privileged containers, and limit system calls (syscalls), ephemeral execution, and no default egress.
- Place agent infrastructure in segmented networks.
- Restrict outbound traffic to approved domains, APIs, and destinations.
- Route outbound requests through inspection or proxy layers when practical.
- Block access to internal admin interfaces, metadata services, and nonapproved internal network resources.
- Separate reasoning from execution so the model cannot directly invoke unrestricted actions.
Defense Against Prompt Injection and Untrusted Content
- Treat all external content as untrusted input, including webpages, emails, PDFs, retrieved documents, user attachments, and tool outputs.
- Prevent external content from overriding system or developer instructions.
- Use content boundaries, contextual labeling, and instruction hierarchy enforcement.
- Limit agent behavior with allowlisted tools and explicit action constraints.
- Filter inputs and outputs for exfiltration attempts, credential requests, policy bypass language, and hidden or obfuscated instructions.
- Do not let retrieved content directly trigger actions without a separate policy decision.
- Track data provenance and source trust levels so that low-trust content does not receive high action authority.
Data Protection, Secrets Management, and Memory Security
- Never place secrets in prompts.
- Use a secrets manager and scoped tokens for all credentials.
- Apply data classification, minimization, retention limits, and encryption for prompts, logs, traces, memory, and outputs.
- Redact or tokenize sensitive data in logs and observability platforms.
- Isolate memory and context by tenant, user, and use case.
- Apply retention limits and time to live (TTL) values to persistent memory.
- Require authorization checks before an agent can retrieve stored memory or prior interactions.
- Prevent cross-session and cross-tenant leakage.
- Validate the provenance and integrity of stored memory and retrieved context to reduce poisoning risk.
Secure Tool and API Integrations
- Put agent-accessible tools behind an API gateway, action broker, or policy enforcement layer.
- Validate requests with authentication, authorization, schema enforcement, parameter constraints, quotas, and rate limits.
- Use explicit allowlists for domains, endpoints, commands, repositories, and file paths.
- Block dangerous primitives such as arbitrary shell access, unrestricted file reads, and unrestricted URL fetches.
- Protect against SSRF, internal reconnaissance, and unintended access to cloud metadata services.
- Validate outputs from tools before using them as context for later decisions.
Policy Enforcement, Output Control, and Human Oversight
- Implement a deterministic policy enforcement point (PEP) between agent outputs and action-capable systems.
- Ensure the PEP validates the action type, target system, actor identity, authorization, business rules, risk thresholds, and required approvals.
- Require human in the loop (HITL) approval for destructive, financial, legal, regulated, or irreversible actions.
- Use step-up authentication, dual control, or transaction confirmation for sensitive workflows.
- Present clear execution summaries so humans can verify exactly what will happen.
- Support read-only mode or recommendation-only mode for higher-risk agent use cases.
Logging, Monitoring, Detection, and Incident Response
- Log, with redaction where needed, prompts, responses, retrieved sources, content hashes, tool invocations, action decisions, approvals, the identity context, and policy violations.
- Use centralized, tamper-resistant logging.
- Monitor for unusual tool use, excessive data retrieval, repeated bypass attempts, anomalous outbound traffic, sudden changes in model behavior, and abuse patterns.
- Create agent-specific incident response playbooks such as playbooks for prompt injection, tool compromise, model/provider compromise, data leakage, cross-tenant exposure, and memory poisoning.
- Define escalation paths and containment procedures, including disabling tools or shifting the agent to read-only mode.
Model, Provider, and Software Supply Chain Security
- Pin and verify versions of models, model endpoints, embeddings, frameworks, plugins, and dependencies.
- Use software bill of materials (SBOM), AI bill of materials (AI-BOM), dependency scanning, artifact signing, and provenance validation.
- Restrict who can publish or update tools, prompts, agent policies, and integrations.
- Vet third-party model and tool providers through supplier risk assessment.
- Contractually define security-related obligations such as data handling, retention, model training use, security controls, residency, and breach notification.
- Monitor for provider-side changes, behavior drift, or endpoint spoofing.
- Maintain rollback capability for model or tool changes.
Reliability, Resilience, Kill Switches, and Safe Degradation
- Implement rate limits, quotas, token budgets, conversation limits, timeouts, retries with caps, and circuit breakers.
- Add global and per-capability kill switches to disable unsafe agent behavior quickly.
- Support rollback of model versions, prompts, policies, tool permissions, and dependencies.
- Design safe fallback modes such as read-only mode, recommendation-only mode, and manual approval mode; ensure critical workflows can revert to manual operation if the agent is disabled.
