5 Generative AI Security Risks and Practical Defenses

Generative AI can summarize documents, write code, search data, and take actions through connected tools. Those capabilities also create new attack paths and amplify familiar security problems. The risk grows when an AI system can read sensitive data or change external systems without strong controls.

Security cannot be reduced to a warning in the prompt. Organizations need normal software-security practices plus controls designed for probabilistic models, untrusted natural-language input, retrieval systems, and autonomous tool use.

1. Prompt injection and excessive agency

A prompt-injection attack places instructions in a user message, webpage, email, document, image, or retrieved record that the model may treat as commands. An indirect attack is particularly dangerous when an agent searches external content and can also send messages, read files, or call business tools.

Input filtering alone is not a complete defense because malicious instructions can be subtle and models do not reliably separate data from commands. Limit the consequences:

• give the agent only the tools required for the current job;
• use narrowly scoped, short-lived credentials;
• separate untrusted content from system instructions and label its provenance;
• require confirmation before sending, publishing, purchasing, deleting, or changing access;
• validate tool arguments with deterministic code;
• cap cost, time, retries, recipients, and affected records.

2. Sensitive information disclosure

Employees may paste customer records, source code, contracts, credentials, or internal strategy into an AI service without checking how the provider stores or uses the data. A poorly designed AI application may also expose information through logs, retrieved context, model output, shared chats, or overly broad connectors.

Create a data policy that distinguishes public, internal, confidential, regulated, and prohibited inputs. Configure enterprise retention and training controls where available, redact unnecessary identifiers, scan prompts and outputs for secrets, and enforce access control before retrieval—not after the model has received the data.

Never place API keys or passwords in prompts. Use a secrets manager and let the application call a narrowly scoped service without revealing the credential to the model.

3. Unsafe model output and insecure generated code

A language model can produce incorrect commands, vulnerable code, fabricated package names, unsafe HTML, or text that becomes an injection payload when another system executes or renders it. Treat model output as untrusted input even when the model is internal.

Use schema validation, allowlists, encoding, parameterized queries, static analysis, dependency scanning, unit tests, and sandboxed execution. Do not pass raw model output to a shell, database, browser, template engine, or privileged API.

Human review is essential for security-sensitive code, but review should be supported by automated tests and scanners. A plausible diff is not evidence that authorization, error handling, cryptography, and boundary cases are correct.

4. Model, data, and supply-chain compromise

An AI application depends on more than a model. Risks can enter through training or fine-tuning data, model files, vector databases, third-party APIs, plugins, MCP servers, Python packages, browser extensions, and container images. Poisoned content can bias retrieval or trigger malicious behavior when selected later.

• inventory models, datasets, prompts, tools, and external services;
• verify model and package sources, signatures, hashes, and licenses;
• pin and scan dependencies;
• track the origin and update history of retrieved documents;
• separate trusted knowledge from user-submitted content;
• test updates before production and keep a rollback path.

A popular model or app-store listing is not a security assessment. Review the vendor's architecture, incident process, retention terms, access controls, and independent assurance relevant to your use case.

5. Synthetic media and AI-assisted social engineering

Generated text, voice, images, and video can make phishing and impersonation faster and more convincing. Attackers can imitate an executive, vendor, relative, or support agent and create a sense of urgency around a payment, credential reset, or confidential request.

Do not use voice or appearance as the only authorization signal. Verify unusual requests through a separate trusted channel, require multi-person approval for high-value transactions, use phishing-resistant authentication, and train staff to question changes in payment details or process.

Technical detection can help prioritize content, but it is not infallible. Process controls—known callback numbers, approval thresholds, and separation of duties—remain effective even when a fake is difficult to detect.

A deployment checklist

1. Map the system. Document data sources, model providers, prompts, tools, users, outputs, and downstream actions.
2. Classify impact. Identify what can go wrong and the maximum data, money, or records one run can affect.
3. Reduce access. Apply least privilege, network segmentation, tenant isolation, and short-lived credentials.
4. Validate boundaries. Check input, retrieval, output, and tool parameters with controls outside the model.
5. Add human gates. Require approval for consequential or irreversible actions.
6. Evaluate attacks. Test prompt injection, data extraction, unauthorized tool use, poisoned retrieval, denial of service, and unsafe output handling.
7. Monitor production. Log permitted events, detect unusual access or cost, and create an incident-response and rollback plan.
8. Review continuously. Reassess after model, prompt, data, integration, or permission changes.

Governance and staff training

Maintain an approved-tool list and explain which data may be entered into each product. Training should show realistic examples of accidental disclosure, malicious documents, fabricated citations, unsafe code, and deepfake requests. Provide a safe alternative so employees do not quietly use unreviewed “shadow AI” services.

Assign owners for model risk, application security, privacy, legal review, and business acceptance. A vendor cannot decide an organization's risk tolerance, and a security team cannot validate whether an automated decision is appropriate without the business owner.

Use recognized frameworks

The OWASP guidance for LLM and generative AI applications catalogs risks such as prompt injection, sensitive-information disclosure, supply-chain issues, unsafe output handling, and excessive agency. The NIST Generative AI Profile provides a broader risk-management resource.

These frameworks are starting points, not certifications. Apply them to the actual data, tools, decisions, and threat actors in your system. The safest generative AI deployment assumes that models can be manipulated and mistaken, then limits what any single failure can do.