Security for AI systems

Old security, plus new routes

Everything you already do for software security still applies: access control, encryption, patching, logging. AI adds routes that traditional checks do not cover, because the system’s behavior comes from data and prompts, not just code.

Threats to know

• Prompt injection. Input crafted to make a system ignore its instructions or reveal information. It can be direct (a user types it) or indirect (hidden in a document, email or web page the system reads).
• Sensitive information disclosure. Outputs that reveal personal, confidential or system data, from prompts, retrieval stores or training.
• Data and model poisoning. Tampered training, fine-tuning or retrieval data that changes behavior on purpose.
• Supply chain. Models, plugins and datasets from sources you have not vetted.
• Improper output handling. Passing model output to other systems without checking it, so a crafted output becomes an attack.
• Excessive agency. Giving the system more permissions or actions than it needs.
• Model theft and extraction. Copying a model’s behavior or weights through its interface or an insecure repository.
• Unbounded consumption. Requests or loops that exhaust capacity or budget.

The OWASP Top 10 for LLM Applications and MITRE ATLAS catalogue these in more detail. They are living documents, so check the current versions, and use them as checklists, not as complete lists.

Evidence an auditor will want

Good security evidence is complete over the period, traceable to production and timestamped. Typical items: model access logs, configuration exports showing encryption in transit and at rest, alert rules and investigation records for model endpoints, quarterly access reviews for model repositories, and data-handling and deletion records. Watch for compliance drift: settings that quietly revert, such as log retention resetting to a shorter default after a cloud migration, or monitoring lost in a model update. See the lesson Evidence an auditor will trust.

What to do before launch

1. Threat-model the use. What could an attacker gain? What can the system touch?
2. Limit reach. Give the system only the data, tools and permissions it needs. Separate instructions from untrusted content.
3. Validate outputs. Check anything the model produces before it triggers an action or reaches another system.
4. Test adversarially. Try injection, data extraction, jailbreaks and misuse. Use the LLM security checklist.
5. Protect the pipeline. Control access to training data, weights and prompts. Vet third-party models and plugins.
6. Log and alert. Log requests and actions, and alert on unusual patterns.
7. Plan the response. Add AI incidents to your incident plan, with a clear route to pause the system.

Vendors

A vendor’s security does not cover your use. Ask how they test for injection and leakage, what logs you get, where data goes, and how they will tell you about incidents and model changes. Put the answers in the contract. The vendor due diligence tool includes these questions.

Example

An internal assistant read emails to summarize them. A message contained hidden text telling the assistant to include the CEO’s calendar in its summary. The assistant complied. The fixes were to separate untrusted email text from instructions, limit the assistant to a summary format and restrict its access to calendars. The team added the attack to their regression tests.

What to do now

• Run a one-hour prompt-injection test on your most connected AI system.
• Check what the system can reach and cut anything it does not need.
• Add AI-specific alerts to your monitoring.