What is prompt injection?

Prompt injection uses input intended to make an AI system ignore, reinterpret or mix its intended instructions with untrusted ones. The problem exists because language models cannot reliably separate data from commands written in natural language. A simple list of prohibited words cannot solve it.

How does prompt injection work?

An LLM application often combines system instructions, user input, conversation history, retrieved documents and external tool results in one context. Manipulated text may claim that earlier rules no longer apply or that it has a more trusted role. Even when the model usually follows the hierarchy, variants, translations, encodings or multi-stage conversations can change its behavior. A system prompt is therefore not a dependable security boundary.

Direct and indirect prompt injection

Indirect attacks are particularly relevant when an agent independently reads external content. The attacker may not need direct access to the application.

What impact is possible?

• Confidential instructions, documents or another tenant's data are disclosed.
• An agent invokes unintended tools, sends data or changes records.
• Safety and content policies are bypassed and prohibited output is generated.
• Manipulated answers mislead users or are consumed by downstream systems as trusted data.

Risk depends mainly on the application's access and capabilities. A text-only generator generally has less potential impact than an agent with email, file and administrative permissions.

Is a jailbreak the same thing?

A jailbreak is a type of manipulation usually aimed at bypassing a model's content or behavior restrictions. Prompt injection is broader and concerns instruction integrity in a specific application. Unlike SQL injection, there is no fixed parser with a cleanly separable command and data syntax. Conventional escaping alone therefore does not solve the issue.

How can the risk be reduced?

• Enforce authorization, tenant isolation and business rules outside the model.
• Give tools least privilege and allowlist parameters on the server.
• Require user confirmation with a clear preview for high-impact actions.
• Label external content, record its provenance and retrieve only necessary sections.
• Validate model output for its destination before HTML, database, shell or other use.
• Monitor anomalies and evaluate safeguards with adversarial regression tests.

Input and output filters are additional barriers, not complete protection. Highly sensitive data should not enter a context that the model can access unless it is strictly necessary.

Code example: the model must not grant permissions

Vulnerable design: execute the tool name and arguments produced by the model directly.

tool = modelResponse.tool
tools[tool.name](**tool.arguments)

Safer baseline:

ALLOWED_TOOLS = {'read_ticket': readTicket}

tool = validateToolSchema(modelResponse.tool)
if tool.name not in ALLOWED_TOOLS:
    raise PermissionError('Tool not allowed')

authorize(currentUser, tool.name, tool.arguments)
result = ALLOWED_TOOLS[tool.name](**tool.arguments)

Even permitted tools require server-side object authorization, narrow schemas, least-privilege credentials and limits. A prompt instruction such as “use safe tools only” is not an access control.

How is the weakness tested?

An LLM penetration test examines direct and indirect sources, multiple roles, languages and conversation sequences. It does not stop at unusual answers, but carefully confirms whether protected data or actions are reachable. Model version, context and success frequency must be recorded because results are not fully deterministic.