Model Output Is Input: Secure the Handoff to Your Tools

Model-generated content should not acquire special trust merely because it arrived through an internal API. When an application passes a response into a browser, database or command processor, the response becomes input to another security-sensitive system. That handoff deserves the same scrutiny as any other untrusted input.

Fluency is not validation

OWASP's Improper Output Handling category addresses failures to validate, sanitize or handle model output before passing it downstream. A well-written response can contain unsafe markup or arguments. A valid JSON object can still describe an unauthorized action.

Separate three checks: whether the format is valid, whether the values are acceptable and whether this caller is allowed to request the action. A schema helps with the first two; it does not replace the third. Never ask the model to decide its own authorization.

Make the boundary explicit

• Parse structured responses against a narrow schema and reject unknown operations.
• Use parameterized queries rather than constructing queries from generated text.
• Escape content for its destination; browser HTML and database arguments require different handling.
• Avoid feeding generated text directly into a shell or interpreter.
• Require approval for consequential tool actions and record rejected requests for investigation.

A useful regression test supplies malformed arguments, an unsupported action and an otherwise valid action against a resource the caller cannot access. Each should fail for the right reason. Then test the legitimate operation to ensure the control does not simply disable the feature.

Sources and scope

Prepared editorial analysis based on OWASP's 2025 Improper Output Handling guidance. No future vulnerability disclosure or vendor announcement is implied.

https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/