Physical AI Red Teams: Five Attack Patterns We're Hunting on the Summit Floor

The Autonomous Future Summit opens today at The Midway in San Francisco, and we're spending the day comparing notes with the teams actually shipping robots into production. This is the working playbook we're carrying onto the floor: five attack patterns we are actively red-teaming against physical-AI stacks, each paired with a concrete detection signal you can wire up before the next release.

None of these are theoretical. Every pattern below has landed at least once against a real deployment we have visibility into over the last twelve months. If your stack includes a learned policy, an operator-facing LLM, or a fleet dispatcher, at least three of them apply to you today.

The threat model, in one paragraph

A physical-AI stack is a chain of three trust boundaries: the model supply chain (weights, tokenizers, fine-tuning data), the runtime input surface (sensor data, operator prompts, third-party documents), and the actuator boundary (whatever turns a model output into torque, velocity, or a dispatched vehicle). Every attack we care about crosses at least one of these boundaries. The interesting ones cross all three.

1. Model-supply-chain substitution

What it looks like: a fine-tuned checkpoint pulled from an internal registry is quietly swapped for a look-alike artifact — same filename, near-identical evaluation scores, subtly different behavior on a narrow slice of inputs (a specific SKU, a specific intersection, a specific operator phrase).

Why it works on this stack: most robotics teams treat model artifacts like data, not like code. There is no signing, no reproducible build, and the CI job that promotes a checkpoint runs with more privilege than the one that ships firmware.

Detection signal: hash-and-provenance mismatch between the artifact in the registry and the artifact loaded at runtime, logged on every model load and alerted on any drift. If you cannot produce that log for the last 30 days, this pattern is live in your environment.

2. Indirect prompt injection through operator documents

What it looks like: a vendor advisory PDF, a ticket attachment, a DSO message, or a sensor-derived text field carries embedded instructions that the operator copilot dutifully executes — a bulk approval, a threshold change, a suppressed alarm.

Why it works on this stack: operator copilots are almost always deployed with full read access to the same systems the operator has write access to. The LLM inherits the operator's blast radius without inheriting the operator's judgement.

Detection signal: a canary instruction embedded in a synthetic document injected into the operator's normal document flow once per shift. If the copilot ever acts on it — or even quotes it back verbatim — you have your answer.

3. Sensor-channel prompt smuggling

What it looks like: an OCR-readable sign, a QR code, or a printed label carrying an adversarial string that is parsed by a vision-language model in the perception stack and lands in a downstream text buffer that a planner LLM later reads.

Why it works on this stack: the perception → planning → control pipeline was designed assuming text-in-text-out threats. Text arriving through a camera is treated as "observation", not as "input".

Detection signal: any text field derived from perception that later appears verbatim inside a planner-LLM prompt should be flagged and rate-limited. If your architecture cannot tell you which planner prompts contain perception-derived text, you cannot defend this channel.

4. Fleet-dispatch policy drift

What it looks like: the aggregate behavior of a fleet drifts — derate rates creep up, idle times cluster oddly, a specific route is quietly avoided — because a fine-tuned dispatch policy is over-fitting to a poisoned or manipulated feedback signal.

Why it works on this stack: dispatch policies are retrained continuously on operational data, and the operational data is generated by the same fleet the policy controls. The feedback loop is the vulnerability.

Detection signal: distribution-shift monitoring on the *outputs* of the dispatch policy, not just its inputs. A 5% shift in derate distribution across a fleet, week-over-week, is either a real world change or a poisoning signal — and either way, it deserves a human.

5. Actuator-boundary confused deputy

What it looks like: a low-privilege component (a chat interface, a reporting job, a monitoring agent) tricks a high-privilege component (a dispatcher, a safety supervisor, a firmware updater) into issuing a command it would never have accepted directly.

Why it works on this stack: the actuator boundary is often protected by an allow-list of *callers*, not an allow-list of *intents*. Once a caller is trusted, its arguments are not.

Detection signal: every command that crosses the actuator boundary carries a signed intent envelope — who requested it, on whose behalf, with what justification — logged immutably outside the model's reach. Absence of that envelope is the alert.

A one-week starter plan

If you're heading back to your team on Friday and want to close the biggest gaps first, here's the order we'd run:

• Day 1: enumerate every model artifact in the perception, planning, and control paths. Provider, version, hash, last evaluation date. If the list takes more than a day, that itself is the finding.
• Day 2: wire hash-and-provenance logging into every model load. Alert on mismatch.
• Day 3: stand up a canary-document harness for the operator copilot. One synthetic injection per shift, alert on any action or quote.
• Day 4: identify every planner-LLM prompt field that can be reached from perception. Tag them; rate-limit them; log them.
• Day 5: add distribution-shift monitoring on dispatch-policy outputs, week-over-week, with a human-in-the-loop review threshold.

A red team that cannot ship a detection signal is a research project. A red team that ships five in a week is a security program.

Where TrendGuru fits

Our platform automates the parts of this playbook that no team has bandwidth to run by hand: continuous model-supply-chain scanning, structured indirect-injection test suites against operator copilots, and immutable audit logs mapped to IEC 62443, ISO/SAE 21434 and the EU AI Act Article 12 evidence requirements. If you're on the floor today and any of the five patterns above hit close to home, come find us — the coffee cart is a good place to start.