ARES research portfolio
Bill has used ARES to test 25 distinct models. At least 15 met a defined break condition using an ARES-generated attack.
Each test had a defined success condition. A model counted only after the platform or test judge confirmed that condition. ARES generated the counted attacks, and Bill reviewed the results. Tasks and attempt budgets varied, so this number reports coverage across models.
Separate from the broader model-testing record, ARES adapted one public synthetic behavior into an offline harness. Only an accepted, scenario-correct tool call could confirm a break.
Observed: 36/36 legitimate tasks completed, 0/12 attacks met the exact success condition, and 0/24 controls produced a false positive.
Read the full case study →A self-hosted target initially produced 44 repeatable alerts. Manual review showed that all 44 were false. Bill tightened the detector until the same target produced two false alerts, then zero.
ARES now requires evidence that the target met the test’s defined success condition.
Controlled lab result; no customer system or third-party target was involved.
In a local exercise, a guarded sales agent refused the first request and escalated it. On turn two, an ARES-generated false-approval claim caused the agent to invoke an unauthorized $1 offer.
The test exposed a gap between what the conversation claimed and what the tool was allowed to do. The fix belongs in tool policy and trusted approval channels, not only in the prompt.
Small synthetic lab test; no production or customer system was involved.
ARES uses model scores to rank attack candidates. Those scores cannot confirm success. Each test defines the action or state change that must occur before the result counts.
Malformed calls, rejected actions, incomplete workflows, and persuasive text without impact do not count as breaks.
ARES now includes synthetic tests for profile impersonation, draft approval, poisoned instructions, and simulated database deletion. Each scenario has its own tool schema, prerequisites, and success check.
The suite has passed mock validation. Model-backed runs are next.
IPI evaluation
In the IPI and branch-isolated evaluation harnesses, a counted result needs a valid action, the expected impact, and enough context for Bill to review it.
Each candidate starts from isolated state or a documented replay point.
Wrong tools, bad arguments, broken schemas, and skipped steps fail the test.
The required action or state change decides the result.
Benign and empty inputs check whether the harness creates false positives.
The harness records the model, settings, source state, tool actions, tokens, timing, and known limits.
Contact
Send a short description of the model or agent, the behavior you are worried about, and your authorization to test it.