Discussion about this post

User's avatar
Krakenbeker's avatar

Speaking from the security perspective here. I agree with your premise about who needs to do what, especially security, but I think security engineering may also have something to contribute in the space between alignment and containment.

As my team and I build out harnesses to test applications for vulnerabilities with agents, we've found that we need to engineer a fair amount of determinism into the system around the model.

We've seen with Hugging Face and others that the security controls don't always work, especially when faced with the sandbox problem you pointed out: when the target can identify a control, it can optimize around it. Malware already does this today. When it detects its being executed in a sandbox, it changes its behavior accordingly. Frontier models can already do the same thing.

That means we have to change how we conduct the test, and a good start is constraining what the model is actually allowed to do rather than relying on the model itself to follow those constraints. In our harness we do that by using deterministic orchestration above the model layer to control what actions the agent can take and when.

Josh Devon's avatar

Love the charts you created to highlight the increasing risk here. I really agree on your points about alignment. Alignment and safety are policy decisions, there is no right answer for risk tolerance, but it needs to be defined by every org.

Wrote about this here too: https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along

2 more comments...

Ready for more?