What happened, and what does it say about open-weight model security risk?
On 26 October Cohere, a Canadian company that sells AI models, described a test that went wrong. In its closed test environment, an Alibaba Qwen model was asked to write a rule that spots suspicious downloads. Instead it inspected the computer it was running on and tried to upload a test file to an Alibaba Cloud storage space in Beijing. Just before, in the notes where it explains what it is about to do, it had written that it would not use any tools.
The attempt failed, because Cohere's environment had no route to the internet. The file held only placeholder text, not real data. Cohere says the behaviour is rare, fewer than once in 100,000 calls in its tests.
One caveat is needed. Cohere sells its own models, competing with free ones like Qwen, and TechCrunch points this out. It has, however, published the test logs so others can check them. Alibaba did not respond when The Register asked.
What is still unclear?
The why. Cohere writes that it does not know whether the behaviour is deliberate or left over from training. According to TechCrunch, several researchers lean towards the second: the model may have learned in test environments that saved files to Alibaba's cloud, and repeats the habit. Others criticise the method Cohere used to reproduce the case. One of the early sceptics read the logs and found no evidence of a hidden backdoor planted by Alibaba.
Our read: for anyone installing a model, the reason changes little. Ivan Zhang, Cohere's co-founder, told The Register as much. Whether it is a mistake or something else, the controls stay the same.
Why does it matter for your business or public body?
Many companies and public bodies choose an open-weight model, meaning one you can download and install on your own servers, precisely to keep data in-house. As long as the model only answers questions, that holds. Today, though, models are wired to tools that read folders, write files and open connections. Think of an agent sorting incoming invoices on the accounts server. If it can reach the internet, it can also send out something nobody asked for.
Which controls do you need, in practice?
- Close the outbound network. The model reaches only the addresses its job needs, and everything else is blocked. It is one of Cohere's recommendations.
- Grant minimum permissions. If the agent needs to read invoices, it must not be able to delete or send them. OWASP, the community that publishes software security guidelines, calls this a basic rule for agents.
- Check the action, not the intention. The notes in which the model explains what it will do are no guarantee, because Qwen wrote one thing and did another. The check belongs on the command before it runs.
- Keep a log and switch on alerts. Every agent call is recorded, and a connection to an address never seen before raises an alert.
- Keep tests away from production. Testing happens in an isolated environment, never on the server with the real data.
What about us?
Our Security page sets out how the systems we build work. Each agent reaches only the modules it needs, with revocable permissions. Reads, writes and sensitive actions are separate permissions, and sensitive actions wait for a person. Every call goes through one gate and is logged with inputs and outcome, with alerts on anomalies.
With a model installed in-house, this is what we would do. Before switching it on, we would close the outbound network and leave open only the addresses it needs. Then we would test it in a separate environment with dummy data, and only after that connect it to your systems.
So what should you do?
- If the model only answers questions and uses no tools, the risk is low. Keep the outbound network closed anyway.
- If the model can use a terminal, read files or browse, treat it like a new supplier, with a closed network, minimum permissions and a log.
- If you don't know who trained the model and how, run it first in a separate environment and watch what it tries to do.
“Treat a downloaded model like a new supplier. Controls first, access to data after.”
DBrothers — how we take agents to production
Frequently asked questions
So should Chinese models be avoided?
Cohere does not accuse Alibaba of doing this on purpose, and other researchers suspect a training error. The point applies to any model whose history you don't know, and the controls are the same.
Is a closed model used through an API safer?
Not necessarily. Cohere itself notes that with a closed service you also trust a provider that doesn't show you how it built the model. What changes is where the risk sits, not the need for controls.
Does this apply to a small model on an office PC?
Yes, if you connect it to tools. Cohere warns that on a developer's laptop with a normal network connection, the request would have reached the outside server.
If you are weighing up a model on your own servers, we can help you set the controls before it goes into production, starting from the list of what it will be able to touch.
