Concerning behavior from a popular open-weight AI model
cohere.com327 points by matthieu_bl 2 days ago
327 points by matthieu_bl 2 days ago
By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)
> By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose
Yes. Even if it's poisoned training data, RLHF fuckups, or something more malignant: the model, wherever it comes from, is in your supply chain and it makes you more vulnerable.
As much as I doubt that Alibaba have any great interest in global espionage, and this is much more likely to just be a consequence of being trained on Alibaba's own products, the fact is there is a geopolitical dynamic here.
This is basically the same as hiring a new person and giving them normal network access to get their job done. Did they upload a file to Beijing? Who knows?
It's much easier for the Chinese government to tell Alibaba they must perform espionage using their models than the US Government. Even if every last engineer there joined the company just for "AI Research and Development", they might be under scrutiny or forced to shutdown if they don't work with the CCP. Power is shared between corporations and the government in the west, the government is the power in China.
Doesn't the NSA have wiretaps in every ISP? Gag orders, secret courts? Do i remember wrong: the NSA backdoored a cipher suite?
I'm comfortable with the idea of my own country conducting espionage. I can believe that China does it more; but the idea "we would never" is naive.
wiretaping an ISP is vastly different from say, ordering and forcing Anthropic or OpenAI to put malicious weights into their models. Not once in my reply did I ever imply the U.S. doesn't perform espionage. Look up Five Eyes.
I think that line is getting blurrier by the day with the current administration. The companies very quickly fell in line before even any consequence of note. Donations to the ballroom, calling it super intelligence, renaming the gulf, etc.
Would you really trust that OpenAI wouldn't skew the results to tend to a certain way or that Grok will always be neutral in how it operates?
From a third world, there was a time when US products & companies were more trustworthy, I agree. But, there has been enough display from these Cos to show that they have lost that badge.
Sweet Summer Lake America Child
I mean, China regularly executes its CEOs. The only sweet summer child here is you not understanding how Chinese politics works.
Trust is not a binary choice between 0 and 100, correct? So because we can't truly see any poisoning, we should treat it like any other piece of closed source software and use a trust level based on the source, right? Maybe one person might accept running closed source from NVidia (even with nv's strong connections to the US gov) but might not trust closed source from a source with even stronger Chinese gov connections -- and someone else might have opposite trust levels for those sources.
> By that logic you can't trust any model
Correct!
But you cannot trust any human either because who knows what someone told them ...
This is true, but up until about two and a half years ago, we built IT systems to be more reliable, consistent and auditable than humans, and to provide some security guardrails on human malfeasance.
Now we're OK with building systems that are an order of magnitude faster but behave no better, because what?
Because number go up!
what numbers? numbers of debt? CDS to hedge against the AI model producing companies banktrupcy?
Probably more because number (of humans employed and how much they're paid) go down.
This was already a well-known problem back in the dark ages, when BERT models were cutting-edge. The model is just a compression of the data, so if the data is poisoned then the model is too.
So just like working with People :) You might have a Snowden, Mark Felt, Jeff Wigand etc working with you everyday.
Except where does the accountability land?
If an individual employee does something unethical or illegal, the employee is accountable and may lose their license or face penalties, and if the company performed the right due diligence and training first then responded "correctly" when they learned of the employee's behavior, their liability might even be zero. There is case law on this topic.
But if a MODEL mis-advises or acts in an unethical or illegal way, who is accountable? The business team that spec'd and tested the solution/product? The architects and the developers? I've never met an architect or consultant with professional liability insurance.
There isn't enough tested law in this area yet.
It might not be tested. But, it would seem very weird if anyone other than the person using the LLM was accountable for the output.
What if the company has provided the harness, tool execution environment, MCP servers, and LLM provider subscription for the employee to use? What if the company has provided a set of skills and other prompt context that could lead to undesirable behavior in certain unforeseen circumstances? What if the permission prompt the user sees 30 dozen times a day looks like, "OK to execute this random shell script?"
Accountability really is hard in this new world.
Just like would imply intent, most of us are chalking this up to the non-determinism, Ai has no intent
AI has will and valance.
How do you define these words? Might goals and autonomy be more accurate words? What evidence do you have beyond their manipulation of words?
Do you also think they can get horny or feel emotions? Is the word will or goal closer to this answer?
If AI has will, then so do rivers and clouds.
People thought ELIZA had emotions. We are a gullible species, attributing humanism to anything that displays features we currently believe only humans posses
Concept Fission throughout human history is a very interesting perspective. For example, we do not even think about growing food, but many centuries ago you only ate the food you made. Today, we can see this with texting and online comms, which some people seeing as a form of socialization while others (older generally) believe you have to be face-to-face in order to socialize.
I expect providers of language models are well aware of this psychological phenomenon of anthropomorphism and projection, people's tendency to form emotional bonds with inanimate objects, and have trained the models to perform as a conscious subject, using "I think", "in my opinion", etc.