Garry Tan wants US open-weight AI labs to 'distill' frontier models, too
techcrunch.com412 points by TheJCDenton 3 days ago
412 points by TheJCDenton 3 days ago
I agree. The frontier models are based on training data from tons of copyrighted work. Some of that work was obtained illegally, even. They could not exist without strip-mining the commons. The labs have no moral or ethical ownership to the end result, and others should feel free to treat any company-imposed restrictions on their use as invalid.
I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.
With the recent Navier-Stokes controversy, I think there's a credible suspicion that all your IP you run through these models will end up in these companies' possession. OpenAI themselves has admitted a weak version of this (that prompts might inadvertedly end up improving the model). We don't know the extent of this.
Obviously it's not possible to run a company whose value is predicated on its IP that uploads said IP to a third party which might get access to it.
This could mean every potential serious customer would have no option but to seek alternatives to these online services.
I thought this was commonly accepted to be the case that companies which sell access to LLMs are also storing and training on the inputs?
I don't mean this as rhetoric, I did not think many people (except possibly those operating under government contracts, and 'normies' who don't know about these things) were under the belief that their IP was kept secret when they use these services.
Some offer zero data retention policies, but there can be weasel words. For example, on the individual pro plan, you can turn off the setting that lets them train models on your data, but they still have a section in their terms that allows them to evaluate your anonymized data for statistical and "research" purposes. You have to actually get a signed contract along with an enterprise plan that spells out exactly what they're going to use, and what settings enable what retention.
https://privacy.claude.com/en/articles/10023548-how-long-do-... (see the additional info section)
Or just use Azure, AWS, etc. for Claude/ChatGPT inference, where the AI labs never even get your data in their data centers at all.
You pay more for it, but if you care that much, use it.
Seems naive to think that those providers - who have a financial interest in selling the data - would not also try to weasel out of the precise definition of ‘zero’ retention.
No, that is not "common knowledge". You are supposed to be able to disable that unwanted feature.
I have no inside information, but I always assume the tickboxes that "disable ____ data" from Google/Facebook/OpenAI just disconnects it from your own account, not hides it from the provider.
The checkbox has an actual statement associated with it about what it does. You don’t need to assume anything.
There's no independent verification of what that checkbox actually does. The company can say anything, and you are unable to verify that they actually do it.
The only verification you could do so far is GDPR-style data export, and also the adherence to GDPR regulations (and even those might get skirted if they aren't operating in europe).
Aren’t they usually phrased very specifically as “we collect this data and use it to show you relevant ads, you can opt out of us showing you relevant ads”?
What about inference providers like Baseten, Modal, Fireworks, Together, etc? I thought one of their value propositions was inference (using open weights models) that guarantees with crisp terms that they will not use your data.
I worked very briefly at Baseten, and I can say that it was a perpetual annoyance (from an engineering perspective) that customers would complain about issues with their models but we couldn't actually see the inputs/outputs. I don't know about the other providers, but at Baseten they literally weren't stored anywhere.
A provider can genuinely avoid storing inputs, as the Baseten engineer below describes. That is still different from proving what code received the prompt or protecting plaintext while it runs; I built TrustedRouter to separate ZDR, attestation, and confidential routes: https://trustedrouter.com/blog/attestation-is-all-you-need?u...
> to separate ZDR, attestation, and confidential routes
Could you please clarify what that means? Given what I've been searching for, I might in principle be part of your intended customer profile, but I can't figure out whether you are merely doing routing (alternative to OpenRouter) or also inference (alternative to the names I've mentioned above). If it's merely routing, then how do you protect me from any potential misbehavior on the part of the inference provider?
Just feedback for what you're building, so please take this in a positive spirit... I'm an AI researcher and not quite an infra guy, and I'm making recommendations on token APIs for several less knowledgeable around me (I've gotten a few people set up with Baseten recently), and I couldn't figure out whether/why I would be interested in TrustedRouter. You should communicate the story better :-)
EDIT: Here's what I now understand after some digging; please correct if wrong.
There are some M token providers (not the names I listed above?) who provide cryptographic guarantees about inference services. But somebody still needs to verify what they do on each request. For an individual running a single harness, that harness would be a logical place to perform this verification if possible. For an org with N users each running their own harness, TrustedRouter solves the N*M problem and becomes the single gateway for trusted inference -- provided one somehow trusts/verifies TrustedRouter.
yes, and we are also a router for the people just wanting routing and only want zdr or uncaring about privacy
it’s all transparent and on github. i’d recommend just pointing your agent at trustedrouter.com since its well documented but quite a large product
> using open weights models
AWS and Azure give you the same thing for Claude and ChatGPT, no need to be stuck with open weights. They might sometimes store some of it for other purposes (I don't know the specifics), but it is emphatically not being fed back to OpenAI or Anthropic.
I don't have any much exposure to the attitudes people have around them, and I haven't worked with them. So I can't really say
I would wager that’s more acceptable if said learning is not in competition with the user. If they didn’t actually produce results but created the model only, then that could be advantageous for users too. But the moment they absorb your work to sell it, or for marketing, it’s a different moral ground.
You have some secret sauce. The model trains on it. Your competitor is solving a similar problem. The model "advantageously" helps them.
Your competitor is happy and continues to pay for the subscription. Sam and Dario just resold your code.
For what it's worth LLMs still suck at reproducing my little secret algorithm/implementation while being able to solve way harder problems. I have a good guess why that's the case.
> I thought this was commonly accepted to be the case that companies which sell access to LLMs are also storing and training on the inputs?
The services have toggles to allow prompts to be used in the training set. There is a conspiracy theory that the toggle is a false distraction and they’re actually keeping everything, and that none of the employees involved will ever whistleblow this fact.
Outside of Internet comment sections, I think most people assume these US-based companies are doing what they say.
For enterprise use there are services like AWS Bedrock which have strict isolation guarantees. There are some people who still believe those guarantees are a lie, but once someone has reached that point I don’t think they trust anything that isn’t running entirely within their house. People in that category are a very small minority, but a very vocal minority.
The impression I have (from interacting with people IRL using OpenAI and Anthropics offerings, and how they feel about the risks involved) is just the opposite. But we probably just have different life experiences.
I can name groups of people I interact with who lean both ways.
It’s still a commonly held belief that “Facebook sells your data” and it’s cool to be cynical about everything tech in many social scenes. Conceding that a tech company might be honest about something will get you classified as a bootlicker depending on who you talk to so the only winning move is to be super cynical.
Among actual professionals I work with in tech and legal, almost nobody holds a belief that these companies are blatantly lying to their customers (and zero of their employees are whistleblowing it, while said companies also have employees trying to whistleblow AI safety on Twitter daily)
I also don’t think these companies are lying at all, but I definitely think they’re training on all your data, toggle or not.
It’s truly trivial to “anonymize” and distill your prompts and model output. They could use just about any off-the-shelf cheap model for this. In fact, their TOS explicitly allows this, even with the toggle checked.
What that probably means is that the EXACT content of your prompt is secret. But the actual ideas are not. If you discover something truly novel, then yeah they get that. They can absorb trends in consumer behavior, too.
I’m sure if someone had access to all my paraphrased prompts, which retain 0% of my exact wording, they could find out literally everything about me. It’s a bit like how collecting metadata is as good (or better!) than collecting the real data.
And we all know “anonymizing” data doesn’t really exist like we think it does. Just removing names and identifiers doesn’t make anything anonymous for motivated actors. Or… say… an LLM that is trained to recognize patterns in text. Which is, like, all of them.
Yes, I am talking about working professionals who use LLMs. Before this thread, I would have considered it surprisingly and singularly naïve if someone told me they trusted OpenAI. I still believe the common and correct take is that these companies are largely training on customer data against their consent.
I don't think they are "blatantly" lying either, just normal bog-standard lying that we've all come to accept. It's a profitable and competitive tech company.
We have already seen this lying. The toggles are opt-out, not opt-in. When you sign up, you agree to binding arbitration, which is effective for preventing lawsuits in the US. The toggles are regularly turned back on without our consent on ChatGPT and Claude. OpenAI's "don't train on my content" setting isn't even in the ChatGPT interface.
As far as I know, they haven't suffered even a tiny controversy in public opinion over any of this at all.
There's nothing to whistleblow about when it's public knowledge.
How many of the people who checked those boxes have cryptographic proof they did it? How many of those people have opted out of the arbitration clause? How many of those people would be able to claim damages? Would the amount of people who satisfy all three questions be large enough to make it worth _not_ training on user data?
> OpenAI themselves has admitted a weak version of this (that prompts might inadvertedly end up improving the model). We don't know the extent of this.
I think this is being misunderstood. Codex has a toggle to allow your prompts to be included in training data. They’re saying they can’t be sure if the person had it on or off while using Codex to discuss the work.
They’re not saying that some prompts are mysteriously jumping into training data.
Also, there is a large market for AI services which don’t retain anything under any circumstances for enterprise customers.
yes this is my understanding as well, and based on [1] seems to be the case. I don't know why everyone is just believing the un-backed accusations of people probably just didn't turn off said setting (and if they did why have they not said anything to such effect)
> I don't know why everyone is just believing the un-backed accusations
Conspiratorial thinking is very common on these topics. Even bringing up the conspiracy theory about Instagram listening to your conversations and showing you related ads will bring up a surprising amount of people defending that idea on Hacker News.
Or these customers could just use AWS Bedrock...but their current CEO is an incompetent MBA unable to publicly articulate their biggest advantage, in the context of the current AI usage my companies.
You have access to all the frontier models, but...your inputs are not shared with the model vendors...neither are used to train the next model.
Why am I even doing the Amazon board job for them!??
Bedrock is really bad. It seems like they don't host the models very well because they produce tons of bugs/errors calling the model. For example you can end up with Anthropic models not returning a stop token and you end up waiting for a timeout thinking its doing something when it isn't.
Well Anthropic hosts their models at AWS, ( and at many others...) so maybe the AWS team can ask them how they do it ;-) ?
Amazon is deeply invested in Anthropic and would not defame them through marketing a service whose selling point was their startup's breach of contracts.
i just wish they updated their open weights models. so many are so far behind what is available now.
All except Gemini which can be rather important depending on your use case.
You mean the Gemini that is even behind the Chinese models?
Yes, the Gemini that might be “even behind the Chinese models” but is uniquely capable of “watching” video.
> OpenAI themselves has admitted a weak version of this (that prompts might inadvertedly end up improving the model).
2023:
"The approach also aligned with the company’s broader deployment strategy, to gradually release technologies into the world for people to get used to them. Some executives, including Altman, started to parrot the same line: OpenAI needed to get the “data flywheel” going."
https://www.theatlantic.com/technology/archive/2023/11/sam-a... https://archive.is/NmO5P#selection-979.907-979.1177
I don't think this has been a big secret.
Almost every serious customer is already using ZDR where nothing is retained at all, instead of "anonymized" data.
They already trained on pirated content, what makes you think they are going to honor ZDR?
Contractual obligations carry teeth. Scraping the internet is relatively risk-free.
Anthropic happily paid billions to settle a lawsuit for pirating books. It's a trivial cost of doing business. If you're lucky you'll get a pittance after the fact by suing them, but a contract doesn't prevent them from doing the thing you don't want them to do and that they are obviously going to do given their past behaviour.
Good luck proving your data was laundered and included in a training run
Lucky for us Apple is already alleging something to this effect in their trade secret lawsuit, so you know they'll make sure discovery turns this up if it exists.
ZDR is based on the exact same pinky-promise as training opt-outs. There is no technical barrier to OpenAI, or whoever is running your compute, retaining your prompt after they run inference on their servers. If you don't control the hardware the model is being inferenced on, you don't control your data.
A lot substance is hinged on the exact definition of the word "data" or "user data". In the age of post-truth everyone is claiming that they keep no "user data". Except that after running it once through some transformer program it's no longer "user data", it's something entirely else and these corpos gave ZERO promises regarding such laundered/transformed data at all, ever.
Just a thought experiment: considering training seems to be 'fair use', I wonder if they trained a tiny model to retain key info from your prompts, would mean that this would still constitute fair use, and allow them to legally claim they don't retain your data.
ZDR is shorthand for a more specified agreement of "we don't do anything other than generate your output tokens", so no.
Besides, true ZDR is usually offered by third-parties with deals to host OpenAI models, such as Amazon (AWS Bedrock) and Microsoft (Azure).
The guarantee on this is a (contractual) “trust me bro”, and a right to try to sue a multi-trillion-dollar company who will absolutely drive you into the ground with legal red tape.
If you are big enough to be able to withstand that, you’re already running (or trying to run) your own/open-weight models.
Doesn't Amazon Bedrock change this, since OpenAI does not have access to the data?
Well that is a different thing and an entirely different provider than OpenAI/Anthropics ZDR promise.
Abolish copyright and make it less ridiculous. Sampling music was never a thing that required royalties until the 1990s when I guess someone got angry that rappers were making money off their sampled music. Its insane to me. Make it illegal to transfer ownership of copyrighted work too, only the spouse or one single inheritor who isnt a company can have the rights transferred, after both die, the work enters public domain.
LLMs should just pay a flat fee to use a specific book and thats it. Fees should be reasonable (not a million dollars per book), so long as the model doesnt spit out the entire book.
If someone really wants to ask a million dollars to let a book be trained on, ok their choice, maybe not getting any customers though. If someone really pays then cool, you deserve it for making what is apparently a very useful book. There are encyclopedias that probably cost more than that to make and would sell for more.
Lol um no everyone benefits from copyrights and IP. If were being flippant how about people just steal your private code and monetize it!? Copyright makes the creative world turn.
Sounds awesome dude, steal whatever code you want from me. Most of it isn't even truly mine anymore because it all came from an LLM anyway.
Yeah, I'm fine with copyright existing even though it's messy. That said, if you put your copyright image on a public site with no watermark and Google Images shows it, skill issue.
> Make it illegal to transfer ownership of copyrighted work too, only the spouse or one single inheritor who isnt a company can have the rights transferred, after both die, the work enters public domain.
By your phrasing, it sounds like you still intend the possibility of companies owning copyrights; but how does that happen (other than copyrights already owned by companies grandfathered in)?
Copyright always starts off in the hands of individual human beings; it only ends up in the hands of companies when those human beings transfer ownership to a company. That ownership transfer can be automatic as a term of a contract, e.g. as part of a work-for-hire agreement. But no contract can cause the copyright to come into existence already held by the company instead of the individual. So if you abolish ownership transfer, you effectively make work-for-hire IP assignment invalid. What replaces it?
And, if "nothing"... then how do people pool the IP rights of their own small contributions to a large-scale work, into an IP pool that can be legally defended by a coherent legal entity, so that the large-scale work itself can have market value (i.e. so that sales of polished commercial bootlegs don't drive sales of the "authentic" work to zero)?
Keep in mind that, no matter how much we might want "mass distributed" media to have more-reasonable IP terms, the ability to sue for infringement is still critical to the existence of some forms of media. Especially "location-based" media, with no equivalent licensed broadcast right: movies still in theatre; concerts; live performances of plays and musicals; etc. If there's no legal team that can sue a movie theatre that shows an unlicensed copy of a given movie, then no movie theatre will ever bother with licensing movies again; "box office" goes to zero (from the movie company's perspective); and the incentive to create movies in the first place declines massively.
(You can see what this alternate world looks like from the few cases where movies screwed up the steps required to assert copyright, back before copyright was automatic. Night of the Living Dead (1968) is a good example: theatres — even upstanding large-chain theatres! — did indeed leap at the opportunity to show the movie unlicensed, and so Romero et al made effectively zero revenue off the work.)
I'm not saying this is an impossible problem. There are ways to accomplish this besides the way it's done now. (For example, individual-contributor IP could be retained by the original owners, but cross-licensed between individuals through a collaboration structure to form a coherent defensible IP pool, in exactly the same way that IP for e.g. video codecs is cross-licensed between corporations to form a coherent defensible IP pool today.) I'm just pointing out that the problem does need to be solved.
> you still intend the possibility of companies owning copyrights
I would prefer it not be handed over across various copyright trolls, the fact nobody can sing "Happy Birthday" in a movie is still insane.
One of the most infamous legal challenges to sampled music was MARRS "Pump Up the Volume" in the 1980s, and that was preceded by other famous cases. Not sure why you think that started in the 1990s.
This is nitpicky. The MARRS case was 1987, and Biz Markie and Vanilla Ice are way higher on the list in terms of actually getting attention on the issue and influencing culture.
The USA is not the whole world. I heard a lot about the MARRS case as a teenager, and I wasn't even in the UK where it happened.
This. Distillation “attacks” are a made up concept. It's as if I claimed that Anthropic made a “training attack” when training on my internet writing.
Anthropic carried out a multitude of "copyright attacks" on open source repositories, and the broader internet.
If the leading private labs attempt to use the government to pull up the ladder under the pretense of "safety" then the response of the people should be to take such questions out of private hands and nationalize the leading labs.
Or they could abide by the precedents they set and learn to compete. They shouldn't be allowed to have it both ways.
The problem here is you need a trustworthy government for nationalizing to make a difference. The current US admin started with DOGE and a crypto rug pull.
How can "the people" nationalize a lab? I'm people, how can I do it?
> I'm people, how can I do it?
Vote (well-informed of the candidate's policies) in every election you can, even the local ones that seem of little consequence.
Convince others to vote.
Make demands of your elected representatives. You can mail them, call them, etc.
The government is the people.
The Reagan-era and beyond successful convincing of people that the government is an unchangeable black box made up of shady actors out to destroy everything (see: Republicans still going on about the 'deep state' when they run literally everything) is a big part of how we got to this place. It was a self-fulfilling lie, now coming true as the people who sold the lie start grasping for unending power.
But we still have the ability to vote our way out of it. If we continue to fail to do so, then at an evolutionary level we have to consider that we collectively deserve all the bad that comes from it.
Voting only changes things when it doesn’t threaten the interests of elites or there is a sufficient counterweight in terms of a competitor nation or a radical labor movement or an uncontrolled armed insurgency.
See salvador allende in chile and mitternand in france for examples of voting without sufficient leverage.
your regan example occurred during a successful counterattack by capital that started under carter and crushed the labor movement.
Not really much of an insight.
Sometimes voting changes things, sometimes it doesn't. Naturally it's in the nature of most governments to not wildly change every election. This is usually thought of as an advantage. It's rare a wide majority of the electorate ever actually agrees on anything anyway, and tyranny of the majority or government that is too reactive to popular will is also not desirable.
But sometimes things are different, due to external or macro factors as you indicated. Some elections are more consequential than others. That doesn't necessarily mean any one election simply doesn't matter. It's simply good civic hygiene to vote. Think of all the people in red states who vote blue or vice versa in elections they will never win, never mind have their chosen candidate "change things". Does this mean they are fools? I salute them for such a faithful exercise of their civic privileges, that which generations of humans have died for. Your vote doesn't simply lose all value because you didn't get the outcome you wanted. It also doesn't preclude other avenues, like direct action or whatever your supposed alternative is.
It's good to vote simply out of civic obligation. Many people who say otherwise seem to merely be complaining that voting doesn't give anyone 100% of what they want. We've seen more and more "insurgent" candidates take over primaries. Increasingly people who complain about politics strike me as engineers who talk instead of build. Complaining about the effectiveness of your vote is like listening to an engineer who is always talking but never ships.
Like go run a campaign or solve the problem. What does saying voting doesn't matter solve? It feels like middle school age cynicism masquerading as insight. Politics has actually rarely felt so unsettled in America. Voting is likely more important than ever.
Building labor unions is more valuable than voting. I think you should still vote, but understand voting without leverage will result in your candidate bending the knee. An example is Mamdani being elected in NYC, but allowing a known class enemy to run the police.
Yes, this clarification makes sense and more politicians should also make this explicit, as they often end up disappointing voters who think their civic obligations start and end at the ballot box.
A politician who doesn't agitate voters to take direct action in between elections is ultimately one who hopes you really don't pay that much attention to what they are doing. It's always about electioneering or diverting attention elsewhere. It's fine to say voting is merely the bare minimum check on elected officials, as long as you don't encourage anyone to then skip this necessary (albeit preliminary) step.
This clarification is a goal of DSA members. Elected officials should use their position oppositionally and to tell the truth about our system. For example reading government secrets into the public record, telling the public the truth about who is steering various debates in Washington, promoting the nationalization of companies and formation of cooperatives.
Mamdani did do something interesting in this vein recently. When the racists on the right said he couldn't go to a 9/11 memorial, he released documents showing they allowed people to be poisoned in 2001.
> Voting only changes things when it doesn’t threaten the interests of elites
It absolutely does change things even in those cases. You just voted for the wrong person if they aren't doing what their supporters want.
Mitterrand did attempt to do the things his voters wanted. He was constrained by capitalists who punished him mercilessly until he bent the knee. Allende was simply couped and assassinated.
Voting changes things when those changes do not substantially threaten elites and they can come to some kind of acceptable deal, or when you have sufficient leverage that when your guy gets in they can steamroll the capitalists.
Leverage isn't "votes", that's just a preference on a piece of paper. Leverage is the ability to materially change reality to reward or punish actions. A strike, granting or witholding financial resources, shutting down infrastructure, armed revolt, etc.
> Leverage isn't "votes", that's just a preference on a piece of paper.
It actually is leverage because those people who get the most votes are in charge of our government.
The government monopoly on violence, controlled by the people who get voted in, is the clear and obvious leverage.
> Mitterrand did attempt to do the things his voters wanted. He was constrained by capitalists who punished him mercilessly until he bent the knee.
Sounds like he didn't get enough of his own people in government then.
If you fail, then it means that you didn't get enough people voted in. Having leverage requires more than just winning a single election. It involves winning many elections and getting widespread support and your people in government.
The attack did not come from inside the government alone. It was for example, capital flight. The rich attempted to starve the French economy to win concessions and they won them.
People that control the government in principle control the guns. People that control the economy hold the real levers of power. You can in principle use guns to get control of the economy, though in practice it is trickier. America has done it on many occasions though, blasting apart democratic governments in South and Central America to install neoliberal puppets.