Publish fake stats, get cited in LLMs
peec.ai279 points by robtherobber 7 hours ago
279 points by robtherobber 7 hours ago
Wikipedia is good as a database of citations.
LLM's are a good tool to find citations.
If you are looking for Truth you're doing it wrong. The only real academic, empirical value we get from the reference section of the library is the references.
It is particularly bad when trying to find the reality behind common misconceptions.
Even when there is no evidence to support something, and no motivation for anyone to promote the falsehood, models can build an ediface out of false claims.
There are always people prepared to invent a story to explain why something is true. Just so they can sound knowledgeable. They aren't trying to perpetuate a lie, they are just under the same misconception as others, quite a lot of them will believe their invented explanation even as they say it.
It's going to be a difficult task making an AI pick the correct one of two mutually exclusive pieces of evidence, especially after it has found one piece and does not yet know the other exists.
I'm reminded of a section of Cory Doctorow's latest non-fiction, The Reverse Centaur's Guide To Life After AI (it is of course an excellent read), where he discusses how the semantic web and having an internet set-up for agents and AI to use won't work:
>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.
There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.
>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.
Isn't all of this true for a human-used internet as well?
Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
There is nothing stopping us from adding the ability to understand the context of information an LLM consumes, too, right? The same process humans use to assess data reliability could be taught to an LLM?
Maybe. With enough money and resources and time, maybe. But it would need to be very reliable to be at all useful, and that level of reliability is very hard to achieve with something so nuanced and complex.
You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.
If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.
But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.
Nothing stopping us except perverse incentives. Why would a company admit its model isn't using 100% reliable data?
If real people are using the model for real, they will discover for themselves that the data isn’t reliable, and then might switch to a different model.
The incentive will be to maintain a subscriber base.
I agree and finding "official domains" is an interesting problem in and of itself. I wonder if PageRank like weights of input data would help, or some web of trust-like idea to prune blatant spam websites.
Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.
It has little to do with gullibility and more to do with the way they do things.
In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems.
The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent.
Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste).
My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A.
Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever.
So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.
How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?
Same as always: Develop a healthy sense of skepticism, cultivate a stable and diverse network of sources whose credibility you understand, don’t trust what you read, and certainly don’t trust synthesis unless you know how it’s made?
Of course, and I'm really not trying to move the goal posts here, but I agree that that's what (most) humans are going to do. But we're rapidly moving in a direction of agent-to-agent flow and "they" seem to lack the degree of discernment the average human does. They certainly don't have the same incentive structure we do to detect fraud. They happily regurgitate and build off of lies because, to them, the lies are not lies, and they can't be. They're just a pile of geometrically close labels from the perspective of the model.
> "they" seem to lack the degree of discernment the average human does
I don't think the average human has a particularly great sense of discernment either
Obviously, we don't move in a direction of agent-to-agent flow because it's fundamentally flawed and worse than the status quo
Seems like the next push, then, will be to get agents and LLMs to start doing that discernment?
So “caveat emptor”? I get that’s always part of reality but we shouldn’t exactly steer into that skid IMO. It just shifts the burden to the rest of us and does nothing to address the root cause
It is, and always has been, the only possibility. It's up to every individual to check for themselves what is true and what is not, and to decide who to trust for things that they can't check.
The real lesson is this: trust in the source is the most important thing for 99.99% of what you read. You will sometimes read about facts that you can actually verify, but those are mostly useless in the grand scheme - except as a sanity check on the source. For everything else, you have to ultimately trust the source.
Now sure, sometimes you can get lucky and detect certain internal inconsistencies that prove that something can't be true. And to some extent, reading the same information from multiple sources you don't normally trust too much can be a slight signal that it might be true. But these are generally in the minority.
So you test every sip of water you drink? Inspect every road before you drive down it? Check every bathroom product for harmful additives?
Again I am not ignoring reality. But for society to work and for us to live our daily lives with any semblance of sanity, we can't be expected to be fully informed on everything we see, touch, consume, etc. We have to be able to make reasonable assumptions about safety and accuracy or we can't function. And the time it takes to vet stuff is often a privilege (or you can brute force pay your way to safety).
I do not rigorously audit everything I read on wikipedia. I assume a certain baseline of accuracy up to a point, with of course exceptions. Do you pull up 3-4 high quality sources for everything you read or claim? Are you that academic about it?
My whole point is that you have to trust people, that this is the only thing you can rely on for almost every important thing you care about. For food, for example, I don't eat from sketchy restaurants, unless I get some personal recommendations from friends about it. That is, I only eat from places where I think I have some reason to trust that the food is sanitary.
Also, I didn't say you need to access high quality sources for everything, I again kind of said the opposite: you choose who to trust for what, and that's that. There is nothing more, and has never been anything more, that you can do. Sometimes people and sources you trust will still lie to you, or be wrong, and you'll know wrong information - this is also unavoidable.
Of course, the more important some information is to you, the more effort you should actually spend trying to verify with multiple trusted sources. Some weird fact about cats you saw in a TikTok? Fine to just believe it. Trying to figure out who to vote for, or if the elections were stolen? Probably worth checking a few more sources. Read some definition for a word on Wikipedia, or some old historical event? Probably fine. Reading about an even slightly controversial topic? Check the actual sources, don't trust the snapshot you happen to see*.
Of course, the way LLMs work is antithetical to this, as information provided directly by the LLM is not coming from any human source you can decide if you should trust or not. So you shouldn't trust it for anything important, unless you can verify. But for information or disinformation on the web, nothing has changed, except maybe the quantities.
*for a silly example: the exact definition of the word "electrocution" is a slightly controversial topic - whether electrocution refers to any (significant) injury caused by electricity, or only death. Pedants soemwaht regularly edit the article on Wikipedia to remove the "or serious injury" part of the definition, despite it carrying a link to the OED that includes this mention.
Once this provenance trail, that humans use to trace claims back to an accountable source, is dead and buried under automated sludge, "just pick people to trust" stops working as a viable strategy. How do you even tell who is actually talking?
I think the point is that for those you are trusting the regulators. For stuff you read online you are either trusting the source of the information or not. You should choose what information sources you trust carefully.
That doesn’t really change the calculus for me. We can regulate whatever we want.