Verification Is Not Knowledge: Science needs more than answers
blog.cosmos-institute.org244 points by ersinesen 3 days ago
244 points by ersinesen 3 days ago
When Kepler published his models:
* https://en.wikipedia.org/wiki/Kepler%27s_laws_of_planetary_m...
People did not understand how or why they worked, but they switched over to them from the Ptolemaic (Tychonic, etc†) system anyway because the math was easier. It was only later (Newton, Einstein) that we understood the "why" and "how".
† https://tofspot.blogspot.com/2013/08/the-great-ptolemaic-sma...
Kepler did understand the data he had, where it came from, what his laws were claiming, how to verify that the data fits the laws, how the data dis-proves heliocentricity etc, which is really what the article is talking about.
Kepler's models showed that the planets travelling in elliptical orbits matched observations, but Ptolemy (and Brahe, etc) also produced models that (tried to) match observation.
If Kepler's models explained the how and why, why were Newton and Einstein necessary?
Newton and Einstein answered "how" questions with more generality and more precision.
Kepler's model predicted the behavior of the planets, and so does Newton's model, but Newton's model is more general, applying not only to planets, but also to apples, cannonballs, trains, and other objects with mass.
Einstein's model is more accurate and precise than Newton's model, especially about the orbit of Mercury. Einstein's model also predicts gravitational lensing (e.g. during eclipses), which Newton's model doesn't.
"Why" questions are very hard for physicists. Generally, when we ask "why" questions, we're asking what prior Event X caused Event Y. But for universal laws, there is no prior event that caused the universal law to be true. The best we get is more generality. Planets orbit the sun for the same reason apples fall to the earth, but we don't know why gravity happens, only how. Magnets resist each other for the same reason the floor is hard, but we don't know why magnetism happens, only how.
> "Why" questions are very hard for physicists.
Yup:
* https://en.wikipedia.org/wiki/Principle_of_sufficient_reason
Newtonian mechanics doesn't explain the why of itself, it explains why Kepler's laws are true, an explanation that won't make sense if you don't know what Kepler's laws are saying, and which won't hold any significance to you if you don't know why Kepler's laws are thought to be true.
Weren't Kepler's models worse at prediction, at first? We adopted them because they were more parsimonious.
This is just going to devolve into more tribal elders pontificating to be possessed of knowledge the average person won't be able or allowed to falsify; the expert can clearly see how their interpretation is correct and it's disrespectful to challenge elders nonsense
Tao and such are merely biological beings serving biological imperative to survive. Educated as they are they are well aware social society expects them to engage in knowledge role-play to barter for food
The recursive non-fractal nature of reality where "function exits" when a human dies naturally fosters signal attenuation and the formation of new spoken traditions among the future generations
There is no propagating our social ways 100 years into the future. We can only propagate our material outputs. The semantics we define to describe something are lost to time.
The provenance to future people was past people; that's all that matters in the future not how we communicated together now
> how to verify that the data fits the laws
What? He worked meticulously to match the observations.
I don't understand your claim here at all.
When the article brought up the cancer drug analogy, I thought roughly the same thing. We still don't know why certain drugs work (take gabapentin as a prime example), but we're still comfortable using them on a regular basis.
Yeah. Medical practice is often based on purely statistical statements like "in case of A associated with conditions B if you do C you'll help D% of the patients". Sometimes looking up a drug's mechanism of action yields nothing, literature just says it's unclear and gives us some educated guesses if we're lucky. Makes me a little sad that medicine has been reduced to this instead of the romantic "understand how the body works, how it fails and how to fix it" ideal.
Kepler did make progress - he showed that equal area covered by the planet on the orbit implies that the orbits are elliptical (and vice versa). Enough progress for the next guy to take it further.
The next guy was Newton - he knew the solutions to a central force problem (thanks to his invention of Calculus - notation for calculating with infinitesimals). Movement with conserved angular momentuum <=> movement with the equal-area property <=> happens for a force ~1/r^2. From that he concluded that the central force is "gravity". The deriveration is simple enough for 1st semester physics student, but you will need to cleverly change variables once or twice to get a familiar integral.
https://en.wikipedia.org/wiki/Central_force
https://en.wikipedia.org/wiki/Classical_central-force_proble...
I'd be careful with the inference you're making here. Tbh we could easily conclude that neither Newton nor Einstein explained "why". One could argue we still don't know and won't even after we have a TOE.
I'd take Asimov's "Relativity of Wrong" here. And I also think there's motivation for why scientists care so much about elegance.
> And I also think there's motivation for why scientists care so much about elegance.
It's an intuition you build pretty early on in science courses in undergrad, somewhat related to Occam's Razor. It's why the optimal packing of 11 squares[1] is so unsatisfying. It flies in the face of all the intuition we've built that simple (and symmetric) solutions are more likely to be true. Machine learning practitioners use this intuition when performing model selection as well: the simpler model (i.e. with fewer free parameters) is less likely to be overfit to the data.
[1] https://en.wikipedia.org/wiki/Square_packing#/media/File:Pac...
> It's an intuition you build pretty early on in science courses in undergrad
I think this intuition is much harder to reach than many think, which causes the frequent over confidence [0] > It's why the optimal packing of 11 squares is so unsatisfying.
Idk, maybe one of those "beauty is in the eye of the beholder" kinda things?[1]Does your [1] link not simply prove my point that humans will search for something satisfying symmetric given a result that is less satisfying and less symmetric?
"Beauty is in the eye of the beholder" - but let me just quickly find an alternative representation of the solution that is much more symmetric and satisfying. There, "beautiful"!
Yes, but that "satisfying symmetric" (I'd say elegant since symmetry isn't required) is meaningful and finding these elegant solutions has been shown to be important for solving further problems. Math and physics aren't as concerned with the answer as they are with understanding
> I'd be careful with the inference you're making here. Tbh we could easily conclude that neither Newton nor Einstein explained "why". One could argue we still don't know and won't even after we have a TOE.
Why be careful? What would be wrong with that conclusion (if it's true)? I'm comfortable with arriving at ambiguity:
* https://en.wikipedia.org/wiki/Principle_of_sufficient_reason
* https://edwardfeser.blogspot.com/2023/04/what-is-law-of-natu...
I think Feser makes part of the argument for me (though it is clear he isn't a physicist) though I'd encourage you to read Asimov's argument.
There's the metaphysics problem in that all models are wrong. There's more than one competing version of physics (as will help you read Asimov's argument/rant). There's many valid ways to interpret "why" something happens. Physics is trying to understand the causal change of events so that we can generate a model that makes strong and accurate counterfactual predictions. Were that type of model "complete" would we have answered "why" things happen? Depends which "why" you're asking. You could still proclaim, as Fesser notes, "that's the rules God made", which isn't meaningfully different than "that's the way it is".
As for the PSR I'd claim everything does happen for "a reason". But is that reason always knowable? Certainly not. And arguably my namesake proved that there can't ever be any fully complete physics model.
So why be careful? Because my responses are limited, you must ask the right questions. ;)
We all know Terry Tao, do you know Peter Scholz?
Fields medalist and as Tao listed as one of the most important mathematicians of our age.
Scholz is known to find „precisely the correct and cleanest way to incorporate all the previously done work and find an elegant formulation for that — and then, because he found really the correct framework, go way beyond the known results“
In a way that man is the complement of an AI, he generates understanding rather than just proofs.
Maybe he is the right template for the mathematicians of the future.
https://www.quantamagazine.org/peter-scholze-and-the-future-...
Even Peter Scholz is not always sure of correctness of extremely complicated mathematical statements and welcomes computer assistance in verification of his results.
See the Liquid tensor experiment:
https://xenaproject.wordpress.com/2020/12/05/liquid-tensor-e...
https://www.ma.imperial.ac.uk/~buzzard/xena/pdfs/liquid_tens...
" I spent much of 2019 obsessed with the proof of this theorem, almost getting crazy over it. In the end, we were able to get an argument pinned down on paper, but I think nobody else has dared to look at the details of this, and so I still have some small lingering doubts. "
In a sense, mathematics is not very different from software engineering.
More precisely: Curry–Howard correspondence
https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
He is also a member of the association of human mathematicians.
“ 1. Members commit to not working with commercial AI companies by providing technical labor, mathematical knowledge, consultation, or publicity.
2. Members commit to not publishing mathematical texts (including papers, reviews, referee reports, lecture notes, teaching materials, and books) generated by AI models. Members only publish their own writing, as opposed to outputs of AI tools. We affirm that mathematical writing should have human authorship. “
Are they publishing cryptographic proof they never used AI? They should request the AI companies that their IP addresses, etc blacklisted and hard banned from AI services and present some kind of verifiable cryptographic proof they submitted to such a ban.
Or, maybe, they assume their peers within the organization are operating in good faith because they're not drawn from morally bankrupt communities like Silicon Valley.
How would you know? Its easy to also accidentally use AI. They are mathematicians, they know mathematics demands proofs. If they pledge they shall not use AI, they should provide cryptographic proof otherwise the statement is considered untrue.
How is silicon valley morally bankrupt?
Because they're the only community that requests cryptographic proof that their peers are acting in good faith. (:
Just like small numbers appear in almost all places in mathematics. Small numbers are limited. Sometimes these may appear as coincidences. Similarly small elegant explanations are limited in numbers. As mathematical universe expands, only way to explain these additional results will be to have complex explanations.
That is true in theory, in practice the space of proofs that fit in thirty pages or so is so vast that I find it unlikely that we’ll exhaust it for the foreseeable future.
Ramanujan is another one that did much in his head and lacked a lot of formality in his methods. Many intuitive mathematicians are this way (Shinichi Mochizuki is another although his work may just be so hard for others to undertsand that it appears to lack rigor).
I did not; thank you.
One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.
--
To my eternal shame, philosophy (like this OC) makes my head hurt. Pretzels my brain, ouroboros like (aka "head up my own butt").
Caveats covered, here's my cave man cold take on two aspects of this topic.
#1 As we're all trying to paraphrase here: answers are great; understanding is even better. I like the word "legibility" (h/t James Scott). The current models are not very legible.
Which directly leads to:
#2 Understanding enables predictions.
If LLMs are stochastic parrots, then pi.dev (GPT + reasoning + ralph loop) is a drunken sailors walk (thru a solution space). A kind of messy heuristic for optimization. With some kind of gate for declaring "done!" Pretty useful.
However, I don't expect Pi.dev to make predictions. That'd require something else. Maybe that something is being worked on; am noob, so I wouldn't know. (I'm still unclear about whatever Jeff Hawkins is doing, in practical terms.)
--
Wrapping up: Automated prediction would be cool. That's very different from finding solutions to problems. And the current cohort of AI can't even explain why, much less make predictions.
Assuming predictions requires understanding. Maybe there are other strategies to generate predictions. Like a meta drunken sailor's walk, with drunken sailors all the way down.
(Someone please correct me as needed. TIA.)
[Aside: That article has an ad for their podcast They Joy of ₍₎. Perfect. I have no idea if it lives up to its promise.]
> One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.
In David Deutsch's book The Beginning of Infinity, he argues, that the purpose of science is not to merely predict or to model the world, but to explain it. And he gives in my opinion convincing arguments, why a better explanation is needed to advance science. Therefore, he deduces that pure empiricism has failed. Because no matter how much data you have gathered, an explanation does not follow from it. The explanation is always an insight the scientist has to contribute. Then it is open to scrutiny by criticism or experiments.
I guess he would argue the same is true for mathematicians. Merely constructing proofs to arrive at theorem, is of no intrinsic value, if it doesn't spark an idea of a more elegant formalism that corresponds to a simpler and even more far reaching explanation.
I cannot recommend this book highly enough to anyone who is intellectually curious, not only about epistemology.
Many high stakes problems are blocked by a specific insight, and once known, the people who are highly invested in those problems won't put sufficient care into whether we've built a supporting framework of ergonomic theory for humans.
That's not to say there isn't value in building mathematical insight for humans, just that there is a goldmine and a lot of people know what to do with gold, even if they don't understand the ecosystem surrounding the gold.
True, but shortsighted. If we continue to pile proofs and new theorems on top of each other without taking care of making those new abstractions understandable, then we will limit our reach. Even though AI might be able to progress further.
https://en.wikipedia.org/wiki/Peter_Scholze
> Scholze was a student of Rapoport, who was a student of Deligne, who was a student of Grothendieck, who was a student of Schwartz; in this chain, Scholze, Deligne, Grothendieck, and Schwartz are all Fields medallists.
Ahhh, epistemology... Does any academic researcher even care about these things anymore?
If you're in academia, you've probably noticed a major shift recently. Research production at an industrial scale through AI.
Grants have "always" been awarded to professors based on quantity multiplied by quality. Now, some professors are churning out 1–3 papers a month with the same group of co-authors. All the grant money goes to them. Grant givers are just only counting the numbers of papers being published.
The Industrial Revolution has hit academia, and some academics are busy arguing for the artisanal value of research while some are creating factories for research.
Yes, true scientists do care about epistemology.
The problem is the systemic disincentivation of true science, the pursuit of truth and knowledge unencumbered by greed.
When you speak favorably of "factories for research", you adopt the position of a mindless consumer, relishing the prospect of "abundant" products of science.
You fail to notice, livestock lives in abundance as well.
True Scotsman fallacy. No meaningful argument was made above due to taking the stance of "true".
If "false" scientists end up getting all the money, resources, and social status from spamming LLM-generated papers, and the "true" scientists are shunned as slow idealists with 0 value, the first one is more of a scientist in the eyes of society than the latter.
Nobody will speak favourably about the factorization of anything.
The "publish or perish" culture is going through what is essentially the same transformation brought by the invention of the assembly line. Academia, "as an industry", is experiencing what the automobile industry went through a century ago.
Grant approvals are public. Just go through them, and you will see a pattern emerge.
The argument I keep hearing is, "AI is good in the hands of a good researcher."
5–7 papers based on the same dataset? 5–7 papers on the same core topic, published within a six-month period?
They tell me that this is their core specialization and that I simply don't understand how science works.
Science is a self-governed, self-serving, and artificially defined concept.
You conflate a host of different issues and ultimately miss the point.
Science is an ideal, the pursuit for knowledge for its own sake. Science is a boon for humanity because it provides an unparalleled degree of prediction and control of the natural world.
But it doesn't tell you what to do with that.
A "capitalist" will seek personal, short-sighted gain from it, socializing repercussions, no matter how devastating. They will gladly poison society and the planet and sell it to you as "your own fault". Their life-span is limited still, so what do they care?
Technological capability exceeding human wisdom is the core issue here.
You need scientific understanding on your own side in order to know and judge the alternatives beyond what vested interests feel obliged to tell you.
I think the actual truth may be more important than we give it credit for. For instance I've noticed that most propaganda consists of lying about factual information, rather than what to do with the information.
The decision about what to do only becomes relevant if the underlying information is true.
Well, what to do and why it will accomplish what you (and everybody else) want and nothing else is the point. The facts being actually true is a necessity to pull that off.
Think of AI as a magic wand or a genie that makes your wishes come true.
While that sounds great at first, actually thinking it through reverses that mirage completely.
Whom would you want to have one? Who should get to decide? How do you control the whole matter and prevent it from going horribly wrong?
Being irresponsible doesn't work already when you're a grown-up human with only your bare hands.
As you said, "Science is an ideal." I think you and I have fundamentally different views of science. You are an idealist, whereas I am a practical person who has seen how the sausage of science is actually made.
You're talking about academia, society's attempt to gain economic utility from science. You assume, I didn't know about it based on what?
Exploiting science for economic utility alone misses the incurred second order effects, as I posited above. You completely ignore that crucial part, despite of it being the whole (very much practical) point here.
Still there's something to the notion of people who prefer Academia to Industry. As much as I tend to agree with your broader points that it's a game and people love the prestige game and they now can pull levers to accelerate their winning in that Prestige game.. still they chose a path that is less about money and more about ideals than industry. It may be less and less ideal driven and less and less of a cloistered monastery but it's still a place for people who chose against industry. I say that as an industry guy who can just feel when I speak to academics that they are driven by something different than those of us industry. And to be clear those resident industry are not only driven by money we're also driven by building and changing the world but it's still different
A practical person would be off doing something else. You have an axe to grind.
Don't you think your comment applies better to your behavior than GP's? At least they are actually contributing to the conversation.
I've seen a modest increase in productivity in bioinformatics. While the field is computational, it's ultimately about analyzing physical processes. The biggest bottlenecks have always been figuring out what makes sense to do and getting the data you need.
You can of course try new ideas and make software tools much faster now, but software is much like meetings. They can be productive if properly motivated, but they can also be a way to pretend you are working without contributing anything of real value.
Theoretical computer science and mathematics have changed. I'm not sure if they will remain active fields of research in the future. If they will, I would guess that theorems and proofs will eventually become things you include as supplementary data rather than in the main paper. Outside some rare exceptions, people may not consider them publishable results anymore.
Which is why there is this disorientating feeling. We are all facing now with an ongoing challenge to epistemological trust, like Someone suffering a mental disorder.
Factories for research have existed since Edison. The scale is the only difference.
The disdain for human intelligence is dripping off your comment "some academics are busy arguing for the artisanal value of research "
When AI can blast out major results that humans missed for decades, I think some appreciation for the limitations of human intellect is warranted.
I've noticed a cooling down of the human chauvinism arguments since the QRH results.
The article's perspective reminds me of math class when I was a kid. The teacher didn't want the right answer alone, she wanted to see how I got there. In an absolute sense, the right answer is the right answer with or without my understanding, but that's not how the class was graded. We're approaching a point where AI is to science as my old calculator is to elementary school math -- it's capable of leapfrogging to answers faster and more accurately than we can.
Thing is, though, verification is knowledge itself. And if we can get to a point where the important questions have answers, we can surely have the verifiers explain those to us in ways we can understand them.
> we can surely have the verifiers explain those
This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them well - you can bash your way to understanding with them anyway but they aren't good at it)
> This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...
Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.
(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)
> 5.5 is excellent and leaves very little to be desired.
Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.
In a world of short videos, shallow reads, and now quick answers from LLM's it is easy to accept truth without verifying before moving on to the next thing. I was running into similar cases where the confidence was there but the answer was wrong, and as the article suggests, math separates correctness and understanding, so I thought it would be a great area to test where hallucinations still exist in the models. I made a test on large prime numbers where the LLM's couldn't know the answer and we can easily check the factors it produces to verify if it faked it. Nearly 75% of GPT-6 Luna were faked with a false factor when given a hard number with no tools.
You point at the exact problem here: people are so used to arguing from ignorance, they don't even notice.
Verification isn't knowledge. Far from it. You can verify statements without knowing why they're true. Variants of that concept are e.g. used in cryptology.
You need humans to truly understand (to know the context) in order to make correct judgements. Without actually knowing the context, you have no way to compare.
An infallible oracle tells Archimedes that the speed of light in vacuum is ~1.08 billion kilometers per hour. Archimedes, with the iron age tools and methods at his disposal, might be able to verify this and make some use of it himself. Maybe. In any case, what he has gained is knowledge itself.
The same goes for our own time. A powerful AI can tell us things at the edge of what we can grasp -- or beyond what we can grasp -- and it would nevertheless be knowledge that we possess. We can work to verify it, we can have it explained to us as though we were schoolchildren, or we can simply accept it as a static fact of the natural world.
In truth, we're going to have to learn to coexist with minds that are more powerful than ours. At least in some respects, if not in all respects. The good scenario is that they share information with us, and we receive it however we can. "Head in the sand" and "mutual disdain" are bad outcomes.
No, you're wrong.
You might want to look up the actual definition of knowledge. It's a pretty interesting topic in epistemology really, far from your simplistic take.
You already live in coexistence with minds more powerful than yours. Society works in a way that you never need to notice that, those minds taking care of things beyond your grasp for you.
The downside of that "convenience" is you developing an over-simplified bubble-view of the world, rendering you prone to arbitrarily hazardous misjudgements.
Here, you align foolishly with interests that cannot possibly be your own. Namely, exploiting scientific knowledge for personalized monetary gain on an unsustainable scale.
When a tiny group of people hogs essentially all scientific knowledge, that renders you what?
Your idea of being graciously provided with some "basic income" or whatever is blind copium. You would live as livestock at best, to be discarded as useless whenever your masters see fit.
I don't think the social problems you listed can be solved by redefining knowledge to your convenience - they can play that game better than you can. The definition of knowledge in this context is simply that which describes reality. And the person you're replying to was correct in saying that the thing gained from the oracle was knowledge itself, even though they've not considered the epistemological nuances.
It's not me who's "redefining knowledge to my convenience". You weirdly succumb to "them" doing that and come to wrong conclusions.
Merely claimed "descriptions of reality" lack the crucial property of truth. The description has to actually fit reality, or it's mere fantasy. It has to be exhaustive, or it remains a facade, leaving you blind to what lies beneath.
When you don't actually know whether it does any of that, you're merely a believer, entirely dependent on your "oracle" and its slave in effect.
Not caring about nuance leaves you open to exploitation, among other things.