GPT-5.6
openai.com1029 points by logickkk1 8 hours ago
1029 points by logickkk1 8 hours ago
https://deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf
https://developers.openai.com/api/docs/guides/latest-model
https://x.com/levie/status/2075287443411222628, https://xcancel.com/levie/status/2075287443411222628
The developer's guide (https://developers.openai.com/api/docs/guides/latest-model) has some interesting semantic tips for using the model: > Intent understanding: GPT-5.6 can better infer the user’s underlying goal and intended level of work without you specifying every step. Continue to state important constraints, approval boundaries, and success criteria explicitly. > Original image detail: GPT-5.6 preserves the original dimensions of images sent with original or auto detail instead of resizing them to a patch budget or pixel-dimension limit. > Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%. > Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.” > Control warmth: GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic. > Avoid generic brevity instructions That part is confusing because it's not like they provide an example of how default GPT-5.6 output compares with GPT-5.5 both with default output and prompted for brevity. Whenever I use such prompts, it's usually because I want the model to give me the gist in a few sentences. I'd be stunned if GPT-5.6 was that concise by default. I would think that could "break" a lot of things for developers who didn't know to make prompt changes after upgrading to 5.6. What if you were expecting GPT to be as wordy as it usually is? Then suddenly your output is not wordy enough? Smells like OpenAI trying its best to stave off financial armageddon for another few months. Then again, I'm not sure why they chose to waste so much output computation on verbal diarrhea all this time up to now. If you conceptualize this as “there is an appropriate amount of brevity for each situation” then it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount. My view is that popular models by default output wildly excessive amounts of prose for nearly every use case, so if this changes in a new model that’s a pure win. > wildly excessive amounts of prose Not just prose. I think this is part of the reason why you see ridiculous code with insane error handling and type checking even for impossible cases. The models don't get better, except when a new one is released. Their performance depends solely on the model training before release and how well you curate the context you feed it. That's it. Contrary to popular belief these things are not intelligent. >The models don't get better, except when a new one is released. Their performance depends solely on the model training before release and how well you curate the context you feed it. That's it. Not quite. The hosting side can change reasoning budgets (or re-assign what terms like "high" means), temperature and other decoding parameters, output length limits, finetune internal "hidden" prompt, latency optimizations, finetune attention algorithms, even change quantization - all still serving as the same model. We know (or suspect) Anthropic frequently nerfs models while keeping their name and version the same. >The models don't get better, except when a new one is released. My brother in Christ this entire thread is talking about the new model that was released It was edited. Original talked about the model learning. Glad they managed to clarify. Because the models are quite literally stupid. It seems like the way brevity instructions have changed is mis-aligned with how most people would expect to use them or are currently using them. Here's the example they give: > Instead of asking for the shortest possible answer, replace brevity instructions with prioritization: > Lead with the conclusion. Include the evidence needed to support it, any material
caveat, and the next action. Omit secondary detail and repetition. > Keep all required facts, decisions, caveats, and next steps. Trim introductions,
repetition, generic reassurance, and optional background first. Generally speaking, when I ask for a short answer, I want a short answer because I'm not really willing to read through a bunch of bullshit to get to a summary. Putting the onus back on me to assume what the model will return and write a longer prompt detailing exactly what information I want completely misses the point of why I'm asking for a short answer in the first place. > Lead with conclusion. I would presume (perhaps falsely?) that an instruction like this would lead to the model presenting a conclusion not supported by the evidence, and potentially backtracking as it then tries to justify said conclusion. Yes, if deliberation happens, the model should figure out what it wants to say during that phase; but if you're using auto mode, the model is not going to be doing any deliberating half the time. In those cases, the output blathering is the model's only chance for deliberation. It "thinks as it talks", per se. Given that, I would advise a different approach: let it blather, but then get it to write you a conclusion at the end that the model can guarantee will obviate the need to read any of the blathering. I.e. advise the model to add an "executive summary" to the end of any non-trivial-in-length response. With some wording to carefully navigate the model between "the summary is itself too long" vs "the summary acts more like clickbait, leaving out necessary detail such that it requires actually reading the blather." Not sure exactly what that wording would look like. I imagine something like "write your postscript executive summary as if you were a senior CIA intelligence analyst summarizing ground-level reports into a daily digest for the Joint Chiefs of Staff. Take up as little of their time as possible, but ensure that any detail critical to decision-making is retained." (But that phrasing might only be useful if the model is delivering a certain type of response, and actively counter-productive otherwise. This kind of thing is delicate.) Over hours of experimentation with various LLMs, I've found virtually any system prompt can cause unintended skewing of the model's output. Even just 5 to 8 short, direct words about length, tone or formatting can cause subtle yet significant changes in model output. Longer, more detailed or conditional prompts always introduce an additional cognitive load as it checks every token it generates against the conditions. Making instructions more absolute (like: "Never do...") can increase the duration of compliance but at the cost of creating a significant center of attentional gravity. This can cause far more output distortion as the model devotes increasing portions of its attention budget to ensure compliance with a heavyweight requirement or prohibition. Every word in a global prompt is a trade-off between attention, compliance, drift, etc. As someone used to thinking of computers as natural deterministic rule-followers, it's weird having to carefully wordsmith and A/B test even the simplest global prompts. It feels like coaxing a hyper-literal, emotionally sensitive, spectrum-ish toddler to comply but without being so strict it gets 'upset' or spirals into hyper-focusing. This was a big concern for earlier models, but with modern CoT trained models they should be able to come to the conclusion entirely in the thinking trace. You are absolutely correct. The second suboptimal part of the prompt is this: > Trim introductions, repetition, generic reassurance, and optional background first. It's not possible for the model to "trim" those before they've been output, so this is akin to telling it "not think of an elephant or even take the existence of elephants into consideration while solving this problem". You may be discounting the tokens generated in the thinking trace but not included in the output to users. That would be true of non-iterative models that just emit an output from beginning to end. No reasonable model has worked that way for years. Oh the number of time LLM will, for example, be giving me the list of bugs it found in code, when I ask it for a review, just to decide there’s no big half way through explaining it. Yes this is an extremely well known result for exactly the reason you guessed. It's not just abcktracking, asking an LLM to present a conclusion and then justify is also an excellent way to provoke hallucination as the model con concts "any justification that plausibly justifies the words it's already said". This is the actual reason why openai _invented_ reasoning models, to give them time/space to work out a solution, rather than having to magic a correct solution out of thin air from token 1. It's less important now that all models do reasoning, but it's still almost always better to make the output come out last rather than first. I wonder if it would help to ask it to write a rough draft and then reorder it? It would (and does), yes; but this takes a lot more output tokens than asking for a summary would. The summary approach is only helpful insofar as it can be cheaper than using the thinking model. (You're basically tricking the instant model into thinking, which it can do, after a fashion.) But, unless your desired output is literally a document for others to read, at the point where you're having a model generate a full, lengthy output multiple times over with revisions, you may as well just turn off auto mode and have it always deliberate (i.e. choose the thinking model explicitly from the model selector.) Then it'll be as messy as it needs to be while deliberating, but give you exactly what you want as output. (And if your desired output is literally a document for others to read, that you want to interactively draft and polish, then (in the case of ChatGPT specifically) you should not only be explicitly forcing the "thinking" model, but also should be asking it to activate the "canvas" feature from the start. My understanding is that revising a canvas document involves the model emitting something like editing gestures, rather than simply re-streaming the updated chunks of text. This saves a lot of output tokens on large documents.) Why would auto mode turn off thinking? The "auto" mode is (AFAICT) a per-conversation-turn router. (Presumably via a preliminary pass through a very fast tiny model that spits out an number for how challenging it thinks the next response might be to compute.) On high-challenge turns, the auto mode routes to the "thinking" model. But on low-challenge turns, it routes to the "instant" model. And the "instant" model, by design, has no capacity for deliberation. (If it did, it couldn't guarantee that its responses would begin streaming "instantly.") I don't expect that would be the case. This is what's called BLUF or Bottom Line Up Front: https://en.wikipedia.org/wiki/BLUF_(communication) The model will still have read the entirety of the document before composing its response. And I believe that even in auto mode, there are thinking tokens behind the scenes. Replace 2 word instruction ('be concise') with a 38 word instruction. Human can no longer be concise when asking for a few sentences instead of 20 paragraphs of BS they don't want to read when all they want is a summary to verify the general direction of the prompt-work before digging into the details. such progress! I don't know how intentional it is / was, but LLMs in general just love to hear themselves talk! They do, and I want to encourage them to do so because they think through talking. What I don’t want to do is spend time reading all that. We will probably just get reader-side affordances for this like auto-folded justification and introduction sections and so on. Doubtless some chat interface will add this the way they’ve added reasoning folding. Thinking models think through talking, don't reveal that talking, then answer by again thinking through talking. It's kinda funny in a way. > LLMs in general just love to hear themselves talk! Because that’s what’s in the training set. Reticent humans don’t have blogs. Interesting idea. I think they're getting more wordy over time, personally, so I think it's more to do with the training than the raw data. Is it just a coincidence that the companies creating them charge by the token? The aligned incentive appears to be realigning in favor of the corporation. Pray they do not realign them further. There are times I require single word answers. I will use whatever model responds as I desire and at this point those models are just a few. The cost-per-task benchmarks align incentives toward more efficient output and those are the ones gaining steam. I think instead of "be concise" you could tell it how long the answer should be. I.e. give the answer in one paragraph. Or in 10 lines max. At least before it would listen to instructions like this. > At least before it would listen to instructions like this. Would it actually follow them? IME LLMs are incapable of estimating the length of their own output, the total length of the current context, etc. They just make stuff up unless they have external tools that can inspect those things for them. That was the case for early models (Llama etc), but they got much better since then. Not perfect, but good enough. This is from Ministral 3 14B, a 2025 model without reasoning, that you can run on your PC: > Write a Haiku involving HackerNews, and the capability of large language models like you to reply in an exact number of words or syllables. this is a dependency update. shouldnt you have good testing for that and not deploy a version update when those tests fail? > could "break" a lot of things for developers who didn't know to make prompt changes after upgrading to 5.6. How does this differ from the other changes in behavior in 5.6 that will also break things? New models always break things. It sure is suspicious that both Anthropic (adaptive thinking) and OpenAI (Avoid generic brevity instructions) both seem to be suggesting that the best way to improve outcomes is to entirely leave it to them to decide how many tokens get used. I mean, it's true that it would be ideal of this stuff did just get figured out optimally behind the API, but there is definitely an incentive on their side to burn more tokens. Perhaps the incentive is for variable behavior. When there is low GPU demand, burn more, but reduce when there is contention. > ...tips for using the model: > Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.” I don't follow. Isn't "the model actually cares and will do what you say" a reason to use those kinds of instructions more liberally? Click through to the link - it states that the model tends to over correct on brevity instructions by omitting required information I think they’re saying it’s irrelevant now, possibly because it’s less likely to trail off on meandering thought bubbles. Does anyone else feel each model is like watching your kids grow up. They we're bubbly and fun and weird, you needed to tell them to sit down and be quiet. Now if you tell them too much they go mute or stop telling you important information. Oh intelligence! I'm impressed. It feels like a faster Fable (probably due to the more efficient token usage). It performs roughly the same job, just with 4x less steps (gamedev). Remains to be seen how the "shorter prompts" advice translates to homogeneity/collapse though. I wonder if it will do any better than past versions when one begs and pleads for it to get a job done using a concise, modest amount of code (as an expert human developer might), rather than responding to all prompts by shoveling in a large amount of code. Serious question: what is a short prompt? (For that matter at what point is it "long"? And does the rest of the context matter? Should it be short too?) It creates the context of the request without including language or terms that activate additional areas of knowledge not necessary for an accurate reply. > Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.” What about my favorite, "no yapping"? It might need the longer answer to think about the question, so one approach would be to ask it normally and then ask it to repeat itself shorter. > Intent understanding This will totally make it brain damaged over a certain tasks. Sort of like the same brain damage that prompted OpenAI project managers to destroy ChatGPT.app today. Can you elaborate? It had a tough time updating today. Or this evening. It just wouldn't update. It actually just freaking disappeared from my MacBook. It took some googling and downloading and multiple tries to get it back and working. Because they also combine on a MacBook Codex with ChatGPT app. I guess codex became ChatGPT app or some silliness like that. > destroy ChatGPT.app today. ... What changed, exactly? Codex.app is gone and merged into ChatGPT.app. The upgrade process was... messy... Codex's self-update just deleted the Codex.app w/o further instruction. And ChatGPT updater failed the first time while also bricking the prior installed ChatGPT.app. Seems good/fine once you get through upgrading the app. > can better infer the user’s underlying goal and intended level of work This is a trap. It's the optimistic fallacy that poisons all "consumer scale" machine learning products and what's going to effectively ruin these models as they keep chasing it in the same way that web queries were ruined, social media feeds were ruined, and media recommenders were ruined. For the vendor, optimizing metrics across their whole user base, they always see positive technological progress as their system gets better at making assumptions and accumulating user engagement scores in aggregate. But for the individual user, most of which has some weird tail intent/interest and some of whom have many weird tail intent/interests, the experience quietly but catastrophically degrades. Output/results become more generic, more divergent with the underspecified "weird tail" intent, and more stubbornly hard to ever wrangle towards that "weird tail" altogether. We've been watching this cycle happen for 20 years now and it's proving hard for anybody to escape because it works so well for the trillion dollar company driving it forward. But while each step might feel ergonomic and welcome to individual users, there's a frog boiling enshitification at play. In pursuit of output quality and capability (rather than simply the vendor's user count), what we need rather than "makes better guesses" is "presses for more clarity", even where it feels kind of annoying. Even among human professionals, one of the first hurdles of breaking out of junior tier work is gaining the confidence to press your colleagues and clients to be more specific in their thoughts and expressions despite their desire to have you do it all for them. But they're often coming to you with incomplete, muddy, and conflicting ideas for which there is no safe and correct assumption that you might just run with, and it's your expertise (i.e. relevant "intelligence") that's critical to bringing attention to that. To achieve professional progression, you need to learn to do that and to not just optimize appeasing the ambiguous client/colleague today in exchange for mutual expense tomorrow. To avoid enshitification, which is probably not possible, we need these models to be learning that too. I agree to an extent but it needs to be balanced. Receiving a half-baked, extremely verbose recap of thinking on benign details with Opus 4.8 or GPT 5.5 feels like an extraordinary loss of quality of experience compared with fable 5. Yes it shares less, but I think the trade-off is you pay less in tokens and hopefully it's truly just not needing to say things because it truly does just better get what you're saying, think to read X markdown file or GH issue which contains the info, etc. As long as I can still push back and get it to share its thinking on demand and I'm confident the model isn't actually basing things on poor premises, this is okay for me. I am more productive when not inundated with time-wasting check-ins. That said, I absolutely lament the loss of the ability to access the thinking - I would happily read the "DANGER DANGER DANGER" internal gremlin thoughts fable 5 makes to verify something if they were accessed, and prefer that to a recap presented only for my benefit. Same, I think you both have great points. Idk how you can debug effectively (the model itself) without reasoning traces Control warmth[1] > GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic. Instead of generic instructions such as “Be friendly and warm,” use concrete guidance:
> Be direct and tactful. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs. Soo basically, my new 5.6 custom instructions: Be Jeeves and eliminate all friction from my life through immense processing power. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs. [1] https://developers.openai.com/api/docs/guides/latest-model#c... > Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.” I used to go to a barber and if you said "cut it short", he cut it really short. > Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%. A shorter prompt results in half as much tokens spend? I find this very hard to believe. If it's anywhere close to the same universe as smaller models in its behavior, a lot of time in "thinking" mode is spent on reiterating on any constraints given in a prompt. So the more constraints you give it, the more tokens it will spend going "Hold on, the prompt said I have to dot my i's and cross my t's. Let me go through my work to check that all the i's are dotted." > A shorter prompt results in half as much tokens spend? I find this very hard to believe. Should be relatively easy to test. And if it's true, just first use a very cheap near-SOTA model to first rewrite the prompt to a similar but shorter prompt before sending it to GPT-5.6. pi.dev for example can control other harnesses. An example: the other day for example I didn't understand why Claude Code CLI (which I hadn't used in a while) wouldn't let me cut/paste anymore (turns out they apparently fixed some long-standing scrolling and blinking SNAFU, but this modified how mouse selection/paste worked under Xorg but I didn't immediately realized they changed this)... I had to copy/paste the oauth challenge/response for I was logged out (maybe because I hadn't used Claude Code CLI in a while, dunno). But my usual copy/paste wasn't working and I didn't know how to fix it at first. And because I wasn't logged in, I couldn't use Claude Code itself for this. My prompt was something like: "Screenshot the Claude Code TUI, transform the URL into a link, open that link in a broswer to get the oauth token, copy it character by character by simulating keypresses in the Claude Code CLI". (remember: I had no idea how to paste with the mouse not with the keyboard, no I know but I was pissed off and wanted to be logged in immediately... So: another model / harness to the rescue). (for the curious: it decided to use xdotool and use a 50 ms wait between simulated keypresses to copy the oauth token) This worked just fine. And I that with a cheap model. I think that just like Linux and Git owned many proprietary software, we'll soon have fully open-source harnesses orchestrating everything and delegating the work to proprietary tools (like "ChatGPT now Codex and vice-versa" and Claude Code)... If proprietary tools are even still needed at all. Honestly I begin to wonder if they're even needed at all: the models, sure, while waiting for the open-weight ones to beat them. But those proprietary tools trying to lock people in? I feel like the open source harnesses are already more powerful. Maybe Codex has the same problem I sometimes have focusing while reading and has to reread the same sentence over and over again. > Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.” RIP Caveman skill. Six month good. Now skill dead. A Yoda skill, is there? > Yoda skill, there is? ftfy Is there a Yoda skill? -> A Yoda skill, is there? There is a Yoda skill. -> A Yoda skill, there is. > Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%. When has this ever not been the case? I don't think this is a GPT 5.6 specialty! Information density of the prompt is the most important factor in my experience. And interestingly, LLMs seem particularly bad at writing prompts for other LLMs for this reason (you can guide them to be more dense, just speaking by default). Conciseness is usually a byproduct of information density though. > LLMs seem particularly bad at writing prompts for other LLMs for this reason Claude is terrible at this! Probably for the same reason that its writing style in prose is so annoying and full of claudisms. Claude use to be leader, too. Their metaprompt was great at the time with opus 3 Lexical-priming->semantic-space-constraint;specialized-lexis+=sharp distributional-signature;∴ tight concept-cluster; generic-lexis->diffuse-activation, broad candidate-set;Attention-heads key/query-match domain-tokens;"Hamiltonian"->{operator,eigenstate,quantum,energy}->register+domain locked;Net:constrained-decoding,vocab=soft-prior over output-distribution; register-matching;#taskdef=decompress->continue Information density of the interpretability of the intent from the perspective of a human (or human-like). If the intent is not easy to understand, it's information sparse. Because it takes a lot of CPU (or brainpower) to interpret. You can run gzip on an English sentence to make it more textually dense, but clearly it is not more information dense in this context. Chatbot expanded this into something that made sense, but I've no idea if it's what you meant. There's an irony there somewhere. There was a fad a while back of building insanely long prompts - tens of thousands of tokens - including having models write prompts for themselves. I always thought it was counterproductive, especially if you're going to use the prompt more than a couple of times. (That said, the e.g. Claude Code system prompt is insanely long, so if you genuinely have a lot of information to provide maybe it's beneficial. Like, shorter is better, but you don't want to be under-specified.) For Gemini 2.5 and ~GPT5.0-5.1, longer prompts with lots of explicit instructions and examples produced better conformance. Seems like heavily second guessing the models started to get counter productive around the end of last year. GPT-5.6 Sol sets a new SOTA on ARC-AGI-3: 7.8% Sol is the first verified frontier model to ever beat an ARC-AGI-3 game Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning.
We are probably going to need a lot more GPUs. These aren’t raw base models they are the result of a ton of RLHF and various adjustments. Bitter lesson wildly overstated in this context. More RLHF is in fact scaling. Yes, but not in the “dump another chunk of all written language in the bucket and stir”-sense which is what bitter lesson became synonymous with. That may not be the intent of the original article, but over the past few years that’s what the phrase turned into. GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong. Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date. The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL. While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient. I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis) There goes my plan to buy a PC for the next decade The whole of knowledge work is being automated. We've barely begun to see the GPU build out. This is just the start. I'd imagine they're going to 10x this, maybe 100x this. Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context. It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse. Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads). I mean, theoretically you can solve every finitary problem with a brute force solution... Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far > Dramatic difference Isn't this just the difference between getting 0 right and getting 1 right? This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective. > We are probably going to need a lot more GPUs. Or a breakthrough in algorithms etc. The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence. The human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat. The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges. I think you're helping GPs point: there is a lot of efficiency gains to be made to match the processing power of the brain, given it's size and power draw. Moreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2]. [1] https://www.sciencedirect.com/science/article/pii/S193459091...
[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/ The real question is not how many "weights" the human brain has (neurons+synapses may or may not translate into "weights", and brain might be also inefficient for what it is), but rather how much evolutionary and social "compute" was necessary to pack everything into that capacity. For intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes. This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"? Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9... It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has. I'm surprised it is that low. Are not all top AI labs "cheating" and workaround LLMs's low sample efficiency by hiring people to generate more data points - similar problems with answers, so they can train models on those and improve scores? A good benchmark for general intelligence probably should be a complete black box, no sample data given/leaked at all. Very interesting. My prediction is that Mythos would outperform Sol. Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs. At some point his claim should be fully falsified no? Mythos probably wouldn't, otherwise they'd have included it in their release. Next version of Mythos probably will though. And yeah.. Reality has not been kind to LeCun. Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meaningful embeddings to the point where image segmentation is just PCA. The spend on DINO cannot be more than five million (correct me if I'm wrong) JEPA is just getting started His main anti-LLM predictions have been consistently either wrong or misleading. There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point. His main LLM predictions have almost nothing to do with Arc AGI... What exactly was he dead wrong about that is proven by any of this? GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying. He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3. He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value. Spending $5B to train a model that scores better than an older model does not disprove any of that in any way. >He's merely said they don't think He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example. > He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000. What things specifically and when? https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
minimaxir - 8 hours ago
ravenstine - 8 hours ago
tshaddox - 5 hours ago
ComputerGuru - 6 minutes ago
daveguy - 3 hours ago
coldtea - an hour ago
Cycl0ps - 2 hours ago
daveguy - 2 hours ago
anticorporate - 7 hours ago
derefr - 7 hours ago
mrandish - an hour ago
mattnewton - 5 hours ago
sebastiennight - 3 hours ago
SgtBastard - 2 hours ago
brookst - 39 minutes ago
prymitive - 6 hours ago
Farmadupe - 5 hours ago
skybrian - 5 hours ago
derefr - 5 hours ago
cma - 6 hours ago
derefr - 6 hours ago
radlad - 6 hours ago
spathi_fwiffo - 7 hours ago
osigurdson - 7 hours ago
arjie - 7 hours ago
bcrosby95 - 7 hours ago
mr_toad - 5 hours ago
flir - 5 hours ago
jimbokun - 7 hours ago
pizzafeelsright - 7 hours ago
minimaxir - 7 hours ago
Romario77 - 7 hours ago
isityettime - 6 hours ago
PhilippGille - 2 hours ago
Across multiple tries it got it wrong a couple times (by ~2 syllables). But syllables are extra tricky (because of how LLMs use tokens) and the point is that for things like "summarize in 5 bullet points" you will mostly get 5 bullet points, maybe 6, but not 10 or 20, and no need for a tool that count bullet points. Silicon whispers,
exact words in code’s embrace—
Haiku blooms anew.
8note - 28 minutes ago
wrs - 3 hours ago
clhodapp - 5 hours ago
newshackr - 2 hours ago
zahlman - 3 hours ago
rrvsh - 2 hours ago
greggsy - 3 hours ago
kinnth - 2 hours ago
avaer - 6 hours ago
kylecordes - 3 hours ago
__mharrison__ - 4 hours ago
bsenftner - 4 hours ago
mlmonkey - 7 hours ago
astrange - 5 hours ago
egorfine - 6 hours ago
oezi - 5 hours ago
vardalab - an hour ago
zahlman - 3 hours ago
apinstein - 2 hours ago
swatcoder - 6 hours ago
loufe - 4 hours ago
mdgld - an hour ago
artisin - 7 hours ago
stillpointlab - 6 hours ago
elAhmo - 7 hours ago
bulder - 7 hours ago
TacticalCoder - 4 hours ago
zeven7 - 7 hours ago
postalcoder - 8 hours ago
delichon - 7 hours ago
froh - 5 hours ago
FreakLegion - 2 hours ago
epihelix - 7 hours ago
adam_arthur - 7 hours ago
astrange - 5 hours ago
irthomasthomas - 2 hours ago
docjay - 5 hours ago
adam_arthur - 4 hours ago
flir - 4 hours ago
daemonologist - 7 hours ago
CuriouslyC - 6 hours ago
meetpateltech - 7 hours ago
10xDev - 7 hours ago
bevekspldnw - 6 hours ago
fastball - 5 hours ago
bevekspldnw - 4 hours ago
avarun - 3 hours ago
linkregister - an hour ago
cma - 5 hours ago
altcognito - 7 hours ago
navigate8310 - 5 hours ago
echelon - an hour ago
XCSme - 3 hours ago
Salgat - 6 hours ago
vatsachak - 7 hours ago
energy123 - 5 hours ago
emp17344 - 6 hours ago
Razengan - 7 hours ago
ryandvm - 6 hours ago
thunky - an hour ago
dbspin - 5 hours ago
orbital-decay - 4 hours ago
aeyes - 6 hours ago
monk_grilla - 2 hours ago
maxnevermind - 3 hours ago
simianwords - 7 hours ago
osti - 7 hours ago
vatsachak - 7 hours ago
Tenoke - 7 hours ago
onlyrealcuzzo - 6 hours ago
Tenoke - 6 hours ago
onlyrealcuzzo - 6 hours ago
Tenoke - 6 hours ago