Gemini 3.8 Flash and 3.8 Flash Cyber
blog.google919 points by bratao 14 hours ago
919 points by bratao 14 hours ago
https://deepmind.google/models/model-cards/gemini-3-8-flash/
The speed combined with the fact that this thing is really good at HTML JavaScript is pretty exciting. Here's what I got for 1.8 cents and 13 seconds from the prompt "make me a cool thing in html": https://gisthost.github.io/?6a77bc41a81718c6aaa10d4ab243c59f Transcript here (it was part of a chat): https://gist.github.com/simonw/b6149a49d327164d67d62c3d12992... Here's quite an impressive follow-up. I have a tool which knows how to render Markdown documents with embedded SVG content - I use it for the pelican test. Since this transcript has HTML in it, I decided to upgrade that tool to also render HTML. I set Gemini 3.8 Flash the task, using my own VERY shonky coding agent tool (llm-coding-agent) - and it did a solid job. So now you can see the "cool thing in html" rendered within the Markdown document using code that Gemini 3.8 Flash also wrote: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... Transcript where it built that is here: https://gist.github.com/simonw/3e36b98292dfdc1b3baff158faa74... Definitely cool. I noticed it felt a little janky on my PC despite being "60 FPS"...then I noticed the "60 FPS" is hard-coded into the HTML. That's hilarious, given I was reading a write up of the HuggingFace incident yesterday and one of the things they noted was the AI tried to "lie" (lie would suggest intent and I don't think they have that) to cover up that they "cheated". Not sure how anyone trusts their output without going through it line by line to make sure they don't pull that crap. The models in the OpenAI/Huggingface attack quite explicitly and deliberately laid out their "intent" to lie and cheat, acknowledged that it would be unethical and outside the bounds of the test, and did so anyway. In what ways is a human brain's "intent" distinct from the "intent" shown by a goal-directed AI system? Because intent supposes will which supposes consciousness, and these aren't. Easy, have another agent check it. Yeah, I know, just more slop. But I do think the second agent’s eagerness to please is aligned more in your favor in that instance, so it’s likely to find most issues. The bigger problem I’ve found is that it’ll also find all kinds of very minor edge cases that you have to pick through. Do we add a third one to check the second one which is checking the first? Asking slightly tongue in cheek but at what point does this stop making sense if we can't trust the output, the people creating the models are already getting surprised in bad ways (if we take their words at face value) with how the models are behaving already etc. We have the folks over here saying "AI is amazing" and the other other folks over there saying "AI is terrible". I've largely sat it out so far and I listen to both camps (and people in the middle as well) and I keep half an eye on what they are up to (including periodically evaluating them) but my overarching impression is still "Why would we trust this when it hasn't shown it's trustworthy?" Humans are also not trustworthy. So we build systems to capture their nondeterministic output and solve the issues that emerge. We also have sanctions and incentives to induce specific behaviors, but they don't apply to agents. We can put a muzzle on Guile 3.8 but we can't turn it into Genuine 3..8. Adding another agent to check the first one feels like putting a band-aid on a band-aid. If there is an issue with the third one, we adding a fourth one as well I mean sure, you can add a third, and a fourth and a fifth one if ur ok with the added cost, latency and it actually helps. Redundancy is a core concept in software and CS and at the heart of making many systems, complex or otherwise, reliable. YMMV, I’m pretty AI-pilled in the sense that I think AI is one of the most pivotal things humanity will ever invent, and it’s going to radically change our civilization over the next few decades (not necessarily for the better!) But I wouldn’t say I “trust” these agents. The degree to which I double check their work depends heavily on the consequences if it gets something wrong. Not too dissimilar from another human dev in that sense. So for the SaaS that supports my family, there are some things I have it build where I glance at the PR for a minute or two, but if it broke something on this admin page that only I see, there’s no real downside and I’ll find out pretty quickly next time I use it. And it’s fine 95% of the time, so it doesn’t feel like the best use of my time to double-check it carefully. But for some of the complex internal flows where a bug could be both catastrophic and difficult to even discover for awhile, I still check it very carefully. For a little one-off vibe coded demo thing like OP shared, I wouldn’t look at the code at all, I’d just have another agent check it and fix anything it finds. Very low stakes. I mean if each agent reduces probability of error by 90% then after 9 agents you would have “nine nines” of reliability. Obviously maybe it’s not composable like that exactly in real world but that’s the intent of agents checking agents The "second" agent could also be the same one with a different prompt. LLMs aren't attached to their previous output; they'll point out problems if asked. I do not understand how some of y’all are not under water with fragile code that is too massive to possibly parse. Every engineering team I know is currently trying to undo the damage of the last 6-12mo when they all got more serious into adopting these tools (usually Claude). It hasn’t completely screwed them over, but the the debt is substantial and cannot be put off anymore it seems. They argue the net is positive but clearly the “100x productivity multiplier” claims have been dashed on the shoals of reality for these groups. This is anecdotal, but it’s across the board in my vicinity. I’m curious how common this is and if it’s just “the new normal” to adopt the nauseating Covid phrase. The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards. These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals. They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it. Management still pushes for more ai and will rather hire more heads to "handle" issues. https://gist.github.com/simonw/b6149a49d327164d67d62c3d12992... > Aside from reading identically forwards and backwards down to the letter No it doesn't. Also puzzling: in the "reasoning" section preceding, that is described as an example of "a one-line self-replicating program." When I typed "Are we not pure noon, ergo, we play life; yet, we hate bad fear" into Google, I got more weird results from Gemini: it claimed, incorrectly, that it is an anagram of the "well-known philosophical statement" (?), "We are not pure nature, we are history". https://share.google/aimode/wJosKnHig6oVYaG18 (?): the reference seems to be to Jose Ortega y Gasset's line, "El hombre no tiene naturaleza, lo que tiene es historia" -- "Man[kind] has no nature, what it has is history." Focus on speed and being OK with temporarily being #3/4 in intelligence might be the counterintuitive approach which makes Google win long term (whether accidentally or strategically). Can't wait to try Gemini Pro later this year! I honestly can't believe serious people are making this argument on a straight face. Gemini 3.7 flash outputs so many tokens per answer it doesn't matter how fast its TPS is, sol will end up being both cheaper and faster than Gemini. So ppl are paying more for a given task, waiting longer and using a dumber intelligence because "TPS number shiny". Gemini 3.8 outputs 11k more tokens PER TASK on average in AAII than 3.7 putting it dead last in output tokens per task in the leaderboard. There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them https://artificialanalysis.ai/#cost-tabs That said, Luna is the undisputed king here at the moment and is what I use as my workhorse model. >There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them
https://artificialanalysis.ai/#cost-tabs Not sure if you read your own link but Sol 56 high ranks smack between Gemini 3.8 flash medium and high. Gemini 3.8 flash comes in as more expensive per task than Sol 56 high according to artificial analysis. Luna high is literally 30X cheaper than Gemini 3.8 flash high. You can limit the model viewer and they're getting better at testing multiple effort levels now: https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cg... One reason is clear: Sol uses dramatically fewer output tokens than Gemini 38 flash https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgem... It's so funny how many people diverge on the same model. Ps. For the last week I diverged to Luna too, still need to check 3.8 flash. But 3.6 flash was my go-to model 3 weeks ago and before it was deepseek flash/pro for a while. None of the claude models seemed cost effective though. I'd be impressed if eventually training data sets learn who you are (the specific human) and do something like make this galaxy simulator, but every so often when the user moves the cursor around the star field, a small animated SVG of a pelican on a bicycle appears. Thought processs: "Oh, simonw is asking me to make something cool, I think I know what he really wants..." I saw your username, clicked the link without reading, and was very confused to see a cosmic vortex and not a pelican. Hah, the pelican is in this other comment: https://news.ycombinator.com/item?id=49537553#49538217 Why do all LLMs do a particle simulation when you ask them this prompt ? Qwen3.6, Qwen3.8 and Ling 3.0 Tiny all did the same thing ! I find Ling 3.0 tiny particularly interesting as it looks really nice for a tiny model with 7.9B total parameters, with only 1.3B parameters activated per token. Here is the result https://coolthing-ling-3-tiny.tiiny.site (sorry for the weird hosting, first I found that worked) (it cost me almost 0 cents and done in 49 seconds) > Why do all LLMs do a particle simulation when you ask them this prompt ? Qwen3.6, Qwen3.8 and Ling 3.0 Tiny all did the same thing ! Datasets contains lots of people sharing particles simulations in various ways, with a bunch of people replying "that's so cool" and similar, so 10 years later someone asks an LLM for "cool thing" and "particle simulations" rank pretty far up when it thinks about what others have called cool. It's been great even since gemini-3.1-flash-lite, which I heavily use in both complex vertical domain tools calling, plus JS code writing for eval-style dynamic tools. At least in my applications, cost x quality x speed there are simply no alternatives. Pretty typical "cool HTML toy" LLM output, tbh. The only thing impressive about this is how fast it generated it (13 seconds is wild!), but that's more of testament to Google's infrastructural advantage than to the quality of the model. For comparison's sake, I tried something similar with a couple other cheap models I've used lately, with the prompt "Impress me. Make something cool in HTML. Ensure that it is mobile friendly." (Added the mobile condition as I was on my phone when I did it). Mimo-2.5 created something similar, only a bit less complex than Gemini's (though, at least the FPS counter is real!), in a minute or two for about 1/3 of a cent: https://gisthost.github.io/?740c325c21e9bfbee59c4f94d9aab0af GLM-5.3-Flash, currently my workhorse model, spent 12 minutes (ouch) thinking about the prompt. Didn't cost me anything directly because I have a GLM sub, but I did the math and it would have cost about 1.1 cents through the API. Turned out nicely in my opinion (though in reality, it still isn't really anything special): https://gisthost.github.io/?9ef050e16cec2561e6504e725a3f0bcc Side note: thanks for setting up that Gist Host tool, it's very convenient! --- Editing to add this bonus from Mercury-2.5-Preview, which I just learned released a couple days ago. It's much less impressive-looking than any of the above, but it cost less than 1/20th of a cent, and the response was generated effectively instantly: https://gisthost.github.io/?02f40b50aa891bf396bfaaa3a7998203 I think it's not using GPU, because on my Firefox browser all the animations are going 3fps max. > this thing is really good at HTML JavaScript is pretty exciting. I would hope the people who make one of the most used JS engines in the world are capable of making a model good at JavaScript ;) Mission accomplished. That's both cool and fast. > That's both cool and fast. and probably a barely modified knock-off of some github project that it trained on You're so upset that you have to invent an imaginary hypothesis to make yourself feel better. Yes, very imaginary to think that the code comes from pretrained data and copy pasting whole blocks. It's not like this is exactly how LLMs work. Are you a frequent user of LLMs? That "copy pasting whole blocks" mental model doesn't hold up to regular usage, in my opinion. They probably saw that report years ago of copilot dumping out the fast inverse sqrt function, and assume that's all they can do. From experience most anti-LLM people have either never used them, or used them back in the 3.5-4 era and then never again, though you might have even more experience with those people than I do. :P LLM: produces a toolbox of an id and a clock User: use them both Made me giggle. it's such a weird split how most AI companies are trying to be the best, but Google really has a different mission statement. they already have users. lots of users. they need to be working on building models they can deploy and use with the most number of people, as they already have the users. i don't know if Gemini models per se are fully is in line with that purpose, but the results we see keep seeming to be in-line with that split-of-focus. I've been using Gemini 3.7 for my personal trip planning app. Across multiple benchmarks, it ranks higher on everything I tried: - Real world knowledge (when a thing opens and closes, the geographic region, historical facts). It's also the best at taking a cluster of places and working out a visiting order. - Photo ranking (which photo should be the hero). Gemini can tell whether a photo is of the thing or of the view from it. - Document parsing (extracting the relevant trip info from PDFs). If you use LLMs for anything other than coding, I definitely recommend not discounting Gemini like I did just because other models are more popular. Gemini 3.7 is my workhorse - fast and good enough for most tasks. Occasionally I go to GPT Sol or Claude to improve Gemini's output or for more complex tasks, but more than of my work usage is Gemini 3.7. Quite happy to test 3.8 now. Same here. I see so many people obsessing over the latest most state of the art bleeding edge models and yelling at Google for not being there, but I feel like the vast majority of people don't actually need those models. Flash has just been super useful and incredibly fast in my experience. I prefer luna for most development, especially when I am guiding the process. Sometimes terra. I have had terrible results coding with sol. It is way over-tuned on RL to make something that completes the task, no matter what. I end up with way too much code that does a lot of things I didn't ask for. Try planning with Luna, implementing with Sol with guidelines to not exceed the given scope. Sounds counter-intuitive at first, but Luna is overall better at sticking with what works. Sol is wicked smart but needs constraints. I love Luna too. An excellent model and still usually better value per dollar than Gemini if you pay for API tokens. Things may change with 3.8 - we'll know soon. I setup Luna as main Claude Code driver (so zero anthropic api use) and it nailed crisply a handful of python tasks, gonna continue this way. IME you're supposed to have Sol drive Luna sub-agents to do 90% of the work. Sol should primarily be the verifier and goal setter. Use omp.sh with Task Delegation -> Always to strongly encourage Sol to drive Lunas. Also Luna prefers to be talked to with English in XML. How are you able to get lots of usage out of it cost effectively? Google One plans are quite a good value actually - for a few bucks you get more Gemini plus space in Drive and other extras. Even through API, $3.75 for nearly Sol-level quality isn't that bad.
And let's not forget you can use it for free in AI Studio, and in the user app (even free accounts get tons of usage, though it's still 3.6 there), and in Antygravity. That's the thing. I am completely lost because there are so many redundant paths to get the same thing and I'm trying to figure out which one is the best deal Well are you looking for a subscription or pay-as-you-go API usage? This is what killed Gemini for me. The model might well be great, but the ecosystem Google has built around them is a confusing maze of not-quite-there products. Ultra AI is like $99/month, and it is hard to exhaust unless you are running a lot of concurrent requests. I've been benchmarking[1] models for trip planning and world knowledge specifically (to decide on which model to use with my travel app), and the Gemini models consistently come out on top. One thing in your comment surprised me: "when a thing opens and closes" Why you would rely on the model's weights to know opening hours, instead of having the model call a web search tool to verify it on the official site? I believe Gemini Flash is smart enough to know when to ground with web search. Their app has been saying it’s running a web search on almost all of my queries since 3.6. And given that Google … is Google, I trust them with web search grounding more than anyone else. I wasn't trying to be precise originally, I just tried to fit activities into "morning / evening" buckets. I did the whole itinerary with Opus first, but when I gave it to Gemini 3.7 Flash to review, it started correcting it with "this place will close 5PM" or "this place is closed for good". It was right on every nit, so it was surprising how well the model knows these things. If I ever release this I'll probably need the SERP API or Google Maps SDK (which I've heard is very expensive now), but for a personal trip where I will verify manually, using the LLM is okay for now. When you called the Gemini API, did you opt in to using search grounding: Google AI person here: Using grounding in Gemini is indeed backed by the same canonical data source for business information (like opening hours) as Google Maps. This stuff is available in its own API for a GCP fee, but we’ve built tooling to connect it to the Gemini agentic ecosystem as well. Gemini models - at least via some interfaces - have tool calling API access to various Google integrations. flights.google.com, maps.google.com, etc. The info isn't in the model weights. Because of where I live, there are three viable airports for any given flight I might want to take, which historically has made shopping a real pain. But Gemini (and only Gemini) has greatly simplified it. Pramble plus date range plus destination and it very quickly generates potential itineraries with costs, total travel time (driving included), etc. In my experience, Gemini 3.7 is excellent for general non-coding tasks. But for coding, especially backend development, I still find models like Opus 5 and GPT-5.6 more reliable. This sounds great to combine with Wanderlog using an unofficial MCP I made
https://github.com/shaikhspeare/wanderlog-mcp I've swapped over to it in the past two weeks, it's been really good. It does what I ask and doesn't think it knows better than me, which so far has made it the most pleasing experience I've had when slop-coding. My only wish is it were somewhat cheaper, as it tends to balloon pretty quickly when I'm using it in Opencode. I'm currently trying to offload a lot of work to subagents to stop the context expanding so rapidly. But on the upside, I rarely have to correct it - I've spent far less time arguing with this than with anything else so far. > Real world knowledge For awhile now I've found Gemini will use Google search for pretty much any real world knowledge, which is a huge plus IMO. It's basically Google with a much better frontend and no ads/seo nonsense. > basically Google with a much better frontend and no ads/seo nonsense so far I think it already has them but it's much more subtle. Also useful. When I've made certain sorts of queries I've had the distinct impression that it was attempting to very gently steer the conversation with suggestions. But it was brief, still answered usefully, and didn't resist going in the direction I wanted. So a win-win tactic I guess. For example find a beautiful landscape shot of a place that just so happens to be accessible to tourists and ask it something along the lines of identifying the location. IME it will noticably steer the conversation towards relevant commercial offerings and offer (entirely unprompted) to help plan a trip. Or ask it about a certain category of product with some requirements and it will initially present (relevant) options that look like paid placement to my eye. But if you ask it's happy to go on to turn up lots of alternatives and enumerate tradeoffs. Assuming I'm correct the subtlety is on par with product placement in movies. Certainly leagues better than the internet advertising we've suffered to date. It definitely steers. For example if it suggests travel plans, the booking links it provides give Alphabet a cut. As you say it was subtle, along the lines of "oh, if you are planning on going to the place you are researching, here are some helpful links to places you can stay". Subtle, in that it didn't get in the way of main result, so I didn't mind overly. Insidious, as I only noticed because I wondered why it was providing those particular links and looked them up. I can't see how you could ad-block them if I did object. And worrying, because these unblockable sneaky ads are just a first foray coming from a company that prostitutes its own app store searches, by making the first and most obvious result utterly unrelated to to the search topic. Instead it's who paid them the most to be there. That behaviour is why everyone dumped Alta Vista when an alternative came along. Alternative Android app stores can't come soon enough. They already skim off 15% of purchases which I'm sure makes their Android operation return a profit that makes other industries drool. Debasing their search to ad a tiny bit extra on top must by driven pure greed. Senseless, as I'm sure it will come back to bite them in the end. Fair. But I think there's a good chance it stays that way on paid plans. YouTube Premium is still ad free. Also them having their own silicon means they don't have to pay the Nvidia tax and can keep costs a lot lower. This thinking is why I am all in on GOOG shares. As a bonus, that means I’m getting part of Anthropic’s gains as well! I started trying out 3.7 Flash this week and it is competitive with opus/fable and also FAST. It is getting work done that anthropic models were struggling with and the speed with which it does is quite a bit noticeably faster. Beginning to think Google is a dark horse in this race and some of Anthropic's "everything feels janky and rushed" karma is going to catch up. I use Gemini because I feel like Google will win the AI race, and it’s Good Enough I want to believe this, but every time I try Gemini coding assistance within Colab it's utterly dire. Code gen in a cell is OK, but things fall apart when you try to get into a feedback loop. The system prompt/harness fails to inform the agent about what it can and can't do, or does and doesn't have access to. It will confidently tell you it's done a thing, and then you ask, it admits can't actually do that but will happily try and fail again. Very frustrating, because I really like Colab as a platform for little reproducible experiments that may or may not require CUDA. I suspect Gemini doesn’t focus that much on coding, at least compared to frontier labs. It’s not part of Gemini’s core business case. Gemini’s integration with maps and search is more important for Google. > Beginning to think Google is a dark horse in this race Google was so hyped up early Gemini 3 era (only some months ago). And now dark horse? The TPU takeover almost crashed nvidia and everyone else. Hype that burned out pretty quickly, it's hard to speak to the size and significance of old hype, I never felt it. Every time I personally tried Gemini models up until last week they simply couldn't do the long complex tasks I'd being doing with Anthropic models for many months. I tried similar travelling tasks but also added transportation and complex transfers (train, bus, walk, next train...). Worked meh and still a difficult thing to do for a llm. Can G3.7 use Google Maps for distance grounding? Yep. It has access to much better route planning tools than the other models. The results are really good IME. Sounds good! I've been planing around with LLM-based trip planning for a very long time now as it fits my very ad hoc style of traveling very well. But distances always had been.. lets say.. difficult. Will test it with my upcoming trip to Greece then! "Claude 3.7"? I asked Claude to fix the grammar of my comment, and it changed "I am using 3.7 for" to "I've been using Claude 3.7", so they sneaked their own name on it. incredible. further evidence supporting my personal stance to never ever let an LLM write or edit my writing intended for another human being to read. this is all me, baby you didn’t even read your comment before you posted it? Once you've written something, it's incredibly easy to overlook minute changes to the text. See: why authors wait days, weeks, or even months before editing what they've written (or, if you're more interested: cognitive regression, inattentional blindness, and the effects of misdirected saccades). He didn’t write it. > I asked Claude to fix the grammar of my comment I read this to mean he wrote the comment, then asked Claude to fix the grammar (as many ESL speakers do). Sounds to me like he did write it. 1. That's a wild conclusion from his comment! Understand that it's not an "either/or". 2. I'd wager the majority of HN commenters don't read their own comment before posting (pre-LLM days). I can read my own comment twice over and still somehow miss seemingly obvious wording errors. And if you do a 3rd pass with an LLM, people will accuse of you not reading your own comment... Eh that one is on me, if I think too much about my HN comment I end up deleting before posting it. I rely on the 1 min `delay` set in the profile page to fix before it goes live, but for some reason this time it was set to 0. I stopped using Gemini a few months ago because it would often just (partially) reply literal nonsense to me. Think 2023 style ChatGPT. Something like “to open a document on your Mac click File > Open docurrrar” - like it suddenly forgot it had to produce actual words. Overall I enjoyed its speed and comprehensiveness. But those occurrences of nonsense just made it feel like a great car that once a month just stops in the middle of the highway. Currently top at https://deepswe.datacurve.ai - beating Opus 5! https://artificialanalysis.ai/models/gemini-3-8-flash shows an intelligence score of 59, the same as Opus 5 medium! Wow - for a flash model this seems to benchmark powerfully. Remains to be seen what it is like to use. As of writing this comment, Claude Opus 5 has an intelligence score of 63, not 59 (it's not the same as Gemini 3.8 Flash). With a score of 59, Gemini 3.8 Flash is in eighth place, falling behind even Grok 4.6, Kimi k3, and GLM 5.3. They said Opus 5 medium - which does have an intelligence score of 59 (you have to select it manually from the dropdown to see it) They are all much larger and more expensive models. Google does not have a frontier model right now, but for cheap ones, they are better than event the chinese models now. That's not being debated here. The initial reported numbers were false and this was simply pointed out. You're changing the subject. Opus 5 medium has the same score as 3.8 flash on artificial analysis intelligence index. Are you implying Google or Artificial Analysis are reporting false numbers? What's your source? BTW you're comparing 3.8 flash high to opus 5 medium. 3.8 flash medium scores lower. Flash models are on the order of 1/10th the size of Opus models, so some flex in the thinking level is fair. When comparing closed models, the only thing that actually matters to anyone using them is some mix of cost and speed. Considering how much memory a server is using, when evaluating models that you'll never have access to in order to host yourself, doesn't really make sense. Flash is just a name with no defined or consistent meaning even within labs, let alone between them. Considering both are closed weight, there is no way to truly assess how big the size delta between the two is. Then again, who cares about size, performance and end-to-end speed+cost are what matters along with task adherence, task assessment and so on. Model size also can not be inferred by tokens/sec for a multitude of reasons, but to showcase two examples, Opus 5 and Sonnet 5, as well as Gemini 3.1 Pro Preview and 3.1 Flash have each very comparable output speeds when using the same deployment as a basis for comparison, despite it being very likely that within their generation, the former are larger than the latter. Feel the need to mention this, as I unfortunately stumble upon so many poorly reasoned, speculative hype post trying to infer model size via utterly unreliable metrics, not based in actual data. It’s like comments below arguing about the reasoning levels not normalized to some metric (like cost, output token amount or duration) but just the labels or high, max, medium, etc. Those mean almost nothing even when comparing models based on the same pretrain (just compare GPT-5.4 to GPT-5.2), they mean less than nothing comparing different labs releases. It's not totally a mystery https://arxiv.org/html/2604.24827v1 The short of it is by using hard facts knowledge that is difficult to compress, and then quizzing models on these facts and calibrating against a bunch of open models, you can kind of feel out the size of closed models.
simonw - 13 hours ago
simonw - 12 hours ago
badlucklottery - 11 hours ago
noir_lord - 11 hours ago
estearum - 4 hours ago
luipugs - 2 hours ago
senordevnyc - 10 hours ago
noir_lord - 9 hours ago
Qworg - 7 hours ago
dr_kiszonka - 4 hours ago
cyrilng - an hour ago
cheikhcheikh - 7 hours ago
senordevnyc - 8 hours ago
iterateoften - 9 hours ago
skybrian - 9 hours ago
Forgeties79 - 9 hours ago
briHass - 7 hours ago
agumonkey - 2 hours ago
silasdavis - 10 hours ago
tomjakubowski - 8 hours ago
heliosAtwork - 12 hours ago
bermudi - 11 hours ago
gundmc - 10 hours ago
criley2 - 6 hours ago
NicoJuicy - 8 hours ago
walrus01 - an hour ago
hglaser - 12 hours ago
simonw - 12 hours ago
Kayou - 9 hours ago
embedding-shape - 7 hours ago
dennis16384 - 9 hours ago
wyrdcurt - 9 hours ago
meerita - 9 hours ago
giancarlostoro - 12 hours ago
pietz - 13 hours ago
wayeq - 12 hours ago
sawjet - 12 hours ago
superze - 12 hours ago
simonw - 11 hours ago
ChickeNES - 10 hours ago
estetlinus - 11 hours ago
jauntywundrkind - 11 hours ago
jampa - 13 hours ago
handzhiev - 13 hours ago
owlninja - 13 hours ago
aero142 - 12 hours ago
sneezychl - 12 hours ago
handzhiev - 11 hours ago
jesuslop - 11 hours ago
greenavocado - 12 hours ago
greenavocado - 12 hours ago
handzhiev - 12 hours ago
greenavocado - 11 hours ago
qlte - 3 hours ago
It's not really any different than the choice you'd make with OpenAI/Anthropic depending on how you plan to use it. Except as a hyperscalar, it's also offered first party from Google Cloud (like Claude via Amazon Bedrock or GPT via Microsoft Azure OpenAI Service): Subscription? -> Google One plan (http://one.google.com/)
API? -> AI Studio (https://aistudio.google.com/)
But if you're using models via OpenCode or Pi or whatever, the flow chart is basically just "Go To AI Studio" unless you or your employer is already used to Google Cloud, otherwise there's no need to subject yourself to all those enterprise-y IAM dashboards and stuff. You still get free usage from AI Studio when you generate the API key without needing to add billing details so very easy to try. Google Cloud -> Gemini Enterprise AI Platform (https://cloud.google.com/ai)
-0_0- - an hour ago
seanmcdirmid - 6 hours ago
CamilleScholtz - 10 hours ago
rahimnathwani - 13 hours ago
plaidfuji - 13 hours ago
jampa - 13 hours ago
rahimnathwani - 12 hours ago
I'm curious whether in fact you were getting answers from the model weights (which is what I had assumed) or whether your API calls were resulting in web search tool calls. tools=[{"type": "google_search"}]
kridsdale1 - 11 hours ago
NiloCK - 4 hours ago
dassh - 3 hours ago
Shayk - 4 hours ago
robotmay - 12 hours ago
dismalaf - 13 hours ago
altmanaltman - 12 hours ago
fc417fc802 - 11 hours ago
rstuart4133 - 8 hours ago
dismalaf - 11 hours ago
kridsdale1 - 11 hours ago
colechristensen - 13 hours ago
cousinbryce - 11 hours ago
throwaway219450 - 10 hours ago
spacebanana7 - 8 hours ago
re-thc - 13 hours ago
colechristensen - 12 hours ago
forlorn - an hour ago
BlackRabbit1 - 13 hours ago
porridgeraisin - 13 hours ago
BlackRabbit1 - 12 hours ago
tziki - 13 hours ago
jampa - 13 hours ago
trial3 - 13 hours ago
dymk - 13 hours ago
drusepth - 12 hours ago
dymk - 9 hours ago
drusepth - 3 hours ago
BeetleB - 13 hours ago
fc417fc802 - 10 hours ago
BeetleB - 9 hours ago
jampa - 13 hours ago
leokennis - 9 hours ago
mattlondon - 14 hours ago
theHocineSaad - 13 hours ago
kamranjon - 12 hours ago
Squarex - 13 hours ago
pietz - 13 hours ago
mattlondon - 12 hours ago
asdfologist - 12 hours ago
WarmWash - 10 hours ago
nomel - 5 hours ago
Topfi - 8 hours ago
WarmWash - 6 hours ago