A new Go API for JSON
go.dev392 points by olexsmir 3 days ago
392 points by olexsmir 3 days ago
I'm working on better SIMD support in Dart and one of the motivations for that work is specializing parser specifications based on schema specifications of that language.
Or more concretely, if we had a parser generator + json parser specification that can generate efficient simdjson style parsers, then we should be able to parse json restricted to a subset of json based on a schema at least as efficiently as simdjson can parse general purpose json.
That way we could have the best of both worlds, a (potentially lazy) statically type-safe model and simdjson performance!
I still think that this is definitely doable, it's just that a lot of infrastructure needs to be built for it.
I remember reading that Meta is doing something similar with binary formats. I find it weird that there's nothing like that for json yet.
This approach is called "staged parsing":
I was working on metaparsing earlier this year (specifically over grpc because of its better support for types and better serialization format).
I think most people who use JSON don't want strong typing or to use schemas. If you do, you just use a different serialization format: JSON is designed to be a generic, human-readable way to basically represent any kind of data structure (since it deserializes to JS objects). So everything about JSON is working against you if you do try to enforce type/schemas on it.
A grpc-like protocol with more prescriptive type registries/discovery (over the network) and better interop with http (like grpc-gateway) would be the best approach IMO. Definitely not JSON schema
> think most people who use JSON don't want strong typing or to use schemas. If you do, you just use a different serialization format: JSON is designed to be a generic, human-readable way to basically represent any kind of data structure (since it deserializes to JS objects). So everything about JSON is working against you if you do try to enforce type/schemas on it.
I have been using serde without issues, and never resorted to `serde::Value`[0], the untyped option, so far.
I have yet to experience anything working against me.
[0]: https://docs.rs/serde_json/latest/serde_json/enum.Value.html
> I think most people who use JSON don't want strong typing or to use schemas.
Not necessarily. It's biggest advantage is probably that it has very good interoperability. Basically every language has a readily accessible library for json, and even if you don't, it isn't that hard to write a parser and/or generator for it. And being human readable isn't something to sneeze at either.
> I think most people who use JSON don't want strong typing or to use schemas.
We use JSON because it's standard, human-readable, mostly self-described and works natively with Javascript.
You can read JSON object without having to know the schema even if one exists.
Protobuf is unusablr garbage without a lot of supporting tooling.
I think it may be fairer to say, “most people pick up JSON because of the flexibility.” Those who continue to use it for anything where there’s a consumer of your API probably do come to want types. In my day job I am starting to enforce end-to-end typing as best I can (given it’s Python and JSON) because our payloads are partially content from agents and I want to be able to flag up weird shit if we can. Certainly not the only gate that we have for looking for prompt injection and not even a great one, but it provides some piece of mind.
> I think most people who use JSON don't want strong typing or to use schemas.
Pretty much anywhere I've come across JSON (and xml before that,) everyone expected strong typing. The fact that JSON (and xml) can represent different data than they anticipated "did not compute" in their heads.
In many cases, they were treating JSON (and xml) as dumb serializers and not even paying attention to the output and corner cases.
I definitely want jsonschema, it's a standard and i can use it for many tools. openapi spec is also a standard and built on top of jsonschema. I can use jsonschema to generate models, types, servers, client sdks, automate forms etc. I think it's better than grpc for some of these things, plus protobufs and grpc have bunch of tooling that i can pretend to understand but in depth i don't really.
I generally like `encoding/json/v2` and I'm glad they did it.
I wish they'd left `MarshalIndent` in there though. The equivalent code is super complicated and non-obvious, so it's a pain to migrate, and it's a pain when you're banging out some quick code and want to dump some readable JSON. I suppose these days the LLMs can write the complicated version for us, of course.
I'm not sure if I'm missing something, but isn't this functionality just moved into jsontext? https://pkg.go.dev/encoding/json/jsontext#Value.Indent
v := jsontext.Value(b)
v.Indent()
fmt.Println(string(v)) func WithIndent(indent string) Options
https://pkg.go.dev/encoding/json/jsontext#WithIndentI love paying 10 cents to some billionaire to serialize some json. Screw using my brain!!! Ill rent me some competency
Not an insult to you, its just how I feel things are going, myself inculded at times.
Edit: why do yall downvote this lol, you know its true.
> in any, out any
Completely unrelated: can anyone tell me why language authors introduce in/out/inout function parameters (see also C#).
It feels like this is some rather bad workaround for something that breaks assumptions about parameters being passed into the function.
In Go, it’s a question of who owns the allocation. If you return a pointer type allocated within the function, the answer is “no one,” so the pointer escapes and has to be managed by the garbage collector.
If the allocation is owned by the calling function which uses it internally and then doesn’t return it, it can be a stack variable. Or the same allocation can be reused many times in a loop. Or an array of allocations can be made all at once with foreknowledge of the total number.
If you pass a pointer into a function, the Go compiler can (and sometimes does) allocate it on the heap. Go has no allocation/ownership semantics, tragically, so the best we can do is depend on undefined behavior regarding the escape analyzer and hope the Go team doesn’t break it (which is a relatively safe assumption, considering that it’s unlikely that the Go team would start allocating things on the heap that were previously allocated on the stack—Go rarely regresses performance like this).
The parameters named "out" here are not "output parameters" in the way you seem to be thinking. They are "input parameters" named "out", because they are the buffer or pipe you want the data written to.
`out` params in C# are params that must be set within the function. They are way to implement multiple return values and tell the caller "this is written to, not read from."
But you might be conflating that with in/out for Contra/Co-variance when specifying a generic.
> They are way to implement multiple return values
They are one way to implement multiple return values. For multiple returns, tuples can be used. In fact, async methods can’t declare out params
C# `out` is kinda nice because it can be `var`d and used directly.
Erm... what do you mean?
if (int.TryParse(str, out var value) && value == 5)
{
// value is available
}
// value also availableI used the jsontext package at my company when we needed to make a parser for challenging clients (specific errors for each kind of mistake) while still keeping performance, and it works like a charm.
There is no introspection and speed is superb. I would definitely recommend jsontext as a parser for anything non-straightforward.
I… think this is good?
I'm just getting the neuron-band back together where Golang is concerned. I have done some in the past for little system tasks, but I am switching away from interpreted languages entirely for web stuff, and initially to Go, as I switch away from hand-coding to at least LLM-supported coding.
So I am a bit out of date on the pain points (I never really felt pain points when I last used it, because the jobs I was doing had very small, quite Go-friendly shapes).
To preserve the order when processing JSON a workaround has been introduced:
https://news.ycombinator.com/item?id=44893833
Still searching the true solution:
https://github.com/golang/go/issues/27179#issuecomment-53634...
A proposal of the missing datatype for this operation mode has been just accepted:
The fact that this took 6 years to land stood out to me.
I'm not entirely surprised, and I'm glad the Go team is carefully considering their interfaces before launching. But 6 years seems like a lot of work to land a JSON API.
Sometimes I wonder about Go's velocity - generics (something that I still haven't used beyond the tutorials) took an age as well
JSON processing, quite markedly, is renown in Go for being slow, so I don't really understand why it took so long
For like 99% of use cases encoding/json was fine. Of course improvements are welcomed. But personally I've lead development of the backend of part one of the biggest sites of its kind in the world, and we just used encoding/json and net/http. Most of our use cases are just building APIs, client facing ones, internal ones, trusted third-parties. These systems have pretty high throughput, more than most people will ever work on.
Within our company I saw people in other teams writing their own optimised JSON libraries for whatever hellish reason. I always reiterated the point that as soon as that app made a single IO request all those nanosecond savings are invisible. Yes, it affects your total CPU time for processing a request but compute was never ever a limitation for us and it's likely not a limitation for most people. And if the trade off is your app needs scale slightly larger versus writing custom JSON code or pulling in a third party dependency, then I'm accepting the larger scaling.
The biggest benefit of encoding/json/v2 isn't the speed, it's the improved API.
I think in the case of JSON, it was “good enough”. People who found this to be a problem would use one of the many 3rd party options.
I kind of like that the standard library doesn’t move fast. It makes it easier to fit the ecosystem into my head (unlike, for example, the JavaScript ecosystem where I find code doesn’t survive 2 years before failing to compile).
But I do agree that slow and steady produces its own problems. Your example of generics is a good one. But think the example of Go modules is another, perhaps even better, example of how sometimes the Go team really should have accelerated their efforts on.
lol, I don't mean to be contrary, but I looked into dependency management (partly because of the modules implementation, partly because of early teething issues, and partly out of pure curiousity.
Dependency management is a freaking nightmare, nobody has managed to create something that is bulletproof, not cargo, npm, apt, yum, nix, homebrew, nothing.
The key issue is that you're managing 3rd party software that may or may not adhere to the prescribed lifecycle/versioning
The Linux variants do it by each having dedicated teams watching upstream sources, taking what they believe to be whatever they feel is right, building for their nominated target(s), then testing, before deploying (on top of that, all of these steps vary in importance and rigor depending on numerous variables)
For programming libraries (which is what depmod, goose, vgo, and modules are managing) it's just as bad, if not harder, because their very nature is to provide a base upon which other developers can turn into whatever their imagination.
Add to that, modules was (initially) written by people who were dealing with a slightly different ecosystem; Google rather famously has a monorepo, making their lives a little easier when it comes to managing dependencies.
So, with those things in mind, dependency management is always going to be a hard problem that is opinionated (eg. I have a firm opinion that there should be an easier way to "pin" a dependency, other than vendoring - note that this isn't an invitation to every junior Go developer to come and claim go.sum does that. It doesn't. go.sum is a log of what dependencies were used for the last build by whomever wrote to it)
I’ve written package management tools and I agree that dependency management is hard. Though I wouldn’t say it’s that hard compared to the other problems you encounter with language development.
But difficulty aside, my point was Go waited until there were multiple established 3rd party package management tools for Go already in widespread use before even releasing their own experimental implementation.
Their inaction caused far more disruption to the ecosystem because people got fed up of waiting, implemented their own, integrated it into their workflow and then had to reintegrate a whole new system all over again a year or two later.
Your point about Googles needs is the one often cited for Gos inaction. But I think that actually makes the situation worse. Because if the Go team cannot empathise with use cases outside of one bespoke enterprise, then it really shouldn’t be released as a general purpose language available for all to use.
And I say this as someone who generally likes the language.
Interesting, I came to the opposite conclusions, but for the same reasons. I like that they waited to see what alternative proposals/implementations were out there before they decided on their own.
Especially since "Go team cannot empathise with use cases outside of one bespoke enterprise" — it shows self-awareness of their shorcomings. I think it's wise to let the target audience explore the design space themselves a bit first. Considerably lowers the risk of cementing the wrong design.
I do somewhat remember the drama around this at the time, but I my impression was that of a petulent community of various dependency management projects with too much ego attached to them to see the bigger, long-term picture.
That’s an interesting take. I appreciate the alternative view point and I can’t say I disagree with the way you’re looking at it.
Thanks for sharing.
> there should be an easier way to "pin" a dependency, other than vendoring
Put this in your go.mod:
replace example.com/package-to-pin => example.com/package-to-pin v1.2.3
Then go.sum will do its actual job (along with GOSUMDB) and ensure you only get exactly the revision of the code that you want.> there should be an easier way to "pin" a dependency, other than vendoring
You left out the part that explains why.
> generics (something that I still haven't used beyond the tutorials) took an age as well
Only about a year from when the Haskell guys lent their expertise in designing it and it landing in a release. That's actually fairly quick given that it was a pretty significant change under the hood and constrained by only two releases annually.
The Go team was always clear that they didn't have the necessary capabilities internally, despite a lot of failed efforts trying anyway, and were relying on the open source community to lend a hand. Unfortunately, open source doesn't mean open willingness to contribute. It turns out that for all the experts on HN, posting "go doesn't have generics lol" was preferable to helping out.
Those guys were the same that helped out with generics in Java, they even published a book in 2006,
https://homepages.inf.ed.ac.uk/wadler/topics/gj.html
Before Featherweight Go, there was Featherweight Java.
So maybe a language designed in 2012 (v 1.0 release year), should learn from history instead of hand-waving it away as needless fluff.
> In retrospect, we were biased too much by experience with C++ without concepts and Java generics. We would have been well-served to spend more time with CLU and C++ concepts earlier.
https://github.com/golang/proposal/blob/master/design/go2dra...
And not all on the team were convinced even after Featherweight Go, here Rob Pike,
> I think the transition to getting polymorphism into the language through generics is has still got some years to work through there's still things that don't quite work right
> instead of hand-waving it away as needless fluff.
If you go back and watch the original Go announcement it was made abundantly clear that they believed Go did need generics but that they didn't have the expertise to do it. Yes it is true that the book was already written on it, but that doesn't mean anyone on the Go team was familiar with it or had the time to become familiar with it. Programming languages do not require gods to design them. Imperfect humans can too.
The experts on HN who know all the ins and outs of generics could have contributed but, you know, shitposting "go doesn't have generics lol" was a better use of their time. There are so many hours in the day.
Gonuts is full of content that they were not keen in generics, in fact gccgo has been abandoned, and I expect it to eventually follow gcj and be removed from GCC.
Go culture is not welcoming to experts contributions, Rust, D, C#, Swift and others, heck even Java, are more open arms to language designers.
Nobody is keen about language features. Each new one adds complexity. That isn't the same as failing to recognize a need, though. And we know the Go team recognized the need as they clearly kept working on generics all the way from the very beginnings up until the Haskell muscle showed up. The receipts are there. The results just weren't good, as one would expect from a team who weren't shy to say that they didn't have the expertise necessary.
You'd think one of the HN experts would have shown up to offer that expertise, but I guess there was a lot of "go doesn't have generics lol" needing to get out.
Some did try, and were welcomed with stones, clubs and pitchforks, until leadership had to reach out for help, from the same team that designed Java generics, years before Go v1.0 came to materialise.
The Haskell muscle had proven their skills already with Featherweight Java, in 2006.
The same people who were working on generics in Go prior to v1 were also still there to help out when it eventually landed, so whatever stones, clubs, and pitchforks you saw weren't compelling enough to scare one away from making it happen. Not that anyone would expect them to. Meaningless distractions don't get in the way when doing something is worthwhile.
But obviously "go doesn't have generics lol" comments on HN were even more important. Just think of what would have happened if that wasn't repeated multiple times daily for years and that effort was put into getting Go generics into place much sooner. Count your lucky stars or something.