A 40ms Go garbage collector pause caused by swap
frn.sh115 points by shellpipe 3 days ago
115 points by shellpipe 3 days ago
Unfortunately that's somewhat expected - regardless of implementation, the sole fact that the GC needs to somehow walk the entire tree to trace still referenced sections makes swap highly impractical -- even if you had not had 40ms stop-the-world pauses the LRU cache used by swap would get thrown away at every GC.
I believe that's actually the same reason why Apple stopped using GC in their frameworks in favour of automatic reference counting.
> Unfortunately that's somewhat expected - regardless of implementation
Yes, that's expected and no not "regardless of implementation": the GC implementation CAN be improved.
See this discussion on reddit: https://old.reddit.com/r/programming/comments/1wf2fei/40ms_g...
Copy/pasted here: >>
I remember a research paper about swap and GC, where the GC cooperated with the OS to avoid this kind of issue. AFAIK it went nowhere, too bad.
[–]andreiross[S] 15 points il y a 21 jours
Are you talking about this one? https://cse.buffalo.edu/\~mhertz/bc-pldi-2005.pdf. If so, yes. Too bad. I don't know the repercusions this paper had in the past, though, in the sense of pros and cons of the bookmark collector. Don't know if anyone tried to actually implement it or design it at some point.
[–]renozyx 9 points il y a 21 jours
Yes, congratulations for finding it. And I don't know either,. Except that they did implement it on Linux (of course) https://plasma.cs.umass.edu/emery/cooperative-memory-managem...
<<
Modern OSes need a facility to signal to a thread that it hit a paged out chunk of memory. It's not like it's not possible to create a swap-aware GC (or any other kind of code that touches lots of memory).
I would go further - I think the whole swap lifecycle needs to be communicated. Before the OS swaps out a page, if it could invoke the GC which would clean up that piece of memory so that we dont end up writing garbage to swap.
The OS should also allow marking pages as piority to stop them from being swapped out.
> The OS should also allow marking pages as piority to stop them from being swapped out.
This is IIUC possible using the mlock(2) family of syscalls: https://man7.org/linux/man-pages/man2/mlock.2.html. (On Linux, though I'm guessing other UNIXen and operating systems have it or something equivalent.)
https://www.kernel.org/doc/html/latest/admin-guide/mm/userfa...
I'd guess it ain't gonna perform great either, though.
Reference counting is a GC algorithm, and no this isn't expected, it depends pretty much on the implementation.
Many make the mistake to think there is only one way to do a GC.
One of the authoritative books on the subject, https://gchandbook.org/contents.html
And a quite well known paper on the matter as well, https://dl.acm.org/doi/10.1145/1035292.1028982
Whether reference counting is a GC algorithm depends on how you define what GC is.
I prefer to consider GC only the methods of memory management where reclaiming the no longer used memory is done either asynchronously with the main program or as late as possible, i.e. when new allocation requests cannot be satisfied.
In the normal implementation of reference counting, memory is freed as soon as possible, i.e. exactly like stack memory, when blocks are exited, so I do not consider reference counting as GC.
The problem with GC in the strict sense is that you cannot predict when it will happen. With both stack memory and reference counted heap memory you know that whenever you exit a block, some time will be spent with running destructors and for freeing memory, but such interruptions will not happen in other points of the program.
> Whether reference counting is a GC algorithm depends on how you define what GC is.
Pretty much all the high-performance GC/refcounting algorithms are hybrids in one form or the other; it's a spectrum of choices. https://dl.acm.org/doi/10.1145/1028976.1028982 explores this in some detail.
That is a classic paper and obviously I am aware of it.
However, if you have distinct names it is efficient to use them with distinct meanings.
Making "garbage collection" synonymous with "freeing memory" is bad, because it eliminates a means to distinguish various methods for freeing memory.
Like I have said, I consider useful to define "garbage collection" as any method of freeing memory where the memory is not freed as soon as possible (i.e. when a block is exited), but freeing is deferred to be performed at a later time, even as late as possible (i.e. when new memory allocation requests cannot be satisfied).
Indeed, many garbage collection algorithms use reference counts, where memory deallocation is deferred, but when I use the term "reference counting" without any other qualifier, I mean it in the sense in which it was originally defined in 1960, where the time when memory deallocation is run is predictable, exactly like for stack-allocated memory.
I prefer to write programs with well-defined worst-case behavior, so I normally prefer deterministic algorithms. Thus I always prefer to use reference counts instead of GC. I have never encountered a case when avoiding reference cycles was difficult.
I think it's saner to be literal here. Reference counting is a way to automatically collect garbage for you. Why shouldn't it be called a GC then?
I define the academic view of GC algorithms in Computer Science, not what random developers decide to call GC.
Which in an industry where some folks call themselves Software Engineers after a bootcamp, without any kind of accreditation, I rather stay with the definition from those that do language design and compiler algorithms research.
I prefer to use the terms as they were originally defined by the authors who introduced them, and not with the modified meanings that become fashionable after the authors of some paper written some decades later decide randomly to change the definitions of the old terms. This is also true for the terms "garbage collector" and "reference counts", which were both introduced in 1960 to denote 2 clearly distinct methods of memory management, and the main difference between them does not consist in whether some kind of reference counts exist somewhere, or not.
The paper "A unified theory of garbage collection", which has started the fashion of considering reference counting as a kind of garbage collection, only shows correctly that both tracing and reference counting are complementary implementation techniques for a garbage collector.
It does not mention anywhere the essential practical difference between the traditional standalone memory management with reference counts and a garbage collector, which stays the same regardless whether the garbage collector also happens to use reference counts for some purposes, which is the difference between predictable and unpredictable times when memory reclamation is done.
Many of the modern authors of academic papers are a poor model of using computer terminology (or for the terminology in other domains), because very frequently it is obvious that they have not read the old works where such terms were introduced for the first time, even when such works are cited in the bibliography.
Unlike them, I have done an extensive research to find when and where various computing terms have been used for the first time, and I strive to use most terms with their original meanings, not with corrupted meanings, even in the cases when the latter have become more popular lately.
So as fellow digital archaeologist I would find interesting to find Lisp, BASIC, CLU, SIMULA, or Cedar papers, where the authors mention reference counting not being a form of garbage collection algorithms.
Cedar was already combining reference counting with a cycle collector, as one of the very first systems programming languages with automatic resource management.
> The problem with GC in the strict sense is that you cannot predict when it will happen.
It’s the same with ARC. You also don’t know when the counter will reach zero.
No, that is not true.
In the normal implementation of reference counts, counters can be decremented only at block exits and not at any other program point.
At a block exit some of the local variables that are freed may contain references, so freeing them will decrement some reference counts. Then some counters will reach zero, triggering other deallocations and the decrementing of other counters. This will repeat until no other counters reach zero.
All the memory deallocation happens predictably, only at block exits.
If a variable is not freed immediately when a counter reaches zero, but the deallocation is deferred for a later time, which is not predictable, that is no longer classic memory management with reference counts, but it is a garbage collector, which happens to also use reference counts, probably in combination with some tracing algorithm.
When reference counts are implemented, manual memory deallocation, like with C free() or C++ delete, should be forbidden, but even if it were used that would just introduce other program points besides the block exits, where it is known that memory deallocation will happen.
You've described where deallocation can happen, not when it will. Every block exit is a candidate, but which one drops the last reference depends on runtime state: how many other owners the object has and which of them goes away first. With shared ownership across threads it's a race by construction. The free runs on whichever thread happens to release last.
The cost isn't bounded either. Dropping the last reference to the head of a list or the root of a tree frees the whole structure at that block exit, and its size is a runtime property.
And it isn't only block exits. Every assignment to a variable or field holding a reference decrements the old target, and so does removing an element from a container. Swift's ARC doesn't even promise the scope boundary: the optimizer may release right after the last use, which is why withExtendedLifetime exists.
By the same "where" criterion a non-concurrent tracing GC is predictable too, because it can only run at allocation points. That doesn't tell you which allocation will trigger it, just as knowing the block exits doesn't tell you which one will free.
In common use, GC refers to mark and sweep and similar systems where allocations are not freed at an easy to predict time.
In common use, by those that never learn the theory behind their tools.
That is a tracing GC by the way.
There are also tracing GC implementations with deterministic resource management APIs, .NET and D have them for example.
People use words in a way that is useful. Really this just shows that the pedantic academic definition of GC is not the most useful one.
Useful to them usually, or it goes to show how little our profession cares about learning the proper terms, or its historical background for that matter.
Anyway it doesn't matter any longer, AI is going to replace most of us, and it comes with automatic everything.
You just need to remember that garbage (unreachable reference cycles) isn't collected.
The issue described in the article isn't really GC specific. The Go GC has a critical section that reads metadata.
Any program that has a rarely accessed, but vital chunk of memory is vulnerable to this sort of issue.
A region-based collector like Hotspot's G1 combined with madvise could probably operate in a manner that's more swap-friendly by focusing on a smaller working-set at any given moment and announcing its intent to switch the working set to the OS ahead of time.
Apple should just stop using a silly mark & sweep collector. All the good collectors are copying, if you have enough memory. Only on tiny devices you need mark & sweep.
Discord learned this back in 2020 and published a great blog post on it: https://discord.com/blog/why-discord-is-switching-from-go-to...
I don't understand why they felt necessary to rewrite a core component in another language. If you have a garbage collection problem my first intuition would be to produce less garbage!
For example better using the stack, or pulling out the big gun of manual memory management.
I'm sure they had reasons to choose Go when they first designed this project but they don't go into them at all.
Feels like they just wanted to play with a new toy.
> If you have a garbage collection problem my first intuition would be to produce less garbage!
FTA:
“These latency spikes definitely smelled like garbage collection performance impact, but we had written the Go code very efficiently and had very few allocations. We were not creating a lot of garbage.
[…]
the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references”
> produce less garbage!
They explained in the post why this wasn't an issue: they were producing very little garbage, but there was a very large object graph.
> manual memory management
If you need to do manual memory management in a GC language with no builtin support for it, like Go, that's probably a sign that you should switch to a different language.
Like D, a proper systems language with GC, and builtin support for manual memory management, and there are others as well.
If you read the link you'll see that they found out go forces a GC every two minutes no matter the amount of garbage.
And importantly in a tracing GC the work load is a function of the non-garbage data not of the garbage.
Generational GC can avoid some of it by ignoring the old code entirely, but Go does not implement generations because its relatively strong ability to stack-allocate generally replaces the nursery, and for most work loads you don’t recoup the costs.
Problem with languages like Go is, because they have GC, their API is less tuned to the ability to prevent new allocations.
In C you see lots of functions whose first parameter is a buffer that will be re-used.
And even in Rust you don't see this often, because reusing a buffer means having to do &mut, which means having to re-structure your code. And it's really easy to do Vec::new().
One of big issue with GC is it need to scan every reachable objects to mark it is reachable. That mean even if your code produce no garbage but have a large amount of long lived object the GC still need to traverse all of those objects every cycle.
Only in very bad implementations of GC.
By that definition, Go has very bad GC. Many (myself included) would strongly disagree with this.
Every memory management scheme makes trade-offs (even manual ones), and golang's is no different. It seems to work well in many scenarios.
It is better to scan everything than to copy it.
The op, most likely, was referring to generational GCs. Golang notably doesn't have one.
That’s a great read. And an incredibly annoying “floating footer”
If you're on desktop, you can use uBlock origin's element zapper to remove annoying things like that.
"It hurts when I do this"
"Stop doing that"
If you care about latency, disable swap. System wide or for the specific the cgroup.
This is not entirely correct. If you care about latency, then it doesn't matter on which major fault your application gets paused. Disabling swap protects you from data pages being evicted, but code pages can still be paged out.
If you care about latency, mlock() your memory, do not disable swap. Swap is good and gives the kernel an equal opportunity to evict data and code pages.
For GC enabled languages swap is universally bad. Some gc-pauses are indistinguishable from a system crash. It's a side effect on not having tightly specified memory limits.
I'd rather have applications be oom_killed than having them swap out, the former is rather obvious and demands action.
If you want you application to stay in memory, then make it explicitly with mlock()/mlockall().
Disabling swap will just moves pressere elsewhere: to code pages. And evicted code page is no better: full stall while kernel loads that page from disk.
FYI: This `delamon` account seems to be meat proxying us. Just relaying garbage AI answers without understanding his own words. Doubt he's ever used mlock himself for anything.
His reply below was presumably killed by mods for doing this: https://news.ycombinator.com/item?id=49970841
I'm not. If you think that I said something not correct, do correct me. We're all learning.
I see that you've rewritten your AI generated reply there. So when have you used mlock in production for Go services, the way you're advising others to do? What were the circumstances?
I did not rewrite anything.
Yes, I was and still running go and zig binaries with mlockall. Apps have large in-application caches and also perform large streaming reads of datasets bigger than available RAM. If it happens that several streaming reads running at same time, they push memory pressure hard enough, so kernel swaps out cold parts of in-app caches. That later leads to elevated latencies => mlockall to avoid that.
While I've seen systems grinding to a halt due to evicting code pages, the idea that disabling swap distributes load between code pages and data pages makes no sense to me. Data pages are also disk backed, so they could also be evicted at any time. Why would data pages be written to swap if they came fron the disk in the first place?
So that only leaves anonymous data pages, i.e., regular data in memory, that could be swapped. Ok but you're not easing the load on code pages, those still get evicted during memory pressure whether you have swap or not.
The assumption that data pages can always be easily discarded is incorrect. Dirty data pages can't be discarded until they are written back. Recently accessed data pages are also not discarded until all inactive data and code pages are discarded.
I think discussing it in a vacuum is pointless, it should depend on how much memory you have. You can have a swap on, but with a low `vm.swappiness` number.
The point is that disabling swap does not prevent page eviction. Instead of swap, this discussion should be about the page cache.
Surely as long as it is all in memory these pauses are not going to be that significant, details like that should be left for the language devs to optimize.
You can tell an average Golang user to use mutexes, you shouldn't be telling them to lock memory manually.
vm.swappiness doesn’t work the way most people assume it does. and it is ignored under heavy memory pressure.
Maybe, but it still there and probably has a different value between distros as well. Default value is 60, what is that good for exactly?
If you're in a position where you still want swap to be there, but only used in extreme situations, then you should set it yourself. Otherwise disable swap or switch a language.
What do you think this value of 60 means? Also, as I said, in extreme situations, when the system is right before invoking the OOM killer, swap is used regardless of `vm.swappiness` setting.
> What do you think this value of 60 means?
Some arbitrary number.
https://docs.kernel.org/admin-guide/sysctl/vm.html#swappines...
> At 0, the kernel will not initiate swap until the amount of free and file-backed pages is less than the high watermark in a zone.
I don't see a reason why the default shouldn't be 0 then on modern systems.