Branchless Rust: Making a Filter 4x Faster by Removing an If
greyblake.com276 points by greyblake 4 days ago
276 points by greyblake 4 days ago
Nice post!
You can do even a bit better if you're willing to use intrinsics. In particular this kind of operation is well-suited for compress-type operations, available as a first-class operation in at least AVX512, SVE and RVV; you can also emulate them reasonably quickly on NEON and AVX2.
Here's an example, building on the OP's work:
pub fn filter_compress(input: &[f64], threshold: f64) -> Vec<f64> {
use std::arch::x86_64::*;
let mut out = vec![0.0; input.len()];
let mut n = 0usize;
let (head, tail) = input.as_chunks::<8>();
for chunk in head {
unsafe {
let p = _mm512_loadu_pd(chunk.as_ptr());
let m = _mm512_cmpnle_pd_mask(p, _mm512_set1_pd(threshold));
let compress = _mm512_maskz_compress_pd(m, p);
_mm512_storeu_pd(out.as_mut_ptr().wrapping_add(n), compress);
n += m.count_ones() as usize;
}
}
for &x in tail {
out[n] = x;
n += (x > threshold) as usize;
}
out.truncate(n);
out
}
Nice! I saw the code and thought, I bet there’s a way to do some SIMD here… never touched intrinsics in Rust before so I really appreciate you writing it up!
See the following pdf for example on how to do this with SSSE3 (pages 104-133) or even SSE2 (pages 151-173)
https://deplinenoise.files.wordpress.com/2015/03/gdc2015_afr...
This is interesting. So at a certain scale, CPU optimization becomes irrelevant because you're just waiting for new data to come in?
I heard there's a way to arrange code such that the compiler can autobectorize easier. I wonder if there's a way to do that here?
Would probably have to pass `-C target-cpu=native` to cargo so that llvm is allowed to use AVX512.
Good question. I personally doubt that the compress instruction is easy to coax compilers into generating, as there are many edge cases to consider.
For example, you'll notice here that we perform a full vector store of 8 elements unconditionally, even if only a few of the elements are active. This is safe, though, because the output buffer is as large as the input buffer, and we're chunking by 8, so we'll never trash memory past the end; but this is a tricky analysis. Performance-wise, we rely on the CPU's store buffer to make these overlapping stores cheap.
Instead, you might think that you could just store the elements which are actually active, using a masked store. In fact there is also an intrinsic for this purpose (_mm512_mask_compressstoreu_pd), but it is extremely slow on some CPUs, namely Zen 4, so it's dangerous to use unless you know exactly what CPU you're using. (In my testing, there also seems to be some weird hazard on Zen 5 where multiple memory-destination compress instructions to nearby, even non-overlapping, addresses are serialized. But I haven't looked closer at this.)
I think WUFFS "iterate loops" might help. WUFFS requires that processing a chunk of N items has code to process one at a time, which means it'll work for any N. However you can optionally provide specialisations for doing K at a time and the compiler is responsible for carving the input up as appropriate so e.g. N = K + K + 1 + 1 + 1 your K-at-a-time code runs twice, the extras are handled 1-at-a-time.
So this divides up the problem, the compiler can vectorize your 8-at-a-time code without needing to handle edge cases where N isn't a multiple of 8, and if a later pass notices we actually never end up using those edge cases they're dead code, if it doesn't they're just a rarely-taken branch once.
I believe the bounds check, in particular, is devastating for autovectorization. There are ways around it, but it requires additional code in safe rust.
Edit: actually looks like autovectorization is in play here [1]. Doesn't look like the bounds check gets in the way at all.
There's no autovectorization there; scalar f64-s just are always stored in xmm registers. And the bounds check is still there.
Compress patterns aren't recognized by any open-source compiler autovectorizer as far as I'm aware of. (I think intel's proprietary C/C++ compiler can?)
Thank you for sharing this. How would you emulate this kind of operation on avx2?
Once you have a mask of the positions you want to compress, you can generate a shuffle index vector from that mask to place the desired elements in the low part of the vector. You can expand the mask into nibble-sized indices using pext/pdep and some magic constants, then expand those nibble-sized indices into a vector of indices to use as the shuffle indices.
Yes, that's one approach. Another reasonable approach is to get out a mask from the comparison using `vmovmskpd` and use that to look up a shuffle constant, since there are only 16 possibilities. This also works well on NEON, although I wonder there whether it'd make more sense to find the shuffle dynamically rather than loading it.
Keep in mind it's UB to be:
> Executing code compiled with target features that the current thread of execution does not support
I.e. calling AVX512 on Neon architecture.
You need to wrap it in target attributes to even dream of it being safe.
This particular UB is not one of the subtle cases. You will almost certainly get illegal instruction signals if you mess this up.
Is rust UB different from C UB? C UB must be avoided at all costs even if you think you know the actual behavior.
This article is 100% AI written. The data was interesting, the commentary overly verbose and hard to gain useful insights from.
I'm apparently not good at spotting it. I was put off by the overly dramatic presentation. It gets tiring that the author apparently finds this more exciting than I do, and writes like it's enthralling. I just assumed it was an excess of enthusiasm or the first experience with this kind of thing. If it's AI, I'm way behind the game noticing it.
There are lots of indicators in this text, and this breathless presentation is very much how modern LLMs present things.
Claude also loves to describe things as being "real", particularly saying "X is real".
In this case,
> The reallocations were real, but they were never the bottleneck.
There was never any indication or setup in the text that they weren't real, but it's how it justifies wasted effort, it insists that some phenomenon it corrected but failed to solve the problem "was real".
Another giveaway are nonsensical analogies:
> The predictor is like a barista who starts making your usual order the moment you walk in. If you are a regular, this is fantastic: the coffee is ready when you reach the counter. If you order something random every day, the barista keeps pouring drinks into the sink.
If you order "something random every day", then you don't have a usual order for them to be making, it's an analogy that doesn't work.
And of course, the smoking gun is:
> The smoking gun
It probably won't be a good indicator forever as it has been noticed so much, but it's a particular favourite of the current generation of anthropic models.
"The smoking gun" is right there in the text ;) (but also things like "Same million floats. Same threshold. Same function."). Don't know if other models have that same specific style, but it looks very 'claude-y'.
I wouldn't be surprised though if (especially) non-native speakers unconsciously start adopting the Claude writing style when they stare all day long at Claude generated text at work.
Not sure if it was added later, but there's an even more obvious sign: the top of the blog post literally says that an LLM was used to write it.
I would assume non-native speakers talk to Claude in their own language. Now I'm curious if Claude's weird quirks of speech are unique in each language or if they carry over!
I'm German but still do all my computing in English (partly out of habbit, but also because Germanized technical text is usually painful to work with because it's full of anglizisms (is that actually an English word?)).
One nice thing about English is you can just make up words with plausible etymological roots in Latin/French or Old English and often people will know what you mean. In this case though it would probably be spelled "Anglicism".
This isn't a necessarily true assumption. I my social circles of non native English speakers, most of us use English to talk with models.
I switch languages randomly.
What do Claude's tics look like in your other language? Do they carry over or does each language get its own weird rhetorical flourishes?
> Don't know if other models have that same specific style
An awful lot of the open-weight models also talk in the Claude-y style. Not sure if an artefact of distilling anthropic models, or just a preponderance of slop in the training set...
> I was put off by the overly dramatic presentation. It gets tiring that the author apparently finds this more exciting than I do, and writes like it's enthralling.
That's one of the main tells that AI wrote this. All the stylistic tics that people usually point out combine to make the writing seem more important than it is.
Yep, “instinct“ and “smoking gun” are LLM favourites.
I find it really annoying when the LLM says “good instinct” as if I’m an animal barely able to think.
It was fun to read and insightful for me, not too artificial, and not too verbose.
I'm glad my internal AI detector doesn't win over my curiosity to learn.
The main problem with the article is that the idea to influence backend code generation decisions via specific highlevel code constructs is mostly just mystical bullshit (some compilers do detect specific patterns - usually for bit twiddling hacks, but not on a basic level like control flow optimization).
Using a highlevel language construct like "y += (x > 0) as usize;" doesn't "switch on" branchless code just because the source code looks branchless, compilers are not that dumb anymore.
E.g. I bet that writing
if (x > 0) {
y += 1;
}
The only reliable way is to mostly bypass the optimizer via simd intrinsics, or drop down to assembler, everything else is just cargo culting.
(fwiw I can't shake the feeling now that the article is recycled, I'm pretty sure I saw those exact same code examples in another "branchless" blog post, but maybe for a different language - because the next question was ineviatably "then why is the code using "if" slower? answer: because it also behaves differently). Or maybe I'm just having a strong dejavu ;)
PS:
> fwiw I can't shake the feeling now that the article is recycled
Ok, I remembered wrong. The article I remembered was this: https://tiki.li/blog/blqsort
HN link: https://news.ycombinator.com/item?id=48375445
It's peddling the exact same myth though.
I just scrolled through & read the code snippets. Interesting enough solution at end
Click on their blog index page, see posts going back to early 2010s and use the same writing style. He must have been time travelling and using AI all this time!
I had a look at their blog page out of curiosity, not that you can prove much from the purported dates and text on a blog, which could be edited at any time.
The blog posts from 2010s are in a completely different style and written by a human: https://www.greyblake.com/blog/vim-preview-plugin/ https://www.greyblake.com/blog/how-to-install-firefox-icewea... https://www.greyblake.com/blog/unexpected-ruby-behaviour/ ...
This new blog post is clearly AI edited (probably 'improved' with AI), the old ones are not.
The first link you cited[0] shows up on the Wayback Machine[1] for the first time on 2022-05-16 - more than 11 years after it was purportedly written.
[0] https://www.greyblake.com/blog/vim-preview-plugin/
[1] https://web.archive.org/web/20220516225844/https://www.greyb...
> the same writing style
It's "fake corporate enthusiasm" style. LLMs were just trained in it.
I think he's asked it to write in his specific style, or possibly he has edited parts of it to his style, or maybe used AI to generate the initial draft.
Something like that anyway. There are some very clear AI tells (smoking guns if you like), but most of it does not read like the prose AI produces by default.
Author if you are here I am curious about your writing process, and why you didn't remove the obvious AI tells.
idk why this is getting downvoted, I also got this sense, plugged it into Pangram and indeed, 80% AI-written score.
I guess that's fine, but after awhile I get a spidey-sense reading something that feels like a Claude session.
Sad to see you getting voted down. But I guess both the pro-AI crowd and anti-AI crowd hate Pangram.
I always got voted down when I posted the evaluation of the parent articles I got from my Ouija board. I just want to help people understand whether they should just reject bad articles, without having to bother reading them.
I'm moving on to evaluating articles with a modified lie detector test and tarot cards, I'm sure that'll help my credibility and give my public rejections more authority.
Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong?
1: https://bfi.uchicago.edu/insights/artificial-writing-and-aut... 2: https://arxiv.org/pdf/2501.15654
A good part of the article feels like it was written by Claude indeed:
- "The reallocations were real, but they were never the bottleneck."
- "Note that the villain is not the branch itself. It is the branch that [..]"
- "Same million floats. Same threshold. Same function."
- "Notice the price we paid though."
Agree.
Interesting topic but why destroy your own credibility and reputation by shoveling llm-assisted slop to us here at hn?
The post should be flagged, and in general, i wish hn would adopt a no-tolerance policy to enhanced posting like this.
So what if the original text, if it existed in a human written form at all, had weird textual quirks and prose issues the author wished to hide. That texture's what makes humans interesting to engage with in the first place.
I think I need to build “Hacker News Except All Arguments About Whether Or Not Something Is AI-Written Are Filtered Out”
A much clearer article from yesterday on making casefolding 15x faster by removing an if:
https://github.blog/engineering/architecture-optimization/do...
Discussion: https://news.ycombinator.com/item?id=49127983
Based on the title, I knew the issue as soon as I looked at the first table. Still, great primer for those who don't know about such CPU shenanigans, and I did appreciate the solution, since I knew high level how to solve it, but didn't come up with an actual piece of code before the author presented theirs.
I didn't know about branch prediction or pipelined CPUs back when I was profiling the code I wrote - honestly it probably would have helped.
I've written something like this in C - including resulting assembler code for ARM and x86.
Great explanation of why a branchless approach results in such a speed up. I've never really had to deal with performance optimization at this level. Generally it's probably best not to get too involved letting the CPU black box do its thing.
I do wonder, would the performance characteristics of branchless vs branching be consistent across different CPUs/architectures? If you had a CPU that wasn't trying to be fancy with branch prediction, would the regular algo be faster?
CPUs aren't black boxes. They are actually much better documented than almost all the software that runs on them.
If you want to treat the CPU as a black box, trust me you do not want to use a CPU with out a branch predictor, your slow code will run like molasses frozen in antarctica.
The regular algo will be lightyears slower on any CPU that does not have a branch predictor.
Virtually every CPU has branch prediction, going back to at least the original Pentium (1993), maybe earlier.
If you're running on a very old CPU, yes, the regular algo should be faster.
I think the Pentium is more or less the first microprocessor with branch prediction. Certainly the most mainstream.
PowerPC 601 arrived at more or less the same time, and the Alpha 21064 was a year earlier. There were a few minicomputers and mainframes before that with branch predictors.
Arguably the 486 could have done with a branch predictor (even a single entry loop predictor would have helped), and maybe the 386 too. But microcoded CISC designs didn't benefit much from predictors because they have multiple cycles to work it out.
And RISC cpus were in their "branch delay slots are awesome" phase throughout most of the 80s. With a bit of trickery (very simple branch conditions and a 2 phase clock), your classic 5-stage MIPS design can fully hide all branches with just a single branch delay slot, so they were a little slow to adopt predictors.
I get the impression that CPU designers in the 80s and early 90s massively underestimated just how beneficial even a small predictor can be.
Pentium was the first CPU with a branch predictor that many people could afford to buy.
Before dynamic branch prediction, where the prediction for every branch is updated after each branch execution, depending on its history, static branch prediction had been used for decades, since around 1960, typically using the rule that forwards branches are unlikely to be taken, but backwards branches are likely to be taken. An alternative was to have an instruction bit where the compiler stored its prediction about the probability of a branch being taken.
Dynamic branch predictors began to be used since the mid seventies.
I do not remember now if any notable monolithic CPU had a dynamic branch predictor before Pentium, but prior multi-chip CPUs certainly existed.
> I get the impression that CPU designers in the 80s and early 90s massively underestimated just how beneficial even a small predictor can be.
It's got a lot to do with how cpu clock speeds were getting way faster, but ram wasn't. That's what makes deeper pipelines attractive, and if you give a cpu a deeper pipeline, it's gonna want a good branch predictor.
I'm more thinking about how MIPS were quite late to branch predictors.
They were shipping the high-performance R4000 and R4400 with 8 stage pipelines and no branch predictors.
They could have really done with a branch predictor, each branch took three cycles (and the branch delay slot could fill only one instruction, often a useless NOP).
The Pentium only had a 5 stage pipeline and massively benefited from its branch predictor. IBM was slapping branch predictors on PowerPC designs with 4 stage integer pipelines. You simply don't need a long pipeline to justify the benefits of a branch predictor.
Cortex M0 and microprocessors generally do not. Cortex M3’s looks nothing like the branch prediction you think of when you think consumer or server CPU. Basically branch prediction requires extra power so it’s excluded or greatly simplified in low power use cases.
I'm not sure how your intuition can be that off, if you don't have a branch predictor then any branching code is going to be even slower than it already is, favouring branchless code even more for obvious reasons.
I say this as someone who is interested in a special type of processor architecture that has no branch prediction at all and would need a branchless subset of Rust to meaningfully program it at high performance.
Why no branch predictor at all? Even a brain-dead one that predicts all branches always/never taken is going to provide some benefit, it's not as if the processor can do anything else while it's waiting.
Or am I missing something?
I note the hazard 3 on the pi Pico rp2350 only predicts a branch if it's the last branch and was taken, ie a single loop. Which seems weird to me, so I'm probably lacking understanding somewhere.
Another recent story from github about case folding as part of code search, the simple version of the code had a couple of ifs, and the branchless version was actually slower.
They have a stupendously fast version and it is also branchless, but it just required more than branchless alone.
I'm fuzzy on the details but I think one of the ifs was an early exit, and without that the loop does a memory assignment on every byte instead of skipping most.
The really fast version was also vectorized. The branchless makes it possible to vectorize, but it was the vectorization that actually made it fast.
I really hope all these guns give up smoking sometime soon...
I like how we have pretty much established how branchless coding is superior to branched coding. However I wonder if the compiler itself could recognize these patterns and turn branches into branchless instead, rather than making the code harder to read? as removing if conditions of course have a readability impact on the code.
Branchless coding is superior to branched coding whenever the branches are more or less random, which happens frequently when checking some properties of input numbers, like their sign or whether they fall inside certain intervals, or when sorting an array that comes in random order.
When a branch alternative will be taken much more frequently than the other, then branched coding with an "if" becomes superior.
So neither is better in general than the other, whenever the program must choose between alternatives, you must think about whether one is more likely than the other, or if both have similar probabilities.
For instance, when sorting an array, the optimal algorithm is not the same when you expect the input array to have a random order and when you expect it to be already almost sorted.
This is true, but the example given yesterday showed that even if branches can be very well predicted (e.g. processing UTF-8 text which is 99.9999% ASCII), branchless code can result in speedup by making autovectorisation possible.
If the branchless code didn't transform to vector instructions, it would be strictly slower. But if it does, it allows the CPU to work on 16 bytes at a time instead of 1 at a time.
https://github.blog/engineering/architecture-optimization/do...
For sorting, conveniently we always definitely need to look at all the elements at least once anyway, so although even the early introspective sorts from the end of last century aren't designed this way both the Timsort and a modern sort like a PDQ sort will end up making that decision early.
"Oh, this was mostly already sorted, done"
If you meant exactly rather than almost then you can still squeak a small win from having an algorithm which is optimised for this case but the vast bulk of your runtime is eaten by the unavoidable work of checking. "Don't check" is faster but then you're not a sort algorithm at all.
> The predictor is like a barista who starts making your usual order the moment you walk in. If you are a regular, this is fantastic: the coffee is ready when you reach the counter. If you order something random every day, the barista keeps pouring drinks into the sink.
I laughed out loud reading this. Interesting writeup. I wonder what kinds of tricks like this exist for computation graph compilers like JAX.
I've been doing leetcode in Janet in a (sometimes) tacit (variabless), branchless way:
(def find-shared-gcd
(comp
(fn [e] (max ;(map (fn [d] (* d ;(map |(- 1 (min 1 (mod $ d))) e)))
(range 1 (+ 1 (min ;e))))))
|((juxt* max min) ;$)))
(defn max-diff `where elements increase` [& numbs]
(reduce max
-1 (filter |(< 0 $) # strip 0s and add -1 in case (= true (apply > numbs))
(map - numbs (accumulate2 min numbs)))))that's a really clever trick to write to out[n] multiple times but only move the index after the logical condition is true thus ending up with the correct values in out
Branchless code can indeed sometimes be slower than conventional one, but in this particular case, the article comes to the wrong conclusion. At a 1% kept, the branchless version is slower because it pays the cost of zero-initializing 8 MB of memory when allocating the Vec. This can be easily demonstrated by comparing it with a version that allocates uninitialized memory.
Did you read the article? The author was trying to even out the test cases and successfully did so. He stated that the idiomatic code was faster for the 1 % as you can see by quote below:
The worst case became almost 4 times faster. And look how flat the branchless column is: the running time does not depend on the data anymore, exactly as we wanted.
Notice the price we paid though. At 1% kept the idiomatic version wins, because an almost always correctly predicted branch is nearly free, while the branchless version always pays for one million writes. Branchless code is not faster in general: it trades the best case for the worst case.Even though you used AI to help you with the article, knowing that performance trick is really cool.
Worth noting that as written the "trick" results in memory usage proportional to the size of the input rather than the output. If the filter rejects most of the input the difference could be quite noticeable.
I disagree in the sense that you can rewrite the code to use the trick and also not allocate in advance. Nothing about the trick requires you to allocate up front: before writing to out[n] you can extend the vector if it’s out of bounds. Or, after incrementing n, do out.push(0).
You should try writing it out. Doing it without introducing another unpredictable branch is harder than it looks.
I discussed this with a coworker earlier this week and the best they were able to come up with was
for &x in input {
out.push(x);
n += (x > threshold) as usize;
out.truncate(n);
}
> The reallocations were real, but they were never the bottleneck.
Why do people not write their own blogposts on their own?!
"the smoking gun"... Mr. AI.
Branchless optimization is such a clean way to squeeze out maximum performance for hot loops. Love this approach.
Thanks for sharing, optimisations like these are what keeps the fun in programming. I have been optimising my JSONLogic evaluator in rust and used arena allocator and preallocation tricks that gave me good jump in tuning. Let me see if branchless programming techniques can get any further in my case
This problem is called stream compaction and there is a wealth of research on it. The best methods use prefix scan. They first efficiently compute the index in the output array of each element that satisfies the predicate and then they gather them in one linear operation.
Also, I can tell that you are a good writer. You didn't need the LLM to "polish" your text.
There is clearly LLM-prose involved, but it's pretty well done. Here's one example: "The reallocations were real, but they were never the bottleneck."
LLMs love this pattern. Whether one put it into this text, or the author soaked it up and now used it himself, who knows. But it is one of the few things in the post that gives me the ick.
And then, there's the verbosity.
If I had to guess, an LLM was involved, but the author did a good job with manual writing and editing, too.
If your objects are large, I can see why you would compute indices first. But why do that for floats?
Because what makes stream compaction challenging is the loop carried dependency: the location you write to in a given iteration depends on the locations you wrote to in previous iterations. By first creating a lookup table of source -> destination locations you remove the dependency. Then you can apply extremely efficient parallel methods.
This is good technical content, but it's obvious that an AI wrote it.
Would PGO figure this out?
Generally most forms of PGO does not try to capture number of mispredicted branches (which isn't the same as how often a branch is taken).
They could.
but.... running PGO is just too much pain.
We can't do it "incrementally", can we? How about combining with LTO?
edit: I was thinking profiling individual module on a test driver and link them after PGO
I don't know if an optimization is allowed to "invent" a write, but I would be surprised if an optimizer goes that far because I have to believe that the number of cases where more writes improve performance are pretty slim.
a simple perf stat should show that the "Keep 50% of random data" case will have insanely more branch mis-predictions that the others.
Keep in mind that this is a synthetic benchmark. The task is to remove outliers and for that the roughly 99% correct guesses of the branch predictor is perfectly fine.
This is a common pattern a compiler should recognise and optimise into an efficient data and control flow. So much for a sufficiently smart compiler. shrug
"A branch is cheap. A mispredicted branch is not."
oh hell nah
[dead]
[flagged]
[dead]
[dead]