Almost Always Unsigned
graphitemaster.github.io51 points by gavide a day ago
51 points by gavide a day ago
I have a better solution: address the root cause of unsafe semantics by not using raw indexed for-loops, unless one absolutely needs an index, in which case one should generate it with std::views::enumerate. To reverse it, use std::views::enumerate | std::views::reverse. Ditto for languages with similar semantics.
I almost never write a raw for-i loop any more, especially since 99% of the time I want to enumerate through the entire array or vector, and I can just use a ranged for-loop to do that. It allows me to redesign my code around the data, express things at a higher level, and my code looks far more SIMDable and reminiscent of array programming languages. And yet it is safer, I will never see under/overflow or any of these old-hat problems.
If you are using C, then too bad, you're stuck with a language that doesn't allow the programmer to more meaningfully and more clearly express intent at a higher level of abstraction without paying additional runtime costs.
This stuff compiles to broadly the same assembly.
> 0x7ffffffffffffffff. The typical argument is that such a value would be “pathological”. Not only is this argument incorrect, it’s even more dangerous which we will see later.
The only later thing I see that's somewhat relevant to that appears to be dealing with 32 bit overflow? That doesn't prove the argument incorrect.
No current OS I'm aware of lets you have a size_t that goes over 2^63. I doubt any OS will ever allow it. It makes things easier if virtual memory is capped to 2^62 or so, and if anyone really ends up with a use case for more I expect them to switch to 128 bit numbers.
While I'm a fan of unsigned (size_t mostly) there have been a few times when the tax for converting them to float was shockingly high:
https://godbolt.org/z/96T4jTshc
1-2 instructions for signed vs 11 including a branch for unsigned.
(in times like these I found casting to signed first preferable)
The article over-downplays the need for sentinel values. Surely Rust has Option that also spreads into C++ these days as std::optional. But for plain C using -1 or negative values to denote sentinels or error code is rather nice idiom.
The argument will be more valid if array indexes will be 1-based like in Fortran/Matlab/Julia as then 0 becomes extremely nice sentinel values. But C is C and needs -1.
Not disagreeing about the need for sentinel values, but Rust also has NonZero. When combined with Option, you get a similar result to outcome to C's -1.
Similar because it's a compiler hack, as in the compiler treats std::num::NonZero specially. You can't create your own type with the same properties as you can in C.
In a language that has arbitrary precision integers, you'd pretty much never want them unsigned, or even to have signed and unsigned flavors.
Whether unsigned or signed is better is a matter that is a combination of personal opinion and the quirks of a given systems programming fixed integer language.
The trade-off reasoning would be different, for instance, in a language that requires implementations to provide two's complement signed integers, with wraparound semantics. Or, say, no wraparound semantics but a robust overflow detection system coupled to exception handling.
There are other matters beside overflow, like conversions. In C, mixtures of signed and unsigned bring in some implementation-defined conversion rules, which nudges the argument toward "all unsigned" or "all signed" for the sake of avoiding mixtures.
I like to trot out the following argument.
Suppose a, b and c are small integers close enough to zero that any additive/subtractive combination of them is free of overflow.
If they are signed, then we can make inequality derivations like
a + b < c
b < c - a // subtract a from both sides; "bring to other side"
I believe the main issue lies in most programming languages lacking theorem proving capabilities to prove the safety of integer operations.
The safety conditions for unsigned arithmetic:
Ensure y+x ≤ INT_MAX.
If x ≤ UINT_MAX-y, then x+y evaluates correctly:
∀x∀y(x ≤ UINT_MAX-y → ∃z(z = y+x))
Ensure y-x ≤ INT_MAX.
If x≤y, then y-x evaluates correctly:
∀x∀y(x≤y → ∃z(z = y-x))
Ensure INT_MIN ≤ y+x and y+x ≤ INT_MAX.
To avoid overflow or underflow, first compare x to 0.
In the case x≤0, INT_MIN-x cannot underflow, and y+x cannot overflow. If y compares greater than INT_MIN-x, then y+x evaluates correctly.
In the case 0≤x, then INT_MAX-x cannot overflow, and y+x cannot underflow. And if y compares less than INT_MAX-x, then y+x evaluates correctly.
∀x∀y((x≤0 ∧ INT_MIN-x≤y)∨(0≤x ∧ y≤INT_MAX-x) → ∃z(z = y-x))
Ensure INT_MIN ≤ y-x and y-x ≤ INT_MAX.
To avoid overflow or underflow, first compare x to 0.
In the case 0≤x, INT_MIN+x cannot underflow, and y-x cannot overflow. If y compares greater than INT_MIN+x, then y-x evaluates correctly.
In the case x≤0, INT_MAX+x cannot overflow, and y-x cannot underflow. If y compares less than INT_MAX+x, then y-x evaluates correctly.
∀x∀y((0≤x ∧ INT_MIN-x≤y)∨(x≤0 ∧ y≤INT_MAX+x) → ∃z(z = y-x))
Stroustrup recommends int over unsigned. Dijkstra recommends int over unsigned. Google coding guidelines recommend int over unsigned.
Blogger recommends unsigned over int.
Tough choice.
I liked how the discussion of 'delta = x - y' moved right on to how really you usually want delta = abs(x - y), so let's talk about that instead...
Even beyond Stroustrup, Dijkstra, and Google, this whole panel of C++ luminaries agrees to prefer signed types and explains pretty clearly why:
- 12:12-13:08 - https://www.youtube.com/watch?v=Puio5dly9N8#t=12m12s
- 42:40-45:26 - https://www.youtube.com/watch?v=Puio5dly9N8#t=42m40s
- 1:02:50-1:03:15 - https://www.youtube.com/watch?v=Puio5dly9N8#t=1h2m50s
Thanks for the excerpts! I was trying to understand the reasoning, which seem to just be in the 2nd excerpt:
- The rules of signed/unsigned are complicated and there is too much auto-conversion - does that mean languages that make this more explicit means this is fine? It just seems ideal to have stronger typing. - It is mentioned that you can initialized an unsigned int to "-2" - but that presumably could also be fixed in the language.
I'm trying to separate out which is "don't do this in C/C++" and which is "don't do this in any language".
that argument (and many others) also round to essentially "implicit imprecise integer casting is surprising but we do it anyway" which is... perhaps the actual problem???
I've made lints for Go that simply disallow implicit number casting, and omfg the (real, occurring but unnoticed) bugs it found. those kinds of lints are trivial to build, you can just stop doing it. forcing visible casts made many of these problematic patterns extremely suspicious at a glance, catching issues at review time far more easily.
I believe there's ways to configure clang to flag dangerous implicit casts as well.
-Wconversion perhaps: https://clang.llvm.org/docs/DiagnosticsReference.html#wconve... or -Wimplicit-int-conversion for the main check I've built in other languages (afaict, I have not used C(++) professionally)
Considering Rust doesn’t have such nonsense, I’m inclined C/C++ have utterly broken generations of programmers.
In what world is using a signed value to index a normal array a good idea?
Makes for horrible footguns like:
history[counter % SIZE] = …
(One cursed day counter rolls over, becomes negative, and an out-of-bounds write occurs)
Everything went South as soon as we broke the abstraction of arrays and treated them as pointers.
Commenters here are pretty much arguing which way to hold scissors while running instead of realizing that one shouldn’t do that in the first place…
> Makes for horrible footguns like:
> history[counter % SIZE] = …
The footgun here is that the “modulo” operator does not actually calculate the modulo in C. In Python, this works correctly for negative values.
> In Python, this works correctly for negative values.
They’re both “broken” in different ways. Arguably C’s brokenness is more apparent and less useful but Python also has footguns: C uses truncated division for its “modulo” so the remainder has the sign of the dividend, Python uses floored division so the remainder has the sign of the divisor instead.
The wiki page for modulo has a pretty extensive page on the subject.
I say you want a pair of operations such that (a, b) = quotient_and_remainder(x, y) gives you a and b such that a * y + b = x
The Euclidean division and remainder work, the other division and remainder also work, and they're both identical for the positive integers so people who only think about the positive integers won't even notice there's a choice here. So I like that Rust provides both pairs, in the same way Rust provides both Wrapping<T> and Saturating<T> because maybe you mean wrapping overflow or maybe you mean saturating overflow and we should make you choose not just assume we know best.
> Python uses floored division so the remainder has the sign of the divisor instead
Serious question, how is that a footgun? In decades of software development I have never needed a negative divisor for modulo. What would you use it for?
Commenter implies authority dictates strategy.
We should be engaging with the article's content.
Actually, in that case, no.
Blogger hasn't bothered to refer to those well known and detailed opinions, from very experienced authorities, and provide detailed rebuttal to those authorities claims, so his opinion can be safely ignored.
Commenter comments before reading.
The entire article literally starts by referring to Google coding guidelines and other well known opinions as arguments against unsigned, then tries to provide a rebuttal.
ClickHouse code style recommends unsigned in every case when you don't need the sign: https://clickhouse.com/docs/development/style
signed overflow (or underflow) is frequently undefined behavior. (often because it's undefined in C)
unsigned is frequently defined. (often because it's defined in C)
tough choice.
(honestly I just lean towards "over/underflow should raise unless explicitly allowed", the ratio of unintended to intended-and-fully-checked overflow behavior is almost certainly FAR beyond 100:1)
Of course unsigned is defined. That's besides the point. The point is: how often in your code, do you expect 1 minus 2 to equal a very large number, vs. the number -1.
both seem equally undesirable to me in all cases where I intend neither. though one also risks undefined behavior, so that is strictly worse.
the reason I use a type system is to make error classes unrepresentable (where possible) or a failure. these are both leaky abstractions in the worst possible manifestation: silent misbehavior at runtime.
> Where unsigned does benefit here is when these are used as indices into an array. The signed behavior will almost certainly produce invalid indices which leads to memory unsafety issues. The unsigned way will never do that, it’ll stay bounded, even if the index it produces is actually wrong. This is a much less-severe logic bug, but can still be used maliciously depending on context.
The argument for signed often goes that it's easier to detect an invalid operation on an index by checking for bounds, or the higher probability of segmentation faults, than if your index is always “valid”. Granted, even with signed your invalid operation might still give a valid index. Simply put, seeing a negative index in your debugger is an obvious red flag you lose with unsigned.
In general I want to complain about the idea that a subtle bug is less severe than an obvious bug. Obvious bugs can be caught automatically or manually and are therefore less severe than subtle ones. This is a mistakes all students do btw. Segmentation faults are your friends!
> btw. Segmentation faults are your friends!
Indeed. I can't count how often I told people that segmentation faults are among the best kind of errors. One has the complete state and not a log message missing 90% of relevant info.
Of course you don't want processes to crash all the time and spend time on writing gigabytes of data onto disk. But for the rare failure it's a good thing to have.
What if I’m making a 2d game and need to go left
As a Java developer I'm a bit sad that we don't have a choice :)
You can count down to zero with while(i--)
There was a related article from the other side of the debate a while back: https://news.ycombinator.com/item?id=47989154
It’s pretty sad that after all these years, dealing with fixed size integers is still so complicated. Yes, many of the problems are specific to low level languages with undefined behaviour and numeric for loops. But the issue of subtracting two numbers and possibly having an underflow is both common and a bit absurd.
The code in the article for “safely” calculating the difference of two unsigned numbers, which is simpler than the equivalent for signed integers, is this little ritual:
> delta = max(x, y) - min(x, y);
Seriously??? Two function calls just for the difference of two numbers??
Why can’t such “safe” operations have some of the sweet syntactic sugar, and the underflow-rampant “ordinary” operations have the bitter medicine of ritual?
For the stuff I do, I rarely use signed types. I find the example of finding difference between two numbers to be really silly. I don’t recall ever having to do anything like that, for a simple reason: I design my code so that I always know which number will be larger than the other.
Let’s actually talk more about this max - min example. Let’s say you calculated this delta. What is it useful for in your code? Typically, in real algorithms, what you want is to not only know the delta, but also which number is larger.
> I design my code so that I always know which number will be larger than the other.
That’s good discipline, but with better tooling and languages such conditions could be enforced by the compiler such as with Dafny. But Dafny is overkill for those who only want to avoid underflow and overflow without verifying everything else about their program…
That’s what I mean when I lament the state of affairs for underflow. Subtracting numbers is such a basic operation, but for the tech stacks used by most teams it still requires a level of care to avoid errors or corruption. Our industry is building million line behemoth codebases but these basic operations still have to be designed around.
> Typically, in real algorithms, what you want is to not only know the delta, but also which number is larger.
I agree with you, and the article was more concerned about avoiding underflow than actually having a practical use for its various example idioms.
What I’d actually prefer is a guarantee that subtracting two (X) bit numbers results in an (X+1) bit number. A sort of compile-time-checked bignum implementation where the programmer never thinks about the integer sizes or which number is larger until the final step of a calculation. Obviously this gets wasteful when multiplying numbers several times. EDIT: and I’m referring to an abstraction of integer bit sizes in a programming language, I’m not expecting the instruction set or hardware to have arbitrary sized registers.
I mean, Rust calls this function you want abs_diff
let a = 1234_u32;
let b = 5678_u32;
let c = a.abs_diff(b); // Or equivalently u32::abs_diff(a, b);
Edited: tweaked language
The issue is that you need dozens of minor variants for the myriad cases involving integers, especially in C. That namespace becomes rather busy.
This particular idiom for finding the absolute difference of an unsigned integer pair is pretty clean and rarely needed. In most contexts you know that one argument will always be greater than or equal to the other, so a simple subtraction will do.
> for (size_t i = size - 1; i < size; i--)
Erm... just because you can, doesn't mean you should.
Also, what if you want to go down to something other than 0?
Post-increment inverts to pre-decrement, but for-loops don't support proper syntax sugar for pre-decrement.
for(size_t i = 0; i < size; i++){
// loop body
}
for(size_t i = size; i > 0;){ i--;
// loop body
}> for (size_t i = size - 1; i < size; i--)
Agreed, seeing that example briefly made me consider whether this blog post was a parody. Sure, it works for this exact example, by relying on i wrapping "down" to MAX_INT on the last iteration. But how long will it take the next developer who works on the code base to figure that out? Will they figure it out before or after committing changes that break it? Or worse yet, before or after shipping code?
I really don't see what's supposedly awful about that loop, but if you want to count down to x instead of 0 you just do:
for (size_t i = size - 1; i >= x; i--)> I really don't see what's supposedly awful about that loop
The stopping condition is incredibly confusing and non-obvious. Misleading at first glance, in fact. The whole thing is so unidiomatic that I don't think I've even seen it once in my life. It's a better contender for an underhanded C++ code contest than production code.
> but if you want to count down to x instead of 0 you just do i >= x
No you can't. That fails if x == 0. Which perfectly illustrates why using unsigned everywhere isn't so great. And I say this as someone who likes unsigned types and uses them more than average!
Thinking of it as a "stopping condition" is backwards, that part of the loop is called the invariant:
https://en.wikipedia.org/wiki/Loop_invariant
You should think of it as the condition that's true for all iterations, not a one-time event that halts the loop. The loop is short for this:
for(size_t i = size - 1; 0 <= i && i < size; i--){
}
> You should think of it as the condition that's true for all iterations, not a one-time event that halts the loop.
(a) It's "a" condition that holds true for all iterations, not "the" condition. Plenty of other conditions can hold true across the iterations of any given loop too. In fact the one interesting thing about the loop invariant compared to any other conditions is the very fact that it is guaranteed to cease to hold immediately after the loop, assuming you don't break in the loop. Other conditions can still continue to hold. i.e. The stopping condition is the entire point of the loop invariant.
(b) I'm well aware what a loop invariant is; I've worked on compilers. I am also a human. Humans care about when a loop starts and stops. There's nothing backwards about it, it's the most straightforward way people think of loops. And it's literally why more modern languages have introduced better syntaxes that merely spell the boundaries and/or values, and skip spelling the invariants entirely.
> I really don't see what's supposedly awful about that loop
Exactly! That's precisely the problem with it.
(Hint: think about your code when size = 0.)
It works perfectly fine for size = 0?
They meant x = 0.
Not sure if they meant that, since the “awful” was referring to the original loop that had no x, but the generic solution in that case is:
for (size_t i = size; i—- > x;)There was a golang proposal to change Go's default int type to arbitrary precision big int. It would avoid overflow bugs, but the proposal was closed (after eight years) due to concerns about compatibility reading serialized data and performance.
as much as I'm not really a fan of Go, and I would love to spend some time in a language that defaults to unsigned ints by default to see just how pleasant or painful it really is, I do think Go makes a reasonable tradeoff here.
signed ints are rather obviously the majority decision, so that default is defensible, and their const / compile-time math is arbitrary precision. so as long as you can describe your math as either wholly const or can move all overflow-risky stuff into a const equation, you're good - Go's compiler will fail if it exceeds the bounds of whatever number you try to cast it to (implicitly or explicitly).
ignoring fun with float rounding, of course.
The entire reason for integer under/overflow to be undefined is to enable compiler optimisations. If you're going to be using unsigned anyway, we might as well drop the undefined behaviour from the standard and just say it's machine-defined. That should honestly be the correct choice. If a loop is hot enough to benefit from those optimisations, you can easily rewrite it in a form that makes the compiler assume overflow won't happen. Either using current syntax with unreachable(), or we can add a runtime_assume(expr) expression that signals to the compiler that it can assume expr is true. Though for full safety I would prefer using if(likely(expr)) {fast code} else {panic or return error}.
As an aside, unsigned does not save you from undefined behaviour. When sizeof(short)==2 and sizeof(int)==4 (e.g. x86, x64, arm32, arm64), then multiplying two unsigned short values happens by upcasting them to ints (see integer promotion rules), which can overflow the int.
My personal opinion is that along with making signed overflow defined, unsigned integers should be entirely removed as a type and there should instead be separate signed vs unsigned operators, because at the processor level there is no difference between the two, and there hasn't been a good case to separate them at the hardware level for the last ~half century. Basically, do what Java does with some syntax like unsigned{expr} which forces all integers inside expression to be treated as unsigned. Unsigned literals can stay, but they will be bitcast to signed equivalents if used outside unsigned context.
Related: I have a variable integer length encoding scheme that beats out LEB128, Protobuf, varint and ASN.1 while also encoding/declaring endianness (but notably, leaving signage information up to the application): https://github.com/pmarreck/BLIP
Should be (2022) apparently - surely if HN can automatically screw up titles for various reasons, we can have it add dates automatically [and sometimes get those wrong] too? DanG ?
> for instance C and C++ leave signed integer wrap undefined
I'm pretty sure the way to write what was meant here is "C and C++ leave signed integer overflow undefined". Wrapping would be a definite choice. Overflow is the situation we're considering, not a particular outcome to choose so that is what's undefined.
The confusion gets worse later when it insists that just like in C or C++ these three Rust expressions will produce invalid results because of LLVM:
x / 0
INT_MIN / -1
INT_MAX % -1
INT_MAX - INT_MIN
It mentions Odin too, I know less about Odin but I believe it too will reject this nonsense, on Godbolt it seems to either SIGILL (for zero) or SIGFPE (for other impossible operations)
Edited: Apparently I copy-pasted wrong? Some of those invalid expressions were not as written on the blog post but are now hopefully fixed. They are, of course, still not invalid in Rust, some panic because they aren't valid questions (like dividing by zero), others are fine - neither case is a problem.
https://odin-lang.org/docs/overview/#integer-overflow
> For signed integers, the operations +, -, *, /, and << may legally overflow and the resulting value exists and is deterministically defined by the signed integer representation. Overflow does not cause a runtime panic. A compiler may not optimize code under the assumption that overflow does not occur. For instance, x < x+1 may not be assumed to be always true.
You've linked Odin's documentation. It's good to have documentation, but, because Matt Godbolt is a nice guy we can just try it out and see for ourselves
https://odin.godbolt.org/z/4WezGr7nK
In Odin, Dividing the 16-bit signed integer -32768 by -1 results in taking SIGFPE. Maybe it's not documented to do that, but that's what happens.
> for instance C and C++ leave signed integer wrap undefined
Not in C++29, and I think the big 3 compilers (gcc, clang, MS) will have squashed this long before that version is officially approved.
> Not in C++29
C++ 29 doesn't yet exist. It'll be finalised in (as its name suggests) 2029. Are you referring to some proposal you think has been accepted? Or is this one of those "I have concepts of a proposal?" ideas where you confused wishful thinking with reality ?
[dead]