Much Ado About None: Sentinel Values in Python
florianwilhelm.info56 points by sebg 2 days ago
56 points by sebg 2 days ago
So much trouble to avoid introducing the optional monad.
Python’s type system has union types (including Option), and using them is pretty ergonomic. As TFA points out the problem is that `None` gets contextually reused in all kinds of places where a proper sentinel type is more appropriate.
Pretty sure Python's union types don't nest, though. If you could nest optionals then I feel that reduces the need for separate sentinel types since e.g., you can distinguish between "value not present" (None) and "null value present" (Some(None))
Indeed. Option[Foo] really just means Foo | None - you can’t have an Option[Foo] at runtime. Fundamentally, Python’s static typing system describes a language that is a sort of mediocre match for Python.
Another fun one is List[Foo]. Runtime Python has lists but you can’t ask a list what type of list it is, and you can’t write isinstance(a, List[Foo]).
Can you explain or give an example of nested optional types? I don't quite see how that makes sense
The idea is that optional values are either some value or no value, but sometimes the inner "some value" bit can itself be an optional value (in Rust this would be Option<Option<T>>, in C++ it'd be std::optional<std::optional<T>>, in Python it'd hypothetically be Option[Option[T]], etc.). Such a situation gives you three possible states:
- You have an inner value and the inner value is something non-null (an integer, a string, another optional, etc.)
- You have an inner value and the inner value is null
- You do not have an inner value
The problem, from what I understand, is that Option[T] in Python is equivalent to the union type T | None, and union types are flattened (e.g., (T | U) | V is equivalent to T | U | V) and duplicates are ignored (e.g., T | U | T is equivalent to T | U). As a result, Option[Option[T]] is equivalent to (T | None) | None, which is equivalent to T | None | None, which is finally equivalent to T | None, which means you lose the distinction between "No inner value" and "Inner value is present but is None".
As a more concrete (albeit somewhat contrived) example, say there's some API you can query for the owner of a particular document ID where the document ID may or may not refer to an actual document (e.g., maybe you got it from a cache and the document has since been deleted). One way to express this would be (using Rust syntax)
fn get_owner(id: DocumentId) -> Option<Option<UserId>> { ... }
(This can probably be done better by creating new enums, but it's an example so give me a bit of slack)
A naive port of this signature to Python would be
def get_owner(id: DocumentId) -> Option[Option[UserId]]: pass
def get_owner(id: DocumentId) -> UserId | None: pass
This seems really silly to me. Why does get_owner need to differentiate between the document exists and maybe has an owner or the document doesn’t exist and thus has no owner. In both cases getting no owner should suffice? Why are you using get_owner to try and figure out if the document exists? Can you give an example where this distinction actually matters?
Maybe a different example works better for your mental model:
A user can set custom settings, and those settings may or may not have a value assigned to them. For example, one setting is optimization level, another option is a debug flag.
You want to have a unified interface to get information about settings, queried through some intermediate system. (lets just say it is limited to querying specific settings, rather than getting the entire list, due to legacy constraints in this case) Because for some settings you just care about them existing and for others you care about the actual value, you have to differentiate between a setting that contains something, a setting that contains nothing, and one that doesnt exist.
> In both cases getting no owner should suffice?
Depends on what exactly you plan on doing with the information, I'd imagine. I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).
> Why are you using get_owner to try and figure out if the document exists?
Potential TOCTOU issues, perhaps? As I mentioned the document could have vanished out from under you so detecting this situation could let handle things appropriately (though to be fair that argument could probably extend to future operations as well).
Maybe a versioned filesystem would have made a bit more sense, where None would indicate a new version is available? Another example I've seen mentioned is trying to get a key-value pair from a JSON where the key may be missing.
But in any case, I did say it was a bit of a contrived example, so cut me a bit of slack :P It was just the first thing that came to mind.
The examples sibling comments give are probably more practical.
> I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).
If you care whether a document exists or not, you should use an API that exclusively tells you that instead of expecting everything related to documents telling you if the document exists or not. Otherwise you’re overloading the meaning. A non existent document has no owner. An API to fetch the owner of the document should not care if it exists or not, it does not matter.
As I said, that sounds like a recipe for potential TOCTOU errors.
But again, this was an off-the-cuff example that I explicitly described as contrived from the start. I could probably flesh it out to make more sense, but I think it'd be easier just to refer to sibling comments rather than bikeshed this to death.