Silk: Open-source cooperative fiber scheduler
github.com91 points by animetyan 4 days ago
91 points by animetyan 4 days ago
It's interesting how some ideas very quickly start popping up everywhere at once. One of my colleagues was working on adding support for fibers to systemd for some time (and the PR was merged a few days ago[1]!). From my understanding, this model is really quite useful for programs like pid1 that cannot use threads.
I've been doing this with something very similar in GNOME for years now. It also does io_uring, threadpool schedulers, workstealing, "pollable semaphores" (like IRIX), profiler integration, and makes fibers implemented as futures themselves.
Looks really interesting. I assume this suggests that ClickHouse is going to gradually switch to using this library for network and I/O, thus addressing the main weakness (in my mind) of C++ thread-per-connection servers, which is, they (surprisingly!) create too many threads and can't really handle more than, say, a thousand active connections at the same time. It mostly matters for async INSERTs in this case of course, not for SELECTs, although generally it applies to both.
Is this comparable to Sea star [0]?
I just took a look a both (thanks for the pointer to seastar! cool project). I'd say "sort of but not exactly". It seems like two approaches to solving the same problem.
Silk seems like mostly a scheduler that uses C++'s native coroutines. Seastar is a future-based framework. Both contain a scheduler and enable async coding patterns.
The `Cilk` angle is interesting. There’s still room for small runtimes focused just on fork/join recursion.
I’ve been working on one for C: https://github.com/xtellect/cactus
It’s narrower than Silk/SeaStar: continuation stealing for CPU-bound recursive code, not a general async I/O fiber runtime.
Looks like its using MinIO for the S3 benchmark, which is now archived. I wonder what they'll switch to (if anything)
Seems not exception safe when task switching during unwind.
Play on Cilk?
[flagged]
[dead]
[dead]
[flagged]
[dead]
those who don't learn Erlang/Elixir are bound to reinvent the Erlang VM and OTP model.
or maybe erlang / elixir do not have 1/1000th of the libraries and features available to C++ thus it's more efficient to port what can be ported? would you rather have the whole of C++ ported to erlang?
Honestly, erlang was designed for telephone use cases and for high concurrency, erlang as an idea is fundamentally really awesome.
And elixir on top of erlang really makes the syntax easier to use. Gleam is interesting too!
Depending upon the amount of concurrency needed, erlang can be absolutely good for the situation.
The only gripe that I might have seen online with Elixir might be that it is functional language (and I think has its own way of doing things) and a lot of the times people start with other languages which can scale to only a certain point.
Either way, Erlang VM/OTP model does feel really cool to me. Fun fact: but Erlang was a virtual machine before it was cool (Java)
I definitely wish to learn elixir someday (don't quote me on that though :-D), just because of how unique Erlang/Elixir are compared to many languages in such sense of scalability.
Although not a 1:1 Apples but I also wish to know more when golang is more fit as compared to elixir as currently, I usually prefer golang as a swiss knife with its decent stdlib,cross-compilation,portability features.
Elixir is pretty easy to learn.
Its syntax is very Ruby and everything is message passing / actors, or genservers.
If you have ever used actors at all it doesnt take much to pick it up and even if you havnt its pretty simple.