Show HN: Is_ready – Wait for many services to become available – 0 Dependencies
github.com38 points by stavepan 14 days ago
38 points by stavepan 14 days ago
"poll every N seconds" for readiness checks is an anti-pattern. The problem is that latency cascades. A dependency chain of M services will have an end-to-end latency of N*M in the worst case and (N*M/2) in the average case, if they all rely on polling to check that the next service is ready.
You should instead rely on backpressure to signal readiness. A service accepts connections immediately, and only blocks the client when the service has to block on something else to procure a response. Systemd takes this approach with UNIX sockets - it creates all the UNIX sockets for each service, and then starts all the services in parallel. The socket buffers queue the messages so that clients can send messages to services that haven't yet started, and the OS scheduler will saturate the available CPU/IO so that the startup of all the services are maximally parallelized.
Obviously, this is not always feasible, especially in cases where you don't control the service(s) you depend on. But to the extent that you do, relying on backpressure instead of polling can markedly improve end-to-end latency.
Zero dependencies seems like a weird selling point, especially since it depends on Clap, Tokio, a Rust compiler...
A neat project, though. I think that this is best solved in the application itself (e.g. your server starts but returns HTTP errors while the database is unavailable), but being able to retrofit this behaviour into any existing application seems useful. Feels like something very similar should be built into tools like docker-compose.
>Feels like something very similar should be built into tools like docker-compose.
In docker compose you can use `depends_on` [0] to define dependencies between containers and by default it will wait until a dependent container is "ready".
But you can also use a more verbose syntax to wait until a container is "healthy", as defined by its own healthcheck.
services:
web:
depends_on:
db:
condition: service_healthy
db:
image: postgres
That's true.
For example, in the case of PostgreSQL, there is already a tool called pg_isready [0] to do this inside a healthcheck as you described.
services: postgres-db: image: postgres healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"]
application:
image: image
depends_on:
postgres-db:
condition: service_healthy
[0] https://www.postgresql.org/docs/current/app-pg-isready.html
Regarding zero dependencies, I meant that it is a static binary and does not need any external tools to be installed. For example, some alternatives require netcat to be installed.
This is fine, and what I understood from "zero dependencies", that I can drop it in a directory and it'll just work. As far as I'm concerned, this complaint is moot.
I took the statement to imply no runtime dependencies, as no compile-time dependencies doesn’t really make sense.
The Linux kernel itself to handle the syscalls is probably also implied in this.
The IMO superior https://github.com/F1bonacc1/process-compose project has this built in, while allowing to manage regular programs that don't require containers.
See: https://f1bonacc1.github.io/process-compose/health/?h=port#r...
`process-compose` is a "scheduler and orchestrator". `is_ready` only checks for services to be ready so it seems great when you want to start the services by other means and wait for them to be avialabe (for example, services started in another node of a cluster).
I have systemd manage my containers and system processes. Are there any benefits to process-compose over just using systemd?
It works on MacOS/Windows, unlike systemd. Therefore it's well suited for development environment setups for polyglot teams.
https://devenv.sh/ is one example that uses it to do just that.
Interesting project, I was not familiar with that. Thanks for letting me know :)
Colloquially a lot of people understand “zero dependencies” to mean “you don’t have to deal with pip/virtualenv/npm/yarn” BS. So, a static binary.
> Zero dependencies seems like a weird selling point, especially since it depends on Clap, Tokio, a Rust compiler...
Came here to make the same comment. 0 dependencies would mean/infer (at least, to me) a lack of any dependencies but that doesn't seem to be the case[1]. There are 30 matches for the word `dependencies`.
[1] - https://github.com/Stavrospanakakis/is_ready/blob/main/Cargo...
When you see someone advertise a tool to put in your containers to do something, why are you immediately thinking "no compile-time dependencies", rather than "no runtime dependencies"?
> When you see someone advertise a tool to put in your containers to do something...
The title claims "0 dependencies", not "0 runtime dependencies" nor "0 compile-time dependencies". In other words, the word dependencies - by itself - is encompassing both runtime and compile-time dependencies.
Think of reading the title at face-value - but as if someone else had written it, with no exposure to your project, its use-case, behaviour, etc. - before it was read.
Someone else did write it, and I didn't have any exposure to the project, use case, behaviour, etc, before I read it.
Ah, thought you were the OP. That's my bad.
The "dependencies" point still stands, though, in that the term - generally - encompasses both.
To be honest, I wanted to make the title more specific regarding dependencies but the title was already too big.
With Docker compose, there is a more idiomatic way to achieve this with zero dependencies using healthchecks[0]. Works well!
I used wait_for_it.sh for the purpose described in the OP until I found healtchecks could be used instead.
[0] https://github.com/peter-evans/docker-compose-healthcheck
Healthchecks are a great way to achieve this.
As this repository mentions, this is the example using PostgreSQL.
depends_on: postgres-database: condition: service_healthy
healthcheck: test: ["CMD-SHELL", "pg_isready"] interval: 10s timeout: 5s retries: 5
However, PostgreSQL has already a command for this called pg_isready.
How is this going to work for other cases such as MySQL?
> How is this going to work for other cases such as MySQL?
You could do a query like SHOW DATABASES as a healthcheck for mysql.
Interesting. And it works for external HTTP services as well, I presume, right?
Alternative: write your applications to not depend on the liveness of other services to start up. Your ops will thank you.
That makes a lot of sense for application servers when one controls the stack and can implement something like exponential backoff, and services are expected to already be running.
However it is not realistic to make sure all library dependencies used behave properly for more lambda-like jobs where a library/app like "wait for" is very useful since it reduces the backoff parameters that ops needs to know about, and reduces unnecessary errors in logs.
Take GCP Cloud SQL Auth Proxy [1] for example which simplifies the details of securely connecting to a database without exposing long-term credentials to app code. Typically for the audience that uses the proxy, it is run as a sidecar container in Kubernetes Pod and has a non-deterministic startup time in the range of a few seconds. There is no "depends_on" facility like in Docker (at least when I had to use it). Your batch job could potentially always fail if the proxy cannot establish a connection to the database before your minimal database migration container starts up. Building the backoff inside the migration app makes it more opaque for ops teams. Having this "wait for" run in a startup script before the main app allow ops to tune its parameters, such as through environment variables, so that the job behaves appropriately under health probes [2] for higher level backoffs. All of which can be configured without recompiling the main app code, or embedding the functionality in the first place.
[1] https://cloud.google.com/sql/docs/postgres/connect-auth-prox...
[2] https://kubernetes.io/docs/tasks/configure-pod-container/con...
Can you suggest to me one way for an API service to wait for a database to accept connections to run the database migrations without the API depending on the liveness of the database to start up?
I think the pragmatic answer is you should wait for your database to be live before starting up. One just wants to lean on the side of depending on a few critical services rather than all potentially-utlizied application services.
Start the server and respond 503 every time when your database and other dependencies are not ready. Couldn’t be easier.
Many people have the following setup in docker-compose.yml file.
version: '3' services: mysql: image: mysql:8.0
app:
build: .
command: is_ready --timeout 10 --addr mysql:3306 -- <run migrations command>
Like other say easy to fix this with bash alone. But maybe if you add things like also waiting on other things it can increase your value u a little like also waiting on files, sql queries it can become a pocket knife of waiting for things? Albeit probably people will start saying it will do too much :)
Ooh they could even make it a multi call binary like busybox! I imagine without it, some kind of mode flag would be required.
Sometimes waiting for the address is not enough. For example, you might want to wait for a specific API enpoint to return 200 OK, or you might want to wait for database migrations to get applied. Is `is_ready` aiming to cover those cases in the future?
Interesting idea for a feature request. I would love to read a GitHub issue with your specific case so I can start investigating and think about how to solve a problem like this.
I considered starting to write myself a Rust "wait for it" app myself this past week with more features than the standard one. Not sure how I didn't come across yours during my initial search.
There could be some useful functionality that could be found in startup and health probe topics [1] in Kubernetes for example. But at least in that world, it may be easier, and more transparent, to plug in a startup bash script to run before the main app. For example the Postgres Docker image [2] will run any scripts that may be mounted in a docker-entrypoint-initdb.d directory before starting the server. So putting a bash script in a ConfigMap that gets used as a disk volume, could be easier that downloading/auditing a binary "wait for it" container image.
[1] https://kubernetes.io/docs/tasks/configure-pod-container/con...
Seriously?
is_ready() {
while ! nc -z $1 $2; do
echo "Waiting for $1:$2..."
sleep 1
done
echo "$1:$2 is accessible."
}
is_ready google.com 443 &
is_ready x.com 443 &
waitThis can certainly be zero dependency :)
Replace the use of netcat with pure bash to test a tcp (or udp) connection: </dev/tcp/$1/$2
is_ready is an alternative to this.
I built it for many reasons: - Most docker images do not contain netcat so you would have to download one of them in any case. - In the case of is_ready, you won't have to write this script yourself. - Repositories like this had a lot of traffic so I supposed that engineers need a similar tool but this repository requires wget and netcat as dependencies. For this reason, I built my own without any dependencies. https://github.com/eficode/wait-for