Deploying Temporal on AWS ECS with Terraform
papnori.github.io23 points by norapap 13 days ago
23 points by norapap 13 days ago
> Autoscaling is configured via CloudWatch alarms on CPU usage: > Scale-out policy adds workers when CPU > 30%. > Scale-in policy removes idle workers when CPU < 20%.
Does this handle the case where there are longer-running activities that have low CPU usage? Couldn't these be canceled during scalein?
Temporal would retry them, but it would make some workflow runs take longer, which could be annoying for some user-interactive workflows.
Otherwise I've seen needing to hit the metrics endpoint to query things like `worker_task_slots_available` to scale up, or query pending activities, pending workflows, etc to scale down per worker.
Whats funny is in some sense, temporal replaces alot of the AWS stack. You dont really need queues, step functions, lambdas, and the rest. I personally think its a better compute model than the wildly complicated AWS infra. Deploying temporal on compute primitives is simply better, and allows for you to be cloud agnostic.
I sometimes suspect AWS deliberately looks for ways to extract low-overhead tasks into dedicated services for the simple reason that many people will pay for the service without thinking about whether they really need it.
Its very easy to add AWS services, but after building them into a stack over a few years, its basically impossible to remove them
Why go with Fargate instead of EC2?
We went with Fargate because it keeps things lean — no servers to manage, no patching, no scaling headaches. It’s perfect for our bursty workloads, since we only pay when containers actually run . Plus autoscaling just works .
In the github you can find comments to easily switch to EC2 if your workload needs it
Just as an aside for the fargate convo... we switched from fargate to EC2 auto scaling group so we could run a custom AMI that has our container images (which are larger than I like) pre-baked and we went from a ~3 minute startup in fargate to ~30 seconds with the ASG when it's not triggering a scaling action.
We're using prefect not Temporal and each prefect flow launches in a discrete ECS task so the waiting added up.