At this point, I started to look deeper at performance. Compute instance startup was taking over 30 seconds, and each generation propagation was about 55-75 seconds
I worked this from a few angles:
- parallelize actions (health checks, integrity checks, propagation initiation, and terminating ancestors) [30% gain]
- rebuild and reduce the AMI below Ubuntu minimal to fit in 2GiB EBS volumes (reduced cost and increased speed by also bundling the glider daemon and boto3 SDK in the AMI) [30% gain]
- switched from AWS CLI to boto3 SDK [5-10% gain]
- switched from CloudFormation creation stacks to using EC2 API directly with LaunchTemplate [under 5% gain total, surprisingly, but the time to initial instance startup is faster so I might be able to do better somewhere here]
- WIP: further refactoring of the daemon propagation steps especially around initial health/integrity checks
I still think I'm missing something major that's extending propagation times, despite benchmarking phases through several trials
I also optimized for cost (reduced DDB calls, abandoned KMS, reduced log & audit retention to the latest run, etc), and it's about a penny for a 10 generation run (and it scales sublinearly, mostly, since the instances are terminated afterwards)
@Sustainable2050@mastodon.energy
