TheArchitect

Spot Batch Processing (up to 90 % savings)

AWS architecture template · SQS-driven Auto Scaling Group of Spot EC2 workers. Scale-in on empty queue, handle Spot interruptions via SQS visibility timeout re-queue.

Spot Batch Processing (up to 90 % savings) is a free AWS architecture template built with SQS, Auto Scaling, CloudWatch, EC2 and IAM. Open it in TheArchitect to edit the diagram, estimate monthly cost, and export the CloudFormation or Terraform that deploys it.

Open in TheArchitect All templates
Job QueueDead-Letter Qu…Spot Worker ASGSpot Worker (t…Queue Depth Me…Worker Instanc…
Spot Batch Processing (up to 90 % savings) architecture diagram

AWS services in this template

  • SQS×2
  • Auto Scaling
  • CloudWatch
  • EC2
  • IAM

Deployment notes

Needs input

  • Configure a Spot interruption handler: on SIGTERM (2-min warning), set SQS visibility timeout to 0 to re-queue the in-flight message.
  • Use a mixed-instances policy (3+ instance types in the same ASG) to maximise Spot availability.
  • Target Tracking policy on ApproximateNumberOfMessagesVisible metric: scale out when queue depth > 10.
  • Cost: Spot t3.medium ~$0.013/hr vs on-demand ~$0.042/hr — up to 70 % savings.

Generated CloudFormation — 6 resources

Logical IDResource type
JobQueuesqsAWS::SQS::Queue
DeadLetterQueuesqsAWS::SQS::Queue
SpotWorkerASGautoscalingAWS::AutoScaling::AutoScalingGroup
SpotWorkert3mediumec2AWS::EC2::Instance
QueueDepthMetriccloudwatchAWS::CloudWatch::Alarm
WorkerInstanceProfileiamAWS::IAM::Role
View the full CloudFormation template
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Generated by TheArchitect — https://www.thecloudarchitect.dev",
  "Metadata": {
    "TheArchitect": {
      "Edges": [
        {
          "From": "SpotWorkert3mediumec2",
          "To": "JobQueuesqs",
          "Label": "consume jobs"
        }
      ]
    }
  },
  "Resources": {
    "JobQueuesqs": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "QueueName": "cost-batch-jobs",
        "VisibilityTimeout": 300,
        "MessageRetentionPeriod": 86400,
        "ReceiveMessageWaitTimeSeconds": 20
      }
    },
    "DeadLetterQueuesqs": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "QueueName": "cost-batch-jobs-dlq",
        "MessageRetentionPeriod": 1209600
      }
    },
    "SpotWorkerASGautoscaling": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "AutoScalingGroupName": "cost-spot-workers",
        "MinSize": "0",
        "MaxSize": "20",
        "DesiredCapacity": "0"
      }
    },
    "SpotWorkert3mediumec2": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "InstanceType": "t3.medium"
      }
    },
    "QueueDepthMetriccloudwatch": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
        "AlarmName": "cost-batch-queue-depth",
        "AlarmDescription": "Scale out Spot ASG when job queue depth > 10",
        "Namespace": "AWS/SQS",
        "MetricName": "ApproximateNumberOfMessagesVisible",
        "Statistic": "Average",
        "Period": 60,
        "EvaluationPeriods": 2,
        "Threshold": 10,
        "ComparisonOperator": "GreaterThanThreshold",
        "TreatMissingData": "notBreaching"
      }
    },
    "WorkerInstanceProfileiam": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "Description": "Worker EC2 instance profile — SQS consume + S3 read/write for job artifacts"
      }
    }
  }
}

Generated by TheArchitect from the diagram above. Review values such as names, CIDR ranges and instance sizes before deploying to your account.

Related templates