← Blog

Executing ad-hoc jobs with Amazon Step Functions

March 8, 2023

A client needed to process thousands of database records, generate a custom report, and monitor its execution state — with filtering based on client requests, support for multiple simultaneous jobs, and processing times ranging anywhere from 5 minutes to 5 hours.

The solution

We combined AWS Step Functions with ECS Tasks to run these ad-hoc reporting jobs.

A client submits a request with filtering parameters:

{
    "filter": {
        "client_id": "1266cd5a-1dfa-44f2-83ca-534c30b38555",
        "start_date": "2021-01-01",
        "end_date": "2021-12-31",
        "status": "pending"
    }
}

How it works

1. Containerization — A Python script reads the request from an environment variable and retrieves data via a RESTful API. The script is dockerized and pushed to Amazon ECR.
2. Task definition — An ECS task definition specifies the resources needed (e.g. 4 GB RAM).
3. Step Function configuration — The state machine accepts JSON input, converts it to a string with States.JsonToString($), and passes it as the REPORT_REQUEST environment variable to the ECS task.
4. Execution — The runTask.sync parameter makes the step function wait for task completion, which also allows multiple concurrent executions.

Why it’s worth it

This solution automates the repetitive ad-hoc tasks, and gives much easier visualization and monitoring through the Step Functions console compared to calling the ECS API directly.