← Blog

Ingest Millions of Messages in the Queue using Serverless

January 3, 2023

While working on a serverless project for a client, we needed to ingest millions of messages from an external API into Azure Storage Queue. The initial approach proved problematic due to scalability limitations.

Initial architecture problem

The first design used a single “Ingestion Function” triggered via an HTTP endpoint to fetch messages from an external API and add them to a queue. However, this created a bottleneck: the single ingestion function is a bottleneck as it can only scale up to specific limits.

The core issue was failing to utilize the strength of the actual serverless function, which is multiple parallel small invocations.

Solution: fan-out architecture

We implemented a two-function approach:

1. Chunking Function — Divides the total message set into smaller chunks (5,000 messages each) and uploads each chunk as a separate file to Azure Blob Storage using a helper function that partitions the data.
2. Ingestion Function — Listens for blob storage upload events, downloads files, reads messages, and ingests them into the queue.

Result

By triggering multiple ingestion function invocations in parallel (one per chunk), the architecture achieves true scalability. We essentially followed a fan-out architecture, where we fan out the workload to multiple serverless invocations instead of one.

This pattern transforms a sequential bottleneck into a distributed, parallelized system capable of handling massive message volumes.