Skip to content
Back to Projects
backend

Async Task Queue — Go

High-throughput async job processor using Go routines and Redis Streams with dead-letter queues.

GoRedisPrometheusDocker
Source

Problem

Need a lightweight task queue without Kafka/RabbitMQ operational overhead.

Solution

Built on Redis Streams with consumer groups, exponential backoff retries, and Prometheus metrics.

Challenges

  • Implementing exactly-once delivery with idempotency keys
  • Graceful shutdown draining in-flight jobs

Outcomes

  • 87 GitHub stars
  • 50k jobs/sec throughput on single node

Redis Streams enables consumer group semantics — multiple workers can each claim an entry, and unacknowledged entries flow to a Dead Letter Queue.