Consumer Groups

Verified on Redis 8.10.2 (redis:8.10.2-alpine), ioredis 6.0.0, Node 24, Apple-silicon Mac. The # cli/... transcripts are one session in document order, replayed by the course harness; I used explicit entry IDs (1-0..5-0) so the session is reproducible, and the idle milliseconds are timings from my run, so yours will differ. Stream basics are in Streams.

What is a consumer group

A consumer group lets several workers (consumers) read one stream so that each entry goes to exactly one consumer of the group. Redis remembers, per group, which entries were delivered and to whom, and keeps every delivered-but-unacknowledged entry in the Pending Entries List (PEL) until a consumer calls XACK. That bookkeeping is what turns a log into a work queue with at-least-once processing. Different groups on the same stream are independent: each one sees every entry.

Creating groups and reading as a consumer

XGROUP CREATE key group id [MKSTREAM] sets where the group starts. $ = only entries added from now on; 0 = from the beginning. MKSTREAM creates an empty stream if needed. XREADGROUP GROUP group consumer [COUNT n] STREAMS key > delivers entries the group has never delivered to anyone; consumers are created on first use.

cli/pubsub-streams-consumer-groups-1.redis

127.0.0.1:6379> XGROUP CREATE jobs workers $ MKSTREAM
OK
127.0.0.1:6379> XLEN jobs
(integer) 0
127.0.0.1:6379> XADD jobs 1-0 type email to ada
"1-0"
127.0.0.1:6379> XADD jobs 2-0 type sms to bob
"2-0"
127.0.0.1:6379> XADD jobs 3-0 type email to cy
"3-0"
127.0.0.1:6379> XADD jobs 4-0 type push to dee
"4-0"
127.0.0.1:6379> XADD jobs 5-0 type email to eve
"5-0"
127.0.0.1:6379> XGROUP CREATE jobs audit 0
OK
127.0.0.1:6379> XREADGROUP GROUP workers alice COUNT 2 STREAMS jobs >
1) 1) "jobs"
   2) 1) 1) "1-0"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "ada"
      2) 1) "2-0"
         2) 1) "type"
            2) "sms"
            3) "to"
            4) "bob"
127.0.0.1:6379> XREADGROUP GROUP workers bob COUNT 2 STREAMS jobs >
1) 1) "jobs"
   2) 1) 1) "3-0"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "cy"
      2) 1) "4-0"
         2) 1) "type"
            2) "push"
            3) "to"
            4) "dee"
127.0.0.1:6379> XREADGROUP GROUP workers alice COUNT 10 STREAMS jobs >
1) 1) "jobs"
   2) 1) 1) "5-0"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "eve"
127.0.0.1:6379> XREADGROUP GROUP workers alice COUNT 10 STREAMS jobs >
(nil)
127.0.0.1:6379> XREADGROUP GROUP workers alice COUNT 10 STREAMS jobs 0
1) 1) "jobs"
   2) 1) 1) "1-0"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "ada"
      2) 1) "2-0"
         2) 1) "type"
            2) "sms"
            3) "to"
            4) "bob"
      3) 1) "5-0"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "eve"
127.0.0.1:6379> XREADGROUP GROUP audit reader COUNT 1 STREAMS jobs >
1) 1) "jobs"
   2) 1) 1) "1-0"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "ada"

What that session shows:

• alice got 1-0 and 2-0, bob got 3-0 and 4-0, and alice’s next read got 5-0: no entry went to two consumers. The following > read returned (nil): nothing new.
• > versus 0. > means “new entries never delivered”. A concrete ID such as 0 means “give me my own pending entries after that ID”: alice got back 1-0, 2-0 and 5-0, the entries she was delivered and had not acknowledged. This is how a restarted consumer resumes. (In this run that history read also raised the delivery counter of 5-0 to 2, visible below.)
• The audit group, created at 0, independently received 1-0 again: groups do not share progress.

XACK, XPENDING and XINFO

XACK key group id [id ...] removes entries from the PEL and returns how many it removed (0 if they were not pending, so a repeated ack is harmless). XPENDING key group summarises the PEL: the count, the smallest and largest pending ID, and pending per consumer. The extended form XPENDING key group start end count lists each entry with its consumer, idle time in ms and delivery count:

cli/pubsub-streams-consumer-groups-2.redis

127.0.0.1:6379> XPENDING jobs workers
1) (integer) 5
2) "1-0"
3) "5-0"
4) 1) 1) "alice"
      2) "3"
   2) 1) "bob"
      2) "2"
127.0.0.1:6379> XACK jobs workers 1-0 2-0
(integer) 2
127.0.0.1:6379> XACK jobs workers 1-0
(integer) 0
127.0.0.1:6379> XPENDING jobs workers - + 10
1) 1) "3-0"
   2) "bob"
   3) (integer) 518
   4) (integer) 1
2) 1) "4-0"
   2) "bob"
   3) (integer) 518
   4) (integer) 1
3) 1) "5-0"
   2) "alice"
   3) (integer) 335
   4) (integer) 2
127.0.0.1:6379> XINFO GROUPS jobs
1)  1) "name"
    2) "audit"
    3) "consumers"
    4) (integer) 1
    5) "pending"
    6) (integer) 1
    7) "last-delivered-id"
    8) "1-0"
    9) "entries-read"
   10) (integer) 1
   11) "lag"
   12) (integer) 4
2)  1) "name"
    2) "workers"
    3) "consumers"
    4) (integer) 2
    5) "pending"
    6) (integer) 3
    7) "last-delivered-id"
    8) "5-0"
    9) "entries-read"
   10) (integer) 5
   11) "lag"
   12) (integer) 0

XINFO GROUPS shows per group pending, last-delivered-id, entries-read and lag (entries not yet delivered: audit was 4 behind, workers 0). XINFO CONSUMERS shows each consumer’s pending count and idle time, which is how you spot a dead consumer: a non-zero pending with a growing idle (milliseconds, so this output is from my run and yours will differ):

127.0.0.1:6379> XINFO CONSUMERS jobs workers
1) 1) "name"
   2) "alice"
   3) "pending"
   4) (integer) 1
   5) "idle"
   6) (integer) 482
   7) "inactive"
   8) (integer) 604
2) 1) "name"
   2) "bob"
   3) "pending"
   4) (integer) 2
   5) "idle"
   6) (integer) 665
   7) "inactive"
   8) (integer) 665

XADD jobs 1-0 type "email" to "[email protected]"
XADD jobs 2-0 type "sms" to "+6681234567"
XGROUP CREATE jobs workers 0
XREADGROUP GROUP workers consumer-1 COUNT 10 STREAMS jobs >
XACK jobs workers 1-0
XPENDING jobs workers - + 10

Copy the sequence ⧉

Recovering from a dead consumer

If bob crashes, 3-0 and 4-0 stay in the PEL under his name forever unless someone takes them. XAUTOCLAIM key group consumer min-idle-time start [COUNT n] (6.2+) transfers ownership of entries that have been pending for at least min-idle-time ms and returns them, plus a cursor for the next call. XCLAIM does the same for IDs you name. In the session below alice first acknowledged her own 5-0, then tried a claim with min-idle-time 60000, which found nothing because the entries were only seconds old, and then a claim with 0 (chosen so the transcript is deterministic; production values are seconds to minutes) which took bob’s two entries:

cli/pubsub-streams-consumer-groups-3.redis

127.0.0.1:6379> XACK jobs workers 5-0
(integer) 1
127.0.0.1:6379> XAUTOCLAIM jobs workers alice 60000 0-0 COUNT 10
1) "0-0"
2) (empty array)
3) (empty array)
127.0.0.1:6379> XAUTOCLAIM jobs workers alice 0 0-0 COUNT 10
1) "0-0"
2) 1) 1) "3-0"
      2) 1) "type"
         2) "email"
         3) "to"
         4) "cy"
   2) 1) "4-0"
      2) 1) "type"
         2) "push"
         3) "to"
         4) "dee"
3) (empty array)
127.0.0.1:6379> XPENDING jobs workers - + 10
1) 1) "3-0"
   2) "alice"
   3) (integer) 53
   4) (integer) 2
2) 1) "4-0"
   2) "alice"
   3) (integer) 53
   4) (integer) 2
127.0.0.1:6379> XACK jobs workers 3-0 4-0
(integer) 2
127.0.0.1:6379> XPENDING jobs workers
1) (integer) 0
2) (nil)
3) (nil)
4) (nil)
127.0.0.1:6379> XREADGROUP GROUP audit reader COUNT 1 NOACK STREAMS jobs >
1) 1) "jobs"
   2) 1) 1) "2-0"
         2) 1) "type"
            2) "sms"
            3) "to"
            4) "bob"
127.0.0.1:6379> XPENDING jobs audit
1) (integer) 1
2) "1-0"
3) "1-0"
4) 1) 1) "reader"
      2) "1"
127.0.0.1:6379> XGROUP DELCONSUMER jobs workers bob
(integer) 0
127.0.0.1:6379> XGROUP DESTROY jobs audit
(integer) 1

Notice the delivery counter: the claimed entries now say 2. A counter that keeps growing is your poison message signal; dead-letter entries above a threshold rather than retrying forever. NOACK on XREADGROUP skips the PEL entirely (audit read 2-0 and XPENDING jobs audit still showed only the earlier 1-0): use it only when losing an entry on a crash is acceptable.

A worker in Node.js: crash, recover, and the ack-first trap

The script below plays it end to end in ioredis. Worker A reads two jobs and dies. Worker B processes the rest, acknowledging after processing. After the idle threshold, XAUTOCLAIM gives A’s jobs to B. The last block repeats the crash with XACK before the work:

src/pubsub-streams-consumer-groups-worker.ts

// @run
import { Redis } from 'ioredis';

const redis = new Redis(process.env.REDIS_URL!);
const stream = 'b2:orders';
const group = 'fulfil';
await redis.del(stream);
await redis.xgroup('CREATE', stream, group, '0', 'MKSTREAM');
for (let i = 1; i <= 5; i++) await redis.xadd(stream, '*', 'order', `o-${i}`);

type Entry = [id: string, fields: string[]];
// XREADGROUP / XAUTOCLAIM return nested arrays; the entries part is [[id, [field, value, ...]], ...]
async function readNew(consumer: string, count: number): Promise<Entry[]> {
  const res = await redis.xreadgroup('GROUP', group, consumer, 'COUNT', count, 'BLOCK', 200, 'STREAMS', stream, '>');
  return res ? (res[0]![1] as Entry[]) : [];
}
const processOrder = async (e: Entry) => console.log('  processed', e[1][1]);

// worker A takes two jobs and "crashes": no processing, no XACK
const lost = await readNew('worker-a', 2);
console.log('worker-a read', lost.length, 'and died; pending:', (await redis.xpending(stream, group))[0]);

// worker B handles the rest: process first, THEN acknowledge
for (let batch = await readNew('worker-b', 10); batch.length; batch = await readNew('worker-b', 10)) {
  for (const e of batch) { await processOrder(e); await redis.xack(stream, group, e[0]); }
}
console.log('worker-b done; still pending (worker-a\'s jobs):', (await redis.xpending(stream, group))[0]);

// recovery: claim entries idle for more than 300 ms (in production 30 s or more), process, ack
await new Promise((r) => setTimeout(r, 400));
const [, claimed] = (await redis.xautoclaim(stream, group, 'worker-b', 300, '0-0', 'COUNT', 10)) as [string, Entry[], string[]];
for (const e of claimed) { await processOrder(e); await redis.xack(stream, group, e[0]); }
console.log('reclaimed', claimed.length, '-> pending now:', (await redis.xpending(stream, group))[0]);

// ...the same crash if you ACK BEFORE working: nothing is left to recover
const s2 = 'b2:orders-ackfirst';
await redis.del(s2);
await redis.xgroup('CREATE', s2, group, '0', 'MKSTREAM');
await redis.xadd(s2, '*', 'order', 'o-99');
const res = await redis.xreadgroup('GROUP', group, 'worker-c', 'STREAMS', s2, '>');
await redis.xack(s2, group, (res![0]![1] as Entry[])[0]![0]);      // ack first...
/* worker-c crashes here, before doing the work */
await new Promise((r) => setTimeout(r, 400));
const [, gone] = (await redis.xautoclaim(s2, group, 'worker-d', 300, '0-0')) as [string, Entry[], string[]];
console.log('ack-first: job o-99 recoverable?', gone.length > 0, '| pending:', (await redis.xpending(s2, group))[0]);

await redis.del(stream, s2);
await redis.quit();

Real output (this Mac):

worker-a read 2 and died; pending: 2
  processed o-3
  processed o-4
  processed o-5
worker-b done; still pending (worker-a's jobs): 2
  processed o-1
  processed o-2
reclaimed 2 -> pending now: 0
ack-first: job o-99 recoverable? false | pending: 0

A real worker runs this in a loop with BLOCK and periodically calls XAUTOCLAIM with a min-idle-time longer than your slowest legitimate job; ioredis returns the nested-array replies shown in the types ([[id, [field, value, ...]], ...], or null when BLOCK times out).

🟥 Consumer groups give at-least-once, so make the work idempotent

A job can run twice: a slow worker whose entry is auto-claimed keeps running while the new owner starts it too. The PEL guarantees that a crash never silently drops an entry, not that it runs once. Use an idempotency key (the entry ID is a good one) in whatever the job writes.

Reading AI-generated code

An assistant writes the consumer loop for an order-fulfilment stream. It acknowledges first “so nothing is processed twice”:

🐛 Spot the bug

for (const [id, fields] of entries) {
await redis.xack('orders', 'fulfil', id);   // ack first: avoid duplicates
await shipOrder(fields);                    // may throw or the process may crash
}

Show the answer

Acknowledging removes the entry from the PEL, so if the process dies (or shipOrder throws) after the XACK there is nothing left for XAUTOCLAIM to find. In the last block of the script the ack-first job o-99 ended with recoverable? false | pending: 0: it vanished, while the ack-after jobs were all recovered. Process first, acknowledge after, and make the work idempotent to handle the rare duplicate:

for (const [id, fields] of entries) {
  await shipOrder(fields);                    // idempotent: keyed by the entry id
  await redis.xack('orders', 'fulfil', id);   // ack only once the work is done
}

Mental Model

A consumer group is a queue with a receipt: XREADGROUP hands you an entry and writes your name on it in the PEL; XACK tears the receipt up. If you disappear, the receipt sits there getting older until another worker’s XAUTOCLAIM takes it over. What the group gives you is the receipt; correctness comes from acknowledging only after the work and from idempotent work.

Trade-offs

Choice	Benefit	Cost
Consumer group over plain XREAD	Scales out; each entry to one consumer; crash-safe bookkeeping	You manage XACK, reclaiming and dead-lettering
At-least-once via the PEL	No silent loss on a crash	Duplicates are possible after reclaiming; work must be idempotent
XAUTOCLAIM with a small min-idle-time	Fast recovery	Steals entries from slow-but-alive workers, causing double execution
NOACK	No PEL cost	A crash loses the entry (like Pub/Sub)
Several groups on one stream	Independent consumers of the same events	The stream must retain entries until the slowest group is done (trim carefully)

Common Misconceptions

• “Consumer groups give exactly-once.” They give at-least-once. A reclaimed entry can run twice.
• “XACK deletes the entry.” It removes it from the group’s PEL only; the entry stays in the stream (use XACKDEL/XDELEX, 8.2+, or trimming).
• “A dead consumer’s entries get reassigned automatically.” Nothing moves until some worker runs XAUTOCLAIM/XCLAIM.
• “> and 0 are interchangeable.” > is new entries; a concrete ID replays your own pending ones.

Common mistakes

• XACK before the work is done — a crash then loses the job (shown above).
• Never calling XACK — the PEL grows forever and XINFO GROUPS shows a rising pending.
• No reclaim process — entries of dead consumers are never retried.
• Retrying a poison message forever — watch the delivery counter in XPENDING and dead-letter past a limit.
• Non-idempotent jobs — at-least-once means duplicates happen.

Real-world example

Redis’s own documentation describes the PEL, XAUTOCLAIM-based recovery of dead consumers and the at-least-once semantics in Redis Streams (“Recovering from permanent failures”).

What does XREADGROUP GROUP g consumer STREAMS s > mean?

Deliver new, never-delivered entries to this consumer Re-deliver all pending entries from the beginning Acknowledge all entries in the group Delete the consumer group

What happens to a stream entry after XACK is called?

It is deleted from the stream entirely It is moved to a dead-letter stream It is removed from the group's Pending Entries List It is redelivered to another consumer

What does XPENDING show?

All entries ever written to the stream Entries delivered to consumers of the group but not yet acknowledged Entries waiting to be delivered The number of active consumer groups

In XGROUP CREATE jobs workers 0 MKSTREAM, what does 0 mean?

Only deliver zero entries at a time Start from the latest entry only Start reading from the beginning of the stream Create zero consumers initially

A worker calls XACK first and then crashes before finishing the job. What does XAUTOCLAIM find?

The job, with delivery count 2 Nothing: the entry already left the PEL, so the job is lost The job in a dead-letter stream An error

A consumer has died with 3 entries pending. Which command makes another worker take them over?

XGROUP DELCONSUMER XAUTOCLAIM with a min-idle-time XREADGROUP with > XTRIM

The delivery counter of an entry in XPENDING keeps climbing. What does it usually mean?

The entry is popular A poison message that keeps failing: dead-letter it after a limit The stream needs trimming The group is out of sync

Submit

Mark as complete