ChiragMukhija

The problem

Background jobs fail, workers die mid-task, and two workers must never run the same job at the same time. The goal: at-least-once delivery with retries, priorities and delayed jobs — without a job ever being silently lost.

Decisions & trade-offs

1. Postgres is the truth. Redis is only an index.

   Instead of Using Redis as the queue itself

   Every job lives durably in Postgres. Redis holds one sorted set per priority tier, scored by run-at time, so “what’s due next?” is an in-memory lookup — not a table scan.

   Cost: Every path that makes a job eligible has to write to both stores. Forgetting one is the easiest bug to introduce in this codebase.

2. Claim with a guarded UPDATE, not row locks.

   Instead of SELECT … FOR UPDATE SKIP LOCKED

   Once Redis has picked a single candidate there is nothing left to lock against. UPDATE … WHERE status = 'pending' is the guard; zero rows back means a stale entry, so drop it and try again.

   Cost: The claim path now has a hard dependency on Redis — by design, there is no fallback to scanning the table.

3. Shut down gracefully, don’t hope.

   Instead of Letting the process die on SIGTERM

   The worker awaits its in-flight job before exiting, so scaling down never strands work. The API closes every WebSocket before server.close() — an open socket blocks it forever, which testing caught.

   Cost: Scaling down now takes as long as the slowest in-flight job — shutdown waits for it.

The crux, in code

async function claimNextJob() {
  const redis = await getRedisClient();

  for (let tier = MIN_TIER; tier <= MAX_TIER; tier++) {
    while (true) {
      const jobId = await redisQueue.peekDueJob(redis, tier);
      if (!jobId) break;

      const claimed = await pool.query(
        `UPDATE jobs
         SET status = 'processing', attempts = attempts + 1
         WHERE id = $1 AND status = 'pending'
         RETURNING *`,
        [jobId]
      );
      await redisQueue.removeJob(redis, tier, jobId);

      if (claimed.rows.length > 0) return claimed.rows[0];
      // stale Redis entry — already claimed elsewhere, try again
    }
  }
  return null;
}

What breaks — honestly

• No lease or visibility timeout yet — a SIGKILL’d worker strands its job in “processing”.
• Exponential backoff (5 s · 2ⁿ) has no jitter and no cap, so mass failures retry in lockstep.
• No priority aging: a steady flood of priority-0 jobs can starve lower tiers.
• At-least-once, not exactly-once — job handlers have to be idempotent.

Build log

1. Postgres-backed queue + submission API
2. Separate worker process + handler registry
3. Retries, exponential backoff, dead-letter queue
4. Priorities + delayed jobs
5. Redis sorted-set hot claim path
6. Live dashboard over WebSockets + Pub/Sub
7. Docker, N workers, graceful shutdown

The problem

A feed is the same data read far more often than it’s written. Do you pay at write time — copy each post into every follower’s feed — or at read time, joining the follow graph live on every open?

Decisions & trade-offs

1. Build both fan-out strategies, side by side.

   Instead of Picking one from a blog post

   Fan-out-on-write precomputes a feed_items row per follower, so reads are a single indexed lookup. Fan-out-on-read writes once and joins follows × posts on every request. Both are exposed so they can be compared directly.

   Cost: Write fan-out grows with follower count — the celebrity problem — and it currently runs inside the post request.

2. Keyset pagination everywhere. Never OFFSET.

   Instead of LIMIT … OFFSET …

   OFFSET makes the database walk and throw away every skipped row, and pages drift when new posts arrive. An opaque (created_at, id) cursor stays O(page) at any depth.

   Cost: No “jump to page 40”. Clients can only move forward from a cursor.

3. The app server never touches image bytes.

   Instead of Streaming uploads through Express

   Clients upload straight to S3 with a presigned URL. The server only signs the request — and before saving a post, checks that the image URL points at its own bucket.

   Cost: Orphaned uploads — signed but never attached to a post — need cleaning up separately.

The crux, in code

const fanOutOnWrite = async (post) => {
  const followerIds = await followService.getFollowerIds(post.user_id);
  if (followerIds.length === 0) return;

  const values = [];
  const placeholders = followerIds.map((followerId, i) => {
    values.push(followerId, post.id);
    return `($${i * 2 + 1}, $${i * 2 + 2})`;
  });

  await db.query(
    `INSERT INTO feed_items (user_id, post_id)
     VALUES ${placeholders.join(', ')}
     ON CONFLICT (user_id, post_id) DO NOTHING`,
    values
  );
};

What breaks — honestly

• Fan-out runs synchronously in the post request — a big account makes posting slow. The fix is a queue (see SYS—01).
• Counts are live COUNT(*) rather than counters: right at this scale, with the point to revisit written down.
• Notifications are poll-based; there are no WebSockets here.

Build log

1. JWT access + refresh, Redis-revocable sessions
2. Profiles, posts, S3 presigned uploads
3. Follow graph
4. Feeds: fan-out on write and on read
5. Likes, comments, notifications
6. Keyset pagination, search, explore
7. Docker, Nginx, two-layer rate limiting

The problem

Networks time out and clients retry. A retried payment request has to return the original payment — not quietly create a second charge. And every state change has to leave a trail you can reconcile against a bank later.

Decisions & trade-offs

1. Idempotency keys are unique per merchant.

   Instead of A globally unique key

   UNIQUE (merchant_id, idempotency_key): two unrelated merchants can both send “order_1”. A SELECT pre-check handles the common retry; the constraint’s 23505 violation is the backstop for the race.

   Cost: Two requests in the same millisecond can both do work before one is rejected — closed properly by a Redis lock in phase 3.

2. A payment and its first event commit together.

   Instead of Two independent pool.query() calls

   pool.query() can hand BEGIN and INSERT to different connections, which makes the transaction meaningless. One checked-out client runs BEGIN → INSERT payment → INSERT event → COMMIT.

   Cost: Holding a connection for the whole transaction — the pool is capped at 20.

3. History is append-only. Money is DECIMAL.

   Instead of Overwriting a status column · FLOAT amounts

   payment_events keeps every transition, which is what debugging and reconciliation actually read. DECIMAL(12,2) because binary floats drift on money.

   Cost: A denormalised status column still has to be kept in sync for fast queries.

The crux, in code

const client = await pool.connect();
try {
  await client.query('BEGIN');
  const { rows } = await client.query(
    `INSERT INTO payments (merchant_id, idempotency_key, amount)
     VALUES ($1, $2, $3) RETURNING *`,
    [merchantId, idempotencyKey, amount]
  );
  await client.query(
    `INSERT INTO payment_events (payment_id, from_status, to_status)
     VALUES ($1, NULL, $2)`,
    [rows[0].id, rows[0].payment_status]
  );
  await client.query('COMMIT');
  return rows[0];
} catch (err) {
  await client.query('ROLLBACK');
  if (err.code === '23505') throw conflict(409); // lost the idempotency race
  throw err;
} finally {
  client.release();
}

What breaks — honestly

• The idempotency race isn’t fully closed yet — a Redis lock lands in phase 3.
• A DB write and a bank call can never be atomic. The honest fix is a reconciliation job (phase 6).
• No test suite yet.

Build log

1. Core API, idempotency, transactions, audit trail
2. Mock bank + payment state machine
3. Redis locking + retries on bank calls
4. Per-merchant rate limiting + caching
5. Docker + Nginx
6. Reconciliation batch job