LogPort

The API and the worker are separate processes

The HTTP server never waits on a database write, the worker can be scaled on its own, and a crashed worker doesn't take ingestion down with it: logs simply queue up in Redis.

Trade-offA second process to deploy and watch. Its throughput counters live in its own memory, so the worker writes a metrics snapshot to Redis every 5 seconds (30-second TTL) for the API to serve.

A Redis list as the queue, drained with LMPOP

LPUSH on the left and LMPOP from the right give FIFO order, and LMPOP takes up to 100 items in one atomic round trip. On Redis older than 7 it falls back to a pipeline of RPOPs.

Trade-offThere is no acknowledgement. Once LMPOP returns, those logs exist only in the worker's memory until the flush succeeds.

Batch in the worker, not in ClickHouse

ClickHouse's async_insert is turned off because the worker already batches; buffering twice would only add delay. The worker flushes at 100 logs or every 2 seconds, whichever comes first.

Trade-offAt low volume a log can take about 2.5 seconds (the 2-second flush plus a 500 ms poll) to become searchable. The live tail doesn't wait for it, because it comes from Pub/Sub.

Publish from the API, not from the worker

The live feed shows a log the moment it is accepted, whatever ClickHouse is doing.

Trade-offThe live view and the stored view can disagree: a log can appear on the dashboard and later land in the dead-letter queue instead of ClickHouse.

A sliding window on a Redis sorted set

A fixed-window counter lets a client send a full quota just before a window boundary and another just after it. The sorted set always looks back exactly 10 seconds from now. Because it lives in Redis, the limit holds across API instances. If Redis errors, the limiter fails open.

Trade-offFour commands per request (one pipelined round trip), and every request is recorded, rejected ones included. A client that keeps hammering stays locked out until it backs off.

JWT alongside the legacy API key

Existing integrations keep working with X-API-Key, while new services exchange the master key for a short-lived JWT naming the service and a scope (ingest, read or admin).

Trade-offTwo auth paths to keep consistent, and scopes are issued but not yet enforced.

The dead-letter queue lives in Redis

It has to work while ClickHouse is down, so it can't live in ClickHouse. It is capped at 10,000 entries (the oldest is dropped first) so it can't grow without bound, and POST /api/v1/dlq/replay pushes everything back onto the main queue once the database recovers.

Trade-offPast the cap, the oldest failed batches are discarded.

ClickHouse is down or an insert fails

The batch goes back to the front of the buffer and is retried three times, 500 → 1000 → 2000 ms apart. After the third failure the whole buffer moves to the dead-letter queue with the error and retry count, ready to replay. If even the DLQ push fails, the batch is dropped and logged: the one place loss is explicit.

The worker stops or crashes

The API keeps accepting logs and they wait in the Redis list (append-only file on in Compose). On SIGTERM or SIGINT the worker flushes its buffer before exiting.

Redis errors while the worker polls

The poll loop backs off 500 ms × 2ⁿ, capped at 30 seconds, and resets after the first success.

One API key floods the endpoint

That key gets 429 with Retry-After, computed from the oldest request still in its window, and X-RateLimit-* headers so well-behaved clients can slow down. Other keys are unaffected.

The dashboard's socket drops

It shows the disconnect, reconnects after 3 seconds and keeps at most 500 logs in memory. While paused, logs keep arriving but the view stays still.

The Redis subscriber reconnects

ioredis restores the subscription itself; LogPort subscribes only on the first ready event, which avoids the double-subscribe error an earlier version hit.

Malformed or oversized input

Schema validation returns 400 with details: level must be debug, info, warn or error; service names are restricted to letters, digits, dots, underscores and hyphens; unknown fields are refused. An unparseable timestamp is stored as the current time rather than failing the batch.

Unbounded growth

The DLQ is capped, rate-limit keys expire with their window, and the worker's metrics snapshot expires after 30 seconds so a dead worker can't keep reporting. The metrics endpoint's health flag turns false when the queue passes 10,000 or the DLQ is non-empty.

Acknowledge before forgetting

Up to 100 logs, or 2 seconds of traffic, live only in the worker's memory after LMPOP. Redis Streams with a consumer group (XREADGROUP + XACK), or LMOVE into a per-worker processing list, would let a restarted worker pick up what a crashed one never flushed.

Rate-limit token holders too

The limiter keys on the X-API-Key header and skips requests without one, so callers using a JWT are never limited. Keying it on the token's service name would close that gap.

Enforce scopes, and authenticate reads

Tokens carry a scope, but no route checks it, and GET /api/v1/logs, /logs/stats and the /ws feed have no auth at all.

Don't fail a request that already succeeded

enqueueLog awaits PUBLISH after LPUSH, so a failed publish returns an error for a log that is already queued, and a client retry duplicates it. Publishing best-effort, and pipelining the batch endpoint's LPUSH + PUBLISH pairs instead of awaiting 2 × N commands, would fix both the error and the round trips.

Make inserts idempotent

Every log has a UUID, but a plain MergeTree doesn't deduplicate. An insert that timed out after succeeding is retried and written twice. A ReplacingMergeTree keyed on id would collapse the copies.

Catch up after a reconnect

Logs published during the 3-second reconnect never reach that dashboard. Resuming from a stream ID, or backfilling from ClickHouse by the last timestamp seen, plus jittered backoff, would make the live tail gap-free.