Observability Hub: Event-Driven Observability Platform

Cross-language schema enforcement

Used JSON Schema Draft 7+ to define event structures (Logs, Metrics, Traces). From these schemas, strict TypeScript interfaces and Go structs with json/validate tags are generated. The shared @observability-hub/event-contracts package guarantees both producers and consumers adhere to the exact same data definitions.

Processing 35K+ operations per second

Developed highly optimized validators including a Simple Validator capable of 35,714 validations/second and a Schema Validator reaching 8,333 ops/second. It enforces field-level error reporting, auto-detection of schema types, and batched validations to keep overhead negligible.

Worker pool, multi-sink fan-out, fault isolation

The collector starts a configurable goroutine pool (default: 20) where all workers race on the same RabbitMQ deliveries channel — zero coordination overhead. Each message is fanned out to three sinks: PostgreSQL via pq.CopyIn batch writes (coordinated by an Adaptive Batch Optimizer that tunes batch size based on Redis cache hit ratio), Redis for EventID-based deduplication (24h TTL) and per-service metadata caching, and Elasticsearch via a fire-and-forget goroutine so that ES unavailability never blocks RabbitMQ acknowledgement. Malformed messages that fail JSON unmarshal receive Nack(false, false) and are routed to a Dead Letter Queue, isolating them from the healthy message flow without infinite retry loops.

Seamless backward compatibility

Implemented a rigorous Message Versioning Strategy (SemVer) with a migration framework to ensure older log structures don't break current analytics. A reusable @observability-hub/log-client was created for Node.js Express microservices, enabling developers to easily log business, trace, and metric events with injected correlation IDs.

Three-tier storage: durability, speed, and search

**PostgreSQL** is the source of truth. The collector writes to it in batches using pq.CopyIn (the COPY protocol), which dramatically reduces round-trips compared to individual INSERTs. **Redis** plays a dual role inside the collector: it deduplicates events by storing EventID+CorrelationID keys with a 24-hour TTL, and it caches per-service metadata so the Adaptive Batch Optimizer can tune batch sizes at runtime without additional DB queries. **Elasticsearch** is the search layer — events are indexed asynchronously into per-service, per-month indices (logs-{service}-{YYYY-MM}), and failures there never affect the main pipeline because the goroutine writing to ES is decoupled from RabbitMQ acknowledgement.

Resilience: circuit breaker, DLQ, graceful shutdown

The Node.js @observability-hub/observability package embeds a CLOSED/OPEN/HALF_OPEN circuit breaker in the log publisher. If RabbitMQ becomes unavailable, after N failures the circuit opens and subsequent log calls fail fast instead of timing out — preventing cascading failures into HTTP handlers. On the collector side, malformed messages go to a Dead Letter Queue (DLQ) instead of retrying infinitely. On shutdown (SIGINT/SIGTERM), a 10-second drain window lets all in-flight workers finish and flushes the remaining batch buffer before closing connections.

Full distributed tracing and metrics visibility

**Jaeger** traces each request across service boundaries via the OpenTelemetry-compatible @obs/observability tracing module — a single trace ID reveals exactly where latency accumulates. **Prometheus** scrapes collector-level metrics (messages_processed, flush_duration_seconds, cache_hit_ratio, batch_size_optimized) and service-level metrics (request rate, error rate) from every Node.js service. **Grafana** unifies all of this into real-time dashboards. **RabbitMQ:** As the event backbone, it decouples producers from the collector entirely — traffic spikes buffer in the queue instead of back-pressuring the microservices.