Your Feature Pipeline's Messaging Layer Is a Reproducibility Decision
Why This Matters Now
A systems architecture comparison of RabbitMQ, Kafka, and Pulsar surfaced today with clear implications for anyone owning streaming feature infrastructure. The core insight isn't about throughput benchmarks — it's about replay capability and what that means for ML reproducibility.
| Dimension | RabbitMQ | Kafka | Pulsar |
|---|---|---|---|
| Data Retention | Gone after consumption | Configurable retention | Ledger-based (configurable) |
| Replay | None | Full replay from any offset | Full replay via cursors |
| Scaling Model | Coupled | Coupled (compute + storage) | Separated (compute ≠ storage) |
| ML Pipeline Fit | Job dispatch, batch orchestration | Feature streaming, event sourcing, training data | Elastic inference, multi-pattern workloads |
The Reproducibility Angle
Kafka's offset-based replay lets you reconstruct the exact sequence of events that generated your training features at any historical point. This is non-negotiable for:
- Debugging training-serving skew — "what did the feature look like at training time vs. serving time?"
- Feature backfills after schema changes or bug fixes
- Running offline/online feature consistency audits
- Generating point-in-time correct training datasets from event streams
If your features flow through RabbitMQ, you're relying entirely on downstream storage for historical reconstruction — which works but adds complexity and failure modes. You're one schema change away from an unreproducible training set.
When Pulsar Beats Kafka
Pulsar's separated compute and storage architecture means you can scale broker capacity for inference traffic spikes without paying for proportional storage scaling. If your ML workloads are bursty — think batch retraining jobs that spike GPU inference queues — Pulsar's elasticity is worth evaluating. But Kafka's ecosystem maturity (Kafka Connect, ksqlDB, Flink integration) still makes it the default correct choice for most feature store architectures.
API Design for Model Serving
A secondary but useful signal: if you serve multi-output models (scores + explanations + metadata), GraphQL lets each consumer request exactly the fields it needs. A mobile client fetching a recommendation score doesn't need SHAP values. But GraphQL's caching lives at the application layer, not the HTTP layer — you lose CDN caching and ETag support. For high-QPS prediction endpoints, start with REST and only move to GraphQL when you have 3+ consumer types requesting meaningfully different output subsets.
Kafka's offset-based replay is non-negotiable for any ML pipeline that needs to reconstruct historical training data; if your features flow through RabbitMQ, you're accumulating reproducibility debt with every schema change.
What to do
Audit your feature pipeline's messaging layer this sprint — identify whether you have replay capability for training data reconstruction
If on RabbitMQ for feature streaming, scope a Kafka migration POC this quarter focused on one high-value feature pipeline
For multi-output model serving APIs, evaluate GraphQL only when you confirm 3+ distinct consumer types with different field requirements