Building a Fault-Tolerant Log Pipeline for Security Monitoring

Building a Fault-Tolerant Log Pipeline for Security Monitoring is a reliability problem because a security operations center can make a wrong decision when logs are missing, and the worst part is that the missing logs may not produce an error. A pipeline that drops events during a restart, a disconnected indexer, or a full queue can silently break process trees, hide tactic progression, and turn a multi-stage intrusion into unrelated alerts. This article explains why log loss is a reliability problem, how the lab project designs a fault-tolerant log pipeline for Windows event monitoring, and which failure tests were used to validate the approach. All lab work should be done only in authorized, isolated environments.

Silent log loss breaks security visibility

Windows event logs are the primary data source for host monitoring, but they are not simple. They arrive in high volume across channels such as Security, Sysmon, PowerShell, Task Scheduler, WMI-Activity, and Defender, with inconsistent fields and potential tampering. In the lab project, a Wazuh agent on a Windows host sends event logs to a Wazuh server, and the server writes all raw events to archives.json when logall_json is enabled. Vector then reads that file and forwards logs to a raw archive and to the correlation service for scoring and correlation. If any step loses events, the downstream correlator may fail to connect process activity into an attack chain. The chain rule requires at least one high-confidence event and at least three distinct ATT&CK tactics, excluding unknown. Missing logs can hide the third tactic, break a process tree, or leave a chain incomplete even when the model would have flagged the events. This is why log loss is not just a data quality issue; it is a detection failure.

Reliability goals for a security log pipeline

The system’s reliability objective is narrow and practical: prevent log loss and recover from disconnections or restarts, while logging incidents as alerts. That objective is easier to state than to implement because security pipelines often mix fast analysis with durable storage. The lab project treats delivery as at least once, not exactly once. This means duplicates are possible, and the design must make retries safe. The pipeline also uses backpressure, disk buffers, checkpoints, acknowledgements, and reconciliation. A slow sink should not cause a source to discard data. A service restart should not erase queued batches. A missing analyzed document should be detectable by comparing the analyzed index with the raw archive. The goal is not to promise perfect hardware, but to make failures visible, bounded, and recoverable.

How the lab project moves logs without losing them

Independent pipelines and disk buffering

Vector 0.58.0 is configured with two independent pipelines, R and G, both reading /var/ossec/logs/archives/archives.json. Pipeline R sends raw logs to the indexer in chain-raw-* indices. Pipeline G sends batches to the correlation service at local port 8901/v1/events for normalization, scoring, and correlation. The pipelines have separate checkpoints and disk buffers, so one slow or failing path does not block the other. This separation matters because the raw path is a durability and audit path, while the analysis path is a latency and detection path. The lab settings use batching tuned for near-real-time display: pipeline R batches up to 100 events or 2 seconds, and pipeline G batches up to 10 events or 1 second. The disk buffer is a write-ahead log that syncs every 500 ms, checksums events, and survives restarts. The minimum disk buffer is about 256 MiB, and the lab uses max_size 268435488 bytes. The when_full policy is block, which preserves order instead of dropping newest events.

Checkpoints, acknowledgements, and delivery semantics

The source keeps file_key and offset_key checkpoints so Vector can resume from the saved byte position after a restart. The lab starts with read_from end, then later resumes from the checkpoint. Delivery is at least once, so possible duplicates are expected. End-to-end acknowledgement requires both disk buffering and enabled sink acknowledgements. In the lab, acknowledgements are enabled. If the correlation service rejects a full queue, Vector keeps the batch instead of discarding it. Pipeline G retries with intervals starting at 1 second and rising to a maximum of 30 seconds, with a request timeout of 30 seconds. These settings create pressure upstream when a downstream component is unavailable, which is safer than silent loss.

Fault-tolerance mechanisms inside the system

Disk queues and atomic writes

The correlation service receives logs through POST /v1/events with bearer token authentication and a maximum request body of 20 MB. It drops non-Windows logs, calculates a document identifier, removes duplicates within the batch, scores the event with the two-tier logistic regression model, creates a document, and adds the batch to a disk queue. A background flusher writes to the indexer in bulk, with a maximum of 500 documents per bulk request. The disk queue stores each batch as JSON lines. It uses an atomic pattern: write a temporary file, fsync it, then rename it into the queue. A batch leaves the queue only after the indexer acknowledges it. The queue survives service restarts, partial temporary files are deleted, and permanently rejected batches are moved to a rejected directory. The service also runs under systemd with restrictions such as NoNewPrivileges, PrivateTmp, ProtectSystem, and ProtectHome, and it exits with code 2 on bad configuration so an invalid restart does not loop.

Idempotent identifiers and reconciliation

Idempotent writes are central to safe retries. The system uses _id = sha1(agent.id|channel|eventRecordID|systemTime). The same record produces the same identifier, so repeated delivery does not create a second logical event. If eventRecordID is missing, the service uses a content hash, but that record cannot be reconciled with the raw archive in the same way. duplicate_in_batch counts duplicates inside a batch, and incident_id overwrite avoids duplicate chain incidents. The correlator groups flagged events by process tree using Sysmon ProcessGuid and Security 4688 PID/PPID with lookup windows. A chain becomes an incident when it contains at least one high-confidence event and at least three distinct tactics. Because the process tree can be incomplete after a failure, the correlator rebuilds state on restart from chain-events-* for the last 24 hours by default. This rebuild helps restore chains that were in memory before the service stopped.

Health incidents and alerting

Reconciliation is a periodic safety net. The reconcile job runs every 300 seconds and compares _id values between chain-raw-* and chain-events-* over a 30-minute window, excluding the last 120 seconds. If a raw log exists but the analyzed event is missing for more than 10 minutes, the system records a reconciled_gap incident and re-scores the missing log from raw. If analyzed events exist but raw is missing for two consecutive checks, the system records raw_missing, which is unrecoverable from events alone. Health monitoring runs every 5 seconds and opens or closes incidents based on indexer write failures and successes. Incident types include indexer_unreachable, indexer_auth, queue_full, ai_error, reconciled_gap, raw_missing, and service_down. A clean stop under 2 minutes is not counted as service_down. Health events are written to health.jsonl, chain-health-*, and alerts.json, then surfaced through Wazuh system incident rules.

Failure tests and what they proved

The failure tests follow a simple principle: intentionally stop the indexer or service, fill the queue, delete analyzed records, and restart components, then measure lost logs, recovery time, correctness, and duplicate behavior. In the lab project, all tested scenarios resulted in zero permanently lost logs. The outcomes below show how the pipeline behaved under each failure.

Scenario	Inputs or affected records	Result
Blocking indexer port	284 test logs during 146 seconds	0 lost; queue cleared 7 seconds after unblock
Stopping indexer service	117 test logs during 116 seconds	0 lost; indexer ready after 18 seconds; queue cleared 6 seconds later
Deleting chain-events records	332 documents	0 lost; reconciliation re-ingested missing logs within 48 seconds
Stopping correlation service	210 test logs during 179 seconds	0 lost; Vector buffered logs; service recovered automatically
Restarting Vector	219 test logs	0 lost; temporary mismatch of 1 log resolved automatically
Restarting or killing correlation service	4 chains, 4,244 events, 1,527 nodes	Correlator rebuilt state correctly; no duplicate alerts

The last test is especially important because correlator state is not just storage. If the service restarts and the process tree is rebuilt correctly, alerts can continue without duplicate chain incidents. The lab project also used a controlled attack simulation to confirm that recovered logs can support chain alerts; from 695 total event records, the analyst needed to review only 2 chain alerts.

Practical checklist for building a resilient log pipeline

• Write all raw events to a durable archive before analysis, but account for storage cost.
• Separate raw archiving and detection pipelines so one failure does not block the other.
• Use disk buffering with backpressure and block when full to preserve order.
• Enable acknowledgements and at-least-once delivery, then design for duplicates.
• Use stable idempotent document IDs based on agent, channel, record ID, and system time.
• Keep file offsets and checkpoints so the collector can resume after restarts.
• Add a service-side disk queue that persists batches until indexer acknowledgment.
• Run periodic reconciliation comparing raw and analyzed indices.
• Emit health incidents for indexer unreachable, queue full, AI errors, stale heartbeat, and reconciliation gaps.
• Test indexer failures, service restarts, queue fills, and correlator rebuilds before production.
• Keep response automation in approval mode with never-block lists, TTLs, max active limits, and audit logs.

Key Takeaways

• Log loss can silently break process trees, tactic diversity, and attack-chain alerts.
• Independent pipelines, disk buffers, checkpoints, and acknowledgements reduce silent drops.
• Idempotent identifiers plus reconciliation make retries safe and gaps detectable.
• The lab project’s failure tests showed zero permanently lost logs across indexer, service, queue, and restart scenarios.
• Reliability must be tested as a security control, not assumed because the pipeline is running.

Related reading: Linking Windows Events into Attack Chains with Wazuh, Vector and a Behavior Model · Testing Attack-Chain Detection with MITRE Caldera in a Home Lab

Frequently Asked Questions

About the author