Testing Attack-Chain Detection with MITRE Caldera in a Home Lab

Attack-chain detection is difficult because useful security events rarely arrive as a single obvious signal. They appear as process creation records, registry writes, scheduled task changes, network connections, and other log entries that may be scattered across channels and hosts. The lab project built a defensive system on the Wazuh platform to score Windows event logs, link suspicious behaviors into attack chains, and help an operator review and respond. The reader will learn how the system was validated through deployment, model evaluation, Caldera-driven chain-linking tests, incident-response support tests, and resilience checks. All lab work should be done only in authorized, isolated environments.

A lab architecture built around Wazuh and a custom correlation service

The validation environment used a small set of machines rather than a production-scale deployment. A Windows victim host ran the Wazuh agent and Sysmon, while an Ubuntu host ran Wazuh 4.14.8, the correlation service, and Vector 0.58.0. A pfSense 2.8.1 firewall provided a network response point, and a separate Caldera server generated test behaviors on another network zone. The data path was intentionally simple: the Wazuh agent forwarded Windows event logs to the server, the server wrote raw events to an archive file, Vector sent batches to the correlation service, and the service normalized, scored, correlated, and exposed results through the Wazuh dashboard and a web interface.

Model validation before adversary emulation

Before Caldera was used for end-to-end testing, the lab project evaluated the scoring model itself. The model was trained from public Windows event datasets, Mordor and EVTX-ATTACK-SAMPLES, totaling 373 scenarios and 556,875 events. The malicious portion contained 92,894 events, about 16.7 percent of the training data. After deduplication, the training table contained 107,455 rows. The model is a two-tier logistic regression system. Tier 1 estimates whether an event is malicious, while Tier 2 assigns one of eight MITRE ATT&CK tactics plus an unknown label. The system uses two probability thresholds: events at or above 0.5 become high-confidence alerts, events between 0.3 and 0.5 become suspicious events, and events below 0.3 are treated as benign.

Validation area	Reported result
Tier 1 malicious detection on unseen scenarios	precision 0.844, recall 0.942, F1 0.890, PR-AUC 0.946, benign false positive rate 3.49%
Tier 1 malicious detection on unseen EVTX files	precision 0.667, recall 0.814, F1 0.733, PR-AUC 0.828
Tier 2 tactic assignment on unseen scenarios	macro-F1 0.696 with fill_unknown and 0.678 without fill_unknown
Tier 2 tactic assignment on unseen EVTX files	macro-F1 0.470

Binary detection and tactic assignment

The binary tier performed well on the cross-validation setting, but tactic assignment was less stable. The weakest tactics in the cross-validation result were persistence with an F1 score of 0.367 and execution with an F1 score of 0.483. This matters because attack-chain correlation depends on tactic diversity, not only on whether an event looks malicious. The system also does not predict techniques, and it does not cover collection or exfiltration. Those limitations are acceptable for a defensive lab validation, but they mean the correlator should be read as a prioritization aid rather than a complete narrative of every intrusion stage.

Using Caldera to test chain linking

Caldera was used as the adversary emulation engine for the lab experimentation, not as the training source. The test used two attacker addresses, 10.0.0.10 and 10.0.0.11, against one Windows victim. Two Caldera agents were involved: one with seven abilities and one with four abilities. The system flagged all 11 ability execution events as high-confidence malicious events, with probabilities from 0.96 to 1.0. The correlator then created two separate attack chains, one for each agent root process. Wazuh alert rules 112001 and 112002 generated dashboard alerts within 5 seconds of agent launch. Across the full event set, the analyst only needed to review 2 chain alerts from 695 total event records.

What the correlator had to prove

The chain-linking test was not just about catching individual malicious events. It checked whether the system could connect related events through process lineage and tactic diversity.

• Group flagged events by process tree rather than by time window alone.
• Require at least one high-confidence event before promoting a chain.
• Require at least three distinct ATT&CK tactics, excluding unknown, before creating an incident.
• Keep separate chains for separate root processes.
• Update chain information as new tactics appear in the same lineage.

The process-tree logic used Sysmon ProcessGuid values when available and Security event 4688 process identifiers when needed. The lab also showed why logging configuration matters: if relevant events are not collected, the chain cannot be built from evidence that never reached the system.

Alerting, operator review, and response support

The validation also measured whether the system could support an operator after detection. Chain alerts were written to a local alerts file, then surfaced in the Wazuh dashboard through custom rules. The web interface displayed chain details so an analyst could review the linked events without manually assembling them from raw logs. The responder component extracted indicators such as IP addresses, domains, and hashes from the chain and raw log context. It could enrich those indicators with lab simulation data and public VirusTotal information, then propose blocking malicious IP addresses on pfSense.

Response support was tested in controlled modes. The responder could run in off, dry-run, approve, or auto modes. In the lab, manual blocking of 10.0.0.10 was successfully added to the pfSense LAB_BLOCK alias, and a manual VirusTotal lookup for a sandcat.go file hash returned results. The design also included safeguards such as never-block lists, time-to-live limits, maximum active blocks, retries, and audit records. These safeguards are important because automated response can create availability problems if it is not bounded.

Resilience tests that protect the chain

Attack-chain detection can fail quietly if logs are lost during a restart, disconnection, or indexer outage. The lab project therefore tested fault tolerance with several failure scenarios. Each scenario was measured by whether logs were permanently lost, how quickly queues cleared, and whether the correlator rebuilt correctly.

Blocking the indexer port for 146 seconds while sending 284 test logs resulted in 0 lost logs, and the queue cleared 7 seconds after the block was removed. Stopping the indexer service for 116 seconds with 117 test logs also resulted in 0 lost logs; the indexer became ready after 18 seconds, and the queue cleared 6 seconds later. Deleting 332 chain-events records triggered a reconciliation job that re-ingested all missing logs within 48 seconds, with 0 lost. Stopping the correlation service for 179 seconds while sending 210 test logs resulted in 0 lost logs, and the service recovered automatically.

Restarting Vector with 219 test logs produced a temporary mismatch of 1 log that resolved automatically, again with 0 permanently lost logs. Restarting or killing the correlation service tested the correlator rebuild path. After recovery, the correlator rebuilt state correctly with 4 chains, 4,244 events, and 1,527 nodes, without duplicate alerts. These results show that the pipeline used disk buffering, checkpoints, idempotent identifiers, and reconciliation to tolerate at-least-once delivery while avoiding permanent loss.

What the results say about detection quality

The lab results point to a practical pattern: the system was strongest when it could combine high-confidence event scoring with process-tree correlation. The Caldera test compressed 695 event records into 2 chain alerts, which is a meaningful reduction in analyst workload. The near-real-time path also helped the operator see updates quickly. The Vector pipeline for AI processing used batches of at most 10 events or 1 second, and the measured median latency for that stage was 2.2 seconds with a range of 1.6 to 3.0 seconds, excluding the earlier Windows-to-Wazuh transport and model scoring time.

The evaluation also exposed limits. Binary detection was strong on the cross-validation setting, but tactic assignment dropped on unseen EVTX files, with macro-F1 0.470. The correlator is single-host, so it cannot detect cross-host lateral movement from the tested log sources. It also lacks collection and exfiltration tactics, does not analyze network traffic, and does not perform malware file analysis beyond hashes already present in logs. The lab Sysmon configuration did not collect module loading or process access events, which means some behaviors cannot be directly evidenced. These are not failures of the lab validation; they are boundaries that an operator must understand.

Still, the defensive value is visible. The two-threshold model separates high-confidence alerts from suspicious events, reducing alert fatigue. The chain view gives context by grouping isolated events into a process lineage. The response support path keeps human approval in the loop while still preparing blocking actions. That combination is useful for resource-constrained teams because it automates correlation and indicator extraction without removing operator judgment.

Key Takeaways

• The lab project validated a Wazuh-based system that scores Windows events, links them into attack chains, and supports operator response.
• Caldera-driven testing showed the system could flag all 11 ability execution events and create two separate attack chains from 695 event records.
• Model results were strong for binary detection but weaker for tactic assignment, especially on unseen EVTX files.
• Fault-tolerance tests showed 0 permanently lost logs across indexer, service, queue, and correlator recovery scenarios.
• Response support remained human-in-the-loop, with approval modes, safeguards, and audit records around pfSense blocking.

Related reading: Linking Windows Events into Attack Chains with Wazuh, Vector and a Behavior Model · Building a Fault-Tolerant Log Pipeline for Security Monitoring

Frequently Asked Questions

About the author