Random Forest vs LSTM for DGA Detection: Generalizing to Unseen Malware Families

Detecting algorithmically generated domains is a cornerstone of modern network defense, but choosing the right machine learning model involves critical trade-offs between accuracy, generalizability, and operational practicality. This analysis delves into a direct comparison between two prominent approaches: a Random Forest classifier using handcrafted features and a character-level Long Short-Term Memory (LSTM) neural network. We examine their performance on a balanced dataset of two million domains, their ability to generalize to completely unseen malware families, and the crucial impact of decision threshold selection on false alarm rates. All lab work described should be conducted only in authorized, isolated environments.

Dataset Composition and Model Training

The foundation for this comparison was a meticulously curated dataset of two million domain names, evenly split between one million benign domains from the Alexa Top 1 Million list and one million malicious domains sourced from public DGA datasets representing 58 distinct malware families. After deduplication and balancing, this data was split into an 80/20 stratified partition, resulting in 1.6 million domains for training and a held-out test set of 400,000 domains. Two primary models were trained and evaluated: a Random Forest classifier with 100 trees using five handcrafted linguistic and structural features (label length, Shannon entropy, vowel-to-consonant ratio, digit ratio, and distinct character ratio), and a character-level LSTM network processing raw domain strings.

Performance on the Standard Test Set

Evaluated on the balanced test set of 400,000 domains with a default decision threshold of 0.5, the LSTM model demonstrated a decisive advantage in overall detection capability.

Model	Accuracy	Precision	Recall	F1-score
Random Forest	88.91%	91.18%	86.15%	88.59%
LSTM (Primary Model)	98.80%	99.23%	98.36%	98.79%

The confusion matrices reveal the practical impact of these metrics. The Random Forest model produced 27,700 false negatives (missing 13.85% of DGA domains) and 16,678 false positives (flagging 8.34% of benign traffic). In stark contrast, the LSTM model missed only 3,283 DGA domains (a 1.64% miss rate) and generated 1,524 false positives (a 0.76% false positive rate). This order-of-magnitude improvement in error rates underscores the LSTM’s superior ability to learn complex patterns directly from character sequences, far exceeding the discriminative power of the five handcrafted features alone.

The Crucial Test of Generalization

A model’s performance on a standard test set, while important, does not fully capture its operational value. A more rigorous test is its ability to correctly identify domains from a DGA family that was completely excluded from the training data. This “leave-one-family-out” evaluation was conducted for several key malware families, with revealing results.

Generalization to Unseen DGA Families

The LSTM model showed remarkable resilience when confronted with unseen random-string DGA families. For instance, when the rovnix family was held out, the LSTM’s recall remained at 99.9%, while the Random Forest’s recall plummeted to 1.6%. For other random-string families like emotet and necurs, the LSTM’s recall dropped by a maximum of only about three percentage points. However, both models struggled with more structured or novel DGA techniques. The seed-based banjori family proved challenging, with LSTM recall falling to 52.2% and Random Forest to 26.6% when the family was unseen. The greatest challenge came from word-list based families like suppobox and matsnu; when these families were excluded from training, both models nearly failed to detect them at test time, with recall estimates falling to near zero. This highlights a fundamental limitation: models cannot reliably detect DGA patterns they have never encountered if those patterns closely mimic legitimate language.

Precision-Recall Trade-off and Threshold Tuning

The binary classification output of an LSTM is a probability score between 0 and 1. The default threshold of 0.5 is often not optimal for security operations, where the cost of a false positive (wasting analyst time) must be balanced against the cost of a false negative (missing a threat). A sweep of this decision threshold for the LSTM model quantified this trade-off.

• At default threshold (0.5): Recall was 98.36% with a false positive rate (FPR) of 0.76%.
• At a raised threshold (~0.97): Recall remained high at 95.4%, but the FPR dropped dramatically to 0.10%.

This adjustment results in 7.6 times fewer false alarms for a recall penalty of less than three percentage points. For a security operations center (SOC), this tuning is critical. Deploying the model at the higher threshold of 0.97 would drastically reduce alert fatigue while maintaining a very high detection rate, making the system far more operationally viable.

Operational Performance and Considerations

Beyond pure accuracy, a model must perform efficiently in a live pipeline. In the lab’s integrated monitoring system, the LSTM model processed domains with a median time of 68.6 milliseconds and an average of 78.5 milliseconds per domain, translating to a throughput of approximately 13 queries per second per processing thread. The end-to-end latency—from DNS query to alert—had a median of 1.12 seconds, with the bulk of the delay attributed to logging buffering in the network sensor (Zeek), not the model inference itself. This demonstrates that the computational cost of the LSTM is not a bottleneck for real-time detection in many enterprise settings. Furthermore, the compact size of the trained LSTM model (373 KB) facilitates easy deployment and integration.

The Role of Random Forest and Interpretability

While outperformed by the LSTM, the Random Forest model provides significant value as an interpretable baseline. Feature importance analysis from the Random Forest revealed that among the five handcrafted features, label length, vowel-to-consonant ratio, and entropy were the most significant predictors. This insight is useful for analysts and for crafting simple, explainable detection rules. In fact, a rule generated based on such insights (length >10, entropy >2.8, vowel-consonant ratio <0.4) achieved a recall of 52.95% on the test set, albeit with a higher FPR of 3.69%. This illustrates that while rule-based methods derived from model insights can provide a layer of explainability, they are not a substitute for the high accuracy and generalization of the learned model itself.

Integrating DGA Detection into Defensive Operations

Detecting Domain Generation Algorithms (DGAs) is not merely an academic exercise; it is a critical component of modern Network Security Monitoring (NSM) and directly maps to adversarial behavior documented in the MITRE ATT&CK framework. DNS is a foundational and often poorly monitored protocol, making it a prime vector for malware Command and Control (C2). The technique of using algorithmically generated domains for resilient C2 communications is formally cataloged as MITRE ATT&CK T1568.002: Dynamic Resolution – Domain Generation Algorithms. A dedicated detection system serves as a specialized Network Intrusion Detection System (NIDS) focused on this specific technique.

Deploying such a system requires strategic placement within the network architecture. The primary deployment model involves monitoring traffic at the internal DNS resolver or at network egress points to capture DNS queries from all internal hosts. This allows for the detection of infected hosts attempting to call home to a C2 server. However, the rise of encrypted DNS (DoH/DoT) presents a significant challenge, as the query content is hidden from passive sensors. In such environments, the monitoring point must shift to the endpoint itself or the internal DNS resolver before the query is encrypted and forwarded to an external provider.

The system’s architecture is designed as a modular pipeline: traffic collection (e.g., using Zeek), feature extraction, model inference, and alerting. This modularity allows for components to be swapped or upgraded. For instance, the primary LSTM model, chosen for its 98.80% accuracy and superior generalization, can be replaced with an even more advanced unified model combining an Attention-BiLSTM network with 28 mathematical and linguistic features, which achieved 99.8% accuracy in testing. The output is actionable security alerts that can be fed into a SIEM or SOAR platform for orchestrated response.

Practical Recommendations and Inherent Limitations

To operationalize a machine learning-based DGA detector, defenders must move beyond raw model accuracy and consider real-world performance, tuning, and inherent system constraints. The following are key defensive takeaways and deployment considerations.

• Threshold Tuning is Non-Negotiable: The default model threshold of 0.5 is rarely optimal for production. Analysis shows that for the LSTM model, raising the threshold to approximately 0.97 reduces the false positive rate from 0.76% to 0.10%—a 7.6-fold decrease—while only lowering recall from 98.4% to 95.4%. This drastic reduction in false alarms is essential to maintain analyst trust and operational efficiency.
• Understand Generalization Boundaries: No model detects everything. While the LSTM generalizes exceptionally well to unseen random-string DGA families (e.g., maintaining 99.9% recall for rovnix), its performance drops significantly for structured algorithms like banjori (52.2% recall) and fails almost completely for unseen word-list-based families like suppobox and matsnu. Defenders must supplement ML detection with other intelligence, such as threat feeds, to cover these blind spots.
• Monitor Performance Characteristics: In operational tests, the LSTM model processed domains at a median speed of 68.6 ms each. However, the end-to-end latency from query to alert had a median of 1.12 seconds, with the bulk of the delay coming from logging buffering in tools like Zeek, not the model inference itself. This is acceptable for detection but highlights that real-time blocking requires optimized pipelines, as demonstrated in an enhanced scenario where a lightweight NumPy inference engine achieved processing in under 6 ms per domain.
• Plan for Encrypted DNS: The system monitors standard DNS (UDP port 53). It cannot inspect encrypted DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT) traffic. Defenders must either enforce DNS traffic to an internal, monitorable resolver or deploy endpoint-based solutions to maintain visibility.
• Maintain and Update Dictionaries: The primary weakness against word-list DGAs stems from their use of legitimate dictionary words. The linguistic analysis features (like dictionary match ratio) in advanced models depend on updated word lists to remain effective against new variants.

Furthermore, several technical limitations must be accounted for during deployment. The system extracts the Second-Level Domain (SLD) using a standard IANA TLD list, which may incorrectly parse multi-level suffixes like .co.uk or .com.br, leading to erroneous feature calculation. The training data for benign domains came from global popularity lists (e.g., Alexa), which may not accurately represent the traffic patterns of a specific organization, potentially affecting the false positive rate. Finally, the system performs per-domain analysis in isolation. It does not incorporate temporal or behavioral features, such as query frequency or failed resolution attempts, which could provide valuable additional context for detection.

In summary, a DGA detection system is a powerful tool that excels against a wide range of malware families and can be integrated with automated response workflows. However, its effectiveness is bounded by the types of DGAs, the need for careful threshold calibration, and the evolving challenge of encrypted traffic. It should be deployed as a core component of a layered defensive strategy, not as a standalone solution.

Key Takeaways

• LSTM Superiority for DGA Detection: The character-level LSTM model significantly outperformed feature-based models like Random Forest, achieving 98.80% accuracy and superior generalization to unseen random-string DGA families.
• Generalization is Not Guaranteed: Both models struggled to detect unseen DGA families that use word-list or highly structured generation methods, highlighting a key area for future dataset and model improvement.
• Threshold Tuning is Essential for Deployment: Raising the LSTM’s decision threshold from 0.5 to approximately 0.97 reduced the false positive rate by 7.6x (from 0.76% to 0.10%) with only a minor drop in recall, a critical adjustment for operational SOC use.
• Operational Viability Confirmed: The LSTM model demonstrated sufficient inference speed (~70ms median) to not be a bottleneck in a real-time detection pipeline, with most latency coming from network sensor logging.
• Complementary Value of Interpretable Models: While less accurate, Random Forest provides crucial interpretability, revealing which handcrafted features are most salient and serving as a benchmark for simpler, explainable detection logic.

Frequently Asked Questions

Related reading: DGA Domain Detection with Machine Learning: A Practical DNS Monitoring Guide

About the author