Monitoring OWASP Juice Shop with Rules, Machine Learning and a Local LLM

Modern web applications face a constant barrage of automated and targeted attacks. Defenders need layered monitoring systems that can detect a wide range of threats while providing actionable context to analysts. This article explores a lab project that built an integrated Network Security Monitoring (NSM) system for the OWASP Juice Shop, a deliberately vulnerable training application. The system combines traditional signature-based rules, behavioral analysis, machine learning (ML), and a locally-hosted Large Language Model (LLM) to identify and explain potential security events. We will examine the system’s functional requirements, its Docker-based architecture, the web attack classes it covers, and the insights gained from its operation. This defensive overview is intended for security practitioners and students looking to understand how multiple detection techniques can be combined in a controlled lab environment. All such lab work should be conducted only in authorized, isolated testing environments.

System Requirements and High-Level Architecture

The monitoring system, referred to here as the lab project, was designed with specific functional goals. It needed to capture HTTP traffic directed at the Juice Shop application, apply data redaction for privacy, perform hybrid detection using rules, ML, and behavioral analysis, store events and alerts durably, classify outcomes based on available evidence, leverage an LLM for alert explanation and remediation suggestions, and provide a dashboard and API for monitoring. A key non-functional requirement was deployment via Docker Compose, making the system portable and reproducible.

The architecture uses a microservices approach within a dedicated Docker network. Traffic flows from a user or attacker container through an Nginx ingress to a custom Collector service, which acts as a reverse proxy to the Juice Shop. The Collector is responsible for recording request metadata, applying redaction to sensitive fields, and forwarding normalized telemetry to a Processor service. The Processor is the core detection engine, housing the rule engine, behavioral detector, and ML models. It classifies events, generates alerts, and can call a locally deployed Ollama LLM service for analysis. A separate Fluent Bit agent collects Nginx access logs, while Prometheus scrapes metrics from the services. The state is preserved across restarts using named Docker volumes.

Component	Key Responsibility
Nginx	Ingress point, request ID injection, JSON logging
Collector	Reverse proxy, request/response capture, redaction, telemetry outbox
Processor	Event parsing, hybrid detection, outcome classification, API host
Ollama	Hosts the local LLM (e.g., qwen3:4b) for analysis tasks
Prometheus	Time-series metrics collection and storage
Docker Volumes	Persistent storage for events, alerts, models, and LLM data

Detection Methods: Rules, Behavior, and Machine Learning

The system employs a multi-layered detection strategy where alerts from different sources are combined using a logical OR operation; an alert from any source raises the overall flag.

Signature-Based Rule Engine

A rule engine evaluates incoming events against a set of predefined signatures defined in a YAML file. The rule set is designed to detect indicators of common web attacks without making claims about the existence of specific vulnerabilities in the backend. For instance, rules exist for SQL Injection, Cross-Site Scripting (XSS), Path Traversal, and OS Command Injection. Transformations for URL decoding and comment stripping are applied only as configured per rule. Notably, a rule for a single 404 response was disabled to limit false positives from benign missing resources like favicon requests.

Behavioral Detection

The behavioral detector analyzes request patterns from a source IP address over a sliding 60-second window. It tracks metrics like request rate, distinct paths accessed, and error ratios. Conditions are defined to flag suspicious activity:

• Path Enumeration: Triggered by accessing at least five different paths at a rate of 0.5 requests per second with an error ratio of 0.5 or higher.
• Suspicious Multi-Target Scanning: Requires at least 20 different paths, with at least five containing suspicious indicators from other detection sources.
• Repeated HTTP Errors: Flagged after observing at least 10 error responses at the defined rate threshold.
• Repeated Failed Login: Activated after five or more 401/403 responses from the login endpoint.

It’s important to note that IP address is used as a correlation key in this test environment and is not a reliable method for user attribution in real-world scenarios with NAT or proxies.

Machine Learning Models

Two ML models work in tandem. A supervised LinearSVC model is the primary content classifier. It was trained on a development dataset of 847 samples (split into train, calibration, and test sets) using character n-gram (2-6) features transformed via TF-IDF. The model outputs an attack probability score, and a decision threshold is applied to determine if the request is anomalous. This model is designed to analyze the content of HTTP requests, specifically focusing on extracted paths, parameter values, and structural markers while excluding environmental data like IP addresses or response statuses to avoid learning irrelevant patterns.

An Isolation Forest model serves as a baseline unsupervised anomaly detector. The HybridDetector can run the system in a “shadow mode,” where the baseline model’s result is used for alerting while the supervised model’s output is logged for comparison and evaluation. If the ML inference fails, the system marks the ML source as unavailable and continues evaluation with rules and behavioral analysis, ensuring resilience.

Integrating a Local LLM for Analyst Support

To augment analyst workflows, the system integrates a locally deployed LLM via Ollama. This design keeps potentially sensitive data within the lab environment and avoids external API calls. The LLM is tasked with analyzing generated alerts to provide a plain-language explanation, assess the available evidence, cite specific evidence IDs, and suggest remediation steps.

The LLM service uses a specifically engineered prompt to ground its responses. It is instructed to separate observed facts from hypotheses, list benign alternative explanations, note missing evidence, and recommend actions categorized as “verify,” “mitigate,” or “prevent.” For example, for a SQL Injection alert, the LLM might recommend verifying database audit logs and mitigating by implementing parameterized queries, while clearly stating that the alert itself does not prove successful exploitation.

The LLM can also be used to propose new detection rules based on analyzed events. This proposal process includes preflight validation checks against a schema and a set of benign samples to catch obvious false positives. Importantly, any LLM-generated rule remains in a draft state and requires explicit human operator validation and activation; there is no automatic deployment. This integration highlights the LLM’s role as an assistant, not an autonomous decision-maker, emphasizing that its outputs require human review for accuracy and appropriateness.

Coverage of OWASP Web Attack Classes

The system’s rules and ML model are configured to detect indicators associated with major OWASP Top Ten and other web application security risks. The enabled rule set provides signature-based detection for a wide range of attack classes, including:

• Injection flaws (SQLi, NoSQL, OS Command, LDAP)
• Cross-Site Scripting (XSS)
• Broken Access Control (Path Traversal, File Inclusion)
• Server-Side Request Forgery (SSRF)
• Security Misconfigurations (sensitive file probing)
• Other vulnerabilities like XML External Entity (XXE), Server-Side Template Injection (SSTI), and Prototype Pollution.

The behavioral detector adds a layer for identifying suspicious activity patterns that might not match a specific payload signature, such as credential brute force attempts and scanning/enumeration campaigns. The ML model contributes a content-based anomaly detection layer trained to recognize attack-like patterns in request structures. It is crucial to understand that detecting an “indicator” or “sign” is not equivalent to confirming a successful exploit. The system classifies outcomes based on the evidence available, using labels like “attempted” when a rule fires but independent verification (like an audit log) is absent.

Lab Environment and Experimental Scenarios

The lab environment is orchestrated using Docker Compose. A typical host setup for running the full system with the Local AI profile (including the Ollama LLM) might involve 16 GB of RAM and a GPU with 4 GB of VRAM to support efficient LLM inference, though a CPU-only configuration is also possible. The setup includes management scripts to start, stop, and reset the lab, where a reset clears monitoring data but preserves model files and LLM weights.

Evaluation scenarios were designed to test various aspects of the system:

Normal and Erroneous Traffic (TN01-TN02): These scenarios validate that normal user access does not generate alerts and that single 404 errors do not trigger the disabled rule, checking the system’s baseline false positive behavior.

Injection Attack Indicators (TN03-TN05): Scenarios simulate signs of SQL Injection, XSS, and Path Traversal/File Inclusion attacks. The tests verify that the corresponding rules (WEB-002, WEB-003, WEB-004/WEB-006) trigger appropriately. They also assess the LLM’s analysis, expecting it to explain the detected pattern (e.g., “SQL-like syntax in parameter”) while correctly noting the lack of proof of successful exploitation and suggesting relevant remediation steps from its playbook.

Behavioral Attacks: Other scenarios test the behavioral detector’s conditions, such as repeated failed logins or rapid path enumeration. Success is measured by the system generating the correct behavioral alert label (e.g., credential_brute_force) based on the activity within the defined time window.

In all scenarios, the evaluation focuses on whether the system correctly collects the request, applies the right detection logic, classifies the outcome, and provides useful supporting analysis through the LLM.

Key Takeaways for Defensive Monitoring

• Hybrid Detection Improves Coverage: Combining signature-based rules, behavioral analysis, and ML anomaly detection creates a more robust monitoring system that can catch both known attack patterns and unusual activity.
• ML Requires Careful Data Management: The effectiveness of supervised ML models depends on rigorous dataset construction, avoiding data leakage between training and test sets, and understanding that model performance is tied to the specific data and environment it was trained on.
• LLMs are Assistants, Not Autonomous Agents: Locally deployed LLMs can provide valuable context and speed up analyst workflows by explaining alerts and suggesting rules, but their outputs—including explanations and proposed remediations—require human verification to prevent hallucinated or incorrect guidance.
• Detection is Not Verification: A security alert is an indicator for investigation. The system emphasizes correlating alerts with independent evidence sources (like audit logs) where possible before concluding an incident, adhering to principles of digital evidence handling.
• Design for Data Protection: Building monitoring systems with data minimization, redaction of sensitive fields, and clear provenance tracking from the outset is essential for operational security and forensic readiness.

Related reading: Using a Local LLM to Explain Security Alerts: What Works and What Doesn’t

Frequently Asked Questions

About the author