The anomaly lab

tools/lab/ is NetOtter's regression test for anomaly detection. To prove that a detector works, it needs traffic that actually holds a port scan, a flood or a brute-force attack, and making that on a real network would disturb it. So the lab builds a small virtual network that only exists inside the machine, plays each attack there, records it with NetOtter, and checks what NetOtter reported.

• No root, nothing installed. Everything runs in a private user namespace (unshare --user --map-root-user --mount --net). The script is root there, with no power outside, and the whole network disappears when it ends.
• Nothing leaves the machine. The virtual hosts only see each other.
• Documentation addresses only (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 2001:db8::/32, MACs 00:00:5e:00:53:xx, example.com), so its captures are safe to publish.

Running it

make                         # or make asan: the lab then checks for sanitizer reports
tools/lab/check.sh           # every scenario, PASS / FAIL each
tools/lab/check.sh scan      # only some of them
tools/lab/lab.sh --list      # the scenarios and what they do
tools/lab/lab.sh scan        # one scenario, with its report
NETOTTER=/path/to/netotter tools/lab/check.sh

It needs iproute2, python3 and, for some scenarios, nmap. Each run writes tools/lab/out/<scenario>.* (git-ignored) and tools/lab/samples/<scenario>.nto, the recorded capture, which is kept in the repository as demo material:

./netotter -N -I tools/lab/samples/scan.nto -T tools/lab/trusted.txt

The Learning capture

tools/lab/learning.sh records the capture the user guide and the Learning pages are made from:

• The setup: two hosts on one virtual cable (192.0.2.1 and 192.0.2.2). The client runs a ping, DNS lookups over IPv4 and IPv6, an HTTP download, and an HTTP/3 attempt to a closed port.
• The outputs: tools/lab/samples/learning.nto, plus tools/lab/out/learning.pcap with the whole frames, which the hexdumps on those pages are cut from. The recorder is labnet.py record, standard library only.
• Two settings make it look like a real cable:
  • gso_max_segs 1 on the client's interface, so datagrams a program sends in one block (curl's HTTP/3 with UDP segmentation offload) are captured one by one.
  • The addresses are set before the cable comes up, so NetOtter knows them as its own from the start, and the capture runs past its 30-second refresh, so the link-local address is saved too.
• Rerunning it: a new run gives other ports, ids and random values, so the pages must be redone from it.

The network

            LAN 192.0.2.0/24, 2001:db8:0:1::/64 (a hub: everyone sees every frame)
  +---------+---------+-----------+------------+-------------+---------+
  gw .1     server .10  client .20  bots .30-.44 attacker .66  monitor .250
  |         DNS, HTTP 80/8080, SSH banner 22                   NetOtter listens here
  |   dns2 .11: a resolver answering SERVFAIL to everything
  |   dead .13: an HTTP server whose port 80 traffic is dropped on arrival (tc)
  |
  WAN 198.51.100.0/24, 203.0.113.0/24, 2001:db8:0:2::/64
  outside 198.51.100.7 (authorized cloud), 203.0.113.5 and .9 (foreign)

• The bridges have ageing_time 0, so they forward every frame to every port, like a hub. That way monitor sees the traffic of the whole LAN, as a sniffer on a mirrored switch port would.
• gw routes between the LAN and the WAN.
• The servers are labnet.py (standard library Python): a DNS server answering from zone.txt (NXDOMAIN for anything else), HTTP servers (/ 200, /missing 404, /error 500) and banner servers.
• trusted.txt is the -T file: the LAN and 198.51.100.0/24. 203.0.113.0/24 stays foreign.

What one scenario run does

1. Builds the network and starts the servers.
2. Starts netotter -N -c -i <monitor's eth0> -T trusted.txt -O samples/<scenario>.nto in monitor, with XDG_CACHE_HOME in out/ so the real reverse DNS cache is never read.
3. Runs the scenario's scenario() function.
4. Makes 4 more seconds of DNS traffic from gw (not from a scenario's hosts, so their channels stay clean), so that unanswered attempts reach their 3-second timeout in capture time, then stops NetOtter with Ctrl-C (SIGINT), which prints the list, the statistics and the report, and saves the capture.
5. Loads the capture again (-c -I).
6. Checks that every EXPECT string is in the report and no FORBID string is, that the report of the reloaded capture is the same as the live one (detection only depends on the packets), and that no sanitizer complained.

Writing a scenario

A scenario is tools/lab/scenarios/<name>.sh:

# Desc: password guessing on SSH: 60 connections in a few seconds
EXPECT=("[WARNING]  brute force   192.0.2.66 -> 192.0.2.10:22/tcp")
FORBID=("request flood")
scenario() {
    labnet attacker connect 192.0.2.10 22 --count 60 --interval 0.05 \
        --send 'SSH-2.0-libssh_0.10\r\n'
}

• Helpers: on <ns> <command> runs a command in a namespace, labnet <ns> <subcommand> runs labnet.py there (connect, http-get, dns; see labnet.py --help).
• Expectations: EXPECT and FORBID are fixed strings searched in the report, from Statistics: to the end of the anomaly list.
• The baseline: baseline is ordinary traffic (DNS, web, SSH, pings) and must raise no warning. It's the false-positive guard: when a limit changes, run it first.

Current scenarios

Scenario	Traffic	Expected
baseline	DNS, web, SSH, pings, a browser reaching 30 servers on 443	no warning, no critical
scan	nmap connect scan of 1000 ports (2 open), SYN scan of 100	critical port scan with 998 ports, warning with 100
sweep	nmap port-445 sweep and ping sweep of 64 outside hosts each	two host sweep
bruteforce	60 SSH connections in 3 s	brute force
burst	50 HTTP requests to one API in 2.5 s	request burst
flood	15 bots, 330 connections, 22 each	request flood, no burst
foreign	connections to two hosts outside the trusted networks, one looked up first	two foreign host, one direct IP
errors-tcp	25 connections to a closed port, 22 to the dead host	refused, unanswered
synflood	hping3: 2500 SYNs from random addresses to the web server	critical SYN flood (and request flood)
errors-dns	30 lookups of random domains, 15 queries to the broken resolver	NXDOMAINs for 30 domains, DNS failing
errors-http	25 requests for missing pages and a failing script	HTTP errors with 8 server errors
errors-icmp	25 syslog datagrams to a closed port, pings past the router to absent hosts	refused (UDP), unreachable from the router
beacon	an HTTP check-in every 2 s ±15 % to an outside server, 12 times	beaconing (and the foreign host)
beacon-udp	a DNS TXT query every 1.5 s, and datagrams from a fixed UDP port every 1.5 s	two beaconing
spike	12 calm seconds of pings, then hping3 at about 4000 packets/s for 2 s	traffic spike from the attacker
mismatch	an SSH server on 443, an HTTP server on 53/tcp, random bytes sent to 80	three port misuse
dns-tunnel	40 TXT queries of 50-character random labels under t.example.net	DNS tunnel (and no NXDOMAINs: one domain only)
arp-moved	the client's MAC is changed with ip link, then the server reaches it again	ARP change (warning)
arp-gateway	the client reaches the outside, the gateway's MAC is changed, the client reaches the outside again	critical ARP change for the gateway
arp-conflict	the client's address is also given to another host, then the server looks it up	IP conflict
arp-multi	the server pings the 17 addresses of the bots host	shared MAC (info)
ptr-burst	dig -x on 30 addresses	PTR burst, and no NXDOMAINs (one domain: in-addr.arpa)

Every scenario uses ordinary traffic and configuration changes (ip link, ip addr, ping, dig, nmap, hping3); nothing in the lab forges ARP. The synflood capture holds random IPv4 source addresses (hping3's --rand-source): they're not real hosts, and nothing left the machine. IPv6 is turned off on the bridges and the host-side veth ends, so only the lab hosts' own addresses (made from the documentation MACs) appear.