Anomaly detection

Every packet NetOtter captures or loads goes through a set of detectors that look for traffic that doesn't look normal:

• Connections: port scans, sweeps, floods, password guessing, connections to hosts outside your networks.
• Errors: services that refuse or don't answer, failing DNS, HTTP errors.
• Timing: regular check-ins (beaconing), traffic spikes.
• Content: protocols on the wrong ports, data hidden in DNS names.

New findings are printed as they happen, and a report sums them up at any time.

Everything here is computed from the packets' headers, capture times and the facts read from their data at capture time, which are saved with each packet. So a capture loaded with -I gives exactly the same report as it did live. The examples on this page come from the anomaly lab, a virtual network that only exists inside one machine, and its recorded captures (tools/lab/samples/).

The commands

Where	What
live, both modes	A new warning or critical finding is printed right away, three per tick at most (... and N more anomalies (w to see them)). Info findings stay quiet and only appear in w.
w	Every anomaly found, most severe first, with: its evidence; when it started and ended (in seconds since the first packet, like l); how many packets are part of it; and the packet that raised it (found at #<id>, which i <id> shows).
s	Traffic statistics: totals, a packets-per-second chart, top talkers, top services, top flows, hosts outside the local networks.
f anomaly <kind>	Shows only the packets of that kind of anomaly (the kinds are in the tables below), or of any with all. It also matches the packets sent before the anomaly was detected, like the first probes of a scan. See Display filters.
-c	Prints the statistics and the report after the packet list.
-I	Prints Anomalies in this capture: ... after loading.

In the analysis view, l ends each line that's part of an anomaly with a tag, like !scan, and i adds an Anomaly line per anomaly the packet is part of. Here is f anomaly scan, then l:

#13    [eth0]     0.630981  192.0.2.66:39744      -> 192.0.2.10:199        ttl 64  TCP     40 B  [SYN]  !scan
#14    [eth0]     0.631001  192.0.2.10:199        -> 192.0.2.66:39744      ttl 64  TCP     20 B  [RST,ACK]  !scan
#15    [eth0]     0.631037  192.0.2.66:38812      -> 192.0.2.10:113        ttl 64  TCP     40 B  [SYN]  !scan

Severities and colors

Label	Color	Meaning
[CRITICAL]	bold red	Almost certainly hostile, or very large: a scan of 500+ ports, a SYN flood, a burst ten times the limit.
[WARNING]	bold yellow	Worth a look: it matches a known attack or failure pattern.
[INFO]	cyan	Unusual, often legitimate, kept for context.

The colors only appear on a terminal that shows them (Windows 10 and later consoles included). In a pipe or a file the labels stay and the colors go, so netotter -c ... > report.txt stays readable. Renamings never touch the color codes.

What is detected

The limits are in includes/anomaly.h, one #define each. Rates are counted over a sliding 60-second window of capture time.

Connections

Kind	Name in w	When	Severity
scan	port scan	One source fails on 25 or more ports of one host within a minute. Once it starts, every port is counted exactly.	warning, critical from 500 ports
sweep	host sweep	One source fails on 20 or more hosts on the same port, or pings 20 or more hosts, within a minute.	warning
flood	request flood	One service (address, port, protocol) gets 300 or more new connections in a minute, from at least 3 clients.	warning
burst	request burst	One client opens 30 or more connections to one service in a minute.	warning, critical from 300
bruteforce	brute force	The same, on a login port: FTP, SSH, Telnet, SMTP, POP3, IMAP, LDAP, SMB, databases, RDP, VNC, WinRM, Redis, MongoDB.	warning, critical from 300
foreign	foreign host	A host outside the trusted networks (-T, below) took part in a connection.	warning
directip	direct IP	A host outside the local networks was reached, and no DNS answer or TLS / QUIC name was seen for it within 5 seconds. Malware often uses hardcoded addresses, but so do some legitimate tools.	info

Scans and sweeps count failed attempts only.

• Attempt: a TCP SYN, or the first UDP datagram of a connection.
• Failure: the attempt is refused (a TCP reset, or an ICMP port unreachable for UDP), or gets no reply within 3 seconds of capture time.
• Why: normal traffic succeeds. A browser opening a page reaches 30 servers on port 443 in a few seconds, and that isn't a sweep. A scan mostly hits closed or filtered ports.
• Consequences: unanswered probes only count once 3 more seconds of traffic were captured, and the scan of a host whose ports are all open goes unseen.

A connection only counts as made (for foreign and directip) once its TCP handshake completed, or traffic went both ways. A SYN flood from random addresses, answered by SYN-ACKs, makes no "connection" with those addresses.

DNS, mDNS, LLMNR, NetBIOS, NTP and SSDP are chatty by nature (each DNS query is a new connection), so their flood and burst limits are ten times higher.

Examples, from nmap -sT -p 1-1000 and a 100-port SYN scan, then sweeps, then floods:

[CRITICAL] port scan     192.0.2.66 -> 192.0.2.10: 998 ports/tcp refused or unanswered [+0.631827 s to +0.653212 s], 2004 packets, found at #69
[WARNING]  port scan     192.0.2.66 -> 192.0.2.20: 100 ports/tcp refused or unanswered [+0.799922 s to +0.801497 s], 200 packets, found at #2075
[WARNING]  host sweep    192.0.2.66 -> 64 hosts pinged (icmp) [+1.207331 s to +2.459089 s], 71 packets, found at #111
[WARNING]  host sweep    192.0.2.66 -> 64 hosts on port 445/tcp refused or unanswered [+3.712142 s to +3.854628 s], 67 packets, found at #167
[WARNING]  brute force   192.0.2.66 -> 192.0.2.10:22/tcp: up to 60 connections a minute [+2.059199 s to +3.597499 s], 601 packets, found at #320
[WARNING]  request burst 192.0.2.20 -> 192.0.2.10:8080/tcp: up to 50 connections a minute [+2.120048 s to +3.154422 s], 616 packets, found at #388
[WARNING]  request flood 192.0.2.10:80/tcp: up to 327 new connections a minute, from several clients [+0.743579 s to +0.748837 s], 3548 packets, found at #3273

998 of the 1000 ports, because SSH (22) and HTTP (80) are open on the server, so they answered. The packet count covers every probe and its answer, including the open ports.

Errors

Kind	Name in w	When	Severity
refused	refused	A service refuses at least half of 20 or more connection attempts in a minute: TCP resets, or ICMP port unreachable for UDP.	warning
unanswered	unanswered	A TCP service leaves at least half of 20 or more attempts unanswered (no SYN-ACK within 3 seconds): it's down, or a firewall drops the traffic.	warning
synflood	SYN flood	200 or more half-open connections in a minute, at least half of the attempts: the server answered the SYN, the client never acknowledged it. That's what spoofed SYN floods leave.	critical
nxdomain	NXDOMAINs	A client gets NXDOMAIN for at least half of 20 or more DNS answers, for 10 or more different domains. Malware that generates random domain names looks like this; mistyped names or search domains fail on a few names only.	warning
dnsfail	DNS failing	A resolver answers SERVFAIL or REFUSED to at least half of 10 or more queries.	warning
unreachable	unreachable	A host (usually a router) sends 20 or more destination unreachable messages in a minute (port unreachable aside: it goes to refused).	warning
httperrors	HTTP errors	A web server answers 4xx or 5xx to at least half of 20 or more requests. Web scanners trigger it with their 404s, a broken application with its 500s.	warning

UDP gets no unanswered: some UDP traffic never expects a reply (syslog, broadcasts, streams), so silence proves nothing. TCP must always answer.

[CRITICAL] SYN flood     192.0.2.10:80/tcp: 2309 of 2500 connections half-open (SYN-ACK never acknowledged) (92%) [+4.479969 s to +4.736853 s], 4821 packets, found at #4862
[WARNING]  refused       192.0.2.10:8081/tcp: 25 of 25 connections refused (100%) [+1.619329 s to +1.871481 s], 50 packets, found at #70
[WARNING]  unanswered    192.0.2.13:80/tcp: 22 of 22 connections unanswered (100%) [+7.277606 s to +8.879131 s], 22 packets, found at #120
[WARNING]  NXDOMAINs     192.0.2.20: 30 of 45 DNS answers are NXDOMAIN (66%), for 30 domains (random names?) [+1.543019 s to +2.920064 s], 90 packets, found at #65
[WARNING]  DNS failing   192.0.2.11: 15 of 15 DNS answers are SERVFAIL or REFUSED (100%) [+2.666401 s to +2.920064 s], 30 packets, found at #109
[WARNING]  HTTP errors   192.0.2.10:80/tcp: 25 of 25 responses are errors (100%), 8 server errors (5xx) in all [+1.666512 s to +1.925366 s], 297 packets, found at #260
[WARNING]  refused       192.0.2.10:514/udp: 25 of 25 connections refused (100%) [+1.611537 s to +1.863286 s], 50 packets, found at #69
[WARNING]  unreachable   192.0.2.1: up to 30 destination unreachable messages a minute [+5.055994 s to +5.056037 s], 30 packets, found at #139

In l, the packets themselves say what went wrong:

#12    [eth0]     0.579255  192.0.2.20:38261      -> 192.0.2.10:53         ttl 64  UDP     42 B  [DNS A?]
#13    [eth0]     0.579385  192.0.2.10:53         -> 192.0.2.20:38261      ttl 64  UDP     42 B  [DNS NXDOMAIN]
#15    [eth0]     0.683190  192.0.2.20:44090      -> 192.0.2.10:80         ttl 64  TCP     94 B  [PSH,ACK]  [GET]
#17    [eth0]     0.683662  192.0.2.10:80         -> 192.0.2.20:44090      ttl 64  TCP    176 B  [PSH,ACK]  [HTTP 404]
#17    [eth0]     0.655972  192.0.2.10            -> 192.0.2.20            ttl 64  ICMP    50 B  [port unreachable 192.0.2.10:514/udp]

Timing

Kind	Name in w	When	Severity
beacon	beaconing	A client reaches the same server and port at regular times: 8 intervals or more between connection starts, a median of 1 second or more, and a median deviation of at most 20 % of it.	warning (info for NTP, DHCP, NetBIOS, SSDP, mDNS, LLMNR, periodic by design)
spike	traffic spike	A second with 1000 packets or more, at least 5 times the usual rate and more than 4 standard deviations above it, after a 10-second warm-up. It's named after the host that sent the most packets that second.	warning, critical from 10,000 packets/s

Beaconing. Malware checks in with its command server on a timer, often with random jitter added so it looks less regular.

• What counts as a connection start: a new TCP connection, a new UDP flow (each DNS query is one), or a UDP datagram after a second of silence, so a beacon from a fixed UDP port is seen too. Keep-alives inside a long TCP connection don't count.
• The test: the median and the median absolute deviation of the last 32 intervals. One missed beat (a double interval) changes neither.
• What it means: plenty of legitimate software runs on timers too (update checks, telemetry, monitoring), so a beacon is a lead to follow, not a verdict.

Spikes. The usual rate is an exponentially weighted mean and variance of the packets per second (each second weighs 1/20). A spike second is kept out of it, so a long flood stays a spike instead of becoming the norm. A page load on a quiet network stays below the 1000 packets/s floor; a flood or a large download doesn't. s draws the rate as a chart, with the spike seconds marked:

Packets per second (one column = 1 s; peak 3870/s at +14 s):
  |.............*@+.....|
                ^^^      spike

Each column is one second, or the busiest second of a group of them for longer captures (60 columns at most). The levels go .:-=+*#%@ from nothing to the peak.

[WARNING]  beaconing     192.0.2.20 -> 203.0.113.9:80/tcp: every 1.911 s (jitter 8.0%), 12 connections [+16.381199 s to +22.364072 s], 151 packets, found at #145
[WARNING]  beaconing     192.0.2.20 -> 192.0.2.10:53/udp: every 1.498 s (jitter 2.1%), 12 connections [+12.543302 s to +17.101816 s], 24 packets, found at #77
[WARNING]  beaconing     192.0.2.20 -> 203.0.113.5:4444/udp: every 1.523 s (jitter 4.9%), 12 connections [+12.486085 s to +17.020113 s], 12 packets, found at #75
[WARNING]  traffic spike 192.0.2.66 sent the most: up to 3870 packets/s, usually 3.1/s, 3 seconds [+12.159318 s to +15.159317 s], 8008 packets, found at #2426

The spike filter shows that host's packets during its spike seconds.

Content

Kind	Name in w	When	Severity
mismatch	port misuse	The first data of a connection on a well-known port isn't the protocol that port is for: SSH on 443, HTTP on 53, random bytes on 80, an unknown protocol on 22...	warning
dnstunnel	DNS tunnel	A client sends 20 or more queries under one domain in a minute, for 20 or more different names, at least half of them long (a 24-character label or a 52-character name): data smuggled out in the names.	warning

Port misuse.

• Classification: when a packet is parsed, its data is classified by what it looks like, not by its port:
  • Protocols: TLS record, SSH banner, HTTP request or status line, DNS message, QUIC long header, NTP, SMTP / FTP / POP3 / IMAP greetings, RDP, SMB, the BitTorrent handshake.
  • Fallbacks: printable text, random-looking data (high entropy: encrypted or compressed), or other binary data.
  • In i: i shows the class.
• The check: the first data of each direction of a connection is compared with what the port should carry:

Port	Expected
21	FTP
22	SSH
25, 587	SMTP
53 (TCP and UDP)	DNS
80, 8080	HTTP
110	POP3
123/udp	NTP
143	IMAP
443, 465, 853, 993, 995, 8443	TLS
443/udp, 853/udp	QUIC
445	SMB
3389	RDP (or TLS)

Only the first data counts, so STARTTLS (a plain greeting, then TLS) is fine. Each w line names the first wrong thing seen, for example SSH instead of TLS: an SSH server hidden on 443 to get through firewalls, or a VPN.

DNS tunnels. NetOtter keeps no names, only hashes, so the report can't print the domain. Instead, found at #<id> points at a query, and i reads the name back from that packet.

[WARNING]  port misuse   192.0.2.10:443/tcp: SSH instead of TLS, 3 connections [+1.090547 s to +1.493129 s], 30 packets, found at #25
[WARNING]  port misuse   192.0.2.10:53/tcp: HTTP instead of DNS, 2 connections [+1.820631 s to +2.022970 s], 24 packets, found at #66
[WARNING]  port misuse   192.0.2.10:80/tcp: random-looking data instead of HTTP, 2 connections [+2.348103 s to +4.551631 s], 16 packets, found at #90
[WARNING]  DNS tunnel    192.0.2.20: 40 queries under one domain, 40 different names, 64 chars long on average (data in the names?) [+1.541847 s to +2.556860 s], 80 packets, found at #68

i 25 and i 68 then show:

  Data         looks like SSH
  Anomaly      [WARNING]  port misuse   192.0.2.10:443/tcp: SSH instead of TLS, 3 connections [+1.090547 s to +1.493129 s]

  DNS          query #65187, question dp45dfxm58yd0sxdfu69hlnymb61pmgxo6z2qak5yfrkd9iars.t.example.net (TXT)
  Anomaly      [WARNING]  DNS tunnel    192.0.2.20: 40 queries under one domain, 40 different names, 64 chars long on average (data in the names?) [+1.541847 s to +2.556860 s]

ARP and reverse DNS

Kind	Name in w	When	Severity
arpmoved	ARP change	An IPv4 address now answers ARP with another MAC than before (2 seconds or more apart): ARP spoofing, or a replaced network card or VM.	critical for the gateway, warning otherwise
arpconflict	IP conflict	Two MACs answer for the same address within 2 seconds, or it flips back to a former MAC: a duplicate address, or ARP poisoning fighting the real owner.	warning (critical for the gateway)
arpmulti	shared MAC	One MAC answers for 16 addresses or more within a minute: proxy ARP, a host with many addresses, or a spoofer.	info
ptrburst	PTR burst	One client asks for the names of 20 or more different addresses (reverse DNS) within a minute: a sniffer or a scanner naming what it sees. From one of this machine's own addresses it's info, since NetOtter's own lookups count too.	warning

How the ARP watch works.

• What it reads: every ARP request and reply says "this IP is at this MAC" for its sender. NetOtter remembers the MAC of each IPv4 address (4,096 of them at most) and reports when it changes.
• The gateway: it's found without any setting: the Ethernet destination of the traffic going outside the local networks is the gateway's MAC. A change of the gateway's address is critical, because all the outside traffic follows it, which is exactly what an attacker in the middle wants.
• Passive only: NetOtter never sends ARP. It only sees the ARP traffic that reaches its interface (broadcast requests always, unicast replies only on a hub, a mirrored switch port, or for its own machine). An attacker who poisons only one victim, with unicast replies, can stay invisible from another switch port.

[WARNING]  ARP change    192.0.2.20: now at 00:00:5e:00:53:21, was 00:00:5e:00:53:20 (ARP spoofing, or a replaced NIC / VM) [+4.607816 s], 4 packets, found at #38
[CRITICAL] ARP change    192.0.2.1 (the gateway): now at 00:00:5e:00:53:91, was 00:00:5e:00:53:01 (ARP spoofing, or a replaced NIC / VM) [+4.887763 s], 8 packets, found at #41
[WARNING]  IP conflict   192.0.2.20: claimed by 00:00:5e:00:53:66 and 00:00:5e:00:53:20, 1 change (a duplicate address, or ARP poisoning) [+0.426710 s], 3 packets, found at #9
[INFO]     shared MAC    00:00:5e:00:53:30 answers for 17 addresses (proxy ARP, a multi-address host, or a spoofer) [+0.482508 s to +0.482612 s], 17 packets, found at #70
[WARNING]  PTR burst     192.0.2.66: reverse lookups of 30 different addresses (a sniffer or scanner naming what it sees?) [+0.865103 s to +1.007724 s], 60 packets, found at #57

The arpmoved filter shows the ARP packets of that address, before and after the change:

#9     [eth0]     0.569847  00:00:5e:00:53:20     -> ff:ff:ff:ff:ff:ff             ARP      0 B  [who-has 192.0.2.1 tell 192.0.2.20]
#10    [eth0]     0.569925  00:00:5e:00:53:01     -> 00:00:5e:00:53:20             ARP      0 B  [192.0.2.1 is-at 00:00:5e:00:53:01]
#40    [eth0]     4.887714  00:00:5e:00:53:20     -> ff:ff:ff:ff:ff:ff             ARP      0 B  [who-has 192.0.2.1 tell 192.0.2.20]
#41    [eth0]     4.887763  00:00:5e:00:53:91     -> 00:00:5e:00:53:20             ARP      0 B  [192.0.2.1 is-at 00:00:5e:00:53:91]

Trusted networks: -T

"Unauthorized connections to foreign addresses" needs a list of what is authorized. Give it with -T <file>, one network per line:

# The office LAN and our cloud provider
192.0.2.0/24
198.51.100.0/24
2001:db8:0:1::/64

The file:

• A bare address is a /32 (or /128), and # starts a comment.
• The list isn't saved in .nto files: it's your policy, not part of the capture. Give -T again with -I.

What counts as outside:

• With -T: any host address outside these networks is foreign, private or not. Multicast, broadcast, loopback and link-local addresses never are.
• Without -T: there are no foreign findings. "Outside the local networks" (for directip and s) then means the globally reachable addresses: everything but the private, shared, loopback, link-local, documentation and other special ranges of the IANA registries.

Hosts outside the local networks: 2
  203.0.113.9      1.5 KiB  files.example.net
  203.0.113.5      1.0 KiB  (no name seen)

Anomalies: 2 warnings, 1 info
[WARNING]  foreign host  203.0.113.5: outside the trusted networks, 3 connections (first with 192.0.2.20) [+0.692616 s to +1.693640 s], 20 packets, found at #14
[WARNING]  foreign host  203.0.113.9: outside the trusted networks, 2 connections (first with 192.0.2.20) [+2.389293 s to +2.389998 s], 20 packets, found at #57
[INFO]     direct IP     192.0.2.20 -> 203.0.113.5: reached with no DNS answer or TLS name seen for it [+0.692616 s], 20 packets, found at #14

203.0.113.9 was looked up first (files.example.net), 203.0.113.5 wasn't.

The statistics: s

Statistics: 113 packets, 7.4 KiB at the IP layer, over 13.311889 s; 32 flows, 19 hosts
Packets per second (one column = 1 s; peak 37/s at +2 s):
  |#*@ .. .....:.|
Top talkers (sent / received):
  192.0.2.20    33 pkts / 1.8 KiB  out, 23 pkts / 1.8 KiB  in
  203.0.113.9   8 pkts / 784 B  out, 12 pkts / 752 B  in
  192.0.2.10    8 pkts / 618 B  out, 8 pkts / 490 B  in
  203.0.113.5   8 pkts / 461 B  out, 12 pkts / 612 B  in
  198.51.100.7  4 pkts / 392 B  out, 6 pkts / 380 B  in
Top services (connections):
  192.0.2.10:53/udp    8 connections from 2 clients
  203.0.113.5:443/tcp  3 connections from 1 client
  203.0.113.9:80/tcp   2 connections from 1 client

A flow is one connection: addresses, ports and protocol, both directions together. Its client is the side that sent the first SYN, or else the one with the higher port.

Limits

• Memory is capped. NetOtter tracks at most 65,536 flows, 8,192 hosts and 8,192 services. Past that, the least recently seen are forgotten and s says the counts are partial. At most 4,096 anomalies are kept, and the rate chart covers the first day of a capture.
• Distinct counts are exact for ports. Hosts and domains are counted by 32-bit hash, so two of them can collide, very rarely.
• The capture has to keep up. On Linux, NetOtter reads through an 8 MiB memory-mapped ring, and Npcap has an 8 MiB buffer on Windows. When traffic outruns them, the tick line says how many frames the system dropped (the kernel dropped N), and the counts are lower than reality.
• Encrypted traffic is opaque. Detection works on headers, timing, the class of each connection's first data and the names seen in clear (DNS, TLS / QUIC SNI).
• Detection is passive. A careful attacker can stay under every limit, for example by scanning one port a minute. NetOtter sends nothing to test hosts (no probing for promiscuous network cards), so another sniffer is only seen through what it does, like resolving every address it sees (ptrburst) or poisoning ARP to get the traffic.