Learning: traffic anomalies

The other Learning pages take one packet apart. This one looks at many packets at once: patterns that no single packet shows, but that stand out when you count. Each section says what the pattern is, why it happens, what it looks like on the wire, and how NetOtter recognizes it (Anomaly detection has every limit). The traffic comes from the anomaly lab.

Port scans: knocking on every door

A scanner wants to know which services a host runs, so it tries to connect to many ports and watches the answers:

The scanner sends	The host answers	Meaning
SYN to port 80	SYN-ACK	open: a service listens
SYN to port 199	RST, ACK	closed: nothing there
SYN to port 9	nothing	filtered: a firewall dropped it

#13    [eth0]     0.630981  192.0.2.66:39744      -> 192.0.2.10:199        ttl 64  TCP     40 B  [SYN]  !scan
#14    [eth0]     0.631001  192.0.2.10:199        -> 192.0.2.66:39744      ttl 64  TCP     20 B  [RST,ACK]  !scan

What gives it away:

• Many ports, failing: one source reaches many ports of one host, and almost all fail.
• Fast: 1000 ports took 21 ms here.

A browser also opens many connections, but to a few ports, and they succeed. NetOtter therefore counts only failed attempts: 25 distinct failing ports in a minute make a scan.

A sweep is the same idea turned sideways: one port (or a ping) on many hosts, to find out which machines exist or which run a given service.

Floods and SYN floods: too much at once

Request floods. A flood tries to overwhelm a service with more requests than it can handle. Many clients opening connections to one service in a short time is a request flood. One client doing it alone is a burst, and on a login port (SSH, RDP, databases...) it's usually someone guessing passwords.

SYN floods. A SYN flood is cheaper for the attacker:

• The trick: it sends SYNs with random source addresses. The server answers each with a SYN-ACK to an address that never asked, so no final ACK comes back, and the half-open connection waits in the server's memory.
• On the wire: thousands of SYNs from addresses never seen before, and a server sending SYN-ACKs that are never acknowledged:

[CRITICAL] SYN flood     192.0.2.10:80/tcp: 2309 of 2500 connections half-open (SYN-ACK never acknowledged) (92%) [+4.479969 s to +4.736853 s], 4821 packets, found at #4862

The spoofed addresses never "connected" to anything (the handshake never completed), so NetOtter doesn't report them as foreign hosts.

Errors: what failures say

Failures are a signal of their own:

Seen on the wire	Usual cause
A TCP service answers RST to most connections	the program is down, or the port is wrong
SYNs get no answer at all	the host is off, or a firewall drops the traffic
ICMP port unreachable after UDP datagrams	nothing listens on that UDP port
ICMP host unreachable from the router	the destination doesn't exist, or the route is broken
Many DNS answers NXDOMAIN, for many different domains	random domain names: malware looking for its server (a domain generation algorithm)
A resolver answering SERVFAIL	the resolver is broken, or can't reach the authoritative servers
HTTP 404 by the dozen	a web scanner trying paths, or broken links
HTTP 500 by the dozen	a failing application

The NXDOMAIN row needs a closer look. A typo fails once. Search domains (www.example.com.corp.example) fail often, but always on the same suffix. Malware generating domains fails on a different domain every time. That's why NetOtter counts distinct domains, not just failures:

#12    [eth0]     0.579255  192.0.2.20:38261      -> 192.0.2.10:53         ttl 64  UDP     42 B  [DNS A?]
#13    [eth0]     0.579385  192.0.2.10:53         -> 192.0.2.20:38261      ttl 64  UDP     42 B  [DNS NXDOMAIN]
[WARNING]  NXDOMAINs     192.0.2.20: 30 of 45 DNS answers are NXDOMAIN (66%), for 30 domains (random names?) [+1.543019 s to +2.920064 s], 90 packets, found at #65

Beaconing: the clock gives it away

Malware on a compromised machine checks in with its command server on a timer: "any orders?" every minute. To look less mechanical, it adds random jitter (±15 %, say).

On the wire: each check-in is short and ordinary. The pattern only appears in the gaps between them, roughly like this for the lab's beacon:

connection starts:  0.0   1.9   3.8   5.7   7.6   9.5  11.4 ...  (seconds)
intervals:             1.9   1.9   1.9   1.9   1.9   1.9 ...

The test:

• What NetOtter measures: the median interval of each channel (client, server, port) and the median absolute deviation, i.e. how far a typical interval strays from the median.
• Why medians: with jitter the deviation stays small next to the median, and a missed check-in (one double interval) doesn't move either.

Human browsing has no such rhythm.

[WARNING]  beaconing     192.0.2.20 -> 203.0.113.9:80/tcp: every 1.911 s (jitter 8.0%), 12 connections [+16.381199 s to +22.364072 s], 151 packets, found at #145

Legitimate software beacons too: update checks, telemetry, NTP. A beacon is where to look first, not proof.

Spikes: the rate itself

A traffic spike is a second much busier than usual: a flood, a scan at full speed, a worm, or a large transfer. "Usual" has to adapt to the network, so NetOtter keeps a moving average of the packets per second and of their spread, and flags a second far above both. It keeps that second out of the average, so a long flood doesn't become the new normal.

Packets per second (one column = 1 s; peak 3870/s at +14 s):
  |.............*@+.....|
                ^^^      spike

Protocol misuse: the port is a promise

Port 443 promises TLS, port 53 DNS, port 22 SSH. Firewalls trust these promises, which is exactly why they get broken:

• Hiding a service: an SSH server on 443 passes where 22 is blocked.
• Bypassing filters: a tunnel over 53 passes where only DNS is allowed out.
• Unknown data: random bytes on port 80 (encrypted traffic where plain HTTP should be) deserve a look.

The first bytes of a connection tell the protocol, whatever the port:

Starts with	Protocol
16 03 0x	a TLS record (handshake)
SSH-2.0-	SSH
GET , POST , HTTP/1.1	HTTP
a 12-byte header and a valid question	DNS
220 and SMTP / FTP	a mail or FTP server greeting

So NetOtter classifies the first data of each direction and compares it with the port:

[WARNING]  port misuse   192.0.2.10:443/tcp: SSH instead of TLS, 3 connections [+1.090547 s to +1.493129 s], 30 packets, found at #25

DNS tunnels: data in the names

Almost every network lets DNS out, so DNS can smuggle data. The client encodes the data in the name it asks about, under a domain whose server the attacker controls:

1929jcuuts46z7b93emsi9mxw7spsxf5sb9iyqqkfhel929933.t.example.net

Every query carries a new chunk. The answers bring data back, often in TXT records.

What gives it away: many queries under one domain, every name different, with long labels that look random.

• Normal browsing: asks for a few names, again and again.
• A tunnel: asks for a new, long, random name every time.

[WARNING]  DNS tunnel    192.0.2.20: 40 queries under one domain, 40 different names, 64 chars long on average (data in the names?) [+1.541847 s to +2.556860 s], 80 packets, found at #68

Some legitimate services also put hashes in DNS names (anti-spam lookups, some CDNs), which is why the finding is a warning to check, not a verdict.

ARP: who is at this address?

On a local network, a machine finds the MAC behind an IP address by asking everyone: who has 192.0.2.1? The owner answers 192.0.2.1 is at 00:00:5e:00:53:01. Nobody checks the answer, and machines also accept answers they never asked for. That's ARP spoofing: an attacker announces "the gateway is at my MAC", and the victim sends it all its outside traffic, which it can read before forwarding it. It's how a sniffer gets to see traffic on a switched network.

What gives it away: an address suddenly answering with another MAC, or two MACs answering for the same address, one after the other, as the attacker and the real owner fight. NetOtter remembers who answers for each address and finds the gateway on its own (the destination of the traffic going outside), so a change there is critical:

#9     [eth0]     0.569847  00:00:5e:00:53:20     -> ff:ff:ff:ff:ff:ff             ARP      0 B  [who-has 192.0.2.1 tell 192.0.2.20]
#10    [eth0]     0.569925  00:00:5e:00:53:01     -> 00:00:5e:00:53:20             ARP      0 B  [192.0.2.1 is-at 00:00:5e:00:53:01]
#40    [eth0]     4.887714  00:00:5e:00:53:20     -> ff:ff:ff:ff:ff:ff             ARP      0 B  [who-has 192.0.2.1 tell 192.0.2.20]
#41    [eth0]     4.887763  00:00:5e:00:53:91     -> 00:00:5e:00:53:20             ARP      0 B  [192.0.2.1 is-at 00:00:5e:00:53:91]
[CRITICAL] ARP change    192.0.2.1 (the gateway): now at 00:00:5e:00:53:91, was 00:00:5e:00:53:01 (ARP spoofing, or a replaced NIC / VM) [+4.887763 s], 8 packets, found at #41

The same signals have innocent causes: a replaced network card, a VM moved to another host, a failover router, two machines given the same address by mistake. The report says both.

Another sniffer. A sniffer that names what it sees asks the DNS for the name of every address it captures (reverse lookups, PTR queries). Dozens of reverse lookups from one host in a minute are that, or a network scanner doing the same:

[WARNING]  PTR burst     192.0.2.66: reverse lookups of 30 different addresses (a sniffer or scanner naming what it sees?) [+0.865103 s to +1.007724 s], 60 packets, found at #57

NetOtter does it too in its analysis view (unless -N), so a burst from your own machine is reported as information only.