Hm, this is an interesting one. We received a complaint today from some user about one of our servers. His IPCop/ Snort IDS had reported that one of our machines was portscanning his network. Naturally, this is a Bad Thing(tm), so we jumped into the OpMobiles and hit the OpPhones for some Late Night Colo Action(r)! (To clarify: I went to the home office. Mike went to the colo, the lucky bugger.)
...however, on first (and second) examination, the machine in question looks to be fine. We've checksummed all manner of binaries against known goods, checked the logs, etc. etc. To be safe, we downed and isolated the machine and sucked up the downtime while bringing up a warm backup.
My firewall shows numerous SYN FLOOD attacks against our network right around the time that the complaint scans occurred, so my first (well, second, after "FUCKITY FUCK") thought was that we've been hit with some manner of reflective attack or spoof, such as idlescan. That would be a bear, since there's no real easy way to prove that's what happened.
Next step: See if the 'easy script kiddie' version of idlescan works using us as a zombie. It shouldn't; we run current versions of the TCP/IP stack and custom-compiled versions of our servers, and I can say with confidence that we do not use sequential fragment/packet ID numbers, which renders us fairly useless as a zombie for idlescan. Of course, idlescan isn't new, and there's always the chance that the blackhats have figured out a new, snazzy means of doing it that I can't verify with a bog-standard Nmap and my own devious brain.
I have another look at the logs from the complainant. I take the time to curse the fact that we have been given super-basic info (essentially, 'detected a portscan from your host:port 80 to our host: 21 ports scanned in x seconds'). Now, a third possibility has occurred to my sleep-fogged brain: This could, in fact, be purely legitimate traffic.
Here's the scoop. The machine he's complaining about serves as, among other things, a public webserver for Ximian Red Carpet. It spends (some of) its time serving http requests for lots and lots of linux distro and software rpms that come in from machines running the Red Carpet client. This client is fairly quick, uses some really fast http client libraries, and can multithread.
Posit the following situation. I know nothing about this guy's network, but I can probably assume that he put his IDS somewhere near a network bottleneck (like his gateway) so as to catch the maximum number of packets. Now, if there are even a few machines on that network running a Red Carpet update simultaneously, then it is quite plausible that our server will be spraying large numbers of http packets back at his net. Let's look at the complaint again. In all cases, there seem to be '21 ports on 1 host' scanned. That number is significant. I don't know the defaults for Snort off the top of my head, but scanlogd assumes that accesses are portscans if 7 privileged ports or 21 unprivileged ports (or some weighted combo of the two) are accessed in a row with three seconds or less between accesses.
So it seems safe to assume, given that similar number of 21 ports, that whatever his IPCop is doing, it's using similar trigger numbers. The problem is this: let's suppose he has, somewhere on his network, several machines behind a NAT gateway. These machines are all running Red Carpet (for example, suppose they've all been set up to auto-update themselves late at night - maybe even at the same time). From his IDS' point of view, there will be a flurry of packets originating from our machine's http port, addressed to various high-numbered ports on the NAT gateway - and there will easily be more than the 21-in-63-seconds-on-different-ports required to trip the 'portscan' alert.
Well, that's it for now. I'm going to request additional data from the complainant. This is just my thought process during a potentional intrusion; I dunno if it's interesting or even relevant to anyone else, but it's useful for me to record for later perusal and reference.Posted by jbz at December 18, 2003 1:14 AM