The useful netstat
In order to know when your box has been hacked, you have to know
what your box is like under normal conditions. Something's got to
trigger the "that ain't right" reflex. [grin] There are times when the
hack is really obvious -- things like your website being replaced by
"Chinese hackers overcome pitiful US defenses!", to paraphrase a recent
common phenomena. Other times, it's not so obvious. One of my servers
was hacked this past summer, with surpassing subtlety. If I hadn't been
really paranoid (and somewhat lucky), I never would have realized they
So, what do you look for? Well, let's start with the list of
services that your box is running. Most of the time, successful remote
hacks are exploits against a service that your box is running. (I'm
talking attacks that give the black-hat priviliges and access to your
server, not ones that will just drop your server off the Internet map
for a while.) So it makes sense not to run more network services than
you need. Running everything under the sun increases your chances of a
hole being found in one of those services, and your box becoming
vulnerable to attack.
On most Linux systems, you'll have a command called netstat.
Simply put, it lets you know the status of your network. Running just
the bare command, like so:
[root@linuxbox /root]# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 www.linuxbox.:2944 astro.dal.net:ircd ESTABLISHED
tcp 0 0 www.linuxbox.o:ssh dragonhome.humanit:1594 ESTABLISHED
tcp 0 256 www.linuxbox.o:ssh scarlett.demon.di:39735 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 0 [ ] DGRAM 157112 /dev/log
unix 0 [ ] DGRAM 272
shows you the open sockets -- the connections between a local IP/port
pair and a remote IP/port pair. It doesn't include the servers, which
is what we're really interested in here. But we can see that someone on
this box is connected to dal.net for IRC, and that two users are ssh'd
in -- one from dragonhome.humanit[something], port 1594, and one from
scarlett.demon.di[something], port 39735. (The latter is me connected
to the box to use it as an example.) You can see that the remote
hostnames are truncated if they're too long. Both of these are.
Much more helpful is to run netstat with the -pl options. This
shows you the server processes that are listening too, and the processes
that are associated with them.
[root@linuxbox /root]# netstat -pl
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:ftp *:* LISTEN 532/tcpserver
tcp 0 0 *:smtp *:* LISTEN 462/tcpserver
tcp 0 0 *:www *:* LISTEN 436/httpd
tcp 0 0 *:ssh *:* LISTEN 421/sshd
raw 0 0 *:icmp *:* 7 -
raw 0 0 *:tcp *:* 7 -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
So we can see that this server is running ftp, mail, www, and
ssh. I'm using tcpserver to run my ftp and mail daemons, and Apache and
ssh are running standalone, without a superserver.
There are lots more options out there -- check out the man page
for netstat for a rather exhaustive list. If anyone feels like posting
the netstat info from their system for comment, we can go over what you
should and shouldn't see here. For the most part, if you don't know
what it is, you probably shouldn't have it listening on a port here.
Try killing and restarting some of your services. You'll see
the ports appear and disappear from netstat if they're running
standalone. (Some superservers answer for the service they maintain
whether it's up or not.)
Really common things that are hackable that you often see here
To remove something from here, you first have to kill the
process and then remove it from your startup scripts (usually in
/etc/init.d and then symlinked to /etc/rc?.d and/or /etc/rc.d/rc?.d).
- Rpcbind and other rpcservices -- portmap. If you're
deliberately running them, make them only accessible to your internal
network or whoever needs them with a firewall. If you don't know what
they are, you probably don't need them.
- Sendmail. Make sure you have a recent version, and keep it
patched. Securing sendmail is a whole other week's worth of discussion.
- Bind. Older versions are notorious for having holes. If you
don't need it, turn it off.
- Xwindows services. Unless you're acting as an X server to
remote Xwindows clients, X doesn't need to be listening on a port. You
can forward X connections over ssh if you need to -- it's much safer.
If you want to run these services but don't want to have them
listening on a port, there's usually some way to do that within the
configuration for the service itself. We can get into detail for those
if anyone has a specific service in mind.
There are other programs that do the same sort of thing -- lsof,
for example. But one thing that's really important is to check your
system both from the inside and from the outside. Use nmap or a similar
tool to portscan yourself. You should see the same list of services
available. If you see a service in one and not the other, start to be
suspicious. (I'll cover nmap in another mail -- this one's getting
I realize that I should say a word about basic troubleshooting
procedures, though I'm sure that most of you already know this.
Whenever possible, make only one change at a time. That way if
something breaks, you know what the likely culprit is. So if you're not
sure if you need a bunch of these services, turn one off, make sure
everything you need that might have been using it still works, then turn
the next one off, and so forth.
What's raw mode?
Our discussion of netstat output sparked a discussion
about raw sockets.
Selected responses to netstat output
> tcp 0 0 *:x11 *:* LISTEN 2751/X
With X11, you don't want to stop X, you just want to stop it
from listening on a port for other machines to connect to. (I'm
assuming that you don't want to serve X to any machine other than your
local one.) I actually did a Web page about this a few months back,
since this is such a common problem.
> tcp 0 0 *:auth *:* LISTEN 825/identd
You'll need it if you IRC from this box. Some people disable it
as a "leaks information" security concern. It's not a big deal either
way, I don't think. For a very locked-down system you'd want to turn it
off, but for a home system you're probably okay.
> tcp 0 0 *:x11-ssh-offset *:* LISTEN 2371/sshd
> tcp 0 0 *:6011 *:* LISTEN 2415/sshd
> tcp 0 0 *:https *:* LISTEN 1052/httpd
> tcp 0 0 *:6012 *:* LISTEN 2495/sshd
Do you use X forwarding for ssh? That's what the 6010
(x11-ssh-offset), 6011, and 6012 ports are. If you don't use it, turn
it off in /etc/sshd_config, or wherever such things live for your flavor
of ssh. I use OpenSSH, so for me it's
A false sense of security?
> I'll also note that this system is behind a NAT router, so maybe
> running unneeded services isn't a humongous risk, but I want to know
> how to do it right anyway.
NAT will save you from some things, but not everything. It's no
replacement for a good firewall. Much depends on how your router does
the NAT. If it's one-to-one address mapping, then you're not
significantly safer. If it's port forwarding, that is considerably
safer, because outside attackers are less likely to know your box is
there. Your NAT device can still get portscanned, of course.
> This computer was SuSE 7.1, then upgraded to 7.3. Nothing special has
> ever been done to make it secure because it runs behind a firewall.
> It operates on a network where the majority of computer run NT.
The firewall will protect you from external threats, but most
hacks happen from inside the company. I've seen varying statistics for
how many that is, but most estimates range from 60% to 90%. Internal
security is almost always weaker than external security, so it's easier.
And just think of how many people you know that hate their
I'm a big believer in defense in depth. Yes, I have a firewall
that protects my home network. My workstation has a firewall too. That
way, if somehow my husband's Windows box gets compromised, I'm okay. I
run logcheck (I agree with the plug; it is a great tool) so I'm aware of
problems on all my boxes. I have set key files on my workstation so
that they can't be changed, even by root. That way, if my box gets
compromised, the black hat will have to reboot it to trojan ssh. I'll
notice that reboot. Defense in depth basically means that you're
constantly thinking, "what if, what if, what if" , and you have lines of
defense to fall back to.
Of course, this also means that you spend a lot of time and
effort on network security. That's worth it for me, because I think
it's fun. It's worth it for my company -- they pay for me, after all.
[grin] But many people just don't see it as that much of a priority.
You just have to decide if the risk of being hacked is worth the cost of
maintaining a relatively secure network.
> I use rlogin and rcp, on my local net -- we have a router/firewall (a
> Linkys somethingorother) which doesn't pass most incoming ports, so I'm
> under the illusion that I can get away with things like that) because
> they're so darned convenient and I hate having to type passwords every
> time I want to move a file someplace. (Yes, I even have a .rhosts with
> the names of our local hosts in it.) Am I being foolish in trusting the
Only somewhat. [grin] Like most things in security, it
depends on how paranoid you want to be. You're probably okay for most
script kiddies and such with that, but if a skilled black hat decided to
have a go at you for some reason, that might be trouble. A quick google
for "Linksys Bugtraq" yields these...
> tcp 0 0 *:https *:* LISTEN 4299/httpd
> ### Why does httpd show up twice here with seemingly
> identical entries? ###
Look closely -- it's http and https. Ports 80 (TCP) and 443
(TCP), respectively. Your web server is listening for both sorts of
connections, normal and secure.
> tcp 0 0 *:www-http *:* LISTEN 1234/httpd
Keep it if you're meaning to run a Web server on this box.
Otherwise turn it off. (Note to the newbies -- you don't have to run a
Web server in order to see Web pages. You only need a Web client for
that, like a browser. Netscape, Opera, etc.)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 *:login *:* LISTEN 1110/inetd
Aaaah! login! Kill it! Kill it! [grin]
Er, I mean, how do you normally log in to your box? If this is
what you use, I would highly recommend using ssh remotely and/or a local
console login that doesn't listen on a port instead. There are a good
number of programs that transparantly use login or the r-tools (rlogin,
rsh, things like that), but they are usually reconfigurable to use ssh
> tcp 0 0 *:time *:* LISTEN 1110/inetd
Another one you probably don't need if you're not actively using
it. Time services and NTP are things that I constantly dither about.
They are insecure and hackable. But they're also really useful,
especially in a larger network, and as far as I know there's not a good
replacement for the insecure xntpd. So this one's a judgement call --
how paranoid do you feel like being?
> tcp 0 0 *:sco-sysmgr *:* LISTEN 859/ypbind
Ypbind isn't necessary unless you're using Sun's yp directory
> tcp 0 0 *:finger *:* LISTEN 1110/inetd
Another one that you probably don't need. It gives out
information about you -- though some people put their PGP keys in their
.plan file and then say "Finger me for my PGP key". There have been
holes in it. I generally turn it off, but it's not a tragedy if you
> tcp 0 0 *:sunrpc *:* LISTEN 531/portmap
More Sun networking. Turn it off if you're not using it.
> tcp 0 0 *:723 *:* LISTEN 543/rpc.statd
Again, unless you're doing Sun networking, kill it.
> tcp 0 0 *:telnet *:* LISTEN 1110/inetd
Oooh! Kill it, and especially kill it if you haven't patched
it recently. There was an exploit in pretty much all versions of telnet
derived from the BSD code base found about six months to a year ago.
This includes Linux, Cisco, and a bunch of other OSs. There are many
exploit scripts out there for it. Even if telnet weren't a really bad
idea from a security POV because of the cleartext passwords, the exploit
means you really don't want to be running it.
> tcp 0 0 *:ipp *:* LISTEN 597/cupsd
If you're not using print services, kill it.
> tcp 0 0 *:smtp *:* LISTEN 823/sendmail: accep
If you're running a local mail server that needs to accept mail,
keep it. If you just need to send mail using your local server, set
sendmail not to listen on a port:
to your sendmail.cf. That way it will only be listening to your
machine, and not to the Internet. (In your netstat output, you'll see
127.0.0.1:smtp instead of *:smtp.)
You may also want to add
to disable the VRFY command. (This keeps the SMTP server you send the
mail to from learning about the other accounts on your server. Some
SMTP servers need this to be on to accept mail, but I haven't run into
> tcp 0 0 *:700 *:* LISTEN 944/rpc.mountd
> Do I need the font server? I had trouble
> getting X to work without it, so I just ran it and everything was happy,
> but I got the impression that it wasn't really needed for a personal
> machine... maybe I misread the docs.
You do want the font server, so that you can have fonts working
properly for your local machine, just like you do want to keep the X
server so that you can have X working for your local machine. This is
mostly a problem grown out of the way that computing in Unix has evolved
-- when Xwindows was written, a common setup was to have one machine
doing most of the processing and a bunch of dumb terminals using its X
and XFS power. Nowadays it's common to only have the one machine you're
on needing to access X and XFS. So you do still need the server (as far
as I know -- if someone knows a way to do it without, please do speak
up) -- it's just that it's only serving the one machine it's on.
You can still run X without xfs as long as you're not sharing the Xfonts with
other machines (XDMCP or dumb terminals, this doesn't apply if you run X
look at your XF86Config (This is for XFree86 4.1 but works on all X that I
know of), find the "Files" section
look for FontPath, if you see it looking for a socket or a port,you're using
xfs, you can simply change it to list all your font directories.
Here's my section as an example:
# Multiple FontPath entries are allowed (which are concatenated together),
# as well as specifying multiple comma-separated entries in one FontPath
# command (or a combination of both methods)
# If you don't have a floating point coprocessor and emacs, Mosaic or other
# programs take long to start up, try moving the Type1 and Speedo directory
# to the end of this list (or comment them out).
> If one has a need for ftp, what's a good server? Redhat uses wu-ftp,
> which I've heard is about the worst as far as security holes?
That is pretty much the worst. I use Pure-FTPd
(http://www.pureftpd.org/), which has a really good security history.
It's not a terribly mature program, but I've had good luck with it for a
couple of years. It also supports the ability to create its own
"virtual users", if you wanted to have FTP-only users with no shell.
I know other sysadmins that hated the way Pure does virtual
users, though. One such favors Pro-FTPd (http://www.proftpd.org/),
which had a showstopper bug a year or so back, but that's been fixed
now. (It was the same file globbing bug that was in wu-ftpd -- a long
string of ls's gave the remote hacker root.)
There's also a program called nc-FTPd that's an FTP server
coded with security in mind. It's not free (in either sense), so I
haven't checked it out yet. http://www.ncftpd.com/ if that's something
you're interested in.
Raven has already listed Pure-FTPd, Pro-FTPd, and nc-FTPd.
One I've heard about in this connection is vsftpd. I am told
the name Chris Evans is familiar to bugtraq readers. He's the
Copyright (c) 2002 by Raven Alder. This material
may be distributed only subject to the terms and
conditions set forth in the Open Publication License,
v1.0 or later (the latest version is presently
available at http://www.opencontent.org/openpub/).