I will never forget my first ping. Working on Unix systems was never the same after that. Working on the command line was always a thrill, but understanding how Unix systems work together and how to do basic network troubleshooting brought the work to a new high. Let's run through the most useful networking commands that you'll find on Unix systems and maybe suggest some options you maybe never tried.
ifconfig
One of the most basic networking commands is ifconfig. It will tell you about your network interfaces, the state that they're in, your assigned IP address(es), and even provide some counts of packets that have crossed the interface since the system was last booted. These days, you may see both ipv4 and ipv6 addresses.
bond0 Link encap:Ethernet HWaddr 78:2B:CB:44:BF:DA
inet addr:10.20.30.40 Bcast:10.20.30.255 Mask:255.255.254.0
inet6 addr: fe80::7a2c:deac:fe22:bbda/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:957499349 errors:0 dropped:17641 overruns:0 frame:0
TX packets:946779445 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:106377372276 (99.0 GiB) TX bytes:1399270021829 (1.2 TiB)
You might have a "bonded" interface (like the one shown above) in which case, multiple physical interfaces are combined to provide more throughput.
Notice the RX (receive) and TX (transmit) packets counts as well as the same basic statistics shown in bytes. These numbers may seem huge but the system was been running for roughly a year. The uptime command shows you this along with the load averages that indicate how hard this system is working to keep up with the load. In this case, the system isn't breaking a sweat. The 0.10 one minute load basically tells you that every 10th time the system checks, there is one process having to wait for acess to the CPU. The 0.01 fifteen minute numbers tells you we're seeing something waiting only one time in a hundred.
$ uptime 09:56:19 up 325 days, 18:16, 1 user, load average: 0.10, 0.06, 0.01
ping
Ping is one of the most basic networking troubleshooting commands. It might tell you that you can reach a remote system, but you can't assume a system is down if you don't get a response -- especially if it belongs to some other company as the ping command and, often, all members of ths icmp protocol family may be blocked somewhere along the route. If you get a response, you know the system you're pinging is up, but you also know that your network connection is good. I often ping a system somewhere else in the US (one on which I had an account mny years ago) as a test of my network connection. If I can get a response from a system several states up the coast, I must be doing OK. If I'm not OK, then it's time to do a little troubleshooting.
The ping command generally behaves in one of two ways. It runs four times in succession and then tells you how many of the requests got through or it runs until you type ^c to stop it and then gives you the statistics. You can also provide the command with a -c option to specify how many times you want to ask. If I'm trying to verify that the systems that I'm expecting to find on a particular subnet are indeed up, I generally don't want to send each more than one ping. It takes too long. Sending one ping per system in a for sys in `cat syslist`; do ping -c 1 $sys; done or, even better, maybe for sys in `cat syslist`; do echo -n "$sys: "; ping -c 1 $sys | grep loss; done.
traceroute
The traceroute command will attempt to provide a list of all the routers your connections cross when reaching out to a remote system. The output also provides some information on how long each segment of the path takes, thus giving you some notion of the quality of a connection.
$ traceroute fermion traceroute to fermion (10.10.10.10), 30 hops max, 40 byte packets 1 coresys (10.10.10.2) 0.965 ms 1.010 ms 1.006 ms 2 rtr2 (10.10.8.10) 0.613 ms 0.627 ms 0.642 ms 3 10.210.1.14 (10.20.1.11) 78.226 ms 77.779 ms 77.734 ms 4 fermion (10.1.2.3) 77.257 ms !X 81.914 ms !X 73.626 ms !X
If traceroute gets to aa point at which it is unable to provide further information, you will start to see asterisks where you used to see system names and timing.
netstat
The netstat command can tell you about ongoing connections on the local system and ports (i.e., services) that are listening, indicating that services are waiting for requests to come through. By itself, netstat gives you a *lot* information. With a -a option, it gives you even more.
To see listening ports (i.e., services), you may have to do something like this:
$ netstat -a | grep LISTEN | grep -v LISTENING tcp 0 0 localhost.localdomain:32000 *:* LISTEN tcp 0 0 localhost.locald:bootserver *:* LISTEN tcp 0 0 *:cbt *:* LISTEN tcp 0 0 *:38821 *:* LISTEN tcp 0 0 *:8009 *:* LISTEN
Some useful options include -n (don't translate IP addresses). When you use this option, the output shows IP addresses and port numbers rather than system and service names.
$ netstat -an | more Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:32000 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2016 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:7777 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:38821 0.0.0.0:* LISTEN
The netstat -i command shows interface statistics for each of your active network interfaces.
$ netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg bond0 1500 0 957543349 0 17641 0 946780933 0 0 0 BMmRU eth0 1500 0 957543349 0 17641 0 946780933 0 0 0 BMsRU eth1 1500 0 0 0 0 0 0 0 0 0 BMsU lo 16436 0 945046526 0 0 0 945046526 0 0 0 LRU
nslookup
The nslookup command will do a DNS query for you. Assuming the system you are working on has a proper /etc/resolv.conf to let it know where to go to ask this type of question, it can help you find the IP addresses for particular systems and, often, the system names that go along with particular IP addresses. I've found that companies are often lax in setting up the reverse lookup (PTR) records that tie IP addresses back to hostnames, but they are wonderfully useful. By the time you are watching over hundreds or even thousands of servers, you're going to occasionally ask yourself something like "What system is 10.94.1.7?" and having an easy way to get an answer can be very helpful.
whois
You can often find a lot more information about a domain using the whois command, the output below is truncated. Here I'm picking on my local gas station. Let's see what I can find out about Valero.
$ whois valero.com [Querying whois.verisign-grs.com] [Redirected to whois.networksolutions.com] [Querying whois.networksolutions.com] [whois.networksolutions.com] Domain Name: VALERO.COM Registry Domain ID: Registrar WHOIS Server: whois.networksolutions.com Registrar URL: http://networksolutions.com Updated Date: 2012-02-06T00:00:00Z Creation Date: 1996-02-21T00:00:00Z Registrar Registration Expiration Date: 2021-02-22T00:00:00Z Registrar: NETWORK SOLUTIONS, LLC. Registrar IANA ID: 2 Registrar Abuse Contact Email: abuse@web.com Registrar Abuse Contact Phone: +1.8003337680 Reseller: Domain Status: clientTransferProhibited Registry Registrant ID: Registrant Name: Valero Energy Registrant Organization: Registrant Street: One Valero Way Registrant City: San Antonio Registrant State/Province: TX Registrant Postal Code: 78249-1616 ...
hostname
Generally, you know the name of the system you're logged into. After all, you just logged into it and, often, server names are set up as the system's command line prompt. But you might be logged into several systems at once or you may want to know just a bit more information. Here are some hostname command options that you might not be using.
$ hostname boson.particles.org $ hostname -d particles.org $ hostname -i 10.20.30.40
The -i option provides and easy way to grab the system's IP address without having to separate it from ifconfig output.
tcpdump
The tcpdump command can print out the headers of network packets as they reach your server or can be used with various filters to select just the packets you want to see. You can also save packets for later analysis. I'll get into some interesting uses of tcpdump in my next post.