Troubleshooting IMCP

What is ICMP?

ICMP stands for Internet Control Message Protocol and is also a common IP transport protocol. It seems pretty explicit, although most people reduce ICMP to ping reply commands, it is a good way to test whether a host can be reached through a network and how much it takes for a packet to make a round trip through the network. Obviously, ping and trace-route-like tools are very useful for network administrators but there is much more to say about ICMP and the help it can provide for network administration and diagnosis. ICMP can be used to send more than twenty types of control messages. Some are just messages; some others are a way for IP devices or routers to indicate the occurrence of an error.

Error Messages

Lets describe the most typical ICMP error messages you can find on networks.

ICMP Network Unreachable

Let’s take the simplest example: one machine sitting on a LAN (192.168.0.7), has one default gateway (192.168.0.254) which is the router. It is trying to reach a server, which does not sit on the LAN (10.1.0.250) and which cannot be reached because 192.168.0.254 does not know how to route this traffic.

ICMP Host Unreachable

Let’s take the simplest example: one machine sitting on a LAN (10.1.2.23), has one default gateway (10.1.2.254/24) which is the router. It is trying to reach a server which does not sit on the LAN (192.168.1.15). The traffic flows and reaches the last router before the server (192.168.1.254/24); this router cannot reach 192.168.1.15 (because it is unplugged, down or it does not exist).

ICMP Port Unreachable

Let’s take a second example: one machine sitting on a LAN (192.168.0.7). It is trying to reach a server 192.168.0.254, which sits on the LAN on port UDP 4000, on which the server does not respond.

Where is the challenge with ICMP?

You may be tempted to say: if it is that simple, why do we need Cisco Provider Connectivity Assurance Sensor Capture (formerly Skylight sensor: capture) on top of any sniffer? All the information sits in the payload. But in every network, you will find some ICMP errors. They may be due to a user trying to connect to a bad destination, or trying to reach a server on the wrong port. The key is in having a global view of how many errors you have normally and currently and from where to where. The key to leveraging ICMP information is in having a relevant view of it and understanding what it means.

How can ICMP help in network diagnostic and security monitoring?

By analysing ICMP errors, we can identify machines that try to connect to networks or machines that are routable from the LAN’s machine or ones that try to connect on actual servers but for services whose ports are not open. Here are some examples of phenomena that can be identified that way:

Misconfigured Workstation

A workstation repeats a large volume of missed attempts to connect to a limited number of servers: it may be that this machine does not belong to the company’s workstations (external consultant on the network whose laptop is trying to reach common resources on his home network – DNS, printers, etc.), or it may be the machine of someone coming from a remote site with its own configuration or a machine that has been simply wrongly configured.

How would we see it?

A large number of ICMP Host Unreachable errors coming from one or several routers to this machine or this group of machines. The ICMP information contained in the payload of each of these errors would probably show they are trying to reach a certain number of hosts for some services or applications.

Migration Legacy

A certain number of machines keep requesting DNS resolution to a DNS server that has been migrated (this could be true for any application available on the network). Their users certainly experience worse performance when trying to use these services.

How would we see it?

A large number of ICMP Host Unreachable errors coming from one or several routers to a group of machines. The ICMP information contained in the payload of each of these errors would probably show they are all trying to reach the previous IP address of a given server.

Network Device Misconfiguration

A router does not have a route configured; some machines are trying to reach some resources, unsuccessfully.

How would we see it?

A large number of ICMP Network Unreachable errors coming from one router to many machines. The ICMP information contained in the payload of each of these errors would probably show they are all trying to reach the same network through the same router.

Port Scanning

A machine is trying to complete a network discovery. It is trying to connect to all servers around to see which ports are open.

How would we see it?

A large number of ICMP Port Unreachable errors coming from one or several routers corresponding to a single machine (the one which is scanning).

Spyware / Worms

An infected machine is trying to propagate its spyware, virus or worm throughout the network; obviously, it has no previous knowledge of the network architecture.

How would we see it?

A large number of ICMP Host Unreachable errors coming from one or several routers corresponding to a limited number of hosts, trying to reach a large volume of non-existing machines on a limited set of ports.

Server Disconnected/Reboot

A service on UDP (DNS, Radius...) is interrupted because the server program is temporarily stopped or the host machine is temporarily shutdown. Many requests are then discarded.

How would we see it?

Many ICMP Port Unreachable messages (preceded by some unreachable host if the host itself was shut down) are emitted during a short period of time for this service host/port.