How do you troubleshoot IoT devices?

Continuing on the subject of troubleshooting the network. Troubleshooting MPEG video has the benefit of a user that can tell you if it doesn't work and you can simply ask that user if the problem persists once you have fixed it. But what if there isn't any obvious way to determine if things are working, for example is that trashcan really signalling that its' full or does the temperature device really update the building climate control properly?

With more and more IoT devices entering the networks, troubleshooting becomes increasingly difficult. Headless (no screen or keyboard) devices depend on their preprogramed start-up sequence and their ability to connect to central resources to work properly. If it is a simple matter of broken cable, the problem might be easy to track down and resolve. But if the problem is more complicated, then so is the troubleshooting. You can't see if the device gets it's IP, you cant test ping and traceroute from it. Its just a black box and you have no way of knowing if it is properly configured.

So what if the device is connected, link is up, the routing in the network should work and yet there is no data received? That waste bin's status report does not reach to the waste management server.

At this point the seasoned network engineers usually pulls out the heavy artillery - Wireshark. Let's figure out if the bin gets its IP and even attempt to send the report and to the right destination, and let's see if there is an answer to it by looking at the packets on the wire.

By running a packet analysis tool such as Wireshark you will be able to pick up the packets and debug them one by one as it happens in the network. While the IoT device might be headless, you can determine from the packet flow what actually happens. Does the device send a DHCP packet, is the server responding, are the configuration parameters correct, does the device attempt to connect to its cloud controller.

But hold on you say, I have to bring half the office equipment with me to go out and get a Wireshark session going on the link connecting to that waste bin. It's not worth the time or effort to do that! Cables, computers, getting access to the facility… bah!

This is precisely why a feature such as the remote mirroring in Waystream's MS and ASR switches are so useful, particularly with the increasing number of IoT devices in the networks.

With the remote mirroring feature you can actually make that remote port very local. Here's how it works; Configure the switch to mirror the traffic on the port where the IoT device is connected over a tunnel to your desktop computer. This allows you to see the packets in and out on the port without leaving your comfortable chair. You can even apply an access-list to capture only the traffic that you are interested in.

You do not have to leave the office and go on-site with computer, cables and stuff, you do not have to bother getting access to the facilities with keys and passcodes, you can avoid having that janitor look over your shoulder stressing you out with all sorts of questions while you try to figure out what is happening.

Instead, you get going with the troubleshooting in minutes and you can debug everything from the comfort of your own desk with access to all resources you need including the ability to see both ends of the communication at the same time, usually very difficult when in the field. You can verify the DHCP, try the ping and traceroute between the server and the client to verify connectivity and see the actual report packet being sent by the client but not reaching the server.

The remote mirroring feature is a valuable tool to troubleshoot more advanced problems than simple cable issues. With more and more headless IoT devices this is precisely the kind of features needed to make troubleshooting easier.