How do you troubleshoot IoT devices?

Submitted by fredrik.nyman on Fri, 02/15/2019 - 13:00

Continuing on the subject of troubleshooting the network. Troubleshooting MPEG video has the benefit of a user that can tell you if it doesn't work and you can simply ask that user if the problem persists once you have fixed it. But what if there isn't any obvious way to determine if things are working, for example is that trashcan really signalling that its' full or does the temperature device really update the building climate control properly?

Image removed.

With more and more IoT devices entering the networks, troubleshooting becomes increasingly difficult. Headless (no screen or keyboard) devices depend on their preprogramed start-up sequence and their ability to connect to central resources to work properly. If it is a simple matter of broken cable, the problem might be easy to track down and resolve. But if the problem is more complicated, then so is the troubleshooting. You can't see if the device gets it's IP, you cant test ping and traceroute from it. Its just a black box and you have no way of knowing if it is properly configured.

So what if the device is connected, link is up, the routing in the network should work and yet there is no data received? That waste bin's status report does not reach to the waste management server.

At this point the seasoned network engineers usually pulls out the heavy artillery - Wireshark. Let's figure out if the bin gets its IP and even attempt to send the report and to the right destination, and let's see if there is an answer to it by looking at the packets on the wire.

By running a packet analysis tool such as Wireshark you will be able to pick up the packets and debug them one by one as it happens in the network. While the IoT device might be headless, you can determine from the packet flow what actually happens. Does the device send a DHCP packet, is the server responding, are the configuration parameters correct, does the device attempt to connect to its cloud controller.

But hold on you say, I have to bring half the office equipment with me to go out and get a Wireshark session going on the link connecting to that waste bin. It's not worth the time or effort to do that! Cables, computers, getting access to the facility… bah!

This is precisely why a feature such as the remote mirroring in Waystream's MS and ASR switches are so useful, particularly with the increasing number of IoT devices in the networks.

Image removed.

With the remote mirroring feature you can actually make that remote port very local. Here's how it works; Configure the switch to mirror the traffic on the port where the IoT device is connected over a tunnel to your desktop computer. This allows you to see the packets in and out on the port without leaving your comfortable chair. You can even apply an access-list to capture only the traffic that you are interested in.

You do not have to leave the office and go on-site with computer, cables and stuff, you do not have to bother getting access to the facilities with keys and passcodes, you can avoid having that janitor look over your shoulder stressing you out with all sorts of questions while you try to figure out what is happening.

Instead, you get going with the troubleshooting in minutes and you can debug everything from the comfort of your own desk with access to all resources you need including the ability to see both ends of the communication at the same time, usually very difficult when in the field. You can verify the DHCP, try the ping and traceroute between the server and the client to verify connectivity and see the actual report packet being sent by the client but not reaching the server.

The remote mirroring feature is a valuable tool to troubleshoot more advanced problems than simple cable issues. With more and more headless IoT devices this is precisely the kind of features needed to make troubleshooting easier.

Blog posts

How to see what your users see

Submitted by fredrik.nyman on Mon, 02/11/2019 - 10:21

Live broadcast TV is one of the most popular services in fibre networks. You can get high quality pictures because there is enough bandwidth to send video uncompressed. But the nature of broadcast media is that it is very sensitive to packet loss or jitter. There is no retransmission of packets because it is live – you can’t hold the stream to get a lost packet back.

FTTH is not like any other network

Submitted by fredrik.nyman on Fri, 01/25/2019 - 13:34

If you are working in network engineering, hands-on with the routers and switches in the network, you probably have seen your fair share of network problems. However well you build it there is always some intermittent issue, some complaining user, some application that doesn’t get the throughput, some website that is unreachable.

It’s part of the everyday chaos of running a network to deal with big and small issues.

The Way Better Blog

Submitted by fredrik.nyman on Fri, 01/25/2019 - 10:02

In this blog I will be writing about some of the topics, big and small, facing network engineers and fibre networks and the kind of challenges I have encountered working with our customers over the past 20 years or so.