Streaming sensor data to dashboard | 100 ms sampling

In this post we look at adding streaming sensor data visualisation to the dashboard. The data we look at is the temperature sensor measurements from the cpu, for each core. Data is sampled at 100 ms i.e 10 samples per second. Each data point has a time stamp. 

Tools used:

- Django 2.1.13 and Python 3.7.4 on docker images for back end.

- Front end is built using java script and libraries like dc.js and d3.js

When the cpu load goes up we can see the data updated on the dashboard. Some points for building a visualisation that updates very frequently like this includes

1) Data size and network distance: It is a good idea to check the impact on the network or if the network is capable of it. Data is streamed in small units but, is 4TB per 24 hours translating to 370 Mbps or 280 Mbps. Would the network be able to handle this if there are multiple sources for data. As the distance to stream to gets longer latency will start to show.

2) Continuous connection: Streaming response means a continuous connection is maintained between the client and server.

3) Data storage: Here the data is continuously send to the client as and when it becomes available. This depends of the data type. At times dashboards fetch data from a database and update the visualisation every 5 seconds or 10 seconds. If the database is involved then the polling interval will affect the database with that load. Also, fetching data from the database every time can include disk latency. So although the data can be saved to disk asynchronously, here it is just streamed back to the client.