Updated: June 8, 2019
In the world of Linux, the typical trend is to diverge, create new tools, fork and split, leading to an ever growing number of programs and utilities revolving around identical or similar functionality. But sometimes, you get software that does the opposite - trying to combine the usage of multiple programs under a single umbrella, but WITHOUT becoming cumbersome or complicated.
One such handy tool is Dstat, a monitoring application that is designed to replace, or at least supplement, a whole range of system monitoring utilities like vmstat, iostat and ifstat. Dstat aims to be simple, extensible, robust, and accurate. All of these make it a good candidate for a dedicated article. Follow me.
How do I dstat?
If you've used the other *stat software, you are probably familiar with the syntax. You invoke the program with some optional flags, you specify the polling interval (like 1 second or 1 minute), and you specify the amount of cycles you want the tool to run before exiting. The first line of the output is the average value for the collected metric since last boot.
Dstat follows the same principle. There are many useful flags and options available, including advanced usage and workload-specific functionality, enabled through a dazzling range of plugins. Now, you may think the sheer amount of options would make dstat a bit busy. Well, it escapes clutter by offering nicely delineated, grouped data statistics. Let us exemplify.
The default dstat configuration is quite useful. It will show total CPU and disk usage, paging activity, network receive and send data, and system behavior (interrupts and context switches). This gives a good first glance overview of how your machine is performing.
You can also try something like:
dstat -c -d -D total, sda -i 1 20
This command will show CPU data (-c), disk read and write values, both total figures as well as specific data for the sda device. If you only have one disk, the numbers will be identical. But you could have multiple disks, in which case, this can be quite useful. Dstat also supports RAID configurations. Finally, I've also selected -i to show interrupts. You really have a lot of flexibility.
If you're interested in memory and swapping, then you would use:
dstat -g -m -s "delay" "count"
Advanced options, plugins & output files
If you find the default values insufficient, you can use advanced options. For instance, --cpu-adv will provide a more detailed overview of the CPU behavior. In combination with system data (-y), you get context switches and interrupts. This is something I've shown you when troubleshooting slow application startup, and the idea remains the same. Dstat allows you to capture this data, and then some.
You may also be interested in battery charge values, or maybe CPU frequency (as percentage). For instance, this can be useful to test if power management works well, if you have Turbo enabled (in BIOS), whether certain sleep states kick in correctly, and so forth.
To see the full list of available plugins, you can run dstat --list. The plugins are stored under /usr/share/dstat, in the form of .py and .pyc files. If you want to add your own custom plugins, you can store them into a hidden directory (.dstat) in your home.
Dstat also lets you output data to files with --output file.csv. This can be useful when you need to perform more detailed data analysis. Finally, the high level of granularity offers insight into really specific subsystems in your machine. For example, dstat supports RPC, NFS, MYSQL5, GPFS, memcache, sendmail, and many other counters.
I did encounter some small issues. Running one of the commands, I got an error that said my terminal width was too small. But even with the console window (Konsole) maximized to my Slimbook's full-HD resolution, the message was still there. Some kind of a bug, it would seem. You can work around it, but it would be nice not to have to encounter it in the first place.
dstat --disk-svctm --disk-wait 1 20
Terminal width too small, trimming output.
Dstat is a wonderful troubleshooting and data collection program. It's extremely helpful in that you don't need half a dozen other utilities (sar, netstat and friends included) to profile your system. There's also no need for any extensive configuration. You just run the tool, save data to a file, and focus on the analysis. Dstat works in a consistent manner, it's simple, elegant and powerful.
You do still need a decent understanding of the Linux operating system and your software and/or workloads, though. You may also require the use of tools like strace, ltrace, perf, or gdb to obtain a complete picture of a problem, be it slowness or any other system behavior. However, dstat removes the noise in having to manage multiple programs of similar nature, capable of grabbing only limited subsets of data. It's quite handy, and if you're keen on mastering Linux administration, or maybe figuring out what your boxen be doing behind the scenes, dstat is an ideal tool for the task. Hopefully, you've learned something useful today, and I've whetted your appetite for knowledge. Take care.