One must know the disk seek patterns of an application when optimizing the storage layer for any application. So when I was working on the performance analysis of ES, one of the first thing was to determine its disk seek patterns. I used blktrace and seekwatcher for that purpose.
blktrace is a utility that traces block io request issued to a given block device. To quote from the blktrace manual:
…blktrace receives data from the kernel in buffers passed up through the debug file system (relay). Each device being traced has a file created in the mounted directory for the debugfs, which defaults to /sys/kernel/debug
For using blktrace, the first step is to mount debugfs. It is a memory based file system to debug linux kernel code.
# mount -t debugfs debugfs /sys/kernel/debug
The next step is to setup the disk on which we need to the tracing. This should be a separate disk as we do not want other OS activity to influence our readings. For this, I’ve used /dev/sdc as the data directory of ES. Now, start ES and wait for a few seconds after the cluster comes into green state, so as to be sure that we are only tracing the disk activity while searching and not the startup. Before firing the ES query, start blktrace on /dev/sdc with the following command
blktrace -d /dev/sdc -o es-ssd-search
This will start tracing the block requests on the disk and send output to es-ssd-search.blktrace.0 .. es-ssd-search.blktrace.n-1 where each file represents requests from one core. Since I was using a quadcore CPU, I got the following files:
es-on-ssd.blktrace.0
es-on-ssd.blktrace.1
es-on-ssd.blktrace.2
es-on-ssd.blktrace.3
Now that we have data from blktrace, the next step is to visualize it. That can be done by blkparse utility. It formats the blktrace output into human readable form. But as they say, a picture is worth a thousand words, there is another tool, namely seekwatcher that can produce plots and movies from the output of blktrace. I used it to visualize the seek patterns. To encode movies with seekwatcher, we also need to install MEncoder. Once seekwatcher and MEncoder are installed, run the following command to generate the movie:
# seekwatcher --movie-frames=10 -m -t es-on-ssd.blktrace.0 -o es-on-ssd.movie
It will produce es-on-ssd.movie that can be played with MPlayer. Following is the output that I got:
As can be seen, apart from a few random reads most of the reads are sequential-reads for which I went on to optimize the storage.
Reference:
- blktrace: http://linux.die.net/man/8/blktrace
- blkparse: http://linux.die.net/man/1/blkparse
- seekwatcher: https://oss.oracle.com/~mason/seekwatcher/
- MEncoder/MPlayer: http://www.mplayerhq.hu/DOCS/man/en/mplayer.1.txt
- and finally, ElasticSearch: http://www.elasticsearch.org/
Bonus video: ES startup seeks
ES Start Seek Patterns from Anand Nalya on Vimeo.