No VM disk statistics

[fumok]

Hi,

I'm testing a setup to monitor kvm with ganglia, everything seems fine except for VM disk statistics (empty graph).
Some detail:

• RHEL 7.2 3.10.0-327.el7.x86_64
• libvirt-1.2.17-13.el7.x86_64
• ganglia-3.7.2-2.el7 (from epel)
• hsflowd-2.0.1-1.x86_64.rpm (from here)

Can you help me? Thanks a lot.

[sflow]

This is the relevant code:
https://github.com/sflow/host-sflow/blob/master/src/Linux/mod_kvm.c#L114-L174

Is is missing the capacity/allocation/available data from virDomainGetBlockInfo(), or the reads/writes/errors counter data from virDomainBlockStats(), or both?

The binary is compiled with "-g -O2" so you should be able to set a breakpoint in gdb and debug like this:
sudo yum install gdb
sudo service hsflowd stop
sudo gdb hsflowd
gdb> set args -ddd
gdb> b mod_kvm.c:169
(say yes it will be loaded later)
gdb> r

If you'd rather build from sources and add print statements then you'll need gcc and something like this:
sudo yum install libvirt-devel libxml2-devel
make FEATURES="kvm ovs"

[fumok]

Thanks for supporting me.
After further digging, I think the disk statistics are populated only if vm uses default storage pool.

I've created a new test vm with a cqow image in default storage pool, and now, dumping data with sflowtool, I can see valorized those counters:
vdsk_capacity
vdsk_allocation
vdsk_available
vdsk_rd_req
vdsk_rd_bytes
vdsk_wr_req
vdsk_wr_bytes
vdsk_errs

Incidentally now the break point works (b mod_kvm.c:169). Before, with not standard storage pool vm, it had never been reached.

EDIT: Perhaps I was too hasty in my diagnosis. I think that the discriminant is the type of storage pool. With qcow images counters are ok, with lvm do not.

EDIT2: I can confirm, disk counters work only with qcow backing storage.

[sflow]

Sounds like it's not recognizing the lvm storage when it parses the XML. If you could send the output of "virsh dumpxml " that would be helpful.

[fumok]

hsflowd in debug mode returns:

dbg1: attribute dev
dbg1: disk.dev=hda
dbg1: ignoring readonly device
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vda
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vdb
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vdc
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vdd
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vde
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vdf
dbg1: attribute dev
dbg1: attribute dev
dbg1: disk.dev=vdg

and this is right: one virtual cdrom (hda) and seven virtual lvm disk (from vda to vdg)

Follow xml of test vm:

test.txt

[sflow]

It looks like we might be able to pick up the disk stats a different way...

It depends on how a qemu VM is treated with respect to Linux cgroups. For example, I have a KVM system running Ubuntu 14.4 with a VM called "test6" and it looks like I can get disk stats like this:

cat /sys/fs/cgroup/blkio/machine/test6.libvirt-qemu/blkio.throttle.*

Can you substitute "test6" with "test" on your system and get numbers this way? Do the numbers look as though they are specific to that VM?

We already pick up cgroup stats this way in hsflowd's mod_docker, so this could be quite straightforward to add. Just need to understand how it appears in different versions of KVM. I'll try Fedora 24 next and see what appears there.

[fumok]

I think cgroup it's the right way. On RHEL7.2 (kernel 3.10.0-327.el7.x86_64) the relevant counter can be found in:

/sys/fs/cgroup/blkio/machine.slice/{weird_name}/

{weird_name} is something like this (in my case the hostname contain a hyphen):

machine-qemu\x2d{host_hostname}\x2dgo\x2d{guest_name}\x2da.scope

Most part of blkio.throttle.* are not valorized, only blkio.throttle.io_service_bytes and blkio.throttle.io_serviced.

I've the same situation on a Centos 7.2, with much more newer libvirtd and qemu-kvm. So i can suppose that only matters host's kernel version. The cgoup approach is used also by others interesting project. Take a look, for example, at https://github.com/firehol/netdata/

P.S.

Systemd systems use different names, see https://libvirt.org/cgroups.html

Thanks.