CloudStack Attach Volumes On Ubuntu Not Working on KVM

Adding new volumes to an Ubuntu Lucid guest instance running under KVM hypervisor seems to not work automatically. The newly attached device is visible to the Ubuntu guest OS only after a reboot.

To replicate the issue…

  1. Create a Ubuntu Lucid guest instance on KVMScreen Shot 2013-12-22 at 1.44.47 pm
  2. Create a new data diskScreen Shot 2013-12-22 at 1.46.25 pm
  3. Attach the new data disk to the Ubuntu guest OSScreen Shot 2013-12-22 at 1.47.35 pm
  4. Run fdisk -l /dev/vdb on the Ubuntu KVM Guest OS. No disks found
    shanu@ubuntu3:~$ sudo fdisk -l /dev/vdb|wc -l
    0
    
  5. Reboot and run fdisk again. Disk is now available as /dev/vdb
    shanu@ubuntu3:~$ sudo fdisk -l /dev/vdb
    
    Disk /dev/vdb: 5368 MB, 5368709120 bytes
    16 heads, 63 sectors/track, 10402 cylinders, total 10485760 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x00000000
    
    Disk /dev/vdb doesn't contain a valid partition table
    

The tests were performed on Apache CloudStack 4.2.0+CentOS 6.5 KVM hypervisor hosts and Ubuntu 12.04 Lucid as guest operating system.

The same steps work correctly on XenServer guest operating systems (tested on Ubuntu,Linux and FreeBSD) and CentOS guest on KVM. Since its working automatically without a reboot in CentOS/KVM, it didn’t seem to be a CloudStack issue per-se.

Maybe its a libvirt/qemu-kvm issue?

Nope, virsh dumpxml for the instance shows that the block device has already been attached to the VM. Also, its a virtio device which means it should hotplug automatically. CloudStack has done what it was asked to do, ie, attach disk to instance.

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/f1de7d10-14b8-39f9-ba2f-4d6f7a05782c/737f1c93-658d-4450-85fa-36cc60858b00'/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x17' function='0x0'/>
    </disk>

So why is hotplug still not working??? A quick search on Google for “KVM+virtio+hotplug” leads to http://www.linux-kvm.org/page/Hotadd_pci_devices. According to the page, 2 modules are required for hotplug to work:

  1. acpiphp
  2. pci_hotplug

Time to take a closer look at Ubuntu now. dmesg shows that pci_hotplug is already built into the kernel and so is virtio

shanu@ubuntu3:~$ dmesg |grep pci_hotplug
[    0.695397] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
shanu@ubuntu3:~$ dmesg |grep virtio-pci
[    0.706821] virtio-pci 0000:00:03.0: setting latency timer to 64
[    0.707271] virtio-pci 0000:00:04.0: setting latency timer to 64
[    0.707762] virtio-pci 0000:00:06.0: setting latency timer to 64
[    0.708285] virtio-pci 0000:00:17.0: setting latency timer to 64
[    0.885157] virtio-pci 0000:00:04.0: irq 40 for MSI/MSI-X
[    0.885223] virtio-pci 0000:00:04.0: irq 41 for MSI/MSI-X
[    1.027816] virtio-pci 0000:00:17.0: irq 42 for MSI/MSI-X
[    1.027888] virtio-pci 0000:00:17.0: irq 43 for MSI/MSI-X
[    1.131989] virtio-pci 0000:00:03.0: irq 44 for MSI/MSI-X
[    1.132096] virtio-pci 0000:00:03.0: irq 45 for MSI/MSI-X
[    1.132166] virtio-pci 0000:00:03.0: irq 46 for MSI/MSI-X

Hmm, but no sign of “acpiphp”.

shanu@ubuntu3:~$ dmesg |grep -c acpiphp
0

The same grep on the working CentOS6 KVM guest shows results.

[root@centos6-kvm64 ~]# dmesg |grep acpiphp:
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5

Lets load the module manually and retry shall we?

shanu@ubuntu3:~$ sudo modprobe -v acpiphp
insmod /lib/modules/3.8.0-29-generic/kernel/drivers/pci/hotplug/acpiphp.ko 
shanu@ubuntu3:~$ dmesg |grep acpiphp:
[ 1450.677666] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5

The dmesg output is looking good. Lets try an attach now…

shanu@ubuntu3:~$ sudo fdisk -l /dev/vdb

Disk /dev/vdb: 5368 MB, 5368709120 bytes
16 heads, 63 sectors/track, 10402 cylinders, total 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn't contain a valid partition table

Bingo!!! Manually loading “acpiphp” module seems to make hotplug event work. Time to make the changes permanent by adding “acpiphp” to /etc/modules.

shanu@ubuntu3:~$ sudo sh -c 'echo acpiphp >> /etc/modules' /etc/modules 
shanu@ubuntu3:~$ cat /etc/modules 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

loop
lp
acpiphp
shanu@ubuntu3:~$ sudo reboot

References:

  1. http://www.linux-kvm.org/page/Hotadd_pci_devices
  2. http://mail-archives.apache.org/mod_mbox/cloudstack-users/201312.mbox/%3c26009708.256.1387630197716.JavaMail.andrei@tuchka%3e

Shanker Balan

Shanker Balan is a devops and infrastructure freelancer with over 14 years of industry experience in large scale Internet systems. He is available for both short term and long term projects on contract. Please use the Contact Form for any enquiry.

More Posts - Website

Follow Me:
TwitterLinkedIn

Published by

Shanker Balan

Shanker Balan is a devops and infrastructure freelancer with over 14 years of industry experience in large scale Internet systems. He is available for both short term and long term projects on contract. Please use the Contact Form for any enquiry.

Leave a Reply