I recently had to patch and reboot my NAS for the “heartbleed” OpenSSL fix. The NAS provides both primary and secondary storage services to my CloudStack 4.2.1 test environments. Being brave, I rebooted the NAS before enabling maintenance mode on the Zone or on the primary and secondary storage resources.
Within a few minutes after the reboot, the CloudStack management server freaked out with the below error messages while trying to start the system VMs (SSVM/CPVM):
System Alert: Number of unallocated private IPs is low in pod POD-XEN1 of availability zone Prod Number of unallocated private IPs is low, total: 10.0, allocated: 10.0 (100%)
Sure enough, all IP addresses for the POD was shown as taken in the op_dc_ip_address_alloc table.
mysql> select * from op_dc_ip_address_alloc; +----+---------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+ | id | ip_address | data_center_id | pod_id | nic_id | reservation_id | taken | mac_address | +----+---------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+ | 1 | 192.168.44.60 | 1 | 1 | 7 | c8fa9939-a7ed-4d7e-978b-606954a8d924 | 2014-04-12 14:12:29 | 51 | | 2 | 192.168.44.61 | 1 | 1 | 7 | f22eaf13-5383-423d-a269-c499f1c5bc75 | 2014-04-12 14:12:28 | 52 | | 3 | 192.168.44.62 | 1 | 1 | 7 | db7d7a46-295b-43f0-aaba-ee33f5d5d34d | 2014-04-12 14:12:28 | 53 | | 4 | 192.168.44.63 | 1 | 1 | 7 | e51bbf11-7000-4117-b309-a60235731fdc | 2014-04-12 14:12:30 | 54 | | 5 | 192.168.44.64 | 1 | 1 | 7 | cad0e89f-34b5-4d8e-9b8b-bcb8ddcb267d | 2014-04-12 14:00:05 | 55 | | 6 | 192.168.44.65 | 1 | 1 | 7 | 2f70a5a9-cd7b-467f-b83b-c3996b72f717 | 2014-04-12 14:12:28 | 56 | | 7 | 192.168.44.66 | 1 | 1 | 7 | 3817c4de-c8cd-42a0-a0bf-dd6779dfe668 | 2014-04-12 14:12:27 | 57 | | 8 | 192.168.44.67 | 1 | 1 | 7 | c727ff6e-f5a7-4a5d-970b-6b3affdad7a5 | 2014-04-12 14:12:30 | 58 | | 9 | 192.168.44.68 | 1 | 1 | 7 | 00e60594-cc35-46e1-8e6f-3e31ca075807 | 2014-04-12 14:12:29 | 59 | | 10 | 192.168.44.69 | 1 | 1 | 7 | b3ddcd9b-6a45-44f9-b14a-90f8f372a6a5 | 2014-04-12 14:12:29 | 60 | | 11 | 192.168.44.70 | 1 | 2 | NULL | NULL | NULL | 61 | | 12 | 192.168.44.71 | 1 | 2 | NULL | NULL | NULL | 62 | | 13 | 192.168.44.72 | 1 | 2 | NULL | NULL | NULL | 63 | | 14 | 192.168.44.73 | 1 | 2 | NULL | NULL | NULL | 64 | | 15 | 192.168.44.74 | 1 | 2 | NULL | NULL | NULL | 65 | | 16 | 192.168.44.75 | 1 | 2 | NULL | NULL | NULL | 66 | | 17 | 192.168.44.76 | 1 | 2 | NULL | NULL | NULL | 67 | | 18 | 192.168.44.77 | 1 | 2 | NULL | NULL | NULL | 68 | | 19 | 192.168.44.78 | 1 | 2 | NULL | NULL | NULL | 69 | | 20 | 192.168.44.79 | 1 | 2 | NULL | NULL | NULL | 70 | +----+---------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+ 20 rows in set (0.00 sec)
According to the CLOUDSTACK-2813 and mailing list discussion on Console Proxy VM failed to start, the fix is to set nic_id and reservation_id to NULL. While CLOUDSTACK-2813, is marked as fixed, my ACS 4.2.1 installation was affected.
After cleaning up the tables, the console proxy and secondary storage VMs started without issues.
mysql> select * from op_dc_ip_address_alloc; +----+---------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+ | id | ip_address | data_center_id | pod_id | nic_id | reservation_id | taken | mac_address | +----+---------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+ | 1 | 192.168.44.60 | 1 | 1 | NULL | NULL | NULL | 51 | | 2 | 192.168.44.61 | 1 | 1 | NULL | NULL | NULL | 52 | | 3 | 192.168.44.62 | 1 | 1 | 2786 | 131d7334-3b81-46ea-80b2-10ff015028a8 | 2014-04-13 07:03:42 | 53 | | 4 | 192.168.44.63 | 1 | 1 | NULL | NULL | NULL | 54 | | 5 | 192.168.44.64 | 1 | 1 | 2785 | 131d7334-3b81-46ea-80b2-10ff015028a8 | 2014-04-13 07:03:42 | 55 | | 6 | 192.168.44.65 | 1 | 1 | 2758 | 52cca31f-dbb7-46c7-8bc8-4133b9dd963d | 2014-04-13 07:03:42 | 56 | | 7 | 192.168.44.66 | 1 | 1 | NULL | NULL | NULL | 57 | | 8 | 192.168.44.67 | 1 | 1 | NULL | NULL | NULL | 58 | | 9 | 192.168.44.68 | 1 | 1 | NULL | NULL | NULL | 59 | | 10 | 192.168.44.69 | 1 | 1 | NULL | NULL | NULL | 60 | | 11 | 192.168.44.70 | 1 | 2 | NULL | NULL | NULL | 61 | | 12 | 192.168.44.71 | 1 | 2 | NULL | NULL | NULL | 62 | | 13 | 192.168.44.72 | 1 | 2 | NULL | NULL | NULL | 63 | | 14 | 192.168.44.73 | 1 | 2 | NULL | NULL | NULL | 64 | | 15 | 192.168.44.74 | 1 | 2 | NULL | NULL | NULL | 65 | | 16 | 192.168.44.75 | 1 | 2 | NULL | NULL | NULL | 66 | | 17 | 192.168.44.76 | 1 | 2 | NULL | NULL | NULL | 67 | | 18 | 192.168.44.77 | 1 | 2 | NULL | NULL | NULL | 68 | | 19 | 192.168.44.78 | 1 | 2 | NULL | NULL | NULL | 69 | | 20 | 192.168.44.79 | 1 | 2 | NULL | NULL | NULL | 70 | +----+---------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+ 20 rows in set (0.00 sec) mysql>
Shanker Balan
Shanker Balan is a devops and infrastructure freelancer with over 14 years of industry experience in large scale Internet systems. He is available for both short term and long term projects on contract. Please use the Contact Form for any enquiry.
Follow Me:
