I recently had to patch and reboot my NAS for the “heartbleed” OpenSSL fix. The NAS provides both primary and secondary storage services to my CloudStack 4.2.1 test environments. Being brave, I rebooted the NAS before enabling maintenance mode on the Zone or on the primary and secondary storage resources.
Within a few minutes after the reboot, the CloudStack management server freaked out with the below error messages while trying to start the system VMs (SSVM/CPVM):
[text]
System Alert: Number of unallocated private IPs is low in pod POD-XEN1 of availability zone Prod
Number of unallocated private IPs is low, total: 10.0, allocated: 10.0 (100%)
[/text]
Sure enough, all IP addresses for the POD was shown as taken in the op_dc_ip_address_alloc table.
[text]
mysql> select * from op_dc_ip_address_alloc;
+—-+—————+—————-+——–+——–+————————————–+———————+————-+
| id | ip_address | data_center_id | pod_id | nic_id | reservation_id | taken | mac_address |
+—-+—————+—————-+——–+——–+————————————–+———————+————-+
| 1 | 192.168.44.60 | 1 | 1 | 7 | c8fa9939-a7ed-4d7e-978b-606954a8d924 | 2014-04-12 14:12:29 | 51 |
| 2 | 192.168.44.61 | 1 | 1 | 7 | f22eaf13-5383-423d-a269-c499f1c5bc75 | 2014-04-12 14:12:28 | 52 |
| 3 | 192.168.44.62 | 1 | 1 | 7 | db7d7a46-295b-43f0-aaba-ee33f5d5d34d | 2014-04-12 14:12:28 | 53 |
| 4 | 192.168.44.63 | 1 | 1 | 7 | e51bbf11-7000-4117-b309-a60235731fdc | 2014-04-12 14:12:30 | 54 |
| 5 | 192.168.44.64 | 1 | 1 | 7 | cad0e89f-34b5-4d8e-9b8b-bcb8ddcb267d | 2014-04-12 14:00:05 | 55 |
| 6 | 192.168.44.65 | 1 | 1 | 7 | 2f70a5a9-cd7b-467f-b83b-c3996b72f717 | 2014-04-12 14:12:28 | 56 |
| 7 | 192.168.44.66 | 1 | 1 | 7 | 3817c4de-c8cd-42a0-a0bf-dd6779dfe668 | 2014-04-12 14:12:27 | 57 |
| 8 | 192.168.44.67 | 1 | 1 | 7 | c727ff6e-f5a7-4a5d-970b-6b3affdad7a5 | 2014-04-12 14:12:30 | 58 |
| 9 | 192.168.44.68 | 1 | 1 | 7 | 00e60594-cc35-46e1-8e6f-3e31ca075807 | 2014-04-12 14:12:29 | 59 |
| 10 | 192.168.44.69 | 1 | 1 | 7 | b3ddcd9b-6a45-44f9-b14a-90f8f372a6a5 | 2014-04-12 14:12:29 | 60 |
| 11 | 192.168.44.70 | 1 | 2 | NULL | NULL | NULL | 61 |
| 12 | 192.168.44.71 | 1 | 2 | NULL | NULL | NULL | 62 |
| 13 | 192.168.44.72 | 1 | 2 | NULL | NULL | NULL | 63 |
| 14 | 192.168.44.73 | 1 | 2 | NULL | NULL | NULL | 64 |
| 15 | 192.168.44.74 | 1 | 2 | NULL | NULL | NULL | 65 |
| 16 | 192.168.44.75 | 1 | 2 | NULL | NULL | NULL | 66 |
| 17 | 192.168.44.76 | 1 | 2 | NULL | NULL | NULL | 67 |
| 18 | 192.168.44.77 | 1 | 2 | NULL | NULL | NULL | 68 |
| 19 | 192.168.44.78 | 1 | 2 | NULL | NULL | NULL | 69 |
| 20 | 192.168.44.79 | 1 | 2 | NULL | NULL | NULL | 70 |
+—-+—————+—————-+——–+——–+————————————–+———————+————-+
20 rows in set (0.00 sec)
[/text]
According to the CLOUDSTACK-2813 and mailing list discussion on Console Proxy VM failed to start, the fix is to set nic_id and reservation_id to NULL. While CLOUDSTACK-2813, is marked as fixed, my ACS 4.2.1 installation was affected.
After cleaning up the tables, the console proxy and secondary storage VMs started without issues.
[text]
mysql> select * from op_dc_ip_address_alloc;
+—-+—————+—————-+——–+——–+————————————–+———————+————-+
| id | ip_address | data_center_id | pod_id | nic_id | reservation_id | taken | mac_address |
+—-+—————+—————-+——–+——–+————————————–+———————+————-+
| 1 | 192.168.44.60 | 1 | 1 | NULL | NULL | NULL | 51 |
| 2 | 192.168.44.61 | 1 | 1 | NULL | NULL | NULL | 52 |
| 3 | 192.168.44.62 | 1 | 1 | 2786 | 131d7334-3b81-46ea-80b2-10ff015028a8 | 2014-04-13 07:03:42 | 53 |
| 4 | 192.168.44.63 | 1 | 1 | NULL | NULL | NULL | 54 |
| 5 | 192.168.44.64 | 1 | 1 | 2785 | 131d7334-3b81-46ea-80b2-10ff015028a8 | 2014-04-13 07:03:42 | 55 |
| 6 | 192.168.44.65 | 1 | 1 | 2758 | 52cca31f-dbb7-46c7-8bc8-4133b9dd963d | 2014-04-13 07:03:42 | 56 |
| 7 | 192.168.44.66 | 1 | 1 | NULL | NULL | NULL | 57 |
| 8 | 192.168.44.67 | 1 | 1 | NULL | NULL | NULL | 58 |
| 9 | 192.168.44.68 | 1 | 1 | NULL | NULL | NULL | 59 |
| 10 | 192.168.44.69 | 1 | 1 | NULL | NULL | NULL | 60 |
| 11 | 192.168.44.70 | 1 | 2 | NULL | NULL | NULL | 61 |
| 12 | 192.168.44.71 | 1 | 2 | NULL | NULL | NULL | 62 |
| 13 | 192.168.44.72 | 1 | 2 | NULL | NULL | NULL | 63 |
| 14 | 192.168.44.73 | 1 | 2 | NULL | NULL | NULL | 64 |
| 15 | 192.168.44.74 | 1 | 2 | NULL | NULL | NULL | 65 |
| 16 | 192.168.44.75 | 1 | 2 | NULL | NULL | NULL | 66 |
| 17 | 192.168.44.76 | 1 | 2 | NULL | NULL | NULL | 67 |
| 18 | 192.168.44.77 | 1 | 2 | NULL | NULL | NULL | 68 |
| 19 | 192.168.44.78 | 1 | 2 | NULL | NULL | NULL | 69 |
| 20 | 192.168.44.79 | 1 | 2 | NULL | NULL | NULL | 70 |
+—-+—————+—————-+——–+——–+————————————–+———————+————-+
20 rows in set (0.00 sec)
mysql>
[/text]
