This morning my prod vROps presented to me the following dialog box:
Thinking quickly I checked the datastore the instance was on, no “datastore full” issues. If there were, multiple VMs on that datastore would’ve been stunned. Status of datastore:
Next, I console’d into the machine with the root account to check the status of the data disk and sure enough the /storage/db directory (note these measurements are in kilobytes) was full:
To increase disk space:
- Shut down your VM
- Edit your VM’s settings, and increase the capacity of your data vmdk file
- Power your VM back on.
- Expanding the /storage/db directory may take a while (mine took about 10 mins) depending on how big/busy your environment is. Sit tight.
- After you see the logon screen in your console session, check the status of the data disk again. Here’s mine:
I used to think vROps would now be able to bring its services online, but it doesn’t. In 5.8 I believe it did, but not in 6.0.2 which is what I’m running. Here’s a shot of the status of the services:
Next, I recalled the admin page allows for nodes to be brought back online. Checked the admin page and sure enough the node’s offline and the master node thinks it’s offline:
Hit bring online and sit tight again for a while. Finally:
Logging back in I saw the alert that should’ve been picked up:
My next step was to configure smtp alerting and have it send alerts to my team’s inbox.