vROps service unavailable! The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Error 503.

This morning my prod vROps presented to me the following dialog box:

vROpsDiskFull2

Thinking quickly I checked the datastore the instance was on, no “datastore full” issues. If there were, multiple VMs on that datastore would’ve been stunned. Status of datastore:

vROpsDiskFull3

Next, I console’d into the machine with the root account to check the status of the data disk and sure enough the /storage/db directory (note these measurements are in kilobytes) was full:

vROpsDiskFull_1

To increase disk space:

  • Shut down your VM
  • Edit your VM’s settings, and increase the capacity of your data vmdk file
  • Power your VM back on.
  • Expanding the /storage/db directory may take a while (mine took about 10 mins) depending on how big/busy your environment is. Sit tight.
  • After you see the logon screen in your console session, check the status of the data disk again. Here’s mine:

vROpsDiskFull5

I used to think vROps would now be able to bring its services online, but it doesn’t. In 5.8 I believe it did, but not in 6.0.2 which is what I’m running. Here’s a shot of the status of the services:

vROpsDiskFull6

Next, I recalled the admin page allows for nodes to be brought back online. Checked the admin page and sure enough the node’s offline and the master node thinks it’s offline:

vROpsDiskFull7

Hit bring online and sit tight again for a while. Finally:

vROpsDiskFull9

Logging back in I saw the alert that should’ve been picked up:

vROpsDiskFull8

My next step was to configure smtp alerting and have it send alerts to my team’s inbox.

Leave a Comment

Your email address will not be published.