Automated Shutdown XenServer 7.x with Network UPS Tool

Having an automated UPS shutdown of critical servers in a power failure is… critical. Unfortunately XenServer, for all of its powerfulness lacks an easy way to configure UPS initiated shutdowns. Keep reading to learn how to keep your servers up and on line, even when the power goes down.

The Goal

We have a XenServer running several virtual guests (VM’s), when we lose power we want to start shutting down the guest VM’s safely and then power down the XenServer.

Tools

We will be using a Cyberpower Network UPS with the RMCARD205 installed. This card allows us to perform SNMP monitoring of the UPS, as well as allows for up to 50 additional devices to monitor the UPS status.

NUT, the Network UPS Toolkit, will be used as a “server” to tell “clients” when to shut down. More specifically it will tell the clients the condition of the battery, and the clients will be able to shutdown on their own based on the information received.

We will be using pfSense as the NUT Master, but you could just as easily use any other system that would always be on to monitor the UPS. You may consider using a Raspberry Pi devices as a dedicated “server”, and have your clients connect to the Pi as well.

Native Software:

Before we jump into the setup, I want to inform you that you “could” install CyberPower’s Client on windows and linux systems, running in your VM’s and use that software to independently manage the shutdown process. With out using NUT, or a NUT Server.

So why are we using NUT? Because installing additional software to your VM Guests, when not needed is bad. Its one additional piece of software that must be maintained, and configured. As you will soon see by using NUT and a little scripting, we can configure the XenServer (host) to shutdown all guests. This has the additional benefit that if you add a new VM and “forget” to install the battery management software, you do not have to worry about having the VM never shutdown in a power failure.

Installing NUT

If you are using pfSense, you can install nut from the Package Manager. Once installed you will be able to access the setup from Services > UPS.

Select “Remote snmp” from the UPS Type, provide the UPS Name as ups, and set the Remote IP address to the IP or hostname of the UPS.

After clicking save, you should be able to see the status on the  UPS Status page.

That wasn’t too hard was it? Okay so now that we have NUT talking to the UPS, we need to tell NUT how to behave, and to configure nut clients to listen to the NUT server.

XenServer goes NUTS

For our test we are using the current XenServer 7.4 (Actually we are using XCP-ng 7.4). We need to install the NUT Client and configure it to talk to the NUT Server running on our pfSense.

We will be installing software in the Dom0. If you are uncomfortable doing this, You may want to modify this setup to run as a dedicated VM. Perhaps using Alpine Linux would be a really good option and not require much resources, or consider deploying a dedicated appliance such as a Raspberry Pi, which I hope to cover in the future.

By default the built in repository does not have the required library, so we need to enable the epel repository before we can install the nut-client

yum --enablerepo=* install epel-release

yum --enablerepo=* install nut-client.x86_64

Now we need to tell NUT that we want it to listen to an existing NUT Server, in other words, we want to be a “client”.

nano /etc/ups/nut.conf
MODE=netclient

Now that NUT knows it will be retrieving information from another network server, we need to tell it what network server to “MONITOR”. You will need to modify only 2 variables in the file /etc/ups/upsmon.conf but take a look around because you can tweak how the NUT Client responds by modifying the settings in this file. For now, lets just tell NUT where to retrieve its information from.

nano /etc/ups/upsmon.conf
MONITOR [email protected] 1 upsremote password slave
SHUTDOWNCMD "/etc/ups/xen-shutdown.sh"

You probably already have a SHUTDOWNCMD in your configuration Just add a comment “#” to the start of the line and add the one provided above. Speaking of the SHUTDOWNCMD, you will notice that we are calling a specific script. This script can be downloaded here.

cd /etc/ups
wget https://raw.githubusercontent.com/serrc-techops/NUT-Configuration/master/slave/xen/xen-shutdown.sh
chmod +x xen-shutdown.sh

This script will loop through all of the VM’s running on our server and shut them down. This does require that you have XenTools installed in your guests, so if you have not done that, go do it, you will have better VM performance as well! You may want to consider modifying the script to fit your needs, such as perhaps pausing the state of the machine rather then shutting it down.

So on a power failure, the NUT Master sends a message to all subscribed NUT Clients, and the NUT Client shuts down based on its configuration. Wonderful Now our Xen Guests can shutdown, and so can our server. But what happens when we power them back up? We need to make sure that NUT starts on our XenServer.

To do so create a new file, /etc/systemd/system/nut-monitor.service and add the following to it.

 

[Unit]
Description=Network UPS Tools - XenServer Shutdown
After=local-fs.target network.target
 
[Service]
ExecStart=/usr/sbin/upsmon
PIDFile=/run/nut/upsmon.pid
Type=forking
 
[Install]
WantedBy=multi-user.target

Now that we have a system startup script in place, lets make sure it runs on start up.

 

systemctl enable nut-monitor.service
systemctl daemon-reload
systemctl start nut-monitor

You can verify that it started up with out issues by entering systemctl status nut-monitor

Testing Shutdown

There are 2 things we need to test. First test the xen-shutdown.sh script from your XenServer. It should shutdown all guests and then shut down the XenServer host. Once the XenServer host is shutdown you will need to manually power it up, use WoL, if it has been enabled, or use IPMI, iLO, Drac to remotely start the system again.

If the system shut down correctly, and you have powered it back up, you will want to test that NUT is actually working for you. Do this by logging in to the console of your pfSense, or NUT server, and issuing the command:

upsmon -c fsd

That should shut everything down, including your pfSense or NUT Server. So make sure you have a way of powering it back up.

Conclusion

NUT allows you to monitor UPS devices, including network attached devices with SNMP management cards. Allowing you to have clients that can retrieve the results from the server and execute actions or scripts based on the UPS Status. We just scratched the surface, next time we will configure a QNap to monitor the status and shut down as needed. You can also setup a NUT Mater to work with multiple UPS units. Installing NUT at the Host level allows you to “forget” to install client tools per VM, and assure all your critical systems shut down safely.