Having an automated UPS shutdown of critical servers in a power failure is… critical. Unfortunately XenServer, for all of its powerfulness lacks an easy way to configure UPS initiated shutdowns. Keep reading to learn how to keep your servers up and on line, even when the power goes down.
We have a XenServer running several virtual guests (VM’s), when we lose power we want to start shutting down the guest VM’s safely and then power down the XenServer.
We will be using a Cyberpower Network UPS with the RMCARD205 installed. This card allows us to perform SNMP monitoring of the UPS, as well as allows for up to 50 additional devices to monitor the UPS status.
NUT, the Network UPS Toolkit, will be used as a “server” to tell “clients” when to shut down. More specifically it will tell the clients the condition of the battery, and the clients will be able to shutdown on their own based on the information received.
We will be using pfSense as the NUT Master, but you could just as easily use any other system that would always be on to monitor the UPS. You may consider using a Raspberry Pi devices as a dedicated “server”, and have your clients connect to the Pi as well.
Before we jump into the setup, I want to inform you that you “could” install CyberPower’s Client on windows and linux systems, running in your VM’s and use that software to independently manage the shutdown process. With out using NUT, or a NUT Server.
So why are we using NUT? Because installing additional software to your VM Guests, when not needed is bad. Its one additional piece of software that must be maintained, and configured. As you will soon see by using NUT and a little scripting, we can configure the XenServer (host) to shutdown all guests. This has the additional benefit that if you add a new VM and “forget” to install the battery management software, you do not have to worry about having the VM never shutdown in a power failure.
If you are using pfSense, you can install nut from the Package Manager. Once installed you will be able to access the setup from Services > UPS.
Select “Remote snmp” from the UPS Type, provide the UPS Name as ups, and set the Remote IP address to the IP or hostname of the UPS.
After clicking save, you should be able to see the status on the UPS Status page.
That wasn’t too hard was it? Okay so now that we have NUT talking to the UPS, we need to tell NUT how to behave, and to configure nut clients to listen to the NUT server.
XenServer goes NUTS
For our test we are using the current XenServer 7.4 (Actually we are using XCP-ng 7.4). We need to install the NUT Client and configure it to talk to the NUT Server running on our pfSense.
We will be installing software in the Dom0. If you are uncomfortable doing this, You may want to modify this setup to run as a dedicated VM. Perhaps using Alpine Linux would be a really good option and not require much resources, or consider deploying a dedicated appliance such as a Raspberry Pi, which I hope to cover in the future.
By default the built in repository does not have the required library, so we need to enable the epel repository before we can install the nut-client
yum --enablerepo=* install epel-release yum --enablerepo=* install nut-client.x86_64
Now we need to tell NUT that we want it to listen to an existing NUT Server, in other words, we want to be a “client”.
nano /etc/ups/nut.conf MODE=netclient
Now that NUT knows it will be retrieving information from another network server, we need to tell it what network server to “MONITOR”. You will need to modify only 2 variables in the file /etc/ups/upsmon.conf but take a look around because you can tweak how the NUT Client responds by modifying the settings in this file. For now, lets just tell NUT where to retrieve its information from.
nano /etc/ups/upsmon.conf MONITOR email@example.com 1 upsremote password slave SHUTDOWNCMD "/etc/ups/xen-shutdown.sh"
You probably already have a SHUTDOWNCMD in your configuration Just add a comment “#” to the start of the line and add the one provided above. Speaking of the SHUTDOWNCMD, you will notice that we are calling a specific script. This script can be downloaded here.
cd /etc/ups wget https://raw.githubusercontent.com/serrc-techops/NUT-Configuration/master/slave/xen/xen-shutdown.sh chmod +x xen-shutdown.sh
This script will loop through all of the VM’s running on our server and shut them down. This does require that you have XenTools installed in your guests, so if you have not done that, go do it, you will have better VM performance as well! You may want to consider modifying the script to fit your needs, such as perhaps pausing the state of the machine rather then shutting it down.
So on a power failure, the NUT Master sends a message to all subscribed NUT Clients, and the NUT Client shuts down based on its configuration. Wonderful Now our Xen Guests can shutdown, and so can our server. But what happens when we power them back up? We need to make sure that NUT starts on our XenServer.
To do so create a new file, /etc/systemd/system/nut-monitor.service and add the following to it.
[Unit] Description=Network UPS Tools - XenServer Shutdown After=local-fs.target network.target [Service] ExecStart=/usr/sbin/upsmon PIDFile=/run/nut/upsmon.pid Type=forking [Install] WantedBy=multi-user.target
Now that we have a system startup script in place, lets make sure it runs on start up.
systemctl enable nut-monitor.service systemctl daemon-reload systemctl start nut-monitor
You can verify that it started up with out issues by entering systemctl status nut-monitor
There are 2 things we need to test. First test the xen-shutdown.sh script from your XenServer. It should shutdown all guests and then shut down the XenServer host. Once the XenServer host is shutdown you will need to manually power it up, use WoL, if it has been enabled, or use IPMI, iLO, Drac to remotely start the system again.
If the system shut down correctly, and you have powered it back up, you will want to test that NUT is actually working for you. Do this by logging in to the console of your pfSense, or NUT server, and issuing the command:
upsmon -c fsd
That should shut everything down, including your pfSense or NUT Server. So make sure you have a way of powering it back up.
NUT allows you to monitor UPS devices, including network attached devices with SNMP management cards. Allowing you to have clients that can retrieve the results from the server and execute actions or scripts based on the UPS Status. We just scratched the surface, next time we will configure a QNap to monitor the status and shut down as needed. You can also setup a NUT Mater to work with multiple UPS units. Installing NUT at the Host level allows you to “forget” to install client tools per VM, and assure all your critical systems shut down safely.
Ben has been building VoIP solutions for over 10 years, has over 15 years of Linux administration experience and enjoys problem-solving. When he is not coding something in Python, or tinkering with some project, you can often find him wandering through the forests and parks of the Pacific Northwest enjoying waterfalls, trails, and animals.