Skip to main content

Administration

Once set up, TwinStor runs on its own. The SR behaves like any shared SR, and this page covers the few things worth knowing day to day.

Using the storage​

From Xen Orchestra or xe, use the TwinStor SR like any shared SR:

  • Create VMs and disks on it, and live-migrate VMs between the two hosts.
  • Take snapshots and run your backup jobs as usual.
  • HA is enabled by the setup: if a host dies, its VMs restart on the other one. TwinStor also restarts protected VMs that an incident left halted, so to keep a protected VM down on purpose, clear its ha-restart-priority first.

VMs with PCI passthrough​

A VM with a passthrough device (GPU, NIC, HBA) can only run on its own host, but HA does not know that and fails to restart it on the other host. Do not protect such a VM with HA; use auto power-on instead (the pool-level setting is required):

root@twinstor-1: auto power-on instead of HA
# xe pool-param-set uuid=<pool-uuid> other-config:auto_poweron=true
# xe vm-param-set uuid=<vm-uuid> other-config:auto_poweron=true
# xe vm-param-set uuid=<vm-uuid> ha-restart-priority=""

HA failover capacity​

The pool setting ha-host-failures-to-tolerate should be 1. XCP-ng then keeps enough room on each host to restart every protected VM, and warns you (HA_POOL_OVERCOMMITTED) when that stops being true. twinstor setup sets it, but enabling HA from Xen Orchestra or by hand can leave it at 0. Check it and set it with:

root@twinstor-1: HA failover capacity
# xe pool-list params=ha-enabled,ha-host-failures-to-tolerate,ha-overcommitted
# xe pool-param-set uuid=$(xe pool-list --minimal) ha-host-failures-to-tolerate=1

With 1, XCP-ng refuses to start or migrate a protected VM when the remaining host could no longer restart them all. If your protected VMs do not fit on one host, set the less important ones to best-effort rather than lowering the setting.

Commands​

root@twinstor-1: everyday commands
# twinstor status            # health of this host and its peer
# twinstor status --json     # the same, for monitoring tools
# twinstor doctor            # check the configuration and print fixes
# twinstor support-bundle    # collect logs and state for a problem report
# twinstor --version

Other commands (discard-and-resync, failback-replication, replace-backing, regenerate-config) are only needed when an alert or these docs tell you to run them.

The readiness: line of twinstor status answers "can this host serve VMs safely?":

State Meaning
ready Serving, and the other host has a healthy copy. The normal state.
serving-without-redundancy Serving, but the other host's copy is not usable right now (resync, host down).
degraded-paths Data is redundant, but the local storage path has a problem. TwinStor repairs it.
not-ready This host's copy is not safe to serve, so the other host serves. The reason is shown.

After a reboot, the returning host resyncs and twinstor status shows the progress.

Monitoring​

Alerts​

TwinStor raises pool alerts, visible in Xen Orchestra. Each alert says what happened and what to do.

Alert Meaning
twinstor_degraded A disk failed or the other host's copy is not usable: running on one copy.
twinstor_split_brain The two copies diverged. Read the alert: it names the VMs involved, and says if a decision is needed.
twinstor_ha_disarmed Pool HA has been off for 30 minutes or more.
twinstor_flapping_link The link between the hosts keeps dropping. Check the NIC, cable and switch port.
twinstor_replication_fallback Arbitrated mode: replication left the direct cable for the switched network. Check the cable.
twinstor_backing_missing The data device is gone (replaced disk). The alert gives the command to use.
twinstor_cannot_go_online A host cannot start serving. The alert gives the cause.
twinstor_peer_daemon_down The TwinStor service on the other host stopped responding.
twinstor_ha_wedge Both hosts think they are pool master. TwinStor is fixing it.
twinstor_no_tiebreaker Arbitrated mode: no ping target answers besides the gateway.
protected_vm_halted A protected VM stayed halted after an incident and could not be restarted.
master_isolated / isolated_master Arbitrated mode: the pool master lost its network. Its VMs are moved to the other host.
host_auto_enable_failed A host stayed disabled after an incident. Run xe host-enable.

Monitoring tools​

twinstor status --json is safe to poll every 30 to 60 seconds on each host. Alert when daemon is false or readiness.status is not-ready, and warn when it stays serving-without-redundancy for more than 15 minutes.

Logs​

The service logs to /var/log/twinstor.log (rotated daily, 14 days kept). Each incident gets an id on every related line, so you can follow it with grep. Fencing decisions are also logged in /var/log/twinstor-fence-peer.log. When you report a problem, twinstor support-bundle collects both.

Planned maintenance​

The rule: HA can stay on for work on one host that comes back at the same version (reboot, firmware, hardware swap). Turn HA off for anything that touches both hosts, and for any update.

One host (reboot, firmware, hardware)​

root@twinstor-1: before maintenance
# xe host-evacuate uuid=<host-uuid>   # live-migrate its VMs to the other host

Do the work and reboot as needed. TwinStor stops and starts with the host and resyncs on its own.

  • Wait until twinstor status shows ready before moving VMs back.
  • TwinStor re-enables a disabled host after 5 minutes. To keep a host disabled on purpose, set xe host-param-set uuid=<host-uuid> other-config:MAINTENANCE_MODE=true, and remove it when you are done.
  • Do not rename a host after setup: the renamed host boots without storage.

Switch maintenance​

In fate-shared mode the switch carries everything, so turn HA off first:

root@twinstor-1: switch maintenance
# xe pool-ha-disable
# switch work: storage pauses on both hosts and resumes when the switch is back
# xe pool-ha-enable heartbeat-sr-uuids=<TwinStor SR uuid> ha-config:timeout=120   # 90 in arbitrated mode

With HA on, a switch reboot longer than the HA timeout reboots both hosts and restarts every VM. In arbitrated mode, replication uses the direct cable and storage does not even pause, but turning HA off is still the safe choice.

Both hosts off (electrical work, moving the hardware)​

  1. Shut down the VMs.
  2. xe pool-ha-disable
  3. Shut down the other host first, then the pool master.
  4. Power both hosts back on.
  5. Wait for twinstor status to show ready on both.
  6. Turn HA back on (command above) and start the VMs.

For updates, see Updates.