Administration

Day-to-day, TwinStor is designed to be invisible: the SR behaves like any shared storage in XCP-ng, and everything else runs unattended. This page covers what a normal day looks like, the CLI, monitoring, and planned maintenance.

Using the storage

The TwinStor SR is a standard shared Storage Repository. From Xen Orchestra (or xe), use it exactly like any other shared SR:

• Create VMs and disks on it.
• Live-migrate VMs between the two hosts, in any direction.
• Snapshots and backups work as usual. Keep running your backup jobs: replication protects against hardware failure, not against deletion or ransomware.
• HA is enabled by the setup: if a host dies, its VMs restart automatically on the survivor. TwinStor also backstops HA by restarting protected VMs that an incident left halted, so to keep a protected VM deliberately down, clear its ha-restart-priority first.

VMs with PCI passthrough (GPU, NIC, HBA)

A VM with a passthrough device is pinned to one host, but XCP-ng's HA does not know that: it judges a VM movable from storage and memory alone, admits it to the failover plan, then fails to restart it on the survivor (the device is not there) and the VM stays halted for the whole outage. For such VMs, do not use HA protection; use auto power-on instead, and note the pool-level flag is required or the per-VM one is silently ignored:

root@twinstor-1 — auto power-on instead of HA

# xe pool-param-set uuid=<pool-uuid> other-config:auto_poweron=true
# xe vm-param-set uuid=<vm-uuid> other-config:auto_poweron=true
# xe vm-param-set uuid=<vm-uuid> ha-restart-priority=""

HA's failover plan and migrations

With HA tolerating one host failure, XCP-ng refuses any migration (manual or from the XO Load Balancer) that would leave the survivor unable to restart every protected VM: the operation fails with HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN. The plan reserves each protected VM's memory-static-max (not its current dynamic memory), so a tight pool hits this earlier than expected. Mark less critical VMs best-effort to keep them out of the plan rather than disabling the protection budget.

Pair it with the XO Load Balancer

Because both hosts share the same SR, VMs can move freely. If you run application-level HA inside your VMs (a replicated database, a pair of front-ends), use the Xen Orchestra Load Balancer's anti-affinity to keep the members on separate hosts, so losing one host never takes both copies of your application down at once.

The CLI

Everything lives in one binary, run from dom0:

root@twinstor-1 — the twinstor CLI

# twinstor --version         # which build am I running (include this in any report)
# twinstor status            # show replication, iSCSI, multipath and sync status
# twinstor status --json     # same, machine-readable (for monitoring)
# twinstor support-bundle    # collect logs + redacted config + state snapshot for a problem report
# twinstor setup             # interactive wizard; on an installed node: settings, status, reconfigure
# twinstor uninstall         # remove TwinStor from this node

Commands you only run when an alert or a release note asks for them:

root@twinstor-1 — recovery commands

# twinstor discard-and-resync     # resolve an undecidable split-brain: run on the node whose copy you discard
# twinstor failback-replication   # arbitrated: bring replication back to the direct link after a repaired fault
# twinstor replace-backing <dev>  # re-point this node at a replacement backing device, full resync from the peer
# twinstor regenerate-config      # bring the generated replication config up to the installed version (dry-run by default)

Readiness states

The readiness: line of twinstor status is the one-word answer to "can this node serve VMs safely":

State	Meaning
ready	Serving, peer holds a healthy redundant copy. Normal state.
serving-without-redundancy	Serving, but the peer is not a healthy copy right now (resync, peer down). A second failure would mean an outage.
degraded-paths	Data is fully redundant but the local path stack is impaired; TwinStor self-heals it.
not-ready	The local copy is not safe to serve, so the node deliberately withholds its storage target; the peer serves. The reason field says why.

twinstor status also names this host, the peer, the current pool master, and whether pool HA is armed.

A healthy node shows Primary / UpToDate / Connected on both sides:

root@twinstor-1 — twinstor status

# twinstor status

health:  ● healthy  — serving, peer redundant

   version:     0.4.16
   daemon:      running
   resource:    twinstor-data
   host:        twinstor-1 (this node), peer twinstor-2
   pool master: twinstor-1 (this node)
   pool HA:     enabled

✓ drbd
     role:       Primary
     disk:       UpToDate
     connection: Connected
     peer role:  Primary
     peer disk:  UpToDate

✓ readiness:   ready (local serving, peer redundant)
✓ backing:     sda: write cache disabled (durable)
✓ iscsi:       active
✓ multipath:   ok (local prio 50 active, remote prio 10 enabled)
✓ tiebreaker:  192.168.1.1 (gateway)

During a resync (expected after any reboot or rejoin), the output shows progress:

root@twinstor-2 — twinstor status

# health:  ● degraded  — peer resyncing, redundancy incomplete

⚠ drbd
     disk:       UpToDate
     connection: Connected
     peer disk:  Inconsistent (SyncTarget)
     sync:       45.2% done, ETA 0:12:34

During a recovery episode an auto-heal: section shows what the daemon is doing and where it stands.

Monitoring

Probe: twinstor status --json

One-shot, safe to poll every 30 to 60 seconds, on each node. Suggested rules:

Condition	Severity	Meaning
daemon == false	critical	daemon not running (systemd restarts it; investigate if persistent)
readiness.status == "not-ready"	critical	node cannot serve; reason says why
readiness.status == "serving-without-redundancy"	warning	one healthy copy only; normal during a resync or peer reboot, alert if it persists more than 15 min
readiness.status == "degraded-paths"	warning	local path stack impaired but still serving; self-heals, alert if it persists
iscsi == false while readiness == ready	critical	target should be up but is not
sync != null	info	resync in progress (percent, eta)

Alerts in Xen Orchestra

TwinStor raises pool-level XAPI alerts, visible in XO (and forwardable from it). All are priority 1:

Alert	Meaning
twinstor_split_brain	split-brain detected and/or resolved; read the body and verify the VMs it names
twinstor_degraded	local disk failed, or the peer is no longer a healthy redundant copy; serving without redundancy
twinstor_ha_disarmed	pool HA has been off for 30 minutes or more; re-arm it
twinstor_flapping_link	repeated link flaps; check NIC, cable, or switch port
twinstor_ha_wedge	dual-master wedge detected; auto-heal is running
twinstor_replication_fallback	arbitrated mode: replication left the direct link for the switch path; check the cable, failback is automatic
twinstor_cannot_go_online	node cannot reach a serving state; the body categorizes the cause
twinstor_no_tiebreaker	arbitrated mode: no ping target beyond the gateway is reachable; arbitration is weakened
twinstor_peer_daemon_down	the peer's daemon stopped responding while its host looks healthy
twinstor_backing_missing	the configured backing device no longer exists (replaced disk, recreated array); the alert names the fix (twinstor replace-backing)
protected_vm_halted	an HA-protected VM was left halted after an incident and could not be auto-restarted
master_isolated / isolated_master	arbitrated mode: the pool master lost its switch path while the direct link stays healthy; auto stand-down engages (see Failure behavior)
host_auto_enable_failed	a host stayed disabled after an incident and the automatic re-enable failed; run xe host-enable

Pool-level checks

Three xe checks complete the picture, per pool:

• xe pbd-list sr-name-label=twinstor currently-attached=false --minimal must be empty: an unplugged PBD means a host lost its storage path.
• xe host-list enabled=false --minimal: a host disabled for more than a few minutes blocks HA VM restarts.
• xe pool-param-get uuid=<pool-uuid> param-name=ha-enabled must be true in normal operation.

Logs

The daemon logs every state transition and recovery action to /var/log/twinstor.log (local time, rotated daily, 14 rotations kept). tail -f /var/log/twinstor.log to follow live. Any ERROR line is worth alerting on. Every incident is a numbered episode: the daemon stamps an episode id (like splitbrain-04680) on every related line, so grep <episode-id> /var/log/twinstor.log reconstructs an incident end to end; a clean rejoin produces no episode at all.

The fencing decisions log separately to /var/log/twinstor-fence-peer.log: exit 4 (survive, serve clients) is a node deciding to keep serving, stay frozen (exit 1) is a node deciding it is the isolated one.

Planned maintenance

The one rule to know: leave HA armed only when the work is confined to a single host and that host comes back at the same version (reboot, firmware, a hardware swap, a disk replacement). Disable HA for anything that involves both nodes, and for any update, even of a single host: an update changes the patch level, and a pool member cannot run a different version from the master.

One node, same version (firmware, hardware swap, reboot)

HA can stay armed. Evacuate first so VMs live-migrate instead of being crash-restarted:

root@twinstor-1 — evacuate before maintenance

# xe host-evacuate uuid=<node-uuid>   # live-migrate VMs to the peer
# do the maintenance, reboot as needed

TwinStor stops on shutdown (the peer keeps serving) and restarts on boot, reconnecting and resyncing automatically.

Three related notes:

• Wait for ready before moving VMs back. After a reboot, give the host a minute: starting or live-migrating a VM onto a host whose storage stack has not finished converging can crash that VM. twinstor status says ready when it is safe.
• Keeping a host down on purpose. TwinStor re-enables a host it finds disabled with healthy storage after 5 minutes (a disabled host blocks HA restarts). To hold a host in maintenance deliberately, set xe host-param-set uuid=<host-uuid> other-config:MAINTENANCE_MODE=true; the auto-enable respects it. Remove the marker and xe host-enable when done.
• Do not rename a pool host after setup. Hostnames are load-bearing for the replication configuration; a renamed node boots without storage. If a rename is unavoidable, ask on the Tech Preview thread for the procedure.

Rolling Pool Update (RPU)

Host updates are fully transparent for TwinStor: follow the Rolling Pool Update procedure and rules in Updates.

Switch maintenance

In fate-shared mode the switch carries everything, so disable HA first:

root@twinstor-1 — disable HA

# xe pool-ha-disable

With HA off, a switch outage freezes storage I/O on both sides (fail-closed, nothing diverges) and the nodes simply wait; VMs ride through a stall of a few minutes. When the switch is back, replication reconnects and I/O resumes on its own. Then re-arm HA, always passing the timeout explicitly:

root@twinstor-1 — re-enable HA

# xe pool-ha-enable heartbeat-sr-uuids=<twinstor SR uuid> ha-config:timeout=60   # 90 in arbitrated mode
# xe pool-param-set uuid=<pool-uuid> ha-host-failures-to-tolerate=1

Skipping the HA-disable turns a 3-minute switch reboot into a full-pool self-fence and a 10-minute recovery with every VM crash-restarted. In arbitrated mode the same procedure applies with a better outcome: replication rides the direct link, so storage never even pauses.

Both nodes off at once (electrical work, moving the machines)

1. Cleanly shut down all guest VMs.
2. xe pool-ha-disable
3. Shut down the slave first, then the pool master (xe host-shutdown).
4. Power both hosts back on (together or staggered; the later node syncs before it serves).
5. Confirm twinstor status is ready on both nodes.
6. Re-enable HA (explicit timeout, as above), then start the VMs.

The one hard rule: disable HA before the shutdown, and re-enable it only after both nodes are back and in sync.