Skip to main content

Troubleshooting and feedback

The Tech Preview is a community torture test: finding the breaking points is the whole point. This page covers where to look when something seems wrong, and how to report what you find so it can be fixed.

First reflexes​

  1. twinstor status on both nodes. A healthy pair shows ready, Primary / UpToDate / Connected on both sides. During a recovery, an auto-heal: section shows what the service is doing; during a resync, a sync: line shows progress.
  2. Check the alerts in Xen Orchestra (or xe message-list). Every state that needs an operator names the decision and the procedure in the alert body. The full list of alerts is in Administration, and the complete list of states that wait for you is in Failure behavior.
  3. Read the log: tail -f /var/log/twinstor.log. Every state transition and recovery action is logged; incidents appear as numbered episodes from open to close. ERROR lines are the ones that matter.

Remember the design posture: when TwinStor holds instead of acting, doing nothing is safe. Both copies of your data are kept until you decide. Never improvise low-level storage commands on a held state; use the procedure named in the alert, or ask on the forum thread first.

Things that look like bugs but are not​

  • A resync after any reboot or rejoin. Expected: the returning node catches up on the writes it missed. Reads keep being served transparently during it; redundancy is simply absent until both sides show UpToDate.
  • A "ghost" VM right after a very fast host reboot. If a host reboots faster than the HA timeout, XCP-ng can report its VMs as running with no live domain behind them for up to that timeout, then resolves it on its own. It is neither a stranded VM nor a double-run.
  • A host disabled for a few minutes after a violent incident. The toolstack can boot before storage converged and leave the host disabled; TwinStor re-enables it within about 5 minutes (alerting if it cannot). If a returned host stays stuck disabled with an "invalid host UUID" error, the fix runs on the pool master (disable then re-arm HA), not on the returned host, and the daemon attempts exactly that on its own first.
  • A protected VM left halted after a node stands down. When a fault makes one node stand down while the other keeps serving, XCP-ng HA can spend its restart attempts on a protected VM and leave it halted, because HA cannot fully re-arm while a node is stood down. From 0.5.6, the surviving node restarts those halted protected VMs on its own once it holds an up-to-date copy, so you no longer have to start them by hand. A VM you deliberately powered off, or one stopped during maintenance, is left alone and only alerts.

Each very short flap can cost a bounded auto-resolved split-brain, and the winner is the pool master, so a flaky link on the master repeatedly punishes the healthy peer. On the twinstor_flapping_link alert: make the healthy node the pool master (xe pool-designate-new-master), evacuate and disable the flaky host, fix the hardware (cable, then port, then NIC), and re-enable. In arbitrated mode, automatic failback to the direct link latches off after repeated strikes; once the hardware is fixed, run twinstor failback-replication.

A prolonged flap or an unlucky reboot during one can also leave pool HA disabled once the dust settles (a fence sets it off, and the pool can be left needing a fresh pool-ha-enable). From 0.5.5, TwinStor re-arms pool HA on its own once storage is healthy again, including the case where the HA daemon was left running while the pool believed HA was off. If you are on an earlier build and HA stays off after the link is fixed, restart the toolstack on both hosts (xe-toolstack-restart) and then re-enable HA with the explicit command from Administration.

A returning master that comes back as a second master (arbitrated)​

Cutting the pool master's client link (not a slave's) while HA is armed can, after the pair reboots, leave the two hosts as two pool masters: the survivor is the real serving master, and the host whose link was cut comes back still reading itself as master but with its management stuck — trying to rejoin in a loop, and refusing manual rejoin because "HA is disabled on this host while the pool has HA enabled". Storage is not at risk while this holds (both copies stay consistent; only the control plane is split), but the pool will not fail over correctly until it is resolved.

From 0.5.8, TwinStor resolves this on its own once the network is back: it clears the stuck local HA state and demotes the empty returning master to a slave of the real master, then HA re-arms across the pool — no operator steps. It only acts when the returning master runs no VMs and the peer is provably the real, serving master, and it waits (rather than acting blindly) while that host is still cut off from the network.

If you are on an earlier build, or you want to resolve it by hand, the manual recovery is: on the stuck former master, clear its local HA state and rejoin — xe host-emergency-ha-disable --force, then xe pool-emergency-reset-master master-address=<the real master's IP> — which restarts its toolstack as a slave of the survivor. Then re-arm HA from the master with the explicit command in Administration. To put mastership back on the original host afterwards, migrate its VMs back and let HA settle, or use xe pool-designate-new-master.

Replacing failed hardware​

  • A failed disk is not an emergency. VMs keep running, served from the peer's copy, and the pool alerts you. Replace the disk when convenient. With an md-RAID backing, the RAID rebuild itself is all that is needed. With a dedicated disk or partition, twinstor replace-backing <new-device> re-points the node at the replacement in one command and starts the full resync from the peer (the twinstor_backing_missing alert names it too). Expect a full resync at roughly 100 MiB/s on 1 Gbps (about 90 minutes for 500 GB), faster on 10 Gbps; VM performance is preserved during it, and running the VMs on the healthy node until it finishes is the prudent choice.
  • A dead host is survived automatically (its VMs restart on the peer). Replacing the machine follows a step-by-step runbook; ask on the TwinStor thread for the current procedure matching your build.

What to test during the Tech Preview​

Functional, realistic failures are what the preview needs: the kind that genuinely happen to two hosts sitting in a closet at a remote site.

  • Yank network cables (one node, both nodes, mid-write).
  • Yank power cables, including a global power cut to both hosts at once.
  • Switch port flapping, switch reboots, full switch power loss.
  • Host reboots, graceful and ungraceful, and Rolling Pool Updates.
  • Re-running install.sh to stage a new build over a running one.
  • VM live migration in every direction, repeatedly, during all of the above.
  • Disk pull, disk failure, disk re-insertion.

If you can corrupt data, lose an acknowledged write, or wedge the cluster into a state it cannot recover from, that is gold, and exactly what the team needs to hear about.

One category is explicitly out of scope: using a firewall to selectively block only some traffic so that auto-heal becomes impossible. No real deployment does that to itself; it is poking the model with tweezers, not testing reality.

Reporting a problem​

When something breaks (please make it break):

  1. Run twinstor support-bundle on both nodes. It packages the logs, redacted configuration, and a full state snapshot for offline analysis; no sensitive credentials are included.
  2. Note what you did, as an exact sequence: "pulled power on A during a migration, then rebooted the switch".
  3. Note what you expected versus what happened (data loss? wedged state? VMs not restarting?).
  4. Post it on the TwinStor thread with the two bundles attached, and include your twinstor --version in the report.