Skip to main content

Troubleshooting and feedback

The Tech Preview is a community torture test: finding the breaking points is the whole point. This page covers where to look when something seems wrong, and how to report what you find so it can be fixed.

First reflexes​

  1. twinstor status on both nodes. A healthy pair shows ready, Primary / UpToDate / Connected on both sides. During a recovery, an auto-heal: section shows what the service is doing; during a resync, a sync: line shows progress.
  2. Check the alerts in Xen Orchestra (or xe message-list). Every state that needs an operator names the decision and the procedure in the alert body. The full list of alerts is in Administration, and the complete list of states that wait for you is in Failure behavior.
  3. Read the log: tail -f /var/log/twinstor.log. Every state transition and recovery action is logged; incidents appear as numbered episodes from open to close. ERROR lines are the ones that matter.

Remember the design posture: when TwinStor holds instead of acting, doing nothing is safe. Both copies of your data are kept until you decide. Never improvise low-level storage commands on a held state; use the procedure named in the alert, or ask on the forum thread first.

Things that look like bugs but are not​

  • A resync after any reboot or rejoin. Expected: the returning node catches up on the writes it missed. Reads keep being served transparently during it; redundancy is simply absent until both sides show UpToDate.
  • A "ghost" VM right after a very fast host reboot. If a host reboots faster than the HA timeout, XCP-ng can report its VMs as running with no live domain behind them for up to that timeout, then resolves it on its own. It is neither a stranded VM nor a double-run.
  • A host disabled for a few minutes after a violent incident. The toolstack can boot before storage converged and leave the host disabled; TwinStor re-enables it within about 5 minutes (alerting if it cannot). If a returned host stays stuck disabled with an "invalid host UUID" error, the fix runs on the pool master (disable then re-arm HA), not on the returned host, and the daemon attempts exactly that on its own first.

Each very short flap can cost a bounded auto-resolved split-brain, and the winner is the pool master, so a flaky link on the master repeatedly punishes the healthy peer. On the twinstor_flapping_link alert: make the healthy node the pool master (xe pool-designate-new-master), evacuate and disable the flaky host, fix the hardware (cable, then port, then NIC), and re-enable. In arbitrated mode, automatic failback to the direct link latches off after repeated strikes; once the hardware is fixed, run twinstor failback-replication.

Replacing failed hardware​

  • A failed disk is not an emergency. VMs keep running, served from the peer's copy, and the pool alerts you. Replace the disk when convenient. With an md-RAID backing, the RAID rebuild itself is all that is needed. With a dedicated disk or partition, twinstor replace-backing <new-device> re-points the node at the replacement in one command and starts the full resync from the peer (the twinstor_backing_missing alert names it too). Expect a full resync at roughly 100 MiB/s on 1 Gbps (about 90 minutes for 500 GB), faster on 10 Gbps; VM performance is preserved during it, and running the VMs on the healthy node until it finishes is the prudent choice.
  • A dead host is survived automatically (its VMs restart on the peer). Replacing the machine follows a step-by-step runbook; ask on the TwinStor thread for the current procedure matching your build.

What to test during the Tech Preview​

Functional, realistic failures are what the preview needs: the kind that genuinely happen to two hosts sitting in a closet at a remote site.

  • Yank network cables (one node, both nodes, mid-write).
  • Yank power cables, including a global power cut to both hosts at once.
  • Switch port flapping, switch reboots, full switch power loss.
  • Host reboots, graceful and ungraceful, and Rolling Pool Updates.
  • Re-running install.sh to stage a new build over a running one.
  • VM live migration in every direction, repeatedly, during all of the above.
  • Disk pull, disk failure, disk re-insertion.

If you can corrupt data, lose an acknowledged write, or wedge the cluster into a state it cannot recover from, that is gold, and exactly what the team needs to hear about.

One category is explicitly out of scope: using a firewall to selectively block only some traffic so that auto-heal becomes impossible. No real deployment does that to itself; it is poking the model with tweezers, not testing reality.

Reporting a problem​

When something breaks (please make it break):

  1. Run twinstor support-bundle on both nodes. It packages the logs, redacted configuration, and a full state snapshot for offline analysis; no sensitive credentials are included.
  2. Note what you did, as an exact sequence: "pulled power on A during a migration, then rebooted the switch".
  3. Note what you expected versus what happened (data loss? wedged state? VMs not restarting?).
  4. Post it on the TwinStor thread with the two bundles attached, and include your twinstor --version in the report.