Skip to main content

Troubleshooting and feedback

The Tech Preview is there to find what breaks. This page covers what to check when something looks wrong, and how to report it.

First checks​

  1. Run twinstor status on both hosts. A healthy pair shows health: healthy and readiness: ready. During a recovery or a resync, status shows what is going on.
  2. Read the alerts in Xen Orchestra. Each one says what happened and what to do (see Administration).
  3. Run twinstor doctor: it checks the configuration and prints the fix for anything wrong.
  4. Look at /var/log/twinstor.log. ERROR lines are the ones that matter.

When TwinStor is waiting instead of acting, doing nothing is safe. Do not run low-level storage commands by hand: follow the alert, or ask on the forum first.

Things that look wrong but are normal​

  • A resync after a reboot. The returning host catches up on the writes it missed. VMs keep running.
  • A VM shown as running right after a very fast host reboot. XCP-ng clears it within the HA timeout.
  • A host disabled for a few minutes after an incident. TwinStor re-enables it within about 5 minutes.
  • A protected VM that stays halted after a shutdown from inside the guest (fate-shared mode). Start it with xe vm-start.
  • A halted VM marked twinstor-disks-suspect. It ran across a split-brain. Check its disks as the alert says, then start it.

On a twinstor_flapping_link alert, the NIC, cable or switch port is failing. Make the healthy host the pool master (xe pool-designate-new-master), move the VMs off the other host, fix the hardware and bring the host back. In arbitrated mode, then run twinstor failback-replication to move replication back to the direct cable.

Replacing failed hardware​

  • A failed disk is not an emergency. VMs keep running from the other copy. With md-RAID, the RAID rebuild is all you need. Otherwise, install the new disk and run twinstor replace-backing <new-device>: it starts a full resync from the other host, about 90 minutes for 500 GB on 1 Gbit/s. Run the VMs on the healthy host until it finishes.
  • A dead host is survived automatically: its VMs restart on the other host. To replace the machine, ask on the TwinStor thread for the procedure matching your version.

What to test​

Real-world failures are the most useful:

  • network cables pulled (one host, both, during writes);
  • power cut, including both hosts at once;
  • switch reboots and port flaps;
  • host reboots, clean or not, and Rolling Pool Updates;
  • live migrations during all of the above;
  • disks pulled and put back.

Firewall rules that block only part of the traffic are out of scope: no real setup does that to itself.

Reporting a problem​

  1. Run twinstor support-bundle on both hosts. It collects logs, configuration (secrets removed) and state.
  2. Write down what you did, in order, and what you expected.
  3. Post it on the TwinStor thread with both bundles and your twinstor --version.

Lost data, a confirmed write that disappeared, or a pool that cannot recover on its own are the reports we want most.