Failure behavior
TwinStor keeps serving through failures and recovers on its own once it is safe. When it cannot be sure what is safe, it stops, keeps both copies of your data and asks you. Every case below was tested on real hardware: power cut, cables pulled, disks removed live.
What heals itself
| Failure | What happens |
|---|---|
| A host loses power | The other host keeps serving. HA restarts the lost host's VMs on it. When the host returns, it resyncs. |
| Power cut on both hosts | Both hosts boot, storage comes back and HA restarts the VMs, about 3 minutes in testing (requires the BIOS power-on setting from Installation). |
| A disk or RAID array fails | VMs keep running from the other host's copy. An alert is raised, and the copy resyncs once the disk is replaced. |
| The network between the hosts is cut | The host that lost the network steps aside and the other keeps serving. In fate-shared mode, the host that lost the network freezes its storage at once, so it never confirms a write the other host does not have. Its VMs are restarted on the other host (or keep running, if the other host ran none during the cut). Everything resyncs when the link is back. |
| A short network blip | A drop of a few seconds causes a short I/O pause, then everything continues. |
| A host reboots | The other host was already serving. The rebooted host catches up on its own. |
| The TwinStor service crashes | systemd restarts it. Storage keeps working meanwhile. |
| The whole switch fails | Fate-shared: if it lasts longer than the HA timeout, both hosts stop cleanly and the pool restarts when the switch is back. Arbitrated: both hosts keep their VMs running over the direct cable and wait for the switch. |
| One host loses its network (arbitrated) | That host stands down, its VMs restart on the other host, and it rejoins once its network is back. |
| The two copies diverged (split-brain) | Rare. TwinStor keeps the copy it has evidence for, stops the VMs on the other host, discards that host's few seconds of different writes, restarts the VMs from the kept copy and raises an alert naming them. |
The one failure nothing can heal is losing both disks at the same time. That is what backups are for.
Can I lose data?
A write is confirmed to a VM only when both hosts have it, so losing a host, a disk or the power never loses a confirmed write.
The only exception is a real split: both hosts keep running VMs while they cannot see each other, and both still reach the gateway. In fate-shared mode the host that lost the network freezes at once, which prevents this on a network cut (no confirmed write was lost in testing). In arbitrated mode it needs the direct cable and the switched network to fail at the same moment. Even then, the loss is limited to the last few seconds written on the losing host, and an alert names the VMs concerned.
When TwinStor waits for you
In a few cases, acting automatically would mean guessing. TwinStor then keeps everything as it is, raises an alert and waits. Doing nothing is always safe: both copies stay intact until you decide.
| Situation | How you know | What to do |
|---|---|---|
| Split-brain where TwinStor cannot tell which copy is right | twinstor_split_brain alert asking for a decision | Choose the copy to keep. The alert gives the command to run on the host whose copy you discard. |
| A VM was found running across a split-brain discard | The VM is halted and marked twinstor-disks-suspect, and an alert names it | Check its disks as the alert says, then start it yourself. |
| A VM cannot start again on one host after a split-brain | twinstor_split_brain alert with a cleanup command | The VM can run on the other host. Run the command before starting it on this host again. |
| Both hosts act as pool master and both run VMs | twinstor_ha_wedge alert | Follow the steps in the alert. |
| A host is frozen and the other host has been gone for a long time | Alert saying the host is held frozen | If the other host is really dead, the alert gives the commands to serve alone. Otherwise, fix its network and the pair heals. |
| Arbitrated: the direct link kept failing and failback is off | twinstor_replication_fallback alert, failback_latched in status | Fix the cable or NIC, then run twinstor failback-replication. |
| HA was turned off by someone | twinstor_ha_disarmed alert after 30 minutes | Turn HA back on when you are ready (see Administration). |
| Hardware is gone (both disks, or a disk while the other host is down) | Alerts naming the failure | Replace the hardware (see Troubleshooting). |
If TwinStor is waiting in a situation that is not in this table, and nothing heals, it is a bug: please report it.
Known limitations
- Fate-shared, single switch: the switch is the one weak point. If it fails longer than the HA timeout, the whole pool stops, cleanly, until it returns.
- Fate-shared, long network cut: a cut longer than about a minute can end with HA rebooting one host, sometimes a few minutes after the pair looks healthy again. No confirmed write is lost, and HA restarts that host's VMs.
- Fate-shared, shutdown from inside a VM: a protected VM shut down from inside the guest stays halted; HA does not restart it. Start it with
xe vm-start. - Arbitrated, direct cable lost: the host that does not keep serving stops its VMs after about a minute, and they restart on the other host. Replication continues over the switched network until the cable is back.
- Arbitrated, pool master cut from the switch: if the cut lasts longer than the HA timeout, both hosts can reboot. VMs are down for about 10 minutes; no confirmed write is lost.