Failure behavior

TwinStor is designed to run unattended for years at sites with no IT staff. The default posture: keep serving through failures, recover automatically once it is safe, and when safety cannot be proven, fail closed and ask. Every scenario below was exercised on real hardware during the Tech Preview campaign: power cables pulled, a real power strip cut, disks removed live.

What heals itself

The most common case, a host going dark, plays out like this:

A power loss, start to finish · no human involvedhealthyfailedresyncing

Failure	What happens
Host power loss	The survivor keeps serving without interruption; the dead host is fenced and its VMs restart automatically on the survivor.
Total site power outage	Both hosts boot, storage converges, VMs restart; measured at about 3 minutes from power cut to all VMs running, with zero human intervention (BIOS power-on policy required, see Installation). After a simultaneous blackout, a host can occasionally boot disabled because its toolstack raced the storage convergence; TwinStor re-enables it within about 5 minutes and restarts the protected VMs (manual fallback: xe host-enable).
Local disk or RAID array failure	VMs keep running, transparently served from the other host's copy, with no path change; an alert is raised, and the disk resyncs and returns to full redundancy on its own when it comes back or is replaced. A failed disk is a "fix it whenever" event, not a 2 a.m. outage. While one side rebuilds, the healthy node is the only copy: prefer running VMs there until the resync completes, and treat the daily re-raised degraded alert as the reminder that a second fault would mean an outage.
Network cut between hosts	The isolated node steps aside (freezes and self-fences), the survivor continues; automatic recovery and resync on rejoin. The decision is functional: a node that can still reach the gateway keeps serving, because its clients can still reach it; a node that cannot steps aside.
Brief network blips	Congestion, packet loss or a switch reconvergence shorter than the detection threshold is absorbed entirely: no disconnect, no resync. An actual carrier drop (cable pull, port bounce) that heals within seconds causes a short I/O stall while the pair reconnects, then writes resume with no divergence.
TwinStor service crash	systemd restarts the service while storage keeps serving from the kernel; zero VM downtime.
Host reboot, graceful or not	The peer was already serving; the rebooted node reconnects and resyncs its backlog automatically.
Both hosts rebooting, staggered	Each node waits to reconnect with its peer before serving, so they come back as a synced pair rather than two independent copies.
Split-brain (the two copies diverged)	Should not happen in normal operation: the fencing above makes the losing side of a partition freeze before it can write, so divergence never accumulates. For the rare windows fencing cannot cover (a link that heals faster than the fence decision, a host-local fault that blocks only replication), TwinStor resolves the divergence safely rather than silently: the copy backed by pool evidence wins, the loser discards its seconds of diverged history and resyncs, any VM affected by the discard is restarted from the authoritative copy, and a loud priority-1 alert names what happened. If the winner is undecidable, TwinStor refuses to discard anything and waits for you.

The only scenario that cannot self-heal is destroyed hardware: both disks dead at the same time means the data no longer exists anywhere. That is what your backups are for.

When TwinStor waits for you

Almost everything heals itself. The deliberate exceptions are states where acting automatically would mean guessing with your data or overriding your intent. There, TwinStor holds safely, tells you loudly (an alert plus twinstor status), and waits. In every case, doing nothing is safe; the hold persists until you decide.

State	How you will know	Your decision
Split-brain, winner undecidable	twinstor_split_brain alert saying an operator decision is needed	Which copy is authoritative. Both copies stay intact until you act; the alert and status name the exact command to run on the node whose copy you discard.
Both nodes yielded to each other	Both nodes alert that they are holding down; neither serves	Which node survives. The alert text carries the procedure.
Dual-master where both hosts run VMs	twinstor_ha_wedge alert	Which host's toolstack yields; escalation steps are in the alert text.
Automatic failback latched off after repeated link flaps (arbitrated)	failback_latched in status; the fallback alert names it	Fix the cable or NIC first, then run twinstor failback-replication.
Pool HA disabled by an operator	twinstor_ha_disarmed alert after 30 minutes	Whether to re-arm (see the maintenance procedures in Administration).
Node frozen, own network healthy, peer long gone	A "holding this node FROZEN" alert	Is the peer truly dead, or just partitioned? If confirmed dead, the alert gives the commands to serve solo; if it might be alive, fix its network and the pair heals on reconnect.
Pool master isolated from the client network (arbitrated: its switch path died, the direct link is alive)	master_isolated alert, then isolated_standdown in status	Usually nothing: after about 10 minutes of confirmed isolation the island stands its storage down so its VMs restart on the reachable peer, then rejoins on its own once its network is stably back. Just reconnect the cable when convenient.
Hardware gone (dead disk with peer unavailable, double disk failure, dead host)	Urgent alerts naming the failure	Physical replacement; see the replacement procedures in Troubleshooting and feedback.

This table is the complete list. A held state that is not in this table and is not healing is, by definition, a bug: please report it with a support bundle from both nodes.

Known trade-offs by topology

• Fate-shared mode (single switch): the switch is a single point of failure. If it dies for longer than the HA timeout, both hosts self-fence into a clean full-pool outage, with intact data and automatic recovery once the switch returns. For planned switch work, follow the switch maintenance procedure in Administration to turn that into a simple freeze-and-resume.
• Arbitrated mode (redundant network with direct link): a cable or NIC failure on the direct link fails replication over to the switched network automatically and loudly, with no interruption. A sustained failure of the whole switched network is still a full-pool outage for compute (the pool heartbeat rides it), but never a corruption.