The database cluster and all web nodes have been stable for over an hour. The web nodes have since been moved to another host without interruption. The incident is resolved.

The database cluster has been complete again since about 00:20 CET (3/3 in sync), shared storage shows no differences, all projects are running again. We will keep watching for a while.

Cause: the host under the three web nodes restarted, all three database nodes were stopped hard at the same time and did not rejoin on their own. All nodes had the same data, nothing was lost. We are rejoining the cluster now.

The host under the web nodes restarted unexpectedly at about 00:08 CET. The web nodes are coming back up; the database cluster (MariaDB/Galera) has to be rejoined. Until then, projects with a database may show errors. We are working on it.