Resolved. Node 2 has been fully connected to our control plane again since 10 September, 15:51: start, stop, console, file manager, backups and the usage display all work. Your servers and bots kept running throughout.

What happens next: the connection still takes a detour through Node 3 in the same data centre. Nothing changes for you and encryption stays end to end.

The actual cause, a filter inside the data centre provider's network between our control server and Node 2, is reported there as a ticket and not yet fixed. We track the removal of the detour as a separate incident so nothing stays open here that is already working.

Node 2 has been connected to our control plane again since 15:51. Start, stop, console, file manager and the usage display work again for servers on Node 2. Your servers and bots were running the whole time.

How: for now the connection takes a detour through Node 3, which sits in the same data centre. Encryption stays end to end, nothing changes for you.

What is still open: the actual cause lies with the data centre provider and is reported there as a ticket. Once the direct path is clear again we remove the detour and close this incident.

Node 2 itself is running and never went down. Your servers and bots on Node 2 are up and reachable. What is missing is the link between our control plane (citexnode.de) and Node 2: packets coming from citexnode.de are dropped inside the data centre's network just before the machine, while the reverse direction works. We have ruled out the machine itself, our side and AWS. The fault sits with the data centre provider, and a ticket is open with them.

What this means for you: servers on Node 2 keep running, but start, stop, console and the file manager in the dashboard do not work for them right now. The usage display also stands still.

Restart at 15:05: a reset of the machine did not bring the link back, but it briefly stopped five servers. We started them again by hand at 15:36, all are running.

Node 2 · Eygelshoven has been offline since 12:01 yesterday.

We are investigating and working to bring the node back online as quickly as possible. As things stand we aim to have it back today by 20:00 at the latest. Until then, thank you for your patience.