History

The cause was not an external attack but a huge amount of primed TNT on the affected game server, which overloaded the port. We removed the TNT; the server runs normally again and is reachable for all players. This incident is resolved.

Node 3 has been quiet for more than two hours: no further attacks, normal load, and connections and voice chat work directly. The protection measures stay in place. This incident is resolved.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

This issue has been fixed. Backups no longer stop after a short wait with "host server unreachable". Larger backups run in the background and show as pending until they finish. If a backup failed during this time, please simply create it again.

Resolved. Inbound UDP is reaching Node 3 directly again, and our external tests are succeeding. If voice chat still does not work for you, please contact support.

Resolved. The discount tier request in the reseller portal's admin area works again.

Resolved. Team accounts without a second factor now see the setup notice in the admin area instead of a loading loop.

Monthly invoice PDF downloads have been working again since the fix. This incident is resolved.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The database cluster and all web nodes have been stable for over an hour. The web nodes have since been moved to another host without interruption. The incident is resolved.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

All CitexNode Deploy services have been stable since the restore. The incident is resolved.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

Resolved: vServers with Intel Xeon have been orderable and provisioned normally since the fix. Thanks for your patience!

The automated check is passing again. The component is reachable.

Custom vServers are fully available again. Ordering new vServers and managing existing servers through the dashboard are working normally again. The certificate change on our virtualization hosts has been completed successfully.

We continue to monitor the platform to ensure all services remain stable.

The automated check is passing again. The component is reachable.

All services have been stable since 01:15 UTC. The backup is now permanently throttled so it no longer slows down normal operation.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

There have been no further stalls since the brief storage hang at 21:16–21:19 CEST, and all systems are running normally. This incident is resolved. If your server is still stopped since then, please restart it from the dashboard.

The automated check is passing again. The component is reachable.

The TeamSpeak servers in Cologne have been running stably since the provider finished its maintenance. This incident is resolved.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

Resolved. The host was unreachable from about 18:20 to 19:03 CEST after a faulty change to its network configuration (the previous update mistakenly said 17:10). The fix is now permanent and has survived a reboot of the host. All services are running normally again. Sorry for the disruption.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

Resolved: ns1.citexnode.de and ns2.citexnode.de have been running stably for almost two hours and are now also monitored on this status page. Domains can be switched to our nameservers again.

Resolved: GitHub has been running normally again for over an hour, webhooks are delivered without delay and deploys from GitHub repositories start as usual. If a push during the disruption did not trigger a deploy, it can be started manually from the dashboard.

Resolved: all affected services have been running stably for over an hour on the patched PHP versions – the Deploy project on PHP 8.5.12, the mail server on PHP 8.2.35 and the Looking Glass on PHP 8.4.27. Webmail and the mail administration are reachable again. Thank you for your patience.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

The automated check is passing again. The component is reachable.

Resolved. The BuildKit cache is capped at 8 GB on every build host, web-04 has joined as a fourth build host, and deployd 0.4.3 now waits up to 5 minutes for nodes to pull the image during rollout instead of 90 seconds. A follow-up bug where every deploy of a project was rolled back immediately after one failed rollout is fixed as well. New builds complete again; please restart any failed builds once.

Resolved. Websites with a database or persistent storage only run on the servers next to the database, and PHP websites serve images, stylesheets and scripts with cache headers. Load times have been normal since the move. If you run a PHP website, the cache applies from your next deploy; clicking "Deploy now" is enough.

Resolved. All cron jobs have run without errors for over an hour. New is the "Command" type: cron scripts run directly in the container as on classic hosting (e.g. php cron/task.php), with no token in the call. Exit code and output are shown under Cron jobs → History.

Resolved. We checked the whole flow with a test payment at Mollie: the payment is reported and assigned correctly. There were no paid Mollie top-ups that went uncredited, so nobody lost any balance.

Resolved. Sign-up and password reset now consistently require and state at least 10 characters.

Resolved. The confirmation email can be requested again after signing up, in the dashboard and when ordering. If you could not confirm your address so far, just request a new email.

The fix is live. All links in our emails lead to the right page again, including in emails you already received. If you could not confirm your email address, just open the link from the email again. If it has expired, you can request a new confirmation email when you sign in.

Resolved. All three database nodes are in sync again; deploys and databases work normally. Duration: about 3 ½ hours.

Deploy now has its own row on this page and is checked every minute, so an outage like this shows up here right away.