Network disruption, TS3 outage

[Bent]

Time of incident: 09.01.2019, ~01:06 - 02:12

Due to a major issue with our internet service provider our Teamspeak 3 server has been down for about an hour.
We have taken counteraction and migrated our TS3 server onto our core infrastructure at Hetzner Germany, back into a datacenter.

We apologize for the downtime. The action we have taken will prevent future issues.

No other services have been affected by this outage.

---

Heatmap of Germany showing current hotspots with outages. (Source: Allestörungen)

[Bent]

Update 20190109 17:20 CET

Due to another mistake on our side the TS3 server has again been unreachable for about 11.5 hours. From 05:12 - 15:38 (09.01.2019). We sincerely apologize for that fail on our side and have fixed the issue causing this mishap.

Technically interested?

The technical reason was us running two separate instances of our TS3 server simultaneously. After the ISP went down (Vodafone) we were forced to migrate our TS3 server onto our core infrastructure at Hetzner Germany (uploading the files via slow 4G). The migration went well and we were back up running in less than an hour. As clients use a domain name to connect to our TS3 server we kept the ‘old’ TS3 server running at the old location/ISP with a message pointing towards the correct IP (just in case someones DNS (servers) cached our domain too long (ts.unitedworldminers.com)).

This is where we failed: The ‘old’ Teamspeak server still used the same license as the Teamspeak server at the new location. Our license allows us to run one single instance of Teamspeak on one IP, not two. Teamspeak periodically checks if there are servers breaking with their licenses and stops all additional instances to avoid multi-using one license (which is completely legitimate of them to do!).

This is what happens if your ISP goes down in the middle of the night and you start doing some quick and dirty workarounds, forgetting to remove your license from the old TS3 instance…