{"id":88840,"date":"2021-10-15T11:46:12","date_gmt":"2021-10-15T09:46:12","guid":{"rendered":"https:\/\/stevinho.justnetwork.eu\/?p=88840"},"modified":"2021-10-15T11:46:12","modified_gmt":"2021-10-15T09:46:12","slug":"diablo-2-resurrected-die-entwickler-aeussern-sich-zu-den-serverproblemen","status":"publish","type":"post","link":"https:\/\/stevinho.justnetwork.eu\/2021\/10\/15\/diablo-2-resurrected-die-entwickler-aeussern-sich-zu-den-serverproblemen\/","title":{"rendered":"Diablo 2 Resurrected: Die Entwickler \u00e4u\u00dfern sich zu den Serverproblemen"},"content":{"rendered":"
Bereits seit der Ver\u00f6ffentlichung von Diablo<\/span> 2 Resurrected<\/span> m\u00fcssen die Spieler dieses Remasters<\/span> mit einer Reihe von Serverproblemen zurechtkommen, die sich negativ auf ihre Spielerfahrung auswirken. Auch wenn das Ganze zu Beginn noch durch einen hohen Andrang erkl\u00e4rt werden konnte, so sind viele dieser Schwierigkeiten aber selbst jetzt noch immer im Spiel vorhanden. Um die Spieler nun \u00fcber die Hintergr\u00fcnde dieser Probleme und die Arbeit der Entwickler zu informieren, hat Community Manager PezRadar<\/span> vor Kurzem einen sehr interessanten Bluepost<\/span> in den Foren ver\u00f6ffentlicht.<\/p>\n Laut diesem Beitrag werden die anhaltenden Schwierigkeiten durch mehrere nacheinander auftauchende Probleme verursacht und die Entwickler l\u00f6sen jeden neu entdeckten Fehler relativ schnell. Das Ganze wird aber wohl dadurch erschwert, dass das Team m\u00f6glichst viel Code aus dem urspr\u00fcnglichen Spiel \u00fcbernommen hat und dieser alte Dienst nicht unbedingt mit dem Verhalten von modernen Spielern mithalten kann. Fr\u00fcher haben Spieler oft einfach nur eine Partie gestartet und gespielt, weil ihnen die Informationen zum Optimieren ihrer Erfahrung fehlten. Heute k\u00f6nnen Spieler aber viele verschiedene Anleitungen zum „korrekten“<\/span> Farmen von Gegenst\u00e4nden im Internet finden und das Spiel auf eine andere Weise angehen. Das moderne Verhalten der Spieler f\u00fchrt leider dazu, dass Partien deutlich schneller erstellt und wieder beendet werden, als es fr\u00fcher der Fall gewesen ist. Auch wenn das Team diesen Umstand im Vorfeld bedacht hat, so reichen die vorgenommenen Anpassungen aber leider nicht aus.<\/p>\n Zus\u00e4tzlich dazu greift das Spiel aktuell zu oft auf die einzige globale Datenbank zu und nutzt die regionalen Datenbanken nicht optimal aus. Dadurch entsteht ebenfalls unn\u00f6tige Last f\u00fcr die Server. Die Entwickler m\u00f6chten in den n\u00e4chsten Tagen daf\u00fcr sorgen, dass diese Kommunikation besser abl\u00e4uft und das System nur noch sehr selten mit der globalen Datenbank des Spiels in Kontakt treten muss.<\/p>\n Ansonsten hat das Team auch noch einige weitere notwendige Anpassungen geplant. Erst einmal wird es ein Limit daf\u00fcr geben, wie viele Partien die Spieler in einem kurzen Zeitfenster erstellen k\u00f6nnen. Ab einer bestimmten Anzahl wird ihnen dann eine Fehlermeldung \u00fcber Probleme mit der Kommunikation des Servers angezeigt und sie d\u00fcrfen erst einmal keine weiteren Partien \u00f6ffnen. Diese Einschr\u00e4nkung ist aber keine dauerhafte L\u00f6sung und sie erst einmal nur die Last der Spielserver reduzieren. Des Weiteren m\u00f6chte das Team viele Funktionen des Spiels aufbrechen und sie auf kleinere unabh\u00e4ngige Dienste aufteilen. Auf diese Weise sollte sich die Leistung verbessern und es gibt weniger gr\u00f6\u00dfere Abst\u00fcrze.<\/p>\n Die letzte Neuerung ist eine kommende Warteschlange f\u00fcr den Login in das Spiel. Aktuell gibt es einfach Zeiten, in denen eine gro\u00dfe Gruppe von Spielern gleichzeitig auf die Server springt. Das Team m\u00f6chte diese Situationen etwas entschlacken und die Server vor zu vielen gleichzeitig auftauchenden Personen sch\u00fctzen. Das Ganze soll \u00e4hnlich wie in World of Warcraft funktionieren und den Spielern bei zu viel Andrang eine Wartezeit aufzwingen. In solchen Moment kann das Team dann die Situation \u00fcberwachen, Limits manuell anpassen und das Spiel vor einem kompletten Absturz retten.<\/p>\n \u00dcber Blizzard Entertainment verteilt arbeiten aktuell auf jeden Fall viele Entwickler an dem Identifizieren und dem Beheben dieser Probleme. Das Team und die gesamte Firma nimmt diese Situation wohl sehr ernst und m\u00f6chte ihren Spielern unbedingt eine bessere Erfahrung bieten. Die Spieler m\u00fcssen sich aber leider etwas gedulden und noch einige Zeit mit m\u00f6glichen Problemen rechnen.<\/p>\n Since the launch of Diablo II: Resurrected, we have been experiencing multiple server issues, and we wanted to provide some transparency around what is causing these issues and the steps we have taken so far to address them. We also want to give you some insight into how we\u2019re moving forward.<\/span><\/p>\n tl;dr: Our server outages have not been caused by a singular issue; we are solving each problem as they arise, with both mitigating solves and longer-term architectural changes. A small number of players have experienced character progression loss\u2013moving forward, any loss due to a server crash should be limited to several minutes. This is not a complete solve to us, and we are continuing to work on this issue. Our team, with the help of others at Blizzard, are working to bring the game experience to a place that feels good for everyone.<\/em><\/span><\/p>\n We\u2019re going to get a little bit into the weeds here with some engineering specifics, but we hope that overall this helps you understand why these outages have been occurring and what we\u2019ve been doing to address each instance, as well as how we\u2019re investigating the overall root cause. Let\u2019s start at the beginning.<\/span><\/p>\n The problem(s) with the servers:<\/strong><\/span><\/p>\n Before we talk about the problems, we\u2019ll briefly give you some context as to how our server databases work. First, there\u2019s our global database, which exists as the single source of truth for all your character information and progress. As you can imagine, that\u2019s a big task for one database, and wouldn\u2019t cope on its own. So to alleviate load and latency on our global database, each region\u2013NA, EU, and Asia\u2013has individual databases that also store your character\u2019s information and progress, and your region\u2019s database will periodically write to the global one. Most of your in-game actions are performed against this regional database because it\u2019s faster, and your character is \u201clocked\u201d there to maintain the individual character record integrity. The global database also has a back-up in case the main fails.<\/span><\/p>\n With that in mind, to explain what\u2019s been going on, we\u2019ll be focusing on the downtimes experienced between Saturday October 9 to now.<\/span><\/p>\n On Saturday morning Pacific time, we suffered a global outage due to a sudden, significant surge in traffic. This was a new threshold that our servers had not experienced at all, not even at launch. This was exacerbated by an update we had rolled out the previous day intended to enhance performance around game creation\u2013these two factors combined overloaded our global database, causing it to time out. We decided to roll back that Friday update we\u2019d previously deployed, hoping that would ease the load on the servers leading into Sunday while also giving us the space to investigate deeper into the root cause.<\/span><\/p>\n On Sunday, though, it became clear what we\u2019d done on Saturday wasn\u2019t enough\u2013we saw an even higher increase in traffic, causing us to hit another outage. Our game servers were observing the disconnect from the database and immediately attempted to reconnect, repeatedly, which meant the database never had time to catch up on the work we had completed because it was too busy handling a continuous stream of connection attempts by game servers. During this time, we also saw we could make configuration improvements to our database event logging, which is necessary to restore a healthy state in case of database failure, so we completed those, and undertook further root cause analysis.<\/span><\/p>\n The double-edged sword of Sunday\u2019s outage was that because of what we\u2019d dealt with on Saturday, we had created what was essentially a playbook on how to recover from it quickly. Which was good.<\/span><\/p>\n But because we came online again so quickly in a peak window of player activity, with hundreds of thousands of games within tens of minutes, we fell over again. Which was bad.<\/span><\/p>\n So we had many fixes to deploy, including configuration and code improvements, which we deployed onto the backup global database. This leads us into Monday, October 11, when we made the switch between the global databases. This led to another outage, when our backup database was erroneously continuing to run its backup process, meaning that it spent most of its time trying to copy from the other database when it should\u2019ve been servicing requests from servers. During this time, we discovered further issues, and we made further improvements\u2013we found a since-deprecated-but-taxing query we could eliminate entirely from the database, we optimized eligibility checks for players when they join a game, further alleviating the load, and we have further performance improvements in testing as we speak. We also believe we fixed the database-reconnect storms we were seeing, because we didn\u2019t see it occur on Tuesday.<\/span><\/p>\n Then Tuesday, we hit another concurrent player high, with a few hundreds of thousands of players in one region alone. This made us hit another incident of degraded database performance, the cause of which is currently being worked on by our database engineers. We also reached out to other engineers around Blizzard to work on smaller fixes as our own team focused on core server issues, and we reached out to our third-party partners for assistance as well.<\/span><\/p>\n Why this is happening:<\/strong><\/span><\/p>\n In staying true to the original game, we kept a lot of legacy code. However, one legacy service in particular is struggling to keep up with modern player behavior.<\/span><\/p>\n This service, with some upgrades from the original, handles critical pieces of game functionality, namely game creation\/joining, updating\/reading\/filtering game lists, verifying game server health, and reading characters from the database to ensure your character can participate in whatever it is you\u2019re filtering for. Importantly, this service is a singleton, which means we can only run one instance of it in order to ensure all players are seeing the most up-to-date and correct game list at all times. We did optimize this service in many ways to conform to more modern technology, but as we previously mentioned, a lot of our issues stem from game creation.<\/span><\/p>\n We mention \u201cmodern player behavior\u201d because it\u2019s an interesting point to think about. In 2001, there wasn\u2019t nearly as much content on the internet around how to play Diablo II \u201ccorrectly\u201d (Baal runs for XP, Pindleskin\/Ancient Sewers\/etc for magic find, etc). Today, however, a new player can look up any number of amazing content creators who can teach them how to play the game in different ways, many of them including lots of database load in the form of creating, loading, and destroying games in quick succession. Though we did foresee this\u2013with players making fresh characters on fresh servers, working hard to get their magic-finding items\u2013we vastly underestimated the scope we derived from beta testing.<\/span><\/p>\n Additionally, overall, we were saving too often to the global database: There is no need to do this as often as we were. We should really be saving you to the regional database, and only saving you to the global database when we need to unlock you\u2013this is one of the mitigations we have put in place. Right now we are writing code to change how we do this entirely, so we will almost never<\/span> be saving to the global database, which will significantly reduce the load on that server, but that is an architecture redesign which will take some time to build, test, then implement.<\/span><\/p>\n A note about progress loss:<\/strong><\/span><\/p>\n The progress loss some players have experienced is due to the way we do character locks both in the regional and global databases\u2013we lock your character in the global database when you are assigned to a region (for example, when you play in the US region, your character is locked to the US region, and most actions are resolved in the US region\u2019s database.)<\/span><\/p>\n The problem was that during a server outage, when the database was falling over, a number of characters were becoming stuck in the regional database, and we had no way of moving them over to the global database. At that time, we believed we had two options: we either unlock everyone with unsaved changes in the global database, therefore losing some progress due to an overwrite that would occur in the global database, or we bring the game down entirely for an indeterminate amount of time and run a script to write the regional data to the global database.<\/span><\/p>\n At the time, we acted on the former: we felt it was more important to keep the game up so people could play, rather than take the game down for a long period of time to restore the data. We are deeply sorry to any players who lost important progress or valuable items. As players ourselves, we know the sting of a rollback, and feel it deeply.<\/span><\/p>\n Moving forward, we believe we have a way to restore characters that doesn\u2019t lead to any significant data loss\u2013it should be limited to several minutes of loss, if any, in the event of a server crash.<\/span><\/p>\n This is better, but still not good enough in our eyes.<\/span><\/p>\n What we are doing about it:<\/strong><\/span><\/p>\n Rate limiting:<\/strong><\/em> We are limiting the number of operations to the database around creating and joining games, and we know this is being felt by a lot of you. For example, for those of you doing Pindleskin runs, you\u2019ll be in and out of a game and creating a new one within 20 seconds. In this case, you will be rate limited at a point. When this occurs, the error message will say there is an issue communicating with game servers: this is not an indicator that game servers are down in this particular instance, it just means you have been rate limited to reduce load temporarily on the database, in the interest of keeping the game running. We can assure you this is just mitigation for now\u2013we do not see this as a long-term fix.<\/span><\/p>\n Login Queue Creation:<\/strong><\/em> This past weekend was a series of problems, not the same problem over and over again. Due to a revitalized playerbase, the addition of multiple platforms, and other problems associated with scaling, we may continue to run into small problems. To diagnose and address them swiftly, we need to make sure the \u201cherding\u201d\u2013large numbers of players logging in simultaneously\u2013stops. To address this, we have people working on a login queue, much like you may have experienced in World of Warcraft. This will keep the population at the safe level we have at the time, so we can monitor where the system is straining and address it before it brings the game down completely. Each time we fix a strain, we\u2019ll be able to increase the population caps. This login queue has already been partially implemented on the backend (right now, it looks like a failed authentication in the client) and should be fully deployed in the coming days on PC, with console to follow after.<\/span><\/p>\n Breaking out critical pieces of functionality into smaller services:<\/strong><\/em> This work is both partially in progress for things we can tackle in less than a day (some have been completed already this week) and also planned for larger projects, like new microservices (for example, a GameList service that is only responsible for providing the game list to players). Once critical functionality has been broken down, we can look into scaling up our game management services, which will reduce the amount of load.<\/span><\/p>\n We have people working incredibly hard to manage incidents in real-time, diagnosing issues, and implementing fixes\u2013not just on the D2R team, but across Blizzard. This game means so much to all of us. A lot of us on the team are lifelong D2 players\u2013we played during its initial launch back in 2000, some are part of the modding community, and so on. We can assure you that we will keep working until the game experience feels good to us not only as developers, but as players and members of the community ourselves.<\/span><\/p>\n Please continue to submit your feedback to the Diablo II: Resurrected forum 347<\/span><\/a>, report your bugs to our Bug Report forum 102<\/span><\/a>, and for troubleshooting assistance, visit our Technical Support forum 54<\/span><\/a>. Thank you for your ongoing communication with us across all channels\u2013it\u2019s invaluable to us as we work on these issues.<\/span><\/p>\n The Diablo community team will keep you updated on our progress via the forums.<\/span><\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":" Bereits seit der Ver\u00f6ffentlichung von Diablo 2 Resurrected m\u00fcssen die Spieler dieses Remasters mit einer Reihe von Serverproblemen zurechtkommen, die sich negativ auf ihre Spielerfahrung auswirken. Auch wenn das Ganze zu Beginn noch durch einen hohen Andrang erkl\u00e4rt werden konnte, so sind viele dieser Schwierigkeiten aber selbst jetzt noch immer im Spiel vorhanden. Um die […]<\/p>\n","protected":false},"author":10647,"featured_media":84118,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[4],"tags":[1112,5315,12268,4274,11185],"class_list":["post-88840","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-gaming","tag-blizzard","tag-diablo-2","tag-diablo-2-resurrected","tag-entwickler","tag-resurrected"],"jetpack_publicize_connections":[],"yoast_head":"\n