Senior Game Producer Tom Ellis published the technical breakdown after the environment had stabilized. The first important detail is that the Forever beta does not run on Blizzard’s normal production infrastructure. It uses a much smaller beta/PTR stack because WoW betas traditionally attract considerably less traffic than live launches. Forever apparently did not receive that memo. Developers described the number of players attempting to enter as extraordinary, and the small test environment suddenly had to behave like something much closer to a live launch.
The Servers Looked Fine While Players Were Disconnecting
The first problem was particularly annoying because Blizzard’s monitoring initially suggested that almost nothing was wrong.
Players could survive the queue only to disconnect while entering the game, yet CPU and memory usage across the relevant services looked perfectly healthy. The Battle.net team eventually traced the issue to a protective metering system used by Battle.net Game Services. Forever’s beta operated with one regional environment and only two connections feeding an enormous number of login attempts into it, which caused that metering system to react incorrectly even though the underlying machines were not overloaded.
Blizzard increased those connections from two to eight.
The disconnect problem disappeared almost immediately.
Ellis noted that the team had never encountered that particular failure without the usual obvious CPU warning signs. Twenty-two years of World of Warcraft infrastructure, and Azeroth can apparently still invent new ways to surprise the people running it.
Then Looting and Quests Started Crawling
Getting inside only revealed problem number two.
Players began experiencing long delays when looting enemies, accepting quests and performing other actions that depended heavily on the database. Blizzard’s Oracle database engineers discovered that the beta’s new tables and enormous volume of inserts had changed normal query patterns enough that database statistics were no longer keeping up.
The team manually analyzed the tables and created automated jobs to repeat the process regularly. According to Ellis, database performance recovered immediately afterward.
That particular problem is especially relevant because Forever’s beta deliberately concentrates everyone inside the first 20 levels. Rather than spreading testers across an entire level 60 world, Blizzard has packed enormous numbers of characters into the same early progression band, all accepting quests, looting enemies and hammering many of the same systems simultaneously.
Empty Maps Were Eating the Servers
Then came the problem that required the big restart.
The WORLD pools running Forever’s actual game simulation began consuming huge amounts of CPU and memory. Some virtual machines were approaching the point where the system’s Out-of-Memory protection could kill them, potentially threatening the underlying hypervisor.
The culprit was wonderfully mundane: empty maps were not shutting down correctly.
World instances that should have disappeared after everyone left were instead continuing to consume resources. As more maps accumulated, CPU and memory usage steadily climbed. Blizzard found the bug, pushed the fix through QA and restarted the environment so it could take effect.
The team also added additional WORLD servers and expanded the regional service while everything was offline.
This Is Exactly Why the Beta Exists
The launch was messy, particularly for anyone who spent an hour in a queue only to reach Azeroth and immediately return to the login screen. But there is a useful difference between finding these failures on September 17 and finding them on November 4, when World of Warcraft: Forever officially launches.
Forever is also attempting infrastructure changes beyond ordinary Classic servers. the launch version replaces traditional realm lists with larger ruleset-based ecosystems, meaning Blizzard needs confidence that the underlying systems can handle populations very differently from a conventional Classic realm structure.
The beta just handed the engineers several very real stress tests.
Battle.net metering broke.
The database struggled.
Empty maps attempted to consume the machines hosting them.
And Blizzard fixed all three while the word BETA was still written on the door.
There are worse ways for a test to earn its name.
Post a Comment