Gaarawarr Gabs with Bruce MacLean
This time, we’re discussing Live Production with Bruce MacLean, WAR Live Producer. Bruce is a very busy guy (obviously, considering just how diverse Live Production is) so we didn’t get a lot of time to chat with him. However, we did get a lot of good information about how Live Production works at Mythic, how they react to bugs with new patches and, amazingly enough, some fairly open discussion about how they treat exploits in WAR.
GG: So, Live Production…gah, see, we were going over what we wanted to talk about this morning and we were like “This is just such a huge thing…” There are a lot of things that go into Live Production.
Bruce: There really are.
GG: So, for people that aren’t well-versed in this, can you give us an overview of everything that’s going into the Live Production of WAR, because a lot of people are like “Live Production…wha?!?”
Bruce: “What is that?”
GG: “They’re just pushing the game out…”
GG: But it’s not as simple as that.
Bruce: Right, really it’s…
Mark Davis: No, it is [that simple]. That’s exactly how it is.
GG: Yeah, and all of you work on server stablity…
Mark: You can make it sound complicated, but it’s not.
All laughing… ~end sarcasm, queue comedic interlude~
Bruce: You know…it’s really hard work…I work really hard!
GG: And everybody should bring you taffy!
~end comedic interlude, back to interview~
Bruce: So, Live Production is really just the coordination of all the individual production that’s going in. It’s also a lot of emergency response.
Mark: Oh yeah.
GG: No way!?!?! To what?? People don’t talk bad about the live game, do they?!?
Bruce: All the hot fixes, all the stuff that’s “Oh crap! We need to fix this right now!”, that all comes through my door a far as categorizing, triage. It’s tough because an issue will crop up and you have to make the assessment. “Is that something we need to drop what we’re doing [to address]?” because we’re working on all kinds of cool stuff, “Do we need to drop what we’re doing and fix this right now or can this wait?” That’s a constant call that Live Production has to make.
Phillip Chan: He secretly hates us because all of the critical issues are from items…
GG: Bastards! But it’s not all of them…
Bruce: No, there are plenty of areas the game can improve on.
GG: I mean, you know, the Auction House ate 300 gold from me. (This was back in July. It hasn’t done that to me since, hehe.)
Bruce: Yeah…that Auction House has issues.
Andy: That is actually true. That is not an item issue.
Bruce: That is not an item issue. You’re right.
GG: So, from the time that…since we’re talking about how these things pop up…obviously everything is going to vary because some things are easier to fix than others. In general, something happens in Live, everybody on the forums freaks out…what’s the turnaround from the time where you hear it and, say it’s one of those things where it’s “Drop everything to do this!” What’s the turnaround from saying that, doing your new build, and getting that into Live? Because a lot of people have unrealistic expectations.
Bruce: That’s a good question. So, the first step is the Rock Stars, our Community Team…
GG: Nice look on your face Andy…
Bruce: They’re constantly watching the forum posts and the blogs…by the way, congrats on 100,000 views.
GG: Thank you, but Mythic doesn’t read forums. We know this, don’t lie to us… 😉
Bruce: So they’re usually the ones that spot it first. Of course, we’re playing the game as well. So if he’s playing his Engineer, he’ll send up an e-mail “Hey, I just spotted this in-game” but a lot of times it’s the Community Team saying “Hey, there’s something going on. We need to look into it.” From there, we have a team (Quality Assurance) that actually confirms that this is a true issue and their turnaround is really fast. We have 24-hour support on this if we need it. From there, the speed really depends on the nature of the issue.
Some issues require us to write new server code and that means, to push it up, we need to bring all servers down. So, that slows us down right there because we’re not going to bring the servers down in the middle of Prime Time unless the Auction House is eating everybody’s gold and not just yours. Sorry. ~laughing~
GG: It’s ok. It only took three weeks for me to get it back…
Bruce: Umm…glad you got it back…
Bruce: Anyway…whereas some other fixes require us to literally just change a bit of data on the server and we can turn around testing of that and get that fix up on the server within an hour. So, there’s a range from an hour to a day, but all the people doing that are the Production Teams that are working on things like…you know, we would like to add new crafting systems into the game, but every time we’re fixing something itemization-related, that’s some of their time we’re eating into. So, we kind of have to measure our response [time] and it’s always a balancing act.
GG: It’s like the bus that leaves on time but then is a minute late at each stop. So, by the time it gets to its last stop, it’s two hours late.
Bruce: Exactly. So, it’s a constant balancing act. The answer to your question is that it totally depends on the nature of the issue, how severe it is, and some issues sound like they’d be really easy to fix but they involve changes to very complex code systems…
Mark: It’s like Jenga.
Bruce: Yes, just like Jenga.
GG: Except you’re not just pulling stuff out…it’s harder to push the stuff back in.
Bruce: Right and it seems like an easy fix, but in order to make this change, before throwing it up to all the Live Servers, we need to thoroughly test the entire system involved to make sure we didn’t break something because that’s a risk when you’re changing code. You have to make sure you didn’t introduce new bugs. So, it depends on the nature of the system it touches. And then some things sound like “Wow! That’s a major bug!” but it’s just a simple, quick fix. So, the answer is: it depends.
Andy: So, remember the Invader/Warlord extravaganza we had on Live Servers?
(Again, this was back in July.)
GG: You mean the one I totally missed? 😦
Andy: That was actually pretty easy. It was a one-line fix. It went up right away and players immediately noticed the difference.
Bruce: Yeah, I missed out too. However, I was able to get on the Auction House right afterwards…but then there are bugs that only happen on one server and it’s like…
GG: Yeah, Iron Rock Flight Masters go on breaks…
Bruce: All of our servers have the same code and the same data…theoretically.
GG: Well, our Flight Master and our Healer like to go on walks out into the RvR lake in Chaos Wastes…
Bruce: It’s a feature…an Iron Rock feature…
Mark: I think, actually, with Iron Rock…I think it was Iron Rock that had Keg End mobs that would not go away.
Mark: I mean, how do we reproduce that? We look on our internal server, Sigmar, they’re not there. We look on internal test, our transitional server, they’re not there.
Phillip: And on our Live mirrors…
Mark: They’re nowhere, but for some reason, on that one server, they kept showing up. We’d turn them off “OK, they’re gone. Great! We fixed it.” A week later, they’re back. “What is it with these drunken Gnoblars…?!”
Bruce: Iron Rock has got a little touch of Chaos, that’s all.
GG: A little!?!?!?
GG: With those issues, so many times we see players just go to the forums and say “Hey! It’s broken.” or people say they put in a ticket and literally they write “It’s broke. Fix it or I’m taking my money somewhere else.”
GG: What are the best ways that players can report issues? Especially when it’s situations where it only happens on a single server or when it’s something that you can’t reproduce.
Bruce: So, one thing we do is, we’re greedy for information Live Production-wise and we’re constantly pulling information from various sources. So, really, there are multiple ways to report it. If you post it on the forums, somebody’s going to see it. If you put in a bug report, we’re constantly pulling from our QA Team whose entire job it is to sift through the bug reports every day and pull out everything that’s critical and, usually, when a serious bug starts popping in the bug report queue, we’ll get word of it via e-mail within an hour or two. Then we can immediately start triage on it, working on producing a solution for it, testing it, then deploying it.
GG: So that leads me to this question…the times I’ve used, well, I tend to go to the forums and post stuff there, but the reason I do that when I’d much rather do it in-game is that the bug report tool has a very small character allotment. So, is there a way, if you guys really want information since we’d love to give it to you sometimes and I’m typing and it stops me and I’m like…sigh.
Bruce: You are a rarity; you are a writer. You’re a writer, you have a blog, come on! Most of them say “Quest no give.” That’s literally what’s in the report. So…
GG: Or, or, “That Empire guy’s not doing the thing he’s supposed to do”…which one?
Andy: There’s a lot on our to-do list, like server stability, fixing Tier 4, Crowd Control…those kind of things are high on our priority list. Making the Report box bigger is not that high up there.
GG: I thought the first thing on the list was to nerf Shadow Warriors and make Bright Wizards better?
Andy: Well, that’s an unspoken. That’s an unspoken always.
The gist of the conversation was that if you have a lot of good information to share about a bug that could help in fixing it, submit it in the Bug Report forums at the Official Forums and it will be read and appreciated. Maybe, one day, we’ll get a bigger in-game reporting tool, but we’ll have to make do until then. 🙂
GG: So, the mythic rumor…not Mythic as in the company…this is why this is broke, because somebody is slamming Print Screen while taking their helmet off and jumping around and spinning 360 degrees, and this is the reason why you lost in combat during a Fortress siege. On the forums, you see a lot of these issues where, like you said, they cannot be reproduced for some reason. They seem to be an anomaly.
Bruce: Actually, for things like that, one of our best assets is that we all play the game, and we all have Rank 40s, and we’re all out there in RvR, and we read the forums, and we see the YouTube videos of people slamming the Print Screen button, so…
GG: That works great, by the way, when you get stuck. You get stuck on something, just hold the forward button down and jam Print Screen over and over while spamming jump. All of a sudden, you’re flying out of whatever you were stuck in. We’re totally going off on a tangent here, sorry.
Bruce: So, issues like that, especially around exploits…one thing we don’t do, as a rule, we don’t message out when we fix exploits. We just fix them. We take exploits very seriously. Not only as people running the game, but as players of the game. There’s nothing worse than knowing, or even thinking, that somebody on the opposing team is cheating to win. So, we have engineers whose job it is to beat those people.
GG: And we’re not talking Dwarfen Engineers…
Bruce: So, you see us constantly throwing up hot fixes, you see us doing our big, major patches…one thing we always try to do is to message every change we make in the game with one exception and that exception is: when we fix an exploit. Because we don’t want to say “Hey look! There was this big, gaping hole in our game that this player was taking advantage of.” We don’t want to give exploiters clues for their exploits essentially. So, as a rule, we don’t message that, but that doesn’t mean we’re not constantly working on that.
GG: So, those are the epic “Ninja Patches”…
Bruce: I don’t know what you’re talking about…I can neither confirm nor deny that…
Phillip: I must refer you to my Public Affairs person…
Andy: One thing to remember is this: To acknowledge the existence of an exploit is to legitimize it. We don’t want to give power to the people who want to ruin the game experience for other players. If that’s their goal in life and that’s their way of enjoying themselves, then, honestly, they can take that somewhere else because we’re going to find them. Trust me. We find those exploiters. We get PM’d information, we go to YouTube, we hit all that stuff very hard like Bruce said.
And we see it in-game and when we see that stuff out there, we squash it as quickly as we can. Sometimes it is a really quick and easy fix, and sometimes it’s one that we’re scratching our head over. That’s not because we’re not good at what we do, it’s because there are some really smart people out there.
Bruce: I was talking earlier about triage and how we triage emergencies. Exploits are considered on-par with crashing out of the game. That is an emergency in our eyes. Exploits get the highest severity ranking possible. Our entire game is built around fighting other players…
GG: So, basically, when it comes to exploits, if it’s continuing to happen, it’s not that you’re ignoring it. It’s that, if it was easy to be fixed, it would have been fixed and nobody would see it anymore.
Andy: A perfect example of that, and I’m going to kind of go against our policy here, because this was a very important thing, was the recent happenings on the Dark Crag server with the Caledor and the Dragonwake zones flipping back and forth so quickly. That was a big issue. That was also an exploit. An exploit, by definition, is manipulation of the game mechanics outside of their intent; to falsely or artificially advance the speed at which you are advancing as a player.
Bruce: In that case, it was a cross-Realming exploit.
Andy: And it was a large one. In that case, we didn’t action the whole server because we can’t really action the whole server on it. We were working on that one pretty much night and day. That was not one of those “Oh, OK, we’ll take care of it Monday morning guys” [things]. That was a ‘people were called in to the office, people were e-mailing back and forth at one in the morning’ [thing]. This was something we were on immediately. It just happened to take a day or so to fix. So, some things go quickly, some things take a little bit longer like that.
GG: So, I mean, we could probably almost guess this, but when it comes to Live Production and making sure the experience is what you want it to be, what is the biggest hurdle just to make sure that you’re doing and staying on top of every day to keep things going and not snagging up?
Bruce: So, apart from emergency response, the other big side of Live Production is the coordination of the major versions and getting all the pieces into the major version, and making sure they’re tested appropriately and that the entire version, as a whole, is ready to go live. Sometimes, we’ve pushed versions live and there have been some pretty bad bugs, so we take smoking out the versions and having the features ready to go very seriously.
That being said, what we’re targeting in our versions now are the issues that we feel are the critical issues for players in our game — the issues that people are posting about, the issues that we’re seeing bug reports about, and the issues that we’re experiencing as players ourselves. That’s the Tier 4 end-game, that’s Fortresses, the frickin’ supply-line debuff which is my personal unhappy guy…yeah…the supply-line debuff…we don’t like the supply-line debuff and we’re working on a solution for that…a solution for the Fortresses altogether in fact. The supply-line debuf is better than everybody rushing to the Fortress and the Fortress crashing, but we’re working on a better long-term solution for that.
Server performance, client performance, AoE, AoE gank groups, Crowd Control, being chain-CC’d, that’s no fun. Those are all things that are on the top of our radar because those are at the top of the players’ radar. Now that we’re a live game, really what drives us is our player-base. We care about what the people that are playing our game care about. So, if it’s something that people feel strongly about enough to post, then we’re all ears. The City Sieges, the entire experience of City Invasion, from top to bottom, all that is up there in our Top Five things to tackle in the next few patches and Stuart Zissu, who I believe you’re going to talk to today, he’s the one you want to hit with questions about the second Keep ramps, the Fortresses, the City changes, open RvR incentives. What they’ve got in the pipe is very exciting and we’re very excited about it.
GG: Are you looking towards making Fortress Sieges back to where they were when the game launched? To where there was no limit to the number of people that could participate if the server could handle it?
Bruce: I’ll let Stuart handle that question. We’re still working on the solution to Fortresses. It may be not having them be a critical part of the campaign where, from your Keeps, you are able to capture the pairing and go straight into the capital city. That was hinted at in the Producer’s Letter.
GG: Ok, one last question: Who do you think would look better as a Witch Elf: me or Mykiel?
Bruce: Oh gosh…ummm…uhhh…
Mark: You can’t answer that question.
GG: Both answers are bad?
GG: It’s like a Presidential election…
GG: Thank you so much, Bruce!
Bruce: My pleasure. I’m really glad you came in.
So, there’s a somewhat brief look at what goes into Live Production in WAR and what is being focused on. Obviously, lately there’s been a lot of talk on the Forum about version control, especially after the 1.3.1 patch. Hopefully we see some improvements in that regard with the next patch. ~crosses fingers~
Maybe I’ll be lucky enough to get a follow-up interview with Bruce after the next patch and we can discuss Live Production some more. There’s definitely a lot more that can be discussed in that regard to flesh out some things for the player-base in a better way so we all know where Mythic is heading. We’ll see.