What Is QA Looking At?
At every game studio where I’ve worked, there came a point in the project when QA leadership was asked the inevitable question: “What is QA looking at?” Now this question can be taken in a myriad of ways, and it was, especially early on in my career. When I was in a leadership position and I was presented with this question, I almost took it personally. “What do they think we’re looking at?” I’d say to myself. But they weren’t just asking what we were testing. They were asking for transparency.
QA works alongside every department in the company. People come to QA for various requests, because QA knows the build inside and out and sees the big picture of the whole game. So when people come to QA asking for information, they do that because QA has one of the broadest cross-functional views of the game. An informative QA update should communicate the current risk, any noticeable changes in quality, major unknowns, blockers, and a general level of confidence in the builds that are being tested. QA should also communicate what is not yet known, such as areas that have not yet been thoroughly tested, the unknown stability of certain features, and risks that cannot yet be measured.
While working at a particular studio, the lead game programmer would come to our QA manager and ask for QA project updates when we entered beta. I can’t remember if it was daily or weekly, but they wanted an emailed update at the end of each day or week. At first, not much context was given on exactly what information they were looking for. So I took an educated guess and gave a summary of how many bugs were entered, what specific areas were being tested and what we planned to test that following day or week. As the updates went on, that lead would reply to the updates, asking more and more questions, until I finally understood what they were really asking for.
They weren’t asking for a bug count, or what we were testing at that moment, or what build we were testing on. They wanted a risk assessment of the build at that point in the project. What concerned QA the most? What was QA seeing in the feature that had been implemented that week? What areas are improving or declining in quality based on Jira data? They wanted to know how QA was reading the project’s heartbeat so that they could get a sense of where we collectively stood, as a dev team.
What also surprised me was who was reading these updates. I was directed to send the updates to a specific email distro that included studio leadership. But the lead programmer was the only person who was replying to the updates. So I assumed they were the only person reading them. Eventually, the studio head started replying to these updates. They would ask me to expand on a specific concern, or ask what blockers we had. And if we did have a blocker, they would help chase it down and see what the issue was.
Transparency in QA is not about reporting more data. It is about making the project’s actual quality state understandable for those who need it.