Earlier in my career, I was doing QA on a particular AAA franchise where QA spent a lot of very late nights and weekends testing the game. As we were getting closer to launch, we were finding new bugs that were being introduced due to sudden changes and updates that were being put in as we got closer to launch. That was making the QA team nervious, but we figured something like “we’ll just spend more time on it, throw more people at it. It’ll be fine”. When it came time to ship, we collectively signed off on the release candidate. For the most part, we were confident, but in the back of my head, I thought to myself “This doesn’t feel right. I’m not confident in this”.
Which was strange to feel, because before this specific game, I had been QA on other projects at another game studio where we absolutely shipped with the utmost confidence in the build. I didn’t think too much about what the difference was until the last few years when I would think about past projects. The difference between these games was simple: evidence. At the previous studio, we were very confident in that build because we had documentation and results on regression tests, platform coverage, known issues, and defect trends. We had a paper trail that led back to our test suites and coverage in all areas that showed the build was stable, in a clean state, and eligible to be a release candidate for submission.
On the later project at the other studio, we had tons of logged hours, weekends of testing, some bugs entered, and a top 10 list of concerning bugs logged. And we signed off with confidence that the build was stable, but without much evidence to support it. If someone were to ask QA “what did you test?”, we didn’t have anything to show for it. Sure, there weren’t many bugs logged, and none of them were high or critical priority. But it doesn’t show exactly everything that was tested and evidence of what we covered.
A release candidate can be ready to ship with known bugs. That was very common in my experience. But readiness doesn’t mean perfection. It means that the risk remaining at the time of ship is collectively accepted and understood. QA shouldn’t be expected to guarantee that a release is defect-free. Its job is to provide enough evidence for the team to understand the condition of the build and make an informed release decision. Release readiness isn’t the absence of defects. It’s an informed understanding of the risk.