By James Whittaker

This is the first in a series of posts on this topic.

The one question I get more than any other is "How does Google test?" It's been explained in bits and pieces on this blog but the explanation is due an update. The Google testing strategy has never changed but the tactical ways we execute it has evolved as the company has evolved. We're now a search, apps, ads, mobile, operating system, and so on and so forth company. Each of these Focus Areas (as we call them) have to do things that make sense for their problem domain. As we add new FAs and grow the existing ones, our testing has to expand and improve. What I am documenting in this series of posts is a combination of what we are doing today and the direction we are trending toward in the foreseeable future.

Let's begin with organizational structure and it's one that might surprise you. There isn't an actual testing organization at Google. Test exists within a Focus Area called Engineering Productivity. Eng Prod owns any number of horizontal and vertical engineering disciplines, Test is the biggest. In a nutshell, Eng Prod is made of:

1. A product team that produces internal and open source productivity tools that are consumed by all walks of engineers across the company. We build and maintain code analyzers, IDEs, test case management systems, automated testing tools, build systems, source control systems, code review schedulers, bug databases... The idea is to make the tools that make engineers more productive. Tools are a very large part of the strategic goal of prevention over detection.

2. A services team that provides expertise to Google product teams on a wide array of topics including tools, documentation, testing, release management, training and so forth. Our expertise covers reliability, security, internationalization, etc., as well as product-specific functional issues that Google product teams might face. Every other FA has access to Eng Prod expertise.

3. Embedded engineers that are effectively loaned out to Google product teams on an as-needed basis. Some of these engineers might sit with the same product teams for years, others cycle through teams wherever they are needed most. Google encourages all its engineers to change product teams often to stay fresh, engaged and objective. Testers are no different but the cadence of changing teams is left to the individual. I have testers on Chrome that have been there for several years and others who join for 18 months and cycle off. Keeping a healthy balance between product knowledge and fresh eyes is something a test manager has to pay close attention to.

So this means that testers report to Eng Prod managers but identify themselves with a product team, like Search, Gmail or Chrome. Organizationally they are part of both teams. They sit with the product teams, participate in their planning, go to lunch with them, share in ship bonuses and get treated like full members of the team. The benefit of the separate reporting structure is that it provides a forum for testers to share information. Good testing ideas migrate easily within Eng Prod giving all testers, no matter their product ties, access to the best technology within the company.

This separation of project and reporting structures has its challenges. By far the biggest is that testers are an external resource. Product teams can't place too big a bet on them and must keep their quality house in order. Yes, that's right: at Google it's the product teams that own quality, not testers. Every developer is expected to do their own testing. The job of the tester is to make sure they have the automation infrastructure and enabling processes that support this self reliance. Testers enable developers to test.

What I like about this strategy is that it puts developers and testers on equal footing. It makes us true partners in quality and puts the biggest quality burden where it belongs: on the developers who are responsible for getting the product right. Another side effect is that it allows us a many-to-one dev-to-test ratio. Developers outnumber testers. The better they are at testing the more they outnumber us. Product teams should be proud of a high ratio!

Ok, now we're all friends here right? You see the hole in this strategy I am sure. It's big enough to drive a bug through. Developers can't test! Well, who am I to deny that? No amount of corporate kool-aid could get me to deny it, especially coming off my GTAC talk last year where I pretty much made a game of developer vs. tester (spoiler alert: the tester wins).

Google's answer is to split the role. We solve this problem by having two types of testing roles at Google to solve two very different testing problems. In my next post, I'll talk about these roles and how we split the testing problem into two parts.

I'll be speaking very soon about innovation and The Pretotyping Manifesto (note: not prototyping). It's a concept that works well for any type of engineering, testing, or idea. Here's a brief preview...

The talk starts off by discussing the odds against innovators and how the deck is stacked against you from the start. Most engineers begin with trying to come up with THE killer idea. But you quickly realize that ideas are cheap. We all believe that our own ideas are good. As a matter of fact, most of us LOVE our own ideas to the point where it clouds our judgement. In parallel, there is a strong desire to jump in and build something after a great idea has been identified. Sometimes swarms of well intentioned people join in and "help." Quickly innovators can find themselves in the weeds or feeling disconnected from their original dream. So, what can be done? The idea is to use pretotyping and to focus on techniques that allow you to: Build the right it vs. Build it right.

Last time I did this talk, it was called: "the best keynote of any tech conference ever!"

I'm looking forward to seeing some of you the week of March7th when I'll be in London. In addition to dropping into the Google office, I'll be speaking at QCon London. If you want to attend (at a cheaper rate than normal) here are the details:

• London event March 7-11, 2011.
• My personal "promotion code" will save you £100 if they enter this code during registration.
• Promotion code is: COPE100

Hope to see you in London,

Patrick Copeland

Senior Engineering Director, Google

By James Whittaker

I know many people who laugh at the concept of resolutions, easily made and easily broken. All true. However, I am a runner now because of a resolution I made about a decade ago and my personality has undergone a successful renovation or two over the years as well. When they stick, resolutions can become habits and the emergence of the occasional butterfly makes them a worthwhile exercise. With the optimism of a new year, I present my Google Testing resolutions for 2011 which I hereby declare the Year of the User.

1. I will listen to users more and developers less.

Developers, by definition, are engineers lost in the details of implementation. When it comes to testing concerns, such implementation details clog a tester's neural pathways with issues that simply should not be relevant. I resolve to take the high road as often as possible and consider user scenarios, integration issues and end-to-end uses of the system above all other concerns. And, yes, that will mean telling developers "sorry, dude, your broken build simply is not my concern."

2. I will push all implementation testing issues to developers.

My first resolution may lead readers to believe that testing implementation details isn't important. Let me be clear. Testing implementation details is important. When they go untested they create enough noise that user-oriented testing is compromised by the constant emergence of silly bugs. Silly bugs mask important ones. Find them at the source: ensure that proper unit testing and automated smoke tests are present and owned by the people most qualified to write and maintain them: developers. I resolve not to be sidetracked by silly bugs but to push back hard on the developers who are happy to write the bug but neglect to write the test for it.

3. I will endeavor to tie together all user-oriented testing.

In the run up to releasing Chrome OS for pilot last year it was clear that many of the bugs found during dogfood (internal testing), crowd-sourced and out-sourced testing had already been found by my test team. Not only is there is a lot of repetitive and wasteful testing being performed, my team isn't getting enough credit for finding these important issues early. I resolve to introduce technology that will allow all testers to share testing tactics and see each other's work, ultimately erasing the boundaries between these common phases and allowing testers who join a project late to build upon the work of those who've been there for a while.

Finally, I resolve to expose more information about how Google tests internally. I am going to return to the conference circuit this year and talk frankly about what we are doing, the good, the bad and the downright embarrassing in the hopes that other testers at other companies do the same. I am also going to push more Google testers to post their experiences on this blog and join me at industry events to discuss these things with anyone struggling with the same issues.

Happy New Year! May it be one that sees a higher level of quality from every corner of our industry.