Leaders in People Analytics: Collaborative Research Approach

Author’s note: This article was originally found here.

Introduction

Much of the work done at Google, and in many organizations, is done collaboratively by teams. The team is the molecular unit where real production happens, where innovative ideas are conceived and tested, and where employees experience most of their work. But it’s also where interpersonal issues, ill-suited skill sets, and unclear group goals can hinder productivity and cause friction.

Following the success of Google’s Project Oxygen research where the People Analytics team studied what makes a great manager, Google researchers applied a similar method to discover the secrets of effective teams at Google. Code-named Project Aristotle – a tribute to Aristotle’s quote, “the whole is greater than the sum of its parts” (as the Google researchers believed employees can do more working together than alone) – the goal was to answer the question: “What makes a team effective at Google?”

Read about the researchers behind the work in The New York Times: What Google Learned From Its Quest to Build the Perfect Team

Define what makes a “team”

The first step in answering this question of “what makes an effective team?” is to ask “what is a team?” More than an existential thought exercise, actually figuring out the memberships, relationships, and responsibilities of individuals all working together is tough but critical to cracking team effectiveness.

The term team can take on a wide array of meanings. Many definitions and frameworks exist, depending on task interdependence, organizational status, and team tenure. At the most fundamental level, the researchers sought to distinguish a “work group” from a “team:”

• Work groups are characterized by the least amount of interdependence. They are based on organizational or managerial hierarchy. Work groups may meet periodically to hear and share information.
• Teams are highly interdependent – they plan work, solve problems, make decisions, and review progress in service of a specific project. Team members need one another to get work done.

Organizational charts only tell part of the story, so the Google research team focused on groups with truly interdependent working relationships, as determined by the teams themselves. The teams studied in Project Aristotle ranged from three to fifty individuals (with a median of nine members).

Define “effectiveness”

Once they understood what constituted a team at Google, the researchers had to determine how to quantitatively measure effectiveness. They looked at lines of code written, bugs fixed, customer satisfaction, and more. But Google’s leaders, who had initially pushed for objective effectiveness measures, realized that every suggested measure could be inherently flawed – more lines of code aren’t necessarily a good thing and more bugs fixed means more bugs were initially created.

Instead, the team decided to use a combination of qualitative assessments and quantitative measures. For qualitative assessments, the researchers captured input from three different perspectives – executives, team leads, and team members. While they all were asked to rate teams on similar scales, when asked to explain their ratings, their answers showed that each was focused on different aspects when assessing team effectiveness.

Executives were most concerned with results (e.g., sales numbers or product launches), but team members said that team culture was the most important measure of team effectiveness. Fittingly, the team lead’s concept of effectiveness spanned both the big picture and the individuals’ concerns saying that ownership, vision, and goals were the most important measures.

So the researchers measured team effectiveness in four different ways:

1. Executive evaluation of the team
2. Team leader evaluation of the team
3. Team member evaluation of the team
4. Sales performance against quarterly quota

The qualitative evaluations helped capture a nuanced look at results and culture, but had inherent subjectivity. On the other hand, the quantitative metrics provided concrete team measures, but lacked situational considerations. These four measures in combination, however, allowed researchers to home in on the comprehensive definition of team effectiveness.

Collect data and measure effectiveness

Using input from executives across the globe, the research team identified 180 teams to study (115 project teams in engineering and 65 pods in sales) which included a mix of high- and low-performing teams. The study tested how both team composition (e.g., personality traits, sales skills, demographics on the team) and team dynamics (e.g., what it was like to work with teammates) impact team effectiveness. Ideas were pulled from existing research as well as Google’s own experience with what makes an effective team.

They conducted hundreds of double-blind interviews with leaders to get a sense of what they thought drove team effectiveness. The researchers then looked at existing survey data, including over 250 items from the annual employee engagement survey and gDNA, Google’s longitudinal study on work and life, to see what variables might be related to effectiveness. Here are some sample items used in the study that participants were asked to agree or disagree with:

• Group dynamics: I feel safe expressing divergent opinions to the team.
• Skill sets: I am good at navigating roadblocks and barriers.
• Personality traits: I see myself as someone who is a reliable worker (informed by the Big Five personality assessment).
• Emotional intelligence: I am not interested in other people’s problems (informed by the Toronto Empathy Questionnaire).

Demographic variables like tenure, level, and location were also collected.

Identify dynamics of effective teams

With all of this data, the team ran statistical models to understand which of the many inputs collected actually impacted team effectiveness. Using over 35 different statistical models on hundreds of variables, they sought to identify factors that:

• impacted multiple outcome metrics, both qualitative and quantitative
• surfaced for different kinds of teams across the organization
• showed consistent, robust statistical significance

The researchers found that what really mattered was less about who is on the team, and more about how the team worked together. In order of importance:

• Psychological safety: Psychological safety refers to an individual’s perception of the consequences of taking an interpersonal risk or a belief that a team is safe for risk taking in the face of being seen as ignorant, incompetent, negative, or disruptive. In a team with high psychological safety, teammates feel safe to take risks around their team members. They feel confident that no one on the team will embarrass or punish anyone else for admitting a mistake, asking a question, or offering a new idea.
• Dependability: On dependable teams, members reliably complete quality work on time (vs the opposite – shirking responsibilities).
• Structure and clarity: An individual’s understanding of job expectations, the process for fulfilling these expectations, and the consequences of one’s performance are important for team effectiveness. Goals can be set at the individual or group level, and must be specific, challenging, and attainable. Google often uses Objectives and Key Results (OKRs) to help set and communicate short and long term goals.
• Meaning: Finding a sense of purpose in either the work itself or the output is important for team effectiveness. The meaning of work is personal and can vary: financial security, supporting family, helping the team succeed, or self-expression for each individual, for example.
• Impact: The results of one’s work, the subjective judgement that your work is making a difference, is important for teams. Seeing that one’s work is contributing to the organization’s goals can help reveal impact.

The researchers also discovered which variables were not significantly connected with team effectiveness at Google:

• Colocation of teammates (sitting together in the same office)
• Consensus-driven decision making
• Extroversion of team members
• Individual performance of team members
• Workload size
• Seniority
• Team size
• Tenure

It’s important to note though that while these variables did not significantly impact team effectiveness measurements at Google, that doesn’t mean they’re not important elsewhere. For example, while team size didn’t pop in the Google analysis, there is a lot of research showing the importance of it. Many researchers have identified smaller teams – containing less than 10 members – to be more beneficial for team success than larger teams (Katzenbach & Smith, 1993; Moreland, Levine, & Wingert, 1996. Smaller teams also experience better work-life quality (Campion et al., 1993), work outcomes (Aube et al., 2011), less conflict, stronger communication, more cohesion (Moreland & Levine, 1992; Mathieu et al., 2008), and more organizational citizenship behaviors (Pearce and Herbik, 2004).

build Tool: Help teams determine their own needs

Beyond just communicating the study results, the Google research team wanted to empower Googlers to understand the dynamics of their own teams and offer tips for improving. So they created a survey for teams to take and discuss amongst themselves. Survey items focused on the five effectiveness pillars and questions included:

• Psychological safety – “If I make a mistake on our team, it is not held against me.”
• Dependability – “When my teammates say they’ll do something, they follow through with it.”
• Structure and Clarity – “Our team has an effective decision-making process.”
• Meaning – “The work I do for our team is meaningful to me.”
• Impact – “I understand how our team’s work contributes to the organization’s goals.”

After completing the survey, team leads received aggregated and anonymized scores to share with team members and inform a discussion. A People Operations facilitator would often join the discussion, or the team lead would follow a discussion guide created by the People Operations team.

Team Effectiveness Discussion Guide

This discussion guide is focused on the five team dynamics Google found to be important for team effectiveness. The guide can help teams identify areas where they might want to improve and elicit ideas of how to do that.

build Tool: Foster psychological safety

Of the five key dynamics of effective teams that the researchers identified, psychological safety was by far the most important. The Google researchers found that individuals on teams with higher psychological safety are less likely to leave Google, they’re more likely to harness the power of diverse ideas from their teammates, they bring in more revenue, and they’re rated as effective twice as often by executives.

Organizational behavioral scientist Amy Edmondson of Harvard first introduced the construct of “team psychological safety” and defined it as “a shared belief held by members of a team that the team is safe for interpersonal risk taking.” Taking a risk around your team members may sound simple. But asking a basic question like “what’s the goal of this project?” may make you sound like you’re out of the loop. It might feel easier to continue without getting clarification in order to avoid being perceived as ignorant.

To measure a team’s level of psychological safety, Edmondson asked team members how strongly they agreed or disagreed with these statements:

1. If you make a mistake on this team, it is often held against you.
2. Members of this team are able to bring up problems and tough issues.
3. People on this team sometimes reject others for being different.
4. It is safe to take a risk on this team.
5. It is difficult to ask other members of this team for help.
6. No one on this team would deliberately act in a way that undermines my efforts.
7. Working with members of this team, my unique skills and talents are valued and utilized.

In her TEDx talk, Edmondson offers three simple things individuals can do to foster team psychological safety:

1. Frame the work as a learning problem, not an execution problem.
2. Acknowledge your own fallibility.
3. Model curiosity and ask lots of questions.

In promoting the results of Google’s research internally, the research team has been running workshops with teams. In the workshops, anonymized scenarios have been used to illustrate behaviors that can support and harm psychological safety. The scenarios are role-played and then the group debriefs. Here’s an example scenario:

Psychological Safety Scenario | Ideas & Innovation

Uli is a long time manager known for his technical expertise. For the past two years he’s worked as manager of team XYZ, which is responsible for running a large scale project. He upholds very high standards, but in the past few months Uli has become increasingly intolerant of mistakes, ideas he considers to be “underpar,” and challenges to his way of thinking.

Recently, Uli publically “trounced” an idea offered by an experienced team member and spoke very negatively about that person to the wider team behind their back. Everyone else thought the idea was strong, well-researched, and worth exploring. Ideas have since dried up.

Uli’s ideas drove the recent project proposal, but it was ultimately rejected by the executives because it lacked creativity and innovation.

Debriefing questions:

• What behaviors do you see that reflect psychological safety?
• What behaviors may signal that psychological safety is lacking in the scenario?
• Why is psychological safety so important? What difference does it make in a team? What have you seen on your teams?

If you’re a manager, consider these recommendations when coaching team members and teammates.

Manager Actions for Psychological Safety

This guide can help managers think about how they model and reinforce psychological safety on their teams. Based on research, this guide offers actionable tips for managers and team members to help create team environments where everyone can contribute.get_app DOWNLOAD PDFopen_in_browser OPEN AS GOOGLE DOC

Help teams take action

The five key dynamics of effective teams that the Google researchers identified are rooted in the wider world of team performance research. Whether you’re coding at Google, riffing in a writers room, preparing for a trip to Mars, or skating in a hockey rink – teams are essential to the work experience and output. At Google, now that the Project Aristotle team has identified what makes for an effective team at Google, they’re conducting research to figure out how take the next steps to create, foster, and empower effective teams.

Whatever it is that makes for effective teams in your organization, and it may be different from what the Google researchers found, consider these steps to share your efforts:

1. Establish a common vocabulary – Define the team behaviors and norms you want to foster in your organization.
2. Create a forum to discuss team dynamics – Allow for teams to talk about subtle issues in safe, constructive ways. An HR Business Partner or trained facilitator may help.
3. Commit leaders to reinforcing and improving – Get leadership onboard to model and seek continuous improvement can help put into practice your vocabulary.

Here are some tips for managers and leaders to support the behaviors the Google researchers found important for effective teams. These are based on external research and Google’s own experience:

Psychological safety:

• Solicit input and opinions from the group.
• Share information about personal and work style preferences, and encourage others to do the same.
• Watch Amy Edmondson’s TED Talk on psychological safety.

Dependability:

• Clarify roles and responsibilities of team members.
• Develop concrete project plans to provide transparency into every individual’s work.
• Talk about some of the conscientiousness research.

Structure & Clarity:

• Regularly communicate team goals and ensure team members understand the plan for achieving them.
• Ensure your team meetings have a clear agenda and designated leader.
• Consider adopting Objectives & Key Results (OKRs) to organize the team’s work.

Meaning:

• Give team members positive feedback on something outstanding they are doing and offer to help them with something they struggle with.
• Publicly express your gratitude for someone who helped you out.
• Read the KPMG case study on purpose.

Impact:

• Co-create a clear vision that reinforces how each team member’s work directly contributes to the team’s and broader organization’s goals.
• Reflect on the work you’re doing and how it impacts users or clients and the organization.
• Adopt a user-centered evaluation method and focus on the user.