When is Twitter like a bad game of telephone?

This morning while perusing LinkedIn status updates, I noticed a retweet3 attributed to Guy Kawasaki that read “Study claims Instant Messaging improves productivity.” I clicked through to the blog post by Rick Broida titled “Instant Messaging improves productivity” that claimed

“..a recent study found that workers who trade instant messages with their bosses are more productive than those who don’t.”

Rick quoted an article about the study that stated:

The researchers analyzed the e-mail traffic, buddy lists, social networking friends, and everything in between of 2,600 [sic] anonymized IBM consultants over 12 months [sic] in order to draw their conclusions. They then compared their communication patters against their performance in billable hours. Those who maintained constant communications averaged an increase in revenue of $588 per month over the average, while those who did not produced $98 per month less than the average.

This sounded great to me, so I wanted to read the full story.

The quote above came from a post by Jacqui Cheng at arstechnica.com, titled “Study: frequent IMs with your boss make you more productive,” who went on to write:

The reason the latter group is underperforming could be because they feel pulled in too many directions with no clear leadership. “It could be that when consultants are linked to many managers, they may have conflicting demands from them on how the project should be executed,” reads the report. “The consultants would be rendered less effective without a clear direction.”

In a way, the findings come as no surprise. People who feel like they know their bosses better are generally happier with their jobs and are more willing to go the extra mile when needed. Similarly, bosses who know their employees’ quirks are better able to assign them to projects that fit their personalities and strengths, while minimizing their weaknesses.

Wah? Pulled in too many directions? IM helps solve this? OK, wait a minute, so what was the study really about?
I located the original MIT paper, which was presented at the Winter Information Systems Conference in Salt Lake City this February. The researchers were interested in measuring the quality and nature of social networks, and trying to assess the notion that all nodes are not created equal. Basic social network theories hold that some nodes in the network are more strategic than others – just as cities that have more highways and rail lines running through them do more commerce. These researchers are taking things a step further, and are interested in the quality of the connections, not just the quantity – where do those roads and railroads lead, and what shape are they in?

They were interested in determining how employees with direct access to managers fared relative to those with indirect access – or those with access to too many manager. In their words:

The strength of connections and attributes of the nodes in the network, such as human capital, power and status, can have a unique relationship with work performance.

Instant message traffic, email, calendars, and other electronic communications were analyzed as well as the organization chart and the corporate project and personal revenue database for about 1000 consultants, selected out of total pool of 400,000 IBM employees in the social network database that was created by previous researchers. The researchers use electronic communications as a proxy for access to management. Their hypothesis is that access leads to activity, and activity leads to revenue:

One important aspect of human capital is status and power endowed to a person. Strong ties to powerful individuals can be beneficial. Authorities such as partners and executives in a consulting organization often have the discretion to pick specific consultants to staff important projects that have high earning potential. Project managers prefer to choose consultants whom they trust to deliver excellent results and therefore, having a good relationship with project managers and gaining their trust is crucial to increase utilization and the likelihood to participate in high valued ventures. Consultants with strong ties to managers who are often experts are also likely to perform better as they can access useful knowledge and subject area expertise. Receiving targeted and useful information directly from the manager with minimum information distortion, consultants with strong ties to management are even more likely to complete a project. This forms a virtuous cycle where strong connections to managers increase the chance of accomplishing a project which then enhances a consultant‘s reputation and attracts even more connections to project managers. Similarly, we expect project teams with strong ties to managers outside of the team to be more successful as well.

So who did they study and what did they find? The researchers studied 1029 consultants (not 2600 as reported in arstechnica) and 2952 projects from June, 2007 to July, 2008 (13 months, not 12). The projects involve 39 countries with the majority of the projects in the US. Of the 1029 consultants, 864 are male, 66 are project managers. Low to mid level consultants represent 69.4% of the population while the rest are senior consultants or executives.

To understand a broader view of how social network is associated with performance, we construct two types of social networks, at person level and project level. In the employee or person level, each node in the network represents an employee and each link between two nodes represents the total number of electronic communications between the two people. To construct project level networks, each node in the networks is a project instead of an individual and links between projects is the sum of all communications between members of the two projects, ignoring all communications happening within the same project team. We hypothesize that the network characteristics that are most conducive for project networks may differ from those at the individual level. For example, the ability of a consultant to generate revenue depends on his ability to bill hours and participate in high valued projects, while project revenue depends not only on the ability of individual team member but also on how they work together to complete the project.

We collect financial performance such as billable hours and revenue they generated on a monthly basis, as well as project characteristics such the difficulty of the tasks, the region where the project is performed, the industry of the client, line of business for the consultants and clients, and duration it takes for the project to be completed. To control for individual characteristics of consultants, we also collected demographic data, such as the technical expertise, geographical locations, managerial roles, and the division they belong within the organization. Unfortunately, we cannot obtain information on how long a consultant has worked in his current job role. This could skew our results as we may expect that employees who are around for a long time are more likely to have a larger network since they have more time and opportunities to build their social network. However, we can eliminate this bias using longitudinal networks.

The findings: First, that a network that is “cohesive” – i.e. incestuous – is less productive than a diverse network. Being connected to people with connections that look like yours lowers productivity.

Network cohesion is negatively correlated with productivity, corroborating the results of previous email network studies (Aral et. al 2006, 2007, Aral and Van Alstyne 2007) demonstrating that structurally diverse networks are positively associated with productivity. One standard deviation increase in network constraint is associated with a decrease of 276% in monthly revenue.

Second, that an individual that has more closer ties or more direct paths to others (i.e. more “betweenness centrality”) is more productive:

Similarly, betweenness centrality is also positively correlated with performance, where one standard deviation in betweenness centrality is associated 513% increase in monthly revenue. Based on interviews with consultants at various stage of their career, we find that accurate and timely access to novel information through a structural diverse network is crucial to project completion by assisting consultants to solve complex problems and discover new opportunities. When an information worker is positioned at the center of the network where she can access other actors quickly, she is in the most effective position to access novel information quickly (Freeman 1979, Brass & Burkhardt 1992, Burt 1992, Hansen 2001), and act on newly discovered opportunities. This is particularly important in consulting business, as lining up projects to avoid bench time is crucial for performance.

So let’s talk a moment about cause and effect. Does “instant messaging increase productivity?” No. It’s not whether or how much you ping or tweet, it’s with whom you’re connected; employees that have networks composed of peers are less productive. Do “frequent IMs with your boss make you more productive?” Nope. Employees that IM with their bosses earn more because they are assigned to more projects. Are employees assigned to more projects because they IM to their bosses more? No, they are assigned to more projects because they have better access, because they have better skills and experience. IM traffic is the result, not the cause.

If you’re a schlumpy third-stringer with weak skills and a butt full of splinters from warming the bench, you can’t IM your way to success, and you can’t network your way there either. Sorry. Go gain some skills, build a track record, and in the process you’ll make some powerful connections along the way. The old-school way. And maybe you’ll even connect with them on LinkedIn get some of their IM handles to boot.

And one more thing – think about going back and thoughtfully reading and understanding the original work before you retweet a retweet of a blog of a blog of a news article on an academic paper.

Oh, and never confuse cause with effect. Go re-read Freakonomics