key: cord-0281636-6205efa1
authors: Ford, Denae; Storey, Margaret-Anne; Zimmermann, Thomas; Bird, Christian; Jaffe, Sonia; Maddila, Chandra; Butler, Jenna L.; Houck, Brian; Nagappan, Nachiappan
title: A Tale of Two Cities: Software Developers Working from Home During the COVID-19 Pandemic
date: 2020-08-25
journal: nan
DOI: 10.1145/3487567
sha: 01449ea6ca0d4f00ede5040ae7e6dda33ad8cf58
doc_id: 281636
cord_uid: 6205efa1

The COVID-19 pandemic has shaken the world to its core and has provoked an overnight exodus of developers that normally worked in an office setting to working from home. The magnitude of this shift and the factors that have accompanied this new unplanned work setting go beyond what the software engineering community has previously understood to be remote work. To find out how developers and their productivity were affected, we distributed two surveys (with a combined total of 3,634 responses that answered all required questions) -- weeks apart to understand the presence and prevalence of the benefits, challenges, and opportunities to improve this special circumstance of remote work. From our thematic qualitative analysis and statistical quantitative analysis, we find that there is a dichotomy of developer experiences influenced by many different factors (that for some are a benefit, while for others a challenge). For example, a benefit for some was being close to family members but for others having family members share their working space and interrupting their focus, was a challenge. Our surveys led to powerful narratives from respondents and revealed the scale at which these experiences exist to provide insights as to how the future of (pandemic) remote work can evolve.

Charles Dickens' "A Tale of Two Cities" begins, "It was the best of times, it was the worst of times. " Adapting Dickens' line to leading an engineering team during the global pandemic, I'd say "We're doing very well, we're barely hanging in there. "

-Shane O'Flynn [1] Software engineering is a complex technical, knowledge-based task that requires focused, uninterrupted work [2] while still coordinating and collaborating with other developers and stakeholders [3] . Despite a need for intense periods of coordination, collaboration, and communication for managing intricate dependencies within and across systems, there are compelling success stories of how developers can effectively develop high-quality, complex software in a distributed fashion. Successful open source communities, globally distributed software projects and fully remote software companies are all testaments to distributed and remote work. Decades of engineering tools (such as version control and continuous integration tools) and knowledge sharing tools (such as email, Stack Overflow, and Wikipedia) were conceived and designed by developers for developers to manage the collaborative and distributed nature of software engineering [4] . Authors Despite having rich tools to support distributed development, many software companies believe there are significant advantages to working in a co-located fashion, with many advocating for close proximity among developers (such as in shared team rooms) [5] . Some of the claimed benefits for co-location are seamless coordination, increased creativity, faster learning, and projects that are easier to manage [6, 7] .

There are numerous studies that give insights about the benefits and challenges of distributed corporate development work compared with co-located development [8] [9] [10] , but these studies tend to focus on specific teams or investigate the velocity or quality of the code developed using quantitative analysis of system trace data. They do not capture the experiences of developers in large companies that have had to switch (over night) from a mostly co-located mode to remote work from home. And yet this is what happened for many developers worldwide with the pandemic forcing technology companies to close their offices.

The COVID-19 pandemic has undoubtedly been a global human and economic disaster, but it has also led to interesting and unexpected revelations about how we work, play and live. Although working from home during a pandemic is not the same as working from home during "normal times" 1 , it is nevertheless an opportunity to study the results from a "natural experiment" and compare the benefits and challenges developers may experience in terms of their development productivity in these two modes. Our work reveals a "tale of two cities" effect, where some developers flourished working at home, while others did not fare as well.

The aim of this research is to understand how and why developers' perceived productivity may have changed over the weeks following the initial work from home directive, and to identify the benefits and challenges they experienced. We report on the very early experiences developers found working from home and their experiences after several weeks of working in this mode. The paper is based on data collected through two surveys with a combined total of 3,634 responses. The first survey was among employees working at Microsoft in Washington State (the location of the company's headquarters), consisted of 30 questions, and received 1,369 responses. The second survey was among Microsoft employees in the United States, consisted of 42 questions, and received 2,265 responses. Our initial survey uncovered the main benefits and challenges they faced, and the second survey revealed the frequency/impact of those benefits/challenges. From the second survey, we also uncovered developers' self-reported changes in productivity since working from home (WFH) and how the benefits and challenges they experienced associate with those changes in productivity. From our survey responses, we also identified improvements that can be made to support the WFH experience.

Notably, we found dichotomous experiences of engineers (developers and program managers) working from home (WFH) during the pandemic from the same factors. For example, for some no commute was a benefit, but for others the lack of a commute removed a period of relaxation or preparation from their day. These differential experiences were in some cases driven by engineers' personal contexts (e.g., if they have school-age children or space for a home office) and the characteristics of their work (such as reliance on team members). However, some factors were experienced as both a benefit and a challenge by some of the same participants (such as time flexibility being both a blessing and a curse).

The main takeaways from our paper are as follows:

• Productivity, when measured using engineering system data, appears to be stable or slightly improved, but some developers, at least initially, appeared to thrive and report being more productive, while others face significant challenges with remote work and feel they are not as productive.

• Some factors (such as schedule flexibility, proximity to family members and more time for work) lead to dichotomous experiences across developers and even by individual developers. • Organizations can support remote work by understanding the varied experiences of developers and the challenges their employees may face, and that there are actionable recommendations they can follow to support developers working from home now or in the future as part of a hybrid model.

Our paper is organized as follows. In Section 2, we provide a background of literature that has studied developer performance and productivity. In Section 3, we describe the methodology we followed for the two surveys we conducted over the first few months of the pandemic. In Section 4, we discuss the changes in productivity reported across the surveys. In Section 5, we report the main benefits encountered, and in Section 6, we report the main challenges encountered as developers working from home. Section 7 summarizes the key recommendations that developers suggested organizations should follow to improve the situation for developers working from home. In Section 8, we delve into the main factors and discuss how these may play a role in the dichotomous experiences of developers. We also discuss how engineering system output data can help triangulate the impact of working from home and how the pandemic may shape the future of remote work. In Section 9, we discuss research on remote work (pre-pandemic) as well as more recent related works about programmers during the pandemic. Finally, we conclude our paper in Section 10.

Understanding developer productivity in software engineering has seen great interest from research and industry, as improving developer productivity may lead to faster development speed, higher quality code, and also higher developer satisfaction. Some of the factors we uncovered in our study overlap earlier research, but the abrupt change for individual developers and the entire organization to working from home reveal new benefits and challenges of working from home, as we discuss later in the paper. A concern during the shift to working from home during the pandemic is that both developer productivity and well-being may have been negatively affected. Existing research has led to insights about developer performance, productivity, satisfaction and developer well-being.

In terms of performance, system engineering activity metrics can provide important signals about developer activity and productivity. Wagner and Ruhe's review of the literature summarize studies that use performance measures such as lines of code or function points as proxies to productivity [11] . However, within the area of organizational behavior, performance and productivity are acknowledged to be related [12] , with higher levels of performance leading to higher levels of productivity, but they are also recognized as distinct concepts. Indeed, many researchers and practitioners emphasize that developer productivity cannot and should not be measured by engineering metrics alone as development work is not mechanized work that can be assessed using a single metric. In fact, doing so may be detrimental to overall and long-term development objectives [13] . For example, developers spend time mentoring newcomers, reviewing each other's work informally, and learning new skills.

Through their systematic literature review, Wagner and Ruhe [11] also identified 51 factors that influence productivity. In addition to the identified technical factors that seem to dominate productivity studies in software engineering, they also distilled a number of soft factors that focus on aspects such as organizational culture and working environment.

Using a different lens, Meyer et al. [14] investigated how developers perceive and think about their own productivity. Through a survey, observations, and interviews, their study brought to the surface that a developer's sense of how productive they are, may be distorted by how many interruptions and context switches they experience. Other research about perceived developer productivity reported that the quality of one's work environment plays a major role [15] , while the effectiveness of a manager [16] is also an important factor on perceived productivity.

More recent research has expanded these factors to include additional elements that may influence perceived productivity and satisfaction, or that can be used in certain contexts to predict productivity. In an earlier study [17] we conducted with developers at Microsoft, the factors that more closely associate with one's satisfaction with their self-reported productivity include job satisfaction, doing impactful work, having autonomy over one's work, the ability to complete tasks, the quality of the engineering system, the ability to complete tasks, personal technical skills, and their work environment. Predictive factors [18] include job enthusiasm, peer support for new ideas, and getting job feedback, while "use of remote work to concentrate" showed the lowest variance across three large software companies in terms of self-reported productivity.

In our research, we build on this previous work by inquiring about factors already shown to be important for productivity while also allowing new factors that are particular to the pandemic (such as having children at home) to emerge from our study. We discuss our methodology and the surveys in the next section.

We investigated the experiences of software engineers at Microsoft during the height of the COVID-19 pandemic in the United States using a set of online surveys. Figure 1 outlines the timeline of our study from March-May 2020.

Design of Survey 1 Privacy/ethics review Survey 1 sent daily over a period of five days to a total of 5,000 full-time employees (3,500 developers, 1,500 program managers) in King County. 1,369 responses (response rate: 27%) Analysis of Survey 1. Design of Survey 2 based on findings from Survey 1. Privacy/ethics review.

Most US states under stay at home orders (94% of the US population) [19] Survey 2 sent weekly to a total of 9,000 full-time employees in the United States (developers, program managers, data scientists). Research Context. The first presumptive positive COVID-19 case in King County, WA (which includes the Microsoft headquarters) was reported on February 29, 2020. In the late afternoon of March 4, Microsoft informed its employees that "Consistent with King County guidance, we are recommending all employees who are in a job that can be done from home should do so" [20] . On March 11, the schools were closed in Washington State (which was made permanent for the school year on April 8). In April, many other restrictions were introduced, and by April 27, the outbreak has reached its peak in Washington State. Some restrictions on outdoor activities were later lifted but social distancing was still recommended. The pandemic also affected the rest of the US: at the end of March, 42 states and a total of 308 million people (94% of the US population) were under stay at home orders [19] . At the time of writing this paper, Microsoft and other large tech companies had extended their recommendations to work from home until at least the Fall of 2020 and in some cases even Summer 2021 [21, 22] .

Research Questions. To understand the effect of WFH on software engineers through our surveys, we answer the following research questions: RQ1 How has engineers' self-reported productivity changed since WFH? RQ2 What are the benefits engineers experience when working from home? How have these benefits affected productivity since WFH? RQ3 What are the challenges engineers face when working from home? How have these challenges impacted productivity since WFH? RQ4 What recommendations should be made to companies whose engineers may wish to work from home? To answer these research questions, we distributed two anonymous surveys to understand the experiences of software developers during the pandemic, their prevalence, and the effect on their work.

Our first survey was designed to understand the types of experiences software developers were having during the pandemic. In this survey, we included a closed-answer question on how productivity has changed with a five-point scale for the responses. Through a following open-ended question, participants were asked to explain their response.

• Compared to working in office, how has your productivity changed? (Q13) 2 (significantly less productive / less productive / about the same / more productive / significantly more productive)

• Please share details about your answer to the previous question on how your productivity has changed. (Q14) Although there has been previous research to understand the factors that affect developer productivity (as discussed above), we anticipated that different benefits and challenges may be more relevant in this period of unexpected and mass transition to working from home for an entire organization. Therefore, the first survey was mainly exploratory to investigate if new factors would emerge through the following open-ended questions:

• What is good about working from home? (Q15)

• What is bad about working from home? (Q16)

• What challenges have you encountered working from home? (Q17)

• What could be improved about how we do work from home at Microsoft? (Q22)

In addition, the survey included questions about Internet connectivity, interruptions and distractions, work times, meetings, and commuting. The full survey is available as supplemental material [23] .

Survey Distribution. This survey was distributed to Microsoft employees in King County during the week of March 16, 2020 (approximately two weeks after the advice to work from home). Each day the survey was sent to 1,000 randomly selected employees in King County, for a total of 5,000 employees (3,500 developers, 1,500 program managers). We received 1,369 survey responses, with all required questions answered, for a response rate of 27% (comparable to the response rates of many other software engineering surveys [24, 25] ). To encourage participation, survey respondents could enter a raffle of multiple $100 Amazon.com gift certificates. No reminder emails were sent.

For the open-ended responses to this first survey, we used an open-coding approach, iterating and refining through multiple rounds of independent coding of an initial sample of responses. We coded all of the open-ended questions listed above, Codes for positive aspects of working from home (benefits, RQ2), negative aspects of working from home (challenges, RQ3) and improvements (RQ4) emerged across the questions.

After several iterations coding and discussing codes, we finalized a unified coding scheme with codes, code definitions and code categories (see Appendix A). We applied these finalized codes to a random selection of 400 responses (see Table 7 ). No new codes emerged during this process. To improve the reliability of our codes, an external researcher used our coding scheme to code a subset of our sample (100) showing an agreement of 81.9%.

The final coding scheme contained 32 codes organized into the following six themes. (The complete list of codes and descriptions can be found in Appendix A.)

• Beyond work -Effects of work from home on non-work aspects of respondents' lives, such as proximity to family, distribution of finances, and access to food. • Collaboration -Aspects of respondents' collaborative tasks of respondents, including challenges being creative with others, being blocked waiting on others, and a range of interactions with co-workers. • Communication -Work-related communication, including channels used, frequency, duration, planned versus ad-hoc, and the result of missing communication. • Well-being -Responses related to the welfare of respondents, including changes to flexibility of schedule and location, and the effects of working from home on health (physical, mental, and emotional) and personal comfort. • Work -Responses related to a direct effect on respondents' technical work output, including codes related to productivity, motivation, and factors affecting focus and distraction. • Work environment -Aspects of the setting in which the respondents accomplish their work when working from home, including the existence or lack of reliable internet connectivity, ergonomically sound furniture, satisfactory hardware, and dedicated space.

Note that some responses were coded with multiple codes when participants raised multiple points. For example, the following response was assigned multiple codes :

1. Avoiding commute, hence more productive, save on fuel (environment friendly). We show the frequency of the main codes from the 400 responses in Table 7 in the Appendix. However, these counts do not represent an accurate description of which benefits or challenges may be more important, and which ones may affect productivity more or less as these are open-ended questions. 3 

To investigate the importance and frequency of the reported benefits and challenges from the first survey and their association with self-reported productivity, we designed and deployed a second survey. Rather than open-ended questions, we included closed-answer questions for the factors that emerged from our coding of the first survey. These closed questions asked about benefits and how important they were to the respondent, as well as challenges and the impact of those challenges. In addition, this second survey inquired about additional benefits and challenges. 4 • Compared to working in office, how has your productivity changed? (Q13) (significantly less productive / less productive / about the same / more productive / significantly more productive)

• What benefits have you experienced working from home and how important are these benefits? (Q15*) (I don't experience this benefit / I experience this benefit but it's *not* important to me / I experience this benefit and it's *important* to me / I experience this benefit and it's *very important* to me)

• What work-related challenges have you experienced working from home and how impactful are these challenges? (Q17*) (I don't experience this challenge / I experience this challenge but it's a *minor issue* for me / I experience this challenge and it's a *major issue* for me)

• What could be improved about working from home (WFH)? Choose up to three (3) items. (Q22*) As items for the questions, we identified a list of 15 benefits (B1..B15), 20 challenges (C1..C20), and 12 improvements/suggestions (S1..S12) based on the thematic analysis of the responses to Survey 1. The items were displayed in random order within a question. The full survey is available as supplemental material [23] .

Survey Distribution. This survey was distributed to 9,000 engineers (consisting of developers, program managers and data scientists) across the entire US over a period of three weeks (a different sample of 3,000 employees was selected for each week). There was no overlap between the samples in Survey 1 and Survey 2. We received 2,265 responses for a response rate of 25%, with all required questions answered (comparable to the response rates of many other software engineering surveys [24, 25] ). To encourage participation, survey respondents could enter a raffle of multiple $100 Amazon.com gift certificates. No reminder emails were sent.

Collecting data across three weeks, and using the same question as in the first survey, allowed us to compare the answers to the closed question about change in productivity so that we could detect if there were any significant changes in productivity (RQ1) as people adapted to or found it harder working from home over time.

Data Analysis. For the quantitative data in the second survey, we present descriptive statistics about the selected benefits and their importance, and the challenges and their impact. We performed a simple subgroup analysis for management responsibility (people manager vs. individual contributor) and job role (software engineers vs. program manager). We did not perform a subgroup analysis for gender because gender information was only collected in the final week of the survey.

We also considered the association of the benefits (Q15*), challenges (Q17*), and suggested improvements (Q22*) with the reported changes in productivity (Q13). We used Wilcoxon Mann Whitney and Fisher Exact Value tests [26] to check for statistically significant differences. To reduce false discoveries due to multiple hypothesis testing, p-values were adjusted with the Benjamini Hochberg correction [27] ; we recorded the original p-values for each hypothesis test and then computed the adjusted p-values using the p.adjust() function in R.

To identify benefits and challenges that are most strongly related to productivity and analyze whether there are important interactions between them, we used a least absolute shrinkage and selection operator (Lasso) analysis [28] . Lasso is a regression analysis method designed to select the most important explanatory variables from a set of variables. It works well in the presence of many explanatory variables. Lasso analysis has been applied in economics and finance [29, 30] and was found to improve prediction and interpretability of models. It was also found to select sometimes neglected variables [31, 32] .

We ran two analyses: for benefits (Q15*) and for challenges (Q17*). In both analyses, the dependent variable was whether a participant reported that productivity stayed the same/increased/significantly increased. Since the outcome is binary, we ran a logit analysis. For potential explanatory variables, we included direct effects and interactions for whether a respondent reported each benefit as important or very important (for the analysis of Q15*) or each challenge as a major issue (for Q17*). With the interactions, we had 240 potential explanatory variables for the benefits regression (420 for challenges). Even with a relatively large survey, that is too many to meaningfully include in a single analysis. The Lasso approach selects the most important variables by first maximizing the likelihood with a penalty for the absolute value of the coefficients. The penalty makes it so a variable is only given a non-zero coefficient if it explains enough of the variation in the dependent variable to outweigh the penalty. As a standard practice, we ran the analysis with a range of penalty parameters and used the maximum penalty parameter that gets a mean cross-validation error within one standard error of the minimum. Then, using the variables with non-zero coefficients given that penalty parameter, 5 we ran a standard logit analysis (logistic regression) to get coefficients that are not shrunk by the penalty procedure. We report the logit coefficients as marginal effects for interpretability. Since all variables in the logit analysis are dichotomous, the marginal effect reports the difference between changing the variable from 0 (absent) to 1 (present).

We describe the threats to validity and limitations of our study.

External Validity. Single-case empirical studies have historically been shown to contribute to scientific discovery [33] and have delivered insights in the social sciences [34, pp. 95 ]. The company we studied employs tens of thousands of software engineers that work on diverse products across many domains (operating systems, databases, cloud software, software tools, to productivity software), and use many tools and diverse processes. By studying a single company, we were able to control for many factors that otherwise may influence the employee experience during the pandemic, such as the region and the company's response to COVID-19. We do not claim that our results are representative of the views of all software engineers and companies in general.

Likewise, we do not include the demographics of race, gender, nor age in this work. This is an opportunity for future work to highlight the experiences of marginalized individuals who have been disproportionately impacted by the pandemic [35] [36] [37] .

It is important to keep in mind that remote work during a pandemic is not the same as regular remote work. While some findings will be specific to the pandemic (e.g., lack of childcare as a challenge because schools and day cares were closed), not all findings are specific to the pandemic. We discuss implications of our work for the future of remote work in Section 8.3.

As with any survey, there may have been non-response bias, i.e., the results might not be representative of the population because the participants disproportionately possess certain traits which affect the outcome. In addition, our survey was advertised as a "Work From Home Survey" and therefore could have been subject to self-selection bias, e.g., participants might have been more likely to participate in the survey if they were more strongly affected by work from home (negatively or positively). To reduce non-response and self-selection bias, we kept the surveys as short as possible, were transparent about the survey length (single-page survey), provided an incentive to participate (raffle), and kept the surveys anonymous.

Construct Validity. Although we could have used existing surveys that inquire about developer productivity and their experience working from home (see Section 2), we recognized from early reports that new factors specific to the pandemic were playing a bigger role in developer productivity and experience (such as not having child care, and the stress of the pandemic). Nothing like this has happened before and so we used an initial open-ended survey to study the factors emerging during this phenomenon, and then conducted a second survey to quantify the change in productivity and frequency/impact of challenges/benefits encountered.

Measuring perceived productivity with a single question has limitations. We consider selfreported changes in their perceived productivity, as objective productivity metrics in software development remain elusive due to the creative and collaborative nature of development work. We chose single response items to keep the survey length reasonable because shorter questionnaires have been found to receive higher response rates [38] . Self-rated, single-item response items for productivity have also been found to correlate with objective productivity measures for software engineers [18] . Later in the paper (in Section 8) we show and discuss an analysis of productivity at the company data by by mining software data collected through the engineering systems to triangulate our findings from the surveys.

Internal Validity. There are additional biases from our survey. Respondents may have shared what they wanted management to hear (in terms of suggested improvements in particular, but also for benefits and challenges), and the wording of our questions may have led to certain responses. Furthermore, our analysis may have been biased by our own experiences (as we were also working from home) and most of us shared the same employer as the respondents. We tried to offset these by using additional coders and by having experts review our survey from outside our team.

Finally, our study involved a single research method (a survey). However, we tried to offset this limitation by considering the findings from other ongoing studies at the company using different methods (including objective quantitative analysis of system data).

There was no overlap between the samples in both surveys to reduce survey fatigue among participants. As a result, the second survey was limited in capturing how things had changed since the first survey (although some extrapolation is possible). To study the long-term impact of work from home, other research designs have used diary studies [39] . The benefit of survey designs like ours is that they require less commitment and time from participants, and as a result, have more participants, which makes it easier to observe statistical effects.

Conclusion Validity. The Lasso approach is designed to select the variables with the strongest relationship to the outcome and uses cross-validation to avoid over-fitting, but it is still possible to find a statistically significant relationship when there is none (since p-values below .01 are still greater than zero). There may also be other relationships in the data that are important but not captured in the results because the sample size is not large enough or there is too much noise Table 1 . Changes in self-reported productivity based on the responses to the question "Compared to working in office, how has your productivity changed?" (Q13, Q13*) The responses to Survey 2 are further broken down by week: April 22-25 (W1), April 26-May 2 (W2), and May 3-9 (W3).

Survey 2 W1 W2 W3

Significantly more productive 8% 11% 13% 10% 10% More productive 22% 26% 23% 28% 26%

About the same 32% 32% 31% 30% 34%

Less productive 32% 26% 26% 26% 24% Significantly less productive 6% 6% 7% 6% 6%

relative to the effect size. The relationships we uncover are all correlational; we cannot distinguish causal effects using the survey.

In this section, we address the research question "How has engineers' self-reported productivity changed since WFH?" (RQ1). In both surveys, we asked participants how their productivity has changed compared to working in office. The results are shown in Table 1 .

• In both surveys, the majority of participants reported that their productivity had not changed or had even improved (62%-68%). However, a substantial portion of participants (32%-38%) reported that they were less productive • The percentage of people reporting to be less productive consistently dropped over the study period: from 38% in Survey 1 to 30% in the last week (W3) of Survey 2. This suggests that some (but not all) people found ways to restore their productivity to their original levels.

Ralph et al. [40] found evidence that developers have lower perceived productivity while working from home during the COVID-19 pandemic. Our findings also support this result but offer a more nuanced view: initially in Survey 1, more people reported lower productivity (38%) than higher productivity (30%); however, this later changed in Survey 2, when more people reported higher productivity (37%) than lower productivity (32%). Similar observations have been made by Forsgren [41] and Bao et al. [42] . We will discuss these papers in more detail in the related work (Section 9).

We performed a simple subgroup analysis of change in productivity across both surveys combined. We found no statistically significant difference between people managers (32% more productive) and individual contributors (35%). However, the difference between software engineers (31%) and program managers (40%) was statistically significant. At Microsoft, program managers define and design products and services, collaborating with many different stakeholders, including engineering teams. Their responsibilities span the entire product/service life cycle [43, 44] .

It is important to recognize that at an individual level, people are affected differently by work from home: productivity can decrease, stay the same, or improve depending on a variety of challenges and benefits. In the next two sections, we discuss the higher level themes in terms of the challenges and benefits experienced that emerged from our qualitative analysis of the additional survey questions.

In this section, we address the research question "What are the benefits engineers experience when working from home? How have these benefits affected productivity since WFH?" (RQ2). To identify the benefits, we analyzed the responses to the open-ended questions in Survey 1, and to quantify the association with productivity, we used the responses to Survey 2.

Respondents identified a wide range of benefits working from home. In this section, we discuss themes that were frequently mentioned in Survey 1 or later emerged as significant in the productivity analysis based on Survey 2.

Commute. Most respondents pointed out benefits related to their well-being. In particular, over half of the participants mentioned the lack of commute as a positive aspect of working from home.

No commute worries. I can focus on my job instead of checking the traffic reports and worrying. (P908)

On average, respondents reported a daily, round trip commute of 67 minutes (median 60 minutes), which is comparable to other people living in the Puget Sound area [45] . In extreme cases, respondents mentioned a daily commute of more than four hours. "I love saving the 14-18hrs/week of commuting and being home when my daughter gets home from school. " (P791)

The time saved on commuting led to a wide range of other benefits. It allowed respondents to work more (Work hours) but also spend time with their families (Family, children, pets) and focus on healthy activities such as physical exercise and more sleep (Healthy habits).

More time with Kids due to reduced commute times. Can workout more when the sun is out. (P48)

Removes the stress of a 30 minute commute to work and saves that wasted time for either more sleep or more work. Let's me sleep in a bit and work in a more rested state than if I got up earlier due to the commute. (P1127)

My health has improved. I am getting more sleep and rest, since there is no commute involved. I can work at my own pace, in a more comfortable environment (my home).

Schedule flexibility. Another frequently mentioned benefit was schedule flexibility. Working from home allowed respondents to plan their day more freely and work at different times than they did before. Activities that required respondents to be physically present at home at certain times were easier to do, for example, accepting deliveries, laundry, or physical exercise.

I can dictate my own schedule, take breaks when I want to, prioritize self-care throughout the day (running errands, taking a quick nap, exercising, calling family + friends) (P267)

Working from home gives me more flexibility to do things when I take 10 minutes break twice a day. I can finish up loading the laundry or dishwasher, I can lie down to straighten up my back if I wish without compromising the productivity. (P833) I can sometimes take a break and do some chores that are often more difficult to do later in the day (e.g. laundry, where all the machines are usually being used in my building).

The flexibility in schedule often led to seamless integration of work and life, where participants completed chores while waiting for builds or during short work breaks.

Sometimes an idea clicks in the middle of the night, and with wfh, implementing that idea is literally 2 seconds away (P593) Being able to quickly task switch for other non-work related tasks and quickly return to work. (P289) I feel like I can solve problems more easily since I don't feel constrained by a clock. I can start a job and cook dinner, then come back to check the job results while I leave something in the oven or when I'm done cooking. (P215)

Focus and Interruptions and Distractions were the second and third most frequent codes. Fewer distractions and interruptions at home made it easier for respondents to concentrate and focus for longer periods of time. Having fewer meetings, the ability to continue work during remote meetings, a quiet work environment, and more control over interruptions further contributed to "Undistracted focus time. " (P1263) for the respondents. Less distraction from others, especially coming from an open office scenario. Teams meetings for some meetings where very little participation is required lets you continue to work while "attending" and listening in, which is better than being idle in a conference room. (P691)

There were plenty of distractions at work too with my office set up. So obviously colleagues interrupting me is easier to manage now as you have to answer the teams chat or email or phone call vs. someone just coming into your office or dragging you to a meeting. It feels more in my control now. (P223) Work Environment. Several respondents pointed out that they preferred their environment at home compared to the office, for example, because the environment is more quiet, more spacious, has a window, more sunlight, or closer bathrooms. Respondents also liked having more privacy and more control at home, for example, over room temperature or decorations, which is more difficult in a shared environment. My apartment is much less dreary than the office. Visuals and decorations are not just about "looking cool", but have a deep effect on how well/fast/creatively/how long I can think. In contrast the office has mostly blank white walls, no windows, and every hallway in the building is identical. Just being in the monotony of that environment is mentally draining, which drops my productivity. Participants also enjoyed the ability to wear comfortable clothes, listen to loud music, and work at their own pace (Personal comfort) in their home work environments.

I'm in the comfort of my own home (I can wear lounge clothing, play music, etc). (P325) Table 2 . Benefits in Survey 2. The column Distribution refers to the distribution of responses that (from left to right) do not experience a benefit (light gray ), experience a benefit and consider the benefit as unimportant (gray ), important (dark gray ) or very important (darker gray ). The column Prevalence indicates the percentage of respondents who experienced the benefit ( ) while the column Importance describes the percentage of participants who indicated this benefit to be important or very important ( percentage of with respect to ). The column Productivity Unchanged or Increased reports the percentage of respondents who reported that productivity stayed "about the same" or increased ("more productive", "significantly more productive") for respondents who did not experience the benefit vs. respondents who did experience the benefit; statistically significant differences ( < .01, with Benjamini-Hochberg correction [27] ) are indicated with an asterisk (*). The benefits are sorted and numbered in descending order by column Prevalence. Family, Children, and Pets. Respondents liked being close to their families, children, and pets. They appreciated being able to see them during breaks or lunch and that they could take care of family needs when required.

Being at home with family, especially with a toddler and baby. I get to spend a bit of time each day every few hours to just say hi and be around them, even if just for a few brief minutes. 

From the themes that emerged in Survey 1, we inferred a list of 15 characteristic benefits (B1..B15) that we included in Survey 2. The results are displayed in Table 2 . The benefits are sorted and numbered in descending order of frequency.

Frequency and Importance of Benefits. We make the following observations from Table 2 about the prevalence and importance of benefits:

• The Prevalence column shows the frequency of the benefits. The most frequently reported benefits (B1-B5) were less time on commute (96%), spending less money (84%), flexible work hours (81%), closer to family (81%), and more comfortable clothing (80%). • The Importance column shows the percentage of participants who indicated a benefit to be important or very important if they experienced it. Almost all benefits were rated as important by the majority of participants who experienced them. The benefits most frequently rated as important were better work life balance (95%, B11), better focus time (93%, B7), and more physical activity (91%, B15). The benefits less frequently rated as important were more comfortable clothing (48%, B5), more breaks (64%, B10), and spending less money (66%, B2).

Relation between Benefits and Productivity. The Delta column of Table 2 shows the difference between the percentage of respondents who reported that their productivity stayed "about the same" or increased ("more productive", "significantly more productive") for those who did not experience the benefit vs. those who did experience the benefit. We make the following observations:

• Almost all benefits had a positive delta on productivity change. This means that, on average, respondents who experienced a benefit also reported being more productive when working from home. The delta was only negative for the benefit more breaks (-1.3 percentage points, B10), but the difference was not statistically significant. • Not all benefits had a statistically significant productivity delta: there was no statistical difference in productivity change for the benefits less time on commute (B1) and more breaks (B10). • The benefits with the highest productivity delta were better focus time (+42.6, B7), less distractions or interruptions (+35.4, B8), and better work environment (+34.1, B12). All of these have been found to be significant predictors of productivity in the past [2, 15, 17, 18] .

Subgroup analysis. We compared the following subgroups to identify differential characteristics in the survey data. Lasso Analysis. We ran a Lasso analysis to see which benefits were most strongly associated with productivity change in a combined model and to check whether interactions between the benefits matter (details in Section 3.2). The marginal effects of the logit analysis with the Lasso-selected variables, shown in Table 3 , are similar to the pairwise relationships. The likelihood that people's productivity stayed the same or improved was increased by the benefits of better focus time (+18.9%) Table 3 . Results from the Lasso logit analysis. The dependent variable was whether a participant reported that productivity stayed the same or increased. The explanatory variables are the direct effects and interactions for experienced benefits that were considered as important or very important. The logit coefficients are converted to marginal effects for interpretability. The marginal effect is the difference between changing the variable from 0 (absent) to 1 (present). The overall mean is the percentage of participants who reported that productivity stayed the same or increased.

Marginal Effect Std. Error and less distractions or interruptions (+10.6%). The coefficient for better work environment at home was not statistically significant, however, the interaction between better work environment and less time spent on commute (+15.7%) was. People who reported benefiting from both less time spent on commute and more time to complete their work were also more likely to report that their productivity remained the same or increased (+12.2%). As were people who benefited from both less distractions or interruptions and closer to family (+8.9%).

In this section, we address the research question "What are the challenges engineers face when working from home? How have these challenges affected productivity since WFH?" (RQ3). To identify the challenges, we analyzed the responses to the open-ended questions in Survey 1, and to quantify the association with productivity, we used the responses to Survey 2.

The survey respondents shared a wide range of challenges and many respondents indicated they experienced multiple challenges working from home. In this section, we discuss challenges that were frequently mentioned in Survey 1 or later emerged as significant in the productivity analysis based on Survey 2.

Connectivity. Of all the challenges, problems with connectivity was the most frequent challenge shared. This included access to remote desktops, special access workstations, and internet bandwidth.

Respondents mentioned they experienced slow speeds due to a high number of users on their internet connections:

Periodic internet disruption due to wi-fi router and modem resetting due to 2 VPN connections for my wife & I, and our kids being online for school work. (P1013) Internet connectivity speed went beyond one's home internet and became a challenge when a colleague's internet connection was not as resilient.

Remote desktop connectivity issues. Coworkers with spotty internet quality are hard to meet with. (P691) Respondents also described workarounds they used to rectify their connectivity issues:

The VPN / Remote tools are not great, crashes often. I especially dont want to Intune my personal device, so working remotely have been challenging with the redmondts gateway down more than 50% of the time, and the WVD features crashing / disconnecting / not allowing correct alt-tab etc. (P37) Although many attempted to find a resolution, at times the connectivity issues they experienced felt out of their control.

Family, Children, Pets. One of the most frequent challenges respondents shared concerned proximity to family life. Being physically co-located with family members, housemates, children, or pets encouraged some to change their work habits:

Staying focused, especially with young kids around. Normally I would only work from home for a few hours occasionally after the kids went to bed. That is the only time I currently feel like I can be productive. (P531) The additional interactions with family have even to be more mindful of supporting family "child care schedules" (267). Respondents also mentioned how there was an implicit expectation of being engaged that often felt at odds with work:

Family in the house means there is also expectations from them to spend time or help around. (P338) The challenge of being physically present with family but mentally focused on other tasks is an experience that was hard to resolve.

Communication Channels. Another frequent challenge that respondents reported having was with the channels they used to communicate with their team members. One issue with communication channels was the increased friction to get ahold of a colleague versus simply walking over to their office:

The hardest thing has been that standard communications/questions and general collaboration take about 2-3 times as long. Something I could just pop over to someone's office to ask now requires an online chat or an email and the response is much slower. (P267) Likewise, for some there was also a higher frequency of using instant messages which made some participants feel like they should be highly responsive at all times:

I also feel like there is no "down time" away from work. I constantly get emails/messages/asks and sometime I have to respond right away. (P384) Managing multiple communication channels and the expectation to be very responsive on many of these channels presents an additional layer of interactions that does not adapt to every respondent's working style.

Work Environment. During the pandemic, most respondents' work environments were their homes, however, the experience drew comparable challenges with in-office work settings.

Many respondents were also not prepared to work from home and improvised their work settings:

I did not own a desk and chair so currently improvising with dining table. Not sure if I want to invest in or have space for expensive home office equipment. I miss having multiple screens but do not have space at home to set up. (P930)

As respondents missed their work office settings, they found themselves under new constraints ranging from financial to square footage when trying to create a comparable home office setting.

Interruptions and Distractions. When software developers are working in the office, interruptions and distractions often come from colleagues stopping by their desks. In a remote work setting, respondents described the advantage and disadvantage of only being available online: Healthy Habits. When respondents described challenges with reduced physical activity, they often mentioned their movements between physical meeting locations which no longer happened:"Sitting for a long time is hard on the body. At work, I'm up and around, moving more. At home all meetings are online so I never (hardly) move.... " (P72). When respondents did find an opportunity to move, it was either only to the restroom or for more coffee so that they can sit down for longer periods of time:

Since I don't do my daily bike ride I sometimes feel I just sit the whole day, and only do very few steps to the toilet [and] coffee machine (P867)

The reduction in what participants referred to as healthy habits also affect their work-life routines.

Work-life Balance and Routine. Respondents described their work-life boundaries blurring outside of the typical eight-hour work day, running late into the evenings: "Unless I impose a strict regimen, I feel like I am working for a lot more hours sometimes way into the night -the line between home and work gets far more blurry. " (P104)

Respondents also reflected on routines they previously had to distinguish boundaries for that were now lost:

To find my time boundaries. Very easily you can end up working much more hours because you don't have the signals of "Time to leave the desk", you don't have the time to decompress your mind in traffic, for example. You just jump from personal to work tasks (and vice-versa) so fast. (P195) In summary, the lost transition time and lack of physical movement between work and home removed a boundary they had before.

In the next subsection of our paper, we report on the association of these challenge with the Survey 2 respondents' productivity. Table 4 . Challenges in Survey 2. The column Distribution shows the distribution of responses that do not experience a challenge (light gray ), experience this challenge as a minor issue (gray ), and experience this challenge as a major issue (dark gray ). The following column Prevalence indicates the percentage of respondents that experienced the challenge ( ), while the column Impact describes the percentage of participants that indicated this challenge presented a major issue (percentage of with respect to ). The column Productivity Unchanged or Increased reports the percentage of respondents who reported that productivity stayed "about the same" or increased ("more productive", "significantly more productive") for respondents who did not experience the challenge vs. respondents who did experience the challenge; statistically significant differences ( < .01, with Benjamini-Hochberg correction [27] ) are indicated with an asterisk (*). The challenges are sorted and numbered in descending order by the Prevalence column. 

From the themes that emerged in Survey 1, we inferred a list of 20 characteristic challenges (C1..C20) that we included in Survey 2. The results are displayed in Table 4 . The challenges are sorted and numbered in descending order of frequency.

Frequency and Impact of Challenges. We make the following observations from Table 4 about the prevalence and impact of challenges:

• The Prevalence column shows the frequency of the challenges. The most frequently reported challenges (C1-C5) were missing social interactions (83%), lack of work-life boundaries (78%), poor ergonomics (70%), less awareness of colleagues work (65%), and less physical activity (65%). • The Impact column shows the percentage of participants that indicated a challenge to be a major issue among the participants who experienced it. The challenges most frequently rated as impactful are lack of childcare (58%, C16), poor ergonomics (52%, C3), and less physical activity (51%, C5). The challenges less frequently rated as a major issue are friction with collaboration tools (22%, C14), lack of dining options (24%, C18), and being blocked waiting on others (28%, C16).

Relationships between Challenges and Productivity. The Delta column of Table 4 shows the difference between the percentage of respondents who reported that their productivity stayed "about the same" or increased ("more productive", "significantly more productive") for those who did not experience the challenge vs. those who did experience the challenge. We make the following observations:

• We found that all of the challenges were associated with lower productivity. For all but two out of the 20 challenges, the difference was statistically significant. Subgroup analysis. We compared the following subgroups to identify differential characteristics in the survey data.

• Lasso Analysis. The previous analysis shows the relationship between individual challenges and change in productivity but does not take into account the presence of multiple challenges and the interaction effects between two challenges, which can be particularly important for productivity. Therefore, we ran a Lasso analysis to see which challenges were most strongly associated with productivity change in a combined model, and to check whether interactions between the challenges matter. Table 5 shows the marginal effects from the logit analysis for the Lasso-selected variables (details on the method are in Section 3.2). People are substantially less likely to report that their productivity is the same or increased when they say that having more distractions and interruptions (-40.9%) or lack of motivation (-26.4%) were major issues; difficult to communicate with colleagues (-13.1%), connectivity problems (-11.9%), missing social interactions (-11.2%), poor home work environment (-8.0%), and less awareness of colleagues work (-6.2%) were also associated with a significantly lower probability of reporting unchanged or increased productivity.

The challenge less time spent to complete work was also selected by the Lasso algorithm, though the coefficient is not significant. However, when combined with a lack of childcare, the interaction between both challenges is associated with a substantial and significant lower probability (-34.0%) of reporting unchanged or increased productivity.

In this section, we address the research question "What recommendations should be made to companies whose engineers may wish to work from home?" (RQ4). To identify these improvements, we first analyzed the open-ended questions in Survey 1, and to identify the most-requested improvements, we used the responses to a closed question in Survey 2. Table 5 . Results from the Lasso logit analysis. The dependent variable was whether a participant reported that productivity stayed the same or increased. The explanatory variables are the direct effects and interactions for major challenges. The logit coefficients are converted to marginal effects for interpretability. The marginal effect is the difference between changing the variable from 0 (absent) to 1 (present). The overall mean is the percentage of participants who reported that productivity stayed the same or increased. Note: * p<0.1; * * p<0.05; * * * p<0.01

Respondents included several improvements that could be made to support their work from home experience. We briefly describe the most frequently mentioned improvements below. Survey 1 was sent within the first two weeks of employees working from home. Microsoft implemented many improvements throughout the pandemic to provide a better work from home experience to its employees, for example, employees facing school closures due to the pandemic were offered up to three months of paid parental leave [46] as well as resources and activities to support their physical, emotional and financial well-being.

Hardware. Although many respondents noted they were able to bring some equipment home or purchase additional devices, the most frequent possible improvement noted by respondents in the first survey was related to hardware. Insufficient hardware was also noted as a key challenge (as discussed in Section 6) and was mentioned by Ralph et al. in the Pandemic Programming study [40] .

I think that this is a reminder that when employing individuals that need certain equipment both at an office and at home, that we come up with a way to fully equip both locations simultaneously. (P994)

Many employees work with multiple monitors and sometimes multiple machines. In contrast, when working from home, employees are sometimes limited to just a laptop or a desktop with a single monitor. Large and/or multiple monitor setups have been found to improve productivity in information workers [47] . Indeed, the most frequently requested type of hardware we noted in the second survey was either larger or more monitors. Developers also asked for more powerful workstations (especially those working on laptops at home) as well as peripherals such as mice, keyboards, and noise-cancellation headphones.

Employees should be provided with equipment to make the experience better for everyone: webcams, good noise-cancelling headsets, etc. (P1174)

Employees were allowed to take any non-confidential property off campus to use at home after informing their manager.

Connectivity. Another frequently requested improvement was better Internet connectivity or improved VPN access from home, and many that mentioned this as an improvement in Survey 1 added that it was their biggest challenge to address: ("Network connectivity is the biggest pain" (P44). Some respondents indicated that ensuring good connectivity at home was expensive. Paying for upgraded home internet connectivity was also a key recommendation from the Pandemic Programming study [40] .

Perhaps subsidize higher speed internet connections, or speak with the broadband providers to get better service in our areas. With multiple people at home, there's only so much bandwidth to be shared. (P79)

Stipend/Budget for home office. The third most frequently selected item was to have a stipend to purchase equipment for a home office, with just slightly more engineers requesting this as an improvement if they experienced lower productivity. Several tech companies, including Microsoft, have been offering such a stipend to their employees [48] [49] [50] .

Provide employees a one time reasonable allowance to set up a home office such as sit/stand desk, ergonomic chair, allowance for monitors, budget for coffee/drinks/snacks. (P86) Improvements to communication tools. Once working from home, engineers were totally reliant on communication tools to collaborate with their colleagues and for meetings. Many noted specific improvements, including some that were engineering specific: Support whiteboard drawing, support multi desktop sharing from multiple people and sharing on split windows or on my local multiple screens. (P1277) Add a bunch of developer specific features. A simple example is how do you go around the room in standup and know that everyone got to talk. (P885)

Provide more ergonomic furniture. Many participants noted in the first survey that their furniture at home was not as ergonomic as their furniture at work (e.g., no standup desk, small desk space, less ergonomic keyboards etc.). To better support employees, Microsoft provides recommendations on how to setup physical workspaces in an ergonomic way and some stipends for the workspace.

Not being able to exercise as much was also an issue, as this respondent mentioned:

I also don't have the ability to stand up and work since my home desk doesn't move. This means that I'm sitting down even more every day, which also leads to back and neck pain and frustration. (P1048) A couple of respondents in our sample suggested treadmill desks as a possible improvement for WFH to address less of exercise.

For some respondents, working from home goes beyond furniture, as one participant noted:

If I am going to continue work from home, I need a new house/space from which I can work w/fewer distractions. (P1224) Support for remote work post-pandemic. In the first survey, many suggested that full or partial remote work should be supported after the pandemic, and that they appreciated the opportunity to work from home and experienced a variety of benefits (as discussed in Section 5). In particular, several mentioned that working at home, at least some of the time, would help them be more productive:

Allow more people to do this long-term after this current crisis ends; I feel I'm more productive for it, I'm contributing better to my team, and keeping more people home will help us meet our sustainability and environmental goals as a company. (P1297) Some provided concrete suggestions for supporting WFH long-term:

Every team should treat their tech stack as if they had to work from home at least 2 days a week. This way things such as VPN, workstations, deploy pipelines, local builds, etc are naturally able to support remote workers. This will lead us to hiring better remote talent and allow us to institute remote working policies during health or environmental changes that hinder the ability of some workers to be in the office. That's what we've done and it's made our org more productive during the WFH period. (P33)

Provide guidance for working from home. Many noted that curated guidance for working from home would benefit not just their own work but also the work of their colleagues. For example, on how to use different communication platforms, one respondent suggested:

Get some primers out so that people can feel comfortable in the space and know the use cases it's for. Lack of this knowledge keeps translating to inefficient use of email threads. (P669) Improving how knowledge is externalized is also more important when everyone is remote:

Encourage documentation as part of our culture. It's difficult to impossible to use libraries from within our org without directly talking to the repository owners. Now the only way to get information about these libraries is to send a message and hope they respond. (P212) Some respondents also noted that it isn't just about improving tools and processes for WFH, there is also a need for organizational guidance regarding maintaining a positive work-life balance:

Broad communication across the company to say "General work ends at 5:00 PM local unless business critical" as a way to force work/life balance now that work and life are in the same place. (P696) Microsoft continuously provided guidance, tips, and resources for employees working from home during the COVID-19 outbreak. In addition, communities were created to connect employees with colleagues around the world for tips and support on working from home.

Other improvements. Other improvements that were suggested by respondents were to improve and encourage social interactions within teams; be more understanding of WFH scenarios beyond the pandemic; encourage people to be more responsive; minimize the number of meetings; and guidance for managers to manage WFH employees.

From the themes that emerged in Survey 1, we inferred a list of 12 characteristic improvements (S1..S12) that we included in Survey 2, in which respondents could select up to three improvements. The results are displayed in Table 6 . The table shows the frequencies for how often an improvement was selected by all respondents (column "All"), by respondents who reported a decrease in productivity (column "Low"), and by respondents who reported an increase in productivity Table 6 . Improvements in Survey 2. Participants could select up to three items. Column "All" indicates the frequency the improvement was suggested among all respondents, "Low" the frequency among respondents who reported a decrease in productivity, and "High" the frequency among respondents who reported an increase in productivity. Differences between the frequency for "Low" and "High" that are statistically significant with < .01 after Benjamini-Hochberg correction [27] are labeled with an asterisk (*).

All Low High S1 Provide more/better hardware for home ( 14.6% 15.7% 15.0% S10 Encourage people to be more responsive 9.8% 8.9% 12.0% S11 Minimize the number of meetings 5.8% 7.6% 3.8% (*) S12 Give guidance to management on how to manage WFH employees 5.6% 5.9% 5.6%

(column "High"). The improvements are sorted and numbered in descending order of frequency by all respondents. We make the following observations:

• The most frequently selected improvement was to provide more/better hardware for home (41.6%), improve connectivity (41.5%), and provide a stipend for improving work from home environment (40.8%). • Several improvements were more frequently selected by respondents who experienced a decrease in their productivity: 6 improve connectivity (45.8% vs. 35.5%, S2), make improvements to communication tools (36.9% vs. 29.3%, S4), provide guidance for successfully working from home (23.7% vs. 16.7%, S7), and minimize the number of meetings (7.6% vs. 3.8%, S11). • Several improvements were more frequently selected by respondents who experienced an increase in their productivity. The improvement provide ergonomic furniture was more than twice as likely to be selected (40.7% vs 17.2%, S5). This may be because more productive respondents were satisfied with other potentially pressing needs. The improvement support remote work better during normal circumstances was selected twice as frequently (30.7% vs. 15.1%, S6). This may be because these respondents' basic needs were met and they were focused on future needs and being able to continue to working from home.

Although the pandemic is an unusual-and hopefully uncommon-event in the lives of software engineers, the sudden work from home directive provides an opportunity to study what happens when engineers at a very large company are suddenly in a remote working condition with the rest of their team and the entire organization. Engineering work is similar to knowledge work in general, but engineers may require highly intense periods of focus work but also rely on tight collaboration to develop modern software. As such, the pandemic and the force to working from home provides an interesting opportunity to understand more about developer productivity, but also to find guidance for developers that work remotely or for developers that collaborate with remote team members. As many companies are anticipating supporting much more remote work in the future, some even declaring they will be entirely remote, the findings from our study are important. However, our study is focused on the study of remote work during the pandemic. We found many overlapping factors from those found in studies conducted before the pandemic, such as control over their work, time to complete work, focus time and work life balance. But some new factors emerged that were specific to the pandemic context, such as commute time and reduced health risks.

Before discussing our findings further, we remind the reader that the context for this study is a large multi-national software company, and that our study focuses on engineers working in the US.

As we saw earlier in Tables 2 and 4, for some developers that previously worked in-office with their co-workers, certain factors that were described as a challenge by some, were described as a benefit by others. In the statistical analysis, we even found that for some factors the corresponding challenge was associated with statistically significant lower levels of productivity, while the corresponding benefit was associated with statistically significant higher levels of productivity. Examples are the ability to focus (B7, C11) and the home work environment (B12, C17).

These dichotomous experiences are expected, which we see from extensive research around the world on people's experiences of lock-down or social distancing during the pandemic. Furthermore, divergent experiences are expected given the varied family life, living conditions/location, job characteristics, and personality characteristics of our studied population. For employers and managers, knowing that "one size does not fit all" is critically important for the future of software development work.

The main divergent factors were:

• Ability to focus. The number and nature of interruptions and distractions varied considerably, with some reporting more focus time at home and higher levels of productivity, and others having less focus time, especially those facing interruptions with family members at home and lower productivity. But even for those that appreciated fewer "randomizations" from colleagues since working from home, at the same time, they missed the knowledge and awareness they gleaned from these and other informal interactions. • Work autonomy and motivation. Increased autonomy and control over tasks and timing increased motivation for some, but reduced motivation for others and also their reported productivity. • Work environment. Some appreciated the novelty of working from home, having natural light and more comfort at home, which was associated with higher productivity, while others missed their office work environment with extra amenities such as the cafeteria and reported lower levels of productivity. • Meetings. Some felt there were too many meetings since WFH, and they missed face-to-face social cues and whiteboards, but others liked the shorter meetings and the associated artifacts they could refer to later. We observed that people managers and product managers reported more frequently having too many meetings. • Work-life balance. Many appreciated having more time due to no commute and being able to use that for extra time with family or to do personal chores or for self care, but others also found it difficult to disconnect from work, worked too many hours and did not have healthy habits since working from home. People managers reported more frequently a lack of work-life balance and benefited less from flexible work hours. • Childcare needs. Having children with a need for childcare led to some surprising dichotomous experiences. From our analysis of the data from Survey 2, we saw that employees with children who had no difficulty handling childcare less frequently reported a drop in productivity (20%) than employees without children (30%). However, for employees with children who had difficulties handling childcare, 40% reported a drop in productivity. We also found that employees who previously had children in school or childcare were MORE likely to indicate these major challenges: lack of childcare, more distractions or interruptions, less time to complete my work, and were LESS likely to indicate lack of motivation as a challenge.

In the subgroup analysis, people managers reported more frequently the lack of childcare as a challenge. • Social connections. Having fewer social connections was reported as a challenge for many, but for others, a minority, they felt MORE connected to their team and appreciated online activities such as standups, social lunches, games, daily check-ins etc.

In this paper, we have presented insights based on surveys with self-reported data through the lens of individual productivity. This is just one of many possible analyses on how remote work affects productivity. To illustrate an alternative perspective, we show an analysis of productivity by mining software data collected through the company's engineering systems. For this section, we analyze trends across Microsoft to see how developer productivity changed during work from home. We compared the pull request counts during the pandemic (March/April 2020) with prior historical values during comparable periods of the fiscal year (March/April 2018 and March/April 2019).

As the Microsoft engineering workforce grew in numbers since 2018, we normalized the pull request counts by the number of engineers to control for the growth. The number of pull requests opened per developer during the pandemic increased compared to previous years: 4.4% compared to 2019 and +3.4% compared to 2018. Similarly, the number of pull requests closed per developer also increased: +4.0% compared to 2019 and +2.1% compared to 2018.

Different parts of the world went into "lockdown" at different times in March and April 2020. Microsoft has development teams spread across the world, spanning all continents. In order to control for geographic differences, we further analyzed the pull request data for the three main Microsoft regions separately: Puget Sound in North America; ASIA, which includes China, India and Japan; and EMEA, which includes UK, Netherlands, Germany, France and Scandinavian countries. We observed that there was no discernible drop in the number of pull requests for all three regions, including when normalized by engineer count.

Overall we observe that there is no clear or significant drop (at statistically significant levels) in terms of the number of pull requests and the pull requests per developer. This data analysis suggests that the pandemic has not significantly influenced productivity at the company level. While this particular analysis shows that productivity has been stable or has slightly improved on average, it is important to recognize that just focusing on the company level alone loses the nuance of how individual people are affected differently. This highlights the need to run a family of experiments that investigate work from home using different types of data and methodologies such as diary studies [51] and workplace analytics [52] .

The pandemic has been a major disruption and will change how engineers work in the future and beyond the pandemic. Of course, this is not unique to software development and this disruption is visible in other professions and for many kinds of knowledge workers. Many companies, software companies in particular, have announced either a shift to full remote work, or to partial remote work 7 in a hybrid fashion where more developers may be allowed or encouraged to work from home several days a week. Working remotely is already the norm for some companies, for example GitHub 8 , Automattic [53] , and many other highly successful open source systems have been designed, written and maintained by distributed developers, many of them volunteers [54] . Much can be learned from these existing success stories, but there is more to learn and many factors to consider in a future hybrid setting. An organization the size of Microsoft that has primarily relied on co-located work must now adapt and find new ways to work in this new hybrid world the pandemic has left behind 9 .

Remote work may suit many developers and projects, but there may be other aspects of software development that are negatively affected and the system engineering output data may not show those limitations. We already see signals from our survey that teams may face collaboration and communication challenges. In our first survey, one of our respondents noted that working at home does not provide the same information about the pulse of work but it could be addressed by tools:

Automate team trends and share across the team, it's difficult to determine the real focus the team is trying to solve without seeing people stressing behind their desks (or not). (P891) And individuals that are part of a team, may be concerned that a lot of the work they do (such as helping others) may not be visible to the entire team, as one noted:

My biggest fear is being "out of sight, out of mind" (P932) Some reported addressing team work challenges through daily stand-up calls over video, virtual coffee hours, and more impromptu meetings (which as we mentioned above, was greatly appreciated by some participants). But some development activities, such as long-term planning and creative aspects of development, may be affected differently as some early work indicates [55] . These aspects of development work need to be studied in a longitudinal fashion, especially if work becomes hybrid for some developers. Managers should also be studied as they may face additional stress working from home and managing a team. For example, it may be harder for managers to give feedback (an important factor for developer satisfaction and productivity [17] ) and maintain awareness of the well-being and productivity of their team members. A shift to hybrid remote work will also have some societal implications-our survey respondents recognized this and appreciated the positive effect on the environment less commuting may lead to.

There has been extensive research that compares distributed/remote development work with co-located work, as well as several other studies that have studied developer productivity and well-being during the pandemic. We summarize the key findings from these prior works below and compare findings across these studies.

Remote work was adopted by many large technology organizations long before the pandemic because of the advantages that working remotely provides to employees. Remote work provides workers with the opportunity to engage with a globally distributed team introducing a wide range of perspectives to the project. Another advantage is the flexibility of how to work. Specifically, workers have the autonomy over when to engage and disengage with colleagues, providing unique opportunities for deeper concentrated work [56] . Focused time to work is often challenging when colleagues face unscheduled interruptions [57] . Remote work also provides the flexibility of where to work, granting workers the ability to work from many parts of the world-which if well supported, can lead to distributed teams being just as effective as co-located teams [9] .

Despite the benefits with remote work, there are also several challenges that remote work presents for workers. For example, the ability to build trust with colleagues while working remotely is critical for collaboration [58] but can be harder to achieve. Close proximity and in-person work provides opportunities for unplanned interactions in-office that build trust. In contrast, interactions in remote settings must be intentional or they will affect the building of social capital across distributed teams [59] . In remote settings, there is a need for more devoted time, resources, communication channels, and events to foster relationships. Although remote work implies the ability to work from any location, working remotely has often been synonymous with working from home-which has its own set of challenges. Some of the challenges with working from home include supporting family members who may be sharing the same working space. For example, Heisman's interviews with remote workers at GitHub identifies how they have been able to take advantage of flexible work hours with use of support groups to support their children [60] .

It is important to note that there were a few multinational technology companies who supported a "remote-first" [61] work environment before the pandemic. Some of these companies have shared their best practices to support other organizations. For example, in 2017, Stack Overflow shared a blogpost about how their organization has created a successful remote work environment. In this post, they proclaim that the most important aspect that has contributed to their success is assigning an employee to be the main point of contact to respond to all remote work-related questions [62] . Their article further describes how effective it has been for someone in the organizations' leadership to advocate on behalf of remote workers.

Similarly, GitLab, an all-remote DevOps technology company, released an inaugural remote work report to reveal the state of distributed work and explore the future of remote work [63] . This timely report was released only days after King County employees were instructed to work from home (see Figure 1 ). The report shares research conducted by a third-party company that provides insights from over 3,000 remote workers across four countries in a variety of industries and roles. Some of the key takeaways from this report are all-remote work is surging, remote work can foster a better sense of work-life harmony, allowing remote work provides a hiring advantage, and that "remote≠alone", meaning that remote work does not have to mean workers are isolated.In response to COVID-19, GitLab produced a "Remote Work Playbook" [64] where they describe strategic tactics to help support their more than 1,200 remote workers across 67 countries feel more supported. It also describes what other now remote companies can do for their newly transitioned remote workers. The playbook outlines guidelines on how to align values with expectations, how to manage remote teams, how to identify tools for effective communication, and how to encourage a healthy remote work lifestyle.

In addition to previously remote companies sharing their insights about effective remote work, several studies have looked at the impact of the pandemic on developers now working from home. We summarize some of these studies below and briefly compare our findings. 9.2.1 Pandemic Programming: Developer Experiences and How Companies Can Help. Ralph et al. [40] conducted an online questionnaire with over 2,000 responses from developers around the world (the largest proportion of 22.7% were from Germany, followed by 16.4% from Russia, 12.2% from Brazil, and 4.4% from US). They aimed to understand how working at home during the COVID-19 pandemic affected developer well-being and productivity. The survey was run at the end of March 2020 and participants were primarily recruited through social media channels frequented by developers.

They found that developers' productivity and their well-being have suffered since working from home, and that well-being and productivity are closely related. Dealing with the pandemic and home office ergonomics affected well-being and productivity and that women/parents/people with disabilities may be disproportionately affected. Their study leads to several recommendations how companies can support their employees: pay for home internet, help with home equipment, pay attention to employee emotional well-being and assure them that their reduced productivity is expected and will not negatively affect their job. In a different study, Machado et al. studied the impact of the pandemic on developers in Brazil and likewise found negative implications on gender equality and on women in particular [65] . [41] . They considered both open source and private project data. They found the following key insights when comparing the first three months of 2020 to the same time period in 2019:

• Developer activity (pushes, pull requests, code review and commented issues) was mainly similar to or slightly increased compared to the previous year. • There was some disruption in the early days of work from home for enterprise projects but this quickly stabilized. • Developer work days were longer by up to an hour per day with more work on weekdays and on weekends. They suggest this could indicate a risk of burnout. • Collaboration had increased on open source projects (in terms of number of users and projects). Their ongoing study from an engineering system performance point of view indicates that developers have stayed productive throughout the pandemic. Their findings align with the findings we found from analyzing system data at Microsoft (see Section 8.2).

The Baidu Pandemic Study. Bao et al. studied the effect of the pandemic and working from home at one of the largest IT companies in China, Baidu [42] . They conducted a quantitative analysis of 139 developers' daily activities (over 138 working days). They found that working from home is associated with positive and negative changes in developer productivity in terms of the number of builds, commits and code reviews. They also considered the influence of different programming languages, project size/age/type, and considered individual developers.

They found working from home was associated with negative changes for large projects and has different effects for different developers. Their data suggests that developer productivity may be more stable working from home than working onsite (less variation in their levels of productivity).

They also considered data from individual developers working from home and before work from home. They found that for the majority (approx. 85%), their productivity is about the same, but different for others (some are more productive, some are less). They asked developers to share feedback on their WFH productivity. The benefits that more productive developers reported were: working from home is exciting and energizing; developers can focus with fewer disturbances; WFH decreases transportation costs and saves time; WFH increases flexibility of when to work and improves work-life balance. The challenges that developers with lower productivity included: more home demands; a need for self discipline; and decreased collaboration with others. For the developers that found no difference with WFH, they experienced no barriers to completing their work, they could keep track of their schedule using online scheduling tools, and they found conferencing tools were powerful and effective for screen sharing.

Working from Home during the Pandemic to Working from Anywhere: A Case Study. Smite et al. conducted a case study with a large international company with offices in Sweden, USA and the UK. They analyzed system data and conducted interviews to study how working from home during the pandemic impacted developer productivity, satisfaction and collaboration [66] . They found from their system data that work activity continued without significant interruptions, but they reported changes in their daily routines. They also report different benefits and challenges speaking to the Tale of Two cities theme, such as less pairing, more loneliness, higher burnout, more communication friction and more scheduled meetings, but they reported more focus time and better work-life balance. They conclude their paper with recommendations for the future of work from anywhere to improve work culture but also to balance individual and team productivity.

During the Pandemic. Russo et al. followed 200 developers across the world working in similar lockdown conditions to understand the impact of the lockdown on their work activity, as well as to understand any changes in their self-reported productivity and well-being over the course of several weeks during the pandemic [67, 68] . They found that developers spent relatively the same amount of time on different activities, but they spent less time in meetings and on breaks. They found no significant relations between productivity, well-being and working activities, suggesting that working from home by itself does not present a significant challenge for organizations or developers, but introverts were potentially more impacted by the lockdown.

9.2.6 Other Pandemic Studies at Microsoft. At the same time that this study was conducted at Microsoft, several other studies (using a variety of methods and focusing on different populations) were also ongoing in parallel. An extensive report by Teevan et al. [69] summarizes the findings from this and other studies. For example, one notable study of developers at Microsoft that was done in parallel to our study, by Butler et al., describes the use of a diary study to investigate not just challenges but also the gratitude felt by developers over the course of the first 10 weeks of the WFH directive [39] . Some challenges reported by developers in this diary study included too many meetings, feeling overworked, and challenges managing their mental and physical health. However, these developers also reported feeling gratitude for their families, their job, increased work flexibility and their team. These authors report how insights from the diaries shape recommendations to improve developer work from home experiences.

We also conducted a followup survey study at Microsoft where we considered in more depth the impact of work from home on team productivity [70] . Similarly, we found that for some developers, team productivity had gone up, while other developers reported that their team productivity was lower. Some of the important factors that were associated with these changes include ability to brainstorm with colleagues, difficulties communicating with team members, and their satisfaction with social activities. Another follow-on study to the surveys we describe in this paper was conducted by Wang et al. [71] to study the impact of returning to the office in China when the country started to recover from the pandemic. They found workers preferred a hybrid style of working as many of the workers they studied were comfortable working at home after the pandemic and were used to using communication technology to do so.

In summary, the work from home studies conducted at Microsoft and across the world agree on a tale of two cities theme, where for some developers they did very well, but others did not, while many adapted over time. The studies led to important insights and recommendations that can shape developer work into a hopefully post-pandemic world.

The COVID-19 pandemic has been and continues to be a worldwide human and economic disaster, with many repercussions that are already evident, but with other effects that we can't even yet imagine. One thing is clear, a return to business as it was before is unlikely, and many predict that the future of development work is likely to be either fully remote, or for many, some form of hybrid work. Thus, it is critical to understand what has worked well and what has not gone well with remote work. We recognize that our study is only a start in understanding the implications of the pandemic on the software developer. In particular, new models of hybrid work are likely to lead to new challenges and benefits over remote work.

Our study reveals a "tale of two cities"-even in a company that has support in place for its developers and remote work-and delivers not just quantitative insights (as we saw from several studies) on how certain factors may be associated with higher and lower productivity, but also deeper insights into the narratives from these differential experiences. The improvements our study participants recommended shine some light on how organizations (and managers) may support their developers, and we hope that the lessons learned from our study and other studies of development work during the pandemic will help others recognize and react to the disruptive changes we see unfolding in our industry. Table 7 . Counts and Ranks of the Codes in Survey 1. The columns under Counts indicate the frequency of each code within questions "Please share details about your answer to the previous question on how your productivity has changed." (Q14), "What is good about working from home?" (Q15), "What is bad about working from home?" (Q16), "What challenges have you encountered working from home?" (Q17), and all four questions combined (Total). The columns under Ranks indicate the rank of each code with respect to the other codes for Q14, Q15, Q16, Q17, and all four questions combined (Total). The most frequent code is #1.

Ranks ( 

What it's like to lead a 100-person engineering team during the pandemic

Software developers' perceptions of productivity

Collaboration in distributed software development

How social and communication channels shape and challenge a participatory culture in software development

Rapid software development through team collocation

Creativity in agile systems development: a literature review, " in Information systems-Creativity and innovation in small and medium-sized enterprises

Shared mental models, familiarity, and coordination: A multi-method study of distributed software teams

A comparative empirical study of communication in distributed and collocated development teams

Does distributed development affect software quality? an empirical case study of windows vista

Use of collaborative technologies and knowledge sharing in co-located and distributed teams: Towards the 24-h knowledge factory

A systematic review of productivity factors in software development

Impact of valid selection procedures on work-force productivity

Why We Should Not Measure Productivity

Software developers' perceptions of productivity

The effect of work environments on productivity and satisfaction of software engineers

What makes a great manager of software engineers?

Towards a theory of software developer job satisfaction and perceived productivity

What predicts software developers' productivity?

An interactive map of the us cities and states still under lockdown -and those that are reopening

Kurt delbene's march 4 guidance to king county employees

Amazon extends work from home policy until january

Google will let employees work from home until at least next summer

Appendix to a tale of two cities: Software developers working from home during the covid-19 pandemic

Conducting on-line surveys in software engineering

Improving developer participation rates in surveys

Testing statistical hypotheses

Controlling the false discovery rate: a practical and powerful approach to multiple testing

Regression shrinkage and selection via the lasso

The welfare effects of nudges: A case study of energy use social comparisons

Sources of geographic variation in health care: Evidence from patient migration

Variable selection and corporate bankruptcy forecasts

Catching gazelles with a lasso: Big data techniques for the prediction of high-growth firms

Five misunderstandings about case-study research

The social science encyclopedia. Routledge

Health equity considerations and racial and ethnic minority groups

Women, caregiving, and covid-19

Struggle with multiple pandemics: Women, the elderly and asian ethnic minorities during the covid-19 pandemic

Response rate and response quality of internet-based surveys: An experimental study

Challenges and Gratitude: A Diary Study of Software Engineers Working From Home During Covid-19 Pandemic

Pandemic programming: How covid-19 affects software developers and how their organizations can help

Octoverse spotlight: An analysis of developer productivity, work cadence, and collaboration in the early days of covid-19

How does working from home affect developer productivity? -a case study of baidu during covid-19 pandemic

What do microsoft program managers do? (and products at microsoft)

Pm at microsoft

Seattle ranks 2nd worst commute time in us, report finds

Microsoft offers 12 weeks of paid parental leave due to pandemic

Toward characterizing the productivity benefits of very large displays

Shopify is giving employees a $1,000 stipend to buy supplies while they work from home during coronavirus pandemic

Playstation employees working from home until 30th april, will receive full pay and equipment budget

Facebook will give employees $1,000 bonuses to support remote work

Challenges and gratitude: A diary study of software engineers working from home during covid-19 pandemic

Microsoft analyzed data on its newly remote workforce

The year without pants: WordPress. com and the future of work

The cathedral and the bazaar

Challenges and gratitude: A diary study of software engineers working from home during covid-19 pandemic

How remote work can foster a more inclusive environment for transgender developers

Office productivity: a theoretical framework

Effects of four computer-mediated communications channels on trust development

Here or there? how configuration of transnational teams impacts social capital

Remote work: How parents are adapting and working during covid-19

What it means to be a remote-first company

Making remote work: Behind the scenes at stack overflow

The remote work report by gitlab: The future of work is remote

The remote playbook from the largest all-remote company in the world

Gendered experiences of software engineers during the covid-19 crisis

From forced working-from-home to working-from-anywhere: Two revolutions in telework

The daily life of software engineers during the covid-19 pandemic

Predictors of well-being and productivity among software professionals during the covid-19 pandemic-a longitudinal study

The new future of work

how was your weekend?" software development teams working from home during covid-19

Returning to the office during the covid-19 pandemic recovery: Early indicators from china

This includes the difficulty of managing, forming, or maintaining informal and team relationships as well as feeling isolated and missing social connections

Communication Channels: Discussions of the use of various communication channels such as Teams chats and calls, Email, instant messaging, including comments about them such as using too many tools, difficulty of use, and benefits of different tools. In addition, this includes comparisons of tools to working in office such as the lack of in

Communication Gaps: Challenges around communication such as it being difficult to connect with particular people (for example, because schedules are more flexible), hard to communicate, missing communication, and miscommunication. This also includes lack of awareness of what others are working on

Formal Communication: Formal communication such as scheduled chats that are work related

Informal Communication: Unscheduled, informal, or ad-hoc communication that is work related. This includes the inability to drop by someone's office or run into someone in the break room as well as the use of tools (e.g. Teams) for frequent

Well-being Breaks: Taking or needing more or less breaks (including meal breaks or walks); Reasons for less breaks such as having meetings are back to back

diet, explicitly saying being "healthy", physical activity (working out), walking between meetings; too much time on the computer

Mental Health: work related stress; personal stress, anxiety; burnout; fatigue; loneliness No Commute: The impact of not having a commute. This includes benefits such as less wasted time, but also negatives such as missing reading on the bus or calling relatives on the drive into work

Personal Comfort: The impact on working from home on personal comforts such as listening to music without headphones, wearing more comfortable clothes, or creating a more comfortable working space

Routine: Any mentions of routine. May include "missing a routine" such as "I miss having breakfast every morning" or the importance of maintaining a routine

Schedule Flexibility: The ability to and impact of working outside of the traditional "9-5" work day. This includes working outside of non-work hours as well as doing non-work related activities (e.g. laundry) during traditional working hours

Work Hours: Whether the number of hours worked during the day stayed the same, went up or went down. For instance, "I get the same amount done, but I'm working 12 hour days to do it

Work-Life Balance: Changes in boundaries between work and non-work life and the ability (or lack) to not let work concerns or responsibilities interfere with non-work activities

Focus: The impact of working from home on the ability to focus, or the impact of various factors on focus time Interruptions and Distractions: Interruptions or distractions (or the lack of them) when working from home, whether work related or not

Productivity: Discussion of perceived productivity and the impact of various factors on productivity Work Environment Connectivity: The challenges of connectivity such as internet speed and latency, "remoting in" to a machine at work to accomplish work, using secured machines, connections to remote machines going up and down, and the need to reboot machines remotely

Environment: Aspects of the physical work environment such as access to natural light, now having a window

The availability or absense of ergonomic furniture, often in comparison to the work office environment. Furniture: References to furniture that do not mention or allude to ergonomics. Items such as whiteboards or bookcases are also included

Hardware: Differences in displays (quantity and quality), and machines (also quantity and quality) as well as accessories such as mice, keyboards, headphones, and webcams

Acknowledgments. We thank the many respondents who answered our surveys and shared their experiences about working from home with us. We thank Victor Bahl, Peter Bergen, Surajit Chaudhuri, Jacek Czerwonka, Nicole Forsgren, Sam Guckenheimer, Brent Hecht, Donald Kossmann, Courtney Miller, Brendan Murphy, Madan Musuvathi, Paige Rodeghero, Jaime Teevan, and Scott Wadsworth for the great discussions about work from home and their support of this research. We also thank the entire Future of Remote Work v-team for the inspiring meetings. We thank Caroline Davis, Susan Hastings Tiscornia, Tanya Platt, and Kelly Sieben for the privacy and ethics review of the surveys. We thank the anonymous reviewers and Cassandra Petrachenko for suggestions on how to improve this paper.

 Breaks  19  21  17  20  77  14  14  16  15  19  Commute  46 229  5  0  280  6  1  27  31  2  Healthy Habits  7  27  48  21  103  22  12  5  13  14  Mental Health  17  45  22  9  93  15  8  14  23  15  Personal Comfort  3  48  0  7  58  28  7  30  26  23  Routine  7  0  12  4  23  22  26  22  27  28  Schedule Flexibility  21  71  1  25  118  13  4  28  11  11  Work Hours  25  16  45  20  106  12  15  8  15  13  Work-Life Balance  11  10  47  20  88  21  17  6  15  16   Work  Focus  83  80  12  25  200  2  3  22  11  5  Interruptions and Distractions 193  91  87  35  406  1  2  2  6  1  Motivation  5  0  12  10  27  27  26  22  22  27  Productivity  31  24  16  17  88  11  13  19  19  16   Work environment  Connectivity  34  0  34  99  167  10  26  12  1  7  Environment  39  63  26  57  185  7  5  13  4  6  Ergonomics  13  0  36  33  82  17  26  11  7  18  Furniture  16  2  17  33  68  16  23  16  7  21  Hardware  38  5  47  51  141  9  19  6  5  8