If you find during an online survey or after one closes that a good number of respondents didn’t answer a certain question, give extra thought to how to handle that question and its sparse answers. A question on a survey that returns lots of blanks suggests that it caused several types of problems. The respondents:
didn’t understand the question or the instructions, didn’t want to take the time to research or construct an answer, felt comfortable offering their views on a sensitive topic, worried about anonymity, had become fatigued because the survey was too long, answered a conditional question that failed to navigate to the next question, deemed the question to have insignificant value to them and others, did not know enough to answer the question, recognized that the question delved into non-disclosable proprietary knowledge, or didn’t feel the question was applicable to them (“You’re asking me, an administrative assistant, about Continuing Legal Education?
Many surveys seek ratings of a feature, a policy, an action, or some topic. Respondents typically choose from a limited scale, such as a Likert type question, to provide their evaluative rating. Analysis of those responses provides insights and powers many online surveys by law firms, law departments or legal vendors. If a compensation survey asks lawyers to assess their satisfaction with vacation time, pension plans, medical leave, CLE funding, and other benefits, that’s fine as far as it goes.
Some topics are so likely to trigger discomfort among respondents, and thus partial participation or honesty, that you don’t want to ask respondents to answer your question directly for themselves. Say you are interested in whether your firm or law department should create co-ed bathrooms. To ask in the questionnaire point blank, “Do you favor co-ed bathrooms?” could cause many participants to pause, feel uncomfortable, and not answer. Worse, they might abandon the survey at that point.
Here is a quote from Greentarget’s DEI survey report (2023, at page 18):
“From April through August of 2022, Greentarget and Zeughauser Group conducted qualitative and quantitative research to inform the 10th version of this study. The firms conducted 10 in-depth interviews and distributed individual surveys to 200 executives … as well as more than 30 law firm chief marketing officers.”
As the two organizations did, sponsors of surveys not infrequently interleave and bolster their survey results with comments made by people they interview.
The people who read your survey report may not be comfortable with purely numeric ratios or percentages. If you write that the odds of an unsatisfactory rating are six times the odds of a satisfactory rating, some readers will draw a blank. Or if you write that the ratio of lawyers to staff in a law firm is 1.4 to 1, it might pause them, puzzle them, and pass them by.