You might think that if you have skillfully created an online survey, all you need do is wait for the data to shower upon you so that you can start the fun: analysis, visualization, and interpretation. Hate to rain on your parade; you spend a major chunk of your project time cleaning the data before you can revel in the good stuff. No matter how carefully you design the survey, you must accept that the instrument, not to mention the lawyers, paralegals, legal-service vendors, outside counsel, and clients who take it, will confound you.
Let’s turn our attention now to various places where a survey project can help respondents answer questions. Artful instructions will also make the analyst’s job easier when it comes to cleaning, standardizing, and aggregating answers. If people can willy-nilly fill in as they see fit, the analyst will have a fit (yes, a willful pun) and the quality of the information collected will degrade. We will review these instruction methods by starting with the broadest scope and proceeding to the most targetted.
When your internet survey has a question that generates a string of numbers as long as the number of your respondents, such as if 100 law departments report their fees paid to outside counsel or hours committed to pro bono work, you have collected what analysts call a continuous variable. “Continuous” is statistician and programmer parlance for “numbers, with or without decimals.” The term says nothing about the order of the numbers or whether there are gaps or clusters.
We have already discussed invitations to potential participants sent by email. If the survey is conducted within a law firm or law department, all the invitees’ addresses are at hand. If, however, you are a vendor or consultant who wants to survey firms or departments, you want to reach as many as possible beyond a kernel email list that you have developed yourself. Let’s consider a number of techniques to increase your email reach.
Before you unleash your survey into the wild, delay long enough to allow at least one thoughtful person to attack the draft survey, as if he or she were an idiot. More specifically, ask a lawyer (or someone from the group to be surveyed) to review your survey work who is a peer or above so that they are not reluctant to criticize, ask tough questions, disagree, or find fault. Push them to answer all the questions as if they were dumb, unmindful, or a saboteur.