As soon as you have accumulated a few responses to your survey of law firms or law departments, you should download them and scrutinize them. It is far better to find and fix a problem early, or make a change that facilitates your analysis, than to proceed deep into the survey project. After all, we know what an ounce of prevention is worth. Here are several jagged edges to avoid.
Toward the end of most academic articles, the authors reflect on the research that spawned the article and speculate on how they might have done better. They also predict how they would like to extend that research in various ways. It is a self-critical postmortem, and that practice suggests what survey sponsors should also do.
In the four topics below, I suggest general areas of self-analysis and reflection. The animating idea is objectivity improvement, and tough love.
Collaboration has its pluses, but it also can unleash a pack of minuses. By the term collaboration, I mean if a law firm ties up with a trade group to co-launch a survey, or a publisher joins a legal vendor. Two parties agree to jointly produce a survey-based report. Below I outline issues that collaborators ought to address – the sheer number and thorniness of them suggests how good the pluses need to be to outweigh the minuses.
Assume you have a few hundred e-mail addresses of potential participants in your survey. Rather than blast them out in one large volley, you might send them in three waves of one-third each. Assuming you have adequate time to do so, experience has shown many benefits follow from a staged emailing campaign. What I have summarized below as the steps to take at the stages all aim to create a bandwagon effect.
When you produce a survey finding, you should want to understand that number’s reliability. If the research that produced the number were repeated several times (with everything else held constant), how much would the results vary?
Say you survey 100 general counsel and collect their total cash compensation. The median is $600,000, but you recognize that your survey produced only one sample of results from a larger population of all general counsel.