RESEARCH

E-mail or snail mail?

Randomized controlled trial on which works better for surveys

Rachelle Seguin, MA, MPA    Marshall Godwin, MD, MSC, CCFP    
Susan MacDonald, MD, CCFP    Marnie McCall, MD, CCFP

This article has been peer reviewed. Cet article a fait l’objet d’une évaluation externe. Can Fam Physician 2004;50:414-419.

Ms Seguin is a Research Associate at the Centre for Studies in Primary Care, Dr Godwin is an Associate Professor and Director of the Centre for Studies in Primary Care, and Dr MacDonald and Dr McCall are Assistant Professors, all in the Department of Family Medicine at Queen’s University in Kingston, Ont.

The most widespread, and generally accepted as the most useful, application of the Internet is e-mail. We found seven reports1-7 comparing e-mail with paper surveys between 1986 and 2000. Response rates to e-mail surveys ranged from 24% to 73% in these studies; response rates to paper surveys ranged from 41% to 82%. The studies comparing method of response generally found that rates were lower with e-mail surveys, but responses were more “honest” or less “socially acceptable,” perhaps even more extreme. Respondents to e-mail questionnaires made fewer errors in completing them, omitted fewer items, and provided longer responses to open-ended questions. In only two of the seven studies did e-mail achieve the 60% response rate thought to be good for self-administered questionnaires.8 In five of the seven studies, a minimum 60% response rate was achieved for paper questionnaires.

Only one study comparing e-mail with postal service6 surveyed physicians. The study was conducted on subscribers to a medical education listserv at the Michigan State University College of Human Medicine. Physicians comprised 40% of subscribers, and while more than a dozen countries were represented, 90% of those surveyed were from the United States and 6% from Canada. The e-mail response rate was 56%; the post response rate was 77%. The authors determined that e-mail was less costly and had a faster return rate, and that e-mail respondents were more likely to provide written comments. The sample size for the study was small, just 100 in each group, and all subjects in the study used e-mail.

We report on a similar, though larger, study comparing e-mail with postal distribution of a survey to physicians. This is the first study of its kind of family physicians in Canada. We recognize that not all physicians have e-mail, so results of the type of research we present in this paper can be extrapolated only to e-mail users. We expect that, within a few years, an overwhelming majority of physicians and patients will have access to e-mail.

METHODS

Study design

The study was a randomized controlled trial of physicians’ responses to a survey distributed by e-mail or regular post. Outcomes were response rates, degree of completion of the survey, response times, and characteristics of responses.

Study population and group allocation

We obtained our sample from physicians on the College of Family Physicians of Canada’s membership list, which provides postal and e-mail addresses. At the time of the study, 1600 family physicians in Canada had e-mail addresses registered with the College. Of these, three were removed: two because they were study investigators and the other because the e-mail address was listed as “no e-mail.”

The remaining 1597 physicians with e-mail addresses were randomly allocated to receive the questionnaire by either e-mail (group A) or post (group B). In addition, a random sample of 800 physicians who did not have e-mail addresses served as the second control group (group C) and received the questionnaire by post. The second control group was used to determine whether having an e-mail address (being an e-mail user) accounted for differences in responses rather than differences being accounted for by method of distribution. Without this second control group, we could not be certain whether the medium by which the questionnaire was completed or the characteristics of respondents (those with e-mail possibly being different from those without e-mail) accounted for differences found.

Survey structure

After pilot-testing the questionnaire format with various forms of e-mail delivery, we settled upon embedding the questionnaire directly into the body of the e-mail. The other alternative was to send it as an attachment, but not all e-mail programs allow users to handle attachments easily, and retrieval of attachments requires more expertise on computers. We attempted to maximize response by making the e-mail process as simple as possible.

Physicians were instructed to use the “reply” function on their e-mail system, to complete the questionnaire, and then to use the “send” function to return it to us. The return e-mail address for all e-mail questionnaires was the project manager’s. A modified Dillman method9 was followed; all three groups received the questionnaires at first and got a reminder 1 week later. Nonresponders received subsequent mailings at 3 weeks and 8 weeks. We waited 2 months after the 8-week mailing before ending the study.

Content of the questionnaire

The content of the questionnaire is secondary in this study. It gave us, however, an opportunity to obtain valuable information regarding family physicians’ attitudes. We developed a legitimate questionnaire asking legitimate questions concerning use of sildenafil citrate (Viagra), a topic that is of interest to us as primary care researchers, and on which little research has been done. We collected demographic data on physicians and asked respondents about their prescribing behaviour. Space was provided for comments on open-ended questions.

Statistical methods

We used the Statisical Package for the Social Sciences to analyze the data. Response rates were calculated based on the number of valid surveys distributed. Demographics and response characteristics were assessed using frequency distributions; comparisons between groups were made using 2 and odds ratios for categorical variables and t tests for continuous variables. The Queen’s University Research Ethics Board approved the study. Respondents were informed that results were confidential.

RESULTS

Of the 2397 family physicians surveyed, 1071 (44.7%) provided usable responses. By group, 268 (33.6%) e-mail respondents, 421 (52.7%) post respondents who had e-mail addresses, and 382 (47.8%) post respondents who did not have e-mail addresses returned their surveys. Due to invalid addresses, 17% of the e-mail mailings were returned, but less than 0.05% of the postal mailings were returned. We did not attempt to follow up on invalid e-mail addresses, but treated them as we would envelopes marked “Return to sender.” Response rates are shown in Table 1.

Table 2 compares characteristics of respondents and responses by method of response. There were no significant differences in the demographics of group A and group B, as would be expected because physicians were randomly allocated to these groups. Physicians in group A, however, were more likely to provide comments on the open-ended questions, and these comments were significantly longer than the comments provided by group B. Very few data were missing from surveys returned by groups A and B. The post surveys (group B) were more completely filled out (98.4%) than the e-mail surveys (group A) (97.6%). While the difference was just statistically significant (P = .05), it is unclear whether there is any practical difference.

Compared with group C, physicians in group A were more likely to be male and to be in university-based practices (Table 2). This is probably because the underlying group (those with e-mail addresses) from which respondents were selected was more likely to be male and to be in university-based practices. The significant differences seen in comparing group A with group C were the same as were seen in comparing group A with group B: comments were more frequent and longer in the e-mail group than in either of the other two groups.

Table 3 shows responses to opinion statements about sildenafil citrate. There were no significant differences between any of the groups in the proportion of physicians agreeing with the statements. This suggests that neither the format nor the process of the questionnaire (e-mail vs post) affect responses.

Response times were assessed. Since responses sent by post must, due to the nature of the medium, be measured in days or weeks, and e-mail response can be measured in seconds and minutes, it is inappropriate to compare them directly. Three e-mail responses arrived within half an hour of being sent; replies were arriving while the e-mail questionnaires were still being sent out. Within 3 hours, 16% of e-mail responses had been returned, and within 12 hours, nearly 36% of responses were in. Slightly more than 24 hours after e-mail distribution, 50% of the responses that would eventually come in had already arrived. Mean response time was 5.26 days; median response time was 1 day.

Such detailed information on time to response is neither available nor reliable for the posted survey. Paper surveys that get completed quickly might not be mailed until days later. Postal service is variable, and time to delivery depends on day, time of day, and time of year sent. About 50% of mailed surveys were returned after the first mailing, a further 35% after the second mailing, and the remainder after the final mailing at 8 weeks.

DISCUSSION

The high number of invalid e-mail addresses was not surprising; e-mail addresses and mailboxes are far from reliable. People often change providers or servers, or allow their accounts to go dormant.

The finding that significantly more physicians with academic practices had e-mail addresses was not surprising given that academic institutions provide them to their faculty. Community-based physicians have to subscribe to an Internet provider to obtain e-mail service at home or in the office.

As with previous studies,1,2,4-7 the response rate for the e-mail group was lower than for the mailed group. The lower response rate for e-mail might be explained by the ease of deleting unwanted e-mail messages, by e-mail messages never being checked, or by physicians having family accounts that are checked by people other than themselves.

If low response rates could be overcome, e-mail surveying could be the means to obtaining rapid responses to surveys, as was shown in this study where 50% of responses arrived within 1 day of being sent. As well, e-mail questionnaires might be better for surveys with open-ended questions because responders seem willing to write longer responses by e-mail than they do when they have to write them by hand.

The lack of differences in the content of responses sent by e-mail and by post suggests that the format of the questionnaire (e-mail vs paper) does not affect what respondents actually reply to structured-response questions. There might be qualitative differences in responses to open-ended questions.

Limitations

The main limitation of this study is the low response rate in all groups. Another limitation is restriction of the sample to members of the College of Family Physicians of Canada. While most family physicians in Ontario are members of the College, it is possible that non-members would respond differently. The similarity of the outcomes of this study to those of studies done on other groups is a strength. It suggests that the results have external validity and that physicians’ responses to e-mail surveys do not differ substantially from responses of other groups.

Conclusion

The e-mail survey had a lower response rate than the mailed survey, but e-mail responses arrived much faster. There were no practical differences in completeness or content of responses to structured-response questions between e-mailed and mailed surveys. Open-ended questions were more likely to be completed and to have longer responses on the e-mail surveys than on the mailed surveys. Since responses are similar for both methods and e-mail is faster, e-mail might be the preferred method when answers are needed quickly. The response rate, however, might well be lower, and this could affect generalizability of results.

Acknowledgment

We thank our research assistants for their work on this project. We also acknowledge the financial assistance of the physicians of Ontario through The Physicians’ Services Inc Foundation.

Contributors

Ms Seguin and Dr Godwin designed the study, collected and analyzed the data, and prepared the paper for publication. Dr MacDonald and Dr McCall contributed to data collection and analysis and critically reviewed the paper during its preparation.

Competing interests

None declared

Correspondence to: Dr Marshall Godwin, Centre for Studies in Primary Care, Department of Family Medicine, Queen’s University, 220 Bagot St, PO Bag 8888, Kingston, ON K7L 5E9; telephone (613) 544-3400, extension 3974; fax (613) 544-9899; e-mail godwinm@post.queensu.ca

References

1. Kesler S, Sproull LS. Response effects in the electronic survey. Public Opin Q 1986;50:402-13.
2. Rafaeli S. The electronic bulletin board: a computer-driven mass medium. Comput Soc Sci 1986;2:123-36.
3. Parker L. Collecting data the email way. Train Develop 1992;46(7):52-4.
4. Kittleson MJ. An assessment of the response rate via the postal service and email. Health Values 1995;18(2):27-9.
5. Mehta R, Sivadas E. Comparing response rates and response content in mail versus electronic mail surveys. J Market Res Soc 1995;37:429-39.
6. Mavis BE, Brocato J. Postal surveys versus electronic mail surveys. The tortoise and the hare revisited. Eval Health Profess 1998;21(3):395-408.
7. Paolo AM, Bonaminio GA, Gibson C, Partridge T, Kallail K. Response rate comparisons of e-mail and mail-distributed student evaluations. Teach Learn Med 2000;12(2):81-4.
8. Babbie ER. Survey research methods. 2nd ed. Belmont, Calif: Wadsworth; 1990.
9. Dillman DA. Mail and telephone surveys: the total design method. New York, NY: Wiley and Son; 1978.