Disadvantages of Online Surveys

Sample Quality

The most questionable aspect of web-based survey data is whether a representative sample of the target population will have the opportunity to respond. Internet users tend to be younger and more highly educated than the general U.S. population (e.g., see the Pew Internet studies). However, the demographics of the Internet are changing rapidly, and among certain subpopulations, such as college-bound seniors in high school, Internet usage is nearly universal (see Art and Science Group, Inc., studies). Thus, web-based surveys may provide high-quality samples for researchers who are interested in populations who are likely to frequent the Internet, such as college students or people who participate in online romance and dating. For those interested in non-white or older populations, however, the Internet may not provide a representative sample of the population of interest.

Technical Problems

There are a variety of technical glitches that can occur while a respondent is filling out a survey.

• Freezes and crashes. The respondent may be unable to complete the survey due to a browser freeze or server crash, resulting in missing data. To minimize this risk, try to keep the survey relatively short. If you have a long survey, you may want to split it across several pages. A respondent's answers are submitted at the end of each page; thus, if the respondent is disconnected during the fourth page of the survey, you have already safely captured the first three pages of data. If you are willing to spend more effort on programming, it is also possible to write a program that will allow respondents to return to the survey at the same point where they were interrupted.
• Error messages. If you overlook a programming error in your survey, the error may be triggered when a respondent fills out the questionnaire, resulting in an ugly and confusing error message. To avoid this problem, test and re-test your survey rigorously, and have several other testers fill out the survey as well, to ensure that all the "bugs" have been worked out. If you use a programming language such as Cold Fusion to create your survey, you can write more attractive and helpful error messages. For example, if a respondent accidentally skips a question, a customized message can instruct the respondent to return to the question and complete it.
• Double entry. In many surveys, it is possible for the same subject to return to the survey and complete the questionnaire several times. The best way to avoid this problem is to recruit a specific sample of subjects, assign each subject a unique id, and allow each id to be used only once. If you do not have a specific sample, but are recruiting subjects from the Internet at large, you can use the subject's e-mail address to assign a unique id (for more information, see the section below on informed consent). However, this is not a foolproof method, as many respondents may have several e-mail addresses.

Top