SURV627: Experimental Design For Surveys

Data Generating Process

A key tool of methodological research is the split-ballot experiment, in which randomly selected subgroups of a sample receive different questions, different response formats, or different modes of data collection. In theory, such experiments can combine the clarity of experimental designs with the inferential power of representative samples. All too often, though, such experiments use flawed designs that leave serious doubts about the meaning or generalizability of the findings. The purpose of this course is to consider the issues involved in the design and analysis of data from experiments embedded in surveys. It covers the purposes of experiments in surveys, examines several classic survey experiments in detail, and takes a close look at some of the pitfalls and issues in the design of such studies. These pitfalls include problems (such as the confounding of the experimental variables) that jeopardize the comparability of the experimental groups, problems (such as nonresponse) that cast doubts on the generality of the results, and problems in determining the reliability of the results. The course will also consider some of the design decisions that almost always arise in planning experiments — issues such as identifying the appropriate error term for significance tests and including necessary comparison groups.

Course objectives: 

By the end of the course, students will…

  • Learn about basic principles of experimental design
  • Recognize the main types of experimental designs
  • Improve the quality of designs used to carry out methodological research in or for surveys
  • Develop critical skills to spot flaws in experimental and nonexperimental designs to support causal inferences
  • Improve skills at analyzing results of survey experiments
  • Improve skills as both consumer and producer of experiments done to shed light on survey methodological issues

Grading will be based on:

3 Online quizzes (45%)
3 exercises (45%)
Participation in online discussions (10%)



Dates of when assignment will be due are indicated in the syllabus. Extensions will be granted sparingly and are at the instructor's discretion.


At least one prior course in data analysis. Ability to use SAS or STATA

Course Dates


Fall Term (September – November)