Reference no: EM133687212

Experimental Design

• Research Question
• Hypotheses
• Standard
• Alternative
• Design
• Pre-registration
• Preparing the experiment
• Instructions
• Software
• Procedural details
• Run the experiment
• Data analysis
• Writing the paper and presenting the results

The most important thing and the reason why you design an experiment is your RESEARCH QUESTION

You should propose the right hypothesis or hypotheses to be tested that can shed light on the research question.
The hypothesis must be motivated, ideally based on existing research.
You must anticipate the kind of data you will obtain and therefore, the kind of analysis you will perform to test your hypothesis.
Finally, you must PRE-REGISTER your study in a website dedicated to pre-registrations. There are several platforms like RCT registry from the
American Economics Association, OSF and many other places where one can pre-register an experiment.

When you preregister your research, you're simply specifying your research plan in advance of your study and submitting it to a registry.
Preregistration separates hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research.
You must describe your plan in advance: what hypothesis you will test, the sample size you will take, what analysis you will perform, etc.
To determine the sample size, you must perform a power analysis. There are several statistical packages to do so; for instance, Stata or G*Power can perform the power analysis ex-ante to determine the sample size of the study.

Treatment: a particular condition of the experiment
There is a control Treatment and the main Treatment where we include our manipulation.
The simple manipulation is the only change between the control treatment and the main treatment.

An experiment usually consist of several Sessions
Subjects are the participants in the experiment.

Within-Subjects: Same subjects participate in more than one treatment. We test the inclusion of the manipulation to the very same subjects.
Allows for individual comparison
Control for individual fixed effects
Usually provide higher power analysis
BUT: It generates problems with "order effects" unless some procedures are implemented.
WE CAN FIX IT: Balancing the order in which subjects face each treatment in the experiment (CT/TC design).
Between-Subjects: Subjects participate only in one treatment. We compare the effect of the manipulation across different subjects.
PRO: Is a clean approach and provides strong evidence in favour of the effect of our manipulation.
CON: The experiments are more expensive

Distinction between observation and statistically independent observation
Example: 5 Sessions of a market experiment with 6 periods and 10 trades in each market.
We collect 300 price observations
However we have only 5 independent observations

If there are only few independent observations, usually the non-parametric tests are used to analyse data instead of regressions.

Pro One-Shot
Strong incentives for decision
No strategic spillovers across periods
Easy to perform and short

Pro repetitions ("repeated one-shot")
We can measure learning
It possible to observe the dynamics like convergence to equilibrium
More observations (not independent observations)

Partner design: Groups of subjects stay together for more than one period
Finitely repeated game
0 If only selfish types and unique Nash equilibrium in stage game: backward induction gives solution to game (start in last period...).
0 If stage game has multiple Nash equilibria, no unique prediction ("anything goes").
0 If multiple types (e.g., reciprocal and selfish players), many Bayesian Nash equilibria.
"Infinitely" repeated game
0 Implementation with the help of a termination probability
0 Problem: length of the experiment is endogenous
R Do you want to throw dice for five hours?
R What if after the first period the game ends?
R Different sessions have different lengths

Partner (groups of subjects stay together for several periods)
0 Every pair/ group of partners yields one independent observation
0 Allows analysis of strategic considerations
Stranger (pairs or groups are recomposed randomly)
0 Similar behavior/prediction as "one-shot" but more observations
Perfect stranger: probability of being re-matched with the same person is exactly zero (and subjects know that)

Strategy method was first used by Reinhard Selten.

Idea: Instead of just playing the game, subjects are asked to indicate an action at each information set, i.e., the experimenter elicits a strategy in every decision node of the game.
Example: Imagine the sequential version of the prisoner's dilemma and the second mover is asked: What do you do (defect or cooperate)
...if first mover cooperates?
...if first mover defects?

Advantages of strategy method:
0 More information about motivation/behavior of players (Figure out, e.g., that someone is a reciprocal player, even though first movers always defect)
0 Information about how people would play "off equilibrium" or "off action path" (since this is not usually reached, you have no information how they play unless use strategy method)
Problems of strategy method:
0 Incentives are weaker, since each information set is reached only with probability < 1.
0 Hot vs. cold emotions: People might feel and act differently knowing they have reached a particular information set, compared to potentially reaching it.
0 Explaining the strategy method to subjects is tricky (loss of understanding, control)

Does strategy method induce a different behavior relative to a situation where a subject responds to the actual move of an opponent?
0 Jordi Brandts and Gary Charness: "The strategy versus the direct-response method: a first survey of experimental comparisons" (Experimental Economics, 14, 2011, 375-398) report evidence indicating that the strategy method does not induce different behavior.

Moreover: You may use strategy method in all your treatments, and focus on treatment differences.

Role switching: Subjects act in different roles, e.g., in the ultimatum game as a proposer and a responder.
Helps to put oneself in the shoes of the other person. This can be useful for learning in complex games (e.g. signaling games).
May not be a good procedure because you lose information about how people act in a given role (e.g. when focus is on fairness).

In complicated experiments (e.g., with difficult trading rules in markets) it is a good idea to have a demo round first (before starting the actual experiment and without monetary consequences) to check that subjects get the rules of the experiment.
Advantages
0 Guarantees subjects' understanding from the first paid period on
0 Allows answering "new" questions of subjects that arise after learning trials
Disadvantages
0 You lose information about the "true" first period
0 People infer uncontrolled things from the learning trials
0 Subjects may send (costless) signals
Makes a lot of sense if the instructions are really difficult. Maybe it is not necessary to play a full game (e.g., just the complicated part) and maybe it is not necessary to display all information about others' actions.
In any case: if learning trials, then in all treatments

Most important: use credible chance moves
0 If many chance moves are necessary: program a random device at the computer
0 If only few chance moves and if credibility is an issue (e.g., imposing infinitely repeated games): Throwing dice may be better since subjects feel that is just the nature (and not a manipulation of the experimenter) who actually decide the movement.
Risk preferences
0 It is possible to control risk preferences with binary lottery method: Holt and Laury (2002)

Example: Prisoner's dilemma
0 Before subjects make their decisions, both players are asked what they think the other player will do, cooperate or defect?

Advantages
0 Beliefs can be informative to understand the motivation
0 Beliefs can be used to check the rationality of decisions (Example: guessing game)
Problems
0 Experimenter-Demand-Effect (you may make people think about stuff they would not have thought about)
0 Desire to be consistent: people state beliefs to "match" their actions
0 People have a desire to "justify" actions: someone defects and states the other person would defect too

Pros:
0 Subjects have an incentive to state correct beliefs
Cons:
0 Costly and - given a budget - goes at the cost of incentives in the decision part
0 Subjects have no incentive to state wrong beliefs anyway
0 Sometimes complicated to explain (e.g., payment dependent on distance measure between true outcome and expected outcome, quadratic scoring rules, etc)
0 Can pollute incentives in the experiment

Advantages of paper and pencil experiments
0 Flexibility (quickly develop new treatments)
0 Relatively low start up costs
0 Natural environment
R Not a lab with computers etc. but a classroom
R Procedures more visible and credible (e.g., throw dice)
Advantages of computerized experiments
0 Better control
R no communication among subjects
R less interaction with experimenter
0 Running of experiment much simpler (e.g., markets)
0 Fewer mistakes
0 Automatic data collection

Never cheat on subjects, even though it is tempting from a scientific point of view.
Why?
0 You will lose your reputation towards your subjects: If you lie to them once, they will never believe you in the future. This diminishes all incentives.
0 There is a moral code among economic experimentalists not to do that. You will never publish a paper and people won‘t like your research.

In almost all experiments you want to have a set of predictions / hypotheses
Traditional assumptions in game theory:
0 Rationality
0 Selfishness = money maximizing
0 Both are "common knowledge"
Determine equilibria
0 Often simple and unique prediction
0 But often describes behavior not very well
Use the standard prediction as a benchmark

Observations from everyday life, intuition
Previous experimental results (economics, psychology)
Game theoretic analysis under alternative assumptions
0 Prospect Theory (risk behavior, loss aversion)
0 Fairness theories
0 Statistical game theory, QRE (errors depend on cost of error)
0 Level-k model (limited steps of reasoning)
0 Visceral factors, emotions

Pros and cons of Framing
0 Concrete framing (goods market, labor market)
R Easy to understand
R Problem (?): Associations from real life
0 Abstract framing
R Harder to understand the rules of the game
R No control about what subjects really think

Complete description of the rules of the game
0 Sequence of decisions
0 Interaction
0 Payoff consequences
Different ways to explain the payoff function
0 Formula
0 Verbal explanation
0 Picture
Control questions
0 Check understanding
0 Allows knowing who read carefully the instructions
0 One should not be suggestive with examples

Allow to test the understanding of experiment
Allow to infer something about motives
Allow to check for the credibility of experiment
Control
0 How many subjects did know each other?
0 Socio-economic questions (sex, age, money, city, subject of study etc. etc.)
Psychological questionnaires (used to construct particular types)

Use hypothetical currency and convert it into Euro at the end of experiment
Show up fee
0 They should receive something just because they take part in the experiment.

Goal: total payments should cover opportunity costs (typical job)
Ensure anonymity when paying

Tell the subjects that this is an experiment about economics
0 This is important for understanding economic problems
Why should you take part?
0 You can earn money (do not mention concrete amounts of money: this creates expectations and may pollute behavior "if I do not earn at least x, I must have been wrong")
0 Learn about an interesting method in the social sciences

Collect data in systematic way (one master file, which remains unchanged)
Do descriptive statistics
0 Tables
R Title, clear variable names, round numbers properly
0 Figures
R As simple as possible, title, label axes, complete legend
R Figures are often understood and remembered best
Test Hypotheses
0 Frequently used
R Means (t-Test)
R Wilcoxon Signed Rank Test
R Wilcoxon-Mann-Whitney Test
R Kolmogorov-Smirnov Two Sample Test