Controlled experiments
- an epidemiological approach to road safety evaluations
Jens Christian Overgaard Madsen, Traffic Research Group, Aalborg University
Scientific experiments
Ideally road safety evaluations should be performed as scientific experiments
The emblematic of scientific activity
An experiment is a set of observations, conducted under controlled circumstances, in which the scientist manipulates the conditions to ascertain what effect, if any, such manipulation has on the
observations (Rothman et al., 2008)
The problem: The control of circumstances!
Approaches to
evaluations of effects
Before-after studies of incidence occurrences (time series analysis)
– Traditional approach to road safety evaluations
– Challenge to control for systematic and random variations in incidences between baseline and follow up/before and after treatment
Case-control studies/epidemiological studies
– Widespread use within medical and clinical evaluations – Comparison of incidence rates during follow up between
samples/cohorts subjected to different levels or types of treatment – Challenge to control for systematic and random variation in incidence
between treated and untreated samples/cohorts
Epidemiology
Definition:
– The study and analysis of disease occurrence, treatment and causation
Main task
– To minimize the presence of random and systematic, i.e.
confounding/bias, errors in studies of the effects of given
treatments/exposures upon disease occurrence and/or disease complications
Cohort study
Cohorts:
– A cohort is a designated group of individuals (or objects) who are followed or traced over a period of time
– The epidemiological study studies the occurrence of disease/disease complications (or in general incidences of interest) within each cohort within the period of follow up (period after “treatment”)
Cohort effect:
– The effect of a given treatment or exposure is studied by comparing the occurrence of disease/disease complications (or incidences of interest) for cohorts subject of treatment/exposure and untreated/unexposed cohorts
Cohort study
Road deaths amongst males age 18-24 year in rural and urban areas:
Urban Rural
Deaths a b
Population
(man months) C D
Incidence rate a/C b/D
Case-control study
Instead of studying the whole populations only samples of the population are included in the study.
Cases:
– Sample of objects/subjects exposed/treated
– Medical experiments typically includes all treated/exposed subjects.
– Estimator of λTreated (incident rate with treatment/exposure)
Control:
– Sample of objects/subjects from the population which is untreated/unexposed
– Estimator of λUntreated (incident rate without treatment/exposure)
Experimental vs.
non-experimental study
Experimental study
A study in which the researcher
manipulates the exposure, i.e. controls the treatment and non-treatment of
objects/subjects Non-experimental
study
A study in which the researcher has no control of treatment/exposure, e.g.
studies the effects of choices made by others => Observational studies
Experimental vs.
non-experimental studies
In epidemiological research experimental studies have a higher rank than observational studies => Higher quality research
The control, which the researcher possesses, leaves the researcher with better possibilities of controlling for confounding, i.e. better possibilities of isolating the effect of the treatment/exposure under evaluation.
Observational studies:
– The effect is estimated as the relative difference in incidence occurrence
between the treated/exposed group (case) and the untreated/unexposed group (control). Depending on how the cases were selected at the outset, this selection may in itself be a source of systematic variation between the case and the
control group.
– Cases and controls may differ not only by the fact that one is treated and the other is not – but also due to why they are selected. (base line difference)
– Treated black spots should ideally not be compared to a control group including untreated sites, but to untreated black spots.
Experimental studies
Clinical trials Those included in the experiment are
already ill. Studies the development of the disease (disease complications) amongst treated/exposed cohorts/samples and untreated/unexposed cohorts/samples CUREMENT/Prevention of complications Field trials Those included in the experiment are not
ill. Studies the occurrence of illness
amongst treated/exposed cohorts/samples and untreated/unexposed
cohorts/samples.
PREVENTION
Field trials are more expensive than clinical trials, as incidences of interest are more likely to occur amongst those who are already ill
Controlled experiment
Experimental trial based upon case-control group(s)
Effect of treatment/exposure
– The occurrence of incidences of interest in a case sample
consisting treated/exposed objects/subjects, e.g. described by an incidence rate (IRCase = E{IRTreated} = E{λTreated})
– The occurrence of incidences of interest in a control sample
consisting untreated/unexposed objects/subjects, e.g. described by an incidence rate (IRControl = E{IRUntreated} = E{λUntreated})
– Effect described by the Incidence rate ratio (IRR) – E{ε} = IRR = IRCase/IRControl
0 0,5 1 1,5 2 Case
Control
Challenge
0 2 4 6 8 10 12 14
Before After
Before-after:
• Control for sources of systematic
variation before-after not attributable to treatment
• Control for random variation from
before to after (regression-to-the-mean)
Case-control:
• Control for sources of systematic variation between control and case sample not attributable to treatment
• Control for random variation in
incidence occurrence between case and control sample
Confounding
The confusion, or mixing, of effects; implying that the effect of the exposure (treatment) is mixed together with the effect on incidence of interest of another variable (Rothman, 2002)
Confounding arises, when the case and control sample differs on other variables affecting the incidence of interest
Confounding in case-control experimental studies may arise from:
– Differences in base-line characteristics between samples – Differences in follow-up characteristics between samples
t
Treatment Study
Base-line
Follow-up
Confounding
The confusion, or mixing, of effects; implying that the effect of the exposure (treatment) is mixed together with the effect on incidence of interest of another variable (Rothman, 2002)
Confounding arises, when the case and control sample differs on other variables than those of treatment affecting the incidence of interest
Confounding in case-control experimental studies may arise from:
– Differences in base-line characteristics between samples – Differences in follow-up characteristics between samples – Measurement/information bias
Misclassification
Recall bias
Biased follow-up
Controlling for confounding
Randomization
(experiments only)
The samples are chosen randomly from the population – treatment assigned randomly. If
samples are large, the samples should apart from treatment be identical. The risk of confounding decreases as sample sizes increase. Prevents confounding for all risk factors
Restriction
(experiments and observational
studies)
Only individuals who share the same value on variables that are considered possible
confounders are included in the study. Those treated and those selected for control share the same value. May only prevent confounding for known risk factors (variables of restriction).
Problems of representativeness.
Matching Each treated/exposed individual is matched by an identical individual in the control group.
Problematic: In case-control studies this
approach provides no control for confounding. It may even introduce confounding even where there is none.
Confounding
Confounding related to base-line
Confounding related to follow-up
Randomization is believed also to control for confounding related to follow- up
– Blinded/double blinded study – Selection bias (discontinuation) – Hortons? effect
– Placebo effect
Stratification vs. data mining
t
Treatment Study
Base-line
Follow-up
Random variation
Error attributable to random occurrence of incidences in the study groups
(Σ xi)j/I → λij
Conducting an experiment does not eliminate random variation as a source of error!
Error
Study size Random
error
Design a large study
?
Control for
random variation/error
The presence of random variation allows us only to provide estimates of the true effect
Applying statistic analysis to the occurrence of incidences of interest allows us to examine how close we are to describing the true effect
In case-control studies the effects are commonly described through incidence rate-ratios
E{ε} = IRR = IRCase/IRControl
IR = Number of incidences in group/Total time of follow-up (Xi/Ti)
95% CI (IRR) = EXP [ln IRR ± 1.96 * SE(ln IRR)]
Ln IRR = ln IRCase – ln IRControl
SE (ln IRR) = √[(1/XCase) + (1/XControl)]
Estimating the true effect
Use an experimental design
Choose the study samples by randomization
Use large study groups
Controlled experiment/randomized experiment
Bicycle running lights
Background
October 1st 1990: Mandatory use of daytime running lights for motorized vehicles in Denmark
A reduction in accidents were recorded – especially for multiparty accidents (Hansen 1993;1995)
3-12% reduction in multiparty daytime accidents (Elvik, 1996)
Mandatory running lights for mopeds and motorcycles; 7% reduction in multiparty daytime accidents involving mopeds and motorcycles (Elvik et al., 2009)
Hypothesis
Introducing permanent running lights for bicycles may improve their visibility and hence their safety
Poor visibility likely to be an accident factor (Danish Road Traffic Accident Investigation Board, 2008)
Permanent bicycle running lights will reduce accidents involving bicycles because:
– Their daytime visibility will be improved
– The problem of cyclists forgetting their bicycle lights during twilight hours and at night will be eliminated
Research design
Prior studies of the safety effects of permanent running lights had been done by observational before-after studies
Observational before-after studies may not have provided sufficient control for confounding factors that may affect the outcome of the evaluation (Elvik 1993; 1996)
Controlled experiments provide a better control for confounding (Hauer, 1997)
It was rather obvious to conduct the study as a controlled experiment
– Readily comparable to epidemiological/medical research study
– Flashing front lights were prohibited at the time; hence they would only be allowed, if it could be documented that a flashing front light would not impair the safety of the cyclists
Case-control, randomized experiment (field study)
Demands
Cyclists to participate in the study should be chosen for treatment and non-treatment at random
Large study groups should be applied in order to control for random effects – 2.000 in treated sample and 2.000 in control sample
The occurrence of incidences (accidents) amongst those treated and those not treated should be as accurate as possible
Follow-up period of minimum 1 year of duration
Participants
Resident in the Municipality of Odense
Men, women and children from the age of 5 and above
Owners of a bike
Frequent bike users
Access to the Internet
Recruitment:
– TV-spots – Radio-spots
– Newspaper adds
– Personal letters to all public school pupils
Incentive: Free set of lights
11.800 persons from a total population of 180.000 persons volunteered to participate
Selection
Household selection
Randomized selection
Cluster randomization
– Large clusters may undermine randomization – Average cluster size 1.7
Recruitment survey
Name
Gender
Age
Car ownership
Age of bike
Bicycle use frequency
Purpose of bicycle trips
Address
Postal code
Participants
Treatment group: 1.845 persons had the running lights fixed to their bike by November 1st 2004
Control group: 2.000 persons had accepted to participate as members of the control group
Gender
53,52% 53,75%
46,48% 46,25%
0%
20%
40%
60%
80%
100%
Treatment group
Control group
Males Females
Participants
Car ownership
0%
20%
40%
60%
80%
100%
Treatment group Control group
3 2 1 0
Cycling - winter
0%
20%
40%
60%
80%
100%
Treatment group
Control group
Every 14th day 1-2 times a week
3-4 times a week
Daily
Self-reporting of accidents
Only 5-10% of all cycle injuries recorded at hospitals and emergency rooms are recorded by the police
Web-surveys are:
– Cheap
– Conditional design – Data processing – Reminders
Report back every second months
– Recall-bias
Do as you use to!
Spørgeskema
Reporting rates
Treatment group Control group
Rep. Answ. Part. % Answ. Part. %
1. 1.774 1.845 96,2% 1.918 2.000 95,9%
2. 1.767 1.845 95,8% 1.903 2.000 95,2%
3. 1.760 1.845 95,4% 1.862 2.000 93,1%
4. 1.719 1.845 93,2% 1.828 2.000 91,4%
5. 1.704 1.845 92,4% 1.812 2.000 90,6%
6. 1.679 1.845 91,0% 1.802 2.000 90,1%
Reporting
Treatment group Control group
Months Persons Percentage Persons Percentage
0 (none) 34 1,8% 44 2,2%
2 16 0,9% 33 1,7%
4 15 0,8% 38 1,9%
6 51 2,8% 44 2,2%
8 39 2,1% 42 2,1%
10 98 5,3% 85 4,3%
12 (full) 1.592 86,3% 1.714 85,7%
Data analysis
Only persons with full reporting record included – complete subject analysis
Statistical analysis of drop outs in order to investigate (de-)selection bias – same dropout rate, same characteristics, same IR
Effects described through incidence rates; accident rates
E{ε} = IRR = IRCase/IRControl
IR = Number of incidences in group/Total time of follow-up (Xi/Ti)
95% CI (IRR) = EXP [ln IRR ± 1.96 * SE(ln IRR)]
Ln IRR = ln IRCase – ln IRControl
SE (ln IRR) = √[(1/XCase) + (1/XControl)]
Accident
Definition:
Accidents are defined as incidences were you have collided with other road users or unwillingly been forced of your bike (solo crash). Accidents must be reported even, if you are not hurt
Accident data
Accident characteristics
Treatment group Control group
All Personal injury All Personal injury
Accidents 98 69 157 125
Winter period 60 38 87 70
Summer period 38 31 70 55
Daylight accidents 57 45 101 81
Accidents during twilight hours 13 5 24 15
Accidents during dark hours 27 19 31 28
Solo accidents 64 51 91 75
Multiparty accidents 34 18 66 50
Accidents reported to police 1 - 4 4
Accidents reported to insurance company 10 9 18 17
Injuries treated at hospital/emergency
Rooms 10 10 23 23
Injuries treated by general practitioner
Only 1 1 7 7
Accident data
Accident characteristics
Treatment group Control group
All Personal injury All Personal injury
Accidents 98 69 157 125
Winter period 60 38 87 70
Summer period 38 31 70 55
Daylight accidents 57 45 101 81
Accidents during twilight hours 13 5 24 15
Accidents during dark hours 27 19 31 28
Solo accidents 64 51 91 75
Multiparty accidents 34 18 66 50
Accidents reported to police 1 - 4 4
Accidents reported to insurance company 10 9 18 17
Injuries treated at hospital/emergency
Rooms 10 10 23 23
Injuries treated by general practitioner
Only 1 1 7 7
Effects
All accidents Treatment
group Control
group Total
Recorded accidents 98 157 255
Man months 19.104 20.568 39.672
Incidence rate –
IR * 103 5.13 7.63 6.43
Incidence rate ratio –
IRR 0.67 -
95% CI (IRR) [0.52 ; 0.86] -
90% CI (IRR) [0.54 ; 0.83] -
Effects
All accidents
Accident type Incidence rate * 103
95% CI Treatment IRR
group Control group
All accidents 5.13 7.63 0.52; 0.86
Daytime accidents 2.98 4.91 0.44; 0.84
Twilight accidents 0.68 1.17 0.30; 1.15
Night time accidents 1.41 1.51 0.56; 1.57
Multiparty accidents 1.78 3.21 0.37; 0.87
Solo accidents 3.35 4.24 0.55; 1.04
Effects
All personal injury accident
Accident type Incidence rate * 103
95% CI Treatment IRR
group Control group
All P. I. accidents 3.61 6.08 0.44; 0.80
Daytime accidents 2.36 3.94 0.42; 0.86
Twilight accidents 0.26 0.73 0.13; 0.99
Night time accidents 0.99 1.36 0.45; 1.19
Multiparty accidents 0.94 2.43 0.23; 0.66
Solo accidents 2.67 3.65 0.51; 1.05
Security/confidence
0% 20% 40% 60% 80% 100%
Daylight Reduced sigth Twilight Lighting-up period
Clearly increased Increased Unchanged Reduced Clearly reduced
Level of satisfaction
0% 20% 40% 60% 80% 100%
Very satisfied Satisfied Medium Unsatified Very unsatisfied
Bias
All solo accidents with personal injury
Accident type Incidence rate * 103 Treatment IRR
group Control group
All solo P. I. accidents 2.67 3.65 0.73
Daytime solo P. I. accidents 1.62 1.94 0.83 Twilight solo P.I. accidents 0.16 0.53 0.29 Night time solo P.I. accidents 0.89 1.12 0.80 Summer solo P.I. accidents 2.09 2.92 0.72 Winter solo P.I. accidents 3.25 4.38 0.74
Controlling for measurement bias
A 61% reduction in accident rate is compared to effects for motorcycles and mopeds very large
It seems unlikely that the introduction of bicycle running lights should reduce the occurrence of solo accidents!
Recalibration of data – zero effect on solo accidents
Introducing a correctional factor to accidents with personal injury:
– CCorr = 0.73
Re-estimation of incidence rates for P.I. accidents:
– IR = X * CCorr/Total time
Corrected effects
Corrected – Accidents with personal injury
Accident type Incidence rate * 103
95% CI Treatment IRR
group Control group
All accidents 3.61 4.45 0.61; 1.09
Daytime accidents 2.36 2.88 0.57; 1.18
Twilight accidents 0.26 0.53 0.18; 1.35
Night time accidents 0.99 1.00 0.56; 1.79
Multiparty accidents 0.94 1.78 0.31; 0.91
Solo accidents 2.67 2.67 0.70; 1.43
Multiparty accidents w. personal injury
Multiparty accidents
Incidence rates * 103 IRR
corr. 95% CI (IRR) Treatment group Control group
All 0.94 1.78 0.53 [0.31 ; 0.91]
Winter 0.73 1.78 0.41 [0.18 ; 0.95]
Summer 1.15 1.78 0.65 [0.32 ; 1.31]
Daylight 0.73 1.46 0.50 [0.27 ; 0.92]
Twilight 0.10 0.14 0.74 [0.13 ; 4.01]
Night time 0.10 0.18 0.84 [0.11 ; 3.03]
Counterpart:
motorized 0.42 0.82 0.51 [0.23 ; 1.14]*
Counterpart:
cyclist, pedestrian 0.52 0.96 0.55 [0.30 ; 1.00]
Conclusions:
Bicycle running lights
Bicycle running lights improve traffic safety for cyclists
The effect is most evident for multiparty accidents under daylight conditions
Generalisations (?)
Self-selection
– Higher use of bike (higher exposure to cycling accidents) – Cautious cyclists (lower accident risk)
Methodological problems related more to the use of self-reporting than to the appliance of a randomized experimental design
Conclusions:
Randomized experiments
The study shows that randomized experiments apply for road safety evaluations
The approach offers convincing methods in terms of confounders and it is possible to account for the presence of random error
through the design process and the analysis of data
Considerations
The road safety evaluation should be defined as an experiment from the outset
Randomization should be applied in the selection of experimental samples; treatment and control samples
One should opt for large samples in order to reduce random error
Likelihood of random effects should be analyzed through statistical analysis
Randomized experiments are applicable to a wide range of safety treatments:
– Treatments related to road users
– Treatments related to means of transportation
– Treatments related to infrastructure/site-specific treatments
Application?
Observational before-after studies dominate in road safety evaluations
Economical reasons
– Experiments are expensive to conduct
– The analysis of data, control for confounding and the handling of random variation is straight forward
Structural reasons
– Study has to be large in order to reduce random error – The treatments must be induced at the same time – Those to be treated must be chosen at random…
– Those not selected for treatment, must remain untreated through follow- up!
Work ethics
– Road safety: Promise => Implement for treatment (=>) Effect?
– Medical: Promise => Experiment => Effective => Implement for treatment
