THE RCT PERSPECTIVE

The author could not find other attempts to measure the efficacy of net neutrality rules for innovation. There is, in fact, an ideal methodology to investigate such a question called the randomized controlled trial (RCT), though in nascent stages in innovation policy. The idea of randomized trials is that subjects are placed by lottery into “treatment”

and “control” groups. The impact of the program is estimated by comparing the behavior and outcomes of the two.

To be sure, the study of innovation is one of the most fertile areas of academic inquiry. There are dozens, if not not hundreds, of conferences on innovation globally, but fewer on innovation policy, and fewer still on the measurement of innovation policy. In May 2016 there was the first gathering²¹³ of academics attempting to study innovation policy with RCT.²¹⁴ The conference proceedings noted that little is known about what makes innovation policy work. Indeed there is little evidence, little measurement of new policy instruments, and reluctance by policymakers to implement RCT.²¹⁵

213 Making Innovation and Growth Policy Work: IGL Global Conference, May 25, 2015, http://www.nesta.org.uk/event/making-innovation-and-growth-policy-work-igl-global-conference#sthash.kHnBY0et.dpuf and

Making Innovation and Growth Policy Work: IGL Global Conference, May 25, 2015, http://www.nesta.org.uk/event/making-innovation-and-growth-policy-work-igl-global-conference

214 Making Innovation and Growth Policy Work: IGL Global Conference, May 25, 2015, http://www.nesta.org.uk/sites/default/files/igl_global_conference_agenda_13-04-16.pdf

215 Xavier Cirera, “We know very little about what makes innovation policy work:

Four areas for more learning”, World Bank. June 17, 2016,

http://blogs.worldbank.org/psd/we-know-very-little-about-what-makes-innovation-policy-work-four-areas-more-learning

An estimated €150 billion is spent annually by European governments in initiatives to support innovation and economic growth.²¹⁶ The UK’s Innovation Growth Lab (IGL) observes that that there limited study of the efficacy of governments’ innovation policies. Moreover, even if innovation evolves, whether radical or incremental, it is difficult to tell whether it was the result of a specific policy. IGL notes the report of the What Works Centre for Local Economic Growth which reviewed 15,000 policy evaluations and found that only 2.5 percent had a concrete conclusion about the effectiveness of the policy, and of those, 1 in 4 had a positive impact (about 0.06 percent of the total). The implication is clear; without evidence about what works, it’s difficult, if not impossible, to allocate resources to successful programs.

This is not to say that innovation policy is not a useful effort; the point is that it can be improved with research about its effectiveness. For example, if innovation for small and medium enterprises is the goal, it would be helpful to know which policy or intervention is more effective. A nation could make a net neutrality law to support

“innovation” for example, or it could offer innovation vouchers to SMEs based upon a lottery.²¹⁷

The goal of RCT is to reduce selection bias. A perfect study RCT for net neutrality would involve randomly assigning policies across countries and then measuring the effects. However in this investigation, the set of countries with net neutrality rules is given, not random. However there is a significantly large and varied group of

217 An RCT of innovation vouchers in which grants of €15,000 euros were given to 130 SMEs in Denmark found worker productivity 20 percent higher in firms that received the voucher versus those that did not three years after the voucher was received.

https://europa.eu/sinapse/webservices/dsp_export_attachement.cfm?CMTY_ID=0C 46BEEC-C689-9F80-54C7DD45358D29FB&OBJECT_ID=19482B57-BFF0-

08A8-B9CDCD0537CF7180&DOC_ID=208C1EB5-F450-3B3E-7A227F5731B73696&type=CMTY_CAL

countries without rules. This itself is worthy of study and provides a de facto control group. This investigation is not an RCT, but it attempts to capture the essence of RCT, which is study scientifically and to reduce bias.

The (IGL) offers some suggestions for how a nation could proceed to bring a scientific approach to innovation policy. Before introducing large scale programs, policymakers could conduct small-scale testing and pilot programs, and then scale up accordingly should evidence of effectiveness emerge. Moreover governments themselves could conduct R&D on their own initiatives.

But doing pilots projects is not sufficient, notes the report. A real culture of innovation must embrace experimentation. As such policymakers should be looking to conduct experiments to see what works with a rigorous framework for evaluation. RCT is one method of experimentation.

While RCT is best known for health and pharmaceutical research, it has been tried to some extent in public policy. For example the MIT Poverty Action Lab (J-PAL) has conducted some 700 RCT on poverty reduction interventions.²¹⁸ The Education Endowment Foundation has conducted over 100 RCT on different means to improve educational outcomes.²¹⁹ The French government also runs an experimentation fund that focuses on interventions for disadvantaged youth based upon projects that are crowdsourced from around the country.²²⁰

Moreover RCT could be utilized in different parts of the policy development process. For example, RCT can be used to test different

218 “Policy Lessons | The Abdul Latif Jameel Poverty Action Lab,” accessed January 8, 2017, https://www.povertyactionlab.org/policy-lessons.

219 “Site Map,” accessed January 8, 2017,

https://educationendowmentfoundation.org.uk/evaluation/resources-centre/setting-up-an-evaluation.

220 “Le Fonds d’Expérimentation Pour La Jeunesse | Ministère de La Ville, de La Jeunesse et Des Sports,” accessed January 8, 2017,

http://www.experimentation.jeunes.gouv.fr/72-les-resultats-des-experimentations.html.

provisions of the existing policy programs. With net neutrality, there are many provisions and the understanding efficacy of the various provisions could be improved using this technique versus on the entire policy itself.

There is no doubt that RCT is imperfect. The Pearce and Raman critique takes issue with what they believe is tacit acceptance that RCT investigations constitutes “neutral evidence.”²²¹ However peer review is the quality control for bias. In any event, it does not seem unreasonable that some amount of RCT, or some amount of empirical evaluation for that matter, could be adopted for net neutrality policy.

Though a direct RCT method could not be employed, the project attempts to measure the countries with two independent measurement tools. This is done to improve the scientific basis for the study as well as to guard against bias and overconfidence in the results.