After implementation, researchers can measure the impact of a new digital health intervention with summative ex-post evaluation methods to elaborate whether the project objectives were achieved.
Suggestions
In any case, the final post-implementation evaluation is recommended, even if the implementation failed. This gives the researcher and the healthcare setting, as well
as other researchers, the possibility to learn from the project experiences and outcome, and thus contributes to further research and documentation (26). The annex provides information on how to conduct an evaluation.
Annex
Research methodology
The structure of the report follows the framework of the “GoTo navigator” developed by Kayer et al. (2017). It is a method used for the deployment of process
development management, implementation and evaluation of digital health interventions. Initially, it is important to emphasise that the search of needs, recommendations, toolkits and guidelines on digital intervention serves as
background information. Based on the selected literature, it is possible to cover the most important developments in the area of interest and knowledge of methods for improving eHealth projects. The database used for the literature search was
PubMed/MEDLINE. The analysis is done by the selection of relevant information in the title, and the abstract followed by reading relevant literature with the following search terms (guidelines OR recommendation OR “toolkit”) AND (eHealth OR digital health OR digital health intervention). The literature was limited to MEDLINE, peer-reviewed publications, in English only.
A Cultural probes interview design gave the structure that enabled a conversation on the experience working with developing digital health interventions in Denmark.
The “Cultural Probes” method serves for the collection of input from a preselected group of Edith network members. Cultural Probes is a method originally developed by Bill Gaver in 1999, as part of the EU Presence Project. Cultural Probe has
become a form of umbrella concept and now covers several different approaches to collect information in connection with design development. The design of the
approach of the Cultural Probe initiates inspirational responses to a certain topic.
The main idea is that the probes are part of a strategy to address common
challenges in conceptualising projects for unfamiliar groups and topics (27). Inspired by Cultural Probes, this project includes steps of the “GoTo navigator” as "probes."
The purpose of "probes" as a set of stimulus cards is to give the participants the chance to either work with them as an incentive to categorise their experience in different topic areas or speak freely about their experience working on digital health projects.
This method helps to identify which topic areas in the “Go-to navigator” applied to their projects. The most recurring experiences, including obstacles, challenges, benefits and strength in the projects are the basis for the components of the toolkit.
For each participant, set of coloured “navigator”-cards are used, and the supplied information of contributions are analysed with the help of audio-recorded files that are described and analysed accordingly. A thematic analysis approach inspired the analysis of data; first, an inductive step and then a deduction analysis. The inductive step consists of a thematisation (already pre-formulated on topic cards) of all care cards in order to identify cross-cutting problem areas and challenges
(subcategories).
Further, the report includes literature recommended through by the advisory panel and team members of the Edith network. In addition, grey literature via web pages, Google and Google Scholar search engines were searched for relevant information.
TOOLKIT
The following consist of literature, models, and suggestions to support and guide individuals who work with developing, implementing and evaluating digital
interventions for patients.
1. Inception phase Project planning
Stanford Biodesign Model from the book: Biodesign: the process of innovating medical technologies
The Stanford Biodesign model is a great tool to develop humen centred digital health interventions. Stanford Biodesign Model(2) defines innovation as a new idea,
process or product that requires need-based invention and implementation of medical technology innovation (Yock et al., 2015). In general, the scope lies on a systematic evaluation of healthcare needs, invention, and concept development as a process. This process can be applied to any medical discipline. Stanford Biodesign gained recognition in the United States and beyond because of its encouraging elements including creativity, network and synergy effects on process management.
The Stanford Biodesign enables innovative thinkers in the field of medicine and other disciplines, to build a bridge between science comprehension and clinical practices.
The process is published in textbook form with the title The Biodesign: The Process of Innovating Medical Technologies. This book includes and describes the method of making innovation happen.
The process is divided into three main phases namely identification, invention and implementation. The identification phase describes the identification of unmet healthcare related needs. The invention phase is sought to brainstorm how to address the most pressing need and develop and select a viable business model concept for it. The last phase is the implementation phase. The implementation phase is the first step where it is sought to translate the most promising business idea into practice, under consideration of, for example, intellectual property rights and regulatory legal factors.
Reference:
Yock, P. G., Zenios, S., Makower, J., Brinton, T. J., Kumar, U. N., Watkins, F. J., ... & Kurihara, C. Q. (2015).Biodesign: the process of innovating medical
technologies. Cambridge University Press.
West Midland Toolkit (WMT)
The West Midland toolkit is an eight-step innovation model developed for the project concerned with digital health innovation. The model gives a framework with steps to go through when planning a project in concerning health-IT.
Reference:
The West Midland Toolkit, Retrieved from
https://www.togetherforshortlives.org.uk/resource/west-midlands-toolkit/
Recommended literature to project planning and management of health IT projects
Ballard, G., & Howell, G. (2003). Lean project management. Building Research & Information, 31(2), 119-133.
Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A.,
Cunningham, W., Fowler, M., ... & Kern, J. (2001). Manifesto for agile software development.
Boogerd, E. A., Arts, T., Engelen, L. J., & van De Belt, T. H. (2015).
“What Is eHealth”: time for an update?. JMIR research protocols, 4(1).
Ciccarese, P., Caffi, E., Quaglini, S., & Stefanelli, M. (2005).
Architectures and tools for innovative health information systems: the guide project.
International journal of medical informatics, 74(7-8), 553-562.
Fiordelli, M., Diviani, N., & Schulz, P. J. (2013). Mapping mHealth research: a decade of evolution. Journal of medical Internet research, 15(5).
Grew, J. C., Svendsen, M. N., & Mortensen, B. B. (2011).
Teoribaserede patientanalyser i Medicinsk Teknologivurdering: En metodepublikation.
Hughes B, Cotterell M. Software project management. (2002) Software project management. Tata McGraw-Hill Education.
Kayser L, Furstrand D, Rasmussen E, Jensen L, Monberg A, Karnøe A. GoTo Navigator -a process navigation tool for digital health solutions to be used to design and align with an efficient trajectory for goal setting to evaluation of outcomes. (in review).
Martínez-Pérez, B., De La Torre-Díez, I., & López-Coronado, M.
(2013). Mobile health applications for the most prevalent conditions by the World Health Organization: review and analysis. Journal of medical Internet research,15(6).
Morrison, L. G. (2015). Theory-based strategies for enhancing the impact and usage of digital health behaviour change interventions: A review. Digital Health, 1, 2055207615595335.
Schwaber, K., & Beedle, M. (2002). Agile software development with Scrum (Vol. 1). Upper Saddle River: Prentice Hall.
Stellman, A., & Greene, J. (2005). Applied software project management. " O'Reilly Media, Inc.".
Zanaboni, P., & Lettieri, E. (2011). Institutionalizing telemedicine applications: the challenge of legitimizing decision-making. Journal of medical Internet research, 13(3).
Adapting existing studies/projects
Recommended literature to adapting existing studies/projects
Arthur, W. B. (2009). The nature of technology: What it is and how it evolves. Simon and Schuster.
Yock, P. G., Zenios, S., Makower, J., Brinton, T. J., Kumar, U. N., Watkins, F. J., ... & Kurihara, C. Q. (2015).Biodesign: the process of innovating medical technologies. Cambridge University Press.
Organisational challenges
Recommended literature to organisational challenges
Baldwin, C. Y. (2012). Organization design for distributed innovation.
Cardinale, A. M. (2018). The Opportunity for Telehealth to Support Neurological Healthcare. Telemedicine and e-Health.
Cohen, E. (2013). Accelerating digital health innovation: Analyzing opportunities in the healthcare innovation ecosystem (Doctoral dissertation, Massachusetts Institute of Technology).
Lennon, M. R., Bouamrane, M. M., Devlin, A. M., O'Connor, S., O'Donnell, C., Chetty, U., ... & Watson, N. (2017). Readiness for delivering digital health at scale: lessons from a longitudinal qualitative evaluation of a national digital health innovation program in the United Kingdom. Journal of medical Internet
research, 19(2).
The following section gives an overview of touch point In Denmark, sorted by Regions.
Region Hovedstaden (The Capital Region of Denmark)
In January 2012 the Capital Region of Denmark established a unit called research and innovation “Forskning-og-innovation”. The unit offers researchers and health professionals guidance when it comes to:
Financing of projects
Legal assistance to contracts
Commercialization of inventions
Collaborations with higher education institutions
University of Copenhagen
Technical University of Denmark
Metropol University of Applied Sciences
Sund Vækst Huset
Afdeling for Velfærdsinnovation Københavns Kommune
Sundheds- og Omsorgsforvaltningen Center for Innovation og Digitalisering Henrik Pontoppidans Vej 4, 1. sal 2200 København N, Denmark Web:
https://velfaerdsinnovation.kk.dk/artikel/sund-vaekst-huset
SundVækst Huset is a part of the Department of Welfare Innovation at the Municipality of Copenhagen. SundVækst cooperates with citizens, employees, companies and research institutions to jointly develop and support the best welfare technology solutions.
Contact:
Region Hovedstaden-Center for regional udvikling, Vækst & Viden
Copenhagen Health Science Partners
Cooperation between the Capital Region and Copenhagen University in
Copenhagen Health Science Partners (CHSP) aims at promoting synergies and cooperation between excellent research, clinical work and education, thus helping to strengthen the future health care system.
Web:
https://www.regionh.dk/til-fagfolk/Forskning-og-innovation/Strategiske-rammer-og- samarbejde/samarbejde-vidensinstitutioner-CHSP/Sider/Copenhagen-Health-Science-Partners.aspx
Region Sjælland (Region Zealand)
Region Zealand offers under the section Region Sjælland│Sundhed│Forskning│For fagfolk│Værktøjskasse a toolbox for researchers. In the toolbox, there is practical information about statistical guidance, statistical programs, it-tools, templates, legal assistance and databases (see below).
More information:
Region Sjælland
Produktion, Forskning og Innovation (PFI) Alléen 15
4180 Sorø Tlf. 70 15 50 00 Fax 70 15 50 09
Web: https://www.regionsjaelland.dk They offer:
1. Statistical guidance and statistic programs
Researchers and PhD fellows in Denmark are affiliated with an educational
institution and thus have access to their statistics programs. PhD fellows rely on their formal PhD supervisors for statistical guidance. A researcher employed in the
Region Zealand is eligible to receive 10 hours of statistical guidance from a local research support unit.
2. IT Toolkit
Region Zealand made an agreement with a Danish company called “EasyTrial”. The IT-software is developed from Danish doctors to provide an Internet-based system that allows researchers and PhD fellows to manage relevant tasks associated with participants in a clinical study. A license for researchers costs 5.000 Danish kroner.
The same license can be used for different projects for multiple project members and users. There is also the possibility to use the software for a test period for free.
Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6r kt%C3%B8jskasse/Sider/EasyTrial.aspx
Templates
Here they offer templates and other practicalities that can assist you in your research process, for example templates for research projects, records, PowerPoint, consent statements, etc.
Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6rkt%C3%B8j skasse/Sider/Skabeloner-til-foredrag.aspx
3. Legal assistance to contracts Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6rkt%C3%B8j skasse/Sider/Juridisk-bistand-til-kontrakter.aspx
4. Access to registry data Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6rkt%C3%B8j skasse/Sider/Forskerservice-hos-SSI.aspx
5. Good research practice Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6rkt%C3%B8j skasse/Sider/God-videnskabelig-praksis.aspx
6. Access to journals and databases Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6rkt%C3%B8j skasse/Sider/Tidskrift-og-databaser.aspx
7. Clinical trial book Web:
https://www.regionsjaelland.dk/Sundhed/forskning/forfagfolk/v%C3%A6rkt%C3%B8j skasse/Sider/Clinical-Trials-Handbook.aspx
Region Syddanmark (The Region of Southern Denmark)
The Region of Southern Denmark established the Health Innovation Centre of
Southern Denmark.The purpose is to offer information and support when developing eHealth solutions and social care in the Region of Southern Denmark. It also offers a referral to legal guidance and proposes different cases as inspiration material.
The Health Innovation Centre of Southern Denmark has three strategic areas:
Innovation
Cooperation
Funding
The Health Innovation Centre of Southern Denmark Syddansk Sundhedsinnovation
Forskerparken 10G 5230 Odense M
WEB: http://www.syddansksundhedsinnovation.dk
Region Midtjylland (Region of Central Jutland) Centre for Telemedicine and Telehealthcare
Olof Palmes Allé 15 8200 Aarhus N.
Denmark Web:
http://www.rm.dk/sundhed/faginfo/center-for-telemedicin/
The Centre for Telemedicine offers support concerning
Guidance to the good video consultation
Law and Telemedicine
Frequently Asked Questions about Law and Telemedicine
Design and development
Minilab
How to apply for funds
Telemedicine on a large scale
Telemedicine and ethics
Evaluation methods: MAST and checklist
Tips to evaluate your idea
Help with business case
Medtech Innovation Consortium (MTIC) Tueager DK
8200 Aarhus N Web:
http://www.mtic.dk/
The Medtech Innovation Consortium works with private health technology and welfare technology companies and healthcare with business-based health innovation.
Region Nordjylland (The Region of Northern Jutland)
The Danish Center for Healthcare Improvement (DCHI)-“Dansk Center for Forbedringer i Sundhedsvæsenet”
Fibigerstræde 11 9220 Aalborg Øst Web:
https://www.dchi.aau.dk/
The Danish Center for Healthcare Improvement (DCHI) is a research centre
established in collaboration between Aalborg University and the Region of Northern Jutland. Through research, evaluations, studies and student projects, the Center develops new knowledge and new methods for use in the healthcare system.
Forskningens Hus-Ideklinikken Sdr. Skovvej 15
9000 Aalborg Web:
http://ideklinikken.rn.dk/
The Idea Clinic makes an assessment of how best to solve your problem or how best to realize your idea or invention - whether it is a patentable and sold product - or it is about improving a workflow that can facilitate your daily work work and routines in hospitals and hospitals.
Stakeholder involvement
Step-by-step guideline for stakeholder involvement for business modeling in eHealth technology implementation by Van Limburg et al. (2015):
Start with a literature review on comparable interventions to get a feeling for the domains, jargon, and global issues and stakeholders.
Involve 1 or 2 domain experts in the research and development team to reflect future findings, ideally experts who have an affinity with technology and research processes.
Make an overview of all possible stakeholders based on literature on comparable interventions in the domain.
Assign stakeholder types to possible stakeholders, verify if certain types are missing and why.
Validate the entire overview by snowball sampling a complete stakeholder list with these key stakeholders.
Let experts select key stakeholders from the complete stakeholder list.
Organize a focus group with at least one in-person representative of each key stakeholder:
o Start with discussing each stakeholders’ role in the current processes.
o Let them complete the stakeholder list for missing stakeholders based on the process.
o Ask stakeholders to rank the importance of stakeholders, or alternatively let experts do it later.
o Discuss what bottlenecks are experienced.
o Discuss opportunities for improvement and opportunities for eHealth.
Summarise bottlenecks and opportunities and determine with the research team which opportunities are there for eHealth technology and whether these fit the project goals.
Ideate an eHealth technology (when possible, make mock-ups or a prototype of the ideas).
Plan interviews with stakeholders, or if possible, multiple stakeholders of the same stakeholder type, for value co-creation dialogues for the ideated eHealth technology.
Prepare the value co-creation dialogue interview with questions that address all business model components (also prepare subquestions that propose possible ideas or values on each business model component to help the interview along.
Focus on what the technology should contribute to their daily routines, not technical requirements).
Code transcripts of the focus groups and interviews, extract all implementation-related comments and combine all values and critical factors in the business model canvas.
Discuss the resultant business model with the research team.
Optionally, for transparency and extra validation, explain the business model to stakeholders and let them reflect on it or write a document that explains the implementation strategy based on the business model as the model itself may be unclear to share with the relevant stakeholders.
Additional literature to relevant stakeholders, their role and value networks Reference:
Catwell, L., & Sheikh, A. (2009). Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS medicine, 6(8), e1000126.
Feng, W., Crawley, E. F., de Weck, O. L., Keller, R., & Robinson, B.
(2010). Dependency structure matrix modelling for stakeholder value networks.
van Limburg, M., Wentzel, J., Sanderman, R., & van Gemert-Pijnen, L. (2015). Business modeling to implement an eHealth portal for infection control: a reflection on co-creation with stakeholders. JMIR research protocols, 4(3)
Yock, P. G., Zenios, S., Makower, J., Brinton, T. J., Kumar, U. N., Watkins, F. J., ... & Kurihara, C. Q. (2015). Biodesign: the process of innovating medical technologies. Cambridge University Press.
Acknowledgment from the project environment
(See organizational challenges)
Technical feasibility
Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., ... & Fernandez, M. (2009). How we design feasibility studies. American journal of preventive medicine, 36(5), 452-457.
Kayser L, Furstrand D, Rasmussen E, Jensen L, Monberg A, Karnøe A. (under review). GoTo Navigator -a process navigation tool for digital health solutions to be used to design and align with an efficient trajectory for goal setting to evaluation of outcomes.
Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004). Design and analysis of pilot studies: recommendations for good practice. Journal of evaluation in clinical practice,10(2), 307-312.
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., ... &
Goldsmith, C. H. (2010). A tutorial on pilot studies: the what, why and how. BMC medical research methodology, 10(1), 1.
Legal feasibility
Information on laws and regulations involving data security in Denmark Datatilsynets
Web:
www.datatilsynet.dk
GDPR
Health data in the workplace
Retrieved from https://edps.europa.eu/data-protection/data-protection/reference-library/health-data-workplace_en The act on processing of personal data
Web:
https://www.retsinformation.dk/
Web:
http://united4health.eu/wp-content/uploads/2015/10/D5.5-v1.0-U4H-Industry-Report-on-Telemedicine-Legal-and-Regulatory-Framework.pdf
Additional references
Kuchinke, W., Krauth, C., Bergmann, R., Karakoyun, T., Woollard, A., Schluender, I., ... & Ohmann, C. (2016). Legal assessment tool (LAT): an
interactive tool to address privacy and data protection issues for data sharing. BMC medical informatics and decision making, 16(1), 81.
O’Connor, Y., Rowan, W., Lynch, L., & Heavin, C. (2017). Privacy by Design: Informed Consent and Internet of Things for Smart Health. Procedia Computer Science
Region Hovedstaden (The Capital Region of Denmark) Center for Information and Innovation Law (CIIR) Karen Blixens Plads 16
DK-2300 Copenhagen S Telefon: +45 35 32 31 92 Fax: +45 35 32 32 04 Web: http://jura.ku.dk/ciir/
In 2010, the Faculty of Law at the University of Copenhagen established the Centre for Information and Innovation Law (CIIR), as a part of the Faculty of Law at the University of Copenhagen. It serves as a research-based forum for legal issues in the areas of information and innovation, including copyright, trademark law, patent law, business secrecy protection, personal data and personal privacy in the Capital region.
CIIR's research concerns both national and international regulation and is for a number of research projects characterised by interdisciplinary illumination of the individual legal areas.
The Capital Region of Denmark offers under the section Research and Innovation (“Forskning & innovation”), information on legal assistance to contracts, process and a handbook for research contracts.
The Capital Region of Denmark
They provide legal assistance about research and innovation.
Region Hovedstaden
Kongens Vænge 2 - 3400 Hillerød Web:
https: //www.regionh.dk/til-fagfolk/forskning-og-innovation /Juridisk%20bistand%20til%20kontrakter/Sider/default.aspx
Region Sjælland (Region Zealand)
Region Zealand has entered an agreement with the University of Southern Denmark (SDU) on legal assistance for research projects by legalising:
• Cooperation agreements on research and development
• Clinical Trials (Clinical Trials)
• Material Transfer Agreements (MTA)
This means that a researcher can forward all of the above-mentioned agreements and contracts for review by SDU lawyers. For drug trials, it is required that SDU lawyers approve the agreements. The lawyers advise that researcher and PhD fellows consult the SDU lawyers when considering a collaboration in which a contract has to be developed. The lawyers can help to prepare the right type of
This means that a researcher can forward all of the above-mentioned agreements and contracts for review by SDU lawyers. For drug trials, it is required that SDU lawyers approve the agreements. The lawyers advise that researcher and PhD fellows consult the SDU lawyers when considering a collaboration in which a contract has to be developed. The lawyers can help to prepare the right type of