A project starts with the project inception and start-up phase. The following sections describe the challenges experienced by the researchers in the advisory panel, who invented a digital intervention themselves or guided projects in a research context.
Most challenges experienced in the inception phase concern the project planning, understanding organisational structures of a healthcare system as well as adapting existing studies and interventions from other projects to the own project. Also, evaluating the technical and legal feasibility is challenging as it requires technical and legal literacy. Many inexperienced researchers, who try to innovate new digital health interventions, fail to involve stakeholders and communicate the scope of the project in a way that yields positive recognition for starting the projects.
Project planning
“In the beginning, I spent almost one year in trying to find out, what we should do with the technical solution. I am a psychologist and I am able to create all the content but to create the framework that was really difficult.”
The challenge in the initiation of a health IT research project is to establish a project definition and project plan that reflects the project vision. Also, if the project plan is not realistic enough, the risk that the overall process will be delayed increases.
Inexperienced researchers spend a lot of time on pre-project planning and are overwhelmed and challenged by all the required tasks; to navigate through pre-project processes, to understand and adjust the own individual process to already existing processes, procedures and products.
Suggestion
Think about what could be improved with the implementation of the new digital health intervention. One advisor from the advisory panel proposed a set of questions important to plan a digital health intervention (See below). It can be used as
inspiration. Further, identify existing guidelines and models for project planning and management (6) (See references in annex).
Examples of questions stressed by advisors Project planning
What is your project vision?
What is a realistic project duration?
What are the project deliverables?
How do you manage quality, time and costs?
What setting are you planning and designing for?
Who is the hosting organisation?
Is it a treatment, rehabilitation or is it a prevention tool?
How is the assessment done (Feasibility study)?
Who is doing the assessment?
Who evaluates the success of the treatment?
Where lays the responsibility of therapy?
Where lays the legal responsibility?
Data privacy, data security, and GDPR?
How is it all followed-up?
Users
Who is the user (Target group)?
How does the user find you?
What is the user pathway?
What is the user interface?
Organisation
What setting are you planning and designing for?
Who is the hosting organisation?
What sort of software do you plan to use (Desktop or Internet software)?
Adapting existing studies/projects
“The inspiration for this project came from a similar project in Japan. It needed translation and adaption into a Danish context. The project process from planning to the development took about 1.5 years.”
“I think because the primary problem often is that you start these projects as research and then you do not have the framework to continue the treatment afterwards because maybe you have borrowed a treatment platform from, for example, Karolinska University or another hospital. So it is actually not possible to take your source directly into practice.”
It can be a good idea to use existing methods/interventions from other projects, or parts of it, when developing new digital health interventions. However, it can be difficult and time-consuming to extract, translate and customise information and material from pre-existing studies/projects into the specific project context. Another challenge emphasized by the informants, was the problems that arise when the project must continue after the research period is finished. If an existing intervention has been used there might not be an agreement on whether the intervention can be used in the future.
Adaptation of content in a meaningful way
Translations in the Danish cultural and organisational context
Intervention style and format
Suggestion
It is recommended to investigate whether similar interventions exist in Denmark and internationally. Be aware of differences/similarities in practice and validate the digital health intervention so that it fits, for example, into the organizational context and is accepted by the user group. Additionally, be aware of deviating terminology, in terms of keeping it culturally relevant and authentic.
Assess economic and human factors and requirements to gain an understanding of how to translate an existing technical solution in a technically feasible way.
Furthermore, it is essential to have an agreement on what will happen when the research period ends.
Organisational challenges
“All doors were really closed and the hospital hadn’t had the structures and a treatment reimbursement system to offer Internet-based treatment options, yet.”
It can be time-consuming and challenging to understand organisational structures. It is often difficult to get an insight into organisations, companies and individuals ways of operating.
It is also challenging to develop a digital health intervention, which suits the patients, the healthcare professionals and fits into a given organisational structure.
It can also be difficult to convince people working in the current infrastructure to adapt to new digital health interventions.
Suggestion
When innovating for a healthcare setting, it is advisable to analyse the innovative capacity of the specific healthcare setting, as well as the attitude towards novel digital health interventions. Therefore, it is recommended to make a strong business model including topics such as reimbursement systems and to learn about the organisation infrastructure in the early planning phase of the project (7). With a strong business model for the new inventions, you demonstrate the benefits and the capability of innovating an intervention that fits into the current organisational
structure.
In order to make a strong business case, it can be helpful to follow a business model template, such as business model canvas, which includes critical steps that help to think about the value you are generating for an organization (e.g., hospital), the infrastructure of the organization, the users of your digital health intervention, and the reimbursement. Information on how to write a business model can be found in the annex.
Stakeholder involvement
“It took me so long to find out who is relevant to the project and who to contact about legal and organisational requirements, tariffs systems, IT, Illustration and
animations.”
“Over time I compiled a list of relevant contact persons and their functions beneficial for the project.”
Many researchers bring their discipline-specific expertise and rely on other
professionals’ determination to drive the development of a digital health intervention in healthcare. In research projects, it often occurs that the project design
insufficiently defines relevant stakeholders and their roles. Further, inter-and
multidisciplinary collaborations cause different challenges in sensitive areas such as communication, working culture, and job specification.
Failure to establish crucial communication links makes correct and qualified
information less accessible to the researcher and often causes frustration among the people involved in the project.
Another part of the challenge is to convey the feeling of ownership for all parties involved in the project from a very early stage. In the process, it will ensure that all parties, at the individual researcher level, feel understood, engaged and committed throughout the course of the research project.
Suggestion
The success of digital interventions improve with the involvement of stakeholders.
Therefore, it is important to:
Identify project roles of stakeholders with different perspectives involved in the project.
Ensure that project member, internal suppliers (e.g., software development team or external contractors), target users (e.g., patients, health professionals), and other project-related stakeholders are informed about the project details relevant to their role throughout the entire period. This conveys the feeling of ownership by involving the stakeholders (18).
Analyse the communication flow in the organisation. Try to detect potential issues leading to miscommunication with target users, health professionals and
stakeholders. Create a communication plan in order to focus the message and reach the target audience. The annex provides a resource collection with contact information on relevant organisations, services, tools and information for each region of Denmark and a communication plan template in 9 steps.
Analyse how the digital health intervention effects these stakeholders
Understand it as an iteration process
Plan workshops and dialogues with central groups.
Plan and allocate some time to work for this part of the process (2). The annex includes tools and references with information on how to identify relevant stakeholders and their role (2).
Acknowledgment from the project environment
“[…] since it got recognised in a positive way, everybody is very supportive and a lot of things have happened in these five years, also in the organisation, they are actually prioritising this kind of project more and they are interested in implementing it[… ]”
An identified challenge is to receive positive recognition and support from
stakeholders on meta-, meso- and micro-level in a healthcare system. Research projects, well recognised and supported by healthcare authorities and healthcare settings from the very beginning, have a better chance of being finished in time and being successfully implemented in a healthcare setting (19).
Apart from that, there are challenging external factors, for example, fundamentals of politics, policy changes and decision‐making processes on the macro level. The consideration of unforeseeable external factors requires a certain degree of
flexibility, time and resources in the course of a research project. This collides with rigid organisational structures in a healthcare system and researchers’ limited resources.
Suggestion
It requires detailed information on local organisations, partners and relevant stakeholders who provide expertise and consultancy in different subject areas.
The main and most critical part is to build a mutual understanding and transparency on each other's work, to create a well-functioning, meaningful organisational
structure.
Technical feasibility
“We started from scratch and we were supposed to develop both the psychological content but also the technical content to treat patients. We had no experience with how to develop such platform and what requirements that had to be fulfilled to do so.”
Estimating the complexity of technical solutions is difficult. A lack of technical expertise makes it hard to obtain realistic estimates. Thus, hidden costs and risks are easily overlooked and not included in the pre-project planning.
Suggestion
It is useful to design and conduct a feasibility study very early in a project. A
feasibility study is a common tool to test technical complexity and estimate technical challenges that may occur. In addition, it is recommended to define risks and include a risk analysis and a risk mitigation plan. The risk-analysis helps to identify risk related to technical challenges. A risk mitigation plan helps to manage, reduce and eliminate those risks to a level that ensures that the overall project success is not jeopardized. It is recommended to implement the risk mitigation plan, continually monitor it, and adjust it with the intent of changing the course-of-action if needed.
Legal feasibility
“The technical solution caused most problems, how to fulfil the legal requirements [… ]” For example, if we have a two-step login system,[… ] you need a confirmation, so we have chosen to use NEMID but this took many months to implement this part on the platform because of data security[… ].”
The development of digital health interventions has to oblige legal requirements to support privacy and data security requirements, as well as data protection issues for data sharing to assure patients data security and safety (9). Data security issues are one of the major challenges in health IT (9–11).
Suggestion
Since May 2018, Data protection is governed by the EU general data protection regulation (GDPR). Since then, institutions dealing with personal data have
responsible persons (data protection officers) concerned with data security and data protection. They can help to follow the current EU general data protection regulations (GDPR). Before consulting GDPR experts, you can think about how you are going to get the user's consent on data usage, how and where you want to store the data and how you are going to inform the users about data usage and data storage.
Also, it is recommended to consider information from legal services. Most
researchers working with health IT projects are eligible to use the legal department from the local University and/or from the Region. The annex provides a resource list with contact information for each Region in Denmark, as well as information and references to EU general data protection regulations. Further, there is a reference to a legal assessment tool (LAT) that addresses privacy and data protection issues for data sharing.
Determination of assessment tools and evaluation parameters
“It is important to have meaningful and robust measures that generate evidence and measure the impact of an intervention”
“How feasible is a project? If the need is not big enough or the patient group is very small [… ]
According to all consulted advisors, digital health interventions must be based on a robust study design and scientifically robust assessment tools, including comparable parameters that measure clinical utility/effectiveness. Also, it is important to choose an appropriate assessment tool with meaningful evaluation parameters, as it is important to measure the impact of the digital health intervention in order to receive the necessary funding.
It can be challenging to govern a research project that has to follow traditional research guidelines , as innovative projects often require very flexible and quick changes in strategy and organisation as well as adjustments from theoretical parameters translatable to practice(2). Thus, the challenge is to identify reliable
methodologies for the assessments of digital health interventions in the pre-project phase that comply with both and the delivery of reliable research in an agile,
innovative way.
Suggestion
Being dependent on funding requires early determination of reliable methodologies for the assessments (8) (evaluation and success criteria) of digital health
interventions in the pre-project phase.
The annex provides references to international studies that cover context aware methods (11,12), and tools on how to identify and recruit the right target group/user (13–15), as well as on how to translate information into the own cultural and
organisational context (16) and intervention style and format (17).
It includes concepts and guidelines for summative studies (e.g., randomised (mhealth and ehealth) clinical trials) and formative methodologies.
2. Materialisation
The materialisation phase describes the realisation of the project idea (e.g., health app, novel software for online treatment and rehabilitation). The materialisation phase in a digital intervention project includes the development of the intervention itself.
It can be challenging to resolve mismatches between project planning and concept realisation. Based on the experience of the members of the advisory panel, the three most challenging tasks in the materialisation phase in a project involving digital health intervention are:
1) the cooperation between IT professionals and researchers,
2) involving/recruiting users (e.g., research participants) and consider users’
feedback to improve the usability of the intervention, as well as
3) the re-evaluation and adjustment of the project plan to changing circumstances and unmet needs.
Cooperation between IT professionals and researchers
“At first we experienced difficulties with the IT firm and their developers managing our (...) app, as they were sitting far away from us. When problems occurred, nobody could fix it, and we had to wait sometimes for days to get the app running again.
During that time, we could not collect data, and concerned study participants contacted us. That was unfortunate!”
Researchers highly rely on IT firms who, for example, develop the software for the digital health intervention. IT-related issues are common in projects, and cause delays, for example, due to lack of immediate fixing of technical issues. Researchers fear that, due to technical problems, research data may not be collected and that users get uninvolved with the digital health intervention to a point, where they might drop out of the study. In addition, IT firms are powerful suppliers, and as a layman, it can be challenging to negotiate conditions and prices.
One additional difficulty that frequently arises is the communication between healthcare and IT professionals. Healthcare professionals seem to struggle predominantly with the comprehension of technological challenges, while
IT-professionals seem to struggle with the organisational structure given in a research project. In addition, terminology between the two fields can differ and cause a different understanding of context and content.
Suggestions
The software specification has to be evaluated with the IT firm. It is a good idea to develop a legally binding contract with software specifications on project milestones, including hard deadlines and specifications on the software documentation. It is also important to negotiate future IT support services and possible extensions after the project. The contract has to ensure that project specifications, developed in the inception phase, are implemented correctly. The contract should ensure that in the case of mismatches or incomplete implementation of the specification the IT firm can be held liable.
Get support and help from supervisors and authorities when it comes to contracting and negotiating services with IT companies. The annex provides information on contracting laws and regulations involving a digital health intervention in Denmark.
Design project specifications to define mutual goals, especially with IT professionals.
Hand-drawn mock-ups and basic visualisation of the software and its interface can help to clarify communication.
The development process often reveals different understandings of specifications between the IT developer and the researcher. It is crucial to monitor the
development closely to detect and clarify misunderstandings early in the process in order to avoid further mismatches. This can be especially challenging as the
researchers lack the technical literacy and expertise for the software development.
Agree on an iterative and agile development model with the IT firm to define hard deadlines for evaluating the software at different development levels. Additional control can be obtained through models like Scrum that involve “customer”
involvement where researchers can help to formulate and prioritise the iterative development process (20). One advisor found it a positive experience to directly employ a software developer as part of the research team. In the annex contains a list of recommended resources to get started with agile development.
User involvement and feedback
“It is important to get constant feedback from users to improve the platform and solve occurring problems.”
Testing prototypes in the software development process and gather user feedback in pilot studies is crucial to validate that the digital health intervention matches the expectations and needs of the target group (e.g., Health professionals and/or patients). However, it can be challenging to recruit the representative user groups that give useful feedback and to initiate trials in clinical settings where they have to fulfil clinical, administrative and legal standards.
Suggestion
Contact health providers who have experience with piloting and conducting clinical trials and can support the recruiting process to target the right user group. In the process, prepare, for example, questionnaires to gain insights on user satisfaction and users’ needs. Receiving regular feedback can help to prevent and detect hindering features in the digital health interventions architecture, program and configuration (21) (Tools in annex). Thus, actively involving users is crucial when implementing digital health interventions.
Also, it is important to involve relevant users (e.g., health professional and patients) during the whole process. In this way, you make sure that the product matches the needs of the users. Another benefit of involving health professionals is that it can
Also, it is important to involve relevant users (e.g., health professional and patients) during the whole process. In this way, you make sure that the product matches the needs of the users. Another benefit of involving health professionals is that it can