Pilot-testing

Pilot-testing entails testing comprehensibility, relevance and acceptability of the new instrument among the target population. Feasibility as to the difficulty and length of the questionnaire should also be tested (2). Pilot-testing provides important information to identify aspects that need adjusting in the

questionnaire to ensure that it will be applicable in health and social care contexts.

16.1.5.1 Pilot test 1: Expert panel

The first draft of DeCANT was tested using the content validity index (CVI) by asking an expert panel to evaluate relevance, comprehensiveness and comprehensibility (114,115). This evaluation provides

information which can be used to reduce the number and adjust formulation of items. Also, information on feasibility in the settings for use is possible.

16.1.5.1.1 Settings

Representing as many perspectives as possible, participants were sought from among informal dementia carers and professionals in both primary and secondary health care, nongovernmental organisations.

Carers with particular interest in reflecting on their carer role were sought from the participants in Study 2.

Professionals were considered experts within their field, working with specific problems related to dementia health and social care, including collaboration with carers. Experts with different professional backgrounds were desirable in order to get a variety of perspectives on carers’ support needs.

16.1.5.1.2 Participants

A pragmatic approach was undertaken to recruit experts from the networks of the PhD student and supervisors. A minimum of three and up to 10 experts are recommended to be recruited based on specific

46

criteria about who qualifies as an expert or member of the expert panel (115). Four categories of experts were identified, and at least one participant representing each expert category was required. Criteria for each category were:

- Informal dementia carers: A person who provides help on a regular basis because of a personal

relationship rather than financial compensation to a person with dementia, able to communicate in Danish, and >18 years old.

- Nongovernmental organisations: An experienced specialist who works to protect the interests and rights of people with dementia and their carers.

- Primary health care: A professional attending to the health needs of people with dementia and carers through promotive, protective, preventive, curative, rehabilitative and palliative efforts in the local setting of a municipality (116).

- Secondary health care: A professional attending to the specialised health needs of people with dementia and carers in the setting of a dementia outpatient clinic or hospital (117).

16.1.5.1.3 Data collection

An email was sent to each expert encouraging them to give suggestions for improvement. The experts evaluated relevance, comprehensiveness and comprehensibility of the items on a scale ranging from 1=’Not relevant’ to 4=’Highly relevant’ (118,119) independently.

16.1.5.1.4 Analysis

Experts’ scores for each item were dichotomised as relevant (ratings 3-4) or not relevant (ratings 1-2). An Item content validity index (I-CVI) was calculated for each item by summing the dichotomised scores. The proportion of experts in agreement was calculated and kappa statistics were used to measure agreement (118). Criteria for evaluation of agreement followed the recommendations of Fleiss et al. (120). Items with kappa below 0.75 were considered for removal or adjustment (also see Paper III).

16.1.5.2 Pilot test 2: Target population

Cognitive interviews were used to test how the target population responded to DeCANT and if any adjustments were necessary, as seen from the personal and subjective perspective of carers (114).

16.1.5.2.1 Settings

All participants were recruited through key professionals in primary and secondary health care who had a professional relationship with the person cared for. Carers were therefore known to the system because of the person with dementia.

47

16.1.5.2.2 Participants

Purposive sampling (97,121) of carers was used based on the same criteria as for carers in Study 2. A minimum of ten participants was necessary to be able to interpret on face and content validity (114).

16.1.5.2.3 Data collection

Interviews were conducted using a combination of verbal probing and Think-Aloud method (2,114).

Participants were asked to think aloud while filling out DeCANT. The interviewer was not allowed to comment or help. Probing questions were subsequently asked regarding comprehensibility, relevance, completeness, acceptability and feasibility. A semi-structured interview guide was prepared to ensure that specific issues were talked about e.g. if the introduction was relevant and comprehensible and if some words were too technical/professional. At this point in the development process, it was also possible to investigate the meaning of any discrepancies between the findings in the scoping review in Study 1 and the focus groups and individual interviews in Study 2 regarding comprehensiveness of the items. All interviews were audio-recorded.

16.1.5.2.4 Analysis

Interpretation of the data was carried out using deductive content analysis (90,91). Using a deductive approach entailed categorisation of collected data into predefined categories to investigate face and content validity of DeCANT. A categorisation matrix (see Table 4) was developed to code the data based on the key elements of content validity (122). Only content fitting the matrix was included in the analysis (91).

Table 4 Categorisation matrix used for coding of data in the cognitive interviews with carers

Comprehensibility Relevance Comprehensiveness What do carers think is important to ensure

that DeCANT is an adequate reflection of carers’ support needs?

Based on the results of the deductive analysis, items were adjusted in discussion between the PhD student and her supervisors. Emphasis was put on carer’s personal and subjective views of what was important.

16.1.5.2.5 Ethical considerations

All interviews were carried out by an experienced interviewer (THC) in the safe environment of their homes. Also, the interviewer made sure that carers knew whom to contact if questions arose as a result of the interview. Interviews were conducted according to the Declaration of Helsinki (108), and participants gave their informed written consent before participating. Data collection and management were conducted in accordance with the General Data Protection Regulations and registered at the Danish Data Protection Agency (2015-57-0016-020a). In agreement with Danish law, no ethics approval was required (109).

48

16.1.5.3 Pilot test 3: Electronic distribution

A third pilot test was conducted to ensure feasibility of electronic distribution, because DeCANT was only tested on paper in Pilot Test 2. Distribution of DeCANT using an electronic platform will become relevant in the following field-test study requiring a larger sample. Also, it may be relevant when used in future health and social care due to its ease of use.

16.1.5.3.1 Settings

Electronic pilot-testing was conducted in the general population to investigate whether it was feasible to fill in DeCANT in digital form using various private electronic devices such as smart phones, tablets and

computers.

16.1.5.3.2 Participants

Purposeful sampling (97,121) (resembling convenience sampling) in the PhD student’s network was

conducted, representing different types of digital users based on the following criteria: age range (young to old), educational background (short to long) and use of electronic device (smart phone, tablet or

computer). A minimum of 10 tests was desirable to be able to identify any technical challenges or issues of comprehension.

16.1.5.3.3 Data collection

An email with a link to DeCANT in digital form was sent using REDCap electronic data capture hosted by the Odense Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark (123,124).

Participants were given written instructions to fill in DeCANT and comment in free text their thoughts on its comprehensibility and feasibility. If participants expressed any problems, a follow-up telephone interview was conducted.

16.1.5.3.4 Analysis

Registration of successful responses were tracked and frequency distribution was calculated. Qualitative analysis of written comments was conducted following the same rigorous data analysis process as described in Pilot Test 2 (91). Findings pointing to needed adjustments of DeCANT were implemented before field-testing.