PAPER 1 42
6. Framework Evaluation
70 identification of segments and segment groups can also foster the publication planning for the involved researchers. In this case, it was the aim to produce at least one major publication for each research scope.
71
# Title Year Journal Size searchRe questions Artifacts
Related to segments
1
CyberGate: A Design Framework and System for Text Analysis of
Computer-Mediated Communication 2008 MISQ
M 3 5 (1,3,1), (1,3,3), (2,2,1), (2,2,2), (2,2,3)
2
Making Sense of Technology Trends in the Information Technology Landscape: A
Design Science Approach 2008 MISQ
S 2 4 (1,2,1), (1,2,2), (2,2,1), (2,2,2), (2,3,1)
3 Process Gramma as a Tool for Business
process Design 2008 MISQ
S 2 4 (1,3,1), (1,3,2), (2,2,1), (2,2,2)
4 The Design Theory Nexus 2008 MISQ
M-L 2 4
(1,1,1), (1,1,2), (1,2,1), (1,2,2), (1,3,1), (2,2,1), (2,2,2), (2,3,1),
(2,3,2) 5
Using cognitive principles to guide classification in information systems
modeling 2008 MISQ
S 1 3 (1,1,1), (1,2,1),
(2,1,1), (2,2,1)
6
Knowing What a User Likes: A Design Science Approach to Interfaces that
Automatically Adapt to Culture 2013 MISQ
M 4 7
(1,1,3), (1,1,4), (1,2,2), (1,2,3), (1,3,2), (1,3,3), (1,3,4), (2,2,1), (2,2,2),
(2,2,3), (2,3,1) 7
Bridging the gap between decision-making and emerging big data sources: An application of a model-based framework to disaster management in Brazil
2017 DSS
M 3 4 (1,2,2), (1,2,3), (2,2,1), (2,2,3)
8
Counterfeit product detection: Bridging the gap between design science and behavioral science in information systems research
2017 DSS
M 3 7 (1,2,1), (1,3,1), (1,3,2), (2,2,1), (2,3,3)
9
A permissioned blockchain-based implementation of LMSR prediction
markets 2019 DSS
S 3 7* (1,1,1), (1,1,2), (1,2,1), (1,2,3), (1,3,1) 10
Operationalizing regulatory focus in the digital age: Evidence from an e-commerce
context 2019 MISQ
S 1 2 (1,2,1), (1,3,1),
(2,2,1), (2,3,1)
Table 6. Overview of Analysed Papers
72 Figure 12 shows the instantiation of the segmentation framework for paper #6. It refers to the most complex DSR study found and nicely demonstrates the complexity of real DSR projects.
Four research sub-questions were identified that span the RQD. Seven different artifacts related to eleven segments.
Figure 12. Installation of the Segmentation Framework for Paper #6
Table 6 demonstrates that it was possible to instantiate the framework and to identify separate artifacts and their relation to specific segments for each reviewed study. The overall applicabilit y (criterion a) and the framework's usefulness to separate a DSR project into well-defined segments (criterion b) can therefore be confirmed. No studies were identified where applying the framework was assessed to be unsuitable. However, several observations and limitations should be considered when interpreting the evaluation results. It was assumed that enough information could be gained for the evaluation from reviewing the selected articles. This was the case in general, but the level of detail related to the DSR projects, research questions, and designed artifacts varied significantly among the analyzed studies. A trend was observable to more precise and informative descriptions in newer publications, whereas the information, especially for those published as part of the MISQ
73 special issue in 2008, was partly rather rud imentary. The more details were provided, the easier it was to establish the segmentation framework.
Usually, several publications result from a single DSR project. It is unlikely that all research sub-questions relevant for a specific DSR project are identified by reviewing a single research paper resulting from that project. This is likely to be the case, especially for those projects from which only one or two research sub-questions could be identified (#2 to #5 and #10). The instantiated frameworks most likely only provide an overview of parts of the underlying DSR projects.
The cubes representing the instantiated frameworks were sparse. On average, just a third of the identified segments could be linked to artifacts. This is not surprising because a single publication usually only refers to a subset of created artifacts in a DSR project and a specific research scope as reported in related empirical studies (Daeuble et al., 2015; Werner et al., 2014) and not the complete output of the underlying DSR project.
The literature review confirmed previous empirical studies as it revealed that DSR studies usually produce multiple artifacts (Werner, 2019). Several artifacts and sub-artifacts were identified for each study, which is individually listed in Appendix A. This observation emphasizes the inherent complexity of DSR projects. An identified artifact commonly related to several segments (#1, #2,
#4, #5, #6, #7, #9, #10) and different artifacts can relate to the same segment (observed in all reviewed studies), which is in line with the relationships modeled in Figure 7. 39.13 percent of identified artifacts were instantiations and thus level 1 contribution according to Gregor and Hevner’s (2013) classification scheme. 60.87 percent of identified artifacts were constructs, methods, and models constituting level 2 contributions. Although no level 3 contributions in terms of design theories were identified, it could be observed that 45.65 percent of the artifacts contributed to a scientific knowledge base. 60.78 percent of the described artifacts contribut ed to the application domain.16 A higher degree of imbalance may have been expected when taking into account discussions about the nature of contributions and the sometimes discussed lack of theoretical contributions generated by DSR (Avison & Malaurent, 2014; Beck et al., 2013). It also clearly demonstrates that theory is not necessarily a prerequisite for creating scientific knowledge.
16 10,87 percent of identified a rtifa cts contributed to the a pplica tion doma in a s well a s to the knowledge ba se .
74 The majority of identified artifacts related to the system level (69.57 percent), whereas relatively few artifacts related to the infrastructure level (19.57 percent) and about a third to the usage level (32.61 percent).17 This finding is in line with the understanding of ISR as an interdisciplinary research domain located between its reference disciplines, computer science and business economics (Keen, 1980), where less emphasis exists regarding technical issues that relate to fundamental infrastructural aspects, which would primarily fall into the domain of computer science. However, it remains unclear why only a relatively small number of artifacts relate to the usage level. This may be an interesting investigation for future research.
Although the knowledge base's contributions were comparatively balanced on average, this was not the case for all individual studies. All artifacts described in study #9, for example, are primarily related to the application domain, none to the knowledge base. As the cubes representing the instantiated frameworks were sparse, gaps in the underlying research projects in terms of contributions to the knowledge base or application domain at different levels can easily be detected. When inspecting the instantiated framework for study #6, for example, it could be considered whether it is possible to derive generalized knowledge from the designed web application prototype and its evaluation for clustering user behavior due to cultural background.
This may be a relevant contribution to the field of cultural classification or social anthropology in general. The framework can also be used to assess at a detailed level whether the output is balanced on the different dimensions or whether additional research may be necessary to fill in important gaps or at least to consciously decide not to cover certain aspects as described in the case study discussed before.