Limitations and Insights on The Effects of GDEs in Education

Table of Links

Abstract and 1 Introduction

2. Background and Related Work and 2.1. Gamification

2.2. Game Design Elements and 2.3. Gamification Effects

2.4. Related Work on Gamification Negative Effects

3. Systematic Mapping and 3.1. The Research Questions

3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria

3.4. Applying the Search Strategy

3.5. Data Extraction

4. Systematic Mapping Results

5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements

5.1. Context and Participant Characterization

5.2. Focus Group Design

5.3. The Developers’ Perception on The Negative Effects

5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects

5.5. Participant Feedback

6. Limitations

7. Concluding Remarks

7.1. Future Research Directions

Acknowledgements and References

6. Limitations

One of the limitations of our mapping study concerns potentially missing papers. After analyzing 3458 papers (see the list in our online repository), based on our inclusion and exclusion criteria, we initially included 79 papers. While our combined search strategy allowed identifying significantly more papers than the database search strategy employed by Toda et al. [27] (we identified 34 papers published between 2012 and 2016, against 17), the sets had differences. Therefore, we manually included the papers found by Toda et al., which were missed by our first search and extended searches, ending up with a final set of 87 papers.

It is noteworthy that we verified that many of the missed papers would also have been retrieved by subsequent snowballing iterations. Nevertheless, an extension applying subsequent snowballing iterations and investigating different hybrid strategies [15], would require significant additional effort, involving analyzing several thousands of papers which could characterize a different publication instead of an extension. We are confident that our final set of included papers as part of this publication allowed providing an unbiased and meaningful overview of the adverse effects related to GDEs in gamified education software.

Another risk of false negatives concerns the filtering process. We screened all papers considering only titles, abstracts, and keywords, which may not contain sufficient information to decide upon inclusion. We avoided applying EC1 and EC3 during the initial screening to lower this risk, only excluding papers that we had high confidence of not investigating GDE effects (EC1) and reporting negative effects (EC3). In case of any doubt, the paper was left for full text-based assessment. Moreover, the application of the inclusion and exclusion criteria made by the first author was reviewed by the second and fourth authors in meetings. In case of doubt, during the initial screening or full-text-based assessment, discussions were held to reach a consensus.

Furthermore, we chose not to consider grey literature as part of our inclusion criteria. There is the possibility of relevant grey literature that does not have equivalent in non-grey academic papers. On the other hand, even though we did not explicitly evaluate the strength of evidence, the results herein reported are based on peer-reviewed research and backed by empirical studies.

Moreover, while research that reaches negative results is essential because it shows us what does not work [115], there still seems to be a publication bias towards positive outcomes. Research reporting negative results tends to have less scientific interest, fewer citations, and to be less often published [116]. Hence, there may be additional negative results that were not published and which, for that reason, could not be included in our mapping study.

Finally, in our summary of the negative effects of GDEs (cf. Figure 4), we included negative effects that may affect different roles (e.g., the students and the maintainers/teachers responsible for configuring the system and keeping it up and running with appropriate educational resources), as we found them reported in the literature. Our goal was to organize evidence regarding the overall adverse effects of GDEs in gamified education software. While we included an overview of the negative effects for different roles (e.g., RQ1.B and RQ1.C), we considered a fine-grained analysis of the effects of each GDEs for each of these roles out of the scope of our intended overview and part of future work.

With regard to the focus group session, we wanted to gather insights on the mapped negative effects of game design elements from the point of view of practitioners. In a virtual focus group, generally, the number of participants is reduced, four being considered appropriate (our case) [117]. However, it is suggested to plan more online focus groups with fewer participants than when conducting face-to-face groups [118]. While we carefully planned our focus group, followed best practices, and carefully conducted our qualitative analyses, a single session is surely a limitation. Therefore, we interpret the focus group results as preliminary complementary discussions with practitioners, avoiding validity claims. Nevertheless, we had rich discussions that nicely complemented our literature study findings. It is noteworthy that these discussions already revealed interesting facts, such as the potential unawareness of developers about the negative effects of GDEs, and some example arguments on the perceived pros and cons of including the GDEs in gamified software.

Still, we are aware that a single focus group is not be enough to reach generalizable findings. Unfortunately, identifying teams with experience in developing gamified software and willing to collaborate with academia is not a trivial task. Therefore, conducting new focus group sessions is part of future work.