Original Research

A scoping review of digital open badge ecosystems in relation to resource-constrained environments

Preia Motheeram, Marlien Herselman, Adele Botha

Received: 07 Aug. 2017; Accepted: 19 Jan. 2018; Published: 26 Mar. 2018

Copyright: © 2018. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This article presents a scoping review of digital open badge ecosystems. A scoping review can be seen as a methodology for systematically assessing the breadth of available research literature. This article is aimed at identifying the nature, extent and current understanding of digital open badge ecosystems. Scoping reviews follow many of the same methodological steps as systematic reviews where the reliability of results and the potential for replication remain essential. As such, this review follows the eight steps outlined by Okoli and Schabram. From the 1608 studies identified, 41 were considered relevant for inclusion criteria and thus included in a final sample. Some relevant websites and the so-called grey literature were included. These were evaluated according to the inclusion criteria and were categorised. Further subcategories were identified which include the notion of digital badges (n = 35) and open digital badges (n = 18), elements of a digital open badge ecosystem (n = 27) and reported implementations (n = 29). The scoping review presents a literature repository that can be used by practitioners and researchers interested in the application of digital open badges for the establishment of a digital open badge ecosystem. The review also serves to highlight the potential benefits this can have for resource-constrained environments.

Introduction

The purpose of this article is to present a scoping review for systematically assessing the breadth of available research literature regarding the nature, extent and current understanding of digital open badge ecosystems. The results of the scoping review allow for a deeper understanding and better conceptualisation of implementation-specific valuable lessons which can be applied to a digital open badge ecosystem for a resource-constrained environment. The objectives are to (1) apply a scoping review for the literature study; (2) clarify the specific concepts: digital badges, open badges and digital open badge ecosystems; and (3) indicate how digital open badge ecosystem can be relevant to resource-constrained environments.

The pursuit of knowledge and skills has steadily and vigorously expanded, forcing pedagogies to follow in its progressive footsteps (Mayrath 2012). The learning environment is no longer limited to the classroom or governed by seat time (Hess 2011; Morrison & DiSalvo 2014). Knowledge and skills are created, shared and valued in various different ways which continuously breach the context in which current education systems were developed and standardised. Alternate learning opportunities include Internet-based projects, self-directed tinkering, community participation and on-the-job training (The Mozilla Foundation 2012). The knowledge, achievements and skills earned outside the traditional formal education system are often either overlooked or not accredited (Mudavanhu 2015; Rooyen 2011). Outcomes, such as achievements in after-school classes, extra-curricular activities, work-based experience and mentoring, to name but a few, are often not recognised in formal education accreditation and therefore seldom acknowledged as a valuable accomplishment.

This article addressed the following research question: How can a scoping review on digital open badge ecosystems allow for the conceptualisation of such a system for resource-constrained environments?

Badges are often used to recognise learning (that which an individual knows), skill (that which an individual can do) or a role in an organisation or community (that which an individual has become) (Randall, Harrison & West 2013). Gibson et al. (2015) argue that badges and badging systems are emerging phenomena used to incentivise learners to engage in positive learning behaviours, identify progress in learning and content trajectories as well as signify learning and achievement. Digital badges are icons and image files which serve as visual metaphors for skills, roles, accomplishments, experiences and interests (Gibson et al. 2015) and have been touted as a common currency to signify learning outcomes across all contexts (Bowen & Thomas 2014).

In 2011, the Mozilla Foundation, with the support of the MacArthur Foundation, launched the Open Badges Project (Mozilla Foundation 2011). Mozilla Open Badges offers a free, open-source digital badge infrastructure to allow institutions, endorsers and learning providers to issue badges and for job seekers and learners to collect and display said badges (Muilenburg & Berge 2016). The Open Badges Project provides an accreditation system which intends to unlock new career and educational opportunities by promoting the recognition of skills and achievements earned through formal and informal learning (Knight et al. 2014).

The following section offers an overview of the methodology followed.

Method

Brien et al. (2010), applying the argument of Arksey and O’Malley (2005), suggest that scoping reviews are a relatively new research tool through which domain-specific literature overviews can be performed. They posit that scoping reviews are similar to systematic reviews in some respects, as they provide a method through which a body of literature can be consolidated. Brien et al. (2010) provide a number of distinguishing characteristics of scoping reviews in comparison to traditional systematic reviews (see Table 1).

Table 1 indicates the differences between systematic and scoping reviews. Brien et al. (2010) highlight the focused nature of the systematic review in that it mostly attempts to answer a clearly defined question through the use of explicit methodologies to assess the quality of included articles. Scoping reviews, on the other hand, are by and large conducted to examine the extent, range and nature of research activity in a particular field Brien et al. (2010:2). Scoping reviews thus focus on the research findings (Lambert 2006) and not on how the literature was obtained (Arskey & O’Malley 2005) or an assessment of its quality (Sandelowski, Docherty & Emden 1997).

Scoping reviews follow many of the same methodological steps as systematic reviews including the essential tenets of result reliability and replication potential (Glasziou et al. 2001; Linde 2002; White & Schmidt 2005). Ellis and Levy (2010) define a systematic literature review as a reproducible procedure performed to identify, evaluate and synthesise quality literature using an explicit algorithm. They argue that a systematic literature review, conducted with sufficient scientific rigour, can provide a solid grounding for a research area and the selection of a suitable research methodology.

Scoping review aims to review relevant literature for systematically assessing the breadth of available research literature whilst describing the nature, extent and current understanding of digital open badge ecosystems. It was operationalised through Okoli and Schabram’s (2010) eight steps: (1) identify the purpose, scope and goal; (2) develop a protocol with specific steps and procedures to be followed; (3) search and eliminate studies that do not meet the requirements; (4) screen the retrieved studies according to relevance and defined criteria; (5) screen the remaining studies according to the quality of the articles; (6) systematically extract applicable information; (7) aggregate, analyse, discuss and organise studies; and (8) report on the findings in detail to ensure that the systematic literature review can be independently reproducible.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (Moher et al. 2009) guidelines, embodied in the four-phase flow diagram, were implemented to guide steps 2–5 of Okoli and Schabram’s (2010) eight-step model (see Table 2).

This was particularly relevant to address the purpose and objectives of this article.

Data sources

After taking the purpose of the scoping review into consideration, search keywords were identified and used. These keywords were ‘digital badges’, ‘digital badge ecosystem’ and ‘open badges’.

The keywords were traced using Google Trends (Figure 1) to identify a search data range. The results displayed a shared spike in interest from April 2011 to January 2015. Accordingly, the search was limited to studies, articles and books published between 2011 and 2016. The keyword search strategy used the following search terms: ‘digital badges’ or ‘badge ecosystem’ or ‘digital badges ecosystem’ or ‘open badges’ and ‘framework’ or ‘implementation’. The word ‘framework’ was used as a subcategory, as it was found that digital badge frameworks existed and this was also applied to refine the search. The word ‘environment’ was used as a subcategory and was only applicable to how digital badges were used in resource-constrained environments. An initial search of only educational databases Educational Resource Information Centre (ERIC) (via EBSCOhost), Professional Development Collection, Professional Development Collection and Academic Search Premier rendered results with a pedagogical implementation in professional practice and was thus deemed outside the scope of the study. The search was consequently extended by not limiting it to specific discipline or database and using frequently cited publications as a measure of credibility. The keyword search was redone using Harzing’s Publish or Perish software (Harzing 2010), which offered a substantial set of publications as well as the number of citations per publications. A high number of citations suggest that a publication has impacted significantly the field. A low number of citations may, but not necessarily, suggest a limited current impact. A low citation per publication metric can be attributed to a limited or recently developed working field or a restricted audience as a result of the article having been published in a language other than English (Harzing 2010). The use of Harzing’s The publish or Perish Book (2010) allowed the research to assess the impact of the publication and thus the resultant conceptualisation of a domain understanding. The assumption made was that articles, which were cited more often, were inclined to influence this domain understanding. Researcher bias is acknowledged in this step of the process, as the researcher had to make judgement calls regarding the perceived value of specific articles. This value call was guided by the h factor (Harzing 2010) and the research purpose. The keyword search yielded 1586 results as shown in Table 3.

FIGURE 1: Google Trends search results for keywords.

After removing duplicate studies, the remainder were screened for eligibility using a method similar to systematic review. Studies were assessed for eligibility based on responses to the questions detailed in Table 4. It was decided to limit the study language to English, as it is the home language of all three authors and the cost of translating the paper exceeded both budget and time constraints. If the publication was not written in English and the response to question 1 thus ‘No’, it was immediately excluded. If the response to any of questions 2–5 was ‘Yes’, the publication was deemed relevant and shortlisted for quality appraisal as indicated in Table 4.

Approximately 58% of the articles related to Bio Med Central (BMC) Public Health, which presented limited relevance to the research purpose. Hence, publications were excluded if they were not written in English, if they fell in the BMC Public Health field and if the content was not pertinent to the research purpose (see Figure 2).

FIGURE 2: Results of eligibility assessment.

The remaining 167 eligible studies were examined in detail. The quality appraisal of the publication was not considered as an inclusion criterion; however, the publications were broadly assessed for relevance. After screening for duplicates and eligibility, the quality appraisal identified 41 publications which were then used in the final review. Figure 3, the PRISMA four-phase flow diagram as relevant to this research, illustrates the flow of information through the phases of searching, evaluating and filtering.

FIGURE 3: Preferred reporting items for systematic reviews and meta-analyses flow diagram.

A standard coding template was used to extract data from each of the subsequent selected 41 publications. Four subcategories emerged: digital badges, open digital badges, elements of a digital open badge ecosystem and reported implementations.

In an effort to systematically assess the breadth of available research literature and to identify the nature, extent and current understanding of digital open badge ecosystems, a summary of the publications and their focus areas is presented in Table 5 under the four subcategories that emerged.

A qualitative descriptive approach as described by Sandelowski (2000) and applied by Weeks and Strudsholm (2008) was used to review the results, grouping together statements that linked to each of the identified subcategories (Sandelowski, Barroso & Voils 2007).

The outcome is presented and discussed in the following sections.

Results

This section presents a synthesis of the statements linked to the identified subcategories: digital badges, open digital badges, elements of a digital open badge ecosystem and reported implementations.

Mind maps were used to organise and summarise the most important points and concepts. Having met stricter inclusion criteria, the 41 publications were processed using the guidelines outlined by Ellis and Levy (2010:192), namely know the literature, comprehend the literature, apply, analyse, synthesise and evaluate. These mind maps are presented in Figures 4 and 5 with literature linked through the article number as allocated in Table 5.

FIGURE 4: Overview of digital badges as extracted from the selected literature.

FIGURE 5: Literature linked to case studies and implementations.

Digital badges

Devedžić and Jovanović (2015) argue that a badge is an image that serves as an indicator of skills, competencies, interests, achievements or hierarchy acquired over time and across all contexts. The affordances of digital badges can be categorised into four key areas:

• motivation – to foster discovery, promote engagement, drive the acquisition of knowledge and incentivise learning (Gibson et al. 2015; Knight & Mozilla Foundation, 2012; Muilenburg & Berge 2016; Randall et al. 2013; Tran, Schenke & Hickey, 2014)
• recognition and credentialing – to validate, measure and accredit knowledge and skills gained across all contexts of learning, to build and formalise an identity and reputation, to symbolise an association with a community or group (Davis & Singh 2015; Gibson et al. 2015; Law et al. 2015; Pedro et al. 2015)
• evidence of achievement – by linking a digital badge with metadata containing evidence of skills, accomplishment or knowledge, creating a granular representation of capabilities, to map an individual’s progress (Hole 2014; Ifenthaler et al. 2016; Pedro et al. 2015)
• research – to form a pathway model for achievement, to make learning more transparent and accessible (Davis & Singh 2015; Sullivan 2013; The Mozilla Foundation, 2012).

Digital badges can be embedded with metadata which provide information regarding the issuer, recipient and why the badge had been awarded, along with the associated evidence. The granularity associated with awarding badges for competencies creates a broader representation of an individual’s capabilities, thus allowing him or her the opportunity to present a more complete picture of his or her competencies by signalling specific critical skills.

The value of the digital badge is backed by the issuing authority and the decided assessment of recognition. A competence- or educational-based assessment must be linked to evidence of activities, learning, experiences, artefacts and skills development (Gibson et al. 2015:404; Mayrath 2012:46). This is an important issue for resource-constrained environments as well (objectives 2 and 3 of the article). Badges are issued in accordance with success criteria which vary in definition, level, weight, quality, rigour, motivation and reward. A single badge may be designed and issued by more than one authority, making the badge achievable via multiple paths and assessment options.

In conclusion, this article will adopt the following definition for a digital badge: a digital badge is a validated representation or indicator of an accomplishment, competency, interest, affiliation, experience or skill that is visual and available online (Gibson et al. 2015; Jovanović & Devedžić 2015).

Open badges

Building on the merits and potential of digital badges and the badging systems, Mozilla and the MacArthur Foundation developed the Open Badges Project. This badge system displays the following key elements: badges, assessment, collecting and sharing tools as well as criteria and evidence. Mozilla Open Badges has refined the concept of the digital badge, as a static image, by hard-coding metadata on the badge which communicates the specific skill or achievement (Knight & Mozilla Foundation 2012:8). This provides a level of security and reliability (Randall et al. 2013). The metadata details, illustrated in Figure 6, include: the badge name, issuer, endorser, competency statement, performance criteria, method of assessment, evidence of performance, date issued and standards with which the badge is aligned (Badges/FAQs – Mozilla Foundation 2014a).

FIGURE 6: Open Badges anatomy – Badge metadata.1

Although learning is similarly validated via standards, evaluation and evidence, in contrast to the current top-down and closed accreditation system, the open badges accreditation system uses a bottom-up, open and distributed approach (Knight & Mozilla Foundation 2012). The Open Badges badging process is outlined below (Mozilla Foundation 2014b):

• A badge issuer creates a certifiable badge and makes it available online to their audience of earners.
• When an earner meets the defined criteria of a badge, the badge is awarded to an earner, and the earner can choose to store the badge in his or her Backpack.
• The Backpack is used to accumulate and manage badges, allowing earners to select privacy and publishing preferences (Goligoski 2012; Niehaus et al. 2017).
• Earners can share and display authenticated badges publicly on social network sites, blogs, profiles and resumes.
• Potential employers, recruiters, organisations and peers can view badges that are displayed publicly and click on the badges to view the badge metadata (Crafford & Matthee 2016; Ifenthaler et al. 2016; Jabbar, Gasser & Lodge 2016).

The following definition is adopted for an open badge: open badges are standards-based badges that have hard-coding metadata on reliability and securely communicating a specific skill or achievement (Knight & Mozilla Foundation 2012; Randall et al. 2013).

Digital open badge ecosystem elements

The literature documenting digital badge ecosystems is relatively scarce and, with the exception of the Mozilla Open Badges ecosystem, consists mainly of single institutionally localised ecosystems. These seem to function within a single institution, enabling the use and application of open digital badges for institutional consumption.

A digital badge ecosystem allows tertiary institutions and employers to bridge the identified skills shortage gap with badge ecosystems that are developed around higher education outcomes (Pearson Education 2013). The Mozilla Foundation, Peer 2 Peer University and MacArthur (2012) have indicated that a definition of digital badge ecosystem has to include descriptions of badges, assessment and infrastructure. The infrastructure of a digital badge ecosystem needs to provide an online, open and decentralised vehicle to issue, collect and display badges online. This is operationalised as a badge backpack.

Universities and higher education institutions can leverage the connected learning outcomes to contextualise an individual’s achievements and readily demonstrate the return on investment and economic impact of their programmes (Jovanović & Devedžić 2015). Universities and education providers who endorse badges will acknowledge and credit the learner’s soft skills and job-ready competencies, and therefore increase the transparency of learning pathways (Glover & Latif 2013). A badge ecosystem can help students make better-informed decisions through connected learning pathways between tertiary education providers and the career-specific skills required by employers (Crafford & Matthee 2016; Itow & Hickey 2013). For employers and advanced education providers, the required skills, competencies and outcomes, articulated through badges, will simplify the communication of changing needs to the learners, tertiary institutions and training providers (Pearson Education 2013). As a result, curricula can easily be tailored to suit the market needs and afford learners the opportunity to better equip themselves with a more market-responsive skill set.

The infrastructure must accommodate the issuing of badges from all contexts of learning environments (e.g. schools, universities, online learning, professional bodies, employers, non-formal education and learning providers); the display and management of badges to afford the learner the power to control what badges are publicly displayed; the impact of metadata attached to a badge; the authentication of badges to verify their validity; the interface to external websites to support badge display and the endorsement of badges to signify the weight and value of a badge.

The Open Badge Infrastructure (OBI) defines three user roles for the open badges ecosystem (Mozilla Foundation 2014b) as the badge earner, the badge issuer and the badge displayer.

From the above literature overview, the following components are identified for an open badge ecosystem, which are regarded as the final components that should also be considered for resource-constrained environments as well when using digital badges (see the ‘Digital open badges ecosystem for resource-constrained environments’ section):

• the badge issuer
• the badge (badge metadata)
• the badge earner
• the badging infrastructure (to facilitate the badging process)
• the badge displayer.

Existing implementations of digital badges internationally

There are several small-scale, mostly institutional, initiatives which have already implemented open badges to incentivise learning and introduce micro-credentialing to increase the scope of learning recognition:

• Makewaves is a safe, social learning platform for schools that provides social media for education to develop digital literacy skills, learn about Internet safety and how to use social media responsibly. It has formed a community of thousands of schools sharing their creativity and raising achievement with badges (Makewaves 2016).
• Kent University has used badges to validate and capture competencies gained through workshops for faculty professional development (Nestor 2014).
• Aurora Public Schools (APS) are working with community colleges and regional employers in the state of Colorado to award badges for 21st century skills and to gain endorsement for these badges (Smore 2016).
• DigitalMe is using the Open Badges platform to work together with teachers, charities and employers to transform the learning landscape and prepare young people with skills and competencies to enable real-world opportunities. Current badge programmes include ‘Badge the UK’ which allows learners to showcase all achievements using digital badges, and ‘Young Carers in Focus’ which empowers young carers to share stories and campaign for change (DigitalMe 2016).
• Newport City (Wales) Homes provides community services to residents and has created a set of badges for key competencies that are linked to industry standards, thus developing and rewarding employee skills (Price 2014).
• The University of California is aware that traditional testing and grading would not accurately reflect the outcomes of its agriculture and food systems curriculum, and linked competency badges to the outcomes defined in collaboration with targeted employers (Muilenburg & Berge 2016).
• The Grading Soft Skills (GRASS) project develops innovative pedagogical approaches to support and represent the soft skills of learners. GRASS is investigating the idea of digital badges and open badges to formally measure, validate and recognise these skills (The GRASS Project 2016).
• Providence after School Alliance (PASA 2016) launched a badge system targeted at the Providence youth to expand and improve learning opportunities by motivating, tracking, recognising and validating learner interests to create connected learning pathways.
• Badge Europe! (2016) is an international initiative working towards promoting open badges as the infrastructure to provide formal and informal learning recognition, to increase the transparency and quality of learning outcomes and to create opportunities for learning, employment and social inclusion.

Digital open badges ecosystem for resource-constrained environments

The experiences, merits and challenges documented, contain valuable lessons which can be applied to conceptualising a digital open badges ecosystem for a resource-constrained environment. Resource-constrained environments in context are environments characterised by these economic, technological and cultural conditions: low-income communities where power and network connectivity are scarce and expensive with low bandwidth, and which also present unique cultural constraints where people are unfamiliar with technology or are technophobic (Niehaus et al. 2017). The following should be considered:

• Randall et al. (2013) stated that the use of expert assessment before a badge is awarded, albeit a reliable evaluation of high quality comes at a high price and financial sustainability should be treated as a major concern. Rosewell (2012) proposed partnering with existing Open Educational Resource (OER) providers and using existing evaluation models to obviate the cost implications of designing a robust validation and assessment rubric.
• The design of the digital open badges ecosystem needs to stimulate and sustain motivation in badge earners. Haaranen et al. (2014) attested that a simple design reduced the learning curve for student badge earners and encouraged learning; however, providing more statistics for earners (e.g. leader boards and notifications about badges earned) might increase motivation. In a pilot test, Pedro et al. (2015) discovered that users of the SAPO Campus (the major Internet portal and ISP in Portugal) badging systems (developed at the University of Aveiro, Portugal) felt more engaged and motivated during participatory learning exercises as a group or community.
• Learners embrace informal learning and the value paid for certification and recognition from a Massive Open Online Learning (MOOC) provider. As experienced by Law et al. (2015), a challenge exists with demonstrating digital badges as a currency of achievement and credibility and not just as a motivational tool. On the contrary, Davis and Singh (2015) reported that the credibility of badge metadata and the gaining of recognition for skills earned across contexts were met with enthusiasm and are a major drawcard for badge adoption.
• A shared challenge concerns the assessment element of a badging system, specifically the success criteria for conceptual learning, critical thinking, creativity and other soft skills (Mewburn et al. 2014; Myllymäki & Hakala 2014; Niehaus et al. 2017; Randall et al. 2013; Rosewell 2012). Abramovich, Schunn and Higashi (2013) claimed that the ‘potential benefit of an assessment is determined by its ability to both maintain learning motivation and accurately communicate a student’s learning’. Competency-based learning, which a badge will validate, has to reflect and validate the true meaning of the competence or skills gained. Reputation is built on valid, verified and quality judgements – not only for badge earners but also for the badges themselves and the medium of exchange within which they have to have currency. The openness of such a system is regarded as essential to long-term sustainability so that exporting and importing of badges can be modified and adapted by the open source community (De Villiers & Sauls 2017; Meyer & Marais 2015; Niehaus et al. 2017).
• It is also important that validating institutions and organisations are not attached to commercial constraints when doing validation. The validation process of badges should minimise the amount of data necessary for checking badges whilst also mapping progress, signalling reputation beyond the community where it was earned and incentivising learners to engage in pro-social behaviours (Crafford & Matthee 2016; Jabbar et al. 2016; Muilenburg & Berge 2016).
• The ability to set an expiration date on a badge has been beneficial for badges that are awarded for the mastery of skills in a field that is rapidly changing. The badge expiration date, if not carefully determined, can result in a lack of interest, making earners less likely to collect this badge because of its lack of longevity (Myllymäki & Hakala 2014; Randall et al. 2013).

This section addressed the research question of this article. It is evident that a digital open badge ecosystem can meaningfully contribute to accreditation challenges in resource-constrained environments if the above issues are addressed appropriately.

Conclusion

Knight and Mozilla Foundation (2012:1), the senior director of learning at Mozilla, describes our current accreditation systems as a ‘shared monopoly across education where you have to go down a very prescribed path to get learning that quote-unquote counts’ and that Mozilla wants to ‘open that up’. Digital badges can advance the reimagining of accreditation practices by measuring competencies accurately and with a finer granularity than seat time and formal degrees do. Mozilla’s Open Badges offers the infrastructure to create and develop a digital badge ecosystem that incentivises and recognises accomplishments and experiences in all learning contexts. This is particularly relevant for resource-constrained environments as these badges can assist them to show their own credibility on social media. A digital open badges ecosystem can potentially create a connected learning environment where skills, achievements and competencies are fully recognised and accredited in specifically resource-constrained environments so that, once community members have a digital identity, their paper-based badges can be replaced with digital badges which acknowledge their competency learning.

Acknowledgements

The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.

P.M. was the main author and mostly did the literature searches and the literature study. M.H. wrote the methodology section and contributed to the overall editing of the article. A.B. contributed to the scientific rigour of the literature, the structure of the article and the referencing.

References

Footnotes

1. Used under CC-BY-ND License.