Discovering Open Access Engineering Journals: An Examination of Indexing Rates for DOAJ and Non-DOAJ Content

James Bierman
Engineering Librarian
University of Oklahoma
Norman, Oklahoma
jbierman@ou.edu

Abstract

Introduction

Open access (OA), as a core principle, is very easily one of the most significant and impactful movements within scholarly communication in the last decade. The first wave of open access journals launched in the early and mid-2000s, and in the intervening years, the number of open titles has risen dramatically (Laakso et al. 2011; van Noorden 2014). These new publication venues now provide scholars expanded choices for the sharing of their research. For the moment, few open journals confer prestige to their authors by masthead alone. Scholars, therefore, may take into account other considerations when scrutinizing open access publication outlets, notably those of reach and impact. A journal's reach and impact can be governed by several internal factors, including qualities of editing, graphics, marketing savvy, and its track record for attracting influential contributions. Outside of these internal variables, a journal's reach and impact may be aided by the availability of its content. On this point, open journals have an advantage over their traditional subscription-based counterparts, those requiring readers to navigate through pay walls, either through institutional affiliation, access fees, or interlibrary loan. However, before a research article can be accessed and read, it must first be discovered. For the STEM disciplines, publications that are indexed and discoverable by major commercial databases in the field have a key advantage over those publications that are not.

This study examines the indexing of 187 open access engineering journals by four prominent science and technology databases: Compendex, Web of Science, Inspec, and Scopus. The purpose is to determine how the reach and impact of open access research is affected by discoverability within prominent engineering databases.

Literature Review

The discoverability of open access publications is not an entirely new field of inquiry. Fisher et al. (2009) examined the indexing of 127 Directory of Open Access Journal (DOAJ) listed journals, 96 in the area of library and information science and 31 in music. This was a follow-up study to an earlier effort (Yontz and Fisher 2007), so the authors were able to compare indexing rates over time, finding that 60 journals were indexed by at least one database in 2009 while only 36 were indexed by at least one database just two years earlier. The authors also found a correlation between journal age and indexing rate, noting that journals that started before 1990 were indexed at the highest rate (81%) while the newest journals, those established 2005 or later, had the lowest indexing rate (38%), positing that it takes time for a new journal to be picked up by major indexing databases. Researchers have examined the discoverability of OA content in other subject areas. Hart (2010) analyzed the indexing of open access law journals (n=61), Evans et al. (2011) analyzed open access art journals (n=31), Poulin and Tomaszewski (2014) analyzed OA journals in communication studies (n=147), and most recently Testa (2016) analyzed OA journals in music (n=84). Cummings (2013) took a much broader approach to open access analysis by downloading the full DOAJ journal list and looked for indexing on three multidisciplinary databases, Ebscohost Academic Search Complete, Proquest 5000 International, and Gale Onefile, reporting indexing percentages of 25.7%, 5.3%, and 2.5%, respectively.

While the discovery and indexing of content is the focus of this study, the open access movement has faced many challenges, from questions regarding the integrity and sustainability of various business and distribution models (e.g., "gold OA" versus "green OA"), as well as concerns about the rigor of OA peer review and overall quality of the research. In short, "gold OA" refers to peer-reviewed, published content funded by the author via article processing charges (APCs), while "green OA" most often refers to preprint and/or postprint content hosted by institutional or discipline-specific repositories. Suber (2015) provides an excellent resource for a nuanced look at these OA models. Cusker and Rauh (2014) examined the author-pay model from the author's perspective by surveying 123 STEM faculty, postdocs, and graduate students, finding both uncertainty regarding author fee requirements and "cautious optimism" about open access journals in general. Beall (2012) provided the most ardent criticism of the author-pays model, arguing that it exposed authors to predatory publishers, that is, publishers who engage in unethical practices resulting in compromised scholarly output.

Berger and Cirasella (2015) described efforts to police open access content as "blacklisting," which would include efforts such as the recently decommissioned Beall's List, and "whitelisting," the creation of title lists made up of open access titles that have been vetted and verified. The authors cite the DOAJ and Open Access Scholarly Publishers Association (OASPA) as two high profile whitelisters. For that reason, the DOAJ was used as the source listing of open engineering journals in this study. OA publishers wishing to list their content with the DOAJ must meet basic standards and best practices, which cover aspects such as peer review, accessibility, copyright, and operational transparency (DOAJ 2018a). OASPA (2018) and the Committee on Publication Ethics (COPE 2018) also provide OA best practices.

Methods

The first list of open access titles was pulled from the DOAJ web site in late 2013. This initial title list was assembled using all journals from five subject areas: Chemical Technology (34 journals), Electrical and Nuclear Engineering (65 journals), Industrial Engineering (23 journals), Materials (44 journals), and Transportation (31 journals). These five subject areas captured the majority of engineering-related journals listed by the DOAJ at the time. The impetus for this exercise was simple: to see if these open access titles were being indexed by Compendex or Web of Science. Although a few notes were committed to paper, the investigation remained informal and unpublished. A concern at the time was that many of these open access titles were relatively new, and that it might be problematic to give too much weight to some of the low indexing numbers that were noted at the time. Low indexing rates may not be due to an open journal's lack of quality or to a database producer's willingness to index open access content, but to the simple fact that some titles had not been in the marketplace long enough to be picked up by leading indexers. So the title list was set aside for four years and revisited in late 2017, providing more than enough time for databases to index (or decline to index) their content.

The starter title list had a total of 197 open access journals. After accounting for title mergers and redundancies, i.e., publications listed multiple times under titles both old and new, the final title count dropped to 187 open access journals. Indexing checks were first made against Compendex, on the Engineering Information Village platform, using the ISSN search function. If the search successfully retrieved the journal content, results were sorted by date to see if content for the current year (2017) was indexed. If a search failed, a second search was made by journal title. Next, this process was repeated on Web of Science, again using the ISSN search option on the Advanced Search tab, then moving on to a title search if necessary. Inspec and Scopus were not accessible to the researcher, therefore OA journal indexing checks were made against each provider's publicly available indexing lists, IET (2018) and Elsevier (2018). All reported results for Inspec and Scopus were based on these title lists; indexing could not be independently verified.

Last, searches for the journals were performed on the DOAJ web site to see if they were still listed in the directory. If a journal was not found on the DOAJ web site, possible title and ISSN changes were considered before making a final determination of dropped coverage. Journal titles that were dropped by the DOAJ were moved to a new title list so that their indexing could be compared with those titles that continue to be listed in the DOAJ.

Results and Discussion

Publication and DOAJ Status

Before examining journal indexing, it is worthwhile to consider attrition rates, both in terms of publication status and presence in the DOAJ web site. Of the 187 journals, 41 (22%) have either ceased or are inactive. A journal was determined to be inactive if no content was published in the previous twelve months--the 2017 calendar year, in this case.

Figure 1 shows that of the 187 journals, 92 (49%) are no longer listed in the DOAJ web site. Having already noted a 22% attrition rate in this sample, it would not be hazardous to venture that journal title cessation or inactivity was a contributing factor in the steep decline in DOAJ coverage. However, the DOAJ provides a Google Spreadsheet (DOAJ 2018b) that lists both new titles added and titles dropped, including explanatory notes for those titles that were dropped, so there is little need for conjecture. Of the 92 engineering titles that are no longer listed by the DOAJ, 38% were removed in May 2016 "...because the publisher failed to submit a valid reapplication within the given time frame." Other reasons for removal include "suspected editorial misconduct by publisher," journal inactivity/publication ceased, and "Journal not adhering to Best Practice." The remaining 13 journals fell into Other, which includes four titles that were dropped for non-functioning URLs, one for lacking an OA statement, one for requiring registration, one for lack of content, and six journals that were indeterminate, i.e., not listed in the spreadsheet.

Figure 1. Attrition of Open Titles Listed in DOAJ, 2013 to 2017

OA Journal Indexing

Appendix A provides title-by-title indexing notes for the 95 journals that are currently still listed in the DOAJ web site. This includes titles that are active, inactive, inaccessible, and ceased. In the journal status column, an asterisk (*) next to the status denotes DOAJ indexing that is not current to 2017. Two asterisks (**) indicates that an entry for the journal was found in the DOAJ web site, but no content at the article level was available. As discussed earlier, several journals in this list were removed from the DOAJ web site for inactivity or lack of content, so the absence of article-level content may be a signal that the journal will soon be dropped from the site. However, an explanation for journals with no article content is provided in the DOAJ FAQ page: "If a journal in DOAJ does not have any articles linked to it, this is simply because the article metadata hasn't been provided to us by the publisher or group responsible." Checkmarks in columns "Comp", "WoS", "Inspec", and "Scopus" indicate some degree of article-level indexing by the four databases -- Compendex, Web of Science, Inspec, and Scopus; indexing may not cover a journal's full content or be current through 2017.

Appendix B provides title-by-title indexing notes for the 92 journals that were dropped from the DOAJ web site. An examination of the indexing reveals a lack of checkmark density in comparison to the indexing seen in Appendix A.

Figure 2 provides an indexing overview of all 187 open access journals in this study, by database. While none of these indexing figures are particularly impressive, it should be noted that this sample includes 41 ceased or inactive journals. Further, 92 journals (49%) were dropped from the DOAJ.

Figure 2. Percentage Indexed, All Open Access Journals (n = 187)

Figure 3 accounts for both of those considerations by focusing on those open journals that are still active and by dividing all active titles into two subcategories, those that are still listed in the DOAJ and those that have been dropped by the DOAJ. As one might expect, the percentage indexed for all databases improved when the ceased/inactive journal titles were removed from the sample (Figure 2 vs. Figure 3, All Active). Figure 3 also shows wide margins between active titles that are listed in the DOAJ and active titles that were removed from the DOAJ: Compendex indexing dropped 26%, while Web of Science dropped 37%, Inspec 23%, and Scopus 38%.

Figure 3. Percentage Indexed, Active Open Access Journals

Figure 4 provides a view of aggregate indexing for active journals currently in the DOAJ and for active titles that have been dropped from the DOAJ. For active titles currently listed in the DOAJ, 87% are indexed by at least one of the four benchmark databases, 70% are indexed by two or more databases, and 49% are indexed by three or more databases. Active journals not listed by the DOAJ are 52%, 36%, and 13%, respectively.

Figure 4. Aggregate View of Percentage Indexed, Active Open Access Journals

This analysis of 187 open engineering journals began with one simple question: Are the major science and engineering databases indexing open access scholarly output? As the analysis proceeded, several new queries emerged. How volatile is the open access journal marketplace, specifically in the engineering disciplines? Does that volatility, to whatever degree, affect the discoverability of open research? Will the DOAJ serve as marker of, or perhaps, a check against, the supposed volatility of this scholarly marketplace? While these questions cannot be answered from this relatively small sample, the journal list pulled from the DOAJ four years previous has some insights to offer, beginning with the evidence of the DOAJ's efforts to maintain a directory that meets a standard of best practices. The rapid growth of the open access marketplace is well known, but less known, perhaps, is the DOAJ attrition rate.

The 187 title sample used for this study represented most of the engineering related titles that were listed by the DOAJ in late 2013. Using a four-year-old title list for an investigation provided a few advantages. First, by excluding the very newest open access journals from the study, the indexing advantage of established journals versus non-established noted by Fisher et al. (2009) was minimized. Second, the aged list made it possible to see a stark contrast between the indexing rates for titles that remained current in the DOAJ to the indexing rates for titles that had, in the intervening four years, been removed from the DOAJ.

Although this study is unable to pin down a causal relationship between an open access journal's DOAJ status and its indexing by some of the major databases, it does uncover a correlation between the two. OA journals that are currently listed in the DOAJ are indexed at a higher rate by Compendex, Web of Science, Inspec, and Scopus, than titles that were dropped by the DOAJ. This is true even if all ceased and inactive journals are removed from the sample. For example, Scopus, the database that indexes the highest number of OA titles in this study, indexes 65 of the 82 (79%) active titles that are listed in the DOAJ, but only 30 of the 64 (47%) active titles that were dropped by the DOAJ. Similar indexing patterns for the other three databases can be seen in Figure 3. This opens up the proposition that OA journals that are listed in the DOAJ are more likely to be indexed by leading databases, at least in the science and engineering disciplines.

Only one database, Scopus, indexes at least half of the 187 journals in this study, with Web of Science coming in second with 37%, Compendex third with 30%, and Inspec last with 17%. Those percentages improve slightly when focusing exclusively on those journals that are actively publishing new content, but it is Figure 4, which shows the indexing of active journals in aggregate, that offers real encouragement. Seventy-one of the 82 (87%) active, DOAJ-listed journals are indexed by at least one of the four benchmark databases. As expected, the indexing figures are much lower for active journals that are not listed in the DOAJ. The most remarkable difference can be seen with comparisons at the three or more indexing level. Of the 82 active, DOAJ listed journals, 40 (49%) are indexed by three or more of the benchmark databases, while only eight of the 64 (13%) active, non-DOAJ titles enjoy that same level of indexing. This further demonstrates, if only in the engineering disciplines, a correlation between DOAJ status and database indexing.

Conclusion

A key finding in this study was the discovery of a correlation between journal indexing by the major databases and DOAJ status. The unique sampling in this study had, however, at least one notable drawback. When the engineering titles were pulled from the DOAJ site in late 2013, it was meant to be a comprehensive representation of the site's engineering content. Four years later, the sample represents less than half the number of engineering journals listed by the DOAJ. Future studies will feature a more comprehensive sample size and may bring into focus other considerations, such as publisher outlet, OA funding models, and journal impact factor. In the interim, engineering researchers looking for OA outlets for their research are encouraged to focus on journals listed by the DOAJ.

References

Beall, J. 2012. Predatory publishers are corrupting open access. Nature 489 (7415): 179. DOI: 10.1038/489179a

Berger, M. & Cirasella, J. 2015. Beyond Beall's list. College & Research Libraries News 76(3): 132-135. Available from: https://crln.acrl.org/index.php/crlnews/article/view/9277/10342

COPE. 2018. Core Practices. Available from: https://publicationethics.org/core-practices

Cummings, J. 2013. Open access journal content found in commercial full-text aggregation databases and Journal Citation Reports. New Library World 114 (3/4):166-178. DOI: 10.1108/03074801311304078

Cusker, J. & Rauh, A.E. 2014. A survey of physical sciences, engineering and mathematics faculty regarding author fees in open access journals. Issues in Science and Technology Librarianship 78. DOI: 10.5062/F4VH5KTQ

DOAJ. 2018a. Principles of Transparency and Best Practices in Scholarly Publishing. Available from: https://doaj.org/bestpractice

DOAJ. 2018b. Frequently Asked Questions: About. Available from: https://doaj.org/faq#list

Elsevier. 2018. How Scopus Works: Content. Available from: https://www.elsevier.com/solutions/scopus/how-scopus-works/content

Evans, S., Thompson, H., & Watkins, A. 2011. Discovering open access art history: a comparative study of the indexing of open access art journals. The Serials Librarian 61: 168-188. DOI: 10.1080/0361526X.2011.592666

Fisher, J., Hart, E., & Yonyz, E. 2009. Are they being indexed II? A follow-up to tracking the indexing and abstracting of open access journals. Proceedings of the Charleston Library Conference. DOI: 10.5703/1288284314740

Hart, E. 2010. Indexing open access law journals...or maybe not. International Journal of Legal Information 38 (1): 19-41. Available from: https://scholarship.law.cornell.edu/ijli/vol38/iss1/5

IET. 2018. Documentation: Free Lists of the Journals Covered by Inspec. Available from: https://www.theiet.org/resources/inspec/support/docs/index.cfm

Laakso M., Welling P., Bukvova H., Nyman L., Bjork B-C. & Hedlund T. 2011. The development of open access journal publishing from 1993 to 2009. PLoS ONE 6(6): e20961. DOI: 10.1371/journal.pone.0020961

OASPA. 2018. Principles of Transparency and Best Practice in Scholarly Publishing. Available from: https://oaspa.org/principles-of-transparency-and-best-practice-in-scholarly-publishing-3/

Poulin, S. & Tomaszewski, R. 2014. Open access journals in communication studies: indexing in five commercial databases. Behavioral & Social Sciences Librarian 33 (3): 3-14. DOI: 10.1080/01639269.2014.866019

Suber, P. 2015. Open Access Overview. Available from: http://bit.ly/oa-overview

Testa, M. 2016. Availability and discoverability of open-access journals in music. Music Reverence Services Quarterly 19 (1): 1-17. DOI: 10.1080/10588167.2016.1130386

van Noorden, R. 2014. Open-access web site gets tough. Nature 512 (7512): 17. DOI: 10.1038/512017a

Yontz, E. & Fisher, J. 2007. Are they being indexed? tracking the indexing and abstracting of open access journals. Charleston Conference Proceedings 2007: 126.

Appendices

Appendix A: Indexing of Journals Currently Listed in DOAJ (PDF)

Appendix B: Indexing of Journals No Longer Listed in the DOAJ (PDF)