610 Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior among Researchers and Master’s Students in Engineering Yasuko Hagiwara, Emi Ishita, Yukiko Watanabe, and Yoichi Tomiura* This study focuses on differences in the scholarly search approaches of researchers and students in contemporary information environments by examining use of academic resources and elements in search results (such as title or abstract) to identify scholarly search knowledge and skills. To examine how users select documents for different search situations, an online survey of resource selection and the use of search result elements was conducted among 48 researchers and 40 master’s students in the field of engineering in Japan. The results show that researchers use resources and search result elements differently for each situation, while students lack that ability. Introduction Literature reviews are an indispensable part of research. An effective scholarly search using bibliographic databases is important for finding academic papers related to the research topic. There are important differences between a scholarly search and an ordinary information search. They involve the use of different sources and different results assessments. When conducting an ordinary information search, users are often satisfied by reviewing several results from a search engine.1 In the case of a scholarly search, users mainly access bibliographic databases and specific academic resources.2 When carrying out a Web search, researchers understand reliable sources.3 Researchers conduct an exhaustive search frequently to confirm the novelty of their study and evaluate all relevant documents.4 After selecting appropriate databases, researchers need to check many search results5 as well as the elements in those results (such as title, author name, or abstract) to decide which documents to read. Changes in the digital environment and possible means of information access may affect how users select academic resources and check the search results. For example, these days, * Yasuko Hagiwara is Librarian in Shinshu University Library; email: 8giwara@gmail.com. Emi Ishita is Associate Professor in the Department of Library Science, Graduate School of Integrated Frontier Sciences, at Kyushu University; email: ishita.emi.982@m.kyushu-u.ac.jp. Yukiko Watanabe is Associate Professor and Deputy Director of Adminis- tration in Kyushu University Library; email: watanabe.yukiko.935@m.kyushu-u.ac.jp. Yoichi Tomiura is Professor in the Department of Informatics, Faculty of Information Science and Electrical Engineering, at Kyushu University; email: tom@inf.kyushu-u.ac.jp. ©2022 Yasuko Hagiwara, Emi Ishita, Yukiko Watanabe, and Yoichi Tomiura. Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 611 researchers also use Google in addition to various academic databases provided for scholarly search.6 For search result elements, the full text is usually preferred in addition to the title and abstract.7 This preference may be encouraged by some databases (such as Web of Science and Scopus), which display links to full-text versions in the search results. Google Scholar also provides PDF links for direct access to full-text versions, if available. This functionality has emerged through the spread of online bibliographic data and e-journals. Experienced researchers adapt their scholarly search approaches to the current digital environments; thus, full-text versions are easily accessible, and the use of search engines has expanded. Some studies show that students who have just started research struggle to adjust to search environment changes. For instance, students find it hard to search for academic information us- ing general search engines because of the large number of results and their uncertain reliability.8 Other research shows that many students are overconfident in their search skills; in reality, they struggle to find proper resources.9 They also lack research-related search skills such as check- ing documents from nonacademic resources or using important figures to identify and select relevant documents.10 These difficulties may arise because students are familiar with ordinary information searches but lack the specific skills and knowledge needed for scholarly search. By studying and documenting the knowledge and skills used by experienced researchers, our work will support students in performing scholarly searches more effectively and efficiently. The present study examines differences between the current scholarly search methods of researchers and graduate students. Researchers’ behavior varies11 in different academic fields: different databases are appropriate for each field. Our study focuses on engineering: it has the largest number of graduate students in Japan,12 and engineering researchers tend to adopt the latest search technologies.13 Among graduate students, we regarded doctoral students as researchers and master’s students as research beginners. In Japan, master’s students are ex- pected to conduct research at a higher level than simply taking courses; they write a master’s thesis for graduation in general. Thus, master’s students are at the beginning of their careers and need to acquire the knowledge and skills to conduct scholarly searches. Accordingly, we compared those students with researchers. Our goal was to identify among research beginners the knowledge and skills necessary to undertake scholarly searches (such as understanding which elements need checking to efficiently select documents to read in full-text format among many search results). The ACRL Framework for Information Literacy for Higher Education14 offers a comprehensive model for teaching students about evaluating information. However, our focus was scholarly search behavior. In a preliminary investigation,15 we conducted an observational study with 10 researchers to explore their search behavior with academic data- bases. In the present study, we surveyed researchers and master’s students: RQ1, to identify what information resources they used when searching for relevant documents; and RQ2, to determine how they selected which documents to read in full-text versions. This study contributes to understanding current user behavior and contemporary issues in scholarly search. The results also have implications for the improvement of library services including information literacy education. Literature Review This study focused on two issues: 1) what information resources (that is, search engines or databases) the subjects used; and 2) what search result elements they used to select docu- ments. Several previous studies have addressed the first issue in the areas of science, technol- 612 College & Research Libraries July 2022 ogy, engineering, and mathematics. Survey results about database selection have revealed that researchers use Google in addition to academic databases.16 Interview studies among researchers found they employed Google and Google Scholar in different situations, such as beginning a new research project,17 conducting literature reviews,18 and finding information related to specific problems.19 Athukorala et al.20 revealed that researchers use different data- bases depending on the purpose of their information search. They used interviews, the diary method, and observations to study six information engineering researchers and identified the following six search purposes: keeping up to date with research; exploring unfamiliar research areas; collaborating; reviewing literature; preparing lectures; and recommending materials for student reading. Subsequently, they conducted a survey of these search purposes among 76 researchers. According to their results, researchers use Google Scholar most for “staying up to date with research” and “collaborating” while they use Google the most for “exploring unfamiliar research areas” and “reviewing literature.” The results indicated that different databases are used depending on the search purpose. In the present study, we examined the use of databases for specific search purposes. In an observational study, we found that the search process appeared to be related to the research stage (such as beginning research and writing papers).21 The questionnaire in the present study determined whether the purposes of the search and the research stage influenced the choice of databases; it also assessed the search results. With regard to the second issue of assessing documents from their search result elements, the following three studies found that title and abstract were important. Hsin22 interviewed three groups of researchers having different years of experience and found that experienced researchers tended to rely on title and abstract to choose documents relevant to their re- search. Wang et al.23 revealed that the four elements most frequently used to assess document popularity and target audience were title, abstract, journal, and author. Macedo-Rouet et al.24 observed researchers’ search strategies using PubMed. This study showed that 84 percent of participants checked abstracts after reviewing titles, and 34 percent of them tried to access the full text of the reference list. Jamali et al.25 also found that article content (in other words, figures/tables or methods) available from a full-text version was equally or more important than the publication journal and author in selecting a document. Thus, in an environment of- fering easier access to full-text versions, the process for evaluating search results might differ from the past. In this study, we include full text as a search result element and also explore which full-text structures researchers consider. Many related studies on search result elements have investigated researchers and doctoral students. The present study also examined master’s students, and it compared such students’ results with those of researchers in terms of scholarly search skills. We considered that our approach would help identify information resources and document selection behavior based on the methods adopted by the users. We believed that our findings would help research beginners learn experienced researchers’ techniques. Methodology Large-scale surveys are typically employed for the study26 and log analysis27 of scholarly search behavior. They are useful in identifying the overall situation related to such behavior. By con- trast, the present study sought to determine scholarly search behavior with respect to research stage and search purpose. Thus, we conducted a preliminary observational study between Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 613 November 2015 and September 2016 among 10 researchers who demonstrated their document search processes with respect to their information needs.28 We recruited those participants (five faculty members, five doctoral students) from the fields of information science and engineering. From the results of that observational study, we developed questions concerning academic resources and checked elements among the search results. We also referred to questions used in related studies with respect to respondent demographics, background experience in scholarly searches,29 and elements.30 The questionnaire consisted of 14 items: respondent demograph- ics (3 items), background experience in scholarly search (3 items), resources for obtaining documents (3 items), checked elements in search results (3 items), and others (2 items). Some question items were based on the observational study or on questions from related studies. See appendix for the full questionnaire. To understand respondent profiles, we asked about the following three items: academic position/doctoral program/master program, department, and period of research experience. For information on participants’ search background, we asked three questions about their satisfaction with finding documents, their experiences par- ticipating in a library workshop, and their database search experience. To learn about the respondents’ use of resources, we asked about information resources used routinely and about databases used for scholarly search (multiple responses allowed for each question). From the observational study, we learned that participants distinguished search needs by the research stage they faced and chose information sources accordingly. We used the insights that emerged from interviews31 to define three research stages: beginning research; considering appropriate methods (to solve problems); and writing papers. For example, one participant who had started a new project about half a year previously, explained, “I am unfa- miliar with this field, so I will look for the standard literature cited in article references selected among the search results.” We considered this beginning research. Another participant stated, “I’d like to use new materials for my experiments, so I’ll search for methods that employ such materials.” We regarded this as considering appropriate methods. Yet another participant com- mented, “I want to search for articles about patented devices because I am writing my paper”; that participant then selected a database to begin the search. We categorized this as writing papers. The observational study also revealed that most (7/10) participants searched several databases. Therefore, the relevant survey question included databases mentioned by participants, such as digital libraries run by academic associations (such as IEEE or ACM), Google Scholar, and Google in questionnaire options in addition to subscription databases (such as Web of Science). To understand how researchers assess their search results, we asked about their use of different search result elements. These elements included bibliographic items (such as title or abstract) and other Web of Science result elements (such as research field or cited counts). Based on the observational study,32 we included three additional elements. First, “full text” was a factor, which was also mentioned in prior studies.33 Respondents who selected “full text” were then asked which structures or portion of the full text they reviewed, such as the introduction part, references, figures, or tables. Second, “image” was also identified as a fac- tor, following a participant in the observational study who performed an image search with Google and used this to select documents he would read in full. Third, “online availability” was also indicated as a desirable feature because we found that some users decided to read only if they could access an online full text. Respondents were asked to indicate their use of these options according to the purpose of their search. From the observational study,34 we determined that the purpose of search 614 College & Research Libraries July 2022 would influence which search result elements were important. For example, a participant in the study explained, “I’m looking for previous studies. Since I’m not familiar with this field, I don’t know much about authors in this field. I use journal titles to judge whether their fields are related.”. The survey offered three purpose options, elicited during the observational study: 1) “to identify appropriate research methods”; 2) “to find previous research in an unfamiliar field”; and 3) “to learn about a trend of research.” For reference purposes, we asked respondents about the order in which they used search result elements (multiple answers were allowed) and the frequency of their use of subscrip- tion database functions. Here we present characteristic findings of information resources and search result elements. Respondents and Data Collection The survey was conducted with two groups (a researcher and a student group) from the Graduate School of Information Science and Electrical Engineering, Kyushu University and the Graduate School of Electrical and Computer Engineering, Shinshu University. Both schools focus on research and education in information science and electrical engineering. The two universities have more than eight schools; thus, their libraries subscribe to databases in many fields; we assumed that would eliminate database selection bias owing to lack of access. That is why we chose those two universities. Through the mailing list of each school, we sent an invitation to faculty members and graduate students to participate in our survey. Respondents were asked to fill out an online survey form (Google Forms) between October 20 and November 10, 2017. Informed consent (see appendix) was obtained individually from each respondent before they answered the questionnaire. We obtained 88 responses (48 researcher responses and 40 student responses), for an effective response rate of 10.4 percent (see table 1). Results Resources for Obtaining Documents Figure 1 shows the strategies used for finding documents (multiple answers were allowed). In the researcher group, the most highly selected item was “References in academic articles” (87.5%); it was followed by “Search engines (e.g., Google)” (70.8%) and “Google Scholar” (66.7%). Those items were also found to be popular in previous studies.35 By contrast, in the student group, the most used strategy was “Recommendation by supervisors, colleagues, and/ or laboratory members” (82.5%); that rate was overwhelmingly higher than among research- ers. Students also used Google Scholar (72.5%) and Search engines (62.5%) at similar levels to researchers. Academic databases were commonly used by researchers (62.5%) but less so TABLE 1 Sample Size and Response Rate Number of Surveys Sent Number of Respondents Response Rate Total 847 88 10.4% Researcher 304 48 15.8% Faculty 163 41 25.2% Doctoral Student 141 7 5.0% Master’s Student 543 40 7.4% Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 615 FIGURE 1 Use of Resources for Obtaining Documents 616 College & Research Libraries July 2022 FIGURE 2 Primary Information Resources by Research Stage *Scopus and World Contents are available only for Kyushu University members; accordingly, we asked such members about using them (29 researchers, 21 students). **JDreamIII and EBSCOhost are available only for Shinshu University members; accordingly, we asked such members about using them (17 researchers, 13 students). Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 617 by students (42.5%). Moreover, researchers tended to seek out information through various resources that students largely overlooked. For example, 64.6 percent of researchers chose “At- tending conferences,” which means documents are found through attending presentations or through discussion at conferences, compared with only 12.5 percent of students; 52.1 percent of researchers chose “Conference proceedings” versus 7.5 percent of students; and 47.9 percent of researchers chose “Specific journals” compared with 7.5 percent of students. These results indicate that, in engineering, researchers use a wide variety of strategies; master’s students use fewer. Instead, students obtained relevant documents recommended by supervisors and laboratory members rather than searching directly. These findings also suggest that students need to better understand how to use these research resources toward obtaining relevant documents by themselves. For example, students need to learn that academic databases, proceedings, and academic society journals are important resources for research. Primary Information Resources by Research Stage Figure 2 shows primary information resources for each group depending on the research stages. Note that only 80 of total 88 respondents had experience with database searches, so the rates are based on n=46 for researchers and n=34 for students. At the beginning stages of research, both researchers and students use Google highly (80.4% and 70.6%, respectively). This is followed by Google Scholar (researchers; 69.6%, stu- dents; 64.7%). These findings are consistent with previous studies showing that Google is the most frequently used database among researchers36 and employed for scholarly search when research is just getting started.37 The same pattern is present in these results, but Google usage decreases with research progress. Researchers select digital libraries provided by academic or professional associations more often when exploring appropriate methods (67.4%), and when writing papers (65.2%). Unlike researchers, students used Google and Google Scholar more than other sources at any stage of research, and they made limited use of digital libraries compared with these two popular resources. Use of Search Results Elements by Search Purpose Figure 3 shows the findings related to elements in the search results when the respondents checked to select documents to read for three different purposes. The total number of respon- dents was different for each purpose and the rates are based on the respondent numbers shown in table 2. Separately, we have reported this result for researchers only.38 As previous studies showed,39 title and abstract are important elements for both groups, but here we highlight the differences between the practices of researchers and students. When researchers want to explore research methods, they focus on search results with “Full-text” (82.6%) and then “Keywords” (54.3%). In exploring research methods, students TABLE 2 Number of Respondents (Use of Search Result Elements per Search Purpose) Search Purpose Researchers Students Research methods 46 33 Unfamiliar field 44 27 Research trends 43 30 618 College & Research Libraries July 2022 FIGURE 3 Use of Search Result Elements, by Search Purpose Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 619 made less use of the following elements than researchers: “Journal title” (6.1%) “Author” (15.2%), and “Document type” (6.1%). Because understanding the research field was required to assess document relevance by the journal title and author, students rarely used those two elements. Further, if students lack experience in reading academic articles, they may not have known what document types existed. When seeking previous research in an unfamiliar field, researchers favor checking “Re- search area” (40.9%), “Journal title” (40.9%), and “Cited counts” (34.1%) to find core articles in a field. These elements are used less for other search purposes. When examining current research trends, both researchers and students check “Publication year” (55.8% and 70.0%, respectively). This element is indispensable to consider when users wish to discover the latest articles. These results are not surprising; indeed, they support insight into the detail and trends of scholarly search behavior, even with the relatively small number of respondents. Use of Full-text Structures by Search Purpose Figure 4 shows the percentages of each full-text structure checked when deciding whether to read the document more carefully after seeing the full-text version. This question was asked of respondents who selected “full-text” as one of the search result elements that they typically checked, and the rates are based on the number of respondents shown in table 3. We have also previously reported this result for researchers only.40 Here we focus on which full-text structures should be considered an important part of document selection skills for students. Although students indicated that they check “Introduction” most frequently for any search purpose, researchers check different full-text structures depending on their search purposes. When searching for research methods, 78.6 percent of researchers check “Method” and 76.2 percent check “Figure, table, and formula.” In other words, both skimming methods and checking charts seem to be useful to identify appropriate research methods. For finding previous research in an unfamiliar field, 89.7 percent of researchers check “Introduction” and 64.1 percent consider “Related work” compared with other purposes. When seeking infor- mation about current research trends, fewer than half of researchers consider “References” (46.2%). Because cited articles are older than the current article, they would rarely be suitable as a source for learning about the latest information on a topic. These results indicated that the experienced researchers understand which structures are most relevant for each search purpose, in the same way that they consider the relevance of differ- ent information resources. In contrast, students demonstrate fewer skills for document selection. Self-assessment of Scholarly Search Ability We found differences between researchers and students in terms of their use of resources and their ability to use search result elements to evaluate documents for selection. Figure 5 TABLE 3 Number of Respondents (Use of Full-text Structures per Search Purpose) Search Purpose Researchers Students Research methods 42 30 Unfamiliar field 39 23 Research trends 39 20 620 College & Research Libraries July 2022 FIGURE 4 Use of Full-text Structural Elements by Search Purpose FIGURE 5 Self-assessment of Search Ability Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 621 shows the results of respondents’ self-assessments about their ability to search effectively on their own. Students appeared to have more difficulty in finding relevant full-text docu- ments. Among researchers, more than 80 percent claimed that they could find “Almost everything” or “About 3/4” of what they need, whereas less than 50 percent of student respondents felt the same. It is likely that, because students lack knowledge and skills to search effectively, compared with researchers, they are less satisfied with their ability to find suitable documents. Discussion We conducted this survey to examine resource use at each research stage and determined the search result elements for different purposes. The results showed differences between researchers and master’s students. In terms of information resources, researchers used digital libraries provided by academic associations for specific research stages (such as considering appropriate methods and writ- ing papers); students tended to use Google and Google Scholar at any stage. Students might be satisfied with Google Scholar: it has links to access full-text versions if they are available. The results indicated that students need to understand the various resources that experienced researchers use: digital libraries, core journals, prominent authors, and preprint servers re- lated to different research fields. Most digital libraries provide search functions for anyone, although they allow only their members to access full-text versions. Students need to know that digital libraries are important resources and can search them. The most popular resource among researchers was references; thus, it would be useful to teach students how to choose relevant documents from a reference list. Students tended to confirm elements that could be judged easily, such as publication year. Researchers—unlike students—also typically checked different elements in the search results and full-text structures depending on the purpose of their search. It is necessary to explain to students why researchers reviewed certain elements and structures. Our results showed that a number of experienced researchers conducted image searches to find relevant documents; thus, image searches should be introduced as a way of locating documents. Such an approach would be valuable, especially for searches related to experimental devices and figure models. Most students have limited opportunity to practice selecting resources and documents be- cause they largely rely on recommendations from supervisors and/or research collaborators for relevant documents. That is, they seldom have the chance to acquire the knowledge and skills of scholarly search, which researchers have learned through experience. Supervisors or senior laboratory members might be expected to train students in theory. However, it could be difficult for them to visualize and explain distinctive features of their methods41 because they learned these methods tacitly through their own search experiences. The following studies relate to how students acquire expertise. Lajoie noted that students need to be instructed about domain-specific techniques to attain expert competence.42 Alex- ander cited the Model of Domain Learning, which lets students gain expertise.43 Stenberg underlined the importance of gaining real-world expertise.44 Their opinions are not consistent with our survey results. Without instruction, students cannot intrinsically understand how to search properly. It is necessary for them to develop a method for efficiently acquiring re- searchers’ knowledge and skills related to scholarly searches. 622 College & Research Libraries July 2022 Conclusion This study focused on researchers’ scholarly search approaches, particularly their choices of resources and how they evaluated documents based on search result elements, to identify knowledge and skills of scholarly search suited to different situations. Our study involved a case study with specific targets in the engineering field. This method allowed us to examine how researchers conduct scholarly search by referring to their actual behavior. It also enabled us to compare how experienced and novice researchers find academic articles in the contemporary digital information environment. This study revealed which resources researchers use most to find academic articles and how they select documents for relevancy. The findings have implications for choosing ap- propriate database subscriptions at university libraries and for improving database interfaces. The findings also can inform the development of library workshops to teach research knowl- edge and skills, creating opportunities to learn about other researchers’ and/or laboratories’ knowledge and skills. Although we focus on the engineering field in this research, it could be applied to other fields by considering databases relevant in those fields. User studies focusing on user characteristics, needs, and behavior are important for developing future library services. Therefore, it is indispensable to communicate regularly with students and faculty. Accumulating further cases like our study will contribute to the enhancement of library services. Acknowledgments This work is based on a master’s thesis submitted by the first author to the Department of Library Science, Graduate School of Integrated Frontier Sciences, Kyushu University and is supported by JSPS KAKENHI Grant Number JP15H01721. We would like to thank researchers who took part in the observational study. We also wish to express our gratitude to members of the Graduate School of Information Science and Electrical Engineering, Kyushu University, and the Graduate School of Electrical and Computer Engineering, Shinshu University, who cooperated on the survey and for being so generous with their time. Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 623 APPENDIX Informed Consent for the Survey on Document Selection Yasuko Hagiwara Master’s Program, Department of Library Science, Graduate School of Integrated Frontier Sciences, Kyushu University If you agree to participate in the survey, please complete the consent form below. We would appreciate your cooperation once you understand the purpose and details of the study. Purpose of the Study The purpose of the study is to understand how researchers use databases for scholarly search and which search result elements they use to select academic articles to read in full. The re- sults will be summarized as part of a master’s dissertation. The study is supported by JSPS KAKENHI Grant Number JP15H01721. Procedures and Duration Please answer the following questions. The survey will take about 10 minutes. Voluntary Participation and Right to Withdraw Participation is voluntary. You may decline to answer any questions and may withdraw from the survey at any time. We would appreciate it if you could answer as much as possible. Risk There are no physical risks involved in participating in the survey. For students, your answers will not affect your grades in any courses you are taking. Publication of the Survey Results The results will be summarized as part of a master’s thesis. We may also publish the results in conferences and/or journals. Privacy and Data Storage This is an anonymous survey, and individual participants will never be identified. The prin- cipal investigator of this research project will store the survey data for up to 10 years. Contact Information If you have any questions about the study, please contact the following: Investigator: Yasuko Hagiwara (Department of Library Science, Graduate School of Integrated Frontier Sciences, Kyushu University) Email: XXX@XXXXXXX Supervisor: Emi Ishita Email: XXX@XXXXXXX Department of Library Science, Kyushu University Tel: XXX-XXX-XXXX Fax: XXX-XXX-XXXX Thank you for your time and cooperation. Please check your answer and write the date before participating in the survey. 624 College & Research Libraries July 2022 Consent form I understand the purpose and details; therefore, I agree to participate in the survey (Please check your answer). □ I Agree □ I Do Not Agree Date: /  /  1. What is your current position? □ Professor □ Associate Professor □ Lecturer □ Assistant Professor □ Research Assistant □ Post-doctoral Fellow, Research Fellow □ Doctoral Student □ Master’s Program Student □ Others (      ) 2. What is your department/program? □ Department of Informatics □ Department of Advanced Information Technology □ Department of Electrical and Electronic Engineering □ Others (      ) 3. How long have you been involved in research? (including master’s program) ______ years 4. How do you typically find academic articles for your research? (Check all that apply) □ Search results from academic databases (e.g., Web of Science, Scopus) □ Search results from Google Scholar □ Search results from search engines (e.g., Google) □ Specific journals □ Books □ References in academic articles □ Conference proceedings □ Attending conferences □ Specific author’s website □ Institutional repositories □ Preprint servers □ Social networking services (e.g., Facebook, Twitter, blog) □ Mailing list/Usenet newsgroup □ Recommendation by supervisors, colleagues, and/or laboratory members □ I have never read academic articles □ Others (      ) 5. To what extent are you able to find academic articles that you need for your research? I can find… □ Almost everything Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 625 □ About 3/4 □ About 1/2 □ About 1/4 □ Very little 6. Have you ever attended library workshops on conducting a literature search? (including instructions at laboratories) □ Yes □ No 7. Have you ever performed a literature search using databases and/or search engines? □ Yes □ No [The following questions concern the use of databases. If you selected “No” to Question 7, you have finished the survey. Thank you for your cooperation.] 8. With which databases do you perform document search for each research stage? (Check all that apply) Stage Beginning research Considering appropriate methods Writing papers Web of Science □ □ □ Scopus □ □ □ SciFinder □ □ □ PubMed Google Scholar □ □ □ Google □ □ □ Digital library provided by academic association (e.g., ACM, IEEE) □ □ □ CiNii Articles* □ □ □ World Contents** □ □ □ Others □ □ □ *CiNii Articles: A Japanese database service that can be searched with bibliographic information of articles. **World Contents: Discovery service provided by Kyushu University Library (Summon). 9. If you use specific databases in addition to the three stages indicated in Question 8, please describe the databases you use for any stage. 10. Which search result elements do you use to select articles that you want to read in full-text version? Please select up to five patterns of elements from #1 to 20 (e.g., If you read the title followed by the abstract, please check #14). *Full-text: i.e., checking necessary parts in full-text 626 College & Research Libraries July 2022 Others (      ) à (      ) à (      ) à (      ) 11. For each search purpose, which search result elements do you check to select articles to read in full? (Check all that apply) Purpose To identify appropriate research methods To find previous research in an unfamiliar field To learn about current research trends Title □ □ □ Abstract □ □ □ Full-text (i.e., checking necessary parts in full-text) □ □ □ Publication year □ □ □ Journal □ □ □ Author □ □ □ Number of pages □ □ □ Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 627 Purpose To identify appropriate research methods To find previous research in an unfamiliar field To learn about current research trends Cited counts (Including sorting search results by times cited) □ □ □ Keyword □ □ □ Research area □ □ □ Document type (e.g., article, review, proceedings paper) □ □ □ Language □ □ □ Online availability of full-text □ □ □ Images (e.g., Google image search, graphical abstract) □ □ □ Others (      ) □ □ □ If you checked “Full-text” in Question 11, please answer the next question. 12. Which full-text structures do you consider? (Check all that apply) To identify appropriate research methods To find previous research in an unfamiliar field To learn the current research trends Introduction □ □ □ Related Work □ □ □ Method □ □ □ Conclusion □ □ □ References □ □ □ Fig., Table, Formula □ □ □ Others (      ) □ □ □ 13. If you consider additional elements and structures for the three search purposes indicated in questions 11 and 12, please describe them and the search purpose. 14. Are you familiar with the following functions of Web of Science and Scopus? If so, how often do you use them? 14-1. Web of Science □ I don’t use Web of Science > Please move on to Question 14-2. Most of the time Some of the time Not very often I know of the function but never use it I don’t know of the function Confirm references □ □ □ □ □ Confirm citing articles □ □ □ □ □ 628 College & Research Libraries July 2022 Most of the time Some of the time Not very often I know of the function but never use it I don’t know of the function Sort search results by times cited □ □ □ □ □ Link through LinQ* □ □ □ □ □ Link to publishers’ website □ □ □ □ □ Confirm impact factor □ □ □ □ □ Export to bibliographic management software □ □ □ □ □ Others (      ) *LinQ: A link resolver icon of Kyushu University. It navigates to a website, providing full-text is available. 14-2. Scopus □ I don’t use Scopus Most of the time Some of the time Not very often I know of the function but never use it I don’t know of the function Confirm references □ □ □ □ □ Confirm citing articles □ □ □ □ □ Sort search results by times cited □ □ □ □ □ Link through LinQ* □ □ □ □ □ Link to publishers’ website □ □ □ □ □ Export to bibliographic management software □ □ □ □ □ Others (      ) *LinQ: A link resolver icon of Kyushu University. It navigates to a website providing full-text if available. This is the end of the questionnaire. Thank you for your cooperation. Notes 1. Bernard J. Jansen and Amanda Spink, “An Analysis of Web Documents Retrieved and Viewed,” Proceed- ings of the International Conference on Internet Computing 1 (2003): 65–69; Lori Lorigo et al., “Eye Tracking and Online Search: Lessons Learned and Challenges Ahead,” Journal of the American Society for Information Science and Technology 59, no. 7 (2008): 1041–52. 2. Laura L. Haines et al., “Information-seeking Behavior of Basic Science Researchers: Implications for Li- brary Services,” Journal of the Medical Library Association 98, no. 1 (2010): 73–81. 3. Saskia Brand-Gruwel et al., “Source Evaluation of Domain Experts and Novices during Web Search,” Journal of Computer Assisted Learning 33, no. 3 (2017): 234–51. 4. Harris M. Cooper, Synthesizing Research: A Guide for Literature Reviews, 3rd ed. (Thousand Oaks, CA: Sage, 1998); Edmund K. Burke and Graham Kendall, Search Methodologies, 2nd ed. (New York, NY: Springer, 2014). 5. Burke and Kendall, Search Methodologies. 6. Christine Wolff-Eisenberg, Alisa B. Rod, and Roger C. Schonfeld, “Ithaka S+ R US Faculty Survey 2015,” Ithaka S+R, last modified April 4, 2016, https://doi.org/10.18665/sr.277685 [accessed 17 August 2021]. 7. David Nicholas, Paul Huntington, and Hamir R. Jamali, “The Use, Users, and Role of Abstracts in the Identifying Scholarly Search Skills Based on Resource and Document Selection Behavior 629 Digital Scholarly Environment,” Journal of Academic Librarianship 33, no. 4 (2007): 446–53. 8. S. Liyana and A. Noorhidawati, “How Graduate Students Seek for Information: Convenience or Guaran- teed Result?” Malaysian Journal of Library & Information Science 19, no. 2 (2014): 1–15. 9. Joe C. Clark and Jennifer Johnstone, “Exploring the Research Mindset and Information-Seeking Behaviors of Undergraduate Music Students,” College & Research Libraries 79, no. 4 (2018): 499–516. 10. Ching-Ting Hsin, Ying-Hsueh Cheng, and Chin-Chung Tsai, “Searching and Sourcing Online Academic Literature: Comparisons of Doctoral Students and Junior Faculty in Education,” Online Information Review 40, no. 7 (2016): 979–97. 11. Wolff-Eisenberg, Rod, and Schonfeld, “Ithaka S+ R US Faculty Survey 2015.” 12. Monbukagakushō, “Monbukagaku Tōkei Yōran: Heisei 29nenban 文部科学統計要覧: 平成29年度版 [Sta- tistical Abstract: 2017 edition],” The Ministry of Education, Culture, Sports, Science and Technology of Japan, last modified April 2017, www.mext.go.jp/b_menu/toukei/002/002b/1383990.htm [accessed 17 August 2021]. 13. Carol Tenopir, Use and Users of Electronic Library Resources: An Overview and Analysis of Recent Research Studies (Washington, DC: Council on Library and Information Resources, 2003). 14. Association of College & Research Libraries (ACRL), Framework for Information Literacy for Higher Education, available online at www.ala.org/acrl/standards/ilframework [accessed 17 August 2021]. 15. Yasuko Hagiwara et al., “An Experiment to Identify How Researchers Select Documents from Search Results,” Proceedings of CiSAP Workshop (The Annual Meeting of the Consortium of iSchools Asia-Pacific) 2015 (2015): 8–11; Yasuko Hagiwara et al., “A Preliminary Study and Analysis to Identify Key Elements in Document Selec- tion” (ISIC: Information Seeking in Context 2016, Zadar, Croatia, Sept. 2016); Yasuko Hagiwara et al., “Identifying Key Elements of Search Results for Document Selection in the Digital Age: An Observational Study,” Lecture Notes in Computer Science 10647 (2017): 237–42. 16. Elaine Sullo, “Engineering Faculty Indicate High Levels of Awareness and Use of the Library but Tend to Consult Google and Google Scholar First for Research Resources,” Evidence Based Library and Information Practice 11, no. 3 (2016): 102–04. 17. Chérifa Boukacem-Zeghmouri and Joachim Schöpfel, “Beyond the Google Generation: Towards Com- munity-specific Usage Patterns of Scientific Information,” in Trends, Discovery, and People in the Digital Age, eds. David Baker and Wendy Evans (Oxford, UK: Chandos Publishing, 2013), 137–51. 18. Anthony Watkinson et al., “Changes in the Digital Scholarly Environment and Issues of Trust: An Ex- ploratory, Qualitative Analysis,” Information Processing & Management 52, no. 3 (2016): 446–58. 19. Hamid R. Jamali and Saeid Asadi. “Google and the Scholar: The Role of Google in Scientists’ Information- seeking Behaviour,” Online Information Review 34, no. 2 (2010): 282–94. 20. Kumaripaba Athukorala et al., “Information-seeking Behaviors of Computer Scientists: Challenges for Electronic Literature Search Tools,” Proceedings of the Association for Information Science and Technology 50, no. 1 (2013): 1–11. 21. Hagiwara et al., “A Preliminary Study and Analysis to Identify Key Elements in Document Selection”; Hagiwara et al., “Identifying Key Elements of Search Results for Document Selection in the Digital Age.” 22. Hsin, Cheng, and Tsai, “Searching and Sourcing Online Academic Literature.” 23. Peiling Wang and Dagobert Soerge, “A Cognitive Model of Document Use during a Research Project: Study I. Document Selection,” Journal of the American Society for Information Science 49, no. 2 (1998): 115–33. 24. Mônica Macedo-Rouet et al., “How Do Scientists Select Articles in the PubMed Database? An Empirical Study of Criteria and Strategies,” European Review of Applied Psychology 62, no. 2 (2012): 63–72. 25. Hamid R. Jamali et al., “How Scholars Implement Trust in Their Reading, Citing and Publishing Activi- ties: Geographical Differences,” Library & Information Science Research 36, no. 3/4 (2014): 192–202. 26. Xi Niu et al., “National Study of Information Seeking Behavior of Academic Researchers in the United States,” Journal of the American Society for Information Science and Technology 61, no. 5 (2010): 869–90. 27. Dogan R. Islamaj et al., “Understanding PubMed User Search Behavior through Log Analysis,” Database 2009 (2009): bap018. 28. Hagiwara et al., “A Preliminary Study and Analysis to Identify Key Elements in Document Selection”; Hagiwara et al., “Identifying Key Elements of Search Results for Document Selection in the Digital Age.” 29. Sandra L. De Groote, Mary Shultz, and Deborah D. Blecic, “Information-seeking Behavior and the Use of Online Resources: A Snapshot of Current Health Sciences Faculty,” Journal of the Medical Library Association 102, no. 3 (2014): 169–76; Xi Niu et al., “National Study of Information Seeking Behavior of Academic Researchers in the United States.” 30. Wang and Soerge, “A Cognitive Model of Document Use during a Research Project. Study I. Document Selection”; Macedo-Rouet et al., “How Do Scientists Select Articles in the PubMed Database?” 31. Hagiwara et al., “Identifying Key Elements of Search Results for Document Selection in the Digital Age.” 630 College & Research Libraries July 2022 32. Hagiwara et al., “Identifying Key Elements of Search Results for Document Selection in the Digital Age.” 33. Nicholas et al., “The Use, Users, and Role of Abstracts in the Digital Scholarly Environment”; Jamali et al., “How Scholars Implement Trust in their Reading, Citing and Publishing Activities.” 34. Ángel Borrego and Jenny Fry, “Measuring Researchers’ Use of Scholarly Information through Social Bookmarking Data: A Case Study of BibSonomy,” Journal of Information Science 38, no. 3 (2012): 297–308. 35. Wolff-Eisenberg, Rod, and Schonfeld, “Ithaka S+ R US Faculty Survey 2015”; Watkinson et al., “Changes in the Digital Scholarly Environment and Issues of Trust”; Boukacem-Zeghmouri and Schöpfel, “Beyond the Google Generation”; Jamali and Asadi. “Google and the Scholar.” 36. Wolff-Eisenberg, Rod, and Schonfeld, “Ithaka S+ R US Faculty Survey 2015”; Watkinson et al., “Changes in the Digital Scholarly Environment and Issues of Trust”; Boukacem-Zeghmouri and Schöpfel, “Beyond the Google Generation.” 37. Jamali and Asadi. “Google and the Scholar.” 38. Emi Ishita et al., “Which Parts of Search Results Do Researchers Check When Selecting Academic Docu- ments?” Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries (2018): 345–46. 39. Hsin, Cheng, and Tsai, “Searching and Sourcing Online Academic Literature”; Wang and Soerge, “A Cognitive Model of Document Use During a Research Project. Study I. Document Selection”; Macedo-Rouet et al., “How Do Scientists Select Articles in the PubMed Database?” 40. Ishita et al., “Which Parts of Search Results Do Researchers Check When Selecting Academic Documents?” 41. John D. Bransford, Ann L. Brown, and Rodney R. Cocking, “How Experts Differ from Novices,” in How People Learn: Brain, Mind, Experience, and School, expanded ed. (Washington, DC: National Academy Press, 2000), 31–50. 42. Susanne P. Lajoie, “Transitions and Trajectories for Studies of Expertise,” Educational Researcher 32, no. 8 (2003): 21–25. 43. Patricia A. Alexander, “The Development of Expertise: The Journey from Acclimation to Proficiency,” Educational Researcher 32, no. 8 (2003): 10–14. 44. Robert J. Sternberg, “What Is an ‘Expert Student’?” Educational Researcher 32, no. 8 (2003): 5–9.