key: cord-0001954-sd0an0z3 authors: Dixon, Brian E.; Pina, Jamie; Kharrazi, Hadi; Gharghabi, Fardad; Richards, Janise title: What’s Past is Prologue: A Scoping Review of Recent Public Health and Global Health Informatics Literature date: 2015-07-01 journal: Online J Public Health Inform DOI: 10.5210/ojphi.v7i2.5931 sha: 41148756482815041a85c11c6b01cb832bf1de9b doc_id: 1954 cord_uid: sd0an0z3 Objective: To categorize and describe the public health informatics (PHI) and global health informatics (GHI) literature between 2012 and 2014. Methods: We conducted a semi-systematic review of articles published between January 2012 and September 2014 where information and communications technologies (ICT) was a primary subject of the study or a main component of the study methodology. Additional inclusion and exclusion criteria were used to filter PHI and GHI articles from the larger biomedical informatics domain. Articles were identified using MEDLINE as well as personal bibliographies from members of the American Medical Informatics Association PHI and GHI working groups. Results: A total of 85 PHI articles and 282 GHI articles were identified. While systems in PHI continue to support surveillance activities, we identified a shift towards support for prevention, environmental health, and public health care services. Furthermore, articles from the U.S. reveal a shift towards PHI applications at state and local levels. GHI articles focused on telemedicine, mHealth and eHealth applications. The development of adequate infrastructure to support ICT remains a challenge, although we identified a small but growing set of articles that measure the impact of ICT on clinical outcomes. Discussion: There is evidence of growth with respect to both implementation of information systems within the public health enterprise as well as a widening of scope within each informatics discipline. Yet the articles also illuminate the need for more primary research studies on what works and what does not as both searches yielded small numbers of primary, empirical articles. Conclusion: While the body of knowledge around PHI and GHI continues to mature, additional studies of higher quality are needed to generate the robust evidence base needed to support continued investment in ICT by governmental health agencies. While they share much in common with other branches within the broader field of biomedical informatics [1] , public health and global health informatics differ in their scope and applications. Public health informatics (PHI; also known as public health & epidemiology informatics in Europe) focuses on the application of information and communications technologies (ICT) to promote the health of populations rather than individual patients. In addition, PHI focuses on disease prevention rather than treatment and typically operates within a governmental rather than private environment [2] . PHI is generally organized and conducted by governmental public health authorities, which is most often a nation's 'ministry of health' except in the United States where public health practice is dispersed among a federation of local, state and federal health agencies. In recent years, global health informatics (GHI) emerged out of the broader biomedical informatics discipline as a distinct field focused on applying ICT to both public health and health care delivery in the context of low-to-middle income countries (LMICs). Its scope includes technologies that support the delivery of public and private health services (e.g., electronic health record, telemedicine, mobile health) as well as the management of health services across the care continuum within as well as across nations (e.g., health information exchange, health worker registries, epidemiology). Thus PHI and GHI share many commonalities with each other, yet each contributes distinctly to the science and practice of informatics, medicine, and public health. Beginning in 2013, the PHI and GHI working groups within the American Medical Informatics Association (AMIA) have offered a 'Year in Review' session at the AMIA Annual Symposium. These sessions highlight advancements and trends in both the science and practice of PHI and GHI. This paper builds upon these Year in Review sessions by providing a semi-systematic review of the PHI and GHI literature over the past three years. Our goal is to summarize recent advancements and trends in both fields and suggest directions for future work that will further stimulate growth in the fields and advancements in the science of informatics as well as the practice of public health in all nations. We conducted a semi-systematic review of the biomedical literature between 2012 and 2014 in accordance with the PRISMA guidelines [3] . Our review is based on the work conducted by the AMIA PHI and GHI working groups to present a 'Year in Review.' Given that our review focuses on a synthesis of multiple mini-reviews performed by the working groups and the heterogeneity of the selected articles, we neither assessed the quality of studies nor aggregated study results, thereby classifying our study as a semi-systematic review instead of a comprehensive systematic review or meta-analysis. Multiple searches were conducted between September 2013 and August 2014 using disparate keywords to identify all possible English-language PHI and GHI peer-reviewed literature indexed in PubMed or MEDLINE that was published between January 1, 2012 and September 30, 2014. Supplemental articles were gathered by reviewing the bibliographies of selected articles and by soliciting suggestions from PHI and GHI working group members using AMIA listservs. For PHI articles, we used the following MEDLINE Subject Headings (MeSH) keywords in various combinations: public health informatics, public health, informatics, and information system. Use of the MeSH headings was preferred, although articles were also identified using the keywords to look within article titles and abstracts. For GHI articles, we used the following keywords: informatics, telemedicine, information system, developing countries, global, national health programs, low resource, and resource-limited. Given the recent emergence of GHI in the literature, MeSH headings for the field are limited; only some of these are official MeSH headings. Furthermore, the searches in 2014 expanded the keyword list to include the name of every LMIC as defined by the World Bank as well as other terms that better reflected GHI activities happening internationally, such as mobile health (mHealth), electronic health records and electronic medical records. This significantly increased the list of candidate articles, but it was determined by CDC health librarians to be the most sensitive method for identifying informatics articles published in LMIC nations. A complete listing of the dates and queries used to identify articles is available in Appendix A. We used a variety of inclusion and exclusion criteria to narrow the lists of candidate articles. To be included and classified as a PHI article: (a) informatics, information science, or computing had to be the primary subject of the study or a main component of the study methodology; and (b) the article needed to focus on a topic related to public health practice or research. We used a broad lens to assess each article's relevance to public health practice and research, including: 1) activities conducted by, with, or involving a local, state or federal health agency; 2) assessment and monitoring of disease and health outcomes; 3) primary and secondary prevention of disease; 4) social determinants of health as well as health disparities; and 5) development of the public health workforce including PHI education and competencies. Articles were excluded from the PHI group if the research occurred principally within a LMIC nation. We further excluded articles that did not constitute original research such as letters, editorials, perspectives, opinions, whitepapers, comments, and study protocols. For a publication to be included and classified as a GHI article, informatics, information science, or information systems had to be the primary subject of the study or a main component of the study methodology. Additionally, the primary research must have focused on activities taking place within an LMIC, and also have a focus on a global health topic. Global health topics follow the framework of Koplan et al. [4] and include: 1) health issues that transcend national boundaries; 2) development and implementation of solutions that often require global cooperation; 3) embraces both prevention of populations and clinical care of individuals; and 4) promote health equity among nations and for all people. Articles were excluded from the GHI group if they did not constitute original research, such as letters, editorials, perspectives, opinions, whitepapers, comments, and study protocols. Multiple volunteers from the PHI and GHI working groups assisted with review of candidate articles. Two authors (BED and JP) divided the candidate articles into relatively equal numbers and assigned them to reviewers. Reviewers examined titles, abstracts, and other article metadata, recommending inclusion or exclusion based on the defined criteria. Each article was reviewed by at least two individuals, and disagreements were resolved after further review by the team leads (BED and JP). Selected articles were abstracted by multiple individuals using a set of criteria established by the PHI and GHI teams. For PHI, trainees in PHI programs volunteered to read through the full text of each selected article and summarize key metadata. They examined the information system implemented or evaluated (e.g., syndromic surveillance system, immunization information system); method of data capture (e.g., electronic health record, mobile device); impact of the technology on practice or research; barriers to adoption; size of the cohort or population; study locations; and jurisdiction of practice (e.g., local health department, ministry of health). Summarized information was captured in a spreadsheet and reviewed by the PHI team leads (BED and HK). For GHI, volunteers had graduate degrees in public health, informatics or informatics-related field and experience in applying ICT in global settings. They reviewed the full text of each selected article published before 2014 (N=83). Given the expanded size of the corpus in 2014 (N=199), volunteers were asked to examine the abstract and full text if possible. Reviewers categorized and summarized the articles using the following metadata: article type (e.g., review, research, methods, models); main keywords; main objective; principal findings; possible impact on the practice of GHI; and other notes. Like PHI, these observations from the reviewers were captured in a spreadsheet and reviewed by the GHI team lead (JP) to identify themes and trends. The selection of PHI and GHI articles is summarized in Figure 1 and Figure 2 , respectively. From a potential pool of 526 PHI articles, we identified 85 articles that satisfied our inclusion criteria. The majority of the candidate PHI articles either (a) did not focus on informatics or an information system; (b) presented research performed within an LMIC; or (c) were classified as a commentary or letter. From a pool of 1241 GHI articles, we identified 282 articles that satisfied our inclusion criteria. Candidate GHI articles were excluded when they (a) did not focus on informatics or an information system; (b) were performed outside of an LMIC; (c) did not focus on a global health topic; or (d) were classified as a commentary or letter. PHI articles focus on a wide range of information systems (Figure 3 ), including: measuring population health using electronic health data (Pop Health / Quality Indicators; N = 20), electronic laboratory reporting (ELR) along with communicable disease case reporting (N=13); syndromic surveillance (N=11), and immunization information system (IIS; N=6). Examining trends across the three years reveals that articles shifted slightly away from a focus on quality indicators towards ELR and case reporting. Yet all four systems remained in the 'Top 4' each year. These are all common information systems or uses of electronic data found in public health practice. They are also specific functions called out in the U.S. Centers for Medicaid and Medicare Services' (CMS) meaningful use program, which incentivizes the adoption and use of electronic health record systems in health care delivery systems and organizations [5, 6] . Other common information systems used by governmental public health agencies but not appearing frequently in the selected articles includes chronic disease registries (e.g., state cancer registry), vital information systems (e.g., death registry), and water quality monitoring systems (e.g., toxicology information system). The year 2012 contained a significant number of 'other' articles which is indicative of the broader search strategy employed at the beginning of our project that became more focused over time. Articles in the other category included topics such as the application of geographic information systems to public health practice [7]; PHI education, training and workforce development [8] [9] [10] ; online information resources [11, 12] ; and social networking [13, 14] . While relevant to the discipline of PHI, we did not classify them as focused on an information system used in routine public health practice. Figure 3 : Count of PHI articles based on the type of information system implemented or evaluated, stratified by year of publication. In Figure-4 , we summarize the methods used in the PHI articles. While 43 articles used a quantitative methodology, just nine articles employed a controlled experimental or comparative design such as the examination of a new influenza-like-illness classifier [15] or changes in syndromic surveillance system use following a change [16] . Twenty articles present initial findings from a pilot study that measured a system's usability [12] or use [17,18] among early adopters. Twelve articles were even more exploratory in nature, outlining simply the design process for a new system [19] [20] [21] or presenting the characterization of a new data source [22, 23] . Twenty-two articles were systematic reviews, including a review of syndromic surveillance classifiers [24], the use of IIS for research [25] , the use of social networking sites in public health [13] , and information needs of public health practitioners [26] . Case studies were also present in 16 articles, summarizing the design or implementation of a PHI system within a single health department or group of organizations [27, 28] . Ten studies were surveys, which employ quantitative methods to analyze respondents' answers to questions about PHI training programs [9]; the role of governance in guiding adoption and use of PHI systems [29]; public health engagement in health information exchange [30]; or characterizing the quality of data in an electronic information system [31,32]. Figure 4 : Count of PHI articles based on study methodology, stratified by year of publication. In Figure-5 we summarize the jurisdictions involved in the implementation or use of the information system described in the selected articles. There is a noticeable trend in the data away from articles that describe systems designed, implemented, or used at the federal level towards systems used at the local or state level. The more recent state and local health department articles tended to focus on information systems implemented in compliance with the CMS meaningful use program: ELR and case reporting [33,34]; syndromic surveillance [16] , and quality reporting [21] . More recent articles also synthesize lessons and implementation strategies at the local level for ELR, IIS, and other meaningful use related information systems [35, 36] . Fourteen articles describe information systems deployed or used by hospitals or health systems. For example, three studies examined information systems in U.S. hospitals used by infection preventionists to monitor and control outbreaks within a hospital or health system [30, 37, 38] . Other studies examined the use of oncology information systems, in combination with as well as independent from cancer registries, to create databases for studying cancer epidemiology and outcomes [39, 40] . While clinical organizations' roles in the design, maintenance or use were emphasized in these articles, the studies either used public health data resources like cancer registries, or they otherwise involved governmental public health agencies in the design or implementation of the system. Other articles employed surveys or interviews that included governmental public health agencies in combination with hospitals or health systems to examine larger topics within the field of biomedical informatics including but not limited to comparative effectiveness research [41,42]. Figure 5 : Count of PHI articles based on the jurisdictions that implemented or used the information systems described in the article, stratified by year. The GHI literature spans a broad range of technologies, health service delivery areas, and foci. Reviewers identified several high-level and recurring themes in the GHI literature through full article review (2013) and abstract review (2014). Major themes that spanned both years of the review included: telemedicine, mHealth, and eHealth. Emerging themes, especially in the 2014 review, included: surveillance, decision support, and geographical information systems (GIS). These themes are summarized in Figure-6 . Furthermore, reviewers noted that the overall volume of articles increased in the 2013-2014 review. Although the overall number of articles increased, the relative proportion of each major theme remained similar. Over the two years of review, the theme of telemedicine remains the largest portion of GHIrelated literature. Telemedicine applications bridge healthcare delivery needs when providers and specialists are not physically present in a specific region. Telemedicine has also been a source for OJPHI access to specialists in fields, such as dermatology and pediatric cardiology, when a country has limited number of specialists available for consultation. Another application of telemedicine technology is to provide continuous medical education to healthcare providers in remote regions. The telemedicine literature in a global context has focused on the development and evaluation of telemedicine programs in various clinical contexts, generally describing implementations of telemedicine and the impact of these tools [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] . The literature suggests that when telemedicine solutions are implemented with adequate infrastructure, financial and local clinical support they are found to be effective in meeting healthcare delivery goals and impacting healthcare providers' ability to learn of new medical practices, although garnering this support is challenging in LMICs [52,53,56,58]. As mobile technologies have proliferated throughout LMICs, their use in a variety of healthrelated activities has been studied [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] . As rapidly as mobile technology use has grown, the mHealth literature also has rapidly increased. Much of the mHealth literature explored the effectiveness of mHealth interventions, with emphasis on using mobile technologies to deliver health-promoting messages, to improve clinic visit attendance for culturally sensitive health issues such as breast cancer, and to increase patient retention in care, treatment, and prevention programs. The mHealth literature focused on scaling up and evaluating mHealth interventions increased in 2014, suggesting growth in maturity for this area of GHI research. mHealth interventions increasingly span numerous areas of clinical and public health focus, including psychiatric disorders, total health/wellness, surveillance of rabies, infant feeding, drug adherence, and measuring the impact of perinatal interventions. Some studies evaluated mobile survey platforms, generally acknowledging the value of this technology for survey distribution [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] . The literature also described mobile applications used to improve specific clinical activities (e.g. waiting time, order entry) [74, 83, [89] [90] [91] [92] [93] . The final major theme, eHealth, focuses on the development, use and impact of electronic health record (EHR) systems on LMICs. Some studies in this category aimed to demonstrate the impact of eHealth applications on patient and population outcomes within various LMIC contexts [47, 89, [94] [95] [96] [97] . Other studies explored more foundational informatics aspects, such as data quality in eHealth applications, factors that lead to adoption of EHR systems, use of standardized minimum data sets to assist with the electronic exchange of clinical data, and development and use of open source, standardized EHRs . Determining the extent of adoption and use of EHRs in LMICs and understanding the level of computer literacy as a barrier to adoption continued in the literature over the two years [98] [99] [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] . Policy research related to e-health was not highly present in the literature; we identified just one systematic review of policy issues [98] . We further examined the methodology used in each selected article, summarized in Figure-7. Our methods for conducting the review varied from 2013 to 2014, yet in each year we sought to distinguish evaluation articles from reviews and methods papers. Review articles, both robust systematic reviews and weaker review articles spanned several topics, including the impact of mHealth interventions on healthcare quality, a general review of mHealth and its potential, the use of open-source EHR systems, health information systems in Sub-Saharan Africa countries, mobile phone interventions for consumer health, and overcoming shortages of human resources in LMICs [74, 81, 84, 89, 99, 100] . We further identified a group papers that focused on new methods of informatics research. Finally we identified model papers, which emphasized the development of new models of evaluation or health information technology (HIT) development. Figure 7 : Count of GHI articles based on study methodology, stratified by year of publication. The largest category of articles included a mix of various primary studies. In addition to studies that assessed the impact of ICT on patient and population outcomes, themes that arose during our discussion of the articles were assessments of infrastructure needed for system implementation and readiness of an environment, such as a clinical facility or a group of system users, to adopt the system. Because access to electricity, telecommunications, hardware and other aspects of infrastructure are sometimes constrained in LMICs, we noted another theme aimed to measure the use and viability of ICT. Specifically in 2014 we observed this theme of enumerating infrastructure challenges and proposing solutions to them; topics included measuring system and power outage, security and privacy enhancements, cloud-based migrations and implementations, measuring a hospital's ability to outsource ICT effectively, and data transformations or mapping to multiple standards [101] [102] [103] [104] [105] [106] [107] . Emphasis was often placed on open-source applications that can be distributed across multiple communities and nations [52, [108] [109] [110] [111] [112] [113] . Discussions examined different aspects of infrastructure as either facilitator or barriers to adoption [94, [114] [115] [116] . We further identified a set of articles (N=7) we refer to as "readiness studies," or studies that assess training; availability of health care workers, health delivery facilities, or ICT; phone ownership; and data access in LMICs [102, [114] [115] [116] [117] [118] [119] . These studies specifically identify the extent to which an environment within an LMIC is prepared to adopt some form of HIT. Finally, as we opened our search in 2014 to include all countries designated as LMICs according to the World Bank, we observed new topics surface in the literature. Specifically, we observed the presence of imaging informatics publications released in 2014 (N=4). These publications described the improvement of imaging services by integrating systems, reducing cost, using mobile devices, applying new imaging techniques in LMICs, and exploring new methods to identify and compare data in images [120] [121] [122] [123] [124] . Reviewers also identified an additional theme related to emergency services supported by information systems in LMICs [125] [126] [127] . We performed a semi-systematic review of the PHI and GHI literature to identify trends in both innovation and research. A total of 85 PHI articles and 282 GHI articles were identified through a search of the available literature and suggestions from AMIA working group members. The OJPHI selected articles suggest growth with respect to both implementation of information systems within the public health enterprise as well as a widening of scope within each informatics discipline. Yet the articles also illuminate the need for more primary research studies on what works and what does not as both searches yielded small numbers of primary, empirical articles. Each field, while maturing, continues to produce a larger volume of articles that describe either a) the design or recent implementation of an information system; or b) a limited review of early systems deployed by a relatively small number of organizations. This limits the conduct of more robust systematic reviews as well as meta-analyses that could inform public health policy as well as health care delivery. At the turn of the twenty-first century, PHI efforts around the world were characterized by a focus on the core public health function of monitoring populations: early detection of bioterrorism [128] , such as the Anthrax attacks in the U.S [129] and the Tokyo subway attacks [130] , as well as global health threats such as SARS [131] and the H1N1 pandemic [132] . While the threat of a large-scale epidemic has not diminished in recent years, as evidenced in 2014 by MERS [133, 134] and Ebola [135] , the scope of PHI has broadened to support the full range of functions performed by governmental public health authorities. As evidenced in this review, PHI today contributes not only to disease detection [15, 136] but also to the delivery of public health care services [137] [138] [139] , the measurement of population health indicators within and across jurisdictions [21, 140] , investigating environmental hazards [141] , and prevention of disease [142] [143] [144] . PHI research to a large extent, just like in many other areas of biomedical research, follows policy and funding patterns within a nation or region. The last few years have witnessed a global recession with shrinking public health budgets and a sharp decline in funds allocated to public health preparedness. Funding patterns explain, in part, what appears to be a more balanced portfolio of PHI activities with a strong but smaller emphasis on surveillance. In developed nations with a strong integrated public health system such as Canada [145, 146] , Australia [147] , Taiwan [27] and several European nations [148] [149] [150] , PHI activities include an emphasis on integrating data and supporting bi-directional communication between clinical and public health services. Whereas in the US, PHI efforts have focused largely on EHR incentive policies that seek to improve how public health agencies gather the data they need to monitor the health of populations [29,33,151]. Thus to stimulate new advancements in PHI, especially in the U.S., policymakers must encourage the development or broader implementation of information systems that span the clinical and public health continuum to support the full range of public health functions. Support for this expanded view of PHI can be found in both the Council of State and Territorial Epidemiologists' 'Blueprint version 2.0' statement on the future of surveillance [152] , as well as the workshop report from the Robert Wood Johnson Foundation on the future of PHI [153] . The focus of GHI research intersects at the crossroads of clinical and public health informatics in the context of LMICs. Given a global emphasis on the development and implementation of national eHealth strategies, recent GHI articles reflect an increase in the development, implementation and use of telemedicine, mHealth, EHR, and other forms of healthcare ICT in LMICs on all continents. Furthermore, the limited studies assessing outcomes or impact of these systems on patient and population health suggests that adoption of EHR and other ICT systems OJPHI in LMICs may be accelerating at a pace that exceeds the rate at which the academic community can evaluate them. Despite a change in our search methodology, we observed similar patterns across all publication years. First, there is significant growth in the publication of GHI articles. While this may be partly due to our change in search strategy to include specific LMIC country names, we believe there is true growth in the development, implementation and use of ICT in LMICs across the globe. External forces, including the World Health Organization (WHO) eHealth Technical Advisory Group (TAG) formed in 2013, are encouraging LMICs to adopt and implement national eHealth strategies that will strengthen health systems through the use of ICT including telemedicine and EHR systems [154, 155] . Second, despite growth in articles many studies continue to focus on implementation rather than outcomes. We observed numerous, although sometimes subtle, descriptions of implementation challenges. For example, reviewers noted that organizational culture can create barriers, which impact the adoption and implementation of informational tools, was often mentioned in the literature, although this subject is not treated methodologically as the focus of many publications [81, 117] . Finally, GHI research is intrinsically linked to public health practice in LMICs, creating the link between the use of ICT and use of clinical and other data for public health activities such as surveillance, health monitoring and public health program evaluation [156] [157] [158] [159] [160] . The literature suggests that the use of ICT will continue to expand in LMICs. This will likely generate new studies on their impact on patient and population health. As these studies are planned and executed, our review suggests the following. First, we note a general dearth of rigorous study designs. Better designed studies that follow the Good Evaluation Practices in Health Informatics (GEP-HI) [161] and are reported using the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) guidelines [162, 163] will support future efforts that can more rigorously systematically review and synthesize outcomes. More rigorous review and synthesis will provide stronger evidence that many policymakers seek as they contemplate policies and funding beyond initial investments in eHealth systems [164] . In addition, given a rich dialogue in the Discussion sections of many articles on the barriers and facilitators of adoption, we suggest that future research more rigorously examine the implementation of ICT in LMICs. There exists a growing body of implementation science literature in the developed world [165, 166] , which would be strengthened by contributions from LMICs if implementation were more rigorously studied in parallel with outcomes. Furthermore, we observed that many studies involve partners from multiple universities, research institutes, and countries with diverse funding streams. Finding harmony across fragmented stakeholders appears to be a goal of the applied work taking place in these LMICs, yet this is not emphasized as a measurable component of research in the current body of literature. Future studies should consider studying this dimension of GHI implementation as a facilitator or barrier. Our review of the PHI and GHI literature contains several limitations. First, our review of the literature was limited to articles published in English, which limited our ability to identify and read relevant articles published in other languages (the Iceberg effect). Second, our review was limited to primarily peer-reviewed articles indexed in MEDLINE, which limited our ability to identify relevant articles published in newer journals or journals which do not meet the scientific scope and quality metrics established by the MEDLINE Literature Selection Technical Review Committee. GHI article review for 2014 was limited to abstract review of selected articles. Although our purpose differs from that of the International Medical Informatics Association's IMIA Yearbook of Medical Informatics, many of our methods are quite comparable [167] . And like IMIA, as we continue to evolve the Year in Review session for AMIA's annual symposium, we will seek to refine our methods. We will consider additional sources beyond MEDLINE to identify more articles from related disciplines which are not indexed in PubMed such as the computer, information, and decision sciences. We will further explore tools such as BibReview to enhance our ability to identify, select and summarize recent publications in the disciplines of PHI and GHI. Although distinct, the PHI and GHI sub-disciplines within the field of biomedical informatics seek to advance health systems' goals of improving the efficiency, costs and outcomes associated with health care delivery to individual patients as well as populations. Studies published over the past three years identify growth in our understanding of ICT development, implementation and adoption in clinical as well as public health settings. Yet the articles also highlight there is much work yet to be done. Namely we need more rigorous studies to generate a robust evidence base demonstrating not only whether certain types of ICT support better population outcomes but also which systems and implementation methods lead to success in terms of system usage as well as health outcomes. Many nations are in the process of either developing or implementing national eHealth strategies, which will undoubtedly require evaluation and refinement in the years ahead. "lesser developed economy" OR "lesser developed economies" OR "under developed economy" OR "under developed economies" OR "underdeveloped economy" OR "underdeveloped economies" OR "middle income economy" OR "middle income economies" OR "low income economy" OR "low income economies" OR "lower income economy" OR "lower income economies" OR "low gdp" OR "low gnp" OR "lower gdp" OR "lower gnp" OR lmic OR lmics OR "third world" OR "lami country" OR "lami countries" OR "transitional country" OR "transitional countries" OR Africa Biomedical Informatics: The Science and the Pragmatics Introduction to Public Health Informatics Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement Towards a common definition of global health Public health surveillance and meaningful use regulations: a crisis of opportunity Meaningful Use. Baltimore, MD: Centers for Medicare & Medicaid Services Understanding public health informatics competencies for mid-tier public health practitioners: a web-based survey Incidence of online health information search: a useful proxy for public health risk perception Improving Access to HIV and AIDS Information Resources for Patients, Caregivers, and Clinicians: Results from the SHINE Project The use of social networking sites for public health practice and research: a systematic review The influence of social networking sites on health behavior change: a systematic review and meta-analysis Nucleating the development of telemedicine to support healthcare workers in resource-limited settings: a new approach Remote measurements of heart and respiration rates for telemedicine A workshop report on HIV mHealth synergy and strategy meeting to review emerging evidence-based mHealth interventions and develop a framework for scale-up of these interventions Effectiveness of mHealth behavior change communication interventions in developing countries: a systematic review of the literature Applying a framework for assessing the health system challenges to scaling up mHealth in South Africa mHealth in Sub-Saharan Africa mHealth Series: Text messaging data collection of infant and young child feeding practice in rural China -a feasibility study Bridging the Human Resource Gap in Primary Health Care Delivery Systems of Developing Countries With mHealth: Narrative Literature Review Influence of mHealth interventions on gender relations in developing countries: a systematic literature review Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low-and middle-income countries: a review mHealth series: New ideas for mHealth data collection implementation in low-and middle-income countries mHealth Series: Factors influencing sample size calculations for mHealth-based studies -A mixed methods study in rural China mHealth Series: Measuring maternal newborn and child health coverage by text messaging -a county-level model for China eHealth and mHealth initiatives in Bangladesh: a scoping study Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa mHealth and global mental health: still waiting for the mH2 wedding? Global Health. 10, 17. Epub 03 An mHealth model to increase clinic attendance for breast symptoms in rural Bangladesh: can bridging the digital divide help close the cancer divide? Understanding the role of mHealth and other media interventions for behavior change to enhance child survival and development in low-and middle-income countries: an evidence review Emergency Department Waiting Times (EDWaT): A Patient Flow Management and Quality of Care Rating mHealth Application. Stud Health Technol Inform Using the lives saved tool (LiST) to model mHealth impact on neonatal survival in resource-limited settings Comparing a paper based monitoring and evaluation system to a mHealth system to support the national community health worker programme, South Africa: an evaluation The effectiveness of mHealth interventions for maternal, newborn and child health in low-and middle-income countries: Protocol for a systematic review and meta-analysis Developing content for a mHealth intervention to promote postpartum retention in prevention of motherto-child HIV transmission programs and early infant diagnosis of HIV: a qualitative study Using Mobile Technology to Support Healthcare Use of mHealth systems and tools for non-communicable diseases in low-and middle-income countries: a systematic review Alves da Cruz de Andrade R, Eler GJ. mHealth data collector: an application to collect and report indicators for assessment of cardiometabolic risk. Stud Health Technol Inform Using Mobile Health (mHealth) and geospatial mapping technology in a mass campaign for reactive oral cholera vaccination in rural Haiti Communication technology use and mHealth acceptance among HIV-infected men who have sex with men in Peru: implications for HIV prevention and treatment Mobile phones and social structures: an exploration of a closed user group in rural Ghana The effectiveness of mobilehealth technologies to improve health care service delivery processes: a systematic review and meta-analysis Mobile health in China: current status and future development Mobile phone intervention reduces perinatal mortality in zanzibar: secondary outcomes of a cluster randomized controlled trial Mobile health-monitoring system through visible light communication A usability study of a mobile health application for rural Ghanaian midwives ITU e-health training program for pacific island community with the support of the Sasakawa peace foundation E-health in low-and middleincome countries: findings from the Center for Health Market Innovations Impacts of e-health on the outcomes of care in low-and middle-income countries: where do we go from here? The effectiveness of mobilehealth technology-based health behaviour change or disease management interventions for health care consumers: a systematic review Scope of policy issues in eHealth: results from a structured literature review Utilization of open source electronic health record around the world: A systematic review Mobile health for non-communicable diseases in Sub-Saharan Africa: a systematic review of the literature and strategic framework for research ICT health 2013: infrastructure and adoption by healthcare providers in Brazil. Stud Health Technol Inform An in-depth, exploratory assessment of the implementation of the National Health Information System at a district level hospital in Tanzania A framework for privacypreserving access to next-generation EHRs. Stud Health Technol Inform How do patients respond to violation of their information privacy? Secure privacy-preserving biometric authentication scheme for telecare medicine information systems Impacts of hospitals' innovativeness on information system outsourcing decisions An adaptive semantic based mediation system for data interoperability among Health Information Systems How can routine health information systems improve health systems functioning in low-and middle-income countries? Assessing the evidence base Feasibility of using teleradiology to improve tuberculosis screening and case management in a district hospital in Malawi A model for the adoption of ICT by health workers in Africa Evaluating the impact of ICT-tools on health care delivery in sub-Saharan hospitals. Stud Health Technol Inform Predicting drug-target interaction networks of human diseases based on multiple feature information Predictors of attitude and intention to use knowledge management system among Korean nurses Information and communication technology and community-based health sciences training in Uganda: perceptions and experiences of educators and students The association between computer literacy and training on clinical productivity and user satisfaction in using the electronic medical record in Saudi Arabia Computer literacy improvement needs: physicians' self assessment in the makkah region An assessment of readiness for pre-implementation of electronic health record in Iran: a practical approach to implementation in general and teaching hospitals Improving the quality of health information: a qualitative assessment of data management and reporting systems in Botswana Resources, indicators, data management, dissemination and use in health information systems in sub-Saharan Africa: results of a questionnaire-based survey Telesonography: virtual 3D image processing of remotely acquired abdominal, vascular, and fetal sonograms Development of a mini-mobile digital radiography system by using wireless smart devices Extracting important information from Chinese Operation Notes with natural language processing methods Integration of federated medical systems for vendor neutral image access in teleradiology applications. Stud Health Technol Inform Study on user interface of pathology picture archiving and communication system Policy-based approach to emergency bio-data management for mobile healthcare Activation of a medical emergency team using an electronic medical recording-based screening system* Design and development of a mobilebased system for supporting emergency triage decision making Systematic review: surveillance systems for early detection of bioterrorism-related diseases Federal Bureau of Investigation, & Amerithrax Investigation Overview of sarin terrorist attacks in Japan The severe acute respiratory syndrome Clinical Advisory Committee H1N1 Working Group NYCIE, Richardson LD. Health information exchange, biosurveillance efforts, and emergency department crowding during the spring 2009 H1N1 outbreak in New OJPHI York City First confirmed cases of Middle East respiratory syndrome coronavirus (MERS-CoV) infection in the United States, updated information on the epidemiology of MERS-CoV infection, and guidance for the public, clinicians, and public health authorities Probabilistic differential diagnosis of Middle East respiratory syndrome (MERS) using the time from immigration to illness onset among imported cases Ebola in the United States: EHRs as a Public Health Tool at the Point of Care How to exploit twitter for public health monitoring? The role of short messaging service in supporting the delivery of healthcare: An umbrella systematic review Implementation of health information technology to maximize efficiency of resource utilization in a geographically dispersed prenatal care delivery system Health information technology: transforming chronic disease management and care transitions The Hub Population Health System: distributed ad hoc queries and alerts Artificial intelligence in public health prevention of legionelosis in drinking water systems Adaptation of an internet-based depression prevention intervention for Chinese adolescents: from "CATCH-IT" to "grasp the opportunity" A question of trust: user-centered design requirements for an informatics intervention to promote the sexual health of African-American youth A coordinated national model for diabetes prevention: linking health systems to an evidence-based community program A qualitative study of health information technology in the Canadian public health system A Web-Based System for Mapping Laboratory Networks: Analysis of GLaDMap Application Timeliness of Salmonella Typhimurium notifications after the introduction of routine MLVA typing in NSW Immunisation registers in Italy: a patchwork of computerisation Health indicators in Europe: availability and data needs Praeventis, the immunisation register of the Netherlands: a tool to evaluate the National Immunisation Programme A Three-Step Approach for Creating Successful Electronic Immunization Record Exchanges between Clinical Practice and Public Health Blueprint version 2.0": updating public health surveillance for the 21st century The Future of Public Health Informatics: Alternative Scenarios and Recommended Strategies. eGEMs (Generating Evidence & Methods to improve patient outcomes World Health Organization and International Telecommunication Union World Health Organization. eHealth Technical Advisory Group The link between information and communication technologies and global public health: pushing forward Use of GIS Mapping as a Public Health Tool-From Cholera to Cancer Web-based integrated public healthcare information system of Korea: development and performance Methods for assessing the quality of data in public health information systems: a critical review Towards an intelligent decision support system for public health surveillance -a qualitative analysis of information needs. Stud Health Technol Inform Guideline for good evaluation practice in health informatics (GEP-HI) STARE-HI -Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration STARE-HI--Statement on reporting of evaluation studies in Health Informatics Ten key considerations for the successful implementation and adoption of large-scale health information technology Bridging organizational divides in health care: an ecological view of health information exchange Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science Toward a formalization of the process to select IMIA Yearbook best papers OR developing countries OR "developing nation" OR "developing nations" OR "developing population" OR "developing populations" OR "developing world" OR "less developed country" OR "less developed countries" OR "less developed nation" OR "less developed nations" OR "less developed population" OR "less developed populations" OR "less developed world" OR "lesser developed country" OR "lesser developed countries" OR "lesser developed nation" OR "lesser developed nations" OR "lesser developed population" OR "lesser developed populations" OR "lesser developed world" OR "under developed country" OR "under developed countries" OR "under developed nation" OR "under developed nations" OR "under developed population" OR "under developed populations" OR "under developed world" OR "underdeveloped country" OR "underdeveloped countries" OR "underdeveloped nation" OR "underdeveloped nations" OR "underdeveloped population" OR "underdeveloped populations" OR "underdeveloped world" OR "middle income country" OR "middle income countries" OR "middle income nation" OR "middle income nations" OR "middle income population" OR "middle income populations" OR We graciously thank the following individuals who assisted in the selection and review of the articles included in the review: Louis Imperiale (AMIA High School Scholar); Anne L. The authors do not have any competing interests to declare. This query in 2013 returned 392 results, after specific inclusion and exclusion criteria were applied 65 articles were fully reviewed. In 2014 the query returned 160 results, with 20 articles meeting the criteria for full review. While this query yielded a reasonable set for the committee to review and discuss within a short timeframe, the group feels the query is not sensitive enough to capture "information system" articles that fail to use the term "informatics." In 2013 to identify articles in GHI, we used the following query (customized for MEDLINE):("exp Informatics"[mh] OR "exp Telemedicine"[mh] OR "information system"[mp]) AND ("Developing Countries"[mh] OR global OR ministry OR "low resource" or "resource- During the review process the GHI reviewers thought this query missed many known articles. To overcome these limitations, we worked closely with an information specialist at the CDC Library to develop this more sensitive query for our 2014 MEDLINE search: