Microsoft Word - March_ITAL_stuart_TC proofread.docx Measuring  Journal  Linking  Success     from  a  Discovery  Service       Kenyon  Stuart,   Ken  Varnum,  and   Judith  Ahronheim       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015             52   ABSTRACT   Online  linking  to  full  text  via  third-­‐party  link-­‐resolution  services,  such  as  Serials  Solutions  360  Link  or   Ex  Libris’  SFX,  has  become  a  popular  method  of  access  to  users  in  academic  libraries.  This  article   describes  several  attempts  made  over  the  course  of  the  past  three  years  at  the  University  of  Michigan   to  gather  data  on  linkage  failure:  the  method  used,  the  limiting  factors,  the  changes  made  in  methods,   an  analysis  of  the  data  collected,  and  a  report  of  steps  taken  locally  because  of  the  studies.  It  is  hoped   that  the  experiences  at  one  institution  may  be  applicable  more  broadly  and,  perhaps,  produce  a   stronger  data-­‐driven  effort  at  improving  linking  services.   INTRODUCTION   Online  linking  via  vended  services  has  become  a  popular  method  of  access  to  full  text  for  users  in   academic  libraries.  But  not  all  user  transactions  result  in  access  to  the  desired  full  text.   Maintaining  information  that  allows  the  user  to  reach  full  text  is  a  shared  responsibility  among   assorted  vendors,  publishers,  aggregators,  local  catalogers,  and  electronic  access  specialists.  The   collection  of  information  used  in  getting  to  full  text  can  be  thought  of  as  a  supply  chain.  To   maintain  this  chain,  libraries  need  to  enhance  the  basic  information  about  the  contents  of  each   vendor  package—a  collection  of  journals  bundled  for  sale  to  libraries—with  added  details  about   local  licenses  and  holdings.  These  added  details  need  to  be  maintained  over  time.  Since  links,   platforms,  contracts,  and  subscriptions  change  frequently,  this  can  be  a  time-­‐consuming  process.   When  links  are  unsuccessfully  constructed  within  each  system,  considerable  troubleshooting  of  a   very  complex  process  is  required  to  determine  where  the  problem  lies.  Because  so  much  of  the   transaction  is  invisible  to  the  user,  linking  services  have  come  to  be  taken  for  granted  by  the   community,  and  performance  expectations  are  very  high.  Failure  to  reach  full  text  reflects  poorly   on  the  institutions  that  offer  the  links,  so  there  is  considerable  interest  for  and  value  to  the   institution  in  improving  performance.     Kenyon  Stuart  (kstuart@umich.edu)  is  Senior  Information  Resources  Specialist,  Ken  Varnum   (kvarnum@umich.edu)  is  Web  Systems  Manager,  and  Judith  Ahronheim  (jaheim@umich.edu)  is   Head,  Electronic  Resource  Access  Unit,  University  of  Michigan  Library,  Ann  Arbor,  Michigan.       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   53   Improving  the  success  rate  for  users  can  best  be  achieved  by  acquiring  a  solid  understanding  of   the  nature  and  frequency  of  problems  that  inhibit  full-­‐text  retrieval.  While  anecdotal  data  and   handling  of  individual  complaints  can  provide  incremental  improvement,  larger  improvement   resulting  from  systematic  changes  requires  more  substantial  data,  data  that  characterizes  the   extent  of  linking  failure  and  the  categories  of  situations  that  inhibit  it.   LITERATURE  REVIEW   OpenURL  link  resolvers  are  “tool[s]  that  helps  library  users  connect  to  their  institutions’   electronic  resources.  The  data  that  drives  such  a  tool  is  stored  in  a  knowledge  base.”1  Since  the   codification  of  the  OpenURL  as  an  ANSI/NISO  standard  in  2004,2  OpenURL  has  become,  in  a  sense,   the  glue  that  holds  the  infrastructure  of  traditional  library  research  together,  connecting  citations   and  full  text.  It  is  well  recognized  that  link  resolution  is  an  imperfect  science.  Understanding  what   and  how  OpenURLs  fail  is  a  time-­‐consuming  and  labor-­‐intensive  process,  typically  conducted   through  analysis  of  log  files  recording  attempts  by  users  to  access  a  full-­‐text  item  via  OpenURL.   Research  has  been  conducted  from  the  perspective  of  OpenURL  providers,  showing  which   metadata  elements  encoded  in  an  OpenURL  were  most  common  and  most  significant  in  leading  to   an  appropriate  full-­‐text  version  of  the  article  being  cited.  In  2010,  Chandler,  Wiley,  and  LeBlanc   reported  on  a  systematic  approach  they  devised,  as  part  of  a  Mellon  grant,  to  review  the  outbound   OpenURLs  from  L’Année  Philologique.3  They  began  with  an  analysis  of  the  metadata  elements   included  in  each  OpenURL  and  compared  this  to  the  standard.  They  found  that  elements  critical  to   the  delivery  of  a  full-­‐text  item,  such  as  the  article’s  starting  page,  were  never  included  in  the   OpenURLs  generated  by  L’Année  Philologique.4  Their  work  led  to  the  creation  of  the  Improving   OpenURLs  Through  Analytics  (IOTA)  working  group  within  the  National  Information  Standards   Organization  (NISO).   IOTA,  in  turn,  was  focused  on  improving  OpenURL  link  quality  at  the  provider  end.  “The  quality  of   the  data  in  the  link  resolver  knowledge  base  itself  is  outside  the  scope  of  IOTA;  this  is  being   addressed  through  the  NISO  KBART  initiative.”5,6  Where  IOTA  provided  tools  to  content  providers   for  improving  their  outbound  OpenURLs,  KBART  provided  tools  to  knowledge  base  and  linking   tool  providers  for  improving  their  data.  Pesch,  in  a  study  to  validate  the  IOTA  process,  discovered   that  well-­‐formed  OpenURLs  were  generally  successful,  however:   The  quality  of  the  OpenURL  links  is  just  part  of  the  equation.  Setting  the  proper  expectations   for  end  users  also  need  to  be  taken  into  consideration.  Librarians  can  help  by  educating  their   users  about  what  is  expected  behavior  for  a  link  resolver  and  end  user  frustrations  can  also  be   reduced  if  librarians  take  advantage  of  the  features  most  content  providers  offer  to  control   when  OpenURL  links  display  and  what  the  links  say.  Where  possible  the  link  text  should   indicate  to  the  user  what  they  will  get  when  they  click  it.7       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   54   Missing  from  the  standards-­‐based  work  described  above  is  the  role  of  the  OpenURL  middleman,   the  library.  Price  and  Trainor  describe  a  method  for  reviewing  OpenURL  data  and  identifying  the   root  causes  of  failures.  8  Through  testing  of  actual  OpenURLs  in  each  of  their  systems,  they  arrived   at  a  series  of  steps  that  could  be  taken  by  other  libraries  to  proactively  raise  OpenURL  resolution   success  rates.  Several  specific  recommendations  include  “optimize  top  100  most  requested   journals”  and  “optimize  top  ten  full  text  target  providers.”9  That  is,  make  sure  that  OpenURLs   leading  to  content  from  the  most  frequently  used  journals  and  content  sources  are  tested  and  are   functioning  correctly.  Chen  describes  a  similar  analysis  of  broken  link  reports  derived  from   Bradley  University  library’s  SFX  implementation  over  four  years,  with  a  summary  of  the  common   reasons  links  failed.10  Similarly,  O’Neill  conducted  a  small  usability  study  whose  recommendations   included  providing  “a  system  of  support  accessible  from  the  page  where  users  experience   difficulty,”11  although  her  recommendations  focused  on  inline,  context-­‐appropriate  help  rather   than  error-­‐reporting  mechanisms.   Not  found  in  the  literature  are  several  systematic  approaches  that  a  library  can  take  to  proactively   collect  problem  reports  and  manage  the  knowledge  base  accordingly.   METHOD   We  have  taken  a  two-­‐pronged  approach  to  improving  link  resolution  quality,  each  relying  on  a   different  kind  of  input.  The  first  uses  problem  reports  submitted  by  users  of  our  SummonTM-­‐ powered  article  discovery  tool,  ArticlesPlus.12  The  second  focuses  on  the  most  commonly-­‐accessed   full-­‐text  titles  in  our  environment,  based  on  reports  from  360  Link.  We  have  developed  this  dual   approach  in  the  expectation  that  we  will  catch  more  problems  on  lesser-­‐used  full-­‐text  sources   through  the  first  approach,  and  problems  whose  resolution  will  benefit  the  most  individuals   through  the  second.   User  Reports   The  University  of  Michigan  Library  uses  Summon  as  the  primary  article  discovery  tool.  When  a   user  completes  a  search  and  clicks  the  “MGet  It”  button  (see  figure  1)—MGet  It  is  our  local  brand   for  the  entire  full-­‐text  delivery  process—the  user  is  directed  to  the  actual  document  through  one   of  two  mechanisms:   1. Access  to  the  full-­‐text  article  through  a  Summon  Index-­‐Enhanced  Direct  Link.  (Some  of   Summon’s  full-­‐text  content  providers  contribute  a  URL  to  Summon  for  direct  access  to  the   full  text.  This  is  known  as  an  Index-­‐Enhanced  Direct  Linking  [Direct  Linking].)   2. Access  to  the  full  text  article  through  the  University  Library’s  link  resolver,  360  Link.  At  this   point,  one  of  two  things  can  happen:   a. The  University  Library  has  configured  a  number  of  full-­‐text  sources  as  “direct  to  full   text”  links.  When  a  citation  leads  to  one  of  these  sources,  the  user  is  directed  to  the   article  (or  as  close  to  it  as  the  content  provider’s  site  allows  (sometimes  to  an  issue       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   55   table  of  contents,  sometimes  to  a  list  of  items  in  the  appropriate  volume,  and—rarely  in   this  instance——to  the  journal’s  front  page;  the  last  outcome  is  rare  in  our  environment   because  the  University  Library  prefers  full-­‐text  links  that  get  closer  to  the  article  and   has  configured  360  Link  for  that  outcome).     b. For  those  full-­‐text  sources  that  do  not  have  direct-­‐to-­‐article  links,  360  Link  is   configured  to  provide  a  range  of  possible  delivery  mechanisms,  including  journal-­‐,   volume-­‐  or  issue-­‐level  entry  points,  document-­‐delivery  options  (for  cases  where  the   library  does  not  license  any  full-­‐text  online  sources),  the  library  catalog  (for  identifying   print  holdings  for  a  journal),  and  so  on.   From  the  user  perspective,  mechanisms  1  and  2a  are  essentially  identical.  In  both  cases,  a  click  on   the  MGet  It  icon  takes  the  user  to  the  full  text  in  a  new  browser  window.  If  the  link  does  not  lead  to   the  correct  article  for  any  reason,  there  is  no  way  in  the  new  window  for  the  library  to  collect  that   information.  Users  may  consider  item  2b  results  as  a  failure  because  the  article  is  not  immediately   perceptible,  even  if  the  article  is  actually  available  in  full  text  after  two  or  more  subsequent  clicks.   Because  of  this  user  perception,  we  interpreted  2b  results  as  “failures.”     Figure  1.  Sample  Citation  from  ArticlesPlus   In  an  attempt  to  understand  this  type  of  problem,  following  the  advice  given  by  O’Neill  and  Chen,   we  provide  a  problem-­‐reporting  link  in  the  ArticlesPlus  search-­‐results  interface  each  time  the  full-­‐ text  icon  appears  (see  the  right  side  of  figure  1).  When  the  user  clicks  this  problem-­‐reporting  link,   they  are  taken  to  a  Qualtrics  survey  form  that  asks  for  several  basic  pieces  of  information  from  the   user  but  also  captures  the  citation  information  for  the  article  the  user  was  trying  to  reach  (see   figure  2).         INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   56     Figure  2.  Survey  Questionnaire  for  Reporting  Linking  Problems   This  survey  instrument  asks  the  user  to  characterize  the  type  of  delivery  failure  with  one  of  four   common  problems,  along  with  an  “other”  text  field:   • There  was  no  article   • I  got  the  wrong  article   • I  ended  up  at  a  page  on  the  journal's  web  site,  but  not  the  article   • I  was  asked  to  log  in  to  the  publisher's  site   • Something  else  happened  (please  explain):   The  form  also  asks  for  any  additional  comments  and  requires  that  the  user  provide  an  email   address  so  that  library  staff  can  contact  the  user  with  a  resolution  (often  including  a  functioning   full-­‐text  link)  or  to  ask  for  more  information.   In  addition  to  the  information  requested  from  the  user,  hidden  fields  on  this  form  capture  the   Summon  record  ID  for  the  article,  the  IP  address  of  the  user’s  computer  (to  help  us  identify  if  the   problem  could  be  a  related  to  our  EZProxy  configuration),  a  time  and  date  stamp  of  the  report’s   submission,  and  the  brand  and  version  of  web  browser  being  used.       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   57   The  results  of  the  form  are  sent  by  email  to  the  University  Library’s  Ask  a  Librarian  service,  the   library’s  online  reference  desk.  Ask  a  Librarian  staff  make  sure  that  the  problem  is  not  associated   with  the  individual  user’s  account  (that  they  are  entitled  to  get  full  text,  that  they  were  accessing   the  item  from  on  campus  or  via  the  proxy  server  or  VPN,  etc.).  When  user-­‐centric  problems  are   ruled  out,  the  problem  is  passed  on  to  the  library’s  Electronic  Access  Unit  in  Technical  Services  for   further  analysis  and  resolution.   Random  Sampling   User-­‐reported  problems  are  only  one  picture  of  issues  in  the  linking  process.  We  were  concerned   that  user  reports  might  not  be  the  complete  story.  We  wanted  to  ensure  that  our  samples   represented  the  full  range  of  patron  experiences,  not  just  that  of  the  ones  who  reported.  So,  to  get   a  different  perspective,  we  instituted  a  series  of  random  sample  testing  using  logs  of  document   requests  from  the  link  resolver,  360  Link.   2011  Linking  Review   Our  first  large-­‐scale  review  of  linking  from  ArticlesPlus  was  conducted  in  2011.  This  first  approach   was  based  on  a  log  of  the  Summon  records  that  had  appeared  in  patron  searches  and  for  which   our  link  resolver  link  had  been  clicked.  For  this  test  we  chose  a  slice  of  the  log  covering  the  period   from  January  30–February  12,  2011.  This  period  was  chosen  because  it  was  well  into  the  academic   term  and  before  Spring  Break,  so  it  would  provide  a  representative  sample  of  the  searches  people   had  performed.  The  resulting  slice  contained  13,161  records.  For  each  record  the  log  contained   the  Summon  ID  of  the  record.  We  used  this  to  remove  duplicate  records  from  the  log  to  ensure  we   were  not  testing  linking  for  the  same  record  more  than  once,  leaving  us  with  a  spreadsheet  of   10,497  records,  one  record  per  row.  From  the  remaining  records  we  chose  a  sample  of  685   records  using  a  random  number  generator  tool,  Research  Randomizer   (http://www.randomizer.org/form.htm),  to  produce  a  random,  nonduplicating  list  of  685   numbers  with  values  from  1  to  10,497.  Each  of  the  685  numbers  produced  was  matched  to  the   corresponding  row  in  the  spreadsheet  starting  with  the  first  record  listed  in  the  spreadsheet.  For   each  record  we  collected  the  data  in  figure  3.                 INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   58   1.  The  Summon  ID  of  the  record   2.  The  raw  OpenURL  provided  with  the  record.   3.  A  version  of  the  OpenURL  that  may  have  been  locally  edited  to  put  dates  in  a  standard   format.   4.  The  final  URL  provided  to  the  user  for  linking  to  the  resource.  This  would  usually  be   the  OpenURL  from  #3  containing  the  metadata  used  by  the  link  resolver  to  build  its   full-­‐text  links.  Currently  it  is  an  intermediary  URL  provided  by  the  Summon  API.  This   URL  may  lead  to  an  OpenURL  or  to  a  Direct  Link  to  the  resource  in  the  Summon  record.   5.  The  classification  of  the  link  in  the  Summon  record.  This  was  either  “Full  Text  Online”   or  “Citation-­‐Only.”   6.  The  date  the  link  resolver  link  was  clicked.   7.  The  page  in  the  Summon  search  results  the  link  resolver  link  was  found.   8.  The  position  within  the  page  of  search  results  where  the  link  resolver  link  was  located.   9.  The  search  query  that  produced  the  search  results.   Figure  3.  Data  Points  Collected   The  results  from  this  review  were  somewhat  disappointing,  with  only  69.5%  of  the  citations   tested  leading  directly  to  full  text.  At  the  time  Direct  Linking  did  not  yet  exist,  so  “direct  to  full  text”   linking  was  only  available  through  the  1-­‐Click  feature  of  360  Link.  The  1-­‐Click  feature  attempts  to   lead  patrons  directly  to  the  full  text  of  a  resource  without  first  going  through  the  360  Link  menu.   1-­‐Click  was  used  for  579  or  84.5%  of  the  citations  tested  with  15.3%  leading  to  the  360  Link  menu.   Of  the  citations  that  used  1-­‐Click,  476  or  82.2%  led  directly  to  full  text,  so  when  1-­‐Click  was  used  it   was  rather  successful.  Links  for  about  30.5%  of  the  citations  led  either  to  a  failed  attempt  to  reach   full  text  through  1-­‐Click  or  directly  to  the  360  Link  menu.  The  2011  review  included  looking  at  the   full-­‐text  links  that  360  Link  indicated  should  lead  directly  to  the  full  text  as  opposed  to  the  journal,   volume  or  issue  level.  When  we  reviewed  all  of  the  “direct  to  full  text”  links  generated  by  360  Link,   not  only  the  ones  used  by  1-­‐Click,  we  found  a  variety  of  reasons  why  those  links  did  not  succeed  in   leading  to  the  full  text.  The  top  five  reasons  found  for  linking  failures  are  the  following:   1. incomplete  target  collection   2. incorrect  syntax  in  the  article/chapter  link  generated  by  360  Link   3. incorrect  metadata  in  the  Summon  OpenURL   4. article  not  individually  indexed   5. target  error  in  targeturl  translation       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   59   Collectively,  these  reasons  were  associated  with  the  failure  of  71.5%  of  the  “direct  to  full  text”   links.  As  we  will  show  later,  these  problems  were  also  noted  in  our  most  recent  review  of  linking   quality.   Move  to  Quarterly  Testing   After  this  review  in  2011,  we  decided  to  perform  quarterly  testing  of  the  linking  so  we  would  have   current  data  on  the  quality  of  linking.  This  would  give  us  information  on  the  effectiveness  of  any   changes  we  and  ProQuest  had  made  independently  to  improve  the  linking.  We  could  see  where   linking  problems  found  in  previous  testing  had  been  resolved  and  where  new  ones  might  exist.     However,  we  needed  to  change  how  we  created  our  sample.  While  the  data  gathered  in  2011   provided  much  insight  into  the  workings  of  360  Link,  testing  the  685  records  produced  2,210  full-­‐ text  links.  Gathering  the  data  for  such  a  large  number  of  links  required  two  months  of  part-­‐time   effort  by  two  staff  members  as  well  as  an  additional  month  of  part-­‐time  effort  by  one  staff  member   for  analysis.  This  would  not  be  workable  for  quarterly  testing.  As  an  alternative  we  decided  to  test   two  records  from  each  of  the  100  serials  most  accessed  through  the  link  resolver.  This  gave  us  a   sample  we  could  test  and  analyze  within  a  quarter  based  on  serials  that  our  patrons  were  using.   We  felt  that  we  could  gather  data  for  such  a  sample  within  three  to  four  weeks  instead  of  two   months.  The  list  was  generated  using  the  “Click-­‐Through  Statistics  by  Title  and  ISSN  (Journal  Title)”   usage  report  generated  through  the  ProQuest  Client  Center  administration  GUI.  We  searched  for   each  serial  title  within  Summon  using  the  serial’s  ISSN  or  the  serial’s  title  when  the  ISSN  was  not   available.     We  ordered  the  results  by  date,  with  the  newest  records  first.  We  wanted  an  article  within  the  first   two  to  three  pages  of  results  so  we  would  have  a  recent  article,  but  not  one  so  new  it  was  not  yet   available  through  the  resources  that  provide  access  to  the  serial.  Then  we  reordered  the  results  to   show  the  oldest  records  first  and  chose  an  article  from  the  first  or  second  page  of  results.  Our  goal   was  to  choose  an  article  at  random  from  the  second  or  third  page  while  ignoring  the  actual  content   of  the  article  so  as  not  to  introduce  a  selection  bias  by  publisher  or  journal.  Another  area  where   our  sample  was  not  truly  random  involved  supplement  issues  of  journals.  One  problem  we  found   with  the  samples  collected  was  that  they  contained  few  items  from  supplemental  issues  of   journals.  Linking  to  articles  in  supplements  is  particularly  difficult  because  of  the  different  ways   supplement  information  is  represented  among  different  databases.  To  attempt  to  capture  linking   information  in  this  case  we  added  records  for  articles  in  supplemental  issues.  Those  records  were   chosen  from  journals  found  in  earlier  testing  to  contain  supplemental  issues.  We  searched   Summon  for  articles  within  those  supplemental  issues  and  selected  one  or  two  to  add  to  our   sample.   One  notable  thing  is  the  introduction  of  Direct  Linking  in  our  Summon  implementation  between   the  reviews  for  the  first  and  second  quarters  of  2012.  ProQuest  developed  Direct  Linking  to       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   60   improve  linking  to  resources  (including  but  not  limited  to  full  text  of  articles)  through  Summon.   Instead  of  using  an  OpenURL  which  must  be  sent  to  a  link  resolver,  Direct  Linking  uses   information  received  from  the  providers  of  the  records  in  Summon  to  create  links  directly  to   resources  through  those  providers.  Ideally,  since  these  links  use  information  from  those  providers,   Direct  Linking  would  not  have  the  problems  found  with  OpenURL  linking  through  a  link  resolver   such  as  360  Link.  Not  all  links  from  Summon  use  Direct  Linking,  and  as  a  result  we  had  to  take  into   account  the  possibility  that  any  link  we  clicked  could  use  either  OpenURL  linking  or  Direct  Linking.   Current  Review:  Back  to  Random  Sampling   While  the  above  sampling  method  resulted  in  useful  data,  we  also  found  it  had  some  limitations.   When  we  performed  the  review  for  the  second  quarter  of  2012,  we  found  a  significant  increase  in   the  effectiveness  of  360  Link  since  the  first  quarter  2012  review.  This  is  further  described  in  the   findings  section  of  this  article.  We  were  able  to  trace  some  of  this  improvement  to  changes   ProQuest  had  made  to  360  Link  and  to  the  OpenURLs  produced  from  Summon.  However,  we  were   unable  to  fully  trace  the  cause  of  the  improvement  and  were  unable  to  determine  if  this  was  real   improvement  that  would  be  persistent.  To  resolve  these  problems,  we  have  returned  to  using  a   random  sample  in  our  latest  review,  but  with  a  change  in  methods.     Current  Review:  Determining  the  Sample  Size   We  wanted  to  perform  a  review  that  would  be  statistically  relevant  and  could  help  us  determine  if   any  changes  in  linking  quality  were  persistent  and  not  just  a  one-­‐time  event.  Instead  of  testing  a   single  sample  each  quarter  we  decided  to  test  a  sample  each  month  over  a  period  of  months.  One   concern  with  this  was  the  sample  size,  as  we  wanted  a  sample  that  would  be  statistically  valid  but   not  so  large  we  could  not  test  it  within  a  single  month.  We  determined  that  a  sample  size  of  300   would  be  sufficient  to  determine  if  any  month-­‐to-­‐month  changes  represent  a  real  change.   However,  in  previous  testing  we  had  learned  that  because  of  re-­‐indexing  of  the  Summon  records,   Summon  IDs  that  were  valid  when  a  patron  performed  a  search  might  no  longer  be  valid  by  the   time  of  our  testing.  We  wanted  a  sample  of  300  still-­‐valid  records,  so  we  selected  a  random  sample   larger  than  that  amount.  So,  we  decided  to  test  600  records  each  month  to  determine  if  the   Summon  IDs  were  still  valid.   Current  Review:  Methods   When  generating  each  month's  sample  we  used  the  same  method  as  in  2011.  We  asked  our  Web   Systems  group  for  the  logs  of  full-­‐text  requests  from  the  library’s  Summon  interface  for  the  period   of  November  2012–February  2013.13  We  processed  each  month’s  log  file  within  two  months  of  the   user  interactions.  To  generate  the  600-­‐record  sample,  after  removing  records  with  duplicate   Summon  IDs,  we  used  a  random  number  generator  tool,  Research  Randomizer,  to  produce  a   random,  nonduplicating  list  of  600  numbers  with  values  from  1  to  the  number  of  unique  records.   Each  of  the  600  numbers  produced  was  matched  to  the  corresponding  row  in  the  spreadsheet  of       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   61   records  with  unique  Summon  IDs.  Once  the  600  records  were  tested  and  we  had  a  subset  with   valid  Summon  IDs,  we  generated  a  list  of  300  random,  nonduplicating  numbers  with  values  from  1   to  the  number  of  records  with  valid  Summon  IDs.  Each  of  the  300  numbers  produced  was  matched   to  the  corresponding  row  in  a  spreadsheet  of  the  subset  of  records  with  valid  Summon  IDs.  This   gave  us  the  300-­‐record  sample  for  analysis.     Testing  was  performed  by  two  people,  a  permanent  staff  member  and  a  student  hired  to  assist   with  the  testing.  The  staff  member  was  already  familiar  with  the  data  gathering  and  recording   procedure  and  trained  the  student  on  this  procedure.  The  student  was  introduced  to  the  library’s   Summon  implementation  and  shown  how  to  recognize  and  understand  the  different  possible   linking  types:  Summon  Direct  Linking,  360  Link  using  1-­‐Click,  and  360  Link  leading  to  the  360   Link  menu.  Once  this  background  was  provided,  the  student  was  introduced  to  the  procedure  for   gathering  and  recording  data.  The  student  was  given  suggestions  on  how  to  find  the  article  on  the   target  site  if  the  link  did  not  lead  directly  to  the  article  and  how  to  perform  some  basic  analysis  to   determine  why  the  link  did  not  function  as  expected.  The  permanent  staff  member  reviewed  the   analysis  of  the  links  that  did  not  lead  to  full  text  and  applied  a  code  to  describe  the  reason  for  the   failure.     Based  on  our  2011  testing,  we  expected  to  see  one  of  two  general  results  in  the  current  round.     1. 360  Link  would  attempt  to  connect  directly  to  the  article  because  of  our  activating  the  1-­‐ Click  feature  of  360  Link  when  we  implemented  the  link  resolver.  With  1-­‐Click,  360  Link   attempts  to  lead  patrons  directly  to  the  full  text  of  a  resource  without  first  having  to  go   through  the  link  resolver  menu.  Even  with  1-­‐Click  active  we  provide  patrons  a  link  leading   to  the  full  360  Link  menu,  which  may  have  other  options  for  leading  to  the  full  text  as  well   as  links  to  search  for  the  journal  or  book  in  our  catalog.     2. The  other  possible  result  was  the  link  from  Summon  leading  directly  to  the  360  Link  menu.     Once  Direct  Linking  was  implemented  after  we  began  this  round,  a  third  result  became  possible  (a   direct  link  from  Summon  to  the  full  text).     For  each  record  we  collected  the  data  shown  in  figure  4.                 INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   62   1.      Date  the  link  from  Summon  record  was  tested.   2.      The  URL  of  the  Summon  record.   3.    *The  OpenURL  generated  by  clicking  the  link  from  Summon.  This  was  the  URL  in  the   address  bar  of  the  page  to  which  the  link  led.  This  is  not  available  when  Direct  Linking  is   used.   4.      The  ISSN  of  the  serial  or  ISBN  of  the  book.   5.      The  DOI  of  the  article/book  chapter  if  it  was  available.   6.      The  citation  for  the  article  as  shown  in  the  360  Link  menu  or  in  the  Summon  record  if   Direct  Linking  was  used.   7.    *Each  package  (collection  of  journals  bundled  together  in  the  knowledgebase)  for  which   360  Link  produced  an  electronic  link  for  that  citation.   8.    *The  order  in  the  list  of  electronic  resources  in  which  the  package  in  #7  appeared  in  the   360  Link  menu.   9.    *The  Linking  Level  assigned  to  the  link  by  360  Link.  This  level  indicates  how  close  to  the   article  the  link  should  lead  the  patron,  with  article-­‐level  or  chapter-­‐level  links  ideally   taking  the  patron  directly  to  the  article/book  chapter.  The  linking  levels  recorded  in  our   testing  starting  with  the  closest  to  full  text  were  article/book  chapter,  issue,  volume,   journal/book  and  database.   10.  *For  article-­‐level  links,  the  URL  that  360  Link  used  to  attempt  to  connect  to  the  article.   11.  For  all  full-­‐text  links  in  the  360  Link  menu,  the  URL  to  which  the  links  led.  This  was  the   link  displayed  in  the  browser  address  bar.   12.  A  code  assigned  to  that  link  describing  the  results.   13.  A  note  indicating  if  full  text  was  available  on  the  site  to  which  the  link  led.  This  was  only   an  indicator  of  whether  or  not  full  text  could  be  accessed  on  that  site  not  an  indicator  of   the  success  of  1-­‐Click/Direct  Linking  or  the  article-­‐level  link.   14.  A  note  if  this  was  the  link  used  by  1-­‐Click.   15.  A  note  if  Direct  Linking  was  used.   16.  A  note  if  the  link  was  for  a  citation  where  1-­‐Click  was  not  used  and  clicking  the  link  in   Summon  led  directly  to  the  360  Link  menu.   17.  Notes  providing  more  detail  for  the  results  described  by  #12.  This  included  error   messages,  search  strings  shown  on  the  target  site,  and  any  unusual  behavior.  The  notes   also  included  conclusions  reached  regarding  the  cause(s)  of  any  problems.   *  Collected  only  if  the  link  resolver  was  used.   Figure  4.  Data  Collected  from  Sample       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   63   Each  link  was  categorized  based  on  whether  it  led  to  the  full  text.  Then  the  links  that  failed  were   further  categorized  on  the  basis  of  the  reason  for  failure  (see  figure  5  for  failure  categories).     1.        Incorrect  metadata  in  the  Summon  OpenURL.   2.        Incomplete  metadata  in  the  Summon  OpenURL.   3.        Difference  in  the  metadata  between  Summon  and  the  target.  In  this  case  we  were  unable  to   determine  which  site  had  the  correct  metadata.   4.        Inaccurate  data  in  the  knowledgebase.  This  includes  incorrect  URL  and  incorrect   ISSN/ISBN.   5.        Incorrect  coverage  in  the  knowledgebase.   6.        Link  resolver  insufficiency.  The  link  resolver  has  not  been  configured  to  provide  deep   linking.  This  may  be  something  that  we  could  configure  or  something  that  would  require   changes  in  360  Link.   7.        Incorrect  syntax  in  the  article/chapter  link  generated  by  360  Link.   8.        Target  site  does  not  appear  to  support  linking  to  article/chapter  level.   9.        Article  not  individually  indexed.  This  often  happens  with  conference  abstracts  and  book   reviews  which  are  combined  in  a  single  section.   10.    Translation  error  of  the  “targeturl”  by  target  site.   11.    Incomplete  target  collection.  Site  is  missing  full  text  for  items  that  should  be  available  on   the  site.   12.    Incorrect  metadata  on  the  target  site.   13.    Citation-­‐Only  record  in  Summon.  Summon  indicates  only  the  citation  is  available  so  access   to  full  text  is  not  expected.   14.    Error  indicating  cookie  could  not  be  downloaded  from  target  site.  This  sometimes   happened  with  1-­‐Click  but  the  same  link  would  work  from  the  360  Link  menu.   15.    Item  does  not  appear  to  have  a  DOI.  The  360  Link  menu  may  provide  an  option  to  search  for   the  DOI.  Sometimes  these  searches  fail  and  we  are  unable  to  find  a  DOI  for  the  item.   16.    Miscellaneous.  Results  that  do  not  fall  into  the  other  categories.  Generally  used  for  links  in   packages  for  which  360  Link  only  provides  journal/book-­‐level  linking  such  as  Directory  of   Open  Access  Journals  (DOAJ).   17.    Unknown.  The  link  failed  with  no  identifiable  cause.     Figure  5.  List  of  Failure  Categories  Assigned       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   64   User-­‐Reported  Problems   In  March  2012,  we  began  recording  the  number  of  full-­‐text  clicks  in  ArticlesPlus  search  results   (using  Google  Analytics  events).  For  each  month,  we  calculated  the  number  of  problems  reported   per  1,000  searches  and  per  1,000  full-­‐text  clicks.  Graphed  over  time,  the  number  of  problem   reports  in  both  categories  shows  an  overall  decline.  See  figures  6  and  7.     Figure  6.  Problems  Reported  per  1,000  ArticlesPlus  Searches  (June  2011–April  2014)     Figure  7.  Problems  Reported  per  1,000  ArticlesPlus  Full-­‐Text  Clicks  (March  2012-­‐April  2014)       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   65   Our  active  work  to  update  the  Summon  and  360  Link  knowledge  bases  began  in  June  2011.  The   change  to  Summon  Direct  Linking  happened  on  February  27,  2012,  at  a  time  when  we  were   particularly  dissatisfied  with  the  volume  of  problems  reported.  We  felt  the  poor  quality  of   OpenURL  resolution  was  a  strong  argument  in  favor  of  activating  Summon  Direct  Linking.  We   believe  this  change  led  to  a  noticeable  improvement  in  the  number  of  problems  reported  per   1,000  searches  (see  figure  6).  We  do  not  have  data  for  clicks  on  the  full-­‐text  links  in  our   ArticlesPlus  interface  prior  to  March  2012,  but  do  know  that  reports  per  1,000  full-­‐text  clicks  have   been  on  the  decline  as  well  (see  figure  7).   FINDINGS   Summary  of  Random-­‐Sample  Testing  of  Link  Success     In  early  2013  we  tested  linking  from  ArticlesPlus  to  gather  data  on  the  effectiveness  of  the  linking   and  to  attempt  to  determine  if  there  were  any  month-­‐to-­‐month  changes  in  the  effectiveness  that   could  indicate  persistent  changes  in  linking  quality.  In  this  section  we  will  review  the  data   collected  from  the  four  samples  used  in  this  testing.  We  will  discuss  the  different  paths  to  full  text,   Direct  Linking  vs.  OpenURL  linking  through  360  Link,  and  their  relative  effectiveness.  We  will  also   discuss  the  reasons  we  found  for  links  to  not  lead  to  full  text.   Paths  to  Full-­‐Text  Access   As  shown  below  (see  table  1)  most  of  the  records  tested  in  Summon  used  Direct  Linking  to   attempt  to  reach  the  full  text.  The  percentage  varied  with  each  sample  tested  but  they  ranged  from   61%  to  70%.  The  remaining  records  used  360  Link  to  attempt  to  reach  the  full  text.  Most  of  the   time  when  360  Link  was  used,  1-­‐Click  was  also  used  to  reach  the  full  text.  Between  Direct  Linking   and  1-­‐Click  about  93%  to  94%  of  the  time  an  attempt  was  made  to  lead  users  directly  to  the  full   text  of  the  article  without  first  going  through  the  360  Link  menu.     Sample  1   November  2012   Sample  2   December  2012   Sample  3   January  2013   Sample  4   January  2013   Direct  Linking   205   68.3%   210   70.0%   184   61.3%   190   63.3%   360  Link/1-­‐Click   77   25.7%   70   23.3%   98   32.7%   87   29.0%   360  Link/360  Link  Menu   18   6.0%   20   6.7%   18   6.0%   23   7.7%   Total   300     300     300     300     Table  1.  Type  of  Linking       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   66   Attempts  to  reach  the  full  text  through  Direct  Linking  and  1-­‐Click  were  rather  successful.  In  the   testing,  we  were  able  to  reach  full  text  through  those  methods  from  79%  to  about  84%  of  the  time   (see  table  2).  The  remaining  cases  were  situations  where  Direct  Linking/1-­‐Click  did  not  lead   directly  to  the  full  text  or  we  reached  the  360  Link  menu.       Sample  1   November  2012   Sample  2   December  2012   Sample  3   January  2013   Sample  4   January  2013   Direct  Linking   197   65.7%   204   68.0%   173   57.7%   185   61.7%   360  Link/1-­‐Click   45   15.0%   47   15.7%   64   21.3%   55   18.3%   Total  out  of  300   242   80.7%   251   83.7%   237   79.0%   240   80.0%   Table  2.  Percentage  of  Citations  Leading  Directly  to  Full  Text   Table  3  contains  the  same  data  but  adjusted  to  remove  results  that  Summon  correctly  indicated   were  citation-­‐only.  Instead  of  calculating  the  percentages  based  on  the  full  300  citation  samples,   they  are  calculated  based  on  the  sample  minus  the  citation-­‐only  records.  The  last  row  shows  the   number  of  records  excluded  from  the  full  samples.       Sample  1   November  2012   Sample  2   December  2012   Sample  3   January  2013   Sample  4   January  2013   Direct  Linking   197   65.9%   204   69.2%   173   59.0%   185   62.5%   360  Link/1-­‐Click   45   15.1%   47   15.9%   64   21.8%   55   18.6%   Total   242   80.9%   251   85.1%   237   80.9%   240   81.1%   Records  excluded   1     5     7     4     Table  3.  Percentage  of  Citations  Leading  Directly  to  Full  Text,  Excluding  Citation-­‐Only  Results   Link  Failures  with  Summon  Direct  Linking  and  360  Link  1-­‐Click   The  next  two  tables  show  the  results  of  linking  for  records  that  used  Direct  Linking  and  the   citations  that  used  1-­‐Click  through  360  Link.  Records  that  used  Direct  Linking  were  more  likely  to   lead  testers  to  full  text  than  360  Link  with  1-­‐Click.  For  the  four  samples,  Direct  Linking  led  to  full   text  more  than  90%  of  the  time  while  1-­‐Click  led  to  full  text  from  about  58%  to  about  67%  of  the   time.   For  those  records  using  Direct  Linking  where  Direct  Linking  did  not  lead  directly  to  the  text,  the   result  was  usually  a  page  that  did  not  have  a  link  to  full  text  (see  table  4).         MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   67     Sample  1   Nov.  2012   n  =  205   Sample  2   Dec.  2012   n  =  210   Sample  3   Jan.  2013   n  =  184   Sample  4   Jan.  2013   n  =  190   Full  Text/Page  with  full-­‐text  link   197   96.1%   204   97.1%   173   94.0%   185   97.4%   Abstract/Citation  Only   6   2.9%   5   2.4%   6   3.3%   5   2.6%   Unable  to  access  full  text  through   available  full-­‐text  link   1   0.5%   1   0.5%   3   1.6%   0   0.0%   Error  and  no  full-­‐text  link  on  target   1   0.5%   0   0.0%   0   0.0%   0   0.0%   Listing  of  volumes/issues   0   0.0%   0   0.0%   1   0.5%   0   0.0%   Wrong  article   0   0.0%   0   0.0%   1   0.5%   0   0.0%   Minor  results14   0   0.0%   0   0.0%   0   0.0%   0   0.0%   Table  4.  Results  with  Direct  Linking   For  360  Link  with  1-­‐Click,  the  results  that  did  not  lead  to  full  text  were  more  varied  (see  table  5).   The  top  reasons  for  failure  included  the  link  leading  to  an  error  indicating  the  article  was  not   available  even  though  full  text  for  the  article  was  available  on  the  site,  the  link  leading  to  a  list  of   search  results  and  the  link  leading  to  the  table  of  contents  for  the  journal  issue  or  book.  In  the  last   case,  most  of  those  results  were  book  chapters  where  360  Link  only  generated  a  link  to  the  main   page  for  the  book  instead  of  a  link  to  the  chapter.           INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   68     Sample  1   Nov.  2012   n  =  77   Sample  2   Dec.  2012   n  =  70   Sample  3   Jan.  2013   n  =  98   Sample  4   Jan.  2013   n  =  87   Full  Text/Page  with  full-­‐text  link   45   58.4%   47   67.1%   64   65.3%   55   63.2%   Table  of  Contents   12   15.6%   6   8.6%   10   10.2%   6   6.9%   Error  but  full  text  available   6   7.8%   11   15.7%   10   10.2%   18   20.7%   Results  list   6   7.8%   2   2.9%   10   10.2%   4   4.6%   Error  and  no  full-­‐text  link  on  target   6   7.8%   1   1.4%   2   2.0%   2   2.3%   Wrong  article   1   1.3%   1   1.4%   1   1.0%   2   2.3%   Other   1   1.3%   0   0.0%   0   0.0%   0   0.0%   Abstract/Citation  Only   0   0.0%   0   0.0%   1   1.0%   0   0.0%   Unable  to  access  full  text  through  available   full-­‐text  link   0   0.0%   1   1.4%   0   0.0%   0   0.0%   Search  box   0   0.0%   1   1.4%   0   0.0%   0   0.0%   Minor  results15   0   0.0%   0   0.0%   0   0.0%   0   0.0%   Table  5.  Results  with  360  Link:  Citations  using  1-­‐Click   Link  Analysis  for  all  360  Link  Clicks   Unlike  the  above  tables,  which  show  the  results  on  a  citation  basis,  the  table  below  shows  the   results  for  all  links  produced  by  360  Link  (see  table  6).  This  includes  the  following:   1. links  used  for  1-­‐Click.   2. links  in  the  360  Link  menu  that  were  not  used  for  1-­‐Click  when  360  Link  attempted  to  link   to  full  text  using  1-­‐Click   3. links  in  the  360  Link  menu  where  clicking  the  link  in  Summon  led  directly  to  the  360  Link   menu  instead  of  using  1-­‐Click             MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   69     Sample  1   Nov.  2012   n  =  167   Sample  2   Dec.  2012   n  =  158   Sample  3   Jan.  2013   n  =  184   Sample  4   Jan.  2013   n  =  172   Full  Text/Page  with  full-­‐text  link   81   48.5%   84   53.2%   103   56.0%   87   50.6%   Abstract/Citation  Only   0   0.0%   0   0.0%   1   0.5%   0   0.0%   Unable  to  access  full  text  through  available   full-­‐text  link   0   0.0%   1   0.6%   0   0.0%   1   0.6%   Error  but  full  text  available   9   5.4%   14   8.9%   17   9.2%   23   13.4%   Error  and  full  text  not  accessible  through   full-­‐text  link  on  target   1   0.6%   0   0.0%   0   0.0%   0   0.0%   Error  and  no  full-­‐text  link  on  target   10   6.0%   1   0.6%   6   3.3%   5   2.9%   Failed  to  find  DOI  through  link  in  360  Link   menu   3   1.8%   5   3.2%   5   2.7%   8   4.7%   Main  journal  page   22   13.2%   24   15.2%   17   9.2%   15   8.7%   Other   2   1.2%   0   0.0%   1   0.5%   2   1.2%   360  Link  menu  with  no  full-­‐text  links   0   0.0%   2   1.3%   3   1.6%   3   1.7%   Results  list   9   5.4%   4   2.5%   10   5.4%   3   1.7%   Search  box   6   3.6%   7   4.4%   5   2.7%   8   4.7%   Table  of  Contents   12   7.2%   6   3.8%   10   5.4%   9   5.2%   Listing  of  volumes/issues   9   5.4%   9   5.7%   5   2.7%   6   3.5%   Wrong  article   3   1.8%   1   0.6%   1   0.5%   2   1.2%   Table  6.  Results  with  360  Link:  All  Links  Produced  by  360  Link   In  addition  to  recording  what  happened,  we  attempted  to  determine  why  links  failed  to  reach  full   text.  Even  though  Direct  Linking  is  very  effective,  it  is  not  100%  effective  in  linking  to  full  text.   When  excluding  records  that  indicated  that  only  the  citation,  not  full  text,  would  be  available   through  Summon,  most  of  the  problems  were  due  to  incorrect  information  in  Summon  (see  table   7).  Either  the  link  produced  by  Summon  was  incorrectly  leading  to  an  error  or  an  abstract  when       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   70   full  text  was  available  on  the  target  site  or  Summon  incorrectly  indicated  access  to  full  text  was   available.       Sample  1   Nov.  2012   n  =  8   Sample  2   Dec.  2012   n  =  6   Sample  3   Jan.  2013   n  =  11   Sample  4   Jan.  2013   n  =  5   Citation-­‐Only  record  in  Summon   1   12.5%   3   50.0%   4   36.4%   1   20.0%   Incomplete  target  collection   1   12.5%   0   0.0%   1   9.1%   1   20.0%   Incorrect  coverage  in  knowledgebase   0   0.0%   0   0.0%   2   18.2%   0   0.0%   Summon  has  incorrect  link   3   37.5%   1   16.7%   2   18.2%   2   40.0%   Summon  incorrectly  indicating  available  access   to  full  text   3   37.5%   2   33.3%   2   18.2%   1   20.0%   Table  7.  Reasons  for  Linking  Failure  to  Link  to  Full  Text  through  Direct  Linking   Table  8  shows  the  reasons  links  generated  by  360  Link  and  used  for  1-­‐Click  did  not  lead  to  full  text.   Most  of  the  failures  were  caused  by  three  general  problems:   1. incorrect  metadata  in  Summon   2. incorrect  syntax  in  the  article/chapter  link  generated  by  360  Link   3. target  site  does  not  support  linking  to  the  article/chapter  level           MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   71     Sample  1   Nov.  2012   n  =  32   Sample  2   Dec.  2012   n  =  23   Sample  3   Jan.  2013   n  =  34   Sample  4   Jan.  2013   n  =  32   Incorrect  metadata  in  the  Summon  OpenURL   2   6.3%   4   17.4%   3   8.8%   4   12.5%   Incomplete  metadata  in  the  Summon   OpenURL   0   0.0%   2   8.7%   0   0.0%   0   0.0%   Difference  in  metadata  between  Summon  and   the  target   1   3.1%   5   21.7%   0   0.0%   2   6.3%   Inaccurate  data  in  knowledgebase   0   0.0%   0   0.0%   0   0.0%   1   3.1%   Incorrect  coverage  in  knowledgebase   0   0.0%   0   4.3%   0   0.0%   0   0.0%   Link  resolver  insufficiency   2   6.3%   0   0.0%   1   2.9%   0   0.0%   Incorrect  syntax  in  the  article/chapter  link   generated  by  360  Link   6   18.8%   3   13.0%   10   29.4%   7   21.9%   Target  site  does  not  support  linking  to   article/chapter  level   11   34.3%   4   17.4%   5   14.7%   6   18.8%   Article  not  individually  indexed   0   0.0%   1   4.3%   3   8.8%   5   15.6%   Target  error  in  targetURL  translation   0   0.0%   0   0.0%   5   14.7%   3   9.4%   Incomplete  target  collection   8   25.0%   1   4.3%   1   2.9%   3   9.4%   Incorrect  metadata  on  the  target  site   0   0.0%   1   4.3%   0   0.0%   1   3.1%   Citation-­‐Only  record  in  Summon   0   0.0%   0   0.0%   0   0.0%   0   0.0%   Cookie   2   6.3%   0   0.0%   0   0.0%   0   0.0%   Item  does  not  appear  to  have  a  DOI   0   0.0%   0   0.0%   0   0.0%   0   0.0%   Miscellaneous   0   0.0%   0   0.0%   4   0.0%   0   0.0%   Unknown   0   0.0%   1   4.3%   2   5.9%   0   0.0%   Table  8.  Reasons  for  Linking  Failure  to  Link  to  Full  Text  through  1-­‐Click       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   72   Broadening  our  view  of  360  Link  to  include  all  links  generated  by  360  Link  during  the  testing,  not   only  the  ones  used  by  1-­‐Click  (see  table  9),  we  see  more  causes  of  failure  than  with  1-­‐Click.  Most  of   the  failures  were  caused  by  five  general  problems:   1. Incorrect  metadata  in  Summon.   2. Link  resolver  insufficiency.  We  mostly  used  this  classification  when  360  Link  only  provided   links  to  the  main  journal  page  or  database  page  instead  of  links  to  the  article  and  we   thought  it  might  have  been  possible  to  generate  a  link  to  the  article.  Sometimes  this  was   due  to  configuration  changes  that  we  could  have  made  and  sometimes  it  was  because  360   Link  would  only  create  article  links  if  particular  metadata  was  available  even  if  other   sufficient  identifying  metadata  was  available.   3. Incorrect  syntax  in  the  article/chapter  link  generated  by  360  Link.   4. Target  site  does  not  support  linking  to  the  article/chapter  level.   5. Miscellaneous.  Most  of  the  links  that  fell  in  this  category  were  ones  that  were  intended  to   go  the  main  journal  page  by  design.  These  were  for  journals  that  are  not  in  vendor-­‐specific   packages  in  the  knowledgebase  but  in  large  general  packages  with  many  journals  on   different  platforms.  Because  there  is  no  common  linking  syntax,  article-­‐level  linking  is  not   possible.  This  includes  packages  containing  open  source  titles  such  as  Directory  of  Open   Access  Journals  (DOAJ)  and  packages  of  subscription  titles  that  are  not  listed  in  vendor-­‐ specific  packages  in  the  knowledgebase.             MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   73     Sample  1   Nov.  2012   n  =  86   Sample  2   Dec.  2012   n  =  74   Sample  3   Jan.  2013   n  =  81   Sample  4   Jan.  2013   n  =  89   Incorrect  metadata  in  the  Summon  OpenURL   9   10.5%   5   6.8%   4   4.9%   8   9.0%   Incomplete  metadata  in  the  Summon   OpenURL   0   0.0%   2   2.7%   1   1.2%   3   3.4%   Difference  in  metadata  between  Summon  and   the  target   1   1.2%   6   8.1%   2   2.5%   2   2.2%   Inaccurate  data  in  knowledgebase   0   0.0%   0   0.0%   1   1.2%   5   5.6%   Incorrect  coverage  in  knowledgebase   3   3.5%   1   1.4%   2   2.5%   1   1.1%   Link  resolver  insufficiency   20   23.3%   15   20.3%   9   11.1%   8   9.0%   Incorrect  syntax  in  the  article/chapter  link   generated  by  360  Link   7   8.1%   3   4.1%   10   12.3%   11   12.4%   Target  site  does  not  support  linking  to   article/chapter  level   17   19.8%   6   8.1%   9   11.1%   10   11.2%   Article  not  individually  indexed   0   0.0%   1   1.4%   3   3.7%   5   5.6%   Target  error  in  targeturl  translation   1   1.2%   3   4.1%   7   8.6%   3   3.4%   Incomplete  target  collection   11   12.8%   2   2.7%   5   6.2%   5   5.6%   Incorrect  metadata  on  the  target  site   0   0.0%   1   1.4%   0   0.0%   1   1.1%   Citation-­‐Only  record  in  Summon   0   0.0%   2   2.7%   3   3.7%   3   3.4%   Cookie   2   2.3%   0   0.0%   0   0.0%   0   0.0%   Item  does  not  appear  to  have  a  DOI   2   2.3%   4   5.4%   5   6.2%   7   7.9%   Miscellaneous   13   15.1%   22   29.7%   18   22.2%   17   19.1%   Unknown   0   0.0%   1   1.4%   2   2.5%   0   0.0%   Table  9.  Reasons  for  Linking  Failure  to  Link  to  Full  Text  for  all  360  Link  Links       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   74   Comparison  of  User  Reports  and  Random  Samples   When  we  look  at  user-­‐reported  problems  during  the  same  period  over  which  we  conducted  our   manual  process  (November  1,  2012–January  29,  2013),  we  see  that  users  reported  a  problem   roughly  0.2%  of  the  time  (0.187%  of  searches  resulted  in  a  problem  report  while  0.228%  of  full-­‐ text  clicks  resulted  in  a  problem  report).  See  table  10.   Sample  Period   Problems   Reported   ArticlesPlus   Searches   MGet  It   Clicks   Problems   Reported  per   Search   Problems   Reported  per   MGet  It  Click   11/1/2012– 11/30/2012   225   111,062   95,218   0.203%   0.236%   12/1/2012– 12/31/2012   105   74,848   58,346   0.140%   0.180%   1/1/2013– 1/29/2013   100   44,204   34,692   0.226%   0.288%               Overall   430   230,114   188,256   0.187%   0.228%   Table  10.  User  Problem  Reports  During  the  Sample  Period   The  number  of  user-­‐reported  errors  is  significantly  less  than  what  we  found  through  our   systematic  sampling  (see  table  2).  Where  the  error  rate  based  on  user  reports  would  be  roughly   0.2%,  the  more  systematic  approach  showed  a  20%  error  rate.  Relying  solely  on  user  reports  of   errors  to  judge  the  reliability  of  full-­‐text  links  dramatically  underreports  true  problems  by  a  factor   of  100.     CONCLUSIONS  AND  NEXT  STEPS   Comparison  of  user  reports  to  random  sample  testing  indicates  a  significant  underreporting  of   problems  on  the  part  of  users.  While  we  have  not  conducted  similar  studies  across  other  vendor   databases,  we  suspect  that  user-­‐generated  reports  likewise  significantly  lag  behind  true  errors.   Future  research  in  this  area  is  recommended.     The  number  of  problems  discovered  in  full-­‐text  items  that  are  linked  via  an  OpenURL  is   discouraging;  however,  the  ability  of  the  Summon  Discovery  Service  to  provide  accurate  access  to   full  text  is  an  overall  positive  because  of  its  direct  link  functionality.  More  than  95%  of  direct-­‐ linked  articles  in  our  research  led  to  the  correct  resource  (table  3).  One-­‐click  (OpenURL)   resolution  was  noticeably  poorer,  with  about  60%  of  requests  leading  directly  to  the  correct  full-­‐ text  item.  More  alarming,  we  found  that,  of  full-­‐text  requests  linked  through  an  OpenURL,  a  large       MEASURING  JOURNAL  LINKING  SUCCESS  FROM  A  DISCOVERY  SERVICE  |  STUART,  VARNUM,  AND   AHRONHEIM   75   portion—20%—fail.  The  direct  links  (the  result  of  publisher-­‐discovery  service  negotiations)  are   much  more  effective.  This  discourages  us  from  feeling  any  complacency  about  the  effectiveness  of   our  OpenURL  link  resolution  tools.  The  effort  spent  maintaining  our  link  resolution  knowledge   base  does  not  make  a  long-­‐term  difference  in  the  link  resolution  quality.     Based  on  the  data  we  have  collected,  it  would  appear  that  more  work  needs  to  be  done  if  OpenURL   is  to  continue  as  a  working  standard.  While  our  data  shows  that  direct  linking  offers  improved   service  for  the  user  as  an  immediate  reward,  we  do  feel  some  concern  about  the  longer-­‐term  effect   of  closed  and  proprietary  access  paths  on  the  broader  scholarly  environment.  From  the  library’s   perspective,  the  trend  to  direct  linking  creates  the  risk  of  vendor  lock-­‐in  because  the  vendor-­‐ created  direct  links  will  not  work  after  the  library’s  business  relationship  with  the  vendor  ends.   An  OpenURL  is  less  tightly  bound  to  the  vendor  that  provided  it.  This  lock-­‐in  increases  the  cost  of   changing  vendors.  The  emergence  of  direct  links  is  a  two-­‐edged  sword:  users  gain  reliability  but   libraries  lose  flexibility  and  the  ability  to  adapt.   The  impetus  for  improving  OpenURL  linking  must  come  from  libraries  because  vendors  do  not   have  a  strong  incentive  to  take  the  lead  in  this  effort,  especially  when  it  interferes  with  their   competitive  advantage.  We  recommend  that  libraries  collaborate  more  actively  on  identifying   patterns  of  failure  in  OpenURL  link  resolution  and  remedies  for  those  issues  so  that  OpenURL   continues  as  a  viable  and  open  method  for  full-­‐text  access.  With  more  data  on  the  failure  modes   for  OpenURL  transactions,  libraries  and  content  providers  may  be  able  to  implement  systematic   improvements  in  standardized  linking  performance.  We  hope  that  the  methods  and  data  we  have   presented  form  a  helpful  beginning  step  in  this  activity.   ACKNOWLEDGEMENT   The  authors  thank  Kat  Hagedorn  and  Heather  Shoecraft  for  their  comments  on  a  draft  of  this   manuscript.   REFERENCES     1.     NISO/UKSG  KBART  Working  Group,  KBART:  Knowledge  Bases  and  Related  Tools,  January  2010,   http://www.uksg.org/sites/uksg.org/files/KBART_Phase_I_Recommended_Practice.pdf.     2.     National  Information  Standards  Organization  (NISO),  “ANSI/NISO  Z39.88  -­‐  The  OpenURL   Framework  for  Context-­‐Sensitive  Services,”  May  13,  2010,   http://www.niso.org/kst/reports/standards?step=2&project_key=d5320409c5160be4697dc 046613f71b9a773cd9e.     3.     Adam  Chandler,  Glen  Wiley,  and  Jim  LeBlanc,  “Towards  Transparent  and  Scalable  OpenURL   Quality  Metrics,”  D-­‐Lib  Magazine  17,  no.  3/4  (March  2011),   http://dx.doi.org/10.1045/march2011-­‐chandler.       INFORMATION  TECHNOLOGY  AND  LIBRARIES  |  MARCH  2015   76     4.     Ibid.   5.     National  Information  Standards  Organization  (NISO),  Improving  OpenURLs  through  Analytics   (IOTA):  Recommendations  for  Link  Resolver  Providers,  April  26,  2013,   http://www.niso.org/apps/group_public/download.php/10811/RP-­‐21-­‐2013_IOTA.pdf.   6.     NISO/UKSG  KBART  Working  Group,  KBART:  Knowledge  Bases  and  Related  Tools.   7.     Oliver  Pesch,  “Improving  OpenURL  Linking,”  Serials  Librarian  63,  no.  2  (2012):  135–45,   http://dx.doi.org/10.1080/0361526X.2012.689465.   8     Jason  Price  and  Cindi  Trainor,  “Chapter  3:  Digging  into  the  Data:  Exposing  the  Causes  of   Resolver  Failure,”  Library  Technology  Reports  46,  no.  7  (October  2010):  15–26.   9.     Ibid.,  26.   10.    Xiaotian  Chen,  “Broken-­‐Link  Reports  from  SFX  Users,”  Serials  Review  38,  no.  4  (December   2012):  222–27,  http://dx.doi.org/10.1016/j.serrev.2012.09.002.     11.    Lois  O’Neill,  “Scaffolding  OpenURL  Results,”  Reference  Services  Quarterly  14,  no.  1–2  (2009):   13–35,  http://dx.doi.org/10.1080/10875300902961940.   12.    http://www.lib.umich.edu/.  See  the  ArticlesPlus  tab  of  the  search  box.   13.    One  problem  we  had  in  testing  was  that  log  data  for  February  2013  was  not  preserved.  This   would  have  been  used  to  build  the  sample  tested  in  April  2013.  To  get  around  this  we  decided   to  take  two  samples  from  the  January  2013  log.   14.    The  “Minor  results”  row  is  a  combination  of  all  results  that  did  not  represent  at  least  0.5%  of   the  records  using  Direct  Linking  for  at  least  one  sample.  This  includes  the  following  results:   Error  but  full  text  available,  Error  and  full  text  not  accessible  through  full  text  link  on  target,   Main  journal  page,  360  Link  menu  with  no  full  text  links,  Results  list,  Search  box,  Table  of   Contents,  and  Other.   15.   The  “Minor  results”  row  is  a  combination  of  all  results  that  did  not  represent  at  least  0.5%  of   the  records  using  360  Link  for  at  least  one  sample.  This  includes  the  following  results:  Error   and  full  text  not  accessible  through  full  text  link  on  target,  Main  journal  page,  360  Link  menu   with  no  full  text  links,  Listing  of  volumes/issues.