UKSG Conference Summary

This is the summary I put together for my SAGE colleagues. I am reposting here in case it is of use. Comments on theme omissions welcome.

The 33rd Annual UKSG Conference was held in Edinburgh a couple of weeks ago, with a varied programme and over 850 attendees. Themes gleaned from the sessions and discussions I attended are summarised below. The very active twitter stream from the conference can be found here. I was amazed at the ability of delegates to listen, assimilate and then tweet or blog all at the same time!

Social Media

An increase in focus on Social Media and new ways of sharing information and research was very evident. There was a much higher engagement and understanding of newer communication channels. Interestingly, it was not younger researchers quoted as spending more time blogging or commenting online but more established researchers who have already secured a reputation and can afford to spend time in this way. They are contributing to knowledge and growth in their field but not just in the tradition of peer-reviewed high impact journals. An interesting development would be a unique researcher identifier such as ORCID which would help tie all researcher output together.

Researchblogging is interesting, its focus is on serious posts about peer reviewed academic research. It has over 1200 blogs and has doubled in the last year. Adam Bly from Researchblogging explained that more collaborative science and the creation of new knowledge has the scientific information industry running to keep up. We’ll be contacting them to ensure posts on Sage content can be linked back to the original research.

Open Access

OA feels like it is slowly gaining traction as a publishing model. The vibe from various sessions was less if than when. Tony Hirst was emphatic on the new openness of communication channels. Why use traditional journal articles to share ideas he asked? If he or his colleagues do decide to publish an article it will be in OA jnls. However, no clear wide-ranging path for institutions to fund OA was apparent yet.

Usage

JISC Collections have secured funding to progress with the JISC usage stats portal or “make it real”. We'll be contacted for our input, the prototype involved data from Elsevier, OUP and Springer. Its aim is to present usage data in user-friendly ways for librarians and include new ways to benchmark usage. There is not clarity on confidentiality issues and was no resolution proposed on the question of the confidential nature of usage data. It’s early days.

Big Deals, Value & Pricing

An interesting session on value had Ted Bergstrom from UCSB explaining that he is securing information about big deal pricing so he can publish information about outliers in the public domain. His reason: “as citizens of the academic community, we are interested in helping librarians to understand the dynamic economic problem that they face and aiding them in negotiating effectively with large publishers. We plan to release a collection of information and analyses that will serve this purpose.” See his Big Deal Contract Project page.

As in previous years the mood seemed to be that the Big Deal bubble must burst, as it is unsustainable for many institutions, but there was no clear way forward proposed still.

Carol Tenopir talked about developing real tools for librarians to demonstrate value of their collections. She pointed us to the ARL website for more information.

Jill emery from Texas talked about patron-driven ebook and journal article acquisition stating 'the age of the article is here'. She explained they need “aggregated article access”. She wants publishers to listen as they have to purchase “Just in time not just in case'.

Quality Metrics

Frustration with the reliance on the Impact Factor and the fact it ranks journals and not articles was apparent. There is a desire to produce new ranking metrics. the Australian Research Assessment is no longer using the Impact Factor we were told.

Pete Binfield (PLOS) ran a session on how they have introduced article-level metrics - it’s worth a look if you haven’t seen. They recognise they are very much at the beginning but are very keen to do whatever they can to help users decide which content is highly valued by the community. Also, these metrics not just about evaluation but to help users filter and discover articles of value.

Pete talked about how they have work to do on working out how to measure “influence”. It’s important to demonstrate influence beyond the scientific community. This ties in with our work to show the value of social science research. How do we ensure the research we publish is credited appropriately when it influences Government policy for instance?

As Hannah Whaley says on her blog: The discussion around these issues is healthy, as is the growing volume with which librarians and researchers are willing to speak them out loud. However these key themes are notable for representing problems, not solutions. It is clear that licensing models, researcher metrics, electronic and open access still have some way to evolve to meet the growing needs and expectations of the community.

Hannah Whaley's UKSG summary

Hannah Whaley is an Assistant Director in the University of Dundee’s Library and Learning Centre, with responsibility for Research and Systems. She specialises in system design, service development and innovation within HE teaching and research. Hannah recently wrote a great blog posting identifying key themes at this year's UKSG conference, and she's kindly agreed that we can syndicate her posting here. Read on for a snapshot of the conference from Hannah's point of view - does it tally with yours? More snapshots coming soon!

The 33rd Annual UKSG Conference was in Edinburgh this week, with a varied programme and over 850 attendees. A number of themes started to recur through the sessions and discussions, as summarised:

• Big deal bubble must burst, as it is unsustainable for many institutions
• We must move further towards open access, but it is not yet clear how
• Journal impact factor isn’t good enough anymore, we need to review the commentary and produce new ranking factors
• Linked information is nearly here, allowing informal and pre-publish conversations to be viewed and measured in a structured way on the web
• The age of the article is here, meaning metrics, usage and discoverability will increasingly be at article level rather than the ‘journal container’
• Just-in-time must replace just-in-case, as no one can maintain a full array of items that may only occasionally be required

The discussion around these issues is healthy, as is the growing volume with which librarians and researchers are willing to speak them out loud. However these key themes are notable for representing problems, not solutions. It is clear that licensing models, researcher metrics, electronic and open access still have some way to evolve to meet the growing needs and expectations of the community.

(Syndicated with permission from http://www.hannahwhaley.com/2010/04/18/uksg-main-themes/)

Richard "call me Dimbleby" Gedye and the Four Horses of the Research Quality Apocalypse

We're not too thin on the ground for this morning's 9am session, which suggests either that the conference dinner was a washout (it wasn't, as witnessed by my slight queasiness and faint headache) or that this morning's session is big draw (it is - a new, Question-Time-style format for UKSG).

Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:

• attention (citations, but also more generally)
• aggregation (researcher -> institution, article -> journal)
• relation (links, related content, metadata - lots of data stored around the institution can be useful in assessing researcher value / impact).

The Impact Factor is "foundational" in this space, and "will continue to be the touchstone for the evaluation of journals for a great deal of time to come." But the area is a "playground for mathematicians, these days". Eigenfactor, for example (network-based factors) consider the journals as a node in a network, and look at relationships between different journals. These metrics show different results, in part due to the way in which different citations are weighted. Normalised aggregates enable us to compare collections across disciplines, and are arguably more flexible / useful.

Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."

COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.

Article-level usage is becoming more relevant because:

• more journal articles in repositories
• interest from authors and funding agencies
• online usage growing in credibility e.g. PLoS reporting, Knowledge Exchange
• increased practicality - COUNTER, SUSHI

Challenges include the widespread distribution of articles - dispersed usage data to be captured. PIRUS aims to create guidelines for creating and consolidating article-level usage reports. Ongoing issues are technical (scaling previous work), organisational (underpinning business models), economic (allocating costs among stakeholders) and political (engaging stakeholders). Peter closes by quoting Joel Best - statistics are not magical, and need to be considered in context.

Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.

Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.

Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"

(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).

The other side of the story: is usage data all it's cracked up to be?

"I collect them - but I don't want them, I don't need them, and I won't use them," said Ian Bannerman, likening promotional giveaway items to usage statistics. We implicitly assume that usage data is consistent, credible and compatible, and that usage factors will be a meaningful indicator of something.

In terms of credibility, Release 3 of the COUNTER guidelines is making inroads into the robot problem (crawlers distorting usage data) by publishing a list of robots and encouraging publishers to exclude these known robots from their list: but is this enough, asks Ian? It's the "amateur" crawler efforts within university IP ranges that cause the most damage and the most distortion, and COUNTER's code will not help exclude these crawlers. And if you identify a crawler (Ian's example showed an article being downloaded 6,372 times in one day), should you retrospectively exclude its activities from previously published usage statistics? Had this crawler been a little more sophisticated, behaving a little more like a "normal" user, the activity would remain unnoticed.

There's a lot going on that we don't understand and that we probably won't ever identify, and interface effects can also distort statistics: HighWire's practice of automatically displaying the HTML version of an article, whilst offering the user the ability to download a PDF from that page, was wrongly giving the impression that the ratio of PDF:HTML downloads was 1:1 (similar stats from Wiley Interscience during the same period gave a more credible 20:1). Given that COUNTER stats are part of the toolkit used by libraries when managing collections, falsely inflating one's statistics in this way is ethically dubious (upon realising this Highwire discontinued its practice).

There is a danger that in measuring the literature, we change the literature. Impact Factors can be abused, for example, by increasing self-citations to the journal in other articles/editorials; alerting authors to content they should cite (however positive the intention); publishing cite-able papers early in the year to maximise the number of citations they could garner before they become eligible for impact factoring; targetting topical areas rather than long-term studies; and so on - see Chronicle of Higher Education Oct 2005 (Monastersky's article) for additional thoughts. The Usage Factor will also not be immune to such "observer effects": authors may encourage everyone they know to download their article and improve their ranking; publishers may "sex up" keywords or seek to double downloads per the HighWire example above, or encourage online coursepacks over printed ones to maximise usage ... and so on. Ian concludes that not all attempts to influence the impact factor are necessarily bad, while those attempts to influence the usage factor are - and are less traceable. Ultimately, citations (on which IF is based) are more meaningful than downloads (on which UF is based).

Recommendations

• Be cautious; don't over-interpret usage data.
• Let's carry out further research into factors that influence article downloads
• Let's improve guidelines on detection/blocking/filtering of robots
• Let's watch out for the Observer Effect when developing usage-based metrics.

During questions, Peter Murray-Rust noted that some robotic usage is for non-aberrant purposes (e.g. data mining) and we need to be careful to distinguish between different forms of robotic activity and ensure that we are not excluding valid usage from metrics.

Reconsidering scholarly impact: MESUR

"Usage data totally rocks", chirped the endearingly-passionate Herbert van de Sompel as he attempted to rouse us all from our hangovers on this second morning of UKSG. van de Sompel's team in Los Alamos has explored interoperability, OpenURL, OAI-PMH, repository architecture and more but is now focussing on MESUR, a project intended to enhance methods of assessment of scholarly impact.

In the paper age, our best attempt to quantify scholarly impact was to count citations. But in a networked environment, we have many more metrics to deploy:

Usage-based metrics
can include numbers of accesses to scholarly material, where the come from and so forth. We can factor in usage of multiple content types (preprints, blog postings, datasets alongside journals and articles) and maintain a comprehensive record from the moment of an item's digital publication. Usage data can, however, present significant challenges. What *exactly* constitutes usage? Can we be sure to protect users' privacy? How do we standardise and aggregate data records?

Network-based metrics
can leverage citation networks, co-authorship networks and so on to assess behaviour. We need to select metrics that characterise the network and define the importance of specific nodes within that network. Tools like Google PageRank and the Eigenfactor can help us to assess networks and assign appropriate levels of significance to nodes within them.

MESUR has accumulated a vast dataset (1 billion usage events, relating to 50 million documents/100,000 publications, spanning up to five years) from multiple stakeholders in the information community. It is important to avoid bias in sampling and analysing this data. Cross-validation against existing indicators, to ensure that there is an appropriate level of correlation, allows the team to check whether their results are broadly valid. The project's goal is to assess whether metrics can be defined from usage data and how these could be used if so.

Networks are identified based on tracking a user's behaviour through a session - for example, creating a connection between the documents downloaded by a single user. Once this type of analysis has been extrapolated to a billion usage events, patterns emerge. This helps to confirm our expectations that, for example, practitioners use the literature differently to researchers. It also shows that whilst users read across multiple disciplines, their citations tend to stick to their own discipline. Correlating different metrics on maps shows that usage-based metrics tend to cluster together (basically in agreement with one another), whereas citation metrics vary both from usage metrics and from each other. Overall, there is an indication that the traditional Impact Factor (IF) "is a completely different animal" to multiple network-based metrics.

July issue of Serials now published

The Editorial Board of Serials thought it would be a good exercise to alert blog readers to the new issue of Serials which has just been published and contains many of the popular presentations from the recent conference (including T Scott Plutchak's and Melinda Kenneway's highly regarded papers). We've also included some of the breakout sessions, including Paul Harwood's briefing on the UK consortia landscape with a guide for librarians, publishers and intermediaries, and Nol Verhagen's briefing on "the licensing battlefield" from a consortium perspective.

Other papers concern the scholarly community in China, e-textbooks, the usage factors project, journal collection evaluation and much more..

Read it here (subscription or members only for current three issues) - http://uksg.metapress.com/openurl.asp?genre=issue&issn=0953-0460&volume=20&issue=2
Subscription information here: http://www.uksg.org/serials/subscribe

Let us know what you think!

Caution: statistics operating in this area

Jason Price, Claremont Colleges' Life Science Librarian and today's Duke of Hazards, took us on an entertaining journey through the potential pitfalls of over-reliance on journal usage data. He opened by warning us of some general hazards to beware:

• narrow definition of use
• COUNTER JR 1 (full text article requests) is only one dimension of use; others might be:
• A-Z list click-throughs
• citations from your faculty
• impact factor
• which journals your faculty publish in, and how much
• surveying faculty/researchers
• Page Rank-type
  
• vagaries of user behaviour
• did the user actually get any value out of something to which they clicked through
  
• Google Accelerator preloads links from pages users visit
  
• different dissemination styles of teaching
• does the lecturer download and circulate (or post internally) the PDF, or circulate a link to it?
  
• granularity of usage reports
• if report is at title-level, there's no indication of whether accesses are e.g. to purchased frontfile or free backfile

Price then moved onto some more specific hazards that may be encountered:

• determining cost per use
  
• take an annual COUNTER report - divide the package fee by the article views. But what package fee? The *stated* annual fee, or the actual cost, factoring in the additional lock-in fee?
  
• comparing to ILL cost
• the views measures in an online environment cannot be directly to correlated to what would otherwise be ordered via ILL
  
• comparing across publishers
• different interfaces can affect number of article deliveries, for example if linking to the article immediately renders its HTML version - so a user then choosing to access the PDF as well could count as two full text downloads
• Price notes that the COUNTER code of conduct does require providers to de-dupe statistics to provide a "unique article requests" figure
• exposure in Google Scholar can also skew usage
  
• (at this point, Price's laptop popped up a helpful flag to let us all know that it is hazy but warm in California this afternoon, which we all enjoyed)
  
• ignoring by-title data
• Price showed a classic long-tail curve with low use titles having very high cost per use - these would be the ones to be excluded from/replaced in future packages
  
• lack of benchmarks
• your concerns about the price per use you're paying could amplify or be assuaged by finding out what other institutions' cost per use is for the same publisher
• it's better to evaluate both cross-institution and cross-publisher to get a more general picture for comparison

Price summarised his recommendations to close before ceding the floor to COUNTER's Peter Shepherd, who opened with an overview of the COUNTER codes of practice, its recommendations and processes, and the current level of compliance within the industry. He outlined some projects which have been undertaken using COUNTER-compliant statistics, for example, the NESLi2 analysis which was able to output cross-publisher comparisons of per-article download costs and growth in full-text article downloads.

Shepherd then overviewed global metrics for evaluating e-journals, including the impact factor, and the potential for a new Usage Factor. Preliminary conclusions of UKSG's recent research in this area indicate that there is considerable support from author, librarian and publisher communities. The UKSG project outcomes also suggest that the COUNTER codes of practice may not be adequately robust, and that there remains frustration at the lack of comparable, quantitative data - particularly given that continuing print usage is not included.

Shepherd proceeded to identify some of the current issues faced by COUNTER, including:

• interface effects on usage statistics as referenced by Price earlier
• filter project concluded that it's not best practice to render the HTML regardless of the user's potential choice
• at most, usage statistics are inflated by no more than 30% as a result of multiple formats
  
• separate reporting for archives
• can currently be requested as a supplementary/sub-report
  
• usage of content within institutional repositories
• involvement with SUSHI protocol to automate retrieval of usage data from providers

And he finished with some future challenges:

• continued evolution of codes of practice - perhaps with respect to federated searching, "pre-fetching" (Google Accelerator), usability, additional data, new categories of content
• deriving metrics from the codes of practice e.g. cost per use, Usage Factor for journals