Richard "call me Dimbleby" Gedye and the Four Horses of the Research Quality Apocalypse

We're not too thin on the ground for this morning's 9am session, which suggests either that the conference dinner was a washout (it wasn't, as witnessed by my slight queasiness and faint headache) or that this morning's session is big draw (it is - a new, Question-Time-style format for UKSG).

Jim Pringle from Thomson Reuters opens the show. We're talking about the value / quality of research, and the ways in which it can be measured. Jim suggests:

• attention (citations, but also more generally)
• aggregation (researcher -> institution, article -> journal)
• relation (links, related content, metadata - lots of data stored around the institution can be useful in assessing researcher value / impact).

The Impact Factor is "foundational" in this space, and "will continue to be the touchstone for the evaluation of journals for a great deal of time to come." But the area is a "playground for mathematicians, these days". Eigenfactor, for example (network-based factors) consider the journals as a node in a network, and look at relationships between different journals. These metrics show different results, in part due to the way in which different citations are weighted. Normalised aggregates enable us to compare collections across disciplines, and are arguably more flexible / useful.

Tools are emerging that make use of metadata relationships (funders, co-authors, patents, etc) to more powerfully and accurately measure value. "But metrics are only as good as the people who use them, and can support judgements but should never be the sole ground for judgements."

COUNTER's Peter Shepherd walks us through PIRUS2, a project that is developing a standard for reporting usage at the individual article level. Journal-level metrics may not be representative of individual articles and therefore author, and citation data is not adequate for measuring some fields. The H-Index is author-centred but can be biased towards older researchers. Overall reliance on any one metric is misleading, and distorts author behaviour.

Article-level usage is becoming more relevant because:

• more journal articles in repositories
• interest from authors and funding agencies
• online usage growing in credibility e.g. PLoS reporting, Knowledge Exchange
• increased practicality - COUNTER, SUSHI

Challenges include the widespread distribution of articles - dispersed usage data to be captured. PIRUS aims to create guidelines for creating and consolidating article-level usage reports. Ongoing issues are technical (scaling previous work), organisational (underpinning business models), economic (allocating costs among stakeholders) and political (engaging stakeholders). Peter closes by quoting Joel Best - statistics are not magical, and need to be considered in context.

Alain Peyraube from CNRS picks up the baton, talking about the European Reference Index for the Humanities project. "We need appropriate tools (for measuring impact) or we are going to lose valuable grants." In the humanities in particular, such a methodology needs to address monographs, book chapters, edited volumes etc as well as journals. The ERIH steering committee took some time to set up the project, identifying which disciplines would be considered (lots of overlap with social sciences), selecting peer review as "the only practicable method of evaluation in basic research", setting up panels for each of 15 disciplines, providing guidelines and soliciting categorised lists of journals. The project suffered from considerable misunderstanding and criticism, especially from the UK scientific community, and particularly around the categorisation of journals. Both the panels and the journal lists have been revised since the project's inception.

Hugh Look takes the stage in a fabulous red blazer (Good Morning Campers!). Trying to attach numbers to things, says Hugh, is a difficult business - numeric targets are always open to abuse, and he points us to the LSE's Michael Power for a widespread analysis of this. What's the benefit of measuring - and who benefits? What are the underlying structures, and what behaviours do they engender? How strong is the link between measurements and real-world impact? We're not currently measuring performance or quality - we're setting up a structure of control, and this risks unproductive use of public money. "The managerial class are the primary beneficiaries of the measurement culture." They manage risk to the institution - but also the risk to themselves, so they use metrics to safeguard their own position / avoid bad PR, and coerce practitioners into compliance by making it part of annual reviews. There is a flight from judgement in the way that metrics are used. Managers cannot do peer review themselves, so they mistrust it.

Hugh acknowledges that measuring things can break "who you know" elitism, but both he and Peter referred to "fetishism" around statistics. Some organisations, e.g. the RAF, are stepping back from using metrics to assess research quality - a "dawning of common sense". Metrics divert attention away from "other things that are more useful to do" - "what aren't we doing?"

(Questions / floor discussion I will post separately, or I'll be verging on LiveSerials' longest posting).

Reconsidering scholarly impact: MESUR

"Usage data totally rocks", chirped the endearingly-passionate Herbert van de Sompel as he attempted to rouse us all from our hangovers on this second morning of UKSG. van de Sompel's team in Los Alamos has explored interoperability, OpenURL, OAI-PMH, repository architecture and more but is now focussing on MESUR, a project intended to enhance methods of assessment of scholarly impact.

In the paper age, our best attempt to quantify scholarly impact was to count citations. But in a networked environment, we have many more metrics to deploy:

Usage-based metrics
can include numbers of accesses to scholarly material, where the come from and so forth. We can factor in usage of multiple content types (preprints, blog postings, datasets alongside journals and articles) and maintain a comprehensive record from the moment of an item's digital publication. Usage data can, however, present significant challenges. What *exactly* constitutes usage? Can we be sure to protect users' privacy? How do we standardise and aggregate data records?

Network-based metrics
can leverage citation networks, co-authorship networks and so on to assess behaviour. We need to select metrics that characterise the network and define the importance of specific nodes within that network. Tools like Google PageRank and the Eigenfactor can help us to assess networks and assign appropriate levels of significance to nodes within them.

MESUR has accumulated a vast dataset (1 billion usage events, relating to 50 million documents/100,000 publications, spanning up to five years) from multiple stakeholders in the information community. It is important to avoid bias in sampling and analysing this data. Cross-validation against existing indicators, to ensure that there is an appropriate level of correlation, allows the team to check whether their results are broadly valid. The project's goal is to assess whether metrics can be defined from usage data and how these could be used if so.

Networks are identified based on tracking a user's behaviour through a session - for example, creating a connection between the documents downloaded by a single user. Once this type of analysis has been extrapolated to a billion usage events, patterns emerge. This helps to confirm our expectations that, for example, practitioners use the literature differently to researchers. It also shows that whilst users read across multiple disciplines, their citations tend to stick to their own discipline. Correlating different metrics on maps shows that usage-based metrics tend to cluster together (basically in agreement with one another), whereas citation metrics vary both from usage metrics and from each other. Overall, there is an indication that the traditional Impact Factor (IF) "is a completely different animal" to multiple network-based metrics.

July issue of Serials now published

The Editorial Board of Serials thought it would be a good exercise to alert blog readers to the new issue of Serials which has just been published and contains many of the popular presentations from the recent conference (including T Scott Plutchak's and Melinda Kenneway's highly regarded papers). We've also included some of the breakout sessions, including Paul Harwood's briefing on the UK consortia landscape with a guide for librarians, publishers and intermediaries, and Nol Verhagen's briefing on "the licensing battlefield" from a consortium perspective.

Other papers concern the scholarly community in China, e-textbooks, the usage factors project, journal collection evaluation and much more..

Read it here (subscription or members only for current three issues) - http://uksg.metapress.com/openurl.asp?genre=issue&issn=0953-0460&volume=20&issue=2
Subscription information here: http://www.uksg.org/serials/subscribe

Let us know what you think!