Usage Factors - Break Out Session

Richard Gedye, Research Director Oxford Journals

The Usage Factor: How we can enhance the relevance of usage as an indicator of relative value?

6 years after the launch of COUNTER (which aimed to be consistent, credible and compatible) it seemed a good time to take stock and determine how successful it has been:

• Consistency - nearly 100 publishers and hosts are now producing the reports in standards
• Credibility - formal auditing process started in 2007 and details on compliance available on website
• Compatibility - reliable comparison of amount of use, but not meaningful measure of relative quality or value

Addressing the challenge - ISI impact factor compensates for size of journal in a way COUNTER doesn't; we (the UKSG) determined that we should seek an additional measure of utility/value: the usage factor.

Usage factor = total usage over period x published during period y

Is there a demand and would it be cost-effective?

Some initial evidence about the demand from CIBER survey on behalf of STM and the PA; authors in the survey were asked to comment on the statements: article downloads are a good measure of utility and citations are a good measure of utility -70% agreed re: article downloads were a useful measure - slightly more than agreed with that statement as opposed to author citations. Download metrics would have considerable credibility amongst author community; alternatives to the impact factor... would certainly be of great appeal to librarians and many publishers"

In 2007, UKSG and COUNTER commissioned research looking at ways in which journal quality is currently assessed and the degree to which additional metrics would be of value. The resulting report, following in depth interviews with librarians and authors

Advantages
- Useful counterweight to impact factors esp. A&H journals not well covered by ISI
- For journals with high use by those who are using it but not for citation purposes (more practical journals)
- Especially helpful for journals with relatively few articles
- Data available potentially sooner than with Impact Factors
- Useful to communicate usage information to journal owners in order to compare with other journals (where they are societies etc)

Issues to address
- Only COUNTER statistics should be used and not all publishers are COUNTER compliant
- COUNTER data needs to be made more robust
- would another global measure, such as usage half-life per journal or per discipline not be of greater value?
- Concern from those publishers with strong impact factors that this would adversely affect their rankings - this will favour lower quality journals
- This will stimulate publishers to inflate their usage by every means
- How would print usage be taken into account?
- needs to amalgamate usage from all available hosting sites
- Bigger publishers have huge resources selling licenses to institutions and consortia, societies and smaller publishers do not (research shows the more you drive up usage the more you drive up the impact factor)

Who should do the calculations?

"It would be difficult for librarians to consolidate global usage statistics? Could not the publishers do this?" Publishers fed back that they would like control over the process- centralizing it would add considerable cost to the industry... Publishers on the whole were unwilling to provide their stats to a third party, but willing to have their own calculations audited... 84% of publishers were willing to calculate and publish their Usage Factors - 8% said "no", 8% "perhaps"; details needed were how would you define total usage; specified usage period and the total number of articles

What are the optimal values to measure?

Having done initial interviews, we widened it out to a web based survey - similar results to CIBER report - ~70% of authors would welcome the measures; strongest in biomedical sciences, least in arts and humanities

For librarians came in at number 2 in terms of its importance in selecting new journals (IFs were number 4); for renewing existing journals it came in at number 3, below feedback from library users and usage; but above Price, cost per download, impact factor and reputations/status of publisher.

Recommendations from the report - that the UF concept be developed further with a view to testing it as a practical implementable measure of journal quality value and status and if satisfactory results were obtained from the investigations and tests we would then need to scale up and test the system for a couple of years behind the scenes to ensure it would scale up.

In 2007 we began testing and modelling the UF concept with real data - 6 publishers, 1 aggregator, and 1 hosting service formed the Project Steering Group

Plan is that usage logs will be converted to some form of uniform standard report for third party analysis - RFP being drafted for third parties to bid for the work involved - RFP to be published later this month - still need to agree uniform standard report and details to ensure data consistency, integrity and fitness for purpose - e.g. measuring number of "items" published; assigning a correct publication year for each item (e-first can be up to 2 years ahead of "print"); excluding spiders and crawlers etc

Summer 2008, we hope to deliver a report which outlines potential usage measures which can be assessed and recommend which ones are robust enough to scale up, and any changes that publishers need to make to make the UF reliable and comparable... to propose ways in which it can be audited.

Further information at http://www.uksg.org/usagefactors or contact richard.gedye@oxfordjournals.org

Q: How do you define usage?
A: by adopting COUNTER definition - the number of successful requests for full text items (ANSI answers in the 200s and one in the 300s which covers downloading from your own cache)

Q: does this take into account purpose of downloads
A: no, and it can't - measures how much something is desired or wanted, not why. Shibboleth might be able to look at sub-dividing usage into audience type (under grads, faculty etc) - data protection may cause problems here

Q: With the expected rise of data and text mining, will this activity be excluded?
A: this is a question for COUNTER rather than the Usage Factor project; they are aware of the increase in non-traditional usage such as pre-fetching articles by Google etc. It does have to be acknowledged and scoped. Nature has a separate version solely for data mining where all the words are in the wrong order so that only a machine can read it

Q: any of the libraries are using ERM systems which are COUNTER compliant where they locally host material
A: publishers would be keen on libraries becoming COUNTER compliant and feeding this usage back to them

Q: How would you take account of "illegal" downloads?
A: raises a number of questions: assume you mean someone getting around a publishers access management system? Publishers have a built in interest in putting a stop to these; most illegal downloads come from rogue activity within existing institutional customers- robots to download large number of articles etc. COUNTER already can catch most of this and it is excluded; if it is an individual doing it manually, then we have statistical and technological means of stopping this. COUNTER need to address retrospective corrections once this abuse has been detected.

Q: as a collective activity between publishers and librarians, what is the audit process and will it be transparent/trustworthy
A: We don't know yet; one of the tasks for the appointed third party is to come up with recommendations for audits that will satisfy both libraries and publishers so that it is "properly benchmarkable". COUNTER has a relationship with ABC-e - the Audit Bureau of Circulation E Division who audit the COUNTER standards for reporting and advise on their development

Q: What about alternative versions of the articles?
A: Ebscohost are on board and of course publish their own versions of the articles; also on publishers' sites you can have an advanced version of the final articles. We are unlikely to look at author versions on Institutional Repositories, but PubMed Central will be an issue. There needs to be a technical solution to logging usage of distributed copies including authors' copies of articles.
Comment: - as use in IRs grow, these need to become COUNTER compliant and within scope, but small scale at the moment.

Q: what do you think the problem will be with eBooks etc - i.e. non-articles
A: could extend to this, but tricky with no continuity but should look again as they become more established- although perhaps perfect for major textbooks which is the closest parallel? Not much demand or impetus? Article downloads could impact authors careers, less so with books

Q: what happens if a publisher isn't COUNTER compliant (asked by a librarian)?
A: include a clause in the license agreement to encourage them to become compliant..