Citation Impact of Open Access Articles vs. Articles available only through subscription ("Toll-Access")

Tim Brody <tdb01r@ecs.soton.ac.uk> [Project members: Tim Brody, Heinrich Stamerjohanns, François Vallières, Stevan Harnad, Yves Gingras, Charles Oppenheim, Steve Hitchcock]

Warning: The data presented here are preliminary unrefereed results that are still being analyzed and corrected (we welcome any suggestions or questions). This is not yet the "definitive" version of our findings. Please do not cite them without consultation with the authors.

See also The effect of open access and downloads ('hits') on citation impact: a bibliography of studies.

Introduction

The following graphs show the size of the "OA advantage" in different fields of physics and mathematics across the years from 1992-2003.

The data come from the Institute for Scientific Information (ISI) CD-ROM citation database, which covers the main journals in all fields, and from the OA Physics/Mathematics Archive http://arxiv.org/ where authors can self-archive their articles to make them Open Access (OA).

The method was to take all the physics and mathematics articles indexed by ISI from 1992-2003 and first calculate how much each article is cited. Then all these articles are divided into those that are and are not in Arxiv, hence are or are not OA.

The OA advantage is then calculated from a comparison of the citation counts for the OA versus the non-OA articles: 100(OA/non) - 100% (OA divided by non-OA citation counts, minus 100%). This gives the percentage by which citation impact is altered, positively or negatively, by making the article OA (by self-archiving it in Arxiv).

As will be seen, virtually all of the OA impact effect (red) is positive: OA enhances citation impact substantially, sometimes by several hundred percent. This is to be expected, because increasing accessibility increases rather than decreases potential usage.

Another measure of interest is the percentage of the yearly articles in a field that are self-archived and thereby made OA (green) relative to the total number of articles in the field (gray). This percentage is (slowly) increasing across the years and is already especially high in nuclear and high-energy physics, the fields in which Arxiv began. (Institutional self-archiving, especially when mandated by the institution and research funder, will help accelerate this.)

The data are analysed in four different ways. The first column is the raw averages, all OA articles in the field compared to all non-OA articles, by year. The second column is a more sensitive, controlled comparison, comparing OA vs. non-OA articles only within the same journal, by year. The third column removes all self-citations. The fourth column is based on a random sample of non-OA articles within the same journal, rather than all of them (to make the numbers more equal, because there are far more non-OA articles in most journals than OA ones). The patterns for the four analyses are much the same, but some interesting differences do emerge.

The first and last entry in each graph is (first) the average across all 12 years and (last) the average for the most recent three years (2001-2003). There are also the correlations between the OA proportion and time (the OA proportion is growing across the years), between the OA advantage and time (?) and between the OA proportion and the OA advantage (?). The OA advantage seems to be the strongest within the first 3 years of publication (including a year before publication for the preprint). For the correlation between usage impact (downloads) and later citation impact, see: http://citebase.eprints.org/analysis/correlation.php

The ISI CD-ROM data

We thank ISI for providing us with access to the ISI's bibliographic database to perform this study.

As with all bibliographic databases the ISI data we use has limitations. The ISI data available covers articles published between 1980-2004. The journals included varies each year (with new journals being added). The Science database - which these initial results are from - has articles from 6443 distinct journals. The ISI WoS (the Web-based citation index tool) covers around 2000 additional journals.

To provide a summary of how the ISI data is used, I will answer the questions posed by Henk F. Moed (who estimates an error rate of 7% for referenced citations) in "The impact-factors debate: the ISI's uses and limits," Nature, 2002:

1. Which version of the SCI was used?
   ISI CD-ROM Science and Social Science Databases
2. How were publication data collected?
   Our study compares articles and authors from only the ISI database. Those articles found to be also in arXiv.org are found by comparing an author's name and initials, and a normalised title. A "normalised" article title is generated from the three longest words in the title, in order. e.g. Witten, E "Anti De Sitter Space And Holography" is normalised to "WITTEN-E-SITTER-SPACE-HOLOGRAPHY".
3. What percentage of the total publication output is covered by the SCI and included in the dataset analysed?
   arXiv.org has around 300,000 full-text eprint articles (pre-prints, post-prints and articles that will never been published). Of these, 90,000 were found in the ISI database.
4. How were cited references matched to target articles?
   References were matched to citations by author name, year of publication, volume and starting page. It is possible that further matches could be made by using the journal abbreviation, however this has not been tested.
5. Do the indicators take into account differences in citation and publication characteristics among scientific fields?
   Different fields, with differing cultures, have different citation behaviours. Fields will also reflect differing degrees of Open Access adoption, and whether authors make use of Open Access literature. e.g. it is unlikely that highly technical fields, that require specialised databases, will make much use of Web-based OA articles.
6. What is the policy question to be answered or problem to be solved?
   See the Introduction.
7. What quality factors are used by evaluators and what is their relative weight?
   This study is interested only in citation impact. A distinction has been made for author self-citations, other than that any citation is scored as 1.
8. Do the procedures allow for comments by the scientists subjected to the analysis?
   While we welcome feedback, this study is not intended to grade or score scientists. We are only interested in whether an increase in accessibility leads to an increase in citation impact.

Notes