Journal IFs Are Behind Black Boxes.

What is a Journal IF?

“Impact Factor” (IF) is a term that aims to do what it says on the tin; it’s a rating of the impact of an article. But it is more than just a number for an individual article. IF is used to determine an academic’s success in a given job position, their right to a grant or future job, and on a more day-to-day level it gives them bragging rights. In a quest to increase the IF of a paper, researchers often aim to get their work published in journals that are not always the most appropriate. Instead, they would rather publish in a journal with the highest IF that will consider their paper.

IF is supposedly a metric that captures the academic importance of a journal. It is reported as being the number of citations each article gets in the previous 2 years. Wikipedia explains IF very thoroughly, and I wouldn’t want to repeat all of that here.

Rossner et al. believe that the IF is not exactly as straight forward as implied. They touch on some very troubling issues that should undermine the authority of the journal IF.

• The numerator of the IF includes different article types to the denominator. This means that articles can be factored into increasing the numerator without increasing the denominator, for example front matter can validly gather citations, but does not count as an article, allowing the journal to accumulate “free” citations.
• Articles, reviews, and front matter are curated by hand by Thomson Scientific. These designations were found to be routinely incorrect when the group purchased data from Thomson Scientific for scrutiny.
• These designations can be negotiated in a non-transparent way. This has allowed journals to vastly reduce their article number inflating the citations per article.
• Retracted articles still count.
• The mean average is skewed by blockbuster publications, meaning the IF does not reflect the majority of articles in a journal.

In a self-analysis of their 2005 IF, Nature noted that 89% of their citations came from only 25% of the papers published.

• The data you can purchase from Thomson Scientific is not the data they use to calculate the IF.

When we requested the database used to calculate the published IFs (i.e., including the erroneous records), Thomson Scientific sent us a second database. But these data still did not match the published IF data. This database appeared to have been assembled in an ad hoc manner to create a facsimile of the published data that might appease

I find it astonishing scientists tolerate such an unscientific method of deciding the scientific quality and validity of individuals and groups.

The article ends in an optimistic tone, calling for other metrics to emerge that may usurp the opaque and ill-defined IF.

Rossner, M., Van Epps, H. & Hill, E. Show me the data. J. Cell Biol. 179, 1091–1092 (2007).

Retrieved from http://jcb.rupress.org/content/179/6/1091.full