Archive for December, 2011

Goodbye F1000, Hello Faculty of a Million

Dr. Seuss' The Sneetches

In the children’s story The Sneetches, Dr. Suess’ presents a world where certain members of society are marked by an arbitrary badge of distinction, and a canny opportunist uses this false basis of prestige for his financial gain*. What does this morality tale have to do with the scientific article recommendation service Faculty of 1000?  Read on…

Currently ~3000 papers are published each day in the biosciences**. Navigating this sea of information to find articles relevant to your work is no small matter. Researchers can either sink or swim with the aid of (i) machine-based technologies based on search or text-mining tools or (ii) human-based technologies like blogs or social networking services that highlight relevant work through expert recommendation.

One of the first expert recommendation services was Faculty of 1000, a service launched in 2002 with the aim of “identifying and evaluating the most significant articles from biomedical research publications” though a peer-nominated “Faculty” of experts in various subject domains. Since the launch of F1000, several other mechanisms for expert literature recommendation have also come to the foreground, including academic social bookmarking tools like citeulike or Mendeley, the rise of Research Blogging, and new F1000-like services such as annotatr, The Third Reviewer PaperCritic and TiNYARM.

Shortly after I started my group at the University of Manchester in 2005 I was invited to join the F1000 Faculty, which I gratefully accepted. At the time, I felt that it was a mark of distinction to be invited into this select club, since I felt that it would be a good platform to voice my opinions on what work I thought was notable. I was under no illusion that my induction was based only on merit, since this invitation came from my former post-doc mentor Michael Ashburner. I overlooked this issue at the time, since when you are invited to join the “in-club” as a junior faculty member, it is very tempting since you think things like this will play a positive role in your career progression. [Whether being in F1000 has helped my career I can't say, but certainly it can't have hurt, and I (sheepishly) admit to using it on grant and promotion applications in the past.]

Since then, I’ve tried to contribute to F1000 when I can [PAYWALL], but since it is not a core part of my job, I’ve only contributed ~15 reviews in 5 years. My philosophy has been only to contribute reviews on articles I think are of particular note and might be missed otherwise, not to review major papers in Nature/Science that everyone is already aware of. As time has progressed and it has become harder to commit time to non-essential tasks, I’ve contributed less and less, and the F1000 staff has pestered me frequently with reminders and phone calls to submit reviews. At times the pestering has been so severe that I have considered resigning just to get them off my back. And I’ve noticed that some colleagues I have a lot of respect for have also resigned from F1000, which made me wonder if they were likewise fed up with F1000′s nagging.

This summer, while reading a post on the Tree of Life blog, Jonathan Eisen made a parenthetical remark about quitting F1000, which made me more aware of why their nagging was really getting to me:

I even posted a “dissent” regarding one of [Paul Hebert's] earlier papers on Faculty of 1000 (which I used to contribute to before they become non open access).

This comment made me realize that the F1000 recommendation service is just another closed-access venture for publishers to make money off a product generated for free by the goodwill and labor of academics. Like closed access journals, my University pays twice to get F1000 content — once for my labor and once for the subscription to the service. But unlike a normal closed-access journal, in the case of F1000 there is not even a primary scientific publication to justify the arrangement. So by contributing to F1000, essentially I take time away from my core research and teaching activities to allow a company to commercialize my IP and pay someone to nag me! What’s even more strange about this situation is that there is no rational open-access equivalent of literature review services like F1000. By analogy with the OA publishing of the primary literature, for “secondary” services I would pay a company to post one of my reviews on someone else’s article. (Does Research Blogging for free sound like a better option to anyone?)

Thus I’ve come to realize that is unjustified to contribute secondary commentary to F1000 on Open Access grounds, in the same way it is unjustified to submit primary papers to closed-access journals. If I really support Open Access publishing, then to contribute to F1000 I must either must either be a hypocrite or make an artificial distinction between the primary and secondary literature. But this gets to the crux of the matter: to the extent that recommendation services like F1000 are crucial for researchers to make sense of the onslaught of published data, then surely these critical reviews should be Open for all, just as the primary literature should be. On the other hand, if such services are not crucial, why am I giving away my IP for free to a company to capitalize on?

Well, this question has been on my mind for a while and I have looked into whether there might be evidence that F1000 evaluations have a real scientific worth in terms of highlighting good publications that might provide a reason to keep contributing to the system. On this point the evidence is scant and mixed. An analysis by the Wellcome Trust finds a very weak correlation between F1000 evaluations and the evaluations of an internal panel of experts (driven almost entirely by a few clearly outstanding papers), with the majority of highly cited papers being missed by F1000 reviewers. An analysis by the MRC shows a ~2-fold increase in the median number of citations (from 2 to 4) for F1000 reviewed articles relative to other MRC-funded research. Likewise, an analysis of the Ecology literature shows similar trends, with marginally higher citation rates for F1000 reviewed work, but with many high impact papers being missed. [Added 28 April 2012: Moreover, multifactorial analysis by Priem et al on a range of altmetric measures of impact for 24,331 PLoS articles clearly shows that the "F1000 indicator did not have shared variability with any of the derived factors" and that "Mendeley bookmark counts correlate more closely to Web of Science citations counts than expert ratings of F1000".] Therefore the available evidence indicates that F1000 reviews do not capture the majority of good work being published, and the work that is reviewed is only of marginally higher importance (in terms of citation) than unreviewed work.

So if (i) it goes against my OA principles, (ii) there is no evidence (on average) that my opinion matters quantitatively much more than anyone else’s, and (iii) there are equivalent open access systems to use, why should I continue contributing to F1000? The only answer I can come up with is that by being a F1000 reviewer, I gain a certain prestige for being in the “in club,” as well as by some prestige-by-association for aligning myself with publications or scientists I perceive to be important. When stripped down like this, being a member of F1000 seems pretty close to being a Sneetch with a star, and that the F1000 business model is not too different than that used by Sylvester McMonkey McBean. Realizing this has made me feel more than a bit ashamed for letting the allure of being in the old-boys club and my scientific ego trick me into something I cannot rationally justify.

So, needless to say I have recently decided to resign from F1000. I will instead continue to contribute my tagged articles to citeulike (as I have for several years) and contribute more substantial reviews to this blog via the Research Blogging portal and push the use of other Open literature recommendation systems like PaperCritic, who have recently made their user-supplied content available under a Creative Commons license. (Thanks for listening PaperCritic!).

By supporting these Open services rather than the closed F1000 system (and perhaps convincing others to do the same) I feel more at home among the ranks of the true crowd-sourced “Faculty of 1,000,000″ that we need to help filter the onslaught of publications. And just as Sylvester McMonkey McBean’s Star-On machine provided a disruptive technology for overturning perceptions of prestige by giving everyone a star in The Sneetches, I’m hopeful that these open-access web 2.0 systems will also do some good towards democratizing personal recommendation of the scientific literature.

* Note: This post should in no way be taken as an ad hominem against F1000 or its founder Vitek Tracz, who I respect very much as a pioneer of Open Access biomedical publishing

** This number is an estimate based on the real figure of ~2.5K papers/day in deposited in MEDLINE, extrapolated to the large number of non-biomedical journals that are not indexed by MEDLINE.  If any has better data on this, please comment below.

Time Management Tips from Francis Crick

Academic researchers nowadays are asked to participate in a multitude of tasks outside the core remit of a scholar, which have succinctly been summarized by Scott Hawley as “to learn, to write and to teach.” Deftly handling requests for participation in “non-core” activities is an art, but is essential if one wishes to maintain an active research programme. While it is clear that email and computers have made things worse, this problem is indeed not new and we can look to history for good strategies to cope with it. Chris Beckett writes of:

A response strategy [Francis] Crick adopted in the 1960s to cope with an enormous post and to make a serious point playfully was the occasional use of a pre-printed postcard offering a number of reply options. The seventeen listed (see Figure 3) are a faithful reflection of the requests he regularly received.

Francis Crick's all-purpose reply card.

While I don’t expect to have opportunity to use many of these myself in the future, and there are some that I don’t agree with rejecting outright (e.g. read your manuscript), this list serves as a checklist for non-core academic activities and a useful reminder of what we didn’t go into science for in the first place.

Inaugural

For the last two years, my lab has run a blog on issues relating to technical matters in bioinformatics and evolutionary biology. Increasingly, I have felt that there are issues that deserve to be commented on in science and society that go beyond this remit, and in order to separate my work and personal views I’ve decided to establish a new blog for these more opinionated posts. Posts with a timestamp earlier than this one are opinions migrated over from my work site. I also hope to get involved more in the Research Blogging network to share my views on interesting or controversial work being published in my area of expertise.

Enjoy!

Just Say No – The Roberts/Ashburner Response

UPDATE: see follow-up post “The Roberts/Ashburner Response” to get more of the story on the origin of this letter.

I had the pleasure of catching up with my post-doc mentor Michael Ashburner today, and among other things we discussed the ongoing development of UKPMC and the importance of open access publishing. Although I consider myself a strong open access advocate, I did not sign the PLoS open letter in 2001, since at the time I was a post-doc and not in a position fully to control where I published. Therefore I couldn’t be sure that I could abide by the manifesto 100%, and didn’t want to put my name to something I couldn’t deliver on. As it turns out this is still the case to a certain degree and (because of collaborations) my freely-available-article-index remains at a respectable 85% (33/39), but alas will never reach the coveted 100% mark.

Nevertheless, I have steadily adopted most of the policies of the open letter, especially as my group has gotten more heavily involved in text-mining research over the years. This became especially true after a nasty encounter with one publisher in 2008 caused campus IT to shutdown my office IP for downloading articles from a journal for which our University has a site license, which radicalized me into more of an open access evangelist. After discussing this event at the time with Ashburner, he reminded me of the manifesto and one of its most powerful tools for changing the landscape of scholarly publishing – refusing to reviewing for journals/publishers who do not submit their content to PubMed Central (see the white-list of journals here).

I have dug this letter out countless times since then and used versions of it when asked to review for non-PMC journals, as it expresses the principles in plain and powerful language. I had another call to dig it out today and thought that I’d post the “Ashburner response” so others have a model to follow if they chose this path.

Enjoy!

From: “Michael Ashburner” <michael.ashburner@xxx.xxx>
Date: 30 August 2008 13:48:03 GMT+01:00
To: “Casey Bergman” <casey.bergman@xxx.xxx>
Subject: Just say No

Dear Editor,

Thank you for your invitation to review for your journal. Because it is not open access and does not provide its back content to PubMed Central, or any similar resource, I regret that I am unwilling to do this.

I would urge you to seriously reconsider both policies and would ask that you send this letter to your co-editors and publisher. In the event that you do change your policy, even to the extent of providing your back content to PubMed Central, or a similar resource, then I will be happy to review for you.

The scientific literature is at present the most significant resource available to researchers. Without access to the literature we cannot do science in any scholarly manner. Your journal refuses to embrace the idea that the purpose of the scientific literature is to communicate knowledge, not to make a profit for publishers. Without the free input of manuscripts and referees’ time your journal would not exist. By and large, the great majority of the work you publish is paid for by taxpayers. We now, either as individuals or as researchers whose grants are top-sliced, have to pay to read our own work and that of our colleagues, either personally or through our institutes’ libraries. I find that, increasingly, literature that is not available by open access is simply being ignored. Moreover, I am very aware that, increasingly, discovering information from the literature relies on some sort of computational analysis. This can only be effective if the entire content of primary research papers is freely available. Finally, by not being an open access journal you are disenfranchising both scientists who cannot afford (or whose institutions cannot afford) to pay for access and the general public.

There are now several good models for open access publication, and I would urge your journal to adopt one of these. There is an extensive literature on open access publishing, and its economic implications. I would be pleased to send you references to this literature.

Yours sincerely,

Michael Ashburner

Related Posts:


Twitter Updates


Follow

Get every new post delivered to your Inbox.

Join 72 other followers