In the children’s story The Sneetches, Dr. Suess’ presents a world where certain members of society are marked by an arbitrary badge of distinction, and a canny opportunist uses this false basis of prestige for his financial gain*. What does this morality tale have to do with the scientific article recommendation service Faculty of 1000? Read on…
Currently ~3000 papers are published each day in the biosciences**. Navigating this sea of information to find articles relevant to your work is no small matter. Researchers can either sink or swim with the aid of (i) machine-based technologies based on search or text-mining tools or (ii) human-based technologies like blogs or social networking services that highlight relevant work through expert recommendation.
One of the first expert recommendation services was Faculty of 1000, a service launched in 2002 with the aim of “identifying and evaluating the most significant articles from biomedical research publications” though a peer-nominated “Faculty” of experts in various subject domains. Since the launch of F1000, several other mechanisms for expert literature recommendation have also come to the foreground, including academic social bookmarking tools like citeulike or Mendeley, the rise of Research Blogging, and new F1000-like services such as annotatr, The Third Reviewer PaperCritic and TiNYARM.
Shortly after I started my group at the University of Manchester in 2005 I was invited to join the F1000 Faculty, which I gratefully accepted. At the time, I felt that it was a mark of distinction to be invited into this select club, since I felt that it would be a good platform to voice my opinions on what work I thought was notable. I was under no illusion that my induction was based only on merit, since this invitation came from my former post-doc mentor Michael Ashburner. I overlooked this issue at the time, since when you are invited to join the “in-club” as a junior faculty member, it is very tempting since you think things like this will play a positive role in your career progression. [Whether being in F1000 has helped my career I can’t say, but certainly it can’t have hurt, and I (sheepishly) admit to using it on grant and promotion applications in the past.]
Since then, I’ve tried to contribute to F1000 when I can [PAYWALL], but since it is not a core part of my job, I’ve only contributed ~15 reviews in 5 years. My philosophy has been only to contribute reviews on articles I think are of particular note and might be missed otherwise, not to review major papers in Nature/Science that everyone is already aware of. As time has progressed and it has become harder to commit time to non-essential tasks, I’ve contributed less and less, and the F1000 staff has pestered me frequently with reminders and phone calls to submit reviews. At times the pestering has been so severe that I have considered resigning just to get them off my back. And I’ve noticed that some colleagues I have a lot of respect for have also resigned from F1000, which made me wonder if they were likewise fed up with F1000’s nagging.
This summer, while reading a post on the Tree of Life blog, Jonathan Eisen made a parenthetical remark about quitting F1000, which made me more aware of why their nagging was really getting to me:
I even posted a “dissent” regarding one of [Paul Hebert’s] earlier papers on Faculty of 1000 (which I used to contribute to before they become non open access).
This comment made me realize that the F1000 recommendation service is just another closed-access venture for publishers to make money off a product generated for free by the goodwill and labor of academics. Like closed access journals, my University pays twice to get F1000 content — once for my labor and once for the subscription to the service. But unlike a normal closed-access journal, in the case of F1000 there is not even a primary scientific publication to justify the arrangement. So by contributing to F1000, essentially I take time away from my core research and teaching activities to allow a company to commercialize my IP and pay someone to nag me! What’s even more strange about this situation is that there is no rational open-access equivalent of literature review services like F1000. By analogy with the OA publishing of the primary literature, for “secondary” services I would pay a company to post one of my reviews on someone else’s article. (Does Research Blogging for free sound like a better option to anyone?)
Thus I’ve come to realize that is unjustified to contribute secondary commentary to F1000 on Open Access grounds, in the same way it is unjustified to submit primary papers to closed-access journals. If I really support Open Access publishing, then to contribute to F1000 I must either must either be a hypocrite or make an artificial distinction between the primary and secondary literature. But this gets to the crux of the matter: to the extent that recommendation services like F1000 are crucial for researchers to make sense of the onslaught of published data, then surely these critical reviews should be Open for all, just as the primary literature should be. On the other hand, if such services are not crucial, why am I giving away my IP for free to a company to capitalize on?
Well, this question has been on my mind for a while and I have looked into whether there might be evidence that F1000 evaluations have a real scientific worth in terms of highlighting good publications that might provide a reason to keep contributing to the system. On this point the evidence is scant and mixed. An analysis by the Wellcome Trust finds a very weak correlation between F1000 evaluations and the evaluations of an internal panel of experts (driven almost entirely by a few clearly outstanding papers), with the majority of highly cited papers being missed by F1000 reviewers. An analysis by the MRC shows a ~2-fold increase in the median number of citations (from 2 to 4) for F1000 reviewed articles relative to other MRC-funded research. Likewise, an analysis of the Ecology literature shows similar trends, with marginally higher citation rates for F1000 reviewed work, but with many high impact papers being missed. [Added 28 April 2012: Moreover, multifactorial analysis by Priem et al on a range of altmetric measures of impact for 24,331 PLoS articles clearly shows that the “F1000 indicator did not have shared variability with any of the derived factors” and that “Mendeley bookmark counts correlate more closely to Web of Science citations counts than expert ratings of F1000”.] Therefore the available evidence indicates that F1000 reviews do not capture the majority of good work being published, and the work that is reviewed is only of marginally higher importance (in terms of citation) than unreviewed work.
So if (i) it goes against my OA principles, (ii) there is no evidence (on average) that my opinion matters quantitatively much more than anyone else’s, and (iii) there are equivalent open access systems to use, why should I continue contributing to F1000? The only answer I can come up with is that by being a F1000 reviewer, I gain a certain prestige for being in the “in club,” as well as by some prestige-by-association for aligning myself with publications or scientists I perceive to be important. When stripped down like this, being a member of F1000 seems pretty close to being a Sneetch with a star, and that the F1000 business model is not too different than that used by Sylvester McMonkey McBean. Realizing this has made me feel more than a bit ashamed for letting the allure of being in the old-boys club and my scientific ego trick me into something I cannot rationally justify.
So, needless to say I have recently decided to resign from F1000. I will instead continue to contribute my tagged articles to citeulike (as I have for several years) and contribute more substantial reviews to this blog via the Research Blogging portal and push the use of other Open literature recommendation systems like PaperCritic, who have recently made their user-supplied content available under a Creative Commons license. (Thanks for listening PaperCritic!).
By supporting these Open services rather than the closed F1000 system (and perhaps convincing others to do the same) I feel more at home among the ranks of the true crowd-sourced “Faculty of 1,000,000” that we need to help filter the onslaught of publications. And just as Sylvester McMonkey McBean’s Star-On machine provided a disruptive technology for overturning perceptions of prestige by giving everyone a star in The Sneetches, I’m hopeful that these open-access web 2.0 systems will also do some good towards democratizing personal recommendation of the scientific literature.
* Note: This post should in no way be taken as an ad hominem against F1000 or its founder Vitek Tracz, who I respect very much as a pioneer of Open Access biomedical publishing
** This number is an estimate based on the real figure of ~2.5K papers/day in deposited in MEDLINE, extrapolated to the large number of non-biomedical journals that are not indexed by MEDLINE. If any has better data on this, please comment below.
An Assembly of Fragments 
On it’s own, I don’t necessarily see a problem with charging for a product like F1000 *if* the product is useful and needs the money to sustain itself. Cameron Neylon’s recent post on the potential loss of valuable data reuse that could be caused by non-commerical creative commons licenses speaks to this ( http://cameronneylon.net/blog/open-access-for-the-other-85/ ). But too often F1000 reviews are not critical, not insightful and don’t add anything that isn’t said in the abstract already. I wonder if it has to do with many of the reviewers being in the “in club” in the traditional publishing sphere and carrying over their habits to F1000 (e.g. being scared to rock the boat).
The CC-NC issue you raise is a bit of a red herring, since the content under consideration here is not Open Access to begin with. Also, in principle I would have no problem with the F1000 model if they were not using free academic labor to generate their “product” and then selling it back to us. Likewise, I would have no problem if they made their back content available for free access to all, like delayed access publishers do in PMC. But in the absence of paying their content generators or making their content freely available, it seems like F1000 is operating a secondary publishing model in the same way that would be considered unacceptable for primary publishing.
Great post Casey – and it is something I have been trying to figure out for a few years now, how to get a viable open-source version of F1000 out there, which is open to everyone.
I envisaged a pretty neat way (if I say so myself) way of stopping spam and letting the cream of the ‘faculty’ rise to the top (be more visible). BUT there is always that question about who pays and how. I guess we look at Wikipedia as a model, but as we can see from their donation drives, that ain’t really free either.
A ‘freemium’ model where recommendations are free for 90 days, then archived, also throws up problems. Will libraries pay for the archives? What would someone think to see ‘their’ content an comments archvied and a price tag slapped on them? The F1M model could be great (especially backed up by ORCID to uniquely identify contributors efforts) – but…
Very interesting post Casey.
I’m really enthusiastic about the idea of OA publishing but do you really consider F1000 to be in the same league as Elsevier or Wiley?
I’m guessing that a lot of your readers have Mendeley accounts, I would also bet that a fair number pay a monthly fee for it as the free storage only goes so far. The £6 p/m that F1000 charge for a personal subscription is pretty similar to the Mendeley fees.
I guess my question is; when is it a matter of paying a reasonable fee for infrastructure etc and when does it become an OA issue?
p.s. thanks for the point towards PaperCritic, looks great!
Hi Phil –
Thanks for the comments. As I noted, I am not in principle opposed to F1000 nor do I see them as a major force acting against OA, per se, just that one needs to consistent when it comes to OA issues, otherwise we may just allow the same old problems to renew themselves in different guises. In fact, I fully acknowledge that F1000 has several innovative OA initiatives (e.g. F1000 posters) and are progressive in giving reduced rate access to institutes in developing countries. And I have no problem paying for infrastructure services that are open, democratic and not subsidized by free academic labor. What irks me about the F1000 model is that it fails on all of these counts, which perhaps why ultimately F1000 reviews are not capturing the majority of the important work being published.
Best,
Casey
Ok, that’s an interesting (and novel, to me at least) expansion of OA principles to include a degree of egalitarianism which F1000 definitely fails on.
I think that PaperCritic have really hit a sweet spot. In allowing anyone with a Mendeley account to comment on articles they stand a much better chance of getting a core of active commenters. It’s great that they are building on the tools that people already use as well, reduces the risk of succumbing to ‘yet another …’ syndrome. i really hope it catches on.
“I would also bet that a fair number pay a monthly fee for it”
Not as many as you may think. We’re starting to think that value-added services might drive more people to sign up for premium accounts than just the added space does. It could be argued that such a paid recommendations service is also extracting value from scholarly output, but no one else could do what Mendeley’s doing (no one else can look at reading activity across disciplines and publishers like we do), so I would argue that makes it a thing apart. I’ve often said that the real thing we’re selling isn’t software or storage space, but design. Kinda like how Dropbox makes file sync so easy, people pay for it gladly, even people who have other more complicated but less expensive options.
Likewise, no one else has been able to elicit such substantial and high-quality post-publication peer review as F1000 has been able to do. The quality of commentary is a direct result of the faculty recruitment and all those annoying emails, so since no one else is going to the effort of soliciting reviews like that, doesn’t that make their corpus of reviews something that’s uniquely theirs and worthy of support?
Of course, there’s more complexity to the funding model than just subscriber or author pays. I know publishers would leap at the chance to pay for F1000 exposure if it drove significant traffic, but there are also societies and companies that would be interested in how their content is being used.
I have posted a response to this interesting discussion on Naturally Selected here: http://blog.f1000.com/2011/12/23/open-letter-on-open-access/.
Hi Casey, enjoyed this post. The “secondary services” (of reviews) to “primary research data” (papers) add considerable value, they are secondary in name but not in value.
But I’m not sure citeulike, mendeley or papercritic are necessarily better places for that data. What about encouraging more readers to leave comments directly on the paper, as with PLoS?
Here’s a broken analogy, If F1000 were amazon.com, they’d charge you to read their book/product reviews…
I still believe that this whole discussion about ‘open’ and ‘closed’ is simply not progressed enough in science, especially if it comes down to proper attribution and granularity of contributions.
Finally, I agree that writing reviews on a blog and enabling a mircopayment service like Flattr http://flattr.com will add a lot. Still, the major hurdle is getting everyone on that boat …
Best crowd-sourcing regards!
F1000 evaluations | Stephens Lab Blog
A Open Archive of My F1000 Reviews « I wish you'd made me angry earlier
Open letter on open access | Naturally Selected
I love F1000, but I am totally with you on this. About 5 years ago, I used F1000 a lot. I even gave a personal subscription to students as a gift. However, Twitter and CiteULike now do the same things so much better. Why are they better? It is because there is no paywall. Why would any top scientist want to bury their opinion behind a paywall when they can share it with more people more quickly using Twitter and CiteULike. It is easy to avoid wasting time on the opinions of those without insight or expertise by changing who I follow, and I’m free to follow people who have not been anointed by someone else.
The general principle is that non-anonymous crowdsourcing works really well in academia. Indeed, that is what academia has always been. Even if I didn’t mind paying (and I do have access through my University) I don’t go to F1000 because it is inferior, and it is inferior because of the paywall.
I find the evaluations at F1000 worthless and unreliable. The vast majority of them are motivated by the reviewers’ need to indirectly promote their own work or field by promoting for the most part papers that represent mediocre or even trivial scientific contributions. In my opinion is one of the most misleading scientific enterprises.
Thanks for your in-depth considerations on the topic! Seems it was the last nail in the coffin I needed (after quite a long period of inactivity) to finally resign from F1000.
This was an interesting read, especially as I am in the process of making an application to work as an Editorial Assistant at F1000. What interested me vastly was their Open Access policy.
What I am referring to here is what is now called F1000prime, the F1000 post-publicatoin peer-review service they offer. F1000research is a new initiative that is distinct from F1000prime, run by the same company and presumably subsidised to some extent by the F1000prime subscription service that is fueled by free academic labor.
Poll: which scientific practices are ethical issues? | Dynamic Ecology
Agree with your analysis. A way to make services like F1000 unecessary is to encourage open-peer review, and pre-publication preprints. That’s the anti-thesis of F1000: where F1000 sells you the free labor or editors, themselves commenting on the free labor of authors and referee, that also make money to the journal. An open system will make all of that available for free, so that we will get _credit_ for our work as referees / commentators.
Funny though, that one of the closest things we have to this system, is actually F1000Research…
Thanks for this comment. I fully agree open peer-review and pre-printing are the way forward, and F1000Reseach (somewhat ironically) is the kind of model we should be moving to in the future.
Wouldn’t it be an approach to use pre-prints, don’t call them “pre-prints” any more, post them on arXiv or local repositories, and let scholars know when a new paper in their specific field will have appeared? Anybody who has read that paper anyway is then invited to submit a comment, rating or questions to foster an academic discourse. All free of charge, all fully open… This would be my vision of an ideal publication system of the future where we do no longer need any classical journals or so-called megajournal platforms. Any chance? Opinions?