Suggesting Reviewers in the Era of arXiv and Twitter

Along with many others in the evolutionary genetics community, I’ve recently converted to using arXiv as a preprint server for new papers from my lab. In so doing, I’ve confronted an unexpected ethical question concerning pre-printing and the use of social media, which I was hoping to generate some discussion about as this practice becomes more common in the scientific community. The question concerns the suggestion of reviewers for a journal submission of a paper that has previously been submitted to arXiv and then subsequently discussed on social media platforms like Twitter. Specifically put, the question is: is it ethical to suggest reviewers for a journal submission based on tweets about your arXiv preprint?

To see how this ethical issue arises, I’ll first describe my current workflow for submitting to arXiv and publicizing it on Twitter. Then, I’ll propose an alternative that might be considered to be “gaming” the system, and discuss precedents in the pre-social media world that might inform the resolution of this issue.

My current workflow for submission to arXiv and announcement on twitter is as follows:

  1. submit manuscript to a journal with suggested reviewers based on personal judgement;
  2. deposit the same version of the manuscript that was submitted to journal in arXiv;
  3. wait until arXiv submission is live and then tweet links to the arXiv preprint.

From doing this a few times (as well as benefiting from additional Twitter exposure via Haldane’s Sieve), I’ve realized that there can often be fairly substantive feedback about an arXiv submission via twitter in the form of who (re)tweets links to it and what people are saying about the manuscript. It doesn’t take much thought to realize that this information could potentially be used to influence a journal submission in the form of which reviewers to suggest or oppose using an alternative workflow:

  1. submit manuscript to arXiv;
  2. wait until arXiv submission is live and then tweet about it;
  3. moniter and assimilate feedback from Twitter;
  4. submit manuscript to journal with suggested and opposed reviewers based on Twitter activity.

This second workflow incidentally also arises under the first workflow if your initial journal submission is rejected, since there would naturally be a time lag in which it would be difficult to fully ignore activity on Twitter about an arXiv submission.

Now, I want to be clear that I haven’t and don’t intend to use the second workflow (yet), since I have not fully decided if this an ethical approach to suggesting reviewers. Nevertheless, I lean towards the view that it is no more or less ethical than the current mechanisms of selecting suggested reviewers based on: (1) perceived allies/rivals with relevant expertise or (2) informal feedback on the work in question presented at meetings.

In the former case of using who you perceive to be for or against your work, you are relying on personal experience and subjective opinions about researchers in your field, both good and bad, to inform your choice of suggested or opposed reviewers. This is some sense no different qualitatively to using information on Twitter prior to journal submission, but is instead based on a closed network using past information, rather than an open network using information specific to the piece of work in question. The latter case of suggesting reviewers based on feedback from meeting presentations is perhaps more similar to the matter at hand, and I suspect would be considered by most scientists to be a perfectly valid mechanism to suggest or oppose reviewers for a journal submission.

Now, of course I recognize that suggested reviewers are just that, and editors can use or ignore these suggestions as they wish, so this issue may in fact be moot. However, based on my experience, suggested reviewers are indeed frequently used by editors (if not, why would they be there?). Thus resolving whether smoking out opinions on Twitter is considered “fair play” is probably something the scientific community should consider more thoroughly in the near future, and I’d be happy to hear what other folks think about this in the comments below.

18 thoughts on “Suggesting Reviewers in the Era of arXiv and Twitter

  1. I wrote a comment that my phone appears to have eaten, so I apologize if this turns into a double post.

    Here’s a bit of a thought experiment: a major rationale for posting preprints and discussing them on social media is to provide an “extra layer of review”. So, suppose someone writes a long-form review of your paper in an online venue like Haldane’s Sieve. It seems reasonable to want this information included in an editor’s evaluation, and if you agree then the question is ‘how?’. Should you suggest that person as a reviewer? Should you try to refer the editor directly to the online review? I’d prefer the latter since it seems less sneaky, but I don’t know how an editor would respond (I’ll ask next time I see one).

    For cases that are less clear cut than this scenario — say, identifying likely positive reviewers via retweets on Twitter — I don’t see how these mechanisms are much different than others ways for coming up with lists of suggested reviewers. With all the grey area here, I think the onus lies on the editor to not rely too heavily on suggested reviewers and to spot uncritical reviews from such people. For what it’s worth, I’ve seen reviews like these backfire; e.g., a short review saying “This is great! No comments.” isn’t very compelling.

    • I was mainly thinking about simple posts on twitter, but you (and Graham) raise an interesting point about longer commentary on preprints, which is another issue that needs more consideration (my response is below). On the issue of tweets/retweets it seems like we are more or less on the same page – they are not really qualitatively different than other ways of suggesting reviewers. And I fully agree that, like any suggested reviewers, editors should use these suggestions judiciously. Part of my goal in opening this discussion is to make editors aware of this issue, and perhaps encourage them do a quick twitter search themselves while making a decision about who to send a manuscript out to.

  2. I don’t see much special about the arXiv and twitter with respect to this. Through talks and sending around papers, and just prior knowledge, people have long guessed what particular reviewers might say.

    if someone was negative about the work at a meeting or on twitter I would try and address their comments. I might not list them as a reviewer, but it isn’t usually grounds to exclude them [I see that mainly as something to be used sparingly for situations where there is a true conflict of interest]. However, in general I’ve usually found that people who comment publicly on my work are positive about it.

    If someone gives a bunch of substantial feedback prior to submission, I won’t list them as a reviewer, as got feedback already. I acknowledge these folks in the paper so the editor can see who has contributed thoughts on the paper already.

    I try to avoid cherry picking reviewers/editors that I think will favour my work [whether they have said anything about a particular article or not], and to go for the people who are the experts on a topic. In the short term it may lead to more work, and some rejections, but in the long term I think I benefit from diverse opinions on my work.

    I think the bigger issue here is how we will deal with twitter comments/blogs/arXived responses as reviewers and editors.

    • OK, another voice supporting the view that using info from twitter is not really different than using other sources to suggest reviewers.

      On the whole I agree that most information you would get from tweets about an arXiv preprint would tend towards the positive – my sense is that fewer people are willing to tweet criticisms about a paper than they are to propogate links (which I take a as a form of endorsement of the work specifically or the field in general).

      I respect and share your view that opposed reviewers should be used sparingly, and I don’t use them often. But there are situations where people have strong differences of opinion, which can unfortunately be wielded to block your work from getting published, so opposing reviewers can be helpful in preventing pitbull reviewing. And as noted above, I don’t think information from twitter is as useful in identifying potential rivals as it is in identifiying potential allies, so maybe the twitter issue is less relevant in this regard.

      As you and Bryan note, a potentially bigger issue than the one I initially raised is how to deal with longer commentaries on preprints, and how this information should be used in reviewer suggestion/selection. I like your idea of listing people who give long prepublication commentary in the acknowledgements. But I wouldn’t necessarily exclude someone who has commented on a paper and who is listed in the acknowledgments from being a potential reviewer. This is mainly because I find informal, prepublication comments are usually at a different level of scrutiny than actual peer review – usually more general (this is generally what I am asking for) and more constructive (presumably since the identity of the reviewer is known). Also, since I’m not forced to take their input on board, I often accomodate informal review in a different manner (as suggestions, more than requests). The same person acting as an anonymous reviewer may not give the same review, and in fact I’d wager would give a harsher review on average, with more specific feedback (what to cite, etc.) and with stronger demands on the essential nature of the requested changes. What this implies about anonymous peer review, I’ll leave to the reader’s imagination.

  3. As a mathematician, I can’t wrap my head around this whole “suggested reviewers” thing. Isn’t peer review supposed to be anonymous and everything? If I were an EiC and someone suggested a reviewer, I’d go out of my way to avoid that reviewer!

    • I guess this is a difference among disciplines then, since in Biology it is often the case the journals request you to list 2-4 suggested reviewers on submission in the manuscript tracking system or cover letter. I don’t have data on the frequency of journals that adopt this practice, but in the PLOS and BMC families of journals, I know this is standard. I also note that this is standard for grant submission in the UK as well (e.g to BBSRC). I agree it seems nonsensical in terms of generating all sorts of potential conflicts and opportunities for gaming, but apparently the need for editors/grant officers to find scientists who are likely to accept the review is high enough to offset this concern.

  4. It’s the journal editor’s call to select appropriate reviewers; if they look at your suggested reviewers and think they’re relevant, then that’s good enough, IMO. In any case, it’s just another form of social networking. And I’d love to get a paper to review that I cared about, frankly ;)

  5. Using the second workflow I wonder if you’d find yourself in the situation where you’d allow someone with a competing or similar project the opportunity to scoop you. If they know your work is coming, they may try to submit their work early as to get ahead of you in the publishing queue as you’re compiling and pondering responses from the Interwebs.

    • The concern about scooping I think has more to do with posting a preprint of your work per se, than the issue of suggesting reviewers based on preliminary feedback on twitter. My view on the potential scoping issue is actually that by posting a preprint you are actually establishing priority, and the more that you publicize a preprint on arXiv, the more difficult you make it for authors or reviewers to legitimately ignore your work. And to think that backroom whispering and leaks about papers that have been submitted never happen via the old boys network, well guess again :)

  6. Do we know how often editors ignore suggested reviewers? my feeling is that they recognise if you’ve suggested all your friends and then ask alternative reviewers anyway

    • This is an important point and one I really don’t have any hard data on, other than informal feedback from people who have said they reviewed my work after the fact, who I had also put down as suggested reviewers.

      But actually identifying potentially favorable reviewers on twitter issue is a way to anonymously identify people who are not specifically your “friends” or even in your social network. Using twitter to poach potential reviewers who you don’t “know” and can therefore be seen to be independent is exactly the kind of issue I worry can be interpreted as gaming the system. As noted in my reply to Bryan, I think the real issue is that editors need to be aware of this general issue and do a quick arXiv and twitter search before sending a paper out for review, in order to balance their decision making and offset any potential gaming that might be at work.

  7. Ian Holmes and Allessandro Rizzo take the view that this issue exposes flaws in the current peer review system:

  8. What a fascinating discussion!

    I think there’s a wider point, which is that pretty much all of the above discussion assumes that the journal editor is a thoroughly unbiased, decent, and ethical individual who wishes to solicit the most objective reviews and to make a decision about accepting the paper purely on its scientific merits, and not to have the decision tainted by using a reviewer who may be biased in favour of or against the manuscript.

    Hands up all those who think the above assumption is realistic. Anyone?

    I wonder how much difference it really makes in some cases who the editor chooses as a peer reviewer. I suspect that often it’s the case that if editors like the manuscript, they’ll simply ignore a negative review and publish anyway, and if they don’t like the manuscript, you can have all the glowing reviews that you like but the paper still won’t get published.

    On the whole, I’m minded to agree with the tweets in the comment above: perhaps the most useful thing about this discussion is that it exposes how flawed our current model of peer review is.

  9. Various Developments – 11/26/2012 « Homologus

  10. Accelerating Your Science with arXiv and Google Scholar « I wish you'd made me angry earlier

  11. Twitter Tips for Scientific Journals | I wish you'd made me angry earlier

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s