Goals of science vs Goals of scientists (& a love letter to PLOS One)

This monster post has been sitting on my computer hard-drive for a few months (seriously). For awhile, I was too scared to publish it. What I’ve written below is based on a (very) informal talk I gave at a graduate student seminar series at University of Maryland. To get the gist, the slides for that talk are below (all the way down) or here.

It’s also based on my stewing thoughts in response to dozens of conversations I’ve had about science and academia over the last year or so. My question is: does being a “good academic” and being a “good scientist” ever conflict? And if so, how often? And more importantly, can we fix academia (or science) to eliminate this conflict?

I’ll get to that in a moment, but I’m going to start with a related discussion of the journal PLOS One. If you are thinking, “Ugh, PLOS One? Bleh” then good, this essay is for you!

I like PLOS One. I’ve been a big fan of Public Library of Science (PLOS) since I first heard of them, and PLOS One is their cross-disciplinary, innovative, transformative (and sadly controversial) online journal. It focuses on rapid publication and article-level metrics, and it has sparked many similar journals.  Since PLOS One was launched in 2006, it has quickly become the largest journal in the world, with a respectable impact factor ((# 2014 citations) / (#2012+2103 papers)= 3.2). It appeals especially to younger scientists, who are publishing their first papers and haven’t fully adopted the norms of how traditional publishing works. As I’ve written previously, if you look at traditional science publishing with a beginner’s mind, it makes no sense. PLOS One intended to change that.

But scientists are still quite divided on this journal and the publishing approach it takes. In fact, many of us have been warned from publishing there or in similar open-access “mega-journals”. The internet is full of posts like this:

Why I published in PLoS ONE. And why I probably won’t again for awhile”. This one is largely about how an author was judged by others for publishing there. He writes,

Even though I personally like PLoS ONE and read a lot of ecology papers they publish, you won’t be seeing my name in there again any time soon. I’m just not brave enough.

 This article states:

…you’ve heard rumors that they’re not peer reviewed, or that they’re “peer-review lite” journals. You’re concerned they’re journals of last resort, article dumping grounds. You’re worried your co-authors will balk, that your work won’t be read, or that your CV will look bad…Well, you’re not the only one. And it’s true: although they’ve got great potential for science as a whole, megajournals… carry some potential career liabilities.

For some people, PLOS One even appears to evoke ill-concealed disgust. “It’s a trash-bin journal” say some. I’ve heard that “publishing a paper there is like throwing it in the trash”.  Much of this attitude comes from some unfair (albeit unsurprising) criticism from competing journals or publishing experts who too often miss the whole point. But I would argue that some of the negative backlash also comes from the fact that PLOS One violates some of the social norms of academia. And like most social norms, academics follow these with the utmost seriousness even when they serve no real scientific purpose. More specifically, I will argue that PLOS One has made itself more useful scientifically but less useful academically.

Explicit criticisms against PLOS One tend to take two basic forms:

  1. PLOS One publishes worse science than other journals.
  2. Publishing in PLOS One is a terrible idea because it does not give your work a good stamp of approval.

Let’s consider them separately.

Is PLOS One full of bad science? No. Well… maybe, but only in that, the world is full of bad science. This myth of PLOS One being particularly bad comes from PLOS One’s “objective” peer-review system (with lacks any subjective evaluation of importance). This means the paper submissions are judged solely on the validity of their methods and results, not on their predicted influence, importance, novelty or the ‘cool’ factor.

In fact, it seems virtually impossible to get a paper rejected from PLOS One: all you have to do is make sure your conclusions match your data and methods. You did a study and didn’t find anything interesting? PLOS One is fine with that. This means PLOS One has a very high acceptance rate of ~70%.

I’ve reviewed a number of PLOS One papers, some of them pretty bad, but all of them ultimately useful and I learned something. So if the authors have made inappropriate claims, you just tell them that they need to reanalyze or reinterpret their data. “You didn’t find what you were looking for. That’s about it. No you can’t say the groups are equal because you found no difference. Rewrite the results.” It’s rare that the data themselves are completely worthless. Bad science is not boring data; it’s making false claims. There is no evidence (that I’m aware of) that papers in PLOS One have a higher rate of false claims than other journals. Afterall, the whole incentive structure of the journal is designed to prevent that from happening. You are way more likely to make false or overconfident claims when the publisher requires that your findings are novel, revolutionary, and surprising. You are less likely to be academically dishonest writing for a journal that expects nothing in the way of novelty and impact.

Another interesting and paradoxical prediction is that PLOS One papers might even be more reliable and yet simultaneously less convincing than papers in other journals. Why? Because to get a paper in a high-impact journal you spend more time refining and strengthening the story and making it sound better. A PLOS One paper can highlight all the limitations and still be published with ease. This leads me to issue #2.

Is publishing in PLOS One terrible for your career because it does not give your work a good stamp of approval? Maybe. I don’t know. I hope not. This idea is perhaps best summarized in an interesting and thoughtful article by Anurag Agrawal. He advices young scientists:

Consider the risks. Because it typically takes some years for most articles to achieve citations, evaluators of academic CVs often use journal metrics as a proxy for quality or likely impact. Although nothing can replace reading and directly evaluating a study, removing the standards associated with selective journals introduces ambiguity to a publication record, especially for young scientists looking for jobs. In other words, when a hiring committee examines a junior scientist’s CV, a publication in a traditional journal carries with it the weight associated with the journal’s reputation for selectivity, rigor, novelty, and yes, likely impact. On the surface, a publication in an open-access journal only imparts ‘not scientifically flawed’.

He then concludes,

…we have not yet arrived at an alternative model of publishing that suits the primary goals of scientists.

My only complaint is that I would argue that he should change the phrase “primary goals of scientists” to “primary goals of academics” because the “goals” he is talking about are related to getting an academic job. If I was a non-academic scientist developing the cure for Ebola, I would not be worrying about the “ambiguity of my publishing record”. That’s an academic concern based on academic goals, and its a concern for all academics, from Professors of Civil Engineering to Historians of French Literature, regardless of how much science they do.

But practical and purely academic career advice for junior scientists is becoming more common and more necessary. If you ask successful younger scientists for advice they have a lot to say about how to survive in today’s highly competitive academic environment, which is some say the worst research funding environment in 50 years. According to a 2007 study by the US government, professional scientists spend about 40% of their time trying (and largely failing) to get grants. It’s tough right now.

But if you ask for advice from older professors closer to retirement (who came up during a very different era), they tend to give you a completely different kind of professional advice: scientific and philosophical suggestions about replication, experimental design, and what to be careful about when drawing conclusions. And the biologists talk more about the animals🙂. The prolific animal behavior experimentalist Jeff Galef told me,

Science is a marathon, not a sprint.”

He elaborated:

“One step at a time, experiment after experiment, frequently replicating your main effect, until you understand what you set out to understand and can be quite sure that, when others attempt to repeat your procedures, they will get the same results you did. And if not, you will know why not.”

I think that’s great advice. But then sadly he added that this is “not exactly the sort of approach likely to reap accolades today” and that “I doubt that I would fare particularly well in today’s academic environment.” And that is very sad in my opinion.

And it rings true. I have never met a young, successful scientist in my field who was not something of a good strategic careerist [I would like to use a softer, less judgmental synonym here, but I can’t think of one. So I’ll use this term assuming that it’s possible to use it in a respectful and affectionate manner, i.e. in a way in which I would also hope to apply it to myself.]

If we all think embracing PLOS One is tantamount to “career suicide”*, then perhaps we need to think less like good academics and more like good scientists. Because quite frankly, by thinking this way we are (ever-so-slightly) creating the problem. We need other methods for third-party validation; journal brands are just not designed well for that purpose. And that change has to come from the scientific community itself.

[*A reader pointed out this phrase was a bit of an exaggeration, which is true. Obviously, having a PLOS One paper is not bad for your career. But if that’s the only journal you publish in, you won’t be able to get a job. That’s what I meant here.]

A more basic problem people have with PLOS One (that no one wants to admit) is that we want to be able to judge a paper just by reading the citation. Here’s how it works:

Smith. 2015. Vampire bats have empathy. Nature. 123:23-24. ==> “Wow, sounds fascinating!”

Smith. 2015. Vampire bats have empathy. Journal of Peruvian Nature Studies. 123:23-27. ==> “Oh, I doubt that! Also, that’s boring.”

If PLOS One has its way and dominates the publishing world, it will be harder to think this way. And scientifically, that’s a good thing. [“Wait, what? You mean I have to read the paper?”]

In fact, I think PLOS One has marked the turning point to a world without journals, where all articles are online and evaluated on a completely individual level.

PLOS is trying to solve big problems, and they are doing it better than most. Many of us in academia complain about academic publishing, but how many of us are also participating and reinforcing the system that we criticize? We say we should publish all our negative results (even preliminary negative results), but who has time for that? We say we should cite and evaluate papers by their content, not by their covers, but we often fail to do that too. We say we should publish in open access journals, but we often don’t want to foot the bill (I certainly don’t want to). How many people are actively trying to fix academic publishing, rather than just complaining about it? (As you can see, I’m more of a complainer than a fixer myself). And how many of those people are actually fixing it successfully?

To fix things, we need to change large-scale incentive structures. That’s how you change behavior, not by asking everyone to voluntarily make sacrifices. PLOS is one of the leading institutions that is actually doing this and pushing science publishing forward in a good direction. They have helped start an important conversation, they sparked the open access revolution, and they changed our expectations of publishers. I support all of that 100%.

So that’s my love letter to PLOS and PLOS One.

Let’s back up though. What exactly is this huge problem I’m claiming that they are solving? And why does it exist. The real issue with publishing is part of a much larger elephant-in-the-room…

Academic incentives often hurt the quality of our science. We don’t want to talk about this because most of us scientists are also academics. Science might be our first love, but success in academia is what pays our bills.

In case you don’t know what I’m talking about yet, I’ll take a moment to explain my language. By “academics” I mean the process of successfully building one’s career in teaching and research at universities. And by “science” I mean the rigorous and systematic process of learning about the world with the highest reasonable standards of logic and evidence. A good scientist pursues falsifiability, repeatability, rigor, and measures of uncertainty. A good academic pursues scholarly impact and prestige. Most of us pursue both these things simultaneously. And ideally, they go together. For the most part, good science leads to impact and prestige. True that.

But does anyone doubt that academic and scientific incentives are also sometimes at odds? I don’t mean tradeoffs in time spent doing research and teaching, nor do I mean conflicts between humanities and the sciences. I mean that: scientific success is not the equivalent of academic success. Being falsifiable, repeatable, and correct is just not the same as being influential, recognized, promoted, and funded.

I would guess that most scientists have a little lab-coat scientist on one shoulder whispering in one ear (“Maybe you should take longer and replicate that result. Technically, you just inflated your alpha! Double-check those analyses. Maybe you should just re-run this experiment.”) and a little tweed-jacket academic on the other shoulder whispering in the other ear (“Publish this before you get scooped. I already know the best spin. You need this on your CV for the next grant application.”)

To put it bluntly, academia has some perverse incentives structures that we would never have intentionally built into science as a process. And we ignore this conflict at our own peril. Science is largely a set of intelligently designed incentives to keep people’s investigations as honest and rigorous as possible. In science, we have the double-blind experimental design, in which neither the observer nor the subject knows which treatments are experimental versus placebo. We have the standard of replication by independent labs. We have requirements to report statistical uncertainty, to report detailed methods, and to calculate the statistical power of our inferences. We have peer-review, which forces us to convince not only our fans, but also our competitors. We have a culture of open and constructive criticism–an environment where a young undergraduate can challenge and question the ideas of the head of the lab. We have a standard of rigor.

These are just some of the useful norms that are built into the structure and culture of science. Of course, not all science adheres to these best practices, but to the extent that something is “scientific” this is what we should mean. These scientific norms, from peer-review to p-values, are designed and revised by a global community of science with the collective purpose of aiding scientific goals.

The same is just not true for academic incentives. Consider for instance the various metrics that we use in hiring: the h-index, journal impact factor, and number of papers on one’s CV. These emerge from necessity in a world of limited time and money, with intense economic competition at every level of academic life. There is competition between publishing companies, between individuals, between labs, between departments, between universities, and between academia and other state-funded institutions. In this context, being a successful academic, like being a successful business, means having a successful brand that gives people confidence in what you say. Being a successful scientist, on the other hand, means not fooling yourself and not trusting your intuitions so you can follow the evidence wherever it leads. Not exactly two sides of the same coin.

For a clear example of how these academic incentives can erode science, consider how they can affect peer review. Several times as a reviewer, I have disagreed with another reviewer on whether a publication should be published even though we agree on the scientific validity and utility of the article. This typically happens because the other reviewer believes the paper is not “important” or “prestigious” enough. In at least two cases, the other reviewer explicitly agreed with me that the paper was correct and useful for their own research, but they then pointed out in private conversation that they had submitted a paper of similar “importance” and it was rejected– so why should this paper be accepted?

This way of thinking only makes sense if papers are viewed entirely as accolades in an academic competition. For working researchers, papers are sources of data and information. For academics, they are accolades. They cannot fulfill both purposes equally well.

In my opinion, other researchers are the most important audience of a paper. As a researcher, I want access to everything. I don’t want just the take-home message or the results in good-story format. Once a paper comes out, it can later be highlighted, summarized, and explained to all other audiences. But other researchers want the data fast, and they don’t need all the spin. Papers are how we communicate ideas and data, so they should all be published if they are scientifically valid and clear, regardless of their importance.

But let’s play devil’s advocate and take this to the extreme. Imagine that I do a “study” where I just measure the legs on 10 beetles of some particular species. That’s it. Just a list of 60 numbers. Surely, this is not meaningful enough to be published. Am I saying that this should be published? Actually, yes! Put it online. It takes no printed space. Maybe someone will find it useful, maybe not. But put it out there if you are not going to do anything else with it. People should share their data. They should share all their findings, their pilot findings, and their doubts. The more data and publications are out there, the better it is for every scientist. Can you imagine doing a search in the future for lengths of legs of beetle X and getting some raw data (with links to the methods and who collected them)?

The fear that we will be inundated to the point of paralysis with terribly boring data and “too much information” is unwarranted. That would have happened by now, and it hasn’t, because the more data and information we have, the more ways we develop for sorting it, filtering it, and parsing the best from the rest. Does a dataset on beetle leg length require peer-review? Probably not. That depends on what kind of conclusions we’re drawing. But why not just allow the data and findings to get published, and then allow further open peer-review over time– a continuous review period–so that a paper’s ‘impact’ is flexible. Basically, we need to make publishing easier and peer-review more rigorous.

One worry people have about open data and open access and open everything, is that the quality will get worse. But ask yourself, what kind of “quality” are we talking about? With more open access, the scientific quality (meaning the integrity of the data) can only get better, because there will be more transparency, and hence more and easier replication. What we have instead is a system that allows for, even encourages, sacrifices in real quality for perceived quality.  Ironically, the easier it is to hide caveats and doubts, the easier the ability of the story to influence others. Daniel Kahneman described a now well-known cognitive bias he called “what you see is all there is“– which explains why a shorter story with less detail can actually be more convincing (and hence higher impact) than a more nuanced detailed one.

So as scientific quality decreases, academic “quality” can increase. And that’s a problem.  Imagine for example that a scientific paper gets picked up by the New York Times. Say someone found a gene linked to psychopathy. At this point, the academic impact will skyrocket as more people are exposed to the story and it becomes famous. But the scientific quality doesn’t change and it may even be diluted as the message becomes exaggerated and distorted by the media over time. As we all know, the take home message can quickly go from “gene weakly linked to some psychopathic traits but more study needed” to “Eureka! Scientists find the serial killer gene!”.

This is not just a problem with how the public perceives science. The same thing happens on a much smaller scale when a scientist takes their results and packages them for Nature or Science, because they have to first sell the story to the journal editors in the same way that the NY Times has to sell a story to its readers. Now, to be clear: I am NOT trying to say that Nature or Science papers are all exaggerated or that they have bad science! In fact, as a academic, I’m always trying very hard to get a Nature or Science paper. It’s my dream to publish in these prestigious journals. But this is not just a bitter rant based on jealousy. My point here is that any desire to be radically and truly intellectually honest and skeptical about one’s findings has to be largely internal, because there are not enough structural incentives. So instead of talking about what’s the best scientific approach, we often talk about doing what’s necessary to convince the reviewers or “getting it past the reviewers”. That language gives insight into how we are often thinking like academics rather than scientists.

Let’s talk about unethical cheating in science. The main incentive for avoiding academic dishonesty is that if you get caught, your career is over (a reality which we hope that every scientist realizes and digests). But take my situation. How many other people are going to replicate my work on vampire bats? Hopefully, it will happen soon, but I’m not holding my breathe, because it took 30 years for me to come along and replicate Wilkinson’s original work on reciprocity. Too often, young researchers don’t realize that before doing the next-step experiment, you have to start by replicating the original finding. This is before we even get to the problem of an undergraduate science education that rewards students that conform their lab exercise results to the ‘correct’ answer. So should we be that surprised that serious scientific dishonesty is a growing problem? And that we have a replication crisis in various fields? Read this excellent story and wonder how often such dishonesty might go undetected. As the author writes,

Not only are most experiments not reproduced, most are probably not reproducible. This statement will shock only those who have never worked in a wet lab. Those who have will already suspect as much.

A few years ago, Glenn Begley put this suspicion to the test. As head of cancer research for pharmaceutical giant Amgen, he attempted to repeat 53 landmark experiments in that field, important work published in some of the world’s top science journals. To his horror, he and his team managed to confirm only six of them. That’s a meagre 11%. Researchers at Bayer set up a similar trial and were similarly depressed by the results. Out of 67 published studies into the therapeutic potential of various drugs (mostly for the treatment of cancer), they were able to reproduce less than a quarter.

As scientists, the incentive is, and should be, to make data and knowledge and ideas as open, accessible, and reproducible as possible. That’s the best way to ensure honesty in science. But it’s also just the best way to make normal everyday science faster and easier.

For that goal, journals themselves don’t really do anything useful. We, the scientists, do the writing, the reviewing, the editing, and even the tedious figure and text formatting. In this age of the internet, journals are merely brands that do not serve their original purpose of helping to disseminate information by printing lots of pages and mailing them out. Journal brands now serve a pivotal academic purpose, but they serve no scientific purpose whatsoever, because ideas in science should be evaluated based on their logic and evidence, not on their journal covers.

Obviously, we do need metrics of quality. Article-level metrics are important. We need to reduce papers to a single number (something like a rotten tomatoes score). The idea of a personal brand (an individual-level metric like an h-index) makes some sense too for hiring purposes, although there are other problems. We do need some kind of metric to compare scientists and their past accomplishments. But metrics of quality at the journal level (impact factors) are just needlessly uninformative. It’s been explained many times, so I won’t repeat it here. Impact factor are increasingly being influenced by gimmicks like publishing controversial papers that draw criticism or extending publication delay times.

There are also some old-fashioned journal requirements that no longer serve a real scientific purpose. One example is ink and paper publishing. My personal favorite journal is Proceedings B (sorry PLOS One). Every time an issue of Proc B comes out, I see something cool, interesting, and relevant to my interests. Plus all the articles are open access after the first year. But when I write a Proceedings B manuscript, I have to write two separate things. First, I have to write the article, which is limited to 6 manuscript pages. Then I write the online “supplement”, which contains all the details I can’t fit into those 6 pages. But the article is supposed to stand alone. And so I have to go back and forth moving text and revising to make it fit without the two being too dependent or too redundant. That’s how I spend (waste) much of the time when writing and revising. Because if you go over the 6 pages, you have to pay for each additional page. And paper is expensive. So is color ink. So yep, make a black-and-white version of every graph.

But wait a sec– why is there a paper version? Does anyone even read paper articles in paper journals?! In fact, why are there even articles that can only be found on paper. Can someone please just start uploading all those to the cloud?

When I write a PLOS One paper, there’s no page limit. And I don’t feel I have to “sell” the paper. I just say: Ok, here’s what I did and here’s what I found. Here’s why it’s interesting and here’s all the limitations. That’s it. And that’s what scientific writing should be. If it’s really brilliant, exciting or interesting, I should let someone else decide that. As the author, I’m probably in the worst possible position to know its real ‘importance’, especially so soon after I write it. Again, peer review is very important for that reason. We need more peer review and better peer review. That was one of the goals of PLOS One from the start. Why stop at two reviewers selected by some random editor who may or may not be my friend or my competitor. PLOS One encourages the notion that every paper is the start of a conversation, not the final word. When you publish there, you get an email stating:

“When your paper is published, you will be able to comment on the paper and respond to any reader comments. We hope that you and your co-authors will participate as widely as possible, as your contributions will be valuable to the community. “

That is great. All papers should be like that.

What we need is a central open-access repository of all curated data and peer-reviewed papers, with the peer-reviews and any revisions attached. Just let those peer reviews pile up (like movie or book reviews). In my imaginary future world, the role of journals would then be to pick through that pile of articles and repackage the best ones into magazine articles, which review a whole series of studies and that are aimed at larger audiences.

This hasn’t happened yet. There’s a perfect storm of converging interests for established journals to continue to act as the current method of third-party validation, which requires restricting publications to high-impact stories. Consider the interests of each shareholder:

From the publisher’s point of view, only the papers that are likely to be cited by many other people are worthy of publication, because only those papers will increase the journal’s brand. Most publisher’s are businesses, and their brand is all they really have. (No offense to Nature, but no one is submitting papers there because the font is nice or the editorials are so great. It’s because a paper in Nature is… a Nature paper.)

For academic authors, these elite brands are hugely important because the journal name acts as a third-party seal of approval. They become the building blocks of one’s personal brand, which will determine all aspect’s of one’s future career.

To understand the reviewer’s point of view, one must first see that most reviewers are authors that also publish in the journal they are reviewing for. So any article in a journal that lowers the journal’s brand value, will also lower the brand value of their own articles in that journal. That means that reviewer will want the articles to be at least as impactful as the article the reviewer has published there. The idea is: “Why should I allow articles in journal X that are not as good as my own article in journal X? They should be as good or better.”

So the collective interest of all three parties (reviewer, author, and publisher) is to keep the number of publications low, and the journal impact high. And to maintain high impact, you need to tell a good story with certainty and confidence. This leads to some irony. I have even seen reviewers (the supposed gatekeepers) who want the authors to simplify or sell the story better by removing inconvenient statistical results and analyses. This makes great academic sense (clearer story, more impact), but it makes no scientific sense (less information).

Science publishing is not about telling stories, it’s about describing a complex world riddled with uncertainty. The easy-to-read stories should come afterwards. The details need to be put out there first and foremost, maybe before the story even makes sense.

Academic incentives that lead science authors to sell their conclusions as confidently as possible cannot add to scientific goals; they can only detract. When articles are rejected on purely subjective measures of impact, they are usually sent to “lower” journals, where they will be eventually published anyway, just after a longer delay. Or they might end up as an unpublished manuscript buried on a computer’s hard drive. I can think of two manuscripts off the top of my head that never saw the light of day, because the authors could not find a ‘suitable’ journal. In one case, after redoing the statistics, I tried to convince the lead author into not making such dramatic claims and just say what the data showed most clearly. His response was to send it to two high-impact journals, and then to just forget about it and move on. In his rush to publish, he even included my name as a co-author when I had not approved (or even seen) the final version! (which is a gross contract violation but that’s another story). The bigger loss is the fact that these data were actually pretty interesting and they were never published. That’s a scientific loss, even if it’s a small one. And those losses add up over time. Whenever papers are rejected or discarded purely on the basis of predicted “low impact” the result is less information available.

Meta-analyses as a case study of problematic academic incentives 

I’ll give a more subtle example of how incentives might shape science in unintended ways. Many a high-impact paper in organismal biology is a meta-analysis looking at patterns across many species. Examples would be comparative studies of cooperative breeding in birds, mating calls in frogs, or social interactions in primates. I love meta-analyses. Aren’t we all most interested in the “big picture”? Yet such studies rely on hundreds or thousands of published data extracted from multi-year field studies of a single species, each of which was not itself “high-impact”, and might not have even been considered publishable by itself. These studies might involve a student sitting in the hot sun and watching a baboon scratch itself for hours (because that’s how those behavioral data on social behavior are collected). Then later someone comes along runs computer code and puts all that hard-won data to work*. That is all good.

The first tragedy is that meta-analyses are affected by the publication bias, because people publish positive more than negative results. Much has already been written on that. The second tragedy is that the original studies providing the data are often not themselves cited in the references of meta-analyses. Instead, they are often hidden in the “supplement” because the journal does not want to use the page space for listing all of them. What are cited instead are more theories and meta-analyses (which are again, higher impact). As a consequence, the hard fieldwork that allowed the meta-analysis to take place, is often not given academic credit or recognition. It does not even get counted in metrics like impact factor or h-index.

[*A reader pointed out that this wording is pretty unfair and dismissive to all the work involved in a meta-analysis, which is a valid point. A meta-analysis is a lot of work. And my point was not to say that analyzing data is always easier than collecting it. This also applies to my argument below about theory vs empirical work. You need both.]

Too much theory as a case study of poor incentive structures

A similar unintended bias exists towards biological theory as opposed to descriptive natural history work (and I’m talking here about studies in ecology, evolution, or behavior where the natural history really matters). It is virtually impossible to build a career in biology by making rigorous but simple natural history observations, regardless of just how much you learn about the natural world. It is much easier to make a career out of publishing risky, ambitious theoretical models, which may all turn out to be wrong in a few years. What’s the incentive for doing those hard (but crucial) field-based observations (sitting and watching baboons)? Any existing incentives for collecting those kind of data are dwindling fast. And that’s a problem, because biology is not physics. There are not a few general simple laws that explain everything (beyond evolution by natural selection). Biology is largely complexity. There’s no simple formula that explain how a cell works. It’s all messy complicated details down and down.

Building a house in science involves both constructing the scaffolding (the theory) and adding the bricks (natural observations). And that process is highly rewarded, yet making the actual bricks is not. Was this planned? Perhaps somewhat. But mostly, I don’t think so; I think it’s an unintended byproduct of academic incentive structures. For scientific progress, we need both scaffolding and bricks. Without theory scaffolding, we have no direction and no big picture. Without enough bricks, we are left with flimsy but popular theories that are supported by many citations but little evidence. (I got this scaffolding-brick metaphor from Bernard Crespi by the way). But academia favors impact, which favors theory over facts.

Academia without science: a world to avoid

It’s getting much better, but the ‘tall-tower-with no-foundational-bricks’ problem has long plagued the social sciences (think Freud). And it’s a problem that largely defines the main reason the humanities have been in decline.

Everything wrong with unchecked academia in one image.
Everything wrong with unchecked academia in one image. This is actually one of his famous quotes.

In a nutshell, there are a lot of very influential people in the humanities with lots of devoted followers and citations, but who have no actual sense, evidence or logic supporting their ideas (think Derrida). In these cases, the humanities can have all the problems of academia with none of the self-correcting practices of science. These influential academics are successful due to a positive feedback loop of being cited, then being cited because others cite you, and then being considered important because you are often cited, and so on. And nobody points out that the ’emperor wears no clothes’. (Well, until they do.) Of course, this ‘self-perpetuating echo chamber feedback loop’ is the sweet spot that all successful brands hope to hit. When you are riding on your own reputation, you don’t need to invest anymore in improving your product quality (think Pepsi).

Needless to say, there should be no place for this process in science. In science we have a different ethic. We do have to deal with imperfection, because science is an endeavor performed by flawed, biased, and irrational humans. But that doesn’t mean we can’t talk about it more openly and work harder to reduce these problems instead of ignoring them. As the writer Sam Harris put it so eloquently,

We need systems that are wiser than we are. We need institutions and cultural norms that make us better than we tend to be. It seems to me that the greatest challenge we now face is to build them. 

PLOS is an organization that is doing a lot of good. And because of them, there is an increasing number of people talking about these issues. The success of PLOS One and similar journals shows that many of us see the problems and want to fix them. We could move things forward by a adopting new sets of social norms, that would make PLOS One a typical journal rather than a strange one. For example, here are some norms I wish we were all incentivized to follow:

  • Cite papers based on their content. Stop citing based on the journal cover. One academic reason to cite high-impact papers is that it raises the impact of your own paper. We all do it. Ok, I’m not afraid to confess that facing a limited number of citations, I’ll cite a Nature paper over an Acta Chiropterologica paper, even if the latter one came before and made the exact same point. It’s wrong, but I do it. This practice creates a circular positive feedback loop where the predicted impact determines actual impact. The same methods and results published in a higher impact journal does not make it more reliable, significant, or interesting than if it were in PLOS One or Journal of Canadian Natural History.
  • Share. Share data. Replicate. Encourage others to replicate your work, rather than discouraging it. I share some of my data, but not as much as I should.
  • Treat papers as communications first and accolades second. Publish negative results.  Post all your own papers online and make them freely accessible to everyone.

The reason doing the right thing scientifically is difficult is that those who do it first will be punished and ostracized, at least until these actions become the new norm. It’s a coordination problem. The first dissenter who says “Hey, the emperor wears no clothes” suffers all the potential risk without enjoying any additional benefits. Yet we would all be better off if the collective norm was shifted over.

So ok, let’s get this academic culture moving forwards and evolving. Let’s question academic incentives and ask if they make the science better or worse. And when groups like PLOS are trying to fix things, let’s embrace that. Let’s think carefully before we advise others by saying, “but it’s better for your academic career if you do it the traditional way…”  

Most of the time you have to adapt yourself to the world, but sometimes it’s the world that needs to be changed. And I’m hopeful because science changes faster than any other world I know.

Here are slides from the informal discussion at University of Maryland this essay is based on:

(Feel free to leave comments below)

About Gerry Carter

I study the behavioral, sensory, and social ecology of vampire bats. http://socialbat.org.
This entry was posted in About science as an activity. Bookmark the permalink.

21 Responses to Goals of science vs Goals of scientists (& a love letter to PLOS One)

  1. You can find many related ideas (and more about how to apply psychology to solve the problems) here. : D


    Hope you find some value in it!😀

  2. Excellent read with many good points that I fully agree with (including the main point about the ideals of academia and science being at odds). There are, however, a few things that either rub me the wrong way, or where I have to disagree.

    The first is that I find the description of PLOS ONE as potential “career suicide” to be very much an exaggeration. Maybe it is a difference of fields, but my experience the people who are most negative about the journal, just think that a paper in PLOS ONE counts very little. I have certainly never run into someone who thought that a PLOS ONE paper would be somehow worse than no paper.

    Also, you seem to consider it a waste of your time to have to write a reasonably short paper and a separate supplementary document with all the details. I agree that this takes more time that just making everything one big manuscript; however, doing the latter would, in my opinion, be a waste of most of the readers’ time. (Pascal: “I have made this longer than usual because I have not had time to make it shorter.”)

    Finally, you describe metaanalyses as “… someone comes along runs computer code …”. This is not really a fair presentation of it. The someone who runs the computer code usually also wrote the computer code, which in turn requires that they had a scientific question. However, that is usually the smallest part of doing a metaanalysis. In my experience, more than 90% of the job is to collect the data and put it into a form that allows you to even begin to consider analyzing it. At the end of a metaanalysis, you usually end up with a list of experimentalists, who will be the first against the wall when the revolution comes😉

    But again, I fully agree with your main point of this piece.

    • Gerry Carter says:

      A few quick responses. Yes, I agree. I did not represent meta-analyses fairly. And in retrospect the term “career suicide” is an exaggeration. I will probably revise that text at some point. I’m not sure where this ‘career suicide’ quote came from, but someone said it to me, I think maybe referring to the idea of publishing *all* of one’s papers in PLOS One. What I do remember is a discussion of a faculty hiring committee where two rival candidates both had multiple high-impact papers, but one candidate had–in addition– a lot of small natural history notes (often with MSc or undergrad students), and someone in this committee (or so I’m told) thought this should score against them! I found this scandalous and backwards. I also know that some very good scientists will ask to not have their names on low-impact papers to which they contributed to avoid “diluting their CV”. This is I think a rational approach to the current academic incentive structure we have, because people are judgmental of CVs that have many low-impact papers, which is perceived as quantity over quality. Academically, it makes sense to wait and package your best results into high-impact papers. But the scientific value in that is more ambiguous.

      • I agree regarding the part about many papers in low-impact journals vs. waiting and bundling it into bigger stories. The latter may be better for your academic career, whereas communicating your results as soon as possible would be preferable from a scientific standpoint. Also, it is bit of a conflict between, e.g., Ph.D. students and their supervisors. For the students, getting more papers on their CVs sooner is preferable, while their supervisor may prefer to wait and make a bigger story. If seen many conflicts result from that.

      • Publishing everything in PLOS ONE would no doubt be problematic for your career. And that is not necessarily unfair: we all do a ton of work and most of it turns out to be solid work that unfortunately does not result in exciting new discoveries. However, if someone never makes any scientific findings that make it over the “subjective importance” bar, it is not entirely unfair to question their ability to ask interesting scientific questions.

      • Mike Taylor says:

        Publishing everything in PLOS ONE would no doubt be problematic for your career.

        For what it’s worth, my policy is to publish everything in PeerJ unless there is a specific, compelling reason not to. PeerJ is much like PLOS ONE, but much cheaper (you buy a lifetime membership once and you’re done), and currently much faster.

        But then I am in an unusual and privileged position: no-one pays me to do research, so I am not beholden to anyone else’s expectations of what I should be doing to climb a ladder. As a result, I am free to behave in a way that best advances my science, instead. (And how abject that are two separate things.)

        If someone never makes any scientific findings that make it over the “subjective importance” bar, it is not entirely unfair to question their ability to ask interesting scientific questions.

        You can’t tell what might have made it over that arbitrary and random bar, if I just go right ahead and send everything to PeerJ or PLOS ONE anyway — yes, anything, up to an including proof that sauropods could fly. If I were ever to make an earth-shattering discovery, I certainly would not give the benefit of that Science’n’Nature, but to one of the journals that are making the hard yards (i.e. PLOS ONE and their ilk).

  3. Great post! True, this: “I have never met a young, successful scientist in my field who was not something of a good strategic careerist” and I’m trying to get there myself, though I’m not so very young anymore.

    The major thing preventing many of us from publishing negative – and even positive, but not complete-story results – is time. Given that there is never enough, one must prioritize – invest time and other resources in those endeavors that will enable you not only to wrap up dissemination of everything you’ve produced to date, but also to pursue and develop your more fruitful discoveries. I consider the incomplete disclosure of all my results over my career as something like procrastination – I do what I think is most important, or most rewarding, and put off the rest for so long, it seems indefinite. Yes, it may help others, and it may be my civic scientific duty, but I’m as susceptible as the rest of putting myself first. When I become aware of others trying to do work in areas that might benefit from any unpublished work I have done, I do share in correspondance, or orally at a conference – a type of unacknowledged and not fully public publication, but a lot less time-consuming.

    • You have a very good point: writing up everything takes too much time. I agree it would be a good thing if we were all better at publishing all our results, including the negative ones. But I am also not convinced that doing so in peer-reviewed journals makes sense. I think that doing peer review of everyones negative or uninteresting results would be a huge waste of scientists’ time.

      • Dear Lars, could you please elaborate what you mean by “uninteresting results”? In my opinion, all results are important, irrespective if they are negative, positive or if it´s a null result. Most important is that the research is sound.

      • What I mean is results that do not meet the subjective criteria of higher impact journals for being sufficiently interesting or novel for the journal to consider publishing them. I lack a better word to describe this.

  4. This is fantastic work, Gerry.

    Minor niggle, the link to your slides at the bottom does not work. Maybe try to re-embed them ?

  5. Mike Hart says:

    Gerry, I came here via a Retraction Watch link in their Weekend Reads today (so late). I was a PLoS ONE subject editor for a year, and I quit because the process and organization of peer review seemed to be very poor compared to two other traditional peer review journals where I am or have been editor. I’m an evolutionary geneticist, but was regularly sent PLoS ONE manuscripts to handle that came from areas of social sciences. After some digging I learned that peer review is not handled by staff at Public Library of Science, instead it is contracted out to private companies staffed by non-scientists (who in many cases don’t seem to care much which subject editors they send manuscripts to). The experience left me deeply skeptical about the quality and reliability of the research published by the journal: if the editors who recruit reviewers are often handling papers far outside their expertise, then only such an expert could ever know whether the published papers are based on sound research. This seems to me to be a general weakness of post-publication peer review: it is hard for non-experts to know whether a particular published paper was ever evaluated for methodological and logical soundness. I acknowledge that such an evaluation is not guaranteed to come from traditional prepublication peer review either, but in traditional peer review one can assume that there has at least been an attempt by the editor to recruit expert reviewers who evaluated the qualities of the research and analysis. My experience at PLoS ONE led me to conclude that this may often not be the case at that journal. I would be interested in what others think of that problem.

    • jeffollerton says:

      Hi Mike – I’m currently a subject editor for PLoS One and wanted to reply to your comment about being sent manuscripts outside your subject area. If that was happening on the extreme level that you say, it suggests that the keywords you had selected in your profile were incorrect. Having said that, mistakes do happen, and I am also occasionally asked to handle manuscripts way outside my expertise. But no editor is obliged to take on any particular manuscript and I’d hope that editors were not dealing with those they did not have sufficient background to handle.

      Really interesting post Gerry, lots of food for thought there.

  6. AgroEcoDoc says:

    Reblogged this on AgroEcoPeople and commented:
    Fantastic insights–ones that seem to me to be underappreciated or even actively avoided by too many scientists (academics).

  7. Thanks for the opinion piece, Gerald.
    I must be naive, but I didn’t realize that publishing in PLoS1 was a career-ender. Surely a journal that sets the quality of the results (assuming that the peer review system isn’t broken) above novelty is one to target? I hate the novelty criterion- if it’s new, then it’s novel by definition. If it’s not new, then reject it. Simple. And, if I want novelty, I’ll go to a circus, or read a….novel.
    Then, I don’t actually care which journal something is published in, but, I am a government employed researcher with permanent tenure and don’t have to worry about impact factors (which, incidentally, are indices with unknown, or possibly no, relationship with impact reality).

    When publishing, I ask, “Who do I want to inform about this?” and “What is the most useful place for the prospective audience?”
    When reviewing, I ask, “Am I a correct person to do this review?”and, if yes, “Do the experimental design and methods allow the conclusions made?” Review effort should be the same for every journal, regardless of impact factor, or else the whole system is broken and we have no more credibility that shock jock journalists.

    Some of my most useful work is unlikely to get cited much because the main users are on-ground managers, rather than other scientists. For the target audience, it is more important that the information is correct than exciting because people’s livelihoods and biodiversity conservation depend on it being the best knowledge at the time, not whether it sets up an academic career. That is also important for the reputation of science and scientists. How can the lay person trust us otherwise?
    I do have students and colleagues who have or seek careers in academia, and it shocks me that they might think it is okay to selectively reference and omit critical papers because they get in the way of their story. My team counsels our students that it is okay for you to disagree with the mainstream thinking and famous names- how does science progress if we all blindly cite (often inappropriately and incorrectly) the references that the previous paper used?
    If you have a good story based on strong evidence, then there should be no problem in publishing in, say, PLoS1.

    Try Flying-foxes; they’re pretty cool too.

    Thanks again, I’ll post a link for my people.

  8. PhD Student says:

    Thank you for this article – it was a very interesting read.

    I have a few comments/additions to your thoughts:

    1) I think it is important to consider the drivers for the metrics of academic success. One of the reasons that research gets shortened to the most punchy story, and that metrics like h-indexes exist is the ‘businessification’ of science. That is, scientists are selling their wares to non-scientists. Often, it isn’t scientists that are doing the hiring (the first pass at least), or the promotions, it is administrators or managers or people looking to the bottom line. They don’t actually care about the science at all, only the prestige. It is those people we are actually trying to impress, not scientists at all! These non-scientists are also the ones that have the best ability to change this system.

    2) Metrics such as impacts factors and h-index are actually NOT a good measure of the impact of science, but rather a measure of how ‘in fashion’ that particular bit of science is. At a base, just getting funded often requires you to propose fashionable science, as funding for blue skies research dries up.

    3) In some areas of science, the data itself IS the publication, rather than the thoughts or theory or analysis of the data being the publishable bit. This is particularly true in areas where the collection of data is particularly arduous or expensive (for example space or the deep-sea), and is a huge barrier to data sharing. Unfortunately, ‘expensive’ data collection is often passed over for funding in favour of less expensive studies, exacerbating this problem.

  9. I’ve intervened in the OA debate a few times, and edit a journal that is completely free and OA (http://jpe.library.arizona.edu) for over a decade. I regard this as part of my job, no need to bring in money from publishing. The biggest point above in the post is about the need for change further up the academic greasy pole than I sit – promotion and tenure review committees, those who choose award winners, etc. Even hiring committees. I do sit on those and in all cases I strongly encourage people to read the work, not just look at the list of journals and books published. Candidates should not be penalised because a brilliant piece of work ended up in a journal with a low Web of Science ranking. Publishing has ethics – and as you know, some publishers are more ethical than others about copyrights, profits, etc. Everybody agreed some variety of OA is the way to go long-term (and no, not all journals charge authors) but we seem unable to wean ourselves off rankings and prestige indicators for what we write. . My further thoughts are here https://simonbatterbury.wordpress.com/2012/12/07/158/

  10. Girl says:

    It is very smart and accurate, I appreciate your work

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s