Increased openness is a welcome trend in academia and in its slipstream we see numerous researchers sharing their early findings no longer only via scientific conferences but also via preprints. Preprints are (academic) manuscripts that have not yet been peer-reviewed or published in a traditional publishing venue. Most academics who post preprints do so to receive early feedback and community review, to speed up dissemination, and to make their work available in open access.

Some disciplines such as physics and astronomy already enjoy a long tradition of publishing preprints, that be considered a de facto standard working procedure; think for instance of arXiv that was founded as early as 1991 and is now hosted by Cornell with financial support from many universities worldwide including Leiden. However, displaying early results is also interesting for researchers in other subject areas, that have so far considered early openness to be not such an obvious choice. In this blogpost we focus on early openness in two ways: the posting of preprints on the one hand, and on the other hand the preregistration of research including a prepublication. How to go about making your research or research plans publicly available at an early stage, and what can you expect to happen after you have done so?

Alex Brandsen, Bjørn Bartholdy (both Archaeology), Eiko Fried (Psychology) and Hugo de Vos (FGGA) each decided to share their work at an early stage as a preprint for their own specific reasons. We asked them what made them decide to publish their research as a preprint rather than to wait for official publication and of course what the outcome has been.

Alex’ research is mainly computer science applied to archaeological data, and in the computer science domain it is much more common to share early findings than in archaeology. His computer science supervisor suggested preprints as a possibility to get research quickly available to the scientific community for citation purposes. For the other three, preprints may have been a less obvious route, but nonetheless, their argumentation is comparable.

4 clear advantages for early sharing emphasized by their experiences are:

1. Speedy dissemination

2. Community feedback

3. Public availability

4. Sustainable findability and accessibility

Speedy dissemination:

Speedy dissemination of results that can be presented or cited is the first step in the preprint publication process. It is an important factor mentioned by Alex and Bjørn who publish in journals with long review times while they seek to defend their PhD theses in time.

The advantage of quick online availability also applies to ‘self-citation’: for Alex and Bjørn the availability of the preprint gave them the opportunity to refer properly to the work they had already done at a conference or in a follow-up article, even though it had not yet been officially published.

Not only did some journals already take a long time to guide articles from submission through peer review towards publication, the current Covid pandemic has only added to that problem: research has been significantly slowed down, peer reviewers are even harder to find, and workflows are delayed. Hard deadlines such as the date of a thesis defense or the end of a project and its funding, turn the use of a preprint to finish up into an attractive alternative, when the final publication is due to come at a later stage. Bjørn elaborates: "COVID caused severe delays. We are now discussing the possibility of including preprints in my dissertation, which will significantly reduce delays associated with publication in a journal."

Alex also used preprint in order to speed up his PhD project: “As my PhD dissertation is a collection of papers, due to university rules at least 4 need to be published or accepted for publication, for the dissertation to be defended. This adds some time pressure to get papers submitted as soon as possible. All my papers build upon my earlier research, so I needed to be able to cite previous papers, but due to slow reviewing and publishing of academic journals, the previous paper generally wasn't published while I was submitting the next one. To still be able to cite my work, I opted to use preprints, specifically on arXiv. This made sure that reviewers of the newest paper were always able to get a background of the previous research, without having to repeat it every time in the introduction.”

Eiko adds the importance of speed in terms of being citable as early as possible, because “citations still matter especially for early career researchers, and it is a nice benefit of preprints that they can be cited immediately, rather than having to wait a year or two before the paper eventually appears online”. On the other hand he also warns for the downsides of speed: loss of academic thoroughness. Traditional peer review is by no means a panacea, but he has also seen plenty of cases of very problematic science and false claims that have been picked up by news but would likely not have made it through peer review.

Arxiv accordingly poses a specific caveat in a statement on its homepage: “e-prints posted on arXiv are not peer-reviewed by arXiv; they should not be relied upon without context to guide clinical practice or health-related behavior and should not be reported in news media as established information without consulting multiple experts in the field”.

Community feedback

Community feedback is the most important factor from the perspective of enhancing scientific progress. Eiko and Alex both stress its great advantage of publishing preprints. Traditional peer review takes a long time, and the outcome of the review process does not always provide a straight indication of the quality of a manuscript: a paper may be good, but simply not fit exactly into the scope of a journal, fall outside the professional interests and expertise of the reviewer, or even receive two different review reports, one enthusiastic and one that rejects the manuscript.

While we know that there are researchers, who feel uncomfortable and vulnerable to expose their work to a large audience before a final acceptance from a journal and fear their ideas might be ‘stolen’, these two find the feedback they received from their colleagues more valuable and actually profited from their early openness, and the useful comments that they received and used further to improve their work. Not to forget that the community of preprint readers also includes journal editors on the lookout for new ideas, which turned out nicely for Eiko: “together with a larger team of colleagues, we conducted a systematic review over the last years. The first 2 journals we submitted the paper to were not interested, which was when we decided to put the paper online as a preprint. Just one day later an editor from a prestigious journal reached out to us, and recommended we submit the paper to their journal.

Alex shares the experience of his work not fitting into a journal's scope: "Another reason I used arXiv was a paper that was rejected from multiple journals due to it being very specific. This paper described an evaluation of a developed system, which was so specific it wasn't applicable to any broader research, and as such journals weren't keen on publishing it. Instead, I published it as a report on arXiv, to at least allow people to read and cite it, outside of the context of my dissertation."

Public availability

The argument that ‘Research funded with public money should also be available to the public’ is one of the main drivers behind the Open Access efforts at the Centre for Digital Scholarship. For Eiko it was also a an important argument: “I work in mental health research, and it’s a disgrace that e.g. people with mental health issues who pay taxes that fund our research cannot access most resulting work published in academic journals”. As other options to publish Open Access are also available to our researchers, a preprint does not need to be the first choice to open up, but its function as such is recognized also by Bjørn: “I only publish in journals where the University has an open access agreement, so Open Science wasn't the main motivation, but I am interested in trying out all aspects of Open Science, and publishing a preprint was something I wanted to try.”

Sustainable findability and accessibility

Hugo reasoned that he should use a preprint to enhance the findability and the accessibility of his work on the long term. ArXiv is properly indexed by internet search engines such as Google and assigning a DOI facilitates a stable link that can also be used to cite his work. As of long much focus in academic publishing lay on articles in journals, publishers more than ever before seek for sustainable archiving solutions and have persistent identifiers assigned to the articles, but this is not yet the case for other publication types, such as workshop papers and conference proceedings.

The volatility of internet based publications, resulting in broken links and error messages, has a negative effect on the possibility to reuse previously created work and the digital longevity of both publication and the link to it, deserve careful attention.

Janna Marie's blogpost triggered us to ask her about her motivation and subsequent experiences in this upcoming way to seek for open practice. If anything it defied the notion that preregistration is an easy way to get ‘stuff’ out in the open: it requires diligent care to define procedures so that all parties involved can work accordingly and the final publication can go ahead independently from the final outcome of the study.

The length of the peer review process that we saw as a motivation for publishing preprints, was also an issue of concern for Janna Marie. Due to the large number of collaborators on her project, it proved difficult for the JAACAP-editors to find reviewers who were not ‘connected’ to the research team. However, a ‘Registered Report’-approach requires that the analyses cannot start before the journal gives a ‘go’ decision. From the perspective of project planning, the time required waiting for the journal’s response is a topic that needs to consider with care.

Janna Marie Bas-Hoogendam’s story

“My first experience with pre-registration was during my PhD: together with a multidisciplinary team of researchers, we performed a family study on social anxiety disorder and we decided to pre-register our study design and hypotheses on the website of the Open Science Framework (osf.io). This website works with a ‘timestamp’ and enabled us to demonstrate that we defined our hypotheses and research-approach before the actual analyses were performed. We chose to do so in order to be transparent in our scientific work and to avoid p-hacking or fishing for results. In all publications from the project, we could refer to the pre-registered hypotheses, and I have the impression this approach was highly valued by the editors and reviewers.

In my present work, a mega-analysis on the neurobiological substrates of childhood inhibited temperament (work supported by a Rubicon grant from NWO), I wanted to take the next step: I chose to submit a ‘Registered Report’ to the Journal of the American Academy of Child and Adolescent Psychiatry (JAACAP). As described in my recent blog on this topic, this Registered Report consists of the Introduction and Methods section of a ‘normal’ paper, as well as a brief Synopsis on the project. Essentially, these parts are submitted and peer-reviewed before the actual analyses take place. Based on the reviewers’ comments, the journal can give an ‘acceptance in principle’, which guarantees that the journal will accept the full paper (‘stage 2 submission’) regardless of the outcomes of the analyses.”

With still a lot of work to deal with in the last stages of getting her prepublication finished, we asked Janna Marie about the advantages: what’s in there for her.

“First of all, I want my scientific work to be impactful and worthwhile, and preregistration of methods and hypotheses is increasingly considered part of good scientific ‘hypothetical-deductive’ research. Furthermore, this approach could reduce the publication bias; that is, it minimizes the chances that the significance of the results will determine whether a paper is accepted (or rejected) for publication. Next, this method increases the value of peer review, as suggestions of the reviewers on the proposed analyses can still be implemented in the analyses.

Another advantage, that applies specifically to my current project, is the following: working on the Introduction and Methods of the paper at an early stage was a great way to involve all collaborators (there are more than 50 international co-authors on this work!) in the project and to think as a team about the possibilities for analyses before actually performing them. I also expect that this approach will help us stay focused when we are in the analysis-stage of the project: when the Registered Report is accepted, we will perform the analyses as specified in the preregistration.

Last but not least, working on this Registered Report enabled me to make good progress with the project while data-collection (i.e., composing the dataset for mega-analysis, which was in some cases complicated due to limited access to data in COVID times), was ongoing.”

In this blog we explored two different ways of opening up research at an early stage: preprints and pre-registration. We think they both offer a valuable contribution to opening up the scientific process, as they each provide their own window to the steps that lead up to final publication of results. Of course we also aimed to give some inspiration to explore these ways to practice open science.

Last but not least, we hope that the practical guide on preprints will prove a useful tool to help meet the increasing demand for scientific transparency, both from within the academic community as well as from the public.

If you have questions please contact us at openaccess@library.leidenuniv.nl.