Peer Review: A Floor, Not a Seal of Truth
Peer review is a quality check performed by two or three anonymous specialists on a manuscript, usually without access to raw data and essentially never by repeating the measurement. It catches incompetence, unclear reasoning and missing controls, which is valuable and limited. Confirmation comes later, from replication.
What Review Actually Checks
An editor receiving a manuscript sends it to two or three researchers working on similar problems. They read it and report on whether the question is worth asking, whether the method can answer it, whether the analysis follows from the data presented, whether the relevant prior work is cited, and whether the conclusions are supported by what is shown. They recommend acceptance, revision or rejection, and the editor decides.
This process is good at specific things. It catches a missing control, an unjustified assumption, an uncited prior result that changes the interpretation, a statistical treatment that does not fit the data structure, and a conclusion stated more strongly than the evidence allows. A competent referee report often improves a paper substantially, and much of the value is in the revision rather than the verdict.
What it does not do is check the work. Referees normally see the manuscript, its figures and whatever supplementary material the authors chose to provide. They do not visit the laboratory, inspect the apparatus, examine the raw data or re-run the analysis - and they are unpaid volunteers with their own research to do, typically giving a manuscript a few hours. Nothing in the process could detect a mis-calibrated instrument or a quietly excluded dataset.
This matters for how a published paper should be read. Publication establishes that two or three specialists found no obvious fault in the account given. It does not establish that the measurement is right, and the system was never designed to establish that. The confusion is encouraged by the phrase peer-reviewed being used as a synonym for verified, which it is not.
Where the Real Scrutiny Happens
In large experimental collaborations, the serious review happens before a journal ever sees the paper. A result from a major detector passes through an internal analysis review: a committee of collaboration members who did not do the work, who do have access to the raw data and the code, and who can and do demand additional cross-checks. This process routinely takes months and is often described by participants as considerably harder than the journal review that follows.
The incentive structure explains the difference. A collaboration of several hundred people puts every author's name on the paper, so a mistake damages everyone, and the institution has both the means and the motive to find it first. A journal referee carries no such exposure and has a fraction of the access.
Physics has also separated the conversation from the publication. The arXiv preprint server, open since 1991, means a result is usually public and being discussed by the relevant community well before it appears in a journal. In practice the community review of a physics preprint begins within days of posting, and formal publication often confirms a judgement that colleagues already reached.
This has a consequence worth stating plainly: in physics, the absence of journal publication does not by itself mean a result is unexamined, and its presence does not mean it has been confirmed. What matters is whether people with the expertise and the means to check it have done so, and said what they found. That can happen on a preprint, in a collaboration meeting, or in a later experiment, and it frequently happens in all three before a journal issue is printed.
Predatory Journals
A substantial industry exists to sell the appearance of peer review. Predatory journals charge a publication fee and accept essentially anything, often within days, with no meaningful refereeing. They exist because the phrase published in a peer-reviewed journal carries weight with audiences who will not examine the journal, and because that weight can be purchased cheaply.
The signs are consistent and easy to check. Acceptance within days of submission, when genuine review takes weeks to months. Unsolicited email invitations praising work the sender has clearly not read. A scope so broad that one journal covers physics, medicine and management. Invented metrics with names resembling the established ones. An editorial board listing researchers who, when asked, did not agree to serve. A fee structure prominent and a review process undescribed.
Checking is straightforward and takes a few minutes: whether the journal is indexed in the recognised databases, whether its publisher belongs to the established publishing-ethics bodies, whether the listed editors acknowledge the role, and whether anything else it has published is of a standard its authors would want to be associated with.
The same caution applies to conference proceedings sold as peer-reviewed, to vanity book chapters and to papers in journals whose names closely imitate well-known ones. The useful question is never whether something was published, but where, and what examination that particular venue actually applies.
Reading a Publication Claim
When a technical claim is supported by a citation, four things are worth establishing. Whether the journal is one the relevant specialists read. Whether the paper actually says what it is cited as saying, which is often not the case. Whether it is a primary measurement or a review that repeats someone else's. And whether anything has independently confirmed it since, which is the question that ultimately decides the matter.
The fourth is the one that counts, and it is also the one most often skipped. A single paper, however well reviewed and however reputable the journal, is one result from one apparatus and one analysis team. The failure modes that peer review cannot catch - a systematic error, a mis-calibration, a quietly made analysis choice - are precisely those that a replication by someone else does catch.
This is why the number and independence of confirming measurements is a better guide to how firmly something is known than any statement about where it was published. A result confirmed by three groups using different methods is in a different category from one published in a prestigious journal and never repeated, even if the second has the more impressive citation.
The standard applies symmetrically, and that is its merit. It does not matter whether a claim is conventional or unexpected, well funded or obscure: the question is what measurement exists, who else has reproduced it, and what uncertainty was stated. Peer review is where that conversation starts rather than where it concludes, and treating it as the conclusion gives a single journal issue authority that no single journal issue has.
Frequently asked questions
What does peer review actually check?
Two or three specialists read the manuscript and report on whether the question is worth asking, whether the method can answer it, whether the analysis follows from the data presented, whether prior work is cited and whether the conclusions are supported. They decide nothing about whether the measurement is correct.
Do referees look at the raw data?
Rarely. They normally see the manuscript, its figures and whatever supplementary material the authors chose to supply. They do not visit the laboratory, inspect the apparatus or re-run the analysis, and they are unpaid volunteers typically giving a paper a few hours. Nothing in the process could detect a mis-calibrated instrument.
Is a preprint less reliable than a published paper?
Not necessarily, and in physics the distinction is weaker than elsewhere. Since the arXiv opened in 1991, results are public and discussed by the relevant community within days of posting, often months before journal publication. What matters is whether people able to check it have done so and said what they found.
How do I spot a predatory journal?
Acceptance within days, when real review takes weeks to months. Unsolicited flattering invitations. A scope covering physics, medicine and management at once. Invented metrics imitating established ones. An editorial board listing people who never agreed to serve. A prominent fee and an undescribed review process.
If peer review does not confirm a result, what does?
Independent replication. The failures peer review cannot catch - a systematic error, a mis-calibration, an undeclared analysis choice - are exactly what a different group with a different apparatus does catch. The number and independence of confirming measurements is a better guide than where something was published.
