How to evaluate sources
Methods for evaluating whether a source is likely to be reliable usually look at authority, motivation, accuracy and relevance.
Information will be true or false regardless of who says it. But if there isn’t an easy way of verifying something (for example, by fact-checking), it can be helpful instead to evaluate the source and how it presents the information.
This article looks at checklists for evaluating sources and questions to ask about a source when getting information from it. It is part of the Commons Library’s good information toolkit, which reviews the strategies and criteria for spotting misinformation and working out what information can be relied upon.
Checklists for evaluating sourcesSeveral groups of researchers and librarians have devised checklists to follow when evaluating sources. Most of them use roughly the same criteria, which are summarised below:
- Accuracy – does it faithfully report the original source?
- Authority – do the authors have relevant expertise or experience?
- Relevance – is the information up to date, and does it directly relate to the matter at hand?
- Transparency – do the authors cite sources and explain the methodology for finding or producing the information?
- Purpose – why are the authors publishing it? Do they tell the full story or are they partial?
Many checklists, particularly the CCOW test, stress the importance of using ‘lateral reading’ to evaluate sources. This means leaving the source being evaluated (often a webpage) to look for information about it elsewhere, such as searching the internet for the names of the authors
The table below shows the criteria used by some common checklists for evaluating sources.
Summary of checklists for evaluating sources
Currency
What
Scope
Rationale
Credentials
Relevance
Who
Motivation
Authority
Claims
Authority
Why
Authority
Date
Objectives
Accuracy
When
Accuracy
Accuracy
Worldview
Purpose
Where
Relevancy
Relevance
How
Timely
The sections below look at three of these checklists in more detail.
CRAAPThe Information Literacy Group, part of the Chartered Institute for Librarians and Information Professionals, advises readers to use the CRAAP test (PDF) when evaluating information. It was created in 2004 by Sarah Blakeslee of the Meriam Library at the California State University Chico:
Currency When was it published? Is the information too old? Does it have a date on it? When was it last updated? How important is it that you have up-to-date information?
Relevancy Does it fit your project? Will your project be stronger if you include this information?
Authority Who has published or written the information? Do you trust them? Is it easy to find out anything about them? Who was it written for?
Accuracy Is the information correct? Check with another source, if you are not sure to see if they say the same thing. Are the details correct?
Purpose Why does the information exist? Is it trying to sell you something, persuade you or give you an opinion? Once you figure this out, you can then decide how to use the information that you have found.
The CRAAP test was adapted from existing checklist criteria used by librarians to evaluate print materials, and it has been criticised for not focusing enough on internet skills, such as lateral reading.
The five WsThe five Ws are: who, what, when, where, and why (and how). They are recommended for writing in journalism, but they are also useful for checking sources.
Similarly to the CRAAP list, the five Ws ask about the currency and relevance of the information (when and what), identity of the authors and their credentials (who), the correspondence with other sources (where), and the source’s intentions (why). It also asks how the source came by its information (how), encouraging readers to examine the methodology for gathering its information and reaching its conclusions.
CCOWCCOW was first proposed in 2022 by Anthony Tardiff of Gonzaga University as an alternative teaching tool to CRAAP. The checklist is designed to encourage readers to do more lateral reading on the internet and to recognise their own biases when approaching sources.
The ‘credential’ criterion in CCOW is similar to the ‘authority’ criterion in CRAAP, ‘claims’ in CCOW is similar to ‘accuracy’ in CRAAP and ‘objectives’ in CCOW is similar to ‘purpose’ in CRAAP.
However, CCOW explicitly includes a criterion for ‘worldview’, which encourages readers to consider their own outlook and biases as well as those of the source and its intended audience. The CCOW test asks readers to examine their reactions to a source and consider whether they are justified. For example, if a source makes the reader angry, is it because its logic is wrong and it doesn’t cite its sources, or is it logical and well-sourced but doesn’t align with the reader’s worldview?
CCOW takes the position that “we all have a worldview”, and that worldview is more complicated than the idea that biased information is bad and unbiased information is good. It says readers should understand that different sources will look at the same information “through a different lens”, and that considering a source’s worldview helps readers evaluate its perspective.
How can I tell whether a source is reliable?The following questions can help us judge whether a source is reliable, based on the principles of authority, credibility and transparency identified in the above checklists.
Do they say who they are and what they want?If organisations and authors are transparent about who they are and what they want, it might be easier to judge whether their information is reliable. It is useful to know:
- their aims
- how they are funded
- their qualifications and relevant experience
Looking at a source’s aims and funding can help readers identify potential biases and conflicts of interest that might affect how they report or produce information. For example, a source might selectively publish information favourable to their aims or the aims of their funders.
QualificationsLooking at a source’s qualifications and experience can help readers judge whether their information is authoritative for a particular issue. For example, if an organisation has a history of producing statistics and it employs qualified statisticians, it might be more likely to have produced reliable statistics.
Concealing informationIf a source does not state its aims, qualifications or funding, it is harder to put their information in context, which could lead readers to draw unreliable conclusions. If a source conceals information about itself, it might suggest that publishing that information would undermine its message.
Do they explain their methods and cite their sources?If sources explain the methods and sources they used to produce their information, it’s easier to understand what can and can’t be concluded from it.
Methods for producing informationReaders might not always have the expertise to evaluate research methods themselves. However, if a source publishes and explains its methods, it shows confidence that the methods support its conclusions and that it is willing to have other independent experts scrutinise them. This includes explaining how much uncertainty there is in any measurements or conclusions.
In social sciences and economics, which are governed by complex human behaviour that is difficult to measure, uncertainty might be greater and harder to evaluate than in physical sciences like chemistry. Reliable sources in these fields are expected to inform readers how they have evaluated the uncertainty in their research; for example, by including confidence intervals and exploring alternative reasons for an observation.
Without knowing how uncertain a measurement is, it is impossible to know what to conclude from it.
Peer reviewMost academic publishers subject their articles to peer review, where independent experts scrutinise the methods and conclusions before publication. An academic article that hasn’t invited experts to review, or one with lower standards for peer review, might be less reliable than one which has been rigorously reviewed.
Publishers’ websites will usually include their policy on peer review for readers to check. For example, at the medical journal the Lancet, research articles are usually peer reviewed by three clinical or subject specialists and one statistician. Some journals also publish the results of peer review and the identity of reviewers, although there are arguments about whether this is better than anonymous and closed peer review.
Guidelines from the Committee on Publication Ethics (COPE), a non-profit organisation promoting ethics in academic publishing, say that reviewers should be fair, honest, unbiased and transparent with publishers about their qualifications, expertise and any conflicts of interest.
However, peer review is not a guarantee that the information is reliable; the website Retraction Watch monitors academic articles that have been retracted by their publisher for a variety of reasons, such as problems with the results of a paper that were not picked up by peer reviewers. Because peer review relies on good faith, it can be difficult to know whether a peer reviewer was qualified or gave their full attention to an article. More peer reviewers are reportedly using artificial intelligence in their reviews, leading to some concerns about the integrity of peer review.
The website PubPeer gives a forum for (usually anonymous) users to comment on the methods and results of academic articles after publication. These have sometimes led to publishers retracting articles after users identified significant problems with them.
Do they make sense, or are they incoherent?If a source makes points incoherently, contradicts itself, or otherwise presents its information in a confusing way, this can indicate that it is unreliable.
Incoherence also suggests that the authors might not have dedicated time to thinking through the information thoroughly, that it might contain errors, or that they are trying to fit pre-conceived conclusions to the information, rather than drawing conclusions from the information.
There is also a risk that the information was generated by artificial intelligence (AI), rather than by research or investigation.
Regardless of the reason for incoherence, it suggests that the source does not consider it a reputational risk to publish incoherent information.
What happens to them if their information is wrong?If a source expects there to be negative consequences for publishing false or misleading information, it might be more likely to put more effort into ensuring its information is reliable.
Reputational damageIf a source is found to have published false or misleading information, it might damage its reputation as a provider of information.
For example, after the Hillsborough disaster in 1989, the Sun newspaper published allegations about actions of Liverpool FC fans under the headline “The Truth”; these allegations were later found to be untrue. Immediately after the publication, people in Merseyside began boycotting the newspaper. The journalist Chris Horrie estimated in 2014 that the headline “was costing News International [owner of the Sun] about £15 million a month” in 1989.
Legal riskA source might also run the risk of legal action for defamation if it published information about someone that was untrue and that harmed their reputation.
In England and Wales, defences against a defamation claim include that the published statement was substantially true, that it was an honest opinion, or that it was in the public interest to publish it.
Sources with a high public profile might be more at risk of defamation action because in many jurisdictions (including England, Wales and Scotland, though not Northern Ireland) a statement cannot be defamatory unless it causes or is likely to cause “serious harm” to the claimant. Case law suggests that publications with wide reach and influence are more likely to cause serious harm than publications that are not widely read.
Sources with low risk of consequencesSources might not face such negative consequences of publishing false or misleading information if they have lower public profiles or business models that don’t depend on being seen as reliable. For example, someone publishing information anonymously on social media is less likely to face consequences if their information is false.
Corrections policySources that are concerned about their reputation might have a policy on publishing corrections and clarifications. An organisation that clearly and willingly communicates corrections might be more reliable than one that doesn’t issue corrections or only does so when forced to.
The code of practice for news editors published by the UK’s independent press regulator, IPSO, requires that a “significant inaccuracy, misleading statement or distortion must be corrected, promptly and with due prominence”. Some publishers that are not members of IPSO, such as the Financial Times and the Guardian, have the same or similar clauses in their editorial policies.