A Systematic Review of Systematic Reviews

Types of Goal

A huge amount of human knowledge is accumulated in books, documents, diagrams, videos, audio and people’s heads. When we want to find something out, we should start by looking through that material to see if someone already has an answer.

What we look through, and how we look, is shaped by our motives: why are we searching the body of knowledge? Perhaps we are looking for:

  • A ‘good enough’ solution  – for an engineering problem
  • Aggregated reliable research on an intervention, to assess confidence in it – for a medical treatment
  • Aggregated qualitative research on an intervention, to assess confidence in it – for social policy
  • What is known and what is not known about a topic – for scoping an academic study
  • An understanding of a particular situation – for intelligence to support decision makers
  • What happened around a particular event, or establishing patterns of life (a set of events) – for criminal investigations
  • Source material to summarise – for introducing a new topic to busy people.
  • Jumping off points (organisations/individuals, or terms, or physical places, or sites/journals) for more detail.

(If your goal is ‘for an assignment’ there may be suitable methods here, but in such cases you will need to conform to the guidelines, practices and styles that you have been given).

Note also that this particular analysis will concentrate on evidence-based or evidence-derived sources. Some communities are strongly founded in community concepts and discussion rather than objective research (see eg the relativist social sciences). While methods will be covered here that may be suitable for them, this guide is intended for those who assume the world is real, objective and shared.

Even so literature, video and expertise are not evidence. The gap between collecting evidence and presenting it can be large, particularly in academia where information assurance practices are typically missing. That is, there are no auditable paths from the presented conclusions back to the collected measurements, and so few ways to check fraud, mistakes or misrepresentation in the presented material.

Types of Search

For our full analysis of current review types please view our ‘Review Types Comparison‘ table. We are continuing to collect, collate and assess academic literature review methods, and will bring in other approaches in other domains.

Before deciding on a review method, a scoping review can be useful. Scoping finds out where suitable information lies in external forms, who the experts are, what investigation types would be useful and the limits to where we can look. It might also find the ‘how’ to look, ie the search terms that should be used to reach this scope.


Searches are largely limited to what has been recorded as text, video, audio or combinations. Much knowledge is held in people’s heads and methods should identify and interview experts, but the key to evidence-based assessments is a survey of what has been recorded, audited and replicated.

Even so, and allowing for limitations such as language, access to materials, missing skills (such as statistics), there is also sometimes far too much material to read let alone assimilate in the time available. This means that searches should be suitably focussed to reduce unnecessary reading, and capture representative material rather than the complete material.


Most searches are an exploratory cycle;  this can be expressed for example as the British Army intelligence cycle:

…note that this is not a workflow. As staff collect they will analyse, and that may redirect the collect to areas or with methods that are more fruitful or relevant. As this happens staff may need to revisit direction to either tighten or loosen as assumptions in the direction are challenged. In ideal practice (?) early drafts should be disseminated to expert audiences for them to feed back their expertise. And so on. All the same we can conceptually differentiate between:

Direct: setting goals for the search. What questions will we try and answer? what is its scope? The questions should be… questioned.
Collect: the methods used to search/find and initially sample the material. Driven by a Collection Plan.
Assess: Analyse to extract concepts from the material and synthesise to compare and contrast them into a coherent narrative
Disseminate: delivering in the right form to the right audience
Review: How well did the product disseminated satisfy the direction? What else needs to be found? What working practices need adapting? What updates to the situation need to be incorporated into a revised version?
There are a number of supporting activities, such as:
Managing Information
Software support to searching (interrogating several engines and logging results for audit)