Critiquing Research

23/07/07

 

An introduction to critiquing research papers, with resources for further study.

 

This document includes text links to other parts of this document, and to other on-line resources. If your computer is connected to the internet you should be able to view those on-line resources by clicking on the links. All underlined words are links that you can click.

 

Any problems or questions about this document? Then please e-mail me

 

Visit my web site for other research-related links: www.richard-ingram.co.uk

 

Contents:

·       What & Why? ·      Critical Reading Stages
·       When? ·       Glossary
·       How? ·       References
·       Where? ·       Links & resources
·       What Next? ·       Spreadsheet

 

What and why?

 

So that you can decide to what extent research may be useful in practice you will need to be able to read research critically, to see if the findings are trustworthy, and be able to compare it with other related research.

 

The activity of “critiquing” research is therefore fundamental to research utilisation and evidence based practice.

 

To critique research, you first need to have a reasonable knowledge and understanding of the research process and research methods, so that the research paper in question can be interpreted and understood. You then need to ask appropriate critical questions about the way in which the research was conducted. No research will be perfect, and it is usually a compromise between the ideal, and what is actually practical, but the way it has been carried out will greatly influence its validity, reliability, the extent to which it can be applied to other populations, and ultimately, its trustworthiness, and usefulness for practice.

 

top

When?

 

Whenever you read research you need to do so with a critical view – it is just not safe to assume that because research is published it is therefore safe. When writing academic essays it is not good enough merely to “cut and paste” research findings “as is”, as if they are self-evidently trustworthy. Critiquing therefore should occur at some level whenever reading research. This could be fairly informally, whenever reading research; more deliberately and formally when synthesising reading and evidence for essays and dissertations; and in considerable detail when completing assignments that require critique of a particular research paper, or a small number of them. At the informal level, the critiquing process will be largely a mental activity, which will become better able to do the more practice you have, whereas for essays and literature reviews you will need to make notes relating to relevant aspects of the research process so you can come to an informed judgement that you can explain and justify in writing.

 

top

 

How?

 

LoBiondo-Wood et al (2002) suggest that there are 4 stages or levels of understanding in critically reading research, I paraphrase these here (click the link).

 

To assist with critiquing, there are a variety of “Critiquing Frameworks” and “Critiquing Criteria” in the literature. These are sets of critiquing questions that deal with the important aspects of the research process. The critical reader can use these questions as a prompt to aid the process of critiquing research. Not all questions they contain will be answerable or appropriate, so these frameworks are a tool to assist critiquing rather than a rigid prescription for the critical reader.

 

There are many frameworks available in the literature; there is no single “correct” or “best” one. Some are very detailed whilst others are brief and succinct. Some ask quite complex questions and may therefore need greater understanding to apply, whilst others are simpler. Some will be suitable for both qualitative and quantitative research, whilst others may not be without some adaptation. The reader should select the framework that best reflects their needs and their level of knowledge and understanding. Sometimes only parts of the framework are applicable, and sometimes it will be a good idea to incorporate the most relevant parts of a few frameworks, so that it offers the necessary coverage. The choice is up to the critical reader; the only rule is that there should be some rationale for the choice, rather than just convenience.

 

One word of caution: some of the simpler critiquing frameworks focus too much on the written presentation of research papers. This can lead to a “textual critique” (a critique of the way it is written and presented, which is often a reflection of the particular journal’s style) rather than a critique of the methods used (i.e. what the researcher actually did). Only the latter can help the reader to decide on the worth of the findings.

 

top

 

Where?

 

Critiquing frameworks appear in many research textbooks, journal articles and on the web. Often they will be called something else (such as “guidelines for the critical evaluation of research”), but they amount to the same thing. A list of some examples and where to find them is attached.

 

top

 

What next?

 

Finally, when some judgement about the particular research is made, it then needs to be related to the other literature on the topic, and the body of knowledge that exists. The reader must ask: how do these findings add to, support or conflict with other sources of evidence?

top

References

 

LoBiondo-Wood, G., Haber, J. & Krainovich-Miller, B. (2002). Critical Reading Strategies: Overview of the Research Process. Chapter 2 In LoBiondo-Wood, G. & Haber, J. (editors). Nursing research: Methods, critical appraisal, and utilization. (5th Edition). St Louis: Mosby.

 

Polit, D.F., Beck, C.T., & Hungler, B.P. (2001).  Essentials of nursing research methods, appraisal, and utilization. (5th Edition). Philadelphia: Lippincott

 

(other books on research)

 

top


Stages in Critical Reading of Research Articles

 

Stage

Purpose

Activities or critical questions:

Preliminary Understanding

(skimming)

Skimming or quickly reading to gain familiarity with the content and layout of the paper

·      Highlight or underline main steps in the research process

·      Make notes (comments & questions)

·      Note down key variables

·      Highlight new or unfamiliar terms & significant sentences

·      Look up unfamiliar terms and write in definitions

Comprehensive Understanding

Increasing understanding of concepts and research terms

·      Review all unfamiliar terms before 2nd reading

·      Clarify any additional terms

·      Read additional sources as necessary

·      Identify how the main concepts relate to each other and the context of the study

·      Write brief summary of the main idea or themes of the article in your own words

·      Identify any further questions or areas that need further clarification

 

Analysis Understanding

(breaking into parts)

Break the study into parts; understand each aspect of the study. Relate to steps in the research process

 

At this point you can start to critique the study using a critiquing framework or criteria, applying them to each step in the research process

·      What is the purpose of this article?

·      Am I clear about the specific design used, so I can apply appropriate critiquing criteria?

·      How are the major parts of the article related to the research process?

·      How was the study carried out? Can I explain it step by step?

·      What are the researchers’ main conclusions?

·      Can I say I understand the parts of the article and summarise them in my own words?

Synthesis Understanding

Pulling the above steps together to make a (new) whole, making sense of it and explaining relationships.

·      Review notes on how each step compared with the critiquing criteria

·      Briefly summarise the study in your own words, identifying the main components, and the overall strengths and weaknesses

·      This is a critical commentary on the study rather than a description or précis of it

 

top

(Adapted from LoBiondo-Wood et al, 2002)

 

 

Using the process above in conjunction with critiquing criteria or frameworks can help you critique individual articles.

 

If you are reviewing the literature, you will need to assemble and synthesise the evidence from all the studies you have read. Using a spreadsheet can help you organise all the studies in a way that helps you see your critiquing notes simultaneously, allowing you to compare and contrast, and pull findings and conclusions together, so you have a sense of the body of knowledge available.

 

An example of how such a spreadsheet might look appears here. (click link).

       

 

top

Links and Resources for Critiquing:

General research resources, understanding research methods:

The Knowledge Base

 

http://www.socialresearchmethods.net/

 

An excellent on-line research methods text book, and other research resources

Project Gold Research Methods Glossary

http://www.bath.ac.uk/e-learning/gold/glossary.html

A user-friendly research glossary on-line.

Statistics on-line

 

http://www.statsoft.com/textbook/stathome.html

A very comprehensive statistics textbook on-line, the entire book can be downloaded for free. You will find a full explanation of all major statistical tests and concepts.

Hyperstat

http://davidmlane.com/hyperstat

As comprehensive, simpler
SOSIG Research Methods

http://www.sosig.ac.uk/roads/subject-listing/World-cat/meth.html

 

The Social Science Information Gateway (SOSIG) research methods resources - lots of resources for learning about qualitative and quantitative research.

Books:

Recommended Text Books

 

http://www.researchbooks.co.uk/recommended%20research%20text%20books.htm

Some suggested research textbooks that may assist with understanding and critiquing research, with details of library shelf numbers and purchase details. This is just a small selection of the many available.

Critiquing skills & resources:

How to read a paper

 

http://www.richard-ingram.co.uk/howto.htm

A British Medical Journal series of 10 articles assisting novices to read and understand research papers. Full text available on-line. Fairly simple to read, very helpful.

Critical Appraisal Resources for assessing Health and Medical Research

 

http://library.ukc.ac.uk/library/info/subjectg/healthinfo/critapprais.htm

A useful web site on critical appraisal from the Library at University of Kent at Canterbury

Critical thinking

 

http://www.canberra.edu.au/studyskills/learning/critical

Not a critiquing framework, but a useful guide to what critical thinking involves. Canberra University, Australia.

Critiquing guides / frameworks:

Guidelines for Critiquing Research Articles

 

http://www.sonoma.edu/users/n/nolan/n400/critique.htm

Sonama State University, California, USA. The guidelines are from Polit, Beck and Hungler 2001: A comprehensive critiquing framework, derived from Polit et al’s book.

Framework for critically appraising research articles

http://hsc.uwe.ac.uk/dataanalysis/critFrame.asp

A critiquing framework within the UWE Data Analysis Web Site (co-authored by me!). User friendly. Example critiques also on the site

Guidelines for critiquing research articles

 

http://www2.msstate.edu/~bsc2/guidelines.htm

From Mississippi State University, USA. Simple critiquing framework, but perhaps a poor example as it tends to focus on textual critique, rather than critique of methods.

CASP Critiquing Tools

 

http://www.phru.nhs.uk/Pages/PHD/resources.htm

CASP (Critical Appraial Skills Programme – an NHS site). Now has an excellent range of tools for evaluating different types of studies and reviews.

Studying systematic reviews

http://www.findarticles.com/cf_0/m0689/11_50/80531397/p1/article.jhtml

An on-line article discussing critical reading of systematic reviews
Ciliska, D., Cullum, N., & Marks, S. (2001). Evaluation of systematic reviews of treatment or prevention interventions. Evidence-Based Nursing. 4(4), October, 100-104

Full text available from Ovid Online (Athens password required)

A useful article on critically appraising systematic reviews.
Ajetunmobi, O. (2002). Making Sense of Critical Appraisal. London: Arnold.

view or purchase on-line

An excellent little book offering criteria / frameworks for critiquing various types of research, plus audits, systematic reviews and economic evaluations. Includes basic research knowledge and explanation of statistics in order to apply these to papers you might be critiquing
Evaluating Web Sites: Criteria and Tools

http://www.library.cornell.edu/olinuris/ref/research/webeval.html

 

Some criteria to help you evaluate web content (usually not primary research)
Example critiques:  
1)   Data Analysis - Critical Appraisal - worked example 1

http://hsc.uwe.ac.uk/dataanalysis/critWorked.asp

 

From the UWE Data Analysis Web Site

 

http://hsc.uwe.ac.uk/dataanalysis/index.asp

2)  Data Analysis - Critical Appraisal - worked example 2

http://hsc.uwe.ac.uk/dataanalysis/critDevelop.asp

 

top

Glossary

 

Population

 

A well-defined group or set that has certain specified properties (e.g. all registered midwives working full-time in Scotland). A selection (sample) of the population may be made for study in research  Go to on-line glossary

 

top

Reliability

Reliability is concerned with the consistency and dependability of a measuring instrument, i.e. it is an indication of the degree to which it gives the same answers over time, across similar groups and irrespective of who administers it. A reliable measuring instrument will always give the same result on different occasions assuming that what is being measured has not changed during the intervening period.  Go to on-line glossary

top

Validity

In research terms, validity refers to the accuracy and truth of the data and findings that are produced. It refers to the concepts that are being investigated; the people or objects that are being studied; the methods by which data are collected; and the findings that are produced. There are several different types of validity: Go to on-line glossary

 top


Mock-up example of a spreadsheet:  

 

Could be used for organising findings for literature review. Headings are not exhaustive, consider what you need. Probably needs to be on A3 Landscape paper or on PC. If it is on A3 paper, you will be able to fit many more rows (articles) on the page. There is absolutely no requirement to use this in your work - its just a tool I have personally found to be useful.  NB: this is NOT a critiquing framework

 

Author(s)

Date

Research Question

Design & Method

Population Sample

(Size, Type)

Data Collection/

instruments

Analysis

Findings

Strengths

Limitations

Conclusion

Implications

(take home message)

Bloggs, J

 

(1998)

Do bladder washouts prevent catheter blockage?

Quasi-experimental

Random assignment to exp / ctrl

Catheterized patients in 1 UK GP practice  (N = 87)

 

Random sample n=26

D.N. questionnaire & notes.

DN’s trained to administer washouts & identify defined “blockages”

 

Good scores for inter-rater reliability

Descriptive & Non-parametric

No significant difference between groups

Appropriate design,

 

good control

Small sample size (type II error possible)

 

No power analysis or other statistical rationale for sample size

 

No blinding or true placebo

Too small sample, may have missed effect of washouts, thus suggesting wrong conclusion (type II error)

Does not support or reject use of washouts

 

Good basis for larger study

Paper 2

 

 

 

 

 

 

 

 

 

 

 

 

 

Paper 3

 

 

 

 

 

 

 

 

 

 

 

 

Paper 4 etc

 

 

 

 

 

 

 

 

 

 

 

top

 

Version  4.5    PAGE CREATED 01/10/2002        LAST UPDATED  23/07/07   © Richard Ingram 2002