research article method

• Interlibrary Loan and Scan & Deliver
• Course Reserves
• Purchase Request
• Collection Development & Maintenance
• Current Negotiations
• Ask a Librarian
• Instructor Support
• Library How-To
• Research Guides
• Research Support
• Study Rooms
• Research Rooms
• Partner Spaces
• Loanable Equipment
• Print, Scan, Copy
• 3D Printers
• Poster Printing
• OSULP Leadership
• Strategic Plan

Scholarly Articles: How can I tell?

• Journal Information
• Literature Review
• Author and affiliation
• Introduction
• Specialized Vocabulary

Methodology

• Research sponsors
• Peer-review

The methodology section or methods section tells you how the author(s) went about doing their research. It should let you know a) what method they used to gather data (survey, interviews, experiments, etc.), why they chose this method, and what the limitations are to this method.

The methodology section should be detailed enough that another researcher could replicate the study described. When you read the methodology or methods section:

• What kind of research method did the authors use? Is it an appropriate method for the type of study they are conducting?
• How did the authors get their tests subjects? What criteria did they use?
• What are the contexts of the study that may have affected the results (e.g. environmental conditions, lab conditions, timing questions, etc.)
• Is the sample size representative of the larger population (i.e., was it big enough?)
• Are the data collection instruments and procedures likely to have measured all the important characteristics with reasonable accuracy?
• Does the data analysis appear to have been done with care, and were appropriate analytical techniques used? 

A good researcher will always let you know about the limitations of his or her research.

• << Previous: Specialized Vocabulary
• Next: Results >>
• Last Updated: Apr 15, 2024 3:26 PM
• URL: https://guides.library.oregonstate.edu/ScholarlyArticle

Contact Info

121 The Valley Library Corvallis OR 97331–4501

Phone: 541-737-3331

Services for Persons with Disabilities

In the Valley Library

• Oregon State University Press
• Special Collections and Archives Research Center
• Undergrad Research & Writing Studio
• Graduate Student Commons
• Tutoring Services
• Northwest Art Collection

Digital Projects

• Oregon Explorer
• Oregon Digital
• ScholarsArchive@OSU
• Digital Publishing Initiatives
• Atlas of the Pacific Northwest
• Marilyn Potts Guin Library  
• Cascades Campus Library
• McDowell Library of Vet Medicine

• Open access
• Published: 07 September 2020

A tutorial on methodological studies: the what, when, how and why

• Lawrence Mbuagbaw   ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
• Daeria O. Lawson 1 ,
• Livia Puljak 4 ,
• David B. Allison 5 &
• Lehana Thabane 1 , 2 , 6 , 7 , 8  

BMC Medical Research Methodology volume  20 , Article number:  226 ( 2020 ) Cite this article

42k Accesses

60 Citations

60 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed   Google Scholar  

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed   PubMed Central   Google Scholar  

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS   PubMed   Google Scholar  

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar  

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS   Google Scholar  

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS   PubMed   PubMed Central   Google Scholar  

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodological study
• Meta-epidemiology
• Research methods
• Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 6. The Methodology
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The methods section describes actions taken to investigate a research problem and the rationale for the application of specific procedures or techniques used to identify, select, process, and analyze information applied to understanding the problem, thereby, allowing the reader to critically evaluate a study’s overall validity and reliability. The methodology section of a research paper answers two main questions: How was the data collected or generated? And, how was it analyzed? The writing should be direct and precise and always written in the past tense.

Kallet, Richard H. "How to Write the Methods Section of a Research Paper." Respiratory Care 49 (October 2004): 1229-1232.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

• Readers need to know how the data was obtained because the method you chose affects the results and, by extension, how you interpreted their significance in the discussion section of your paper.
• Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and, as a consequence, undermines the value of your analysis of the findings.
• In most cases, there are a variety of different methods you can choose to investigate a research problem. The methodology section of your paper should clearly articulate the reasons why you have chosen a particular procedure or technique.
• The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a multiple choice questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
• The method must be appropriate to fulfilling the overall aims of the study. For example, you need to ensure that you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
• The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring. For any problems that do arise, you must describe the ways in which they were minimized or why these problems do not impact in any meaningful way your interpretation of the findings.
• In the social and behavioral sciences, it is important to always provide sufficient information to allow other researchers to adopt or replicate your methodology. This information is particularly important when a new method has been developed or an innovative use of an existing method is utilized.

Bem, Daryl J. Writing the Empirical Journal Article. Psychology Writing Center. University of Washington; Denscombe, Martyn. The Good Research Guide: For Small-Scale Social Research Projects . 5th edition. Buckingham, UK: Open University Press, 2014; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I.  Groups of Research Methods

There are two main groups of research methods in the social sciences:

• The e mpirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences . This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for formulating hypotheses that need to be tested. This approach is focused on explanation.
• The i nterpretative group of methods is focused on understanding phenomenon in a comprehensive, holistic way . Interpretive methods focus on analytically disclosing the meaning-making practices of human subjects [the why, how, or by what means people do what they do], while showing how those practices arrange so that it can be used to generate observable outcomes. Interpretive methods allow you to recognize your connection to the phenomena under investigation. However, the interpretative group requires careful examination of variables because it focuses more on subjective knowledge.

II.  Content

The introduction to your methodology section should begin by restating the research problem and underlying assumptions underpinning your study. This is followed by situating the methods you used to gather, analyze, and process information within the overall “tradition” of your field of study and within the particular research design you have chosen to study the problem. If the method you choose lies outside of the tradition of your field [i.e., your review of the literature demonstrates that the method is not commonly used], provide a justification for how your choice of methods specifically addresses the research problem in ways that have not been utilized in prior studies.

The remainder of your methodology section should describe the following:

• Decisions made in selecting the data you have analyzed or, in the case of qualitative research, the subjects and research setting you have examined,
• Tools and methods used to identify and collect information, and how you identified relevant variables,
• The ways in which you processed the data and the procedures you used to analyze that data, and
• The specific research tools or strategies that you utilized to study the underlying hypothesis and research questions.

In addition, an effectively written methodology section should:

• Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
• Indicate how the approach fits the overall research design . Your methods for gathering data should have a clear connection to your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is not suitable to achieving the stated objective of your paper.
• Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom. Also be sure to explain how older data is still relevant to investigating the current research problem.
• Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors? Describe how you plan to obtain an accurate assessment of relationships, patterns, trends, distributions, and possible contradictions found in the data.
• Provide background and a rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
• Provide a justification for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of data being used? If other data sources exist, explain why the data you chose is most appropriate to addressing the research problem.
• Provide a justification for case study selection . A common method of analyzing research problems in the social sciences is to analyze specific cases. These can be a person, place, event, phenomenon, or other type of subject of analysis that are either examined as a singular topic of in-depth investigation or multiple topics of investigation studied for the purpose of comparing or contrasting findings. In either method, you should explain why a case or cases were chosen and how they specifically relate to the research problem.
• Describe potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

NOTE:   Once you have written all of the elements of the methods section, subsequent revisions should focus on how to present those elements as clearly and as logically as possibly. The description of how you prepared to study the research problem, how you gathered the data, and the protocol for analyzing the data should be organized chronologically. For clarity, when a large amount of detail must be presented, information should be presented in sub-sections according to topic. If necessary, consider using appendices for raw data.

ANOTHER NOTE: If you are conducting a qualitative analysis of a research problem , the methodology section generally requires a more elaborate description of the methods used as well as an explanation of the processes applied to gathering and analyzing of data than is generally required for studies using quantitative methods. Because you are the primary instrument for generating the data [e.g., through interviews or observations], the process for collecting that data has a significantly greater impact on producing the findings. Therefore, qualitative research requires a more detailed description of the methods used.

YET ANOTHER NOTE:   If your study involves interviews, observations, or other qualitative techniques involving human subjects , you may be required to obtain approval from the university's Office for the Protection of Research Subjects before beginning your research. This is not a common procedure for most undergraduate level student research assignments. However, i f your professor states you need approval, you must include a statement in your methods section that you received official endorsement and adequate informed consent from the office and that there was a clear assessment and minimization of risks to participants and to the university. This statement informs the reader that your study was conducted in an ethical and responsible manner. In some cases, the approval notice is included as an appendix to your paper.

III.  Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but concise. Do not provide any background information that does not directly help the reader understand why a particular method was chosen, how the data was gathered or obtained, and how the data was analyzed in relation to the research problem [note: analyzed, not interpreted! Save how you interpreted the findings for the discussion section]. With this in mind, the page length of your methods section will generally be less than any other section of your paper except the conclusion.

Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. An exception to this rule is if you select an unconventional methodological approach; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall process of discovery.

Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data, or, gaps will exist in existing data or archival materials. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose.

Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in and of itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. "How to Write a Scientific Paper: Writing the Methods Section." Revista Portuguesa de Pneumologia 17 (2011): 232-238; Blair Lorrie. “Choosing a Methodology.” In Writing a Graduate Thesis or Dissertation , Teaching Writing Series. (Rotterdam: Sense Publishers 2016), pp. 49-72; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Kallet, Richard H. “How to Write the Methods Section of a Research Paper.” Respiratory Care 49 (October 2004):1229-1232; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section. The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Rudestam, Kjell Erik and Rae R. Newton. “The Method Chapter: Describing Your Research Plan.” In Surviving Your Dissertation: A Comprehensive Guide to Content and Process . (Thousand Oaks, Sage Publications, 2015), pp. 87-115; What is Interpretive Research. Institute of Public and International Affairs, University of Utah; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University; Methods and Materials. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Don't avoid using a quantitative approach to analyzing your research problem just because you fear the idea of applying statistical designs and tests. A qualitative approach, such as conducting interviews or content analysis of archival texts, can yield exciting new insights about a research problem, but it should not be undertaken simply because you have a disdain for running a simple regression. A well designed quantitative research study can often be accomplished in very clear and direct ways, whereas, a similar study of a qualitative nature usually requires considerable time to analyze large volumes of data and a tremendous burden to create new paths for analysis where previously no path associated with your research problem had existed.

To locate data and statistics, GO HERE .

Another Writing Tip

Knowing the Relationship Between Theories and Methods

There can be multiple meaning associated with the term "theories" and the term "methods" in social sciences research. A helpful way to delineate between them is to understand "theories" as representing different ways of characterizing the social world when you research it and "methods" as representing different ways of generating and analyzing data about that social world. Framed in this way, all empirical social sciences research involves theories and methods, whether they are stated explicitly or not. However, while theories and methods are often related, it is important that, as a researcher, you deliberately separate them in order to avoid your theories playing a disproportionate role in shaping what outcomes your chosen methods produce.

Introspectively engage in an ongoing dialectic between the application of theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics . Part 1, Chapter 3. Boise State University; The Theory-Method Relationship. S-Cool Revision. United Kingdom.

Yet Another Writing Tip

Methods and the Methodology

Do not confuse the terms "methods" and "methodology." As Schneider notes, a method refers to the technical steps taken to do research . Descriptions of methods usually include defining and stating why you have chosen specific techniques to investigate a research problem, followed by an outline of the procedures you used to systematically select, gather, and process the data [remember to always save the interpretation of data for the discussion section of your paper].

The methodology refers to a discussion of the underlying reasoning why particular methods were used . This discussion includes describing the theoretical concepts that inform the choice of methods to be applied, placing the choice of methods within the more general nature of academic work, and reviewing its relevance to examining the research problem. The methodology section also includes a thorough review of the methods other scholars have used to study the topic.

Bryman, Alan. "Of Methods and Methodology." Qualitative Research in Organizations and Management: An International Journal 3 (2008): 159-168; Schneider, Florian. “What's in a Methodology: The Difference between Method, Methodology, and Theory…and How to Get the Balance Right?” PoliticsEastAsia.com. Chinese Department, University of Leiden, Netherlands.

• << Previous: Scholarly vs. Popular Publications
• Next: Qualitative Methods >>
• Last Updated: Sep 4, 2024 9:40 AM
• URL: https://libguides.usc.edu/writingguide

• Methodology
• Open access
• Published: 11 October 2016

Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research

• Stephen J. Gentles 1 , 4 ,
• Cathy Charles 1 ,
• David B. Nicholas 2 ,
• Jenny Ploeg 3 &
• K. Ann McKibbon 1  

Systematic Reviews volume  5 , Article number:  172 ( 2016 ) Cite this article

54k Accesses

27 Citations

13 Altmetric

Metrics details

Overviews of methods are potentially useful means to increase clarity and enhance collective understanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness. This type of review represents a distinct literature synthesis method, although to date, its methodology remains relatively undeveloped despite several aspects that demand unique review procedures. The purpose of this paper is to initiate discussion about what a rigorous systematic approach to reviews of methods, referred to here as systematic methods overviews , might look like by providing tentative suggestions for approaching specific challenges likely to be encountered. The guidance offered here was derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research.

The guidance is organized into several principles that highlight specific objectives for this type of review given the common challenges that must be overcome to achieve them. Optional strategies for achieving each principle are also proposed, along with discussion of how they were successfully implemented in the overview on sampling. We describe seven paired principles and strategies that address the following aspects: delimiting the initial set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology used to describe specific methods topics, and generating rigorous verifiable analytic interpretations. Since a broad aim in systematic methods overviews is to describe and interpret the relevant literature in qualitative terms, we suggest that iterative decision making at various stages of the review process, and a rigorous qualitative approach to analysis are necessary features of this review type.

Conclusions

We believe that the principles and strategies provided here will be useful to anyone choosing to undertake a systematic methods overview. This paper represents an initial effort to promote high quality critical evaluations of the literature regarding problematic methods topics, which have the potential to promote clearer, shared understandings, and accelerate advances in research methods. Further work is warranted to develop more definitive guidance.

Peer Review reports

While reviews of methods are not new, they represent a distinct review type whose methodology remains relatively under-addressed in the literature despite the clear implications for unique review procedures. One of few examples to describe it is a chapter containing reflections of two contributing authors in a book of 21 reviews on methodological topics compiled for the British National Health Service, Health Technology Assessment Program [ 1 ]. Notable is their observation of how the differences between the methods reviews and conventional quantitative systematic reviews, specifically attributable to their varying content and purpose, have implications for defining what qualifies as systematic. While the authors describe general aspects of “systematicity” (including rigorous application of a methodical search, abstraction, and analysis), they also describe a high degree of variation within the category of methods reviews itself and so offer little in the way of concrete guidance. In this paper, we present tentative concrete guidance, in the form of a preliminary set of proposed principles and optional strategies, for a rigorous systematic approach to reviewing and evaluating the literature on quantitative or qualitative methods topics. For purposes of this article, we have used the term systematic methods overview to emphasize the notion of a systematic approach to such reviews.

The conventional focus of rigorous literature reviews (i.e., review types for which systematic methods have been codified, including the various approaches to quantitative systematic reviews [ 2 – 4 ], and the numerous forms of qualitative and mixed methods literature synthesis [ 5 – 10 ]) is to synthesize empirical research findings from multiple studies. By contrast, the focus of overviews of methods, including the systematic approach we advocate, is to synthesize guidance on methods topics. The literature consulted for such reviews may include the methods literature, methods-relevant sections of empirical research reports, or both. Thus, this paper adds to previous work published in this journal—namely, recent preliminary guidance for conducting reviews of theory [ 11 ]—that has extended the application of systematic review methods to novel review types that are concerned with subject matter other than empirical research findings.

Published examples of methods overviews illustrate the varying objectives they can have. One objective is to establish methodological standards for appraisal purposes. For example, reviews of existing quality appraisal standards have been used to propose universal standards for appraising the quality of primary qualitative research [ 12 ] or evaluating qualitative research reports [ 13 ]. A second objective is to survey the methods-relevant sections of empirical research reports to establish current practices on methods use and reporting practices, which Moher and colleagues [ 14 ] recommend as a means for establishing the needs to be addressed in reporting guidelines (see, for example [ 15 , 16 ]). A third objective for a methods review is to offer clarity and enhance collective understanding regarding a specific methods topic that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness within the available methods literature. An example of this is a overview whose objective was to review the inconsistent definitions of intention-to-treat analysis (the methodologically preferred approach to analyze randomized controlled trial data) that have been offered in the methods literature and propose a solution for improving conceptual clarity [ 17 ]. Such reviews are warranted because students and researchers who must learn or apply research methods typically lack the time to systematically search, retrieve, review, and compare the available literature to develop a thorough and critical sense of the varied approaches regarding certain controversial or ambiguous methods topics.

While systematic methods overviews , as a review type, include both reviews of the methods literature and reviews of methods-relevant sections from empirical study reports, the guidance provided here is primarily applicable to reviews of the methods literature since it was derived from the experience of conducting such a review [ 18 ], described below. To our knowledge, there are no well-developed proposals on how to rigorously conduct such reviews. Such guidance would have the potential to improve the thoroughness and credibility of critical evaluations of the methods literature, which could increase their utility as a tool for generating understandings that advance research methods, both qualitative and quantitative. Our aim in this paper is thus to initiate discussion about what might constitute a rigorous approach to systematic methods overviews. While we hope to promote rigor in the conduct of systematic methods overviews wherever possible, we do not wish to suggest that all methods overviews need be conducted to the same standard. Rather, we believe that the level of rigor may need to be tailored pragmatically to the specific review objectives, which may not always justify the resource requirements of an intensive review process.

The example systematic methods overview on sampling in qualitative research

The principles and strategies we propose in this paper are derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research [ 18 ]. The main objective of that methods overview was to bring clarity and deeper understanding of the prominent concepts related to sampling in qualitative research (purposeful sampling strategies, saturation, etc.). Specifically, we interpreted the available guidance, commenting on areas lacking clarity, consistency, or comprehensiveness (without proposing any recommendations on how to do sampling). This was achieved by a comparative and critical analysis of publications representing the most influential (i.e., highly cited) guidance across several methodological traditions in qualitative research.

The specific methods and procedures for the overview on sampling [ 18 ] from which our proposals are derived were developed both after soliciting initial input from local experts in qualitative research and an expert health librarian (KAM) and through ongoing careful deliberation throughout the review process. To summarize, in that review, we employed a transparent and rigorous approach to search the methods literature, selected publications for inclusion according to a purposeful and iterative process, abstracted textual data using structured abstraction forms, and analyzed (synthesized) the data using a systematic multi-step approach featuring abstraction of text, summary of information in matrices, and analytic comparisons.

For this article, we reflected on both the problems and challenges encountered at different stages of the review and our means for selecting justifiable procedures to deal with them. Several principles were then derived by considering the generic nature of these problems, while the generalizable aspects of the procedures used to address them formed the basis of optional strategies. Further details of the specific methods and procedures used in the overview on qualitative sampling are provided below to illustrate both the types of objectives and challenges that reviewers will likely need to consider and our approach to implementing each of the principles and strategies.

Organization of the guidance into principles and strategies

For the purposes of this article, principles are general statements outlining what we propose are important aims or considerations within a particular review process, given the unique objectives or challenges to be overcome with this type of review. These statements follow the general format, “considering the objective or challenge of X, we propose Y to be an important aim or consideration.” Strategies are optional and flexible approaches for implementing the previous principle outlined. Thus, generic challenges give rise to principles, which in turn give rise to strategies.

We organize the principles and strategies below into three sections corresponding to processes characteristic of most systematic literature synthesis approaches: literature identification and selection ; data abstraction from the publications selected for inclusion; and analysis , including critical appraisal and synthesis of the abstracted data. Within each section, we also describe the specific methodological decisions and procedures used in the overview on sampling in qualitative research [ 18 ] to illustrate how the principles and strategies for each review process were applied and implemented in a specific case. We expect this guidance and accompanying illustrations will be useful for anyone considering engaging in a methods overview, particularly those who may be familiar with conventional systematic review methods but may not yet appreciate some of the challenges specific to reviewing the methods literature.

Results and discussion

Literature identification and selection.

The identification and selection process includes search and retrieval of publications and the development and application of inclusion and exclusion criteria to select the publications that will be abstracted and analyzed in the final review. Literature identification and selection for overviews of the methods literature is challenging and potentially more resource-intensive than for most reviews of empirical research. This is true for several reasons that we describe below, alongside discussion of the potential solutions. Additionally, we suggest in this section how the selection procedures can be chosen to match the specific analytic approach used in methods overviews.

Delimiting a manageable set of publications

One aspect of methods overviews that can make identification and selection challenging is the fact that the universe of literature containing potentially relevant information regarding most methods-related topics is expansive and often unmanageably so. Reviewers are faced with two large categories of literature: the methods literature , where the possible publication types include journal articles, books, and book chapters; and the methods-relevant sections of empirical study reports , where the possible publication types include journal articles, monographs, books, theses, and conference proceedings. In our systematic overview of sampling in qualitative research, exhaustively searching (including retrieval and first-pass screening) all publication types across both categories of literature for information on a single methods-related topic was too burdensome to be feasible. The following proposed principle follows from the need to delimit a manageable set of literature for the review.

Principle #1:

Considering the broad universe of potentially relevant literature, we propose that an important objective early in the identification and selection stage is to delimit a manageable set of methods-relevant publications in accordance with the objectives of the methods overview.

Strategy #1:

To limit the set of methods-relevant publications that must be managed in the selection process, reviewers have the option to initially review only the methods literature, and exclude the methods-relevant sections of empirical study reports, provided this aligns with the review’s particular objectives.

We propose that reviewers are justified in choosing to select only the methods literature when the objective is to map out the range of recognized concepts relevant to a methods topic, to summarize the most authoritative or influential definitions or meanings for methods-related concepts, or to demonstrate a problematic lack of clarity regarding a widely established methods-related concept and potentially make recommendations for a preferred approach to the methods topic in question. For example, in the case of the methods overview on sampling [ 18 ], the primary aim was to define areas lacking in clarity for multiple widely established sampling-related topics. In the review on intention-to-treat in the context of missing outcome data [ 17 ], the authors identified a lack of clarity based on multiple inconsistent definitions in the literature and went on to recommend separating the issue of how to handle missing outcome data from the issue of whether an intention-to-treat analysis can be claimed.

In contrast to strategy #1, it may be appropriate to select the methods-relevant sections of empirical study reports when the objective is to illustrate how a methods concept is operationalized in research practice or reported by authors. For example, one could review all the publications in 2 years’ worth of issues of five high-impact field-related journals to answer questions about how researchers describe implementing a particular method or approach, or to quantify how consistently they define or report using it. Such reviews are often used to highlight gaps in the reporting practices regarding specific methods, which may be used to justify items to address in reporting guidelines (for example, [ 14 – 16 ]).

It is worth recognizing that other authors have advocated broader positions regarding the scope of literature to be considered in a review, expanding on our perspective. Suri [ 10 ] (who, like us, emphasizes how different sampling strategies are suitable for different literature synthesis objectives) has, for example, described a two-stage literature sampling procedure (pp. 96–97). First, reviewers use an initial approach to conduct a broad overview of the field—for reviews of methods topics, this would entail an initial review of the research methods literature. This is followed by a second more focused stage in which practical examples are purposefully selected—for methods reviews, this would involve sampling the empirical literature to illustrate key themes and variations. While this approach is seductive in its capacity to generate more in depth and interpretive analytic findings, some reviewers may consider it too resource-intensive to include the second step no matter how selective the purposeful sampling. In the overview on sampling where we stopped after the first stage [ 18 ], we discussed our selective focus on the methods literature as a limitation that left opportunities for further analysis of the literature. We explicitly recommended, for example, that theoretical sampling was a topic for which a future review of the methods sections of empirical reports was justified to answer specific questions identified in the primary review.

Ultimately, reviewers must make pragmatic decisions that balance resource considerations, combined with informed predictions about the depth and complexity of literature available on their topic, with the stated objectives of their review. The remaining principles and strategies apply primarily to overviews that include the methods literature, although some aspects may be relevant to reviews that include empirical study reports.

Searching beyond standard bibliographic databases

An important reality affecting identification and selection in overviews of the methods literature is the increased likelihood for relevant publications to be located in sources other than journal articles (which is usually not the case for overviews of empirical research, where journal articles generally represent the primary publication type). In the overview on sampling [ 18 ], out of 41 full-text publications retrieved and reviewed, only 4 were journal articles, while 37 were books or book chapters. Since many books and book chapters did not exist electronically, their full text had to be physically retrieved in hardcopy, while 11 publications were retrievable only through interlibrary loan or purchase request. The tasks associated with such retrieval are substantially more time-consuming than electronic retrieval. Since a substantial proportion of methods-related guidance may be located in publication types that are less comprehensively indexed in standard bibliographic databases, identification and retrieval thus become complicated processes.

Principle #2:

Considering that important sources of methods guidance can be located in non-journal publication types (e.g., books, book chapters) that tend to be poorly indexed in standard bibliographic databases, it is important to consider alternative search methods for identifying relevant publications to be further screened for inclusion.

Strategy #2:

To identify books, book chapters, and other non-journal publication types not thoroughly indexed in standard bibliographic databases, reviewers may choose to consult one or more of the following less standard sources: Google Scholar, publisher web sites, or expert opinion.

In the case of the overview on sampling in qualitative research [ 18 ], Google Scholar had two advantages over other standard bibliographic databases: it indexes and returns records of books and book chapters likely to contain guidance on qualitative research methods topics; and it has been validated as providing higher citation counts than ISI Web of Science (a producer of numerous bibliographic databases accessible through institutional subscription) for several non-biomedical disciplines including the social sciences where qualitative research methods are prominently used [ 19 – 21 ]. While we identified numerous useful publications by consulting experts, the author publication lists generated through Google Scholar searches were uniquely useful to identify more recent editions of methods books identified by experts.

Searching without relevant metadata

Determining what publications to select for inclusion in the overview on sampling [ 18 ] could only rarely be accomplished by reviewing the publication’s metadata. This was because for the many books and other non-journal type publications we identified as possibly relevant, the potential content of interest would be located in only a subsection of the publication. In this common scenario for reviews of the methods literature (as opposed to methods overviews that include empirical study reports), reviewers will often be unable to employ standard title, abstract, and keyword database searching or screening as a means for selecting publications.

Principle #3:

Considering that the presence of information about the topic of interest may not be indicated in the metadata for books and similar publication types, it is important to consider other means of identifying potentially useful publications for further screening.

Strategy #3:

One approach to identifying potentially useful books and similar publication types is to consider what classes of such publications (e.g., all methods manuals for a certain research approach) are likely to contain relevant content, then identify, retrieve, and review the full text of corresponding publications to determine whether they contain information on the topic of interest.

In the example of the overview on sampling in qualitative research [ 18 ], the topic of interest (sampling) was one of numerous topics covered in the general qualitative research methods manuals. Consequently, examples from this class of publications first had to be identified for retrieval according to non-keyword-dependent criteria. Thus, all methods manuals within the three research traditions reviewed (grounded theory, phenomenology, and case study) that might contain discussion of sampling were sought through Google Scholar and expert opinion, their full text obtained, and hand-searched for relevant content to determine eligibility. We used tables of contents and index sections of books to aid this hand searching.

Purposefully selecting literature on conceptual grounds

A final consideration in methods overviews relates to the type of analysis used to generate the review findings. Unlike quantitative systematic reviews where reviewers aim for accurate or unbiased quantitative estimates—something that requires identifying and selecting the literature exhaustively to obtain all relevant data available (i.e., a complete sample)—in methods overviews, reviewers must describe and interpret the relevant literature in qualitative terms to achieve review objectives. In other words, the aim in methods overviews is to seek coverage of the qualitative concepts relevant to the methods topic at hand. For example, in the overview of sampling in qualitative research [ 18 ], achieving review objectives entailed providing conceptual coverage of eight sampling-related topics that emerged as key domains. The following principle recognizes that literature sampling should therefore support generating qualitative conceptual data as the input to analysis.

Principle #4:

Since the analytic findings of a systematic methods overview are generated through qualitative description and interpretation of the literature on a specified topic, selection of the literature should be guided by a purposeful strategy designed to achieve adequate conceptual coverage (i.e., representing an appropriate degree of variation in relevant ideas) of the topic according to objectives of the review.

Strategy #4:

One strategy for choosing the purposeful approach to use in selecting the literature according to the review objectives is to consider whether those objectives imply exploring concepts either at a broad overview level, in which case combining maximum variation selection with a strategy that limits yield (e.g., critical case, politically important, or sampling for influence—described below) may be appropriate; or in depth, in which case purposeful approaches aimed at revealing innovative cases will likely be necessary.

In the methods overview on sampling, the implied scope was broad since we set out to review publications on sampling across three divergent qualitative research traditions—grounded theory, phenomenology, and case study—to facilitate making informative conceptual comparisons. Such an approach would be analogous to maximum variation sampling.

At the same time, the purpose of that review was to critically interrogate the clarity, consistency, and comprehensiveness of literature from these traditions that was “most likely to have widely influenced students’ and researchers’ ideas about sampling” (p. 1774) [ 18 ]. In other words, we explicitly set out to review and critique the most established and influential (and therefore dominant) literature, since this represents a common basis of knowledge among students and researchers seeking understanding or practical guidance on sampling in qualitative research. To achieve this objective, we purposefully sampled publications according to the criterion of influence , which we operationalized as how often an author or publication has been referenced in print or informal discourse. This second sampling approach also limited the literature we needed to consider within our broad scope review to a manageable amount.

To operationalize this strategy of sampling for influence , we sought to identify both the most influential authors within a qualitative research tradition (all of whose citations were subsequently screened) and the most influential publications on the topic of interest by non-influential authors. This involved a flexible approach that combined multiple indicators of influence to avoid the dilemma that any single indicator might provide inadequate coverage. These indicators included bibliometric data (h-index for author influence [ 22 ]; number of cites for publication influence), expert opinion, and cross-references in the literature (i.e., snowball sampling). As a final selection criterion, a publication was included only if it made an original contribution in terms of novel guidance regarding sampling or a related concept; thus, purely secondary sources were excluded. Publish or Perish software (Anne-Wil Harzing; available at http://www.harzing.com/resources/publish-or-perish ) was used to generate bibliometric data via the Google Scholar database. Figure  1 illustrates how identification and selection in the methods overview on sampling was a multi-faceted and iterative process. The authors selected as influential, and the publications selected for inclusion or exclusion are listed in Additional file 1 (Matrices 1, 2a, 2b).

Literature identification and selection process used in the methods overview on sampling [ 18 ]

In summary, the strategies of seeking maximum variation and sampling for influence were employed in the sampling overview to meet the specific review objectives described. Reviewers will need to consider the full range of purposeful literature sampling approaches at their disposal in deciding what best matches the specific aims of their own reviews. Suri [ 10 ] has recently retooled Patton’s well-known typology of purposeful sampling strategies (originally intended for primary research) for application to literature synthesis, providing a useful resource in this respect.

Data abstraction

The purpose of data abstraction in rigorous literature reviews is to locate and record all data relevant to the topic of interest from the full text of included publications, making them available for subsequent analysis. Conventionally, a data abstraction form—consisting of numerous distinct conceptually defined fields to which corresponding information from the source publication is recorded—is developed and employed. There are several challenges, however, to the processes of developing the abstraction form and abstracting the data itself when conducting methods overviews, which we address here. Some of these problems and their solutions may be familiar to those who have conducted qualitative literature syntheses, which are similarly conceptual.

Iteratively defining conceptual information to abstract

In the overview on sampling [ 18 ], while we surveyed multiple sources beforehand to develop a list of concepts relevant for abstraction (e.g., purposeful sampling strategies, saturation, sample size), there was no way for us to anticipate some concepts prior to encountering them in the review process. Indeed, in many cases, reviewers are unable to determine the complete set of methods-related concepts that will be the focus of the final review a priori without having systematically reviewed the publications to be included. Thus, defining what information to abstract beforehand may not be feasible.

Principle #5:

Considering the potential impracticality of defining a complete set of relevant methods-related concepts from a body of literature one has not yet systematically read, selecting and defining fields for data abstraction must often be undertaken iteratively. Thus, concepts to be abstracted can be expected to grow and change as data abstraction proceeds.

Strategy #5:

Reviewers can develop an initial form or set of concepts for abstraction purposes according to standard methods (e.g., incorporating expert feedback, pilot testing) and remain attentive to the need to iteratively revise it as concepts are added or modified during the review. Reviewers should document revisions and return to re-abstract data from previously abstracted publications as the new data requirements are determined.

In the sampling overview [ 18 ], we developed and maintained the abstraction form in Microsoft Word. We derived the initial set of abstraction fields from our own knowledge of relevant sampling-related concepts, consultation with local experts, and reviewing a pilot sample of publications. Since the publications in this review included a large proportion of books, the abstraction process often began by flagging the broad sections within a publication containing topic-relevant information for detailed review to identify text to abstract. When reviewing flagged text, the reviewer occasionally encountered an unanticipated concept significant enough to warrant being added as a new field to the abstraction form. For example, a field was added to capture how authors described the timing of sampling decisions, whether before (a priori) or after (ongoing) starting data collection, or whether this was unclear. In these cases, we systematically documented the modification to the form and returned to previously abstracted publications to abstract any information that might be relevant to the new field.

The logic of this strategy is analogous to the logic used in a form of research synthesis called best fit framework synthesis (BFFS) [ 23 – 25 ]. In that method, reviewers initially code evidence using an a priori framework they have selected. When evidence cannot be accommodated by the selected framework, reviewers then develop new themes or concepts from which they construct a new expanded framework. Both the strategy proposed and the BFFS approach to research synthesis are notable for their rigorous and transparent means to adapt a final set of concepts to the content under review.

Accounting for inconsistent terminology

An important complication affecting the abstraction process in methods overviews is that the language used by authors to describe methods-related concepts can easily vary across publications. For example, authors from different qualitative research traditions often use different terms for similar methods-related concepts. Furthermore, as we found in the sampling overview [ 18 ], there may be cases where no identifiable term, phrase, or label for a methods-related concept is used at all, and a description of it is given instead. This can make searching the text for relevant concepts based on keywords unreliable.

Principle #6:

Since accepted terms may not be used consistently to refer to methods concepts, it is necessary to rely on the definitions for concepts, rather than keywords, to identify relevant information in the publication to abstract.

Strategy #6:

An effective means to systematically identify relevant information is to develop and iteratively adjust written definitions for key concepts (corresponding to abstraction fields) that are consistent with and as inclusive of as much of the literature reviewed as possible. Reviewers then seek information that matches these definitions (rather than keywords) when scanning a publication for relevant data to abstract.

In the abstraction process for the sampling overview [ 18 ], we noted the several concepts of interest to the review for which abstraction by keyword was particularly problematic due to inconsistent terminology across publications: sampling , purposeful sampling , sampling strategy , and saturation (for examples, see Additional file 1 , Matrices 3a, 3b, 4). We iteratively developed definitions for these concepts by abstracting text from publications that either provided an explicit definition or from which an implicit definition could be derived, which was recorded in fields dedicated to the concept’s definition. Using a method of constant comparison, we used text from definition fields to inform and modify a centrally maintained definition of the corresponding concept to optimize its fit and inclusiveness with the literature reviewed. Table  1 shows, as an example, the final definition constructed in this way for one of the central concepts of the review, qualitative sampling .

We applied iteratively developed definitions when making decisions about what specific text to abstract for an existing field, which allowed us to abstract concept-relevant data even if no recognized keyword was used. For example, this was the case for the sampling-related concept, saturation , where the relevant text available for abstraction in one publication [ 26 ]—“to continue to collect data until nothing new was being observed or recorded, no matter how long that takes”—was not accompanied by any term or label whatsoever.

This comparative analytic strategy (and our approach to analysis more broadly as described in strategy #7, below) is analogous to the process of reciprocal translation —a technique first introduced for meta-ethnography by Noblit and Hare [ 27 ] that has since been recognized as a common element in a variety of qualitative metasynthesis approaches [ 28 ]. Reciprocal translation, taken broadly, involves making sense of a study’s findings in terms of the findings of the other studies included in the review. In practice, it has been operationalized in different ways. Melendez-Torres and colleagues developed a typology from their review of the metasynthesis literature, describing four overlapping categories of specific operations undertaken in reciprocal translation: visual representation, key paper integration, data reduction and thematic extraction, and line-by-line coding [ 28 ]. The approaches suggested in both strategies #6 and #7, with their emphasis on constant comparison, appear to fall within the line-by-line coding category.

Generating credible and verifiable analytic interpretations

The analysis in a systematic methods overview must support its more general objective, which we suggested above is often to offer clarity and enhance collective understanding regarding a chosen methods topic. In our experience, this involves describing and interpreting the relevant literature in qualitative terms. Furthermore, any interpretative analysis required may entail reaching different levels of abstraction, depending on the more specific objectives of the review. For example, in the overview on sampling [ 18 ], we aimed to produce a comparative analysis of how multiple sampling-related topics were treated differently within and among different qualitative research traditions. To promote credibility of the review, however, not only should one seek a qualitative analytic approach that facilitates reaching varying levels of abstraction but that approach must also ensure that abstract interpretations are supported and justified by the source data and not solely the product of the analyst’s speculative thinking.

Principle #7:

Considering the qualitative nature of the analysis required in systematic methods overviews, it is important to select an analytic method whose interpretations can be verified as being consistent with the literature selected, regardless of the level of abstraction reached.

Strategy #7:

We suggest employing the constant comparative method of analysis [ 29 ] because it supports developing and verifying analytic links to the source data throughout progressively interpretive or abstract levels. In applying this approach, we advise a rigorous approach, documenting how supportive quotes or references to the original texts are carried forward in the successive steps of analysis to allow for easy verification.

The analytic approach used in the methods overview on sampling [ 18 ] comprised four explicit steps, progressing in level of abstraction—data abstraction, matrices, narrative summaries, and final analytic conclusions (Fig.  2 ). While we have positioned data abstraction as the second stage of the generic review process (prior to Analysis), above, we also considered it as an initial step of analysis in the sampling overview for several reasons. First, it involved a process of constant comparisons and iterative decision-making about the fields to add or define during development and modification of the abstraction form, through which we established the range of concepts to be addressed in the review. At the same time, abstraction involved continuous analytic decisions about what textual quotes (ranging in size from short phrases to numerous paragraphs) to record in the fields thus created. This constant comparative process was analogous to open coding in which textual data from publications was compared to conceptual fields (equivalent to codes) or to other instances of data previously abstracted when constructing definitions to optimize their fit with the overall literature as described in strategy #6. Finally, in the data abstraction step, we also recorded our first interpretive thoughts in dedicated fields, providing initial material for the more abstract analytic steps.

Summary of progressive steps of analysis used in the methods overview on sampling [ 18 ]

In the second step of the analysis, we constructed topic-specific matrices , or tables, by copying relevant quotes from abstraction forms into the appropriate cells of matrices (for the complete set of analytic matrices developed in the sampling review, see Additional file 1 (matrices 3 to 10)). Each matrix ranged from one to five pages; row headings, nested three-deep, identified the methodological tradition, author, and publication, respectively; and column headings identified the concepts, which corresponded to abstraction fields. Matrices thus allowed us to make further comparisons across methodological traditions, and between authors within a tradition. In the third step of analysis, we recorded our comparative observations as narrative summaries , in which we used illustrative quotes more sparingly. In the final step, we developed analytic conclusions based on the narrative summaries about the sampling-related concepts within each methodological tradition for which clarity, consistency, or comprehensiveness of the available guidance appeared to be lacking. Higher levels of analysis thus built logically from the lower levels, enabling us to easily verify analytic conclusions by tracing the support for claims by comparing the original text of publications reviewed.

Integrative versus interpretive methods overviews

The analytic product of systematic methods overviews is comparable to qualitative evidence syntheses, since both involve describing and interpreting the relevant literature in qualitative terms. Most qualitative synthesis approaches strive to produce new conceptual understandings that vary in level of interpretation. Dixon-Woods and colleagues [ 30 ] elaborate on a useful distinction, originating from Noblit and Hare [ 27 ], between integrative and interpretive reviews. Integrative reviews focus on summarizing available primary data and involve using largely secure and well defined concepts to do so; definitions are used from an early stage to specify categories for abstraction (or coding) of data, which in turn supports their aggregation; they do not seek as their primary focus to develop or specify new concepts, although they may achieve some theoretical or interpretive functions. For interpretive reviews, meanwhile, the main focus is to develop new concepts and theories that integrate them, with the implication that the concepts developed become fully defined towards the end of the analysis. These two forms are not completely distinct, and “every integrative synthesis will include elements of interpretation, and every interpretive synthesis will include elements of aggregation of data” [ 30 ].

The example methods overview on sampling [ 18 ] could be classified as predominantly integrative because its primary goal was to aggregate influential authors’ ideas on sampling-related concepts; there were also, however, elements of interpretive synthesis since it aimed to develop new ideas about where clarity in guidance on certain sampling-related topics is lacking, and definitions for some concepts were flexible and not fixed until late in the review. We suggest that most systematic methods overviews will be classifiable as predominantly integrative (aggregative). Nevertheless, more highly interpretive methods overviews are also quite possible—for example, when the review objective is to provide a highly critical analysis for the purpose of generating new methodological guidance. In such cases, reviewers may need to sample more deeply (see strategy #4), specifically by selecting empirical research reports (i.e., to go beyond dominant or influential ideas in the methods literature) that are likely to feature innovations or instructive lessons in employing a given method.

In this paper, we have outlined tentative guidance in the form of seven principles and strategies on how to conduct systematic methods overviews, a review type in which methods-relevant literature is systematically analyzed with the aim of offering clarity and enhancing collective understanding regarding a specific methods topic. Our proposals include strategies for delimiting the set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology, and generating credible and verifiable analytic interpretations. We hope the suggestions proposed will be useful to others undertaking reviews on methods topics in future.

As far as we are aware, this is the first published source of concrete guidance for conducting this type of review. It is important to note that our primary objective was to initiate methodological discussion by stimulating reflection on what rigorous methods for this type of review should look like, leaving the development of more complete guidance to future work. While derived from the experience of reviewing a single qualitative methods topic, we believe the principles and strategies provided are generalizable to overviews of both qualitative and quantitative methods topics alike. However, it is expected that additional challenges and insights for conducting such reviews have yet to be defined. Thus, we propose that next steps for developing more definitive guidance should involve an attempt to collect and integrate other reviewers’ perspectives and experiences in conducting systematic methods overviews on a broad range of qualitative and quantitative methods topics. Formalized guidance and standards would improve the quality of future methods overviews, something we believe has important implications for advancing qualitative and quantitative methodology. When undertaken to a high standard, rigorous critical evaluations of the available methods guidance have significant potential to make implicit controversies explicit, and improve the clarity and precision of our understandings of problematic qualitative or quantitative methods issues.

A review process central to most types of rigorous reviews of empirical studies, which we did not explicitly address in a separate review step above, is quality appraisal . The reason we have not treated this as a separate step stems from the different objectives of the primary publications included in overviews of the methods literature (i.e., providing methodological guidance) compared to the primary publications included in the other established review types (i.e., reporting findings from single empirical studies). This is not to say that appraising quality of the methods literature is not an important concern for systematic methods overviews. Rather, appraisal is much more integral to (and difficult to separate from) the analysis step, in which we advocate appraising clarity, consistency, and comprehensiveness—the quality appraisal criteria that we suggest are appropriate for the methods literature. As a second important difference regarding appraisal, we currently advocate appraising the aforementioned aspects at the level of the literature in aggregate rather than at the level of individual publications. One reason for this is that methods guidance from individual publications generally builds on previous literature, and thus we feel that ahistorical judgments about comprehensiveness of single publications lack relevance and utility. Additionally, while different methods authors may express themselves less clearly than others, their guidance can nonetheless be highly influential and useful, and should therefore not be downgraded or ignored based on considerations of clarity—which raises questions about the alternative uses that quality appraisals of individual publications might have. Finally, legitimate variability in the perspectives that methods authors wish to emphasize, and the levels of generality at which they write about methods, makes critiquing individual publications based on the criterion of clarity a complex and potentially problematic endeavor that is beyond the scope of this paper to address. By appraising the current state of the literature at a holistic level, reviewers stand to identify important gaps in understanding that represent valuable opportunities for further methodological development.

To summarize, the principles and strategies provided here may be useful to those seeking to undertake their own systematic methods overview. Additional work is needed, however, to establish guidance that is comprehensive by comparing the experiences from conducting a variety of methods overviews on a range of methods topics. Efforts that further advance standards for systematic methods overviews have the potential to promote high-quality critical evaluations that produce conceptually clear and unified understandings of problematic methods topics, thereby accelerating the advance of research methodology.

Hutton JL, Ashcroft R. What does “systematic” mean for reviews of methods? In: Black N, Brazier J, Fitzpatrick R, Reeves B, editors. Health services research methods: a guide to best practice. London: BMJ Publishing Group; 1998. p. 249–54.

Google Scholar  

Cochrane handbook for systematic reviews of interventions. In. Edited by Higgins JPT, Green S, Version 5.1.0 edn: The Cochrane Collaboration; 2011.

Centre for Reviews and Dissemination: Systematic reviews: CRD’s guidance for undertaking reviews in health care . York: Centre for Reviews and Dissemination; 2009.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700–0.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.

Article   PubMed   PubMed Central   Google Scholar  

Kastner M, Tricco AC, Soobiah C, Lillie E, Perrier L, Horsley T, Welch V, Cogo E, Antony J, Straus SE. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol. 2012;12(1):1–1.

Article   Google Scholar  

Booth A, Noyes J, Flemming K, Gerhardus A. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions. In: Integrate-HTA. 2016.

Booth A, Sutton A, Papaioannou D. Systematic approaches to successful literature review. 2nd ed. London: Sage; 2016.

Hannes K, Lockwood C. Synthesizing qualitative research: choosing the right approach. Chichester: Wiley-Blackwell; 2012.

Suri H. Towards methodologically inclusive research syntheses: expanding possibilities. New York: Routledge; 2014.

Campbell M, Egan M, Lorenc T, Bond L, Popham F, Fenton C, Benzeval M. Considering methodological options for reviews of theory: illustrated by a review of theories linking income and health. Syst Rev. 2014;3(1):1–11.

Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med. 2008;6(4):331–9.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reportingqualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Article   PubMed   Google Scholar  

Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.

Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365(9465):1159–62.

Alshurafa M, Briel M, Akl EA, Haines T, Moayyedi P, Gentles SJ, Rios L, Tran C, Bhatnagar N, Lamontagne F, et al. Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature. PLoS One. 2012;7(11):e49163.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Gentles SJ, Charles C, Ploeg J, McKibbon KA. Sampling in qualitative research: insights from an overview of the methods literature. Qual Rep. 2015;20(11):1772–89.

Harzing A-W, Alakangas S. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics. 2016;106(2):787–804.

Harzing A-WK, van der Wal R. Google Scholar as a new source for citation analysis. Ethics Sci Environ Polit. 2008;8(1):61–73.

Kousha K, Thelwall M. Google Scholar citations and Google Web/URL citations: a multi‐discipline exploratory analysis. J Assoc Inf Sci Technol. 2007;58(7):1055–65.

Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102(46):16569–72.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Quality Safety. 2015;24(11):700–8.

Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13(1):37.

Carroll C, Booth A, Cooper K. A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;11(1):29.

Cohen MZ, Kahn DL, Steeves DL. Hermeneutic phenomenological research: a practical guide for nurse researchers. Thousand Oaks: Sage; 2000.

Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park: Sage; 1988.

Book   Google Scholar  

Melendez-Torres GJ, Grant S, Bonell C. A systematic review and critical appraisal of qualitative metasynthetic practice in public health to develop a taxonomy of operations of reciprocal translation. Res Synthesis Methods. 2015;6(4):357–71.

Article   CAS   Google Scholar  

Glaser BG, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967.

Dixon-Woods M, Agarwal S, Young B, Jones D, Sutton A. Integrative approaches to qualitative and quantitative evidence. In: UK National Health Service. 2004. p. 1–44.

Download references

Acknowledgements

Not applicable.

There was no funding for this work.

Availability of data and materials

The systematic methods overview used as a worked example in this article (Gentles SJ, Charles C, Ploeg J, McKibbon KA: Sampling in qualitative research: insights from an overview of the methods literature. The Qual Rep 2015, 20(11):1772-1789) is available from http://nsuworks.nova.edu/tqr/vol20/iss11/5 .

Authors’ contributions

SJG wrote the first draft of this article, with CC contributing to drafting. All authors contributed to revising the manuscript. All authors except CC (deceased) approved the final draft. SJG, CC, KAB, and JP were involved in developing methods for the systematic methods overview on sampling.

Authors’ information

Competing interests.

The authors declare that they have no competing interests.

Consent for publication

Ethics approval and consent to participate, author information, authors and affiliations.

Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada

Stephen J. Gentles, Cathy Charles & K. Ann McKibbon

Faculty of Social Work, University of Calgary, Alberta, Canada

David B. Nicholas

School of Nursing, McMaster University, Hamilton, Ontario, Canada

Jenny Ploeg

CanChild Centre for Childhood Disability Research, McMaster University, 1400 Main Street West, IAHS 408, Hamilton, ON, L8S 1C7, Canada

Stephen J. Gentles

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Stephen J. Gentles .

Additional information

Cathy Charles is deceased

Additional file

Additional file 1:.

Submitted: Analysis_matrices. (DOC 330 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gentles, S.J., Charles, C., Nicholas, D.B. et al. Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research. Syst Rev 5 , 172 (2016). https://doi.org/10.1186/s13643-016-0343-0

Download citation

Received : 06 June 2016

Accepted : 14 September 2016

Published : 11 October 2016

DOI : https://doi.org/10.1186/s13643-016-0343-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic review
• Literature selection
• Research methods
• Research methodology
• Overview of methods
• Systematic methods overview
• Review methods

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Insights blog

Method articles

Share methodological innovations, what is a method article .

A method article is a medium length peer-reviewed, research-focused article type that aims to answer a specific question. It also describes an advancement or development of current methodological approaches and research procedures (akin to a research article), following the standard layout for research articles.

This includes new study methods, substantive modifications to existing methods, or innovative applications of existing methods to new models or scientific questions. These should include adequate and appropriate validation to be considered, and any datasets associated with the paper must publish all experimental controls and make full datasets available.   

What are the benefits of publishing a method article?

Maximize the potential of your research by increasing the transparency, accessibility and reproducibility of your research by sharing innovations and new ways of approaching research questions.

Gain credit for your methodological innovations with a citable publication.

You can upload your method steps to protocols.io to increase the findability of your research.

How do I write a method article?

Method articles are medium length (2500-4000 words) article types following the standard layout for research articles and should include:

The abstract, introduction, method, results, and conclusions and/or discussion, within the main body of the article

A data availability statement

Author contributions

Funding/grant information

Any supplementary materials and appendices

Authors submitting methodology articles have the option to share their method steps on  protocols.io . Please note, this is not required for submission but is encouraged. 

Method article template

Here is a downloadable method article template to help you write and prepare your method article for submission: F1000Research method note template

Publishing your method article with F1000Research

F1000Research  enables you to publish your method article rapidly and openly via an author-centric platform. You’ll benefit from:

Open access publication  – for maximum visibility of your method article.

Fast publication times  – your method article will be published online with a DOI following our pre-publication checks in as little as 14 days.

Open data  – All submissions to F1000Research must comply with our open data policy and support FAIR principles.

Open peer review  – Once published, your data note will undergo a robust open peer review process where reviewers are asked specific questions about the description, methodology and availability of the data.

Please read the  guide to publishing a method article with F1000Research  and the  method article submission template , which makes it quick and easy to write your method article for submission for F1000Research.

Taylor & Francis journals

The Taylor & Francis journal portfolio offers a growing range of options for publishing method articles. To maintain quality and provide validation of the methodology and supporting information, method articles will be peer reviewed in line with the relevant submission policy. 

Below is our list of journals for publishing your method article. You can also submit a method article to Dove Medical Press journals .

Life, Earth & Environmental Sciences Please read the aims and scope of the journal to make sure its the right fit for your research

Acta Agriculturae Scandinavica, Section B — Soil & Plant Science

All Life Methods

Annals of GIS

Aquatic Insects

Archives Of Phytopathology And Plant Protection

Arid Land Research And Management

Biocatalysis and Biotransformation

Biological Rhythm Research

Bioremediation Journal

Biotechnology and Biotechnological Equipment

Canadian Journal of Plant Pathology

Carbon Management

Cell Adhesion & Migration

Communicative & Integrative Biology

Compost Science & Utilization

Epigenetics Reports

GIScience & Remote Sensing

International Journal Of Pest Management

International Journal of Sustainable Development & World Ecology

Journal of Integrative Environmental Sciences

Journal of Essential Oil Research

Journal of Herbs, Spices & Medicinal Plants

Journal of Land Use Science

Lake and Reservoir Management

Mitochondrial DNA part B

Physical Geography

Plant Signaling & Behavior

Small GTPases

Soil & Sediment Contamination: An International Journal

Studies On Neotropical Fauna And Environment

Sustainable Environment

Physical Sciences & Engineering Please read the aims and scope of the journal to make sure that it’s the right fit for your research

Aerosol Science and Technology

American Journal of Mathematical and Management Sciences

Biostatistics & Epidemiology

Brain-Computer Interfaces

Combustion Science and Technology

Composite Interfaces

Computer Methods In Biomechanics & Biomedical Engineering

Computer Methods In Biomechanics and Biomedical Engineering: Imaging & Visualization

Econometric Reviews

Experimental Heat Transfer

Enterprise Information Systems

Fullerenes, Nanotubes and Carbon Nanostructures

Georisk: Assessment and Management of Risk for Engineered Systems and Geohazards

Information Systems Management

International Journal of Environmental Analytical Chemistry

International Biomechanics

International Journal of Computer Mathematics

International Journal of Computer Mathematics: Computer Systems Theory

International Journal of Electronics

International Journal of General Systems

International Journal of Polymer Analysis and Characterization

International Journal of Sustainable Engineering

In ternational Journal of Systems Science: Operations & Logistics

Isotopes In Environmental and Health Studies

The Journal of Adhesion

Journal of Adhesion Science and Technology

Journal of the American Society of Brewing Chemists

Journal of Dispersion Science and Technology

Journal of Biological Dynamics

Journal of Electromagnetic Waves and Applications

Journal of Medical Engineering & Technology

Journal of Intelligent Transportation Systems

Journal of Location Based Services

Journal of Organizational Computing and Electronic Commerce

Journal of Wood Chemistry and Technology

Journal of Thermal Stresses

Machining Science and Technology

Materials Research Innovations

Materials Research Letters

Mechanics Based Design of Structures and Machines

Mechanics of Advanced Materials and Structures

Nanoscale and Microscale Thermophysical Engineering

Phase Transitions

Non-Destructive Testing and Evaluation

Nucleosides, Nucleotides and Nucleic Acids

Numerical Heat Transfer, Part A: Applications

Numerical Heat Transfer, Part B: Fundamentals

Petroleum Science And Technology

Polycyclic Aromatic Compounds

Production Planning & Control

Research in Mathematics

Separation Science and Technology

Solvent Extraction and Ion Exchange

Stochastics: An International Journal Of Probability & Stochastic Processes

Supramolecular Chemistry

Medicine & Health Please read the aims and scope of the journal to make sure that it’s the right fit for your research

Annals of Human Biology

Archives Of Physiology and Biochemistry

Artificial Cells, Nanomedicine, and Biotechnology

Autoimmunity

Behavioral Sleep Medicine

Children’s Health Care

Clinical Linguistics & Phonetics

Clinical Toxicology

Cutaneous and Ocular Toxicology

Drug and Chemical Toxicology

Drug Delivery

Drug Development and Industrial Pharmacy

Electromagnetic Biology and Medicine

Endocrine Research

Expert Opinion on Orphan Drugs

Fetal and Pediatric Pathology

Growth Factors

Health, Risk & Society

HIV Research and Clinical Practice

Home Health Care Services Quarterly

Human Vaccines & Immunotherapeutics

Immunological Investigations

Immunopharmacology and Immunotoxicology

Inhalation Toxicology

International Journal of Healthcare Management

International Journal of Neuroscience

International Reviews in Immunology

Issues in Mental Health Nursing

Journal of the American Nutrition Association

Journal of Dermatological Treatment

Journal of Dietary Supplements

Journal of Drug Targeting

Journal of Enzyme Inhibition and Medicinal Chemistry

Journal of HIV/AIDS and Social Services

Journal of Immunotoxicology

Journal of Interprofessional Care

Journal of Liposome Research

Journal of Manual & Manipulative Therapy

Journal of Microencapsulation

Journal of Nutrition and Gerontology and Geriatrics

Journal of Prevention & Intervention in the Community

Journal of Oral Microbiology

Journal of Receptors and Signal Transduction

Medical Teacher

Nanotoxicology

Nutrition and Cancer

Occupational Therapy in Mental Health

Ophthalmic Genetics

Pathogens and Global Health

Pediatric Hematology & Oncology

Pharmaceutical Biology

Physical & Occupational Therapy In Geriatrics

Physical Therapy Reviews

Psychology, Health & Medicine

Seminars In Ophthalmology

Teaching and Learning in Medicine

Tissue Barriers

Toxicology Communications

Toxicology Mechanisms and Methods

Ultrastructural Pathology

Vulnerable Children and Youth Studies

Behavioral Science & Education Please read the aims and scope of the journal to make sure it’s a good fit for your research

Activities, Adaptation & Aging

Assessment & Evaluation in Higher Education

Dynamics of Asymmetric Conflict

Education 3-13

Educational Research and Evaluation

Educational Review

Irish Educational Studies

European Journal of Psychotraumatology

Health Psychology and Behavioral Medicine

Health Psychology Review

International Gambling Studies

International Journal of Bilingual Education & Bilingualism

International Journal of Early Years Education

International Journal of Adolescence and Youth

Journal of Aging & Social Policy

Journal of Child Sexual Abuse

Journal of Elder Abuse & Neglect

Journal of Evidence-Based Social Work

Journal of Geography

Journal of the History of the Neurosciences

Journal of Intergenerational Relationships

Journal of Loss and Trauma

Journal of Moral Education

Journal of Sex & Marital Therapy

Journal of School Violence

Measurement and Evaluation in Counseling and Development

Metaphor and Symbol

Military Psychology

School Psychology Review

The Journal of Positive Psychology

The Journal of Genetic Psychology

Social Sciences Please read the aims and scope of the journal to make sure it’s a good fit for your research

China Economic Journal

International Journal of Advertising

The Chinese Economy

Journal of Applied Economics

Journal of Economic Policy Reform

Political Science

Humanities, Media & Arts Please read the aims and scope of the journal to make sure it’s a good fit for your research

Critical Discourse Studies

Journal of Global Information Technology Management

Journal of Information Technology Case and Application Research

The Information Society

The journal Ecology of Food and Nutrition

Further resources

Article types.

Guide to different types of articles

Registered reports

Publishing tips, direct to your inbox

Expert tips and guidance on getting published and maximizing the impact of your research. Register now for weekly insights direct to your inbox.

• Search Menu
• Sign in through your institution
• Advance articles
• Editor's Choice
• Supplements
• French Abstracts
• Portuguese Abstracts
• Spanish Abstracts
• Author Guidelines
• Submission Site
• Open Access
• About International Journal for Quality in Health Care
• About the International Society for Quality in Health Care
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Contact ISQua
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Primacy of the research question, structure of the paper, writing a research article: advice to beginners.

• Article contents
• Figures & tables
• Supplementary Data

Thomas V. Perneger, Patricia M. Hudelson, Writing a research article: advice to beginners, International Journal for Quality in Health Care , Volume 16, Issue 3, June 2004, Pages 191–192, https://doi.org/10.1093/intqhc/mzh053

• Permissions Icon Permissions

Writing research papers does not come naturally to most of us. The typical research paper is a highly codified rhetorical form [ 1 , 2 ]. Knowledge of the rules—some explicit, others implied—goes a long way toward writing a paper that will get accepted in a peer-reviewed journal.

A good research paper addresses a specific research question. The research question—or study objective or main research hypothesis—is the central organizing principle of the paper. Whatever relates to the research question belongs in the paper; the rest doesn’t. This is perhaps obvious when the paper reports on a well planned research project. However, in applied domains such as quality improvement, some papers are written based on projects that were undertaken for operational reasons, and not with the primary aim of producing new knowledge. In such cases, authors should define the main research question a posteriori and design the paper around it.

Generally, only one main research question should be addressed in a paper (secondary but related questions are allowed). If a project allows you to explore several distinct research questions, write several papers. For instance, if you measured the impact of obtaining written consent on patient satisfaction at a specialized clinic using a newly developed questionnaire, you may want to write one paper on the questionnaire development and validation, and another on the impact of the intervention. The idea is not to split results into ‘least publishable units’, a practice that is rightly decried, but rather into ‘optimally publishable units’.

What is a good research question? The key attributes are: (i) specificity; (ii) originality or novelty; and (iii) general relevance to a broad scientific community. The research question should be precise and not merely identify a general area of inquiry. It can often (but not always) be expressed in terms of a possible association between X and Y in a population Z, for example ‘we examined whether providing patients about to be discharged from the hospital with written information about their medications would improve their compliance with the treatment 1 month later’. A study does not necessarily have to break completely new ground, but it should extend previous knowledge in a useful way, or alternatively refute existing knowledge. Finally, the question should be of interest to others who work in the same scientific area. The latter requirement is more challenging for those who work in applied science than for basic scientists. While it may safely be assumed that the human genome is the same worldwide, whether the results of a local quality improvement project have wider relevance requires careful consideration and argument.

Once the research question is clearly defined, writing the paper becomes considerably easier. The paper will ask the question, then answer it. The key to successful scientific writing is getting the structure of the paper right. The basic structure of a typical research paper is the sequence of Introduction, Methods, Results, and Discussion (sometimes abbreviated as IMRAD). Each section addresses a different objective. The authors state: (i) the problem they intend to address—in other terms, the research question—in the Introduction; (ii) what they did to answer the question in the Methods section; (iii) what they observed in the Results section; and (iv) what they think the results mean in the Discussion.

In turn, each basic section addresses several topics, and may be divided into subsections (Table 1 ). In the Introduction, the authors should explain the rationale and background to the study. What is the research question, and why is it important to ask it? While it is neither necessary nor desirable to provide a full-blown review of the literature as a prelude to the study, it is helpful to situate the study within some larger field of enquiry. The research question should always be spelled out, and not merely left for the reader to guess.

Typical structure of a research paper

The Methods section should provide the readers with sufficient detail about the study methods to be able to reproduce the study if so desired. Thus, this section should be specific, concrete, technical, and fairly detailed. The study setting, the sampling strategy used, instruments, data collection methods, and analysis strategies should be described. In the case of qualitative research studies, it is also useful to tell the reader which research tradition the study utilizes and to link the choice of methodological strategies with the research goals [ 3 ].

The Results section is typically fairly straightforward and factual. All results that relate to the research question should be given in detail, including simple counts and percentages. Resist the temptation to demonstrate analytic ability and the richness of the dataset by providing numerous tables of non-essential results.

The Discussion section allows the most freedom. This is why the Discussion is the most difficult to write, and is often the weakest part of a paper. Structured Discussion sections have been proposed by some journal editors [ 4 ]. While strict adherence to such rules may not be necessary, following a plan such as that proposed in Table 1 may help the novice writer stay on track.

References should be used wisely. Key assertions should be referenced, as well as the methods and instruments used. However, unless the paper is a comprehensive review of a topic, there is no need to be exhaustive. Also, references to unpublished work, to documents in the grey literature (technical reports), or to any source that the reader will have difficulty finding or understanding should be avoided.

Having the structure of the paper in place is a good start. However, there are many details that have to be attended to while writing. An obvious recommendation is to read, and follow, the instructions to authors published by the journal (typically found on the journal’s website). Another concerns non-native writers of English: do have a native speaker edit the manuscript. A paper usually goes through several drafts before it is submitted. When revising a paper, it is useful to keep an eye out for the most common mistakes (Table 2 ). If you avoid all those, your paper should be in good shape.

Common mistakes seen in manuscripts submitted to this journal

Huth EJ . How to Write and Publish Papers in the Medical Sciences , 2nd edition. Baltimore, MD: Williams & Wilkins, 1990 .

Browner WS . Publishing and Presenting Clinical Research . Baltimore, MD: Lippincott, Williams & Wilkins, 1999 .

Devers KJ , Frankel RM. Getting qualitative research published. Educ Health 2001 ; 14 : 109 –117.

Docherty M , Smith R. The case for structuring the discussion of scientific papers. Br Med J 1999 ; 318 : 1224 –1225.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1464-3677
• Print ISSN 1353-4505
• Copyright © 2024 International Society for Quality in Health Care and Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Privacy Policy

Home » Research Methodology – Types, Examples and writing Guide

Research Methodology – Types, Examples and writing Guide

Table of Contents

Research Methodology

Definition:

Research Methodology refers to the systematic and scientific approach used to conduct research, investigate problems, and gather data and information for a specific purpose. It involves the techniques and procedures used to identify, collect , analyze , and interpret data to answer research questions or solve research problems . Moreover, They are philosophical and theoretical frameworks that guide the research process.

Structure of Research Methodology

Research methodology formats can vary depending on the specific requirements of the research project, but the following is a basic example of a structure for a research methodology section:

I. Introduction

• Provide an overview of the research problem and the need for a research methodology section
• Outline the main research questions and objectives

II. Research Design

• Explain the research design chosen and why it is appropriate for the research question(s) and objectives
• Discuss any alternative research designs considered and why they were not chosen
• Describe the research setting and participants (if applicable)

III. Data Collection Methods

• Describe the methods used to collect data (e.g., surveys, interviews, observations)
• Explain how the data collection methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or instruments used for data collection

IV. Data Analysis Methods

• Describe the methods used to analyze the data (e.g., statistical analysis, content analysis )
• Explain how the data analysis methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or software used for data analysis

V. Ethical Considerations

• Discuss any ethical issues that may arise from the research and how they were addressed
• Explain how informed consent was obtained (if applicable)
• Detail any measures taken to ensure confidentiality and anonymity

VI. Limitations

• Identify any potential limitations of the research methodology and how they may impact the results and conclusions

VII. Conclusion

• Summarize the key aspects of the research methodology section
• Explain how the research methodology addresses the research question(s) and objectives

Research Methodology Types

Types of Research Methodology are as follows:

Quantitative Research Methodology

This is a research methodology that involves the collection and analysis of numerical data using statistical methods. This type of research is often used to study cause-and-effect relationships and to make predictions.

Qualitative Research Methodology

This is a research methodology that involves the collection and analysis of non-numerical data such as words, images, and observations. This type of research is often used to explore complex phenomena, to gain an in-depth understanding of a particular topic, and to generate hypotheses.

Mixed-Methods Research Methodology

This is a research methodology that combines elements of both quantitative and qualitative research. This approach can be particularly useful for studies that aim to explore complex phenomena and to provide a more comprehensive understanding of a particular topic.

Case Study Research Methodology

This is a research methodology that involves in-depth examination of a single case or a small number of cases. Case studies are often used in psychology, sociology, and anthropology to gain a detailed understanding of a particular individual or group.

Action Research Methodology

This is a research methodology that involves a collaborative process between researchers and practitioners to identify and solve real-world problems. Action research is often used in education, healthcare, and social work.

Experimental Research Methodology

This is a research methodology that involves the manipulation of one or more independent variables to observe their effects on a dependent variable. Experimental research is often used to study cause-and-effect relationships and to make predictions.

Survey Research Methodology

This is a research methodology that involves the collection of data from a sample of individuals using questionnaires or interviews. Survey research is often used to study attitudes, opinions, and behaviors.

Grounded Theory Research Methodology

This is a research methodology that involves the development of theories based on the data collected during the research process. Grounded theory is often used in sociology and anthropology to generate theories about social phenomena.

Research Methodology Example

An Example of Research Methodology could be the following:

Research Methodology for Investigating the Effectiveness of Cognitive Behavioral Therapy in Reducing Symptoms of Depression in Adults

Introduction:

The aim of this research is to investigate the effectiveness of cognitive-behavioral therapy (CBT) in reducing symptoms of depression in adults. To achieve this objective, a randomized controlled trial (RCT) will be conducted using a mixed-methods approach.

Research Design:

The study will follow a pre-test and post-test design with two groups: an experimental group receiving CBT and a control group receiving no intervention. The study will also include a qualitative component, in which semi-structured interviews will be conducted with a subset of participants to explore their experiences of receiving CBT.

Participants:

Participants will be recruited from community mental health clinics in the local area. The sample will consist of 100 adults aged 18-65 years old who meet the diagnostic criteria for major depressive disorder. Participants will be randomly assigned to either the experimental group or the control group.

Intervention :

The experimental group will receive 12 weekly sessions of CBT, each lasting 60 minutes. The intervention will be delivered by licensed mental health professionals who have been trained in CBT. The control group will receive no intervention during the study period.

Data Collection:

Quantitative data will be collected through the use of standardized measures such as the Beck Depression Inventory-II (BDI-II) and the Generalized Anxiety Disorder-7 (GAD-7). Data will be collected at baseline, immediately after the intervention, and at a 3-month follow-up. Qualitative data will be collected through semi-structured interviews with a subset of participants from the experimental group. The interviews will be conducted at the end of the intervention period, and will explore participants’ experiences of receiving CBT.

Data Analysis:

Quantitative data will be analyzed using descriptive statistics, t-tests, and mixed-model analyses of variance (ANOVA) to assess the effectiveness of the intervention. Qualitative data will be analyzed using thematic analysis to identify common themes and patterns in participants’ experiences of receiving CBT.

Ethical Considerations:

This study will comply with ethical guidelines for research involving human subjects. Participants will provide informed consent before participating in the study, and their privacy and confidentiality will be protected throughout the study. Any adverse events or reactions will be reported and managed appropriately.

Data Management:

All data collected will be kept confidential and stored securely using password-protected databases. Identifying information will be removed from qualitative data transcripts to ensure participants’ anonymity.

Limitations:

One potential limitation of this study is that it only focuses on one type of psychotherapy, CBT, and may not generalize to other types of therapy or interventions. Another limitation is that the study will only include participants from community mental health clinics, which may not be representative of the general population.

Conclusion:

This research aims to investigate the effectiveness of CBT in reducing symptoms of depression in adults. By using a randomized controlled trial and a mixed-methods approach, the study will provide valuable insights into the mechanisms underlying the relationship between CBT and depression. The results of this study will have important implications for the development of effective treatments for depression in clinical settings.

How to Write Research Methodology

Writing a research methodology involves explaining the methods and techniques you used to conduct research, collect data, and analyze results. It’s an essential section of any research paper or thesis, as it helps readers understand the validity and reliability of your findings. Here are the steps to write a research methodology:

• Start by explaining your research question: Begin the methodology section by restating your research question and explaining why it’s important. This helps readers understand the purpose of your research and the rationale behind your methods.
• Describe your research design: Explain the overall approach you used to conduct research. This could be a qualitative or quantitative research design, experimental or non-experimental, case study or survey, etc. Discuss the advantages and limitations of the chosen design.
• Discuss your sample: Describe the participants or subjects you included in your study. Include details such as their demographics, sampling method, sample size, and any exclusion criteria used.
• Describe your data collection methods : Explain how you collected data from your participants. This could include surveys, interviews, observations, questionnaires, or experiments. Include details on how you obtained informed consent, how you administered the tools, and how you minimized the risk of bias.
• Explain your data analysis techniques: Describe the methods you used to analyze the data you collected. This could include statistical analysis, content analysis, thematic analysis, or discourse analysis. Explain how you dealt with missing data, outliers, and any other issues that arose during the analysis.
• Discuss the validity and reliability of your research : Explain how you ensured the validity and reliability of your study. This could include measures such as triangulation, member checking, peer review, or inter-coder reliability.
• Acknowledge any limitations of your research: Discuss any limitations of your study, including any potential threats to validity or generalizability. This helps readers understand the scope of your findings and how they might apply to other contexts.
• Provide a summary: End the methodology section by summarizing the methods and techniques you used to conduct your research. This provides a clear overview of your research methodology and helps readers understand the process you followed to arrive at your findings.

When to Write Research Methodology

Research methodology is typically written after the research proposal has been approved and before the actual research is conducted. It should be written prior to data collection and analysis, as it provides a clear roadmap for the research project.

The research methodology is an important section of any research paper or thesis, as it describes the methods and procedures that will be used to conduct the research. It should include details about the research design, data collection methods, data analysis techniques, and any ethical considerations.

The methodology should be written in a clear and concise manner, and it should be based on established research practices and standards. It is important to provide enough detail so that the reader can understand how the research was conducted and evaluate the validity of the results.

Applications of Research Methodology

Here are some of the applications of research methodology:

• To identify the research problem: Research methodology is used to identify the research problem, which is the first step in conducting any research.
• To design the research: Research methodology helps in designing the research by selecting the appropriate research method, research design, and sampling technique.
• To collect data: Research methodology provides a systematic approach to collect data from primary and secondary sources.
• To analyze data: Research methodology helps in analyzing the collected data using various statistical and non-statistical techniques.
• To test hypotheses: Research methodology provides a framework for testing hypotheses and drawing conclusions based on the analysis of data.
• To generalize findings: Research methodology helps in generalizing the findings of the research to the target population.
• To develop theories : Research methodology is used to develop new theories and modify existing theories based on the findings of the research.
• To evaluate programs and policies : Research methodology is used to evaluate the effectiveness of programs and policies by collecting data and analyzing it.
• To improve decision-making: Research methodology helps in making informed decisions by providing reliable and valid data.

Purpose of Research Methodology

Research methodology serves several important purposes, including:

• To guide the research process: Research methodology provides a systematic framework for conducting research. It helps researchers to plan their research, define their research questions, and select appropriate methods and techniques for collecting and analyzing data.
• To ensure research quality: Research methodology helps researchers to ensure that their research is rigorous, reliable, and valid. It provides guidelines for minimizing bias and error in data collection and analysis, and for ensuring that research findings are accurate and trustworthy.
• To replicate research: Research methodology provides a clear and detailed account of the research process, making it possible for other researchers to replicate the study and verify its findings.
• To advance knowledge: Research methodology enables researchers to generate new knowledge and to contribute to the body of knowledge in their field. It provides a means for testing hypotheses, exploring new ideas, and discovering new insights.
• To inform decision-making: Research methodology provides evidence-based information that can inform policy and decision-making in a variety of fields, including medicine, public health, education, and business.

Advantages of Research Methodology

Research methodology has several advantages that make it a valuable tool for conducting research in various fields. Here are some of the key advantages of research methodology:

• Systematic and structured approach : Research methodology provides a systematic and structured approach to conducting research, which ensures that the research is conducted in a rigorous and comprehensive manner.
• Objectivity : Research methodology aims to ensure objectivity in the research process, which means that the research findings are based on evidence and not influenced by personal bias or subjective opinions.
• Replicability : Research methodology ensures that research can be replicated by other researchers, which is essential for validating research findings and ensuring their accuracy.
• Reliability : Research methodology aims to ensure that the research findings are reliable, which means that they are consistent and can be depended upon.
• Validity : Research methodology ensures that the research findings are valid, which means that they accurately reflect the research question or hypothesis being tested.
• Efficiency : Research methodology provides a structured and efficient way of conducting research, which helps to save time and resources.
• Flexibility : Research methodology allows researchers to choose the most appropriate research methods and techniques based on the research question, data availability, and other relevant factors.
• Scope for innovation: Research methodology provides scope for innovation and creativity in designing research studies and developing new research techniques.

Research Methodology Vs Research Methods

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Limitations in Research – Types, Examples and...

Theoretical Framework – Types, Examples and...

Research Objectives – Types, Examples and...

Research Paper Outline – Types, Example, Template

Research Methods – Types, Examples and Guide

Data Analysis – Process, Methods and Types

What is Research Methodology? Definition, Types, and Examples

Research methodology 1,2 is a structured and scientific approach used to collect, analyze, and interpret quantitative or qualitative data to answer research questions or test hypotheses. A research methodology is like a plan for carrying out research and helps keep researchers on track by limiting the scope of the research. Several aspects must be considered before selecting an appropriate research methodology, such as research limitations and ethical concerns that may affect your research.

The research methodology section in a scientific paper describes the different methodological choices made, such as the data collection and analysis methods, and why these choices were selected. The reasons should explain why the methods chosen are the most appropriate to answer the research question. A good research methodology also helps ensure the reliability and validity of the research findings. There are three types of research methodology—quantitative, qualitative, and mixed-method, which can be chosen based on the research objectives.

What is research methodology ?

A research methodology describes the techniques and procedures used to identify and analyze information regarding a specific research topic. It is a process by which researchers design their study so that they can achieve their objectives using the selected research instruments. It includes all the important aspects of research, including research design, data collection methods, data analysis methods, and the overall framework within which the research is conducted. While these points can help you understand what is research methodology, you also need to know why it is important to pick the right methodology.

Having a good research methodology in place has the following advantages: 3

• Helps other researchers who may want to replicate your research; the explanations will be of benefit to them.
• You can easily answer any questions about your research if they arise at a later stage.
• A research methodology provides a framework and guidelines for researchers to clearly define research questions, hypotheses, and objectives.
• It helps researchers identify the most appropriate research design, sampling technique, and data collection and analysis methods.
• A sound research methodology helps researchers ensure that their findings are valid and reliable and free from biases and errors.
• It also helps ensure that ethical guidelines are followed while conducting research.
• A good research methodology helps researchers in planning their research efficiently, by ensuring optimum usage of their time and resources.

Writing the methods section of a research paper? Let Paperpal help you achieve perfection  

Types of research methodology.

There are three types of research methodology based on the type of research and the data required. 1

• Quantitative research methodology focuses on measuring and testing numerical data. This approach is good for reaching a large number of people in a short amount of time. This type of research helps in testing the causal relationships between variables, making predictions, and generalizing results to wider populations.
• Qualitative research methodology examines the opinions, behaviors, and experiences of people. It collects and analyzes words and textual data. This research methodology requires fewer participants but is still more time consuming because the time spent per participant is quite large. This method is used in exploratory research where the research problem being investigated is not clearly defined.
• Mixed-method research methodology uses the characteristics of both quantitative and qualitative research methodologies in the same study. This method allows researchers to validate their findings, verify if the results observed using both methods are complementary, and explain any unexpected results obtained from one method by using the other method.

What are the types of sampling designs in research methodology?

Sampling 4 is an important part of a research methodology and involves selecting a representative sample of the population to conduct the study, making statistical inferences about them, and estimating the characteristics of the whole population based on these inferences. There are two types of sampling designs in research methodology—probability and nonprobability.

• Probability sampling

In this type of sampling design, a sample is chosen from a larger population using some form of random selection, that is, every member of the population has an equal chance of being selected. The different types of probability sampling are:

• Systematic —sample members are chosen at regular intervals. It requires selecting a starting point for the sample and sample size determination that can be repeated at regular intervals. This type of sampling method has a predefined range; hence, it is the least time consuming.
• Stratified —researchers divide the population into smaller groups that don’t overlap but represent the entire population. While sampling, these groups can be organized, and then a sample can be drawn from each group separately.
• Cluster —the population is divided into clusters based on demographic parameters like age, sex, location, etc.
• Convenience —selects participants who are most easily accessible to researchers due to geographical proximity, availability at a particular time, etc.
• Purposive —participants are selected at the researcher’s discretion. Researchers consider the purpose of the study and the understanding of the target audience.
• Snowball —already selected participants use their social networks to refer the researcher to other potential participants.
• Quota —while designing the study, the researchers decide how many people with which characteristics to include as participants. The characteristics help in choosing people most likely to provide insights into the subject.

What are data collection methods?

During research, data are collected using various methods depending on the research methodology being followed and the research methods being undertaken. Both qualitative and quantitative research have different data collection methods, as listed below.

Qualitative research 5

• One-on-one interviews: Helps the interviewers understand a respondent’s subjective opinion and experience pertaining to a specific topic or event
• Document study/literature review/record keeping: Researchers’ review of already existing written materials such as archives, annual reports, research articles, guidelines, policy documents, etc.
• Focus groups: Constructive discussions that usually include a small sample of about 6-10 people and a moderator, to understand the participants’ opinion on a given topic.
• Qualitative observation : Researchers collect data using their five senses (sight, smell, touch, taste, and hearing).

Quantitative research 6

• Sampling: The most common type is probability sampling.
• Interviews: Commonly telephonic or done in-person.
• Observations: Structured observations are most commonly used in quantitative research. In this method, researchers make observations about specific behaviors of individuals in a structured setting.
• Document review: Reviewing existing research or documents to collect evidence for supporting the research.
• Surveys and questionnaires. Surveys can be administered both online and offline depending on the requirement and sample size.

Let Paperpal help you write the perfect research methods section. Start now!

What are data analysis methods.

The data collected using the various methods for qualitative and quantitative research need to be analyzed to generate meaningful conclusions. These data analysis methods 7 also differ between quantitative and qualitative research.

Quantitative research involves a deductive method for data analysis where hypotheses are developed at the beginning of the research and precise measurement is required. The methods include statistical analysis applications to analyze numerical data and are grouped into two categories—descriptive and inferential.

Descriptive analysis is used to describe the basic features of different types of data to present it in a way that ensures the patterns become meaningful. The different types of descriptive analysis methods are:

• Measures of frequency (count, percent, frequency)
• Measures of central tendency (mean, median, mode)
• Measures of dispersion or variation (range, variance, standard deviation)
• Measure of position (percentile ranks, quartile ranks)

Inferential analysis is used to make predictions about a larger population based on the analysis of the data collected from a smaller population. This analysis is used to study the relationships between different variables. Some commonly used inferential data analysis methods are:

• Correlation: To understand the relationship between two or more variables.
• Cross-tabulation: Analyze the relationship between multiple variables.
• Regression analysis: Study the impact of independent variables on the dependent variable.
• Frequency tables: To understand the frequency of data.
• Analysis of variance: To test the degree to which two or more variables differ in an experiment.

Qualitative research involves an inductive method for data analysis where hypotheses are developed after data collection. The methods include:

• Content analysis: For analyzing documented information from text and images by determining the presence of certain words or concepts in texts.
• Narrative analysis: For analyzing content obtained from sources such as interviews, field observations, and surveys. The stories and opinions shared by people are used to answer research questions.
• Discourse analysis: For analyzing interactions with people considering the social context, that is, the lifestyle and environment, under which the interaction occurs.
• Grounded theory: Involves hypothesis creation by data collection and analysis to explain why a phenomenon occurred.
• Thematic analysis: To identify important themes or patterns in data and use these to address an issue.

How to choose a research methodology?

Here are some important factors to consider when choosing a research methodology: 8

• Research objectives, aims, and questions —these would help structure the research design.
• Review existing literature to identify any gaps in knowledge.
• Check the statistical requirements —if data-driven or statistical results are needed then quantitative research is the best. If the research questions can be answered based on people’s opinions and perceptions, then qualitative research is most suitable.
• Sample size —sample size can often determine the feasibility of a research methodology. For a large sample, less effort- and time-intensive methods are appropriate.
• Constraints —constraints of time, geography, and resources can help define the appropriate methodology.

Got writer’s block? Kickstart your research paper writing with Paperpal now!

How to write a research methodology .

A research methodology should include the following components: 3,9

• Research design —should be selected based on the research question and the data required. Common research designs include experimental, quasi-experimental, correlational, descriptive, and exploratory.
• Research method —this can be quantitative, qualitative, or mixed-method.
• Reason for selecting a specific methodology —explain why this methodology is the most suitable to answer your research problem.
• Research instruments —explain the research instruments you plan to use, mainly referring to the data collection methods such as interviews, surveys, etc. Here as well, a reason should be mentioned for selecting the particular instrument.
• Sampling —this involves selecting a representative subset of the population being studied.
• Data collection —involves gathering data using several data collection methods, such as surveys, interviews, etc.
• Data analysis —describe the data analysis methods you will use once you’ve collected the data.
• Research limitations —mention any limitations you foresee while conducting your research.
• Validity and reliability —validity helps identify the accuracy and truthfulness of the findings; reliability refers to the consistency and stability of the results over time and across different conditions.
• Ethical considerations —research should be conducted ethically. The considerations include obtaining consent from participants, maintaining confidentiality, and addressing conflicts of interest.

Streamline Your Research Paper Writing Process with Paperpal  

The methods section is a critical part of the research papers, allowing researchers to use this to understand your findings and replicate your work when pursuing their own research. However, it is usually also the most difficult section to write. This is where Paperpal can help you overcome the writer’s block and create the first draft in minutes with Paperpal Copilot, its secure generative AI feature suite.  

With Paperpal you can get research advice, write and refine your work, rephrase and verify the writing, and ensure submission readiness, all in one place. Here’s how you can use Paperpal to develop the first draft of your methods section.  

• Generate an outline: Input some details about your research to instantly generate an outline for your methods section 
• Develop the section: Use the outline and suggested sentence templates to expand your ideas and develop the first draft.  
• P araph ras e and trim : Get clear, concise academic text with paraphrasing that conveys your work effectively and word reduction to fix redundancies. 
• Choose the right words: Enhance text by choosing contextual synonyms based on how the words have been used in previously published work.  
• Check and verify text : Make sure the generated text showcases your methods correctly, has all the right citations, and is original and authentic. .   

You can repeat this process to develop each section of your research manuscript, including the title, abstract and keywords. Ready to write your research papers faster, better, and without the stress? Sign up for Paperpal and start writing today!

Frequently Asked Questions

Q1. What are the key components of research methodology?

A1. A good research methodology has the following key components:

• Research design
• Data collection procedures
• Data analysis methods
• Ethical considerations

Q2. Why is ethical consideration important in research methodology?

A2. Ethical consideration is important in research methodology to ensure the readers of the reliability and validity of the study. Researchers must clearly mention the ethical norms and standards followed during the conduct of the research and also mention if the research has been cleared by any institutional board. The following 10 points are the important principles related to ethical considerations: 10

• Participants should not be subjected to harm.
• Respect for the dignity of participants should be prioritized.
• Full consent should be obtained from participants before the study.
• Participants’ privacy should be ensured.
• Confidentiality of the research data should be ensured.
• Anonymity of individuals and organizations participating in the research should be maintained.
• The aims and objectives of the research should not be exaggerated.
• Affiliations, sources of funding, and any possible conflicts of interest should be declared.
• Communication in relation to the research should be honest and transparent.
• Misleading information and biased representation of primary data findings should be avoided.

Q3. What is the difference between methodology and method?

A3. Research methodology is different from a research method, although both terms are often confused. Research methods are the tools used to gather data, while the research methodology provides a framework for how research is planned, conducted, and analyzed. The latter guides researchers in making decisions about the most appropriate methods for their research. Research methods refer to the specific techniques, procedures, and tools used by researchers to collect, analyze, and interpret data, for instance surveys, questionnaires, interviews, etc.

Research methodology is, thus, an integral part of a research study. It helps ensure that you stay on track to meet your research objectives and answer your research questions using the most appropriate data collection and analysis tools based on your research design.

Accelerate your research paper writing with Paperpal. Try for free now!  

• Research methodologies. Pfeiffer Library website. Accessed August 15, 2023. https://library.tiffin.edu/researchmethodologies/whatareresearchmethodologies
• Types of research methodology. Eduvoice website. Accessed August 16, 2023. https://eduvoice.in/types-research-methodology/
• The basics of research methodology: A key to quality research. Voxco. Accessed August 16, 2023. https://www.voxco.com/blog/what-is-research-methodology/
• Sampling methods: Types with examples. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/types-of-sampling-for-social-research/
• What is qualitative research? Methods, types, approaches, examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-qualitative-research-methods-types-examples/
• What is quantitative research? Definition, methods, types, and examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-quantitative-research-types-and-examples/
• Data analysis in research: Types & methods. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/data-analysis-in-research/#Data_analysis_in_qualitative_research
• Factors to consider while choosing the right research methodology. PhD Monster website. Accessed August 17, 2023. https://www.phdmonster.com/factors-to-consider-while-choosing-the-right-research-methodology/
• What is research methodology? Research and writing guides. Accessed August 14, 2023. https://paperpile.com/g/what-is-research-methodology/
• Ethical considerations. Business research methodology website. Accessed August 17, 2023. https://research-methodology.net/research-methodology/ethical-considerations/

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Dangling Modifiers and How to Avoid Them in Your Writing 
• Research Outlines: How to Write An Introduction Section in Minutes with Paperpal Copilot
• How to Paraphrase Research Papers Effectively
• What is a Literature Review? How to Write It (with Examples)

Language and Grammar Rules for Academic Writing

Climatic vs. climactic: difference and examples, you may also like, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide), maintaining academic integrity with paperpal’s generative ai writing..., research funding basics: what should a grant proposal..., how to write an abstract in research papers....

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Res Metr Anal

The Use of Research Methods in Psychological Research: A Systematised Review

Salomé elizabeth scholtz.

1 Community Psychosocial Research (COMPRES), School of Psychosocial Health, North-West University, Potchefstroom, South Africa

Werner de Klerk

Leon t. de beer.

2 WorkWell Research Institute, North-West University, Potchefstroom, South Africa

Research methods play an imperative role in research quality as well as educating young researchers, however, the application thereof is unclear which can be detrimental to the field of psychology. Therefore, this systematised review aimed to determine what research methods are being used, how these methods are being used and for what topics in the field. Our review of 999 articles from five journals over a period of 5 years indicated that psychology research is conducted in 10 topics via predominantly quantitative research methods. Of these 10 topics, social psychology was the most popular. The remainder of the conducted methodology is described. It was also found that articles lacked rigour and transparency in the used methodology which has implications for replicability. In conclusion this article, provides an overview of all reported methodologies used in a sample of psychology journals. It highlights the popularity and application of methods and designs throughout the article sample as well as an unexpected lack of rigour with regard to most aspects of methodology. Possible sample bias should be considered when interpreting the results of this study. It is recommended that future research should utilise the results of this study to determine the possible impact on the field of psychology as a science and to further investigation into the use of research methods. Results should prompt the following future research into: a lack or rigour and its implication on replication, the use of certain methods above others, publication bias and choice of sampling method.

Introduction

Psychology is an ever-growing and popular field (Gough and Lyons, 2016 ; Clay, 2017 ). Due to this growth and the need for science-based research to base health decisions on (Perestelo-Pérez, 2013 ), the use of research methods in the broad field of psychology is an essential point of investigation (Stangor, 2011 ; Aanstoos, 2014 ). Research methods are therefore viewed as important tools used by researchers to collect data (Nieuwenhuis, 2016 ) and include the following: quantitative, qualitative, mixed method and multi method (Maree, 2016 ). Additionally, researchers also employ various types of literature reviews to address research questions (Grant and Booth, 2009 ). According to literature, what research method is used and why a certain research method is used is complex as it depends on various factors that may include paradigm (O'Neil and Koekemoer, 2016 ), research question (Grix, 2002 ), or the skill and exposure of the researcher (Nind et al., 2015 ). How these research methods are employed is also difficult to discern as research methods are often depicted as having fixed boundaries that are continuously crossed in research (Johnson et al., 2001 ; Sandelowski, 2011 ). Examples of this crossing include adding quantitative aspects to qualitative studies (Sandelowski et al., 2009 ), or stating that a study used a mixed-method design without the study having any characteristics of this design (Truscott et al., 2010 ).

The inappropriate use of research methods affects how students and researchers improve and utilise their research skills (Scott Jones and Goldring, 2015 ), how theories are developed (Ngulube, 2013 ), and the credibility of research results (Levitt et al., 2017 ). This, in turn, can be detrimental to the field (Nind et al., 2015 ), journal publication (Ketchen et al., 2008 ; Ezeh et al., 2010 ), and attempts to address public social issues through psychological research (Dweck, 2017 ). This is especially important given the now well-known replication crisis the field is facing (Earp and Trafimow, 2015 ; Hengartner, 2018 ).

Due to this lack of clarity on method use and the potential impact of inept use of research methods, the aim of this study was to explore the use of research methods in the field of psychology through a review of journal publications. Chaichanasakul et al. ( 2011 ) identify reviewing articles as the opportunity to examine the development, growth and progress of a research area and overall quality of a journal. Studies such as Lee et al. ( 1999 ) as well as Bluhm et al. ( 2011 ) review of qualitative methods has attempted to synthesis the use of research methods and indicated the growth of qualitative research in American and European journals. Research has also focused on the use of research methods in specific sub-disciplines of psychology, for example, in the field of Industrial and Organisational psychology Coetzee and Van Zyl ( 2014 ) found that South African publications tend to consist of cross-sectional quantitative research methods with underrepresented longitudinal studies. Qualitative studies were found to make up 21% of the articles published from 1995 to 2015 in a similar study by O'Neil and Koekemoer ( 2016 ). Other methods in health psychology, such as Mixed methods research have also been reportedly growing in popularity (O'Cathain, 2009 ).

A broad overview of the use of research methods in the field of psychology as a whole is however, not available in the literature. Therefore, our research focused on answering what research methods are being used, how these methods are being used and for what topics in practice (i.e., journal publications) in order to provide a general perspective of method used in psychology publication. We synthesised the collected data into the following format: research topic [areas of scientific discourse in a field or the current needs of a population (Bittermann and Fischer, 2018 )], method [data-gathering tools (Nieuwenhuis, 2016 )], sampling [elements chosen from a population to partake in research (Ritchie et al., 2009 )], data collection [techniques and research strategy (Maree, 2016 )], and data analysis [discovering information by examining bodies of data (Ktepi, 2016 )]. A systematised review of recent articles (2013 to 2017) collected from five different journals in the field of psychological research was conducted.

Grant and Booth ( 2009 ) describe systematised reviews as the review of choice for post-graduate studies, which is employed using some elements of a systematic review and seldom more than one or two databases to catalogue studies after a comprehensive literature search. The aspects used in this systematised review that are similar to that of a systematic review were a full search within the chosen database and data produced in tabular form (Grant and Booth, 2009 ).

Sample sizes and timelines vary in systematised reviews (see Lowe and Moore, 2014 ; Pericall and Taylor, 2014 ; Barr-Walker, 2017 ). With no clear parameters identified in the literature (see Grant and Booth, 2009 ), the sample size of this study was determined by the purpose of the sample (Strydom, 2011 ), and time and cost constraints (Maree and Pietersen, 2016 ). Thus, a non-probability purposive sample (Ritchie et al., 2009 ) of the top five psychology journals from 2013 to 2017 was included in this research study. Per Lee ( 2015 ) American Psychological Association (APA) recommends the use of the most up-to-date sources for data collection with consideration of the context of the research study. As this research study focused on the most recent trends in research methods used in the broad field of psychology, the identified time frame was deemed appropriate.

Psychology journals were only included if they formed part of the top five English journals in the miscellaneous psychology domain of the Scimago Journal and Country Rank (Scimago Journal & Country Rank, 2017 ). The Scimago Journal and Country Rank provides a yearly updated list of publicly accessible journal and country-specific indicators derived from the Scopus® database (Scopus, 2017b ) by means of the Scimago Journal Rank (SJR) indicator developed by Scimago from the algorithm Google PageRank™ (Scimago Journal & Country Rank, 2017 ). Scopus is the largest global database of abstracts and citations from peer-reviewed journals (Scopus, 2017a ). Reasons for the development of the Scimago Journal and Country Rank list was to allow researchers to assess scientific domains, compare country rankings, and compare and analyse journals (Scimago Journal & Country Rank, 2017 ), which supported the aim of this research study. Additionally, the goals of the journals had to focus on topics in psychology in general with no preference to specific research methods and have full-text access to articles.

The following list of top five journals in 2018 fell within the abovementioned inclusion criteria (1) Australian Journal of Psychology, (2) British Journal of Psychology, (3) Europe's Journal of Psychology, (4) International Journal of Psychology and lastly the (5) Journal of Psychology Applied and Interdisciplinary.

Journals were excluded from this systematised review if no full-text versions of their articles were available, if journals explicitly stated a publication preference for certain research methods, or if the journal only published articles in a specific discipline of psychological research (for example, industrial psychology, clinical psychology etc.).

The researchers followed a procedure (see Figure 1 ) adapted from that of Ferreira et al. ( 2016 ) for systematised reviews. Data collection and categorisation commenced on 4 December 2017 and continued until 30 June 2019. All the data was systematically collected and coded manually (Grant and Booth, 2009 ) with an independent person acting as co-coder. Codes of interest included the research topic, method used, the design used, sampling method, and methodology (the method used for data collection and data analysis). These codes were derived from the wording in each article. Themes were created based on the derived codes and checked by the co-coder. Lastly, these themes were catalogued into a table as per the systematised review design.

Systematised review procedure.

According to Johnston et al. ( 2019 ), “literature screening, selection, and data extraction/analyses” (p. 7) are specifically tailored to the aim of a review. Therefore, the steps followed in a systematic review must be reported in a comprehensive and transparent manner. The chosen systematised design adhered to the rigour expected from systematic reviews with regard to full search and data produced in tabular form (Grant and Booth, 2009 ). The rigorous application of the systematic review is, therefore discussed in relation to these two elements.

Firstly, to ensure a comprehensive search, this research study promoted review transparency by following a clear protocol outlined according to each review stage before collecting data (Johnston et al., 2019 ). This protocol was similar to that of Ferreira et al. ( 2016 ) and approved by three research committees/stakeholders and the researchers (Johnston et al., 2019 ). The eligibility criteria for article inclusion was based on the research question and clearly stated, and the process of inclusion was recorded on an electronic spreadsheet to create an evidence trail (Bandara et al., 2015 ; Johnston et al., 2019 ). Microsoft Excel spreadsheets are a popular tool for review studies and can increase the rigour of the review process (Bandara et al., 2015 ). Screening for appropriate articles for inclusion forms an integral part of a systematic review process (Johnston et al., 2019 ). This step was applied to two aspects of this research study: the choice of eligible journals and articles to be included. Suitable journals were selected by the first author and reviewed by the second and third authors. Initially, all articles from the chosen journals were included. Then, by process of elimination, those irrelevant to the research aim, i.e., interview articles or discussions etc., were excluded.

To ensure rigourous data extraction, data was first extracted by one reviewer, and an independent person verified the results for completeness and accuracy (Johnston et al., 2019 ). The research question served as a guide for efficient, organised data extraction (Johnston et al., 2019 ). Data was categorised according to the codes of interest, along with article identifiers for audit trails such as authors, title and aims of articles. The categorised data was based on the aim of the review (Johnston et al., 2019 ) and synthesised in tabular form under methods used, how these methods were used, and for what topics in the field of psychology.

The initial search produced a total of 1,145 articles from the 5 journals identified. Inclusion and exclusion criteria resulted in a final sample of 999 articles ( Figure 2 ). Articles were co-coded into 84 codes, from which 10 themes were derived ( Table 1 ).

Journal article frequency.

Codes used to form themes (research topics).

These 10 themes represent the topic section of our research question ( Figure 3 ). All these topics except, for the final one, psychological practice , were found to concur with the research areas in psychology as identified by Weiten ( 2010 ). These research areas were chosen to represent the derived codes as they provided broad definitions that allowed for clear, concise categorisation of the vast amount of data. Article codes were categorised under particular themes/topics if they adhered to the research area definitions created by Weiten ( 2010 ). It is important to note that these areas of research do not refer to specific disciplines in psychology, such as industrial psychology; but to broader fields that may encompass sub-interests of these disciplines.

Topic frequency (international sample).

In the case of developmental psychology , researchers conduct research into human development from childhood to old age. Social psychology includes research on behaviour governed by social drivers. Researchers in the field of educational psychology study how people learn and the best way to teach them. Health psychology aims to determine the effect of psychological factors on physiological health. Physiological psychology , on the other hand, looks at the influence of physiological aspects on behaviour. Experimental psychology is not the only theme that uses experimental research and focuses on the traditional core topics of psychology (for example, sensation). Cognitive psychology studies the higher mental processes. Psychometrics is concerned with measuring capacity or behaviour. Personality research aims to assess and describe consistency in human behaviour (Weiten, 2010 ). The final theme of psychological practice refers to the experiences, techniques, and interventions employed by practitioners, researchers, and academia in the field of psychology.

Articles under these themes were further subdivided into methodologies: method, sampling, design, data collection, and data analysis. The categorisation was based on information stated in the articles and not inferred by the researchers. Data were compiled into two sets of results presented in this article. The first set addresses the aim of this study from the perspective of the topics identified. The second set of results represents a broad overview of the results from the perspective of the methodology employed. The second set of results are discussed in this article, while the first set is presented in table format. The discussion thus provides a broad overview of methods use in psychology (across all themes), while the table format provides readers with in-depth insight into methods used in the individual themes identified. We believe that presenting the data from both perspectives allow readers a broad understanding of the results. Due a large amount of information that made up our results, we followed Cichocka and Jost ( 2014 ) in simplifying our results. Please note that the numbers indicated in the table in terms of methodology differ from the total number of articles. Some articles employed more than one method/sampling technique/design/data collection method/data analysis in their studies.

What follows is the results for what methods are used, how these methods are used, and which topics in psychology they are applied to . Percentages are reported to the second decimal in order to highlight small differences in the occurrence of methodology.

Firstly, with regard to the research methods used, our results show that researchers are more likely to use quantitative research methods (90.22%) compared to all other research methods. Qualitative research was the second most common research method but only made up about 4.79% of the general method usage. Reviews occurred almost as much as qualitative studies (3.91%), as the third most popular method. Mixed-methods research studies (0.98%) occurred across most themes, whereas multi-method research was indicated in only one study and amounted to 0.10% of the methods identified. The specific use of each method in the topics identified is shown in Table 2 and Figure 4 .

Research methods in psychology.

Research method frequency in topics.

Secondly, in the case of how these research methods are employed , our study indicated the following.

Sampling −78.34% of the studies in the collected articles did not specify a sampling method. From the remainder of the studies, 13 types of sampling methods were identified. These sampling methods included broad categorisation of a sample as, for example, a probability or non-probability sample. General samples of convenience were the methods most likely to be applied (10.34%), followed by random sampling (3.51%), snowball sampling (2.73%), and purposive (1.37%) and cluster sampling (1.27%). The remainder of the sampling methods occurred to a more limited extent (0–1.0%). See Table 3 and Figure 5 for sampling methods employed in each topic.

Sampling use in the field of psychology.

Sampling method frequency in topics.

Designs were categorised based on the articles' statement thereof. Therefore, it is important to note that, in the case of quantitative studies, non-experimental designs (25.55%) were often indicated due to a lack of experiments and any other indication of design, which, according to Laher ( 2016 ), is a reasonable categorisation. Non-experimental designs should thus be compared with experimental designs only in the description of data, as it could include the use of correlational/cross-sectional designs, which were not overtly stated by the authors. For the remainder of the research methods, “not stated” (7.12%) was assigned to articles without design types indicated.

From the 36 identified designs the most popular designs were cross-sectional (23.17%) and experimental (25.64%), which concurred with the high number of quantitative studies. Longitudinal studies (3.80%), the third most popular design, was used in both quantitative and qualitative studies. Qualitative designs consisted of ethnography (0.38%), interpretative phenomenological designs/phenomenology (0.28%), as well as narrative designs (0.28%). Studies that employed the review method were mostly categorised as “not stated,” with the most often stated review designs being systematic reviews (0.57%). The few mixed method studies employed exploratory, explanatory (0.09%), and concurrent designs (0.19%), with some studies referring to separate designs for the qualitative and quantitative methods. The one study that identified itself as a multi-method study used a longitudinal design. Please see how these designs were employed in each specific topic in Table 4 , Figure 6 .

Design use in the field of psychology.

Design frequency in topics.

Data collection and analysis —data collection included 30 methods, with the data collection method most often employed being questionnaires (57.84%). The experimental task (16.56%) was the second most preferred collection method, which included established or unique tasks designed by the researchers. Cognitive ability tests (6.84%) were also regularly used along with various forms of interviewing (7.66%). Table 5 and Figure 7 represent data collection use in the various topics. Data analysis consisted of 3,857 occurrences of data analysis categorised into ±188 various data analysis techniques shown in Table 6 and Figures 1 – 7 . Descriptive statistics were the most commonly used (23.49%) along with correlational analysis (17.19%). When using a qualitative method, researchers generally employed thematic analysis (0.52%) or different forms of analysis that led to coding and the creation of themes. Review studies presented few data analysis methods, with most studies categorising their results. Mixed method and multi-method studies followed the analysis methods identified for the qualitative and quantitative studies included.

Data collection in the field of psychology.

Data collection frequency in topics.

Data analysis in the field of psychology.

Results of the topics researched in psychology can be seen in the tables, as previously stated in this article. It is noteworthy that, of the 10 topics, social psychology accounted for 43.54% of the studies, with cognitive psychology the second most popular research topic at 16.92%. The remainder of the topics only occurred in 4.0–7.0% of the articles considered. A list of the included 999 articles is available under the section “View Articles” on the following website: https://methodgarden.xtrapolate.io/ . This website was created by Scholtz et al. ( 2019 ) to visually present a research framework based on this Article's results.

This systematised review categorised full-length articles from five international journals across the span of 5 years to provide insight into the use of research methods in the field of psychology. Results indicated what methods are used how these methods are being used and for what topics (why) in the included sample of articles. The results should be seen as providing insight into method use and by no means a comprehensive representation of the aforementioned aim due to the limited sample. To our knowledge, this is the first research study to address this topic in this manner. Our discussion attempts to promote a productive way forward in terms of the key results for method use in psychology, especially in the field of academia (Holloway, 2008 ).

With regard to the methods used, our data stayed true to literature, finding only common research methods (Grant and Booth, 2009 ; Maree, 2016 ) that varied in the degree to which they were employed. Quantitative research was found to be the most popular method, as indicated by literature (Breen and Darlaston-Jones, 2010 ; Counsell and Harlow, 2017 ) and previous studies in specific areas of psychology (see Coetzee and Van Zyl, 2014 ). Its long history as the first research method (Leech et al., 2007 ) in the field of psychology as well as researchers' current application of mathematical approaches in their studies (Toomela, 2010 ) might contribute to its popularity today. Whatever the case may be, our results show that, despite the growth in qualitative research (Demuth, 2015 ; Smith and McGannon, 2018 ), quantitative research remains the first choice for article publication in these journals. Despite the included journals indicating openness to articles that apply any research methods. This finding may be due to qualitative research still being seen as a new method (Burman and Whelan, 2011 ) or reviewers' standards being higher for qualitative studies (Bluhm et al., 2011 ). Future research is encouraged into the possible biasness in publication of research methods, additionally further investigation with a different sample into the proclaimed growth of qualitative research may also provide different results.

Review studies were found to surpass that of multi-method and mixed method studies. To this effect Grant and Booth ( 2009 ), state that the increased awareness, journal contribution calls as well as its efficiency in procuring research funds all promote the popularity of reviews. The low frequency of mixed method studies contradicts the view in literature that it's the third most utilised research method (Tashakkori and Teddlie's, 2003 ). Its' low occurrence in this sample could be due to opposing views on mixing methods (Gunasekare, 2015 ) or that authors prefer publishing in mixed method journals, when using this method, or its relative novelty (Ivankova et al., 2016 ). Despite its low occurrence, the application of the mixed methods design in articles was methodologically clear in all cases which were not the case for the remainder of research methods.

Additionally, a substantial number of studies used a combination of methodologies that are not mixed or multi-method studies. Perceived fixed boundaries are according to literature often set aside, as confirmed by this result, in order to investigate the aim of a study, which could create a new and helpful way of understanding the world (Gunasekare, 2015 ). According to Toomela ( 2010 ), this is not unheard of and could be considered a form of “structural systemic science,” as in the case of qualitative methodology (observation) applied in quantitative studies (experimental design) for example. Based on this result, further research into this phenomenon as well as its implications for research methods such as multi and mixed methods is recommended.

Discerning how these research methods were applied, presented some difficulty. In the case of sampling, most studies—regardless of method—did mention some form of inclusion and exclusion criteria, but no definite sampling method. This result, along with the fact that samples often consisted of students from the researchers' own academic institutions, can contribute to literature and debates among academics (Peterson and Merunka, 2014 ; Laher, 2016 ). Samples of convenience and students as participants especially raise questions about the generalisability and applicability of results (Peterson and Merunka, 2014 ). This is because attention to sampling is important as inappropriate sampling can debilitate the legitimacy of interpretations (Onwuegbuzie and Collins, 2017 ). Future investigation into the possible implications of this reported popular use of convenience samples for the field of psychology as well as the reason for this use could provide interesting insight, and is encouraged by this study.

Additionally, and this is indicated in Table 6 , articles seldom report the research designs used, which highlights the pressing aspect of the lack of rigour in the included sample. Rigour with regards to the applied empirical method is imperative in promoting psychology as a science (American Psychological Association, 2020 ). Omitting parts of the research process in publication when it could have been used to inform others' research skills should be questioned, and the influence on the process of replicating results should be considered. Publications are often rejected due to a lack of rigour in the applied method and designs (Fonseca, 2013 ; Laher, 2016 ), calling for increased clarity and knowledge of method application. Replication is a critical part of any field of scientific research and requires the “complete articulation” of the study methods used (Drotar, 2010 , p. 804). The lack of thorough description could be explained by the requirements of certain journals to only report on certain aspects of a research process, especially with regard to the applied design (Laher, 20). However, naming aspects such as sampling and designs, is a requirement according to the APA's Journal Article Reporting Standards (JARS-Quant) (Appelbaum et al., 2018 ). With very little information on how a study was conducted, authors lose a valuable opportunity to enhance research validity, enrich the knowledge of others, and contribute to the growth of psychology and methodology as a whole. In the case of this research study, it also restricted our results to only reported samples and designs, which indicated a preference for certain designs, such as cross-sectional designs for quantitative studies.

Data collection and analysis were for the most part clearly stated. A key result was the versatile use of questionnaires. Researchers would apply a questionnaire in various ways, for example in questionnaire interviews, online surveys, and written questionnaires across most research methods. This may highlight a trend for future research.

With regard to the topics these methods were employed for, our research study found a new field named “psychological practice.” This result may show the growing consciousness of researchers as part of the research process (Denzin and Lincoln, 2003 ), psychological practice, and knowledge generation. The most popular of these topics was social psychology, which is generously covered in journals and by learning societies, as testaments of the institutional support and richness social psychology has in the field of psychology (Chryssochoou, 2015 ). The APA's perspective on 2018 trends in psychology also identifies an increased amount of psychology focus on how social determinants are influencing people's health (Deangelis, 2017 ).

This study was not without limitations and the following should be taken into account. Firstly, this study used a sample of five specific journals to address the aim of the research study, despite general journal aims (as stated on journal websites), this inclusion signified a bias towards the research methods published in these specific journals only and limited generalisability. A broader sample of journals over a different period of time, or a single journal over a longer period of time might provide different results. A second limitation is the use of Excel spreadsheets and an electronic system to log articles, which was a manual process and therefore left room for error (Bandara et al., 2015 ). To address this potential issue, co-coding was performed to reduce error. Lastly, this article categorised data based on the information presented in the article sample; there was no interpretation of what methodology could have been applied or whether the methods stated adhered to the criteria for the methods used. Thus, a large number of articles that did not clearly indicate a research method or design could influence the results of this review. However, this in itself was also a noteworthy result. Future research could review research methods of a broader sample of journals with an interpretive review tool that increases rigour. Additionally, the authors also encourage the future use of systematised review designs as a way to promote a concise procedure in applying this design.

Our research study presented the use of research methods for published articles in the field of psychology as well as recommendations for future research based on these results. Insight into the complex questions identified in literature, regarding what methods are used how these methods are being used and for what topics (why) was gained. This sample preferred quantitative methods, used convenience sampling and presented a lack of rigorous accounts for the remaining methodologies. All methodologies that were clearly indicated in the sample were tabulated to allow researchers insight into the general use of methods and not only the most frequently used methods. The lack of rigorous account of research methods in articles was represented in-depth for each step in the research process and can be of vital importance to address the current replication crisis within the field of psychology. Recommendations for future research aimed to motivate research into the practical implications of the results for psychology, for example, publication bias and the use of convenience samples.

Ethics Statement

This study was cleared by the North-West University Health Research Ethics Committee: NWU-00115-17-S1.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

• Aanstoos C. M. (2014). Psychology . Available online at: http://eds.a.ebscohost.com.nwulib.nwu.ac.za/eds/detail/detail?sid=18de6c5c-2b03-4eac-94890145eb01bc70%40sessionmgr4006&vid$=$1&hid$=$4113&bdata$=$JnNpdGU9ZWRzL~WxpdmU%3d#AN$=$93871882&db$=$ers
• American Psychological Association (2020). Science of Psychology . Available online at: https://www.apa.org/action/science/
• Appelbaum M., Cooper H., Kline R. B., Mayo-Wilson E., Nezu A. M., Rao S. M. (2018). Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report . Am. Psychol. 73 :3. 10.1037/amp0000191 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bandara W., Furtmueller E., Gorbacheva E., Miskon S., Beekhuyzen J. (2015). Achieving rigor in literature reviews: insights from qualitative data analysis and tool-support . Commun. Ass. Inform. Syst. 37 , 154–204. 10.17705/1CAIS.03708 [ CrossRef ] [ Google Scholar ]
• Barr-Walker J. (2017). Evidence-based information needs of public health workers: a systematized review . J. Med. Libr. Assoc. 105 , 69–79. 10.5195/JMLA.2017.109 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bittermann A., Fischer A. (2018). How to identify hot topics in psychology using topic modeling . Z. Psychol. 226 , 3–13. 10.1027/2151-2604/a000318 [ CrossRef ] [ Google Scholar ]
• Bluhm D. J., Harman W., Lee T. W., Mitchell T. R. (2011). Qualitative research in management: a decade of progress . J. Manage. Stud. 48 , 1866–1891. 10.1111/j.1467-6486.2010.00972.x [ CrossRef ] [ Google Scholar ]
• Breen L. J., Darlaston-Jones D. (2010). Moving beyond the enduring dominance of positivism in psychological research: implications for psychology in Australia . Aust. Psychol. 45 , 67–76. 10.1080/00050060903127481 [ CrossRef ] [ Google Scholar ]
• Burman E., Whelan P. (2011). Problems in / of Qualitative Research . Maidenhead: Open University Press/McGraw Hill. [ Google Scholar ]
• Chaichanasakul A., He Y., Chen H., Allen G. E. K., Khairallah T. S., Ramos K. (2011). Journal of Career Development: a 36-year content analysis (1972–2007) . J. Career. Dev. 38 , 440–455. 10.1177/0894845310380223 [ CrossRef ] [ Google Scholar ]
• Chryssochoou X. (2015). Social Psychology . Inter. Encycl. Soc. Behav. Sci. 22 , 532–537. 10.1016/B978-0-08-097086-8.24095-6 [ CrossRef ] [ Google Scholar ]
• Cichocka A., Jost J. T. (2014). Stripped of illusions? Exploring system justification processes in capitalist and post-Communist societies . Inter. J. Psychol. 49 , 6–29. 10.1002/ijop.12011 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Clay R. A. (2017). Psychology is More Popular Than Ever. Monitor on Psychology: Trends Report . Available online at: https://www.apa.org/monitor/2017/11/trends-popular
• Coetzee M., Van Zyl L. E. (2014). A review of a decade's scholarly publications (2004–2013) in the South African Journal of Industrial Psychology . SA. J. Psychol . 40 , 1–16. 10.4102/sajip.v40i1.1227 [ CrossRef ] [ Google Scholar ]
• Counsell A., Harlow L. (2017). Reporting practices and use of quantitative methods in Canadian journal articles in psychology . Can. Psychol. 58 , 140–147. 10.1037/cap0000074 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Deangelis T. (2017). Targeting Social Factors That Undermine Health. Monitor on Psychology: Trends Report . Available online at: https://www.apa.org/monitor/2017/11/trend-social-factors
• Demuth C. (2015). New directions in qualitative research in psychology . Integr. Psychol. Behav. Sci. 49 , 125–133. 10.1007/s12124-015-9303-9 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Denzin N. K., Lincoln Y. (2003). The Landscape of Qualitative Research: Theories and Issues , 2nd Edn. London: Sage. [ Google Scholar ]
• Drotar D. (2010). A call for replications of research in pediatric psychology and guidance for authors . J. Pediatr. Psychol. 35 , 801–805. 10.1093/jpepsy/jsq049 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Dweck C. S. (2017). Is psychology headed in the right direction? Yes, no, and maybe . Perspect. Psychol. Sci. 12 , 656–659. 10.1177/1745691616687747 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Earp B. D., Trafimow D. (2015). Replication, falsification, and the crisis of confidence in social psychology . Front. Psychol. 6 :621. 10.3389/fpsyg.2015.00621 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ezeh A. C., Izugbara C. O., Kabiru C. W., Fonn S., Kahn K., Manderson L., et al.. (2010). Building capacity for public and population health research in Africa: the consortium for advanced research training in Africa (CARTA) model . Glob. Health Action 3 :5693. 10.3402/gha.v3i0.5693 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ferreira A. L. L., Bessa M. M. M., Drezett J., De Abreu L. C. (2016). Quality of life of the woman carrier of endometriosis: systematized review . Reprod. Clim. 31 , 48–54. 10.1016/j.recli.2015.12.002 [ CrossRef ] [ Google Scholar ]
• Fonseca M. (2013). Most Common Reasons for Journal Rejections . Available online at: http://www.editage.com/insights/most-common-reasons-for-journal-rejections
• Gough B., Lyons A. (2016). The future of qualitative research in psychology: accentuating the positive . Integr. Psychol. Behav. Sci. 50 , 234–243. 10.1007/s12124-015-9320-8 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Grant M. J., Booth A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies . Health Info. Libr. J. 26 , 91–108. 10.1111/j.1471-1842.2009.00848.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Grix J. (2002). Introducing students to the generic terminology of social research . Politics 22 , 175–186. 10.1111/1467-9256.00173 [ CrossRef ] [ Google Scholar ]
• Gunasekare U. L. T. P. (2015). Mixed research method as the third research paradigm: a literature review . Int. J. Sci. Res. 4 , 361–368. Available online at: https://ssrn.com/abstract=2735996 [ Google Scholar ]
• Hengartner M. P. (2018). Raising awareness for the replication crisis in clinical psychology by focusing on inconsistencies in psychotherapy Research: how much can we rely on published findings from efficacy trials? Front. Psychol. 9 :256. 10.3389/fpsyg.2018.00256 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Holloway W. (2008). Doing intellectual disagreement differently . Psychoanal. Cult. Soc. 13 , 385–396. 10.1057/pcs.2008.29 [ CrossRef ] [ Google Scholar ]
• Ivankova N. V., Creswell J. W., Plano Clark V. L. (2016). Foundations and Approaches to mixed methods research , in First Steps in Research , 2nd Edn. K. Maree (Pretoria: Van Schaick Publishers; ), 306–335. [ Google Scholar ]
• Johnson M., Long T., White A. (2001). Arguments for British pluralism in qualitative health research . J. Adv. Nurs. 33 , 243–249. 10.1046/j.1365-2648.2001.01659.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Johnston A., Kelly S. E., Hsieh S. C., Skidmore B., Wells G. A. (2019). Systematic reviews of clinical practice guidelines: a methodological guide . J. Clin. Epidemiol. 108 , 64–72. 10.1016/j.jclinepi.2018.11.030 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ketchen D. J., Jr., Boyd B. K., Bergh D. D. (2008). Research methodology in strategic management: past accomplishments and future challenges . Organ. Res. Methods 11 , 643–658. 10.1177/1094428108319843 [ CrossRef ] [ Google Scholar ]
• Ktepi B. (2016). Data Analytics (DA) . Available online at: https://eds-b-ebscohost-com.nwulib.nwu.ac.za/eds/detail/detail?vid=2&sid=24c978f0-6685-4ed8-ad85-fa5bb04669b9%40sessionmgr101&bdata=JnNpdGU9ZWRzLWxpdmU%3d#AN=113931286&db=ers
• Laher S. (2016). Ostinato rigore: establishing methodological rigour in quantitative research . S. Afr. J. Psychol. 46 , 316–327. 10.1177/0081246316649121 [ CrossRef ] [ Google Scholar ]
• Lee C. (2015). The Myth of the Off-Limits Source . Available online at: http://blog.apastyle.org/apastyle/research/
• Lee T. W., Mitchell T. R., Sablynski C. J. (1999). Qualitative research in organizational and vocational psychology, 1979–1999 . J. Vocat. Behav. 55 , 161–187. 10.1006/jvbe.1999.1707 [ CrossRef ] [ Google Scholar ]
• Leech N. L., Anthony J., Onwuegbuzie A. J. (2007). A typology of mixed methods research designs . Sci. Bus. Media B. V Qual. Quant 43 , 265–275. 10.1007/s11135-007-9105-3 [ CrossRef ] [ Google Scholar ]
• Levitt H. M., Motulsky S. L., Wertz F. J., Morrow S. L., Ponterotto J. G. (2017). Recommendations for designing and reviewing qualitative research in psychology: promoting methodological integrity . Qual. Psychol. 4 , 2–22. 10.1037/qup0000082 [ CrossRef ] [ Google Scholar ]
• Lowe S. M., Moore S. (2014). Social networks and female reproductive choices in the developing world: a systematized review . Rep. Health 11 :85. 10.1186/1742-4755-11-85 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Maree K. (2016). Planning a research proposal , in First Steps in Research , 2nd Edn, ed Maree K. (Pretoria: Van Schaik Publishers; ), 49–70. [ Google Scholar ]
• Maree K., Pietersen J. (2016). Sampling , in First Steps in Research, 2nd Edn , ed Maree K. (Pretoria: Van Schaik Publishers; ), 191–202. [ Google Scholar ]
• Ngulube P. (2013). Blending qualitative and quantitative research methods in library and information science in sub-Saharan Africa . ESARBICA J. 32 , 10–23. Available online at: http://hdl.handle.net/10500/22397 . [ Google Scholar ]
• Nieuwenhuis J. (2016). Qualitative research designs and data-gathering techniques , in First Steps in Research , 2nd Edn, ed Maree K. (Pretoria: Van Schaik Publishers; ), 71–102. [ Google Scholar ]
• Nind M., Kilburn D., Wiles R. (2015). Using video and dialogue to generate pedagogic knowledge: teachers, learners and researchers reflecting together on the pedagogy of social research methods . Int. J. Soc. Res. Methodol. 18 , 561–576. 10.1080/13645579.2015.1062628 [ CrossRef ] [ Google Scholar ]
• O'Cathain A. (2009). Editorial: mixed methods research in the health sciences—a quiet revolution . J. Mix. Methods 3 , 1–6. 10.1177/1558689808326272 [ CrossRef ] [ Google Scholar ]
• O'Neil S., Koekemoer E. (2016). Two decades of qualitative research in psychology, industrial and organisational psychology and human resource management within South Africa: a critical review . SA J. Indust. Psychol. 42 , 1–16. 10.4102/sajip.v42i1.1350 [ CrossRef ] [ Google Scholar ]
• Onwuegbuzie A. J., Collins K. M. (2017). The role of sampling in mixed methods research enhancing inference quality . Köln Z Soziol. 2 , 133–156. 10.1007/s11577-017-0455-0 [ CrossRef ] [ Google Scholar ]
• Perestelo-Pérez L. (2013). Standards on how to develop and report systematic reviews in psychology and health . Int. J. Clin. Health Psychol. 13 , 49–57. 10.1016/S1697-2600(13)70007-3 [ CrossRef ] [ Google Scholar ]
• Pericall L. M. T., Taylor E. (2014). Family function and its relationship to injury severity and psychiatric outcome in children with acquired brain injury: a systematized review . Dev. Med. Child Neurol. 56 , 19–30. 10.1111/dmcn.12237 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Peterson R. A., Merunka D. R. (2014). Convenience samples of college students and research reproducibility . J. Bus. Res. 67 , 1035–1041. 10.1016/j.jbusres.2013.08.010 [ CrossRef ] [ Google Scholar ]
• Ritchie J., Lewis J., Elam G. (2009). Designing and selecting samples , in Qualitative Research Practice: A Guide for Social Science Students and Researchers , 2nd Edn, ed Ritchie J., Lewis J. (London: Sage; ), 1–23. [ Google Scholar ]
• Sandelowski M. (2011). When a cigar is not just a cigar: alternative perspectives on data and data analysis . Res. Nurs. Health 34 , 342–352. 10.1002/nur.20437 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sandelowski M., Voils C. I., Knafl G. (2009). On quantitizing . J. Mix. Methods Res. 3 , 208–222. 10.1177/1558689809334210 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Scholtz S. E., De Klerk W., De Beer L. T. (2019). A data generated research framework for conducting research methods in psychological research .
• Scimago Journal & Country Rank (2017). Available online at: http://www.scimagojr.com/journalrank.php?category=3201&year=2015
• Scopus (2017a). About Scopus . Available online at: https://www.scopus.com/home.uri (accessed February 01, 2017).
• Scopus (2017b). Document Search . Available online at: https://www.scopus.com/home.uri (accessed February 01, 2017).
• Scott Jones J., Goldring J. E. (2015). ‘I' m not a quants person'; key strategies in building competence and confidence in staff who teach quantitative research methods . Int. J. Soc. Res. Methodol. 18 , 479–494. 10.1080/13645579.2015.1062623 [ CrossRef ] [ Google Scholar ]
• Smith B., McGannon K. R. (2018). Developing rigor in quantitative research: problems and opportunities within sport and exercise psychology . Int. Rev. Sport Exerc. Psychol. 11 , 101–121. 10.1080/1750984X.2017.1317357 [ CrossRef ] [ Google Scholar ]
• Stangor C. (2011). Introduction to Psychology . Available online at: http://www.saylor.org/books/
• Strydom H. (2011). Sampling in the quantitative paradigm , in Research at Grass Roots; For the Social Sciences and Human Service Professions , 4th Edn, eds de Vos A. S., Strydom H., Fouché C. B., Delport C. S. L. (Pretoria: Van Schaik Publishers; ), 221–234. [ Google Scholar ]
• Tashakkori A., Teddlie C. (2003). Handbook of Mixed Methods in Social & Behavioural Research . Thousand Oaks, CA: SAGE publications. [ Google Scholar ]
• Toomela A. (2010). Quantitative methods in psychology: inevitable and useless . Front. Psychol. 1 :29. 10.3389/fpsyg.2010.00029 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Truscott D. M., Swars S., Smith S., Thornton-Reid F., Zhao Y., Dooley C., et al.. (2010). A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995–2005 . Int. J. Soc. Res. Methodol. 13 , 317–328. 10.1080/13645570903097950 [ CrossRef ] [ Google Scholar ]
• Weiten W. (2010). Psychology Themes and Variations . Belmont, CA: Wadsworth. [ Google Scholar ]

• Research Process
• Manuscript Preparation
• Manuscript Review
• Publication Process
• Publication Recognition
• Language Editing Services
• Translation Services

Choosing the Right Research Methodology: A Guide for Researchers

• 3 minute read
• 52.2K views

Table of Contents

Choosing an optimal research methodology is crucial for the success of any research project. The methodology you select will determine the type of data you collect, how you collect it, and how you analyse it. Understanding the different types of research methods available along with their strengths and weaknesses, is thus imperative to make an informed decision.

Understanding different research methods:

There are several research methods available depending on the type of study you are conducting, i.e., whether it is laboratory-based, clinical, epidemiological, or survey based . Some common methodologies include qualitative research, quantitative research, experimental research, survey-based research, and action research. Each method can be opted for and modified, depending on the type of research hypotheses and objectives.

Qualitative vs quantitative research:

When deciding on a research methodology, one of the key factors to consider is whether your research will be qualitative or quantitative. Qualitative research is used to understand people’s experiences, concepts, thoughts, or behaviours . Quantitative research, on the contrary, deals with numbers, graphs, and charts, and is used to test or confirm hypotheses, assumptions, and theories. 

Qualitative research methodology:

Qualitative research is often used to examine issues that are not well understood, and to gather additional insights on these topics. Qualitative research methods include open-ended survey questions, observations of behaviours described through words, and reviews of literature that has explored similar theories and ideas. These methods are used to understand how language is used in real-world situations, identify common themes or overarching ideas, and describe and interpret various texts. Data analysis for qualitative research typically includes discourse analysis, thematic analysis, and textual analysis. 

Quantitative research methodology:

The goal of quantitative research is to test hypotheses, confirm assumptions and theories, and determine cause-and-effect relationships. Quantitative research methods include experiments, close-ended survey questions, and countable and numbered observations. Data analysis for quantitative research relies heavily on statistical methods.

Analysing qualitative vs quantitative data:

The methods used for data analysis also differ for qualitative and quantitative research. As mentioned earlier, quantitative data is generally analysed using statistical methods and does not leave much room for speculation. It is more structured and follows a predetermined plan. In quantitative research, the researcher starts with a hypothesis and uses statistical methods to test it. Contrarily, methods used for qualitative data analysis can identify patterns and themes within the data, rather than provide statistical measures of the data. It is an iterative process, where the researcher goes back and forth trying to gauge the larger implications of the data through different perspectives and revising the analysis if required.

When to use qualitative vs quantitative research:

The choice between qualitative and quantitative research will depend on the gap that the research project aims to address, and specific objectives of the study. If the goal is to establish facts about a subject or topic, quantitative research is an appropriate choice. However, if the goal is to understand people’s experiences or perspectives, qualitative research may be more suitable. 

Conclusion:

In conclusion, an understanding of the different research methods available, their applicability, advantages, and disadvantages is essential for making an informed decision on the best methodology for your project. If you need any additional guidance on which research methodology to opt for, you can head over to Elsevier Author Services (EAS). EAS experts will guide you throughout the process and help you choose the perfect methodology for your research goals.

Why is data validation important in research?

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

You may also like.

Descriptive Research Design and Its Myriad Uses

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

Making Technical Writing in Environmental Engineering Accessible

To Err is Not Human: The Dangers of AI-assisted Academic Writing

Writing a good review article

Scholarly Sources: What are They and Where can You Find Them?

Input your search keywords and press Enter.

How to Write the Methods Section of a Scientific Article

What Is the Methods Section of a Research Paper?

The Methods section of a research article includes an explanation of the procedures used to conduct the experiment. For authors of scientific research papers, the objective is to present their findings clearly and concisely and to provide enough information so that the experiment can be duplicated.

Research articles contain very specific sections, usually dictated by either the target journal or specific style guides. For example, in the social and behavioral sciences, the American Psychological Association (APA) style guide is used to gather information on how the manuscript should be arranged . As with most styles, APA’s objectives are to ensure that manuscripts are written with minimum distractions to the reader. Every research article should include a detailed Methods section after the Introduction.

Why is the Methods Section Important?

The Methods section (also referred to as “Materials and Methods”) is important because it provides the reader enough information to judge whether the study is valid and reproducible.

Structure of the Methods Section in a Research Paper

While designing a research study, authors typically decide on the key points that they’re trying to prove or the “ cause-and-effect relationship ” between objects of the study. Very simply, the study is designed to meet the objective. According to APA, a Methods section comprises of the following three subsections: participants, apparatus, and procedure.

How do You Write a Method Section in Biology?

In biological sciences, the Methods section might be more detailed, but the objectives are the same—to present the study clearly and concisely so that it is understandable and can be duplicated.

If animals (including human subjects) were used in the study, authors should ensure to include statements that they were treated according to the protocols outlined to ensure that treatment is as humane as possible.

• The Declaration of Helsinki is a set of ethical principles developed by The World Medical Association to provide guidance to scientists and physicians in medical research involving human subjects.

Research conducted at an institution using human participants is overseen by the Institutional Review Board (IRB) with which it is affiliated. IRB is an administrative body whose purpose is to protect the rights and welfare of human subjects during their participation in the study.

Literature Search

Literature searches are performed to gather as much information as relevant from previous studies. They are important for providing evidence on the topic and help validate the research. Most are accomplished using keywords or phrases to search relevant databases. For example, both MEDLINE and PubMed provide information on biomedical literature. Google Scholar, according to APA, is “one of the best sources available to an individual beginning a literature search.” APA also suggests using PsycINFO and refers to it as “the premier database for locating articles in psychological science and related literature.”

Authors must make sure to have a set of keywords (usually taken from the objective statement) to stay focused and to avoid having the search move far from the original objective. Authors will benefit by setting limiting parameters, such as date ranges, and avoiding getting pulled into the trap of using non-valid resources, such as social media, conversations with people in the same discipline, or similar non-valid sources, as references.

Related: Ready with your methods section and looking forward to manuscript submission ? Check these journal selection guidelines now!

What Should be Included in the Methods Section of a Research Paper?

One commonly misused term in research papers is “methodology.” Methodology refers to a branch of the Philosophy of Science which deals with scientific methods, not to the methods themselves, so authors should avoid using it. Here is the list of main subsections that should be included in the Methods section of a research paper ; authors might use subheadings more clearly to describe their research.

• Literature search : Authors should cite any sources that helped with their choice of methods. Authors should indicate timeframes of past studies and their particular parameters.
• Study participants : Authors should cite the source from where they received any non-human subjects. The number of animals used, the ages, sex, their initial conditions, and how they were housed and cared for, should be listed. In case of human subjects, authors should provide the characteristics, such as geographical location; their age ranges, sex, and medical history (if relevant); and the number of subjects. In case hospital records were used, authors should include the subjects’ basic health information and vital statistics at the beginning of the study. Authors should also state that written informed consent was provided by each subject.
• Inclusion/exclusion criteria : Authors should describe their inclusion and exclusion criteria, how they were determined, and how many subjects were eliminated.
• Group characteristics (could be combined with “Study participants”) : Authors should describe how the chosen group was divided into subgroups and their characteristics, including the control. Authors should also describe any specific equipment used, such as housing needs and feed (usually for animal studies). If patient records are reviewed and assessed, authors should mention whether the reviewers were blinded to them.
• Procedures : Authors should describe their study design. Any necessary preparations (e.g., tissue samples, drugs) and instruments must be explained. Authors should describe how the subjects were “ manipulated to answer the experimental question .” Timeframes should be included to ensure that the procedures are clear (e.g., “Rats were given XX drug for 14 d”). For animals sacrificed, the methods used and the protocols followed should be outlined.
• Statistical analyses: The type of data, how they were measured, and which statistical tests were performed, should be described. (Note: This is not the “results” section; any relevant tables and figures should be referenced later.) Specific software used must be cited.

What Should not be Included in Your Methods Section?

Common pitfalls can make the manuscript cumbersome to read or might make the readers question the validity of the research. The University of Southern California provides some guidelines .

• Background information that is not helpful must be avoided.
• Authors must avoid providing a lot of detail.
• Authors should focus more on how their method was used to meet their objective and less on mechanics .
• Any obstacles faced and how they were overcome should be described (often in your “Study Limitations”). This will help validate the results.

According to the University of Richmond , authors must avoid including extensive details or an exhaustive list of equipment that have been used as readers could quickly lose attention. These unnecessary details add nothing to validate the research and do not help the reader understand how the objective was satisfied. A well-thought-out Methods section is one of the most important parts of the manuscript. Authors must make a note to always prepare a draft that lists all parts, allow others to review it, and revise it to remove any superfluous information.

m so confused about ma research but now m okay so thank uh so mxh

Mil gracias por su ayuda.

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• AI in Academia
• Infographic
• Manuscripts & Grants
• Reporting Research
• Trending Now

Can AI Tools Prepare a Research Manuscript From Scratch? — A comprehensive guide

As technology continues to advance, the question of whether artificial intelligence (AI) tools can prepare…

Abstract Vs. Introduction — Do you know the difference?

Ross wants to publish his research. Feeling positive about his research outcomes, he begins to…

• Old Webinars
• Webinar Mobile App

Demystifying Research Methodology With Field Experts

Choosing research methodology Research design and methodology Evidence-based research approach How RAxter can assist researchers

• Manuscript Preparation
• Publishing Research

How to Choose Best Research Methodology for Your Study

Successful research conduction requires proper planning and execution. While there are multiple reasons and aspects…

Top 5 Key Differences Between Methods and Methodology

While burning the midnight oil during literature review, most researchers do not realize that the…

How to Draft the Acknowledgment Section of a Manuscript

Discussion Vs. Conclusion: Know the Difference Before Drafting Manuscripts

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

• Industry News
• Promoting Research
• Career Corner
• Diversity and Inclusion
• Infographics
• Expert Video Library
• Other Resources
• Enago Learn
• Upcoming & On-Demand Webinars
• Peer Review Week 2024
• Open Access Week 2023
• Conference Videos
• Enago Report
• Journal Finder
• Enago Plagiarism & AI Grammar Check
• Editing Services
• Publication Support Services
• Research Impact
• Translation Services
• Publication solutions
• AI-Based Solutions
• Thought Leadership
• Call for Articles
• Call for Speakers
• Author Training
• Edit Profile

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

Which among these features would you prefer the most in a peer review assistant?

Research Methodologies

• What are research designs?
• What are research methodologies?

What are research methods?

Quantitative research methods, qualitative research methods, mixed method approach, selecting the best research method.

• Additional Sources

Research methods are different from research methodologies because they are the ways in which you will collect the data for your research project.  The best method for your project largely depends on your topic, the type of data you will need, and the people or items from which you will be collecting data.  The following boxes below contain a list of quantitative, qualitative, and mixed research methods.

• Closed-ended questionnaires/survey: These types of questionnaires or surveys are like "multiple choice" tests, where participants must select from a list of premade answers.  According to the content of the question, they must select the one that they agree with the most.  This approach is the simplest form of quantitative research because the data is easy to combine and quantify.
• Structured interviews: These are a common research method in market research because the data can be quantified.  They are strictly designed for little "wiggle room" in the interview process so that the data will not be skewed.  You can conduct structured interviews in-person, online, or over the phone (Dawson, 2019).

Constructing Questionnaires

When constructing your questions for a survey or questionnaire, there are things you can do to ensure that your questions are accurate and easy to understand (Dawson, 2019):

• Keep the questions brief and simple.
• Eliminate any potential bias from your questions.  Make sure that they do not word things in a way that favor one perspective over another.
• If your topic is very sensitive, you may want to ask indirect questions rather than direct ones.  This prevents participants from being intimidated and becoming unwilling to share their true responses.
• If you are using a closed-ended question, try to offer every possible answer that a participant could give to that question.
• Do not ask questions that assume something of the participant.  The question "How often do you exercise?" assumes that the participant exercises (when they may not), so you would want to include a question that asks if they exercise at all before asking them how often.
• Try and keep the questionnaire as short as possible.  The longer a questionnaire takes, the more likely the participant will not complete it or get too tired to put truthful answers.
• Promise confidentiality to your participants at the beginning of the questionnaire.

Quantitative Research Measures

When you are considering a quantitative approach to your research, you need to identify why types of measures you will use in your study.  This will determine what type of numbers you will be using to collect your data.  There are four levels of measurement:

• Nominal: These are numbers where the order of the numbers do not matter.  They aim to identify separate information.  One example is collecting zip codes from research participants.  The order of the numbers does not matter, but the series of numbers in each zip code indicate different information (Adamson and Prion, 2013).
• Ordinal: Also known as rankings because the order of these numbers matter.  This is when items are given a specific rank according to specific criteria.  A common example of ordinal measurements include ranking-based questionnaires, where participants are asked to rank items from least favorite to most favorite.  Another common example is a pain scale, where a patient is asked to rank their pain on a scale from 1 to 10 (Adamson and Prion, 2013).
• Interval: This is when the data are ordered and the distance between the numbers matters to the researcher (Adamson and Prion, 2013).  The distance between each number is the same.  An example of interval data is test grades.
• Ratio: This is when the data are ordered and have a consistent distance between numbers, but has a "zero point."  This means that there could be a measurement of zero of whatever you are measuring in your study (Adamson and Prion, 2013).  An example of ratio data is measuring the height of something because the "zero point" remains constant in all measurements.  The height of something could also be zero.

Focus Groups

This is when a select group of people gather to talk about a particular topic.  They can also be called discussion groups or group interviews (Dawson, 2019).  They are usually lead by a moderator  to help guide the discussion and ask certain questions.  It is critical that a moderator allows everyone in the group to get a chance to speak so that no one dominates the discussion.  The data that are gathered from focus groups tend to be thoughts, opinions, and perspectives about an issue.

Advantages of Focus Groups

• Only requires one meeting to get different types of responses.
• Less researcher bias due to participants being able to speak openly.
• Helps participants overcome insecurities or fears about a topic.
• The researcher can also consider the impact of participant interaction.

Disadvantages of Focus Groups

• Participants may feel uncomfortable to speak in front of an audience, especially if the topic is sensitive or controversial.
• Since participation is voluntary, not every participant may contribute equally to the discussion.
• Participants may impact what others say or think.
• A researcher may feel intimidated by running a focus group on their own.
• A researcher may need extra funds/resources to provide a safe space to host the focus group.
• Because the data is collective, it may be difficult to determine a participant's individual thoughts about the research topic.

Observation

There are two ways to conduct research observations:

• Direct Observation: The researcher observes a participant in an environment.  The researcher often takes notes or uses technology to gather data, such as a voice recorder or video camera.  The researcher does not interact or interfere with the participants.  This approach is often used in psychology and health studies (Dawson, 2019).
• Participant Observation:  The researcher interacts directly with the participants to get a better understanding of the research topic.  This is a common research method when trying to understand another culture or community.  It is important to decide if you will conduct a covert (participants do not know they are part of the research) or overt (participants know the researcher is observing them) observation because it can be unethical in some situations (Dawson, 2019).

Open-Ended Questionnaires

These types of questionnaires are the opposite of "multiple choice" questionnaires because the answer boxes are left open for the participant to complete.  This means that participants can write short or extended answers to the questions.  Upon gathering the responses, researchers will often "quantify" the data by organizing the responses into different categories.  This can be time consuming because the researcher needs to read all responses carefully.

Semi-structured Interviews

This is the most common type of interview where researchers aim to get specific information so they can compare it to other interview data.  This requires asking the same questions for each interview, but keeping their responses flexible.  This means including follow-up questions if a subject answers a certain way.  Interview schedules are commonly used to aid the interviewers, which list topics or questions that will be discussed at each interview (Dawson, 2019).

Theoretical Analysis

Often used for nonhuman research, theoretical analysis is a qualitative approach where the researcher applies a theoretical framework to analyze something about their topic.  A theoretical framework gives the researcher a specific "lens" to view the topic and think about it critically. it also serves as context to guide the entire study.  This is a popular research method for analyzing works of literature, films, and other forms of media.  You can implement more than one theoretical framework with this method, as many theories complement one another.

Common theoretical frameworks for qualitative research are (Grant and Osanloo, 2014):

• Behavioral theory
• Change theory
• Cognitive theory
• Content analysis
• Cross-sectional analysis
• Developmental theory
• Feminist theory
• Gender theory
• Marxist theory
• Queer theory
• Systems theory
• Transformational theory

Unstructured Interviews

These are in-depth interviews where the researcher tries to understand an interviewee's perspective on a situation or issue.  They are sometimes called life history interviews.  It is important not to bombard the interviewee with too many questions so they can freely disclose their thoughts (Dawson, 2019).

• Open-ended and closed-ended questionnaires: This approach means implementing elements of both questionnaire types into your data collection.  Participants may answer some questions with premade answers and write their own answers to other questions.  The advantage to this method is that you benefit from both types of data collection to get a broader understanding of you participants.  However, you must think carefully about how you will analyze this data to arrive at a conclusion.

Other mixed method approaches that incorporate quantitative and qualitative research methods depend heavily on the research topic.  It is strongly recommended that you collaborate with your academic advisor before finalizing a mixed method approach.

How do you determine which research method would be best for your proposal?  This heavily depends on your research objective.  According to Dawson (2019), there are several questions to ask yourself when determining the best research method for your project:

• Are you good with numbers and mathematics?
• Would you be interested in conducting interviews with human subjects?
• Would you enjoy creating a questionnaire for participants to complete?
• Do you prefer written communication or face-to-face interaction?
• What skills or experiences do you have that might help you with your research?  Do you have any experiences from past research projects that can help with this one?
• How much time do you have to complete the research?  Some methods take longer to collect data than others.
• What is your budget?  Do you have adequate funding to conduct the research in the method you  want?
• How much data do you need?  Some research topics need only a small amount of data while others may need significantly larger amounts.
• What is the purpose of your research? This can provide a good indicator as to what research method will be most appropriate.
• << Previous: What are research methodologies?
• Next: Additional Sources >>
• Last Updated: Aug 2, 2022 2:36 PM
• URL: https://library.tiffin.edu/researchmethodologies

• Introduction
• Article Information

Distribution of methods (A) and equations (B) to estimate kidney function. C, Overall distribution of the kidney function estimation approach according to the enrollment start year. Trials using serum creatinine (sCr) or creatinine clearance (CrCl) level specified that patients could be eligible by meeting either criteria. Vertical bars indicate 1 SE. CG indicates Cockcroft-Gault; eGFR, estimated glomerular filtration rate; and MDRD, Modification of Diet in Renal Disease.

Data Sharing Statement

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Karol AB , Paredes R , Fujiwara Y, et al. Heterogeneity in Methods of Estimating Kidney Function for Cancer Clinical Trial Eligibility. JAMA Netw Open. 2024;7(9):e2433387. doi:10.1001/jamanetworkopen.2024.33387

Manage citations:

© 2024

• Permissions

Heterogeneity in Methods of Estimating Kidney Function for Cancer Clinical Trial Eligibility

• 1 Department of Medicine, Icahn School of Medicine at Mount Sinai, Mount Sinai Hospital, New York, New York
• 2 Department of Medicine, Icahn School of Medicine at Mount Sinai, Mount Sinai Beth Israel, New York, New York
• 3 Department of Medicine, Roswell Park Comprehensive Cancer Center, Buffalo, New York
• 4 Icahn School of Medicine at Mount Sinai, New York, New York
• 5 Division of Hematology and Medical Oncology, The Tisch Cancer Institute, Icahn School of Medicine at Mount Sinai, New York, New York
• 6 Department of Population Health and Policy, Icahn School of Medicine at Mount Sinai, New York, New York
• 7 Division of Nephrology, Icahn School of Medicine at Mount Sinai, New York, New York

Kidney function is frequently impaired in patients with cancer due to preexisting chronic kidney disease, cancer-associated factors, and prior anticancer treatments. Given these multifactorial insults, accurate assessment of kidney function is critical to determining patient eligibility for clinical trial enrollment. Measurement of the glomerular filtration rate (GFR) using clearance of exogenous markers provides the most accurate measure of kidney function but is not practical clinically. Although serum creatinine (sCr) level alone has been used to estimate kidney function, it can be influenced by non-GFR physiologic determinants, such as sarcopenia, and nephrology and oncology organization guidelines recommend against using sCr level alone. 1 Several equations incorporating clinical and demographic variables have been developed to improve the estimation of kidney function. 2 Recently, there have been calls for harmonization of most aspects of cancer clinical trial eligibility to foster feasibility, inclusivity, comparability, and generalizability. 3 We sought to understand the landscape of approaches used to estimate kidney function for contemporary cancer clinical trial eligibility.

In this cross-sectional study, ethics approval and consent to participate were not required because the institutional review board of the Icahn School of Medicine did not qualify this study as human research. We systematically searched the ClinicalTrials.gov database to review trial demographics and kidney eligibility criteria for phase 3 clinical trials. Trials were included if completed between October 28, 2013, and October 28, 2023, and if anticancer drugs for adults were evaluated. Trial demographics and methods of kidney function estimates used for trial eligibility were recorded (eMethods in Supplement 1 ). Descriptive statistical analysis was performed using R, version 4.3.2. This study followed the STROBE reporting guideline.

A total of 436 trials met criteria for review, and 231 trials, enrolling 111 424 patients, met the criteria for inclusion. Characteristics of the included trials are shown in the Table ; 139 of 197 (70.6%), 35 of 231 (15.1%), and 200 of 231 (86.5%) trials used the Cockcroft-Gault (CG) formula, sCr level alone, or either CG or sCr level to define kidney function eligibility, respectively ( Figure , A and B). Trends in methods used to estimate kidney function for trial eligibility over time are shown in the Figure , C.

Among 231 phase 3 cancer clinical trials in our analysis, we observed substantial heterogeneity in the methods used to estimate kidney function for clinical trial eligibility. Although professional societies recommend against its use, and despite an improved trend in recent years, 11 of 32 (34.4%) of cancer clinical trials since 2018 still use sCr level alone to define kidney function eligibility. 1 , 3 , 4 The CG formula was the most frequently used equation to estimate kidney function. The CG formula was developed in 1973 using data from 249 men and is known to underestimate kidney function in patients with cancer, and nephrology societies recommend modern approaches to estimate the GFR. 1 Contemporary measures, such as the Chronic Kidney Disease–Epidemiology Collaboration (CKD-EPI) and Modification of Diet in Renal Disease equations, were used in a few trials. A limitation of our study, however, is that we included only phase 3 trials, which often require less precise kidney function measurements than earlier phase trials.

Despite calls for harmonized clinical trial eligibility across multiple domains, no uniform guidelines currently exist regarding estimation of kidney function. Although various anticancer agents necessitate distinct kidney function thresholds based on their metabolism and toxicity profiles, standardization of methods to estimate kidney function for trial eligibility would foster a comprehensive understanding of the effect of kidney function on adverse events and cancer-related outcomes across clinical trials. Despite momentum to preferentially use the race-free CKD-EPI 2021 formula to estimate GFR, a preferred calculation to estimate kidney function is not well defined. 1 Certain anticancer drugs can also impair tubular secretion of sCr, making accurate estimation of the GFR more challenging. 5 Although cystatin C is another readily available biomarker that may improve GFR estimation, there is concern that certain cancers may affect cystatin C levels endogenouslly. 6 Together, our findings call for a collaborative effort among key stakeholders to establish a standardized approach to estimate kidney function to define cancer clinical trial eligibility.

Accepted for Publication: July 18, 2024.

Published: September 16, 2024. doi:10.1001/jamanetworkopen.2024.33387

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Karol AB et al. JAMA Network Open .

Corresponding Author: Alexander B. Karol, MD, Department of Medicine, Icahn School of Medicine at Mount Sinai, Mount Sinai Hospital, 1 Gustave L. Levy Pl, New York, NY 10029 ( [email protected] ).

Author Contributions: Dr Karol had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Karol, Fujiwara, Abramson, Galsky.

Acquisition, analysis, or interpretation of data: Karol, Paredes, Fujiwara, Argulian, Joshi, Galsky.

Drafting of the manuscript: Karol, Joshi, Galsky.

Critical review of the manuscript for important intellectual content: Karol, Paredes, Fujiwara, Argulian, Abramson, Galsky.

Statistical analysis: Karol, Paredes, Fujiwara, Joshi.

Administrative, technical, or material support: Karol, Argulian, Galsky.

Supervision: Abramson, Galsky.

Conflict of Interest Disclosures: Dr Galsky reported receiving personal fees from Bristol Myers Squibb, Merck, Genentech, AstraZeneca, Pfizer, EMD Serono, Seagen, Janssen, Fujiflim, Gilead, Asieris, Veracyte, Daiichi, Astellas, and Aktis outside the submitted work. No other disclosures were reported.

Data Sharing Statement: See Supplement 2 .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is Qualitative Research? | Methods & Examples

What Is Qualitative Research? | Methods & Examples

Published on June 19, 2020 by Pritha Bhandari . Revised on September 5, 2024.

Qualitative research involves collecting and analyzing non-numerical data (e.g., text, video, or audio) to understand concepts, opinions, or experiences. It can be used to gather in-depth insights into a problem or generate new ideas for research.

Qualitative research is the opposite of quantitative research , which involves collecting and analyzing numerical data for statistical analysis.

Qualitative research is commonly used in the humanities and social sciences, in subjects such as anthropology, sociology, education, health sciences, history, etc.

• How does social media shape body image in teenagers?
• How do children and adults interpret healthy eating in the UK?
• What factors influence employee retention in a large organization?
• How is anxiety experienced around the world?
• How can teachers integrate social issues into science curriculums?

Table of contents

Approaches to qualitative research, qualitative research methods, qualitative data analysis, advantages of qualitative research, disadvantages of qualitative research, other interesting articles, frequently asked questions about qualitative research.

Qualitative research is used to understand how people experience the world. While there are many approaches to qualitative research, they tend to be flexible and focus on retaining rich meaning when interpreting data.

Common approaches include grounded theory, ethnography , action research , phenomenological research, and narrative research. They share some similarities, but emphasize different aims and perspectives.

Note that qualitative research is at risk for certain research biases including the Hawthorne effect , observer bias , recall bias , and social desirability bias . While not always totally avoidable, awareness of potential biases as you collect and analyze your data can prevent them from impacting your work too much.

Prevent plagiarism. Run a free check.

Each of the research approaches involve using one or more data collection methods . These are some of the most common qualitative methods:

• Observations: recording what you have seen, heard, or encountered in detailed field notes.
• Interviews:  personally asking people questions in one-on-one conversations.
• Focus groups: asking questions and generating discussion among a group of people.
• Surveys : distributing questionnaires with open-ended questions.
• Secondary research: collecting existing data in the form of texts, images, audio or video recordings, etc.
• You take field notes with observations and reflect on your own experiences of the company culture.
• You distribute open-ended surveys to employees across all the company’s offices by email to find out if the culture varies across locations.
• You conduct in-depth interviews with employees in your office to learn about their experiences and perspectives in greater detail.

Qualitative researchers often consider themselves “instruments” in research because all observations, interpretations and analyses are filtered through their own personal lens.

For this reason, when writing up your methodology for qualitative research, it’s important to reflect on your approach and to thoroughly explain the choices you made in collecting and analyzing the data.

Qualitative data can take the form of texts, photos, videos and audio. For example, you might be working with interview transcripts, survey responses, fieldnotes, or recordings from natural settings.

Most types of qualitative data analysis share the same five steps:

• Prepare and organize your data. This may mean transcribing interviews or typing up fieldnotes.
• Review and explore your data. Examine the data for patterns or repeated ideas that emerge.
• Develop a data coding system. Based on your initial ideas, establish a set of codes that you can apply to categorize your data.
• Assign codes to the data. For example, in qualitative survey analysis, this may mean going through each participant’s responses and tagging them with codes in a spreadsheet. As you go through your data, you can create new codes to add to your system if necessary.
• Identify recurring themes. Link codes together into cohesive, overarching themes.

There are several specific approaches to analyzing qualitative data. Although these methods share similar processes, they emphasize different concepts.

Qualitative research often tries to preserve the voice and perspective of participants and can be adjusted as new research questions arise. Qualitative research is good for:

• Flexibility

The data collection and analysis process can be adapted as new ideas or patterns emerge. They are not rigidly decided beforehand.

• Natural settings

Data collection occurs in real-world contexts or in naturalistic ways.

• Meaningful insights

Detailed descriptions of people’s experiences, feelings and perceptions can be used in designing, testing or improving systems or products.

• Generation of new ideas

Open-ended responses mean that researchers can uncover novel problems or opportunities that they wouldn’t have thought of otherwise.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Researchers must consider practical and theoretical limitations in analyzing and interpreting their data. Qualitative research suffers from:

• Unreliability

The real-world setting often makes qualitative research unreliable because of uncontrolled factors that affect the data.

• Subjectivity

Due to the researcher’s primary role in analyzing and interpreting data, qualitative research cannot be replicated . The researcher decides what is important and what is irrelevant in data analysis, so interpretations of the same data can vary greatly.

• Limited generalizability

Small samples are often used to gather detailed data about specific contexts. Despite rigorous analysis procedures, it is difficult to draw generalizable conclusions because the data may be biased and unrepresentative of the wider population .

• Labor-intensive

Although software can be used to manage and record large amounts of text, data analysis often has to be checked or performed manually.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square goodness of fit test
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

There are five common approaches to qualitative research :

• Grounded theory involves collecting data in order to develop new theories.
• Ethnography involves immersing yourself in a group or organization to understand its culture.
• Narrative research involves interpreting stories to understand how people make sense of their experiences and perceptions.
• Phenomenological research involves investigating phenomena through people’s lived experiences.
• Action research links theory and practice in several cycles to drive innovative changes.

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

There are various approaches to qualitative data analysis , but they all share five steps in common:

• Prepare and organize your data.
• Review and explore your data.
• Develop a data coding system.
• Assign codes to the data.
• Identify recurring themes.

The specifics of each step depend on the focus of the analysis. Some common approaches include textual analysis , thematic analysis , and discourse analysis .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2024, September 05). What Is Qualitative Research? | Methods & Examples. Scribbr. Retrieved September 16, 2024, from https://www.scribbr.com/methodology/qualitative-research/

Is this article helpful?

Pritha Bhandari

Other students also liked, qualitative vs. quantitative research | differences, examples & methods, how to do thematic analysis | step-by-step guide & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 16 September 2024

Highly efficient CRISPR-mediated genome editing through microfluidic droplet cell mechanoporation

• You-Jeong Kim 1 , 2   na1 ,
• Dayoung Yun 3   na1 ,
• Jungjoon K. Lee 4 ,
• Cheulhee Jung 3 &
• Aram J. Chung   ORCID: orcid.org/0000-0003-4984-0222 1 , 2 , 5 , 6  

Nature Communications volume  15 , Article number:  8099 ( 2024 ) Cite this article

Metrics details

• Biomedical engineering
• Microfluidics

Clustered regularly interspaced short palindromic repeats (CRISPR)-based editing tools have transformed the landscape of genome editing. However, the absence of a robust and safe CRISPR delivery method continues to limit its potential for therapeutic applications. Despite the emergence of various methodologies aimed at addressing this challenge, issues regarding efficiency and editing operations persist. We introduce a microfluidic gene delivery system, called droplet cell pincher (DCP), designed for highly efficient and safe genome editing. This approach combines droplet microfluidics with cell mechanoporation, enabling encapsulation and controlled passage of cells and CRISPR systems through a microscale constriction. Discontinuities created in cell and nuclear membranes upon passage facilitate the rapid CRISPR-system internalization into the nucleus. We demonstrate the successful delivery of various macromolecules, including mRNAs (~98%) and plasmid DNAs (~91%), using this platform, underscoring the versatility of the DCP and leveraging it to achieve successful genome engineering through CRISPR–Cas9 delivery. Our platform outperforms electroporation, the current state-of-the-art method, in three key areas: single knockouts (~6.5-fold), double knockouts (~3.8-fold), and knock-ins (~3.8-fold). These results highlight the potential of our platform as a next-generation tool for CRISPR engineering, with implications for clinical and biological cell-based research.

Introduction

Genome editing has transformed biological research by enabling precise modifications to the genomes of various organisms, especially those of living cells 1 . This capability has allowed scientists to manipulate innate genetic functions, to their specific objectives, with unprecedented accuracy. Key genome editing technologies include zinc-finger nucleases (ZFNs), transcription activator-like effector nucleases (TALENs), and the clustered regularly interspaced short palindromic repeats (CRISPR)–Cas9 system 2 . Notably, the CRISPR–Cas9 system has garnered significant attention due to its simplicity, versatility, and high efficiency in genome manipulation, paving the way for complex and precise genomic engineering 3 . It should be mentioned that the efficacy of CRISPR as a therapeutic modality is contingent upon the successful internalization of the CRISPR systems into the targeted cells. This highlights the importance of developing a suitable delivery method and specifying the structural/morphological format of the CRISPR systems for optimal therapeutic outcomes. CRISPR-associated complexes can be delivered in various forms, including plasmid DNAs (pDNA), mRNAs encoding the Cas9 protein and single-guide RNAs (sgRNA), as well as ribonucleoprotein (RNP) complexes, which involve the fusion of sgRNA with the Cas9 protein 4 . Characterized by its stability and cost-effective large-scale production in laboratory settings, pDNA is a versatile molecular entity. Although its stability contributes to enhanced efficiency, it carries the risk of undesired prolonged expression of Cas9 and sgRNA, potentially increasing the probability of off-target effects 5 , 6 . Furthermore, transfection utilizing plasmids has been documented to induce cell death in specific cell types, including embryonic stem cells 6 , 7 . In contrast, mRNA delivery expedites the gene editing process, as transcription is not required 8 . Although mRNA has the advantage of reducing the chances of unintended genetic modifications when compared with that in pDNA, its low stability renders it susceptible to rapid degradation, reducing the editing efficiency. As an alternative, RNP has garnered widespread adoption as the preferred delivery format for the CRISPR–Cas9 engineering. Introducing RNP into the cells establishes a state ready for CRISPR activity, circumventing the transcription and translation processes required by other delivery forms, thereby enabling rapid editing 4 . In contrast to pDNA, RNP exhibits a relatively short lifespan, mitigating the risks associated with off-target editing 9 . Furthermore, RNP ensures greater stability against degradation when compared with that achieved with the mRNA delivery form 6 , 10 . Although delivering the CRISPR–Cas9 system as RNP offers numerous advantages, the internalization process has certain challenges, mainly due to the considerable size of the Cas9 protein (approximately 160 kDa), which necessitates delivery into the nucleus. Therefore, selection of an appropriate delivery method is of paramount importance 11 .

To facilitate effective RNP delivery for CRISPR-mediated applications, delivery strategies such as lipofection 12 , 13 , electroporation 14 , 15 , and cell-penetrating peptide 16 are commonly employed. Electroporation is predominantly used because of its operational simplicity and relatively high editing efficiency. It is a technique that creates transient nanopores in the cell membrane through electrical pulses, allowing the passage of biomolecules via electrophoretic migration 17 . This method has been demonstrated to be effective across a range of cell types. However, a notable drawback of electroporation is its dependence on high-voltage electrical pulses, which can result in substantial cell death and modulation of gene expression. Furthermore, reports indicate that electroporation alone often induces unintended DNA double-strand breaks, which significantly limits its potential for genomic editing applications 18 .

To address these limitations, several microfluidic platforms have been proposed to provide enhanced efficiency, consistency, stability, and throughput, along with reduced reagent consumption 19 , 20 , 21 . For example, microfluidic mechanoporation, using fluid–cell-structure interactions, has been utilized for RNP delivery. Deng et al. reported an inertial, microfluidic cell hydroporator, which enabled cell elongation to create discontinuities in the cell membrane. Successful editing of the COL11A1 gene in A2780cis cells 22 was demonstrated; however, the wedge-shaped design is susceptible to channel clogging. Another example is constriction-based microfluidic intracellular delivery, known as cell squeezing 23 , 24 , 25 . This platform passes cells through a series of narrow constrictions, allowing the delivery of biomolecules including mRNA and RNPs into different cell types. Since the delivery relies solely on the passive diffusion of RNPs into cells, a limited knockout efficiency via non-homologous end joining (NHEJ) was reported 23 . In addition, it has been documented that this diffusion-based method fails to perforate the nuclear envelope, preventing pDNA-based transfection 26 . Therefore, additional demonstrations and validations, such as pDNA transfection, highly effective knockout, large deletions through multiplexed knockouts, and gene insertion (knock-in via homology-directed repair (HDR)) are warranted. Furthermore, a newly developed method must exceed the capability and scalability of electroporation, which is the current standard for gene editing.

Here, we present a microfluidic platform called the droplet cell pincher (DCP), designed as a highly efficient and safe CRISPR-mediated genome editing tool. Our approach integrates droplet microfluidics with cell mechanoporation, allowing the encapsulation and controlled passage of cells and CRISPR–Cas9 RNPs through a microscale constriction. This process results in the creation of discontinuities in both the cell and nuclear membranes, facilitating the internalization of CRISPR–Cas9 RNPs into the nucleus. We demonstrated that this platform could be used for the delivery of diverse macromolecules, including mRNAs and pDNAs, underscoring the versatility of the DCP. Next, the DCP platform was successfully employed to perform various genome editing operations, including single knockouts, double knockouts, and single knock-ins. Notably, the editing efficiency surpassed that of electroporation, firmly validating its potential as a highly efficient gene editing tool.

Results and discussion

Device characterization.

The DCP platform operates on the principle of droplet cell mechanoporation, integrating droplet microfluidics with physical cell permeabilization. Recently, we introduced a droplet cell squeezing method involving the passage of droplets containing cells through a series of narrow constrictions 27 . Although the system exhibited high transfection/delivery performance in delivering various biomolecules and cell types, it often experienced channel clogging due to the presence of multiple constrictions and slow droplet passage speed. Furthermore, this method demonstrated suboptimal gene delivery outcomes, limiting its applicability to genomic engineering, particularly in CRISPR-mediated applications. A high flow-rate operation is an option to overcome these challenges; however, such flow conditions compromise stable droplet generation and cell encapsulation upstream. Therefore, we hypothesized that these problems could be fundamentally resolved by separately achieving stable droplet generation upstream and subsequently passing the droplets through a single constriction at a high speed downstream. Our redesigned platform not only enhances throughput and addresses clogging issues while preserving cell viability but also significantly improves delivery and transfection efficiency. This demonstrates that the proposed platform is capable of achieving various highly efficient CRISPR-mediated genome editing tasks that have not been accomplished before.

To this end, we designed a DCP platform presented in Fig.  1a, b and Supplementary Movie  1 . The cell suspension, mixed with the target biomolecules (e.g., CRISPR-Cas9 systems) and oil for droplet generation, was pumped independently into the microchannel using conventional syringe pumps. The microfluidic flow-focusing geometry facilitated the generation of uniform droplets, allowing stable droplet formation and upstream cell and biomolecule encapsulation. The generated droplets were accelerated by injecting an additional oil sheath flow. The accelerated droplets containing cells and biomolecules were then rapidly guided through a single constriction, inducing transient permeabilization of the membrane (Fig.  1c and Supplementary Movie  2 ). This high-speed cell mechanoporation process allows the convective internalization of external biomolecules into both the cytosol and nucleus through secondary flows developed in droplets, a capability that conventional cell squeezing methods, which rely solely on diffusion, do not possess. It should be noted that additional oil injection does not require additional biomolecules, and any excessive oil is fully recycled owing to the immiscibility and density difference between the two phases. Moreover, adopting a single constriction with a high-speed droplet passage substantially decreases the risk of channel clogging.

a Schematic representation of DCP. b High-speed microscope images demonstrating stable droplet generation with cell encapsulation, droplet acceleration through additional sheath flow, and droplet cell-pinching process. c Detailed illustration of the intracellular-delivery mechanism through droplet cell mechanoporation, encompassing encapsulation, pinching and transient cell deformation, and internalization. d Bright-field (BF) and fluorescence (FL) images showing the delivery of 3–5 kDa fluorescein isothiocyanate (FITC)-dextran into K562 cells via DCP (scale bar: 200 μm). e Confocal images of a K562 cell (scale bar: 5 μm). f Fluorescence intensity histograms and g Delivery efficiency of different FITC-dextran sizes into K562 cells. h 2000 kDa FITC-dextran delivery efficiency, and i Cell viability for various cell types. j 2000 kDa FITC-dextran delivery efficiency and k Cell viability as a function of cell concentration. l Plots of fluorescence intensity histograms, and m MFI fold change and cell viability for cell squeezing (SQZ) and DCP. All bars indicate mean ± standard error of the mean ( N  = 3 biological replicates). Each data point is based on flow cytometry analysis performed independently on different cell flasks. n.s. stands for no statistical difference; *, **, ***, and **** indicate P values below 0.05, 0.01, 0.001, and 0.0001, respectively. Statistical analyses were performed using one-way ANOVA and Tukey’s post-hoc tests for multiple comparisons. Source data are provided as a source data file. Detailed P values are provided in the source data file.

To validate the effectiveness of the DCP in internalizing the biomolecules into cells, we first attempted to deliver 0.3 mg/mL of 3–5 kDa fluorescein isothiocyanate (FITC)-conjugated dextran (FITC-dextran) into the K562 cells. FITC-dextran was specifically selected for its size variety, strong fluorescence, and minimal cell surface-binding characteristics attributed to its negative surface charge 28 . For evaluating the FITC-dextran delivery after processing the cells via the DCP, imaging of the K562 cells was conducted 18 h post-delivery. As shown in Fig.  1d, e , strong and uniform fluorescence signals were observed in the cytosol and nucleus. In contrast, the endocytosis group (positive control), comprising cells co-incubated with FITC-dextran during the DCP process, exhibited no detectable fluorescence, validating the success of the DCP-induced FITC-dextran delivery.

Next, we investigated the delivery dependence of the DCP platform on the cargo size. We postulated that a single yet rapid cell mechanoporation process could facilitate the effective internalization of large dextrans. To validate this, different molecular weights of 3–2000 kDa FITC-dextran, equivalent to a size range of approximately 3–50 nm, were delivered into the K562 cells under optimized fluidic and geometric conditions (a detailed description of the optimized parameters is presented in the next section). For quantitative characterization, we conducted a flow cytometry-based analysis. As shown in Fig.  1f, g , nearly 100% delivery efficiency was consistently achieved, regardless of the dextran size.

Additionally, we evaluated the effect of the cell type on the delivery efficiency. Under the same experimental conditions, 2000 kDa FITC-dextran was internalized into each cell type at the same concentration (0.3 mg/mL), with cell density of 2 × 10 7 cells/mL. The results depicted in Fig.  1h, i , consistently demonstrate highly efficient delivery exceeding 90%, with concurrent preservation of the cell viability at levels higher than 75% across all tested cell types. The fold changes in the mean fluorescence intensity (MFI) are plotted separately in Supplementary Fig.  1 . All these results highlight that the DCP platform can be used for both suspension and adherent cells in various delivery applications.

Next, we assessed the scalability of the DCP platform by evaluating the delivery efficiency as a function of the cell concentration. This investigation is crucial because high scalability should be guaranteed for an intracellular delivery platform to be used in cell-based therapy. For example, the production of ex vivo CAR-T cells for patient reinfusion requires approximately 1 × 10 8 cells per kg of body weight, indicating the necessity for high-throughput processing 29 . With the DCP, we can routinely process 2 × 10 7 cells/mL, a notably higher density than that possible with existing microfluidic platforms, which typically operate within the range of 10 5 –10 7 cells/mL 26 , 27 , 28 , 30 , 31 , 32 . At increased cell concentrations, the platform exhibited stable performance without compromising the delivery efficiency and viability, achieving levels as high as 6 × 10 7 cells/mL, as shown in Fig.  1j, k (refer to Supplementary Fig.  2 for MFI values). As the platform operates continuously with channel parallelization, it is expected to meet the increased scalability demands for cell-based therapeutic applications.

To characterize the performance of the DCP platform in comparison with that of the representative microfluidic delivery method, namely, cell squeezing 28 , under identical operational conditions, we delivered equivalent quantities of cargo (2000 kDa FITC-dextran) into the K562 cells. As illustrated in Fig.  1l, m , both scenarios (with and without the use of droplets) exhibited effective dextran delivery (see Supplementary Fig.  3 for the delivery efficiency) and no discernible differences in viability; nevertheless, a significant MFI fold change was evident in the DCP method (192.7-fold) when compared with that in cell squeezing (15.1-fold). Substantially higher delivery performance of the DCP platform was observed, which could be attributed to the effective concentration of dextran and secondary flows in droplets which enhance convective transport of cargo into cells 27 . Considering that the DCP microchannel is mostly filled with oil, the use of droplets decreases the aqueous volume; therefore, even with an equivalent amount of analytes, the cells are exposed to an environment with a higher concentration than that in cell squeezing (see Supplementary Fig.  4 ). Collectively, these results suggest that our DCP platform is a competitive solution with superior delivery and minimal analyte consumption.

Optimization of device operation

Delivery efficiency is critically influenced by the operational conditions, specifically the fluidic and geometric parameters, as illustrated in Fig.  2a . To identify the optimized operational conditions, our investigation initially focused on assessing the effects of the flow conditions on delivery efficiency, MFI fold change, and cell viability. The upstream flow condition was fixed for stable droplet generation and encapsulation 33 , and the cell density (2 × 10 7 cells/mL) and FITC-dextran size and concentration (2000 kDa and 0.3 mg/mL, respectively) were set identically. Under these circumstances, we first attempted to internalize dextran into the K562 cells with different oil sheath flow rates while utilizing the same microfluidic channel geometry with a width of 8 μm and length of 70 μm. The downstream oil sheath flow rate was systematically increased from 0 to 1.5 mL/h in increments of 0.5 mL/h. Subsequently, we calculated the corresponding delivery efficiency, MFI fold change, and cell viability. As depicted in Fig.  2b , nearly 100% delivery efficiency was achieved at all flow rates, including the case with no oil sheath fluid. For a comprehensive comparison of the delivery performance, MFI fold changes were calculated relative to those in the control group and the results are presented in Fig.  2c . The introduction of oil sheath fluids resulted in a significant increase in the MFI change. Broadly speaking, a higher oil sheath flow rate resulted in an increased MFI fold change; however, a decrease in MFI was observed as the flow rate was further elevated to 1.5 mL/h. Concurrently, the viability exhibited a monotonous decrease with increasing oil sheath flow rate; a substantial decrease in viability was observed at 1.5 mL/h (Fig.  2d ). Considering both aspects, it is postulated that at the highest oil sheath flow rate, excessive cell deformation is induced beyond the self-repairing capacity of the cell (i.e., membrane resealing), resulting in a diminished MFI fold change.

a Parameters of the DCP platform determining performance. Delivery efficiency, MFI fold change, and viability for different b – d sheath flow rates, e – g gap widths, and h – j gap lengths. All bars indicate mean ± standard error of the mean ( N  = 3 biological replicates). Each data point is based on flow cytometry analysis performed independently on different cell flasks. n.s. stands for no statistical difference; *, **, ***, and **** indicate P values below 0.05, 0.01, 0.001, and 0.0001, respectively. Statistical analyses were performed using one-way ANOVA and Tukey’s post-hoc tests for multiple comparisons. Source data and detailed P values are provided in the source data file.

To determine the optimal flow conditions for effective cargo delivery, we evaluated a new metric, defined as the product of the MFI fold change and cell viability. This metric serves as a practical measure of the net intracellular delivery of biomolecules, assuming both MFI fold change and viability are presented simultaneously. The calculated values for each tested flow rate of the oil sheath fluid were 117.8, 130.2, 143.7, and 110 (A.U.). Upon careful consideration of these calculations, a sheath fluid flow rate of 1.0 mL/h was identified as the optimal condition that maximizes the delivery efficiency while concurrently preserving the cell viability.

Subsequently, we investigated the influence of the geometric parameters of the microchannel on the overall delivery level and cell viability. As previously noted, a single constriction was employed to avoid potential clogging; therefore, we investigated two primary factors: the constriction (i.e., gap) width and length. Initially, three different constriction gap widths of 6, 8, and 10 μm were evaluated while maintaining a constant constriction length of 70 μm and sheath flow rate of 1.0 mL/h. As illustrated in Fig.  2e, f , close to 100% delivery efficiency was attained in delivering 2000 kDa FITC-dextran into the K562 cells; however, the MFI fold change decreased as the channel gap width increased. As expected, a reduction in cell viability was observed with decreasing gap width, as shown in Fig.  2g . The application of the metric describing the net intracellular delivery indicated that a constriction width of 8 μm was optimal.

We also examined the effect of the constriction length on dextran internalization into the K562 cells. The constriction length was varied from 40 to 100 μm, with a constant gap width of 8 μm and sheath flow rate of 1.0 mL/h while keeping other parameters unchanged. The width of 8 μm provides a sufficiently narrow gap for pinching the K562 cells; thus, it was observed that the delivery efficiency exceeded 99% across all cases, as shown in Fig.  2h . We hypothesized that increasing the constriction length would enhance the intracellular delivery level, potentially leading to reduced cell viability. As anticipated, the longest pinching length of 100 μm resulted in the largest fold change in MFI of 307.47 (Fig.  2i ). However, this enhancement in the MFI fold change was accompanied by the lowest viability of 36.8%, as shown in Fig.  2j . Furthermore, only a slight difference was observed in both the MFI fold change and viability ( p  < 0.5) at the pinching lengths of 40 and 70 μm. We postulated that the relatively short constriction lengths of 40 and 70 μm may not allow for maximal cell deformation, thus explaining the subtle differences. Nevertheless, when evaluating the product of the MFI fold change and viability, the pinching length of 70 μm was identified as the most suitable dimension.

In summary, for the delivery of 2000 kDa FITC-dextran into K562 cells, the optimal conditions were determined to be a flow rate of 1.0 mL/h, constriction width of 8 μm, and pinching length of 70 μm. These optimized conditions were employed for various biomolecule delivery experiments, as described in the subsequent sections.

Intracellular delivery of functional biomolecules through DCP

To assess the versatility of the DCP platform beyond its application to FITC-dextran delivery, we aimed to internalize functional biomolecules, such as eGFP-encoded mRNA and GFP-encoded pDNA, into the K562 cells. Transfection efficiency and cellular viability were investigated using the optimized operational conditions identified above. First, a 2 μg/mL of 996-nucleotide eGFP-mRNA construct was internalized into the K562 cells using the DCP platform, with varying mRNA concentrations. As shown in Fig.  3a and Supplementary Fig.  5a , uniform and strong fluorescent signals were observed only in the cells treated with the DCP platform, whereas negligible fluorescence was detected in the positive control.

a Bright-field and fluorescence images showing 2.0 μg/mL of eGFP-mRNA expression via endocytosis and DCP after 24 h. b Fluorescence intensity histograms, and c MFI fold change of mRNA expression as a function of mRNA concentration. d Bright-field and fluorescence images depicting the transfection of 50 μg/mL of GFP-encoding pDNA with a size of 7.9 kbp. e pDNA transfection efficiency of K562 cells with different pDNA sizes. f Comparison of transfection efficiency and MFI fold change for DCP and electroporation (EP). All bars indicate mean ± standard error of the mean ( N  = 3 biological replicates). Each data point is based on flow cytometry analysis performed independently on different cell flasks. n.s. stands for no statistical difference; *, **, ***, and **** indicate the P values below 0.05, 0.01, 0.001, and 0.0001, respectively. Statistical analyses were performed using one-way ANOVA. Source data are provided as a source data file. Detailed P values are provided in the source data file.

To quantify the transfection efficiency based on the mRNA concentration, 0.2, 2, and 20 μg/mL of mRNA were internalized using the DCP under the optimized delivery conditions (see Methods for more details). Subsequently, flow cytometry analysis, presented in Fig.  3b (with N cell  = 5000), was used to systematically measure the signal from each cell. The corresponding MFI fold changes are presented separately in Fig.  3c . At a concentration of 2 μg/mL, a transfection efficiency of ~99% and a 47-fold change in the MFI were achieved; the highest concentration resulted in a 726.1-fold change in the MFI. Notably, this transfection performance surpassed that of previously reported microfluidics-based mRNA transfection results based on substantially higher mRNA consumption and concentrations 27 , 30 , 31 , 34 , 35 .

Next, we prepared the GFP-encoding pDNAs of various sizes ranging from 4.1–9.1 kbp for delivery into the K562 cells. In contrast to mRNA, the transfection of cells with pDNA presents a greater challenge, as pDNA must pass not only through the cellular membrane but also navigate through the nuclear envelope. Moreover, owing to its distinctive morphological shape and larger molecular footprint, successful pDNA delivery presents a notably greater complexity. We hypothesized that the DCP platform can facilitate pDNA cell transfection. To validate this, 7.9 kbp pDNA was first internalized into the K562 cells. As depicted in Fig.  3d and Supplementary Fig.  5b , cells treated with the DCP platform exhibited high fluorescence intensity, indicating successful transfection. At a concentration of 50 μg/mL, the transfection efficiency was evaluated for different pDNA sizes, as shown in Fig.  3e . As anticipated, larger pDNA sizes posed greater challenges for internalization, resulting in a relatively lower transfection efficiency. However, it should be noted that conventional microfluidic platforms, which rely on cell squeezing, fail to demonstrate pDNA-based transfection 26 , 28 ; therefore, the DCP platform exhibits significant potential for application to DNA-mediated engineering.

To assess the performance in comparison with that of electroporation, we tested 4.1 kbp pDNA transfection using a concentration of 50 μg/mL while maintaining the same cell concentration for each method. While both methods appear to have similar transfection efficiencies, as shown in Fig.  3f (with the DCP method statistically exhibiting superior transfection efficiency), the MFI fold change revealed clear distinctions between the two approaches. Specifically, the DCP method achieved an approximately 286.7-fold increase, surpassing that of electroporation, which resulted in an approximately 115-fold increase. In addition to comparing the transfection performance, we analyzed the changes in the cell shape following treatment with DCP and electroporation. Given the documented evidence that the cell mechanoporation process has superior capability to preserve the cell functionality and stability than electroporation 36 , our objective was to directly compare the alterations in the cell morphological phenotype after treatment. As can be seen in Supplementary Fig.  6 , we confirmed that electroporation induced more significant alterations, consistent with previous findings 35 . In short, these differences clearly highlight that the DCP platform is a superior and safer alternative than electroporation.

CRISPR–Cas9-mediated genome editing

The sgRNA guides Cas9 to the targeted DNA sequence, creating double-strand breaks (DSBs). Subsequently, two primary repair mechanisms are activated, NHEJ and HDR 37 . As illustrated in Fig.  4a , NHEJ repairs the DSBs by directly rejoining the ends, often resulting in small insertion and deletion mutations (indels). In contrast, the HDR mechanisms facilitate precise insertion of a foreign genetic sequence at a specific genomic locus. This process involves the utilization of external DNA or RNA sequences introduced alongside the CRISPR–Cas9 complex, which serves as a template for the precise integration of genetic material at the designated genomic site (see Fig.  5a ).

a Schematic of NHEJ repair pathway. Percentage of indel formation verified by T7E1 assay using b Electroporation (EP), DCP, and c Lipofection (LP). d Editing efficiency of each method. e PCR product sequencing data for the EMX1 targeting region in K562 cells. Representative sequences for indels. The inserted sequence is highlighted in red, and the deletions are denoted by the gray box. f Ratio between unmodified and modified genes quantified using next-generation sequencing (NGS). g Schematic of the dual-DSB approach and ligation. h Bands representing the amplified multiplex products using electroporation and DCP for comparison. A 50 bp DNA ladder (left, NEB N3236S) and a 1 kb DNA ladder (right, NEB N3232S) were used as size markers. i Relative band intensity of EP and DCP. j PCR product sequencing data for the EMX1 targeting region. Representative sequences for large deletions (with or without indels). k Ratio between unmodified and modified genes quantified using NGS. All bars indicate mean ± standard error of the mean ( N  = 3 biological replicates). Each data point is based on gel image analysis performed independently on different cell flasks. n.s. stands for no statistical difference; *, **, ***, and **** indicate P values below 0.05, 0.01, 0.001, and 0.0001, respectively. Statistical analyses were performed using one-way ANOVA. Source data and detailed P values are provided as the source data file.

a Schematic of the HDR repair pathway. b Restriction enzyme (RE)-cleaved PCR products indicative of HDR. c Quantification of HDR frequency targeting EMX1 in K562 cells. d Representative sequences for accurate insertion of donor-template sequences and other indels. e Ratio between unmodified and modified genes, including those with accurate insertion, quantified using NGS. All bars indicate mean ± standard error of the mean ( N  = 3 biological replicates). Each data point is based on gel image analysis (RE assay) performed independently on different cell flasks. n.s. stands for no statistical difference; *, **, ***, and **** indicate P values below 0.05, 0.01, 0.001, and 0.0001, respectively. Statistical analyses were performed using one-way ANOVA. Source data are provided as a source data file. Detailed P values are provided in the source data file.

We hypothesized that our DCP platform could effectively facilitate both NHEJ for knockout and HDR for knock-in applications. By employing the optimized operational conditions, we attempted to deliver the CRISPR–Cas9 RNPs at a concentration of 100 μg/mL into the K562 cells, targeting the EMX1 locus (see the Methods section for more details). The CRISPR–Cas9 RNP complex was prepared and directly delivered into the cells through the DCP while ensuring a uniform cell density of 2 × 10 7 cells/mL in all the experiments. Note that genomic editing can only be achieved if the RNP complex is localized in the nucleus. Following the delivery of the RNPs into the cells, the cells were collected after 72 h of incubation, providing sufficient time for the RNPs to engage and induce NHEJ or HDR within the nucleus. Subsequently, for editing efficiency characterization, the cells were harvested to extract the DNA. For a direct one-to-one comparison with other benchtop methods, including electroporation (EP) and lipofection (LP) (Figs.  4 and 5 ), we consistently employed the same experimental conditions unless specified otherwise.

The EMX1 knockout efficiency was evaluated using the T7E1 assay, which is known for its effectiveness in detecting indels greater than 1 nt in length 38 . As shown in Fig.  4b–d , the analysis of the average band intensities indicated an indel frequency of 1.3% for lipofection, 7.6% for electroporation, and over 50% for DCP, thereby demonstrating an editing frequency approximately twenty times greater than that of lipofection and efficiency approximately five times higher than that of electroporation.

To precisely validate the successful occurrence of indels at the target loci facilitated by the DCP, we conducted next-generation sequencing (NGS), specifically focusing on indels within 50 base pairs from the target site (Fig.  4e, f ). Among the reads with no modification, 37.84% (3,227,127 reads) were identified, whereas the remaining reads exhibited evident indel events, accounting for 62.16% knockout editing efficiency (slightly higher than that in the T7E1 analysis). Notably, the most prevalent alteration involved adenine insertions, accounting for 23.8% (2,025,764 reads), followed by 11.1% (951,250 reads), which were characterized by deletions of six nucleotide sequences (Supplementary Fig.  7a ).

Next, we assumed that the DCP platform could generate dual cleavage by facilitating the delivery of two distinct CRISPR–Cas9 RNPs into the nucleus, thereby enabling multiplexed genome editing, as illustrated in Fig.  4g . The assessment of multiplexed NHEJ efficiency was conducted through the internalization of two separate RNP complexes, each with a concentration of 50 μg/mL. Two distinct loci within EMX1 were targeted and delivery was achieved using the DCP platform and an electroporator. Lipofection was excluded owing to its inferior editing efficiency (Fig.  4c, d ) in the single RNP editing trials. Following the induction of DSBs at the designated sites, deletions were observed within the genomic sequences located between the targeted loci. Subsequent ligation mechanisms were activated to repair the genomic remnants resulting from the induced DSBs. Multiplexing efficiency was indirectly evaluated by quantifying the amplification of the cleaved region using polymerase chain reaction (PCR) primers. The primers were designed to amplify a 2069 bp segment encompassing the wild-type target site, which facilitated the detection of bands at the specified “NC” location. The induction of two DSBs led to the targeted deletion of a sizable genomic fragment measuring 1355 bp, as intended (Fig.  4h ). Consequently, a 714 bp residual fragment emerged, which was attributable to either direct ligation or potentially small-scale deletions and insertions. Gel electrophoresis of the PCR amplicons revealed distinct bands, confirming successful genome editing at the anticipated 714 bp location. The identification of small-scale insertions and deletions based on band patterns posed a challenge, whereas larger-scale insertions were evident as smeared band patterns located prominently above the 714 bp threshold.

To assess the editing efficiencies of DCP and electroporation, the band intensities at the specific positions were analyzed for samples prepared using equivalent amounts of amplified genomic DNA (gDNA). The results indicated average values of 11.9 and 45.3 for electroporation and DCP, respectively (Fig.  4i ). In comparison to electroporation, DCP demonstrated approximately 3.8-fold higher efficiency in achieving successful double knockouts, indicating its superior multiplexing editing efficiency over the current state-of-the-art method. Although agarose gel electrophoresis facilitated a qualitative comparison of the two platforms, quantifying the exact efficacies of both CRISPR-Cas9 RNPs remained challenging. Therefore, NGS was employed to scrutinize the modifications within the target region (Fig.  4j, k ). The analysis revealed that only 6.36% of the amplicons remained unmodified, whereas 93.64% exhibited various forms of modification. Direct ligation following cleavage was observed in 6.26% of the cases. The remaining modifications encompassed various sizes of indels, with 6.9% sequence deletions and 53.8% sequence insertions (Supplementary Fig.  7b ).

As another key example of genome editing, we demonstrated gene knock-in through HDR. It is worthwhile to mention that there are limited reports on knock-in demonstrations via microfluidics. Therefore, we hypothesized that our DCP platform is capable of effectively inserting genes at the desired locations. The initial objective was to deliver the same RNP targeting EMX1 , accompanied by single-stranded DNA (ssDNA) serving as the HDR donor template (112 nt) (see Fig.  5a ). This ssDNA template contained a sequence recognized by the restriction enzyme (RE), Eco-RV HF, which facilitates the convenient calculation of the insertion efficiency. Considering that gene insertion is conventionally less efficient than deletion, as reported in a previous study 39 , we opted to increase the concentration of RNP two-fold during the delivery process.

When assessing the gene insertion efficiency by comparing the band intensities in gel electrophoresis and RE assay, electroporation demonstrated 3.9% efficiency, whereas DCP exhibited a significantly higher efficiency of 14.7%, yielding an approximately 3.8-fold increase in the insertion efficiency (Fig.  5b, c ). Subsequently, NGS analysis was conducted to validate the precision of gene insertion (Supplementary Fig.  7c ). The observed occurrences of various types of indel modifications within HDR accounted for 86.03%, as shown in Fig.  5d, e . Notably, instances where the donor template sequence was accurately inserted without indels and subsequently ligated comprised 28% of the outcomes, indicating that a significantly higher editing was achieved than that obtained from the RE assay. It is important to acknowledge that the HDR efficiency can be further improved by synchronizing and capturing cells in the S and G2 phases 40 ; therefore, we expect that the DCP platform has the potential to achieve even greater efficiency. Thus, this outcome underscores the immense potential of the DCP platform for diverse applications in genome editing, providing enhanced efficiency, safety, and cost-effectiveness compared to electroporation.

The genome editing process begins with internalization of the CRISPR–Cas system into the cells. However, the editing efficiency is constrained by the delivery method; this is recognized as a significant obstacle that prevents the realization of the full potential of the editing process. In this study, we developed a microfluidic gene delivery platform, which is a highly efficient genome editing tool capable of gene deletion and insertion along with multiplexing. Our current study primarily demonstrated genetic engineering using CRISPR–Cas9 systems. However, it is crucial to emphasize that the platform is adaptable to next-generation genome editing approaches, such as base editors 41 and prime editors 42 . Given its capacity for facile macromolecule internalization, this platform holds considerable promise, particularly for prime editing, which involves proteins of substantial size (~240 kDa), albeit these are smaller than those tested in this study 43 . Furthermore, our platform demonstrated highly effective mRNA and pDNA cell transfection, underscoring its versatility. In contrast to other microfluidic technologies, our platform leverages the unique secondary flows developed within droplets and employs highly rapid cell mechanoporation to achieve high levels of delivery. Most notably, a comparison with electroporation, which is the current standard for gene editing, indicated that the proposed platform outperforms electroporation in terms of both editing and transfection results. This confirmed the high potential of the DCP platform to establish a new standard for ex vivo genome editing. Additionally, as we previously mentioned, the DCP platform relies on cell mechanoporation for delivery; therefore, it is anticipated to achieve greater cell functionality and stability than electroporation 36 .

While the DCP platform utilizes droplet microfluidics and requires an additional demulsification step for cell retrieval, this requirement can be effectively addressed through system automation, unlike with other microfluidic platforms 31 , 44 . Nonetheless, the DCP platform distinguishes itself due to its near clog-free operation, high scalability, low analyte consumption, elevated editing/transfection efficiency, superior cell viability, and cost-effectiveness. In summary, we introduced a microfluidic platform for gene delivery that holds promise as a next-generation tool for CRISPR engineering, with significant implications for clinical and biological cell-based research.

Microfabrication

The microchannel mold was fabricated using conventional photolithography with an SU-8 photoresist (Microchem Corp., USA). Polydimethylsiloxane (PDMS Sylgard 184, USA) channels were replicated using standard soft lithography. The inlets and outlets were created by punching holes into the PDMS using a pin vise. Subsequently, oxygen-plasma treatment (CUTE, Femto Science, South Korea) was applied to bond the PDMS to a standard glass slide. The bonded chips underwent a baking process for a minimum of 24 h in a 75 °C oven, ensuring robust adhesion.

Cell culture

K562 (KCLN no.: 10243) and Jurkat (KCLN no.: 40152) cells were purchased from the Korean Cell Line Bank. HEK 293 T (Cat. no.: CRL-3216) cells were obtained from the ATCC. NK-92 cells were generously provided by the group associated with Professor J. Doh at Seoul National University (South Korea). The K562 and Jurkat cells were cultured according to the standard protocols in RPMI-1640 medium (Corning, USA) and supplemented with 10% fetal bovine serum (FBS; Gibco, USA) and 1% penicillin–streptomycin (Gibco, USA). HEK 293 T cells were cultured in DMEM with the same supplements. NK-92 cells were grown in an RPMI 1640 medium with L-glutamine (Corning, USA) supplemented with 10% FBS, 1% penicillin–streptomycin, and 100 UI/ml IL-2 (PeproTech, USA). All cell types were incubated, maintained, and cultured at 37 °C with 5% CO 2 .

Intracellular delivery procedure

The droplet generation oil (Bio-Rad, USA) was filtered using a 0.2 μm polytetrafluoroethylene syringe filter (Advantec, Taiwan). The cells were mixed with the target nanomaterials and loaded into 1–3 mL Luer-Lok plastic syringes (BD, USA). Opti-MEM (Thermo Fisher Scientific, USA) was used for 996 nt eGFP-mRNA (TriLink Biotechnologies, USA), pDNA, and CRISPR–Cas9 RNP delivery. FITC-dextran (3–2000 kDa; Sigma-Aldrich, USA) was resuspended in a complete cell medium at a concentration of 2 × 10 7 cells/mL. pCMV-T7-eGFP (4.119kbp, BPK1098) pDNA (cat. No. 133962), 7.9 kbp pGreenPuro pDNA (cat. no. SI505A-1), and 9.1kbp pDNA with the backbone vector pCDH-EF1-MCS-T2A-copGFP (System Biosciences, USA) were kindly provided by the laboratory under Professor S. Cho at Konkuk University (South Korea).

The sgRNA sequence was designed as 5’ GAGTCCGAGCAGAAGAAGAA 3’ to target the EMX1 (chromosome 2, 72933853 – 72933872, NCBI reference) site using chop-chop and Synthego (USA). For NHEJ demonstration, the Cas9 RNP complex was prepared using 30 μM endotoxin-free Cas9 protein, provided by Prof. J. Lee at NUS (Singapore), and 100 μM sgRNA purchased from Synthego. The 500 pmol RNP complex was formed by mixing the components at the same molar ratio for 10 m at room temperature in 1 mL Opti-MEM. In the case of NHEJ multiplexing, an additional sgRNA (5’ AGTAAAGAACCACGGAGTCA 3’) targeting a different site of EMX1 was used. Two types of RNP complexes, each consisting of 250 pmol, were prepared in separate tubes in 500 μL of Opti-MEM using the method mentioned above. For HDR, a 112 nt ssDNA donor template (5’ TACAAACGGCAGAAGCTGGAGGAGGAAGGGCCTGAGTCCGAGCAGAAGAAGATATCAAGCTTGAAGGGCTCCCATCACATCAACCGGTGGCGCATTGCCACGAAGCAGGCCA 3’) containing the restriction enzyme site was utilized, purchased from Integrated DNA Technologies (IDT; USA). After preparing the RNP complex as described above, it was mixed with an equal amount of 500 pmol donor template and transfected into cells. During the experiments, the fluid flow was regulated using syringe pumps (Harvard Apparatus, USA). 1H,1H,2H,2H-Perfluoro-1-octanol (PFO, Thermo Fisher Scientific) was used to demulsify the droplets, allowing for cell retrieval. Subsequently, the collected cells were washed with Dulbecco’s phosphate-buffered saline (DPBS; Modified, Hyclone, USA) and resuspended in the standard cell medium. Incubation durations of 18, 24, and 72 h were implemented for post-analysis of the delivery of FITC-dextran, nucleic acid transfection, and genome editing, respectively. Cell viability was assessed using a 0.4% trypan blue solution (Lonza, Switzerland).

Delivery efficiency calculation

Delivery or transfection efficiency is defined as the portion of the fluorescence signals surpassing the threshold, corresponding to the top 5% of fluorescence of the control group.

Electroporation and lipofection

The Neon transfection system (Thermo Fisher, USA) and CRISPRMAX reagent (CMAX00003, Invitrogen, USA) were used for electroporation and lipofection, respectively. The electroporation parameters, specifically those outlined in recipe #22 for K562 cells, were used according to the manufacturer protocol. Lipofection assays were performed according to the guidelines provided by the manufacturer.

Flow cytometry

The cells were harvested and suspended in DPBS for subsequent flow cytometry analyses. Flow cytometry was performed using a Guava EasyCyte flow cytometer from Luminex, USA. Cells of the control group were initially gated on their FSC and SSC to exclude debris and dead cells. The fluorescence of the gated cells was then measured using flow cytometry. An example of the flow cytometry gating scheme is provided in Supplementary Fig.  8 .

DNA substrate construction and primer design

gDNA was extracted from the cell using the G-spin total DNA extraction kit (Intron Biotechnology, South Korea). Primers for the experiments were purchased from IDT, and their sequences of primers were as follows: EMX1 FP; GGG TCA TAG GCT CTC TCA TTT AC, EMX1 RP; CCA TTG CTT GTC CCT CTG T, Multiplex EMX1 FP; CCA TTG CTT GTC CCT CTG T, Multiplex EMX1 RP; CCA TTG CTT GTC CCT CTG T. The target region was amplified from 150 ng gDNA using the Q5 high-fidelity DNA polymerase (NEB, USA) and the specified primers. For the NHEJ and HDR samples, the PCR program consisted of an initial denaturation step at 98 °C for 30 s, followed by 30 cycles of denaturation at 98 °C for 10 s, annealing at 65 °C for 20 s, extension at 72 °C for 20 s, and a final extension at 72 °C for 2 m. The PCR program for the multiplex editing sample involved an additional extension step at 72 °C for 1 m. The final PCR product was purified using the Monarch PCR and DNA cleanup kit (NEB, USA).

The purified 200 ng PCR product was denatured at 95 °C for 5 m, followed by re-annealing through a two-step ramping process: first from 95 to 85 °C at a rate of 2 °C/s, and then from 85 to 25 °C at a rate of 0.1 °C/s. The resulting heteroduplexed PCR product was subjected to digestion at 37 °C for 15 m with 10 units of T7 endonuclease 1 (NEB, USA). Finally, the analysis was performed using 1.5% TBE agarose gel electrophoresis. DNA sample was loaded together with low molecular weight DNA ladder (NEB N3233S).

Restriction enzyme assay

For HDR analysis using restriction enzyme digestion, the 200 ng purified PCR product was incubated with 5 units of EcoRV-HF(NEB) in rCutSmart buffer at 37 °C. After 2 h, the reaction was stopped with heat inactivation at 65 °C for 20 min. The product was analyzed by electrophoresis using 1.5% TBE agarose gel.

NGS sample preparation

For targeted deep sequencing analysis, a DNA substrate was prepared by following the upper DNA substrate construction protocol using distinct primer sets: EMX1 NGS FP; GCCTCCTGAGTTTCTCATCTG, EMX1 NGS RP; CTAGTCATTGGAGGTGACATCG, Multiplex EMX1 NGS FP; TCTGGAGCAAGAATCCAAGAG, Multiplex EMX1 NGS RP; CTAGTCATTGGAGGTGACATCG. The resulting PCR product was purified using a Monarch PCR and DNA cleanup kit and subsequently subjected to deep sequencing analysis. The genomic sites of interest were amplified from genomic DNA samples and sequenced on a NovaSeq 6000 system (Illumina, USA).

High-speed microscopy, and image and data processing

High-speed images were recorded using a Phantom VEO 701 L high-speed camera (Vision Research, USA), and fluorescence images were captured using a standard inverted microscope (Axio Observer A1, Carl Zeiss, Germany). Both the high-speed and fluorescence images were analyzed using the ImageJ software ( https://imagej.nih.gov/ij/ ). Guava flow cytometry data were analyzed using Guava software, Incyte 3.3.

Statistical analysis and reproducibility

For all experiments, at least three independent biological replicates were used unless otherwise noted. Statistical analysis was performed using a one-way ANOVA and Tukey’s HSD post-hoc test for multiple comparisons using SPSS Statistics 27 (IBM, USA). A P value of < 0.05 was considered as statistically significant. All data were replotted using the OriginPro software (OriginLab, USA). Error bars and symbols in the plots represent the mean ± standard error of the mean.

Confocal microscopy

Fluorescent-stained cells and internalized 3 kDa FITC were imaged using a confocal laser scanning microscope (Zeiss, Germany). The cells were fixed with 4% formaldehyde and stained with DAPI. Subsequently, the processed cells were affixed onto a poly- l -lysine (PLL)-coated glass slide covered with a mounting solution for confocal imaging.

Reporting summary

Further information on research design is available in the  Nature Portfolio Reporting Summary linked to this article.

Data availability

All data generated or analyzed during this study are included in this published article and its supplementary information files. The NGS data generated in this study have been deposited in the NCBI Sequence Read Archive (SRA) under accession numbers SRR28563326 , SRR28563327 , SRR28563328 , and SRR28563329 , associated with NCBI BioProject PRJNA1096627 . Additionally, the GenBank accession number for the relevant sequences is PP645754 . All datasets are publicly accessible and will be permanently available, with no restrictions on data availability. Due to its size, the raw flow cytometry data is available upon request, and requests will be fulfilled within 2 weeks.  Source data are provided with this paper.

Gaj, T., Gersbach, C. A. & Barbas, C. F. 3rd. ZFN, TALEN, and CRISPR/Cas-based methods for genome engineering. Trends Biotechnol 31 , 397–405 (2013).

Article   PubMed   PubMed Central   Google Scholar  

Gupta, R. M. & Musunuru, K. Expanding the genetic editing tool kit: ZFNs, TALENs, and CRISPR-Cas9. J. Clin. Investig. 124 , 4154–4161 (2014).

Doudna, J. A. & Charpentier, E. The new frontier of genome engineering with CRISPR-Cas9. Science 346 , 1258096 (2014).

Article   PubMed   Google Scholar  

Fajrial, A. K., He, Q. Q., Wirusanti, N. I., Slansky, J. E. & Ding, X. A review of emerging physical transfection methods for CRISPR/Cas9-mediated gene editing. Theranostics 10 , 5532–5549 (2020).

Rouet, R. et al. Receptor-Mediated Delivery of CRISPR-Cas9 endonuclease for cell-type-specific gene editing. J. Am. Chem. Soc. 140 , 6596–6603 (2018).

Kim, S., Kim, D., Cho, S. W., Kim, J. & Kim, J. S. Highly efficient RNA-guided genome editing in human cells via delivery of purified Cas9 ribonucleoproteins. Genome Res . 24 , 1012–1019 (2014).

Hughes, T. S. et al. Immunogenicity of intrathecal plasmid gene delivery: cytokine release and effects on transgene expression. J. Gene. Med. 11 , 782–790 (2009).

Yin, K., Gao, C. & Qiu, J.-L. Progress and prospects in plant genome editing. Nat. Plants 3 , 17107 (2017).

Liang, Z. et al. Efficient DNA-free genome editing of bread wheat using CRISPR/Cas9 ribonucleoprotein complexes. Nat. Commun. 8 , 14261 (2017).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Hendel, A. et al. Chemically modified guide RNAs enhance CRISPR-Cas genome editing in human primary cells. Nat. Biotechnol. 33 , 985–989 (2015).

Chen, G. et al. A biodegradable nanocapsule delivers a Cas9 ribonucleoprotein complex for in vivo genome editing. Nat. Nanotechnol 14 , 974–980 (2019).

Zuris, J. A. et al. Cationic lipid-mediated delivery of proteins enables efficient protein-based genome editing in vitro and in vivo. Nat. Biotechnol. 33 , 73–80 (2015).

Yu, X. et al. Improved delivery of Cas9 protein/gRNA complexes using lipofectamine CRISPRMAX. Biotechnol. Lett. 38 , 919–929 (2016).

Yang, R. et al. Monoclonal cell line generation and CRISPR/Cas9 manipulation via single-cell electroporation. Small 14 , 1702495 (2018).

Article   Google Scholar  

Sido, J. M. et al. Electro-mechanical transfection for non-viral primary immune cell engineering. bioRxiv , 2021.2010.2026.465897 (2021).

Suresh, B., Ramakrishna, S. & Kim, H. in Eukaryotic Transcriptional and Post-transcriptional Gene Expression Regulation (ed. Wajapeyee, N. & Gupta, R.), pp. 81-94 (Springer New York, 2017).

Boukany, P. E. et al. Nanochannel electroporation delivers precise amounts of biomolecules into living cells. Nat. Nanotechnol. 6 , 747–754 (2011).

Article   ADS   PubMed   Google Scholar  

Meaking, W. S., Edgerton, J., Wharton, C. W. & Meldrum, R. A. Electroporation-induced damage in mammalian cell DNA. Biochim. Biophys. Acta, Gene Struct. Expression 1264 , 357–362 (1995).

Stewart, M. P., Langer, R. & Jensen, K. F. Intracellular delivery by membrane disruption: mechanisms, strategies, and concepts. Chem. Rev. 118 , 7409–7531 (2018).

Sun, M. & Duan, X. Recent advances in micro/nanoscale intracellular delivery. Nanotechnol. Precis. Eng. (NPE) 3 , 18–31 (2019).

Hur, J. & Chung, A. J. Microfluidic and nanofluidic intracellular delivery. Adv. Sci. 8 , 2004595 (2021).

Deng, Y. et al. Intracellular Delivery of nanomaterials via an inertial microfluidic cell hydroporator. Nano Lett . 18 , 2705–2710 (2018).

Han, X. et al. CRISPR-Cas9 delivery to hard-to-transfect cells via membrane deformation. Sci. Adv. 1 , e1500454 (2015).

Yu, T. et al. Development of a microfluidic cell transfection device into gene-edited CAR T cell manufacturing workflow. Lab Chip . 23 , 4804–4820 (2023).

Sevenler, D. & Toner, M. High throughput intracellular delivery by viscoelastic mechanoporation. Nat. Commun. 15 , 115 (2024).

Ding, X. et al. High-throughput nuclear delivery and rapid expression of DNA via mechanical and electrical cell-membrane disruption. Nat. Biomed. Eng. 1 , 0039 (2017).

Joo, B., Hur, J., Kim, G.-B., Yun, S. G. & Chung, A. J. Highly efficient transfection of human primary T lymphocytes using droplet-enabled mechanoporation. ACS Nano 15 , 12888–12898 (2021).

Sharei, A. et al. A vector-free microfluidic platform for intracellular delivery. Proc . Natl Acad. Sci. USA 110 , 2082–2087 (2013).

Article   ADS   Google Scholar  

Qiu, H.-Y., Ji, R.-J. & Zhang, Y. Current advances of CRISPR-Cas technology in cell therapy. Cell Insight 1 , 100067 (2022).

Hur, J. et al. Microfluidic cell stretching for highly effective gene delivery into hard-to-transfect primary cells. ACS Nano 14 , 15094–15106 (2020).

Liu, A. et al. Microfluidic generation of transient cell volume exchange for convectively driven intracellular delivery of large macromolecules. Mater. Today 21 , 703–712 (2018).

Szeto, G. L. et al. Microfluidic squeezing for intracellular antigen loading in polyclonal B-cells as cellular vaccines. Sci. Rep. 5 , 10276 (2015).

Utada, A. S., Fernandez-Nieves, A., Stone, H. A. & Weitz, D. A. Dripping to jetting transitions in coflowing liquid streams. Phys. Rev. Lett. 99 , 094502 (2007).

Jarrell, J. A. et al. Intracellular delivery of mRNA to human primary T cells with microfluidic vortex shedding. Sci. Rep. 9 , 3214 (2019).

Kang, G. et al. Intracellular nanomaterial delivery via spiral hydroporation. ACS Nano 14 , 3048–3058 (2020).

Hur, J. et al. Genetically stable and scalable nanoengineering of human primary T cells via cell mechanoporation. Nano Lett . 23 , 7341–7349 (2023).

Featherstone, C. & Jackson, S. P. DNA double-strand break repair. Curr. Biol. 9 , R759–R761 (1999).

Sentmanat, M. F., Peters, S. T., Florian, C. P., Connelly, J. P. & Pruett-Miller, S. M. A Survey of validation srategies for CRISPR-Cas9 editing. Sci. Rep. 8 , 888 (2018).

Liu, C., Zhang, L., Liu, H. & Cheng, K. Delivery strategies of the CRISPR-Cas9 gene-editing system for therapeutic applications. J. Control. Release 266 , 17–26 (2017).

Liu, M. et al. Methodologies for Improving HDR Efficiency. Front. Genet . 9 https://doi.org/10.3389/fgene.2018.00691 (2019).

Komor, A. C., Kim, Y. B., Packer, M. S., Zuris, J. A. & Liu, D. R. Programmable editing of a target base in genomic DNA without double-stranded DNA cleavage. Nature 533 , 420–424 (2016).

Anzalone, A. V. et al. Search-and-replace genome editing without double-strand breaks or donor DNA. Nature 576 , 149–157 (2019).

Wolff, J. H. & Mikkelsen, J. G. Prime editing in hematopoietic stem cells—From ex vivo to in vivo CRISPR-based treatment of blood disorders. Front. genome . 5 . https://doi.org/10.3389/fgeed.2023.1148650 (2023)

Aghaamoo, M. et al. High-throughput and dosage-controlled intracellular delivery of large cargos by an acoustic-electric micro-vortices platform. Adv. Sci. 9 , 2102021 (2022).

Download references

Acknowledgements

The authors thank Mr. H. S. Lee and all members of the Biomicrofluidics Laboratory at Korea University for useful discussions and Prof. I. Choi and Mr. H. Kim at the University of Seoul for their technical support. This study was supported by the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT; Ministry of Science and ICT) (2021R1A2C2006224, RS-2023-00242443, and RS-2023-00218543, A.J.C.), Technological Innovation R&D Program (RS-2023-00262758, A.J.C.) funded by the Ministry of SMEs and Startups (MSS, Republic of Korea), and Materials/Parts Technology Development Program (20020278, A.J.C.) funded by the Ministry of Trade, Industry & Energy (MOTIE, Republic of Korea).

Author information

These authors contributed equally: You-Jeong Kim, Dayoung Yun.

Authors and Affiliations

Department of Bioengineering, Korea University, Seoul, Republic of Korea

You-Jeong Kim & Aram J. Chung

Interdisciplinary Program in Precision Public Health (PPH), Korea University, Seoul, Republic of Korea

Department of Biotechnology, Korea University, Seoul, Republic of Korea

Dayoung Yun & Cheulhee Jung

Department of Biochemistry, National University of Singapore, Singapore, Singapore

Jungjoon K. Lee

School of Biomedical Engineering, Korea University, Seoul, Republic of Korea

Aram J. Chung

MxT Biotech, Seoul, Republic of Korea

You can also search for this author in PubMed   Google Scholar

Contributions

Y-J.K. and A.J.C. conceived the research, contributed to the data analysis, and designed the experiments. Y-J.K. conducted all microfluidics-associated experiments. D.Y.Y. and C.J. designed the CRISPR systems, and D.Y.Y. performed CRISPR analyses. Y-J.K. and A.J.C. wrote the manuscript with input from all authors. All the authors discussed the results and contributed to the writing of the manuscript.

Corresponding authors

Correspondence to Cheulhee Jung or Aram J. Chung .

Ethics declarations

Competing interests.

A.J.C. declares the following competing interest: A financial interest in MxT Biotech, which is commercializing the presented technology. The remaining authors declare no competing interests.

Peer review

Peer review information.

Nature Communications thanks Qiang Sun, Dayong Yang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, supplementary movie 1, supplementary movie 2, reporting summary, peer review file, description of additional supplementary files, source data, source data, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kim, YJ., Yun, D., Lee, J.K. et al. Highly efficient CRISPR-mediated genome editing through microfluidic droplet cell mechanoporation. Nat Commun 15 , 8099 (2024). https://doi.org/10.1038/s41467-024-52493-1

Download citation

Received : 27 March 2024

Accepted : 11 September 2024

Published : 16 September 2024

DOI : https://doi.org/10.1038/s41467-024-52493-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

A Meta-Analysis of the Relations Between Achievement Goals and Internalizing Problems

• META-ANALYSIS
• Open access
• Published: 16 September 2024
• Volume 36 , article number  109 , ( 2024 )

Cite this article

You have full access to this open access article

• Loredana R. Diaconu-Gherasim   ORCID: orcid.org/0000-0003-3598-5375 1 ,
• Andrew J. Elliot   ORCID: orcid.org/0000-0002-1664-6426 2 ,
• Alexandra S. Zancu   ORCID: orcid.org/0000-0002-1361-6870 1 ,
• Laura E. Brumariu   ORCID: orcid.org/0000-0002-3389-4288 3 ,
• Cornelia Măirean   ORCID: orcid.org/0000-0001-6895-8627 1 ,
• Cristian Opariuc‑Dan   ORCID: orcid.org/0000-0003-4079-0142 4 , 5 &
• Irina Crumpei-Tanasă 1  

This systematic meta-analytic review investigated the relations between achievement goals and internalizing symptoms and disorders, namely, anxiety and depression. The number of samples for each focal relationship ranged from 3 to 36. The results indicated significant effect sizes for the relations between mastery-approach goals and anxiety ( r  =  − .10) and depression (r =  − .18), as well as performance-avoidance goals and anxiety ( r  = .25) and depression ( r  = .16). A significant effect size was also found for the relation between performance-approach goals and anxiety ( r  = .15), and a non-significant effect size was observed for the relation between performance-approach goals and depression ( r  = .05). Mastery-avoidance goals were not significantly related to either anxiety ( r  = .08) or depression ( r  =  − .13). Several moderators representing the conceptualization of achievement goals (e.g., theoretical model), sample characteristics (e.g., education level), and methodology- and publication-based characteristics (e.g., year of publication) were significant, and suggested avenues for future research. These findings herein have implications for intervention programs that could focus on reducing the links between achievement goals and internalizing problems.

Avoid common mistakes on your manuscript.

The study of achievement goals is central to the achievement motivation literature. Achievement goals are defined as cognitive representations of competence-based end states that individuals strive to approach or avoid (Elliot, 1997 ). Several models of achievement goals (e.g., Dweck, 1986 ; Elliot, 1999 ; Nicholls, 1984 ) have been formulated over the years that vary regarding the focal components or number of achievement goals that individuals adopt in achievement situations. Despite some variation in conceptualization, theorists agree that the definition of competence (i.e., how doing well or poorly is defined) is an important dimension of achievement goals (Dweck & Leggett, 1988 ; Elliot & McGregor, 2001 ; Nicholls, 1984 ). The valence of competence (i.e., the approach-avoidance distinction) is also widely acknowledged as an important dimension of achievement goals (Elliot, 1997 ). Achievement goal frameworks, therefore, center on mastery goals (focusing on the development of competence and task- or self-based standards), performance goals (focusing on the demonstration of competence and other-based standards; Dweck, 1986 ; Nicholls, 1984 ), approach goals (approaching competence), and avoidance goals (avoiding incompetence; Elliot & Harackiewicz, 1996 ). In the present work, we focus on the 2 × 2 model of achievement goals in which the definition of competence is crossed with the valence of competence, resulting in four goals: Mastery-approach (striving to attain task- or self-based competence), performance-approach (striving to attain other-based competence), mastery-avoidance (striving to avoid task- or self-based incompetence), and performance-avoidance (striving to avoid other-based incompetence). This 2 × 2 model is commonly used in theoretical and empirical work in the achievement goal literature, as well as narrative and meta-analytic reviews (Butera et al., 2024 ).

Achievement goals are presumed to guide the way that people engage in achievement situations and how they cognitively, emotionally, and behaviorally respond to these situations (Ames, 1992 ; Dweck & Leggett, 1988 ). A great deal of research has shown that individuals’ achievement goals influence important outcomes such as performance, persistence, intrinsic motivation, and help-seeking behavior (for reviews, see Butera et al., 2024 ; Elliot & Hulleman, 2017 ; Senko, 2016 ). The influence of achievement goals has been extensively documented in a variety of different settings, especially school (e.g., Wirthwein et al., 2013 ), sports (Lochbaum & Gottardy, 2015 ), and work (Payne et al., 2007 ). In the present work we focus on the link between achievement goals and anxiety and depression symptoms and disorders.

Anxiety and depression are the most prevalent mental disorders and symptoms (APA, 2022 ). Theorists use the term “internalizing problems” for the cluster that includes clinical anxiety and depression or a combination of both symptoms or disorders (Yap et al., 2016 ). Anxiety affects a significant number of school-age children, with prevalence rates of an anxiety disorder of 7.1% in middle school and high school children (Ghandour et al., 2019 ), and prevalence rates of symptoms of 32% in college students (Sheldon et al., 2021 ). Depressive disorders have prevalence rates of approximately 3.5% in middle school and high school children (Ghandour et al., 2019 ) and the rate of depressive symptoms in college students reaches 25% (Sheldon et al., 2021 ). The prevalence rates of anxiety in the school-aged population have increased during the past few decades (Bitsko et al., 2018 ; Spoelma et al., 2023 ). Previous research has consistently shown that anxiety (e.g., generalized anxiety, panic disorder, separation anxiety, social anxiety; APA, 2013 ) is related to impairments across domains, such as emotional exhaustion (Koutsimani et al., 2019 ), and relational difficulties (Biswas et al., 2020 ). Similar results are also found for depression (e.g., disruptive mood dysregulation disorder, major depressive disorder; DSM-5; APA, 2013 ; Koutsimani et al., 2019 ; Marx et al., 2023 ). Previous narrative and meta-analytic reviews showed that in the educational context, anxiety and depression are related to a variety of negative outcomes, such as poor school attainment (see Riglin et al., 2014 ), poor school attendance (Finning et al., 2019 ), poor academic competence and greater school dropout (Brumariu et al., 2022 ), and problematic peer functioning (Christina et al., 2021 ), even at non-clinically diagnosed levels. Anxiety and depression show continuity in adulthood and, left untreated, could have severe consequences, even at subclinical levels, across one’s lifespan (e.g., poor quality of life and greater suicide risk; Marx et al., 2023 ).

Several narrative and meta-analytic reviews have tested how achievement goals are related to people’s emotional experiences, conceptualized as achievement emotions or affective states. In their meta-analysis, Payne et al. ( 2007 ) provided evidence of a negative relation between mastery-approach goals and state anxiety, as well as a positive relation between both performance-approach and performance-avoidance goals and state anxiety. Baranik et al.’s ( 2010 ) meta-analysis revealed positive relations of mastery-approach and performance-approach goals with positive affect, as well as positive relations of mastery-avoidance and performance-avoidance goals with negative affect. Huang ( 2011 ) replicated Baranik et al.’s findings, but also found a negative relation between performance-avoidance goals and positive affect, and a positive relation between performance-approach goals and negative affect.

Although these findings reveal that achievement goals are related to state anxiety and overall positive and negative affectivity, they do not address the links between these goals and clinical levels of anxiety and depression, or internalizing problems in general. Importantly, achievement emotions and affective states are conceptually distinct from clinical anxiety and depression, which are not based in a specific situation or event, but instead represent a constellation of symptoms that typically exert a significant impairment on one’s overall functioning (Luttenberger et al., 2018 ). Thus, the question of whether achievement goals are differentially related to internalizing problems has yet to be addressed meta-analytically. A growing number of individual studies have explored how achievement goals are related to internalizing problems (e.g., Madjar et al., 2021 ; Măirean & Diaconu-Gherasim, 2020 ; Sideridis, 2005 ), however, nothing is known about the cumulative nature and strength of these relations. The present meta-analysis advances the literature by synthesizing the existing empirical evidence on the relations between achievement goals and internalizing symptoms and disorders, and by investigating moderators of these links. In our work we focused on clinical anxiety and depression, and the combination of the two (i.e., global internalizing problems), as indicators of internalizing problems (Melton et al., 2016 ).

Basic Components and Models of Achievement Goals

As noted above, several theoretical models have guided research on achievement goals and we will test these different models in the present work. Initially, achievement goals models were dichotomous (Dweck, 1986 ; Nicholls, 1984 ), distinguishing between mastery goals (or task or learning goals) and performance goals (or ego or ability goals). Each goal was conceptualized as focusing on approaching success (Ames, 1992 ). Subsequently, the dichotomous model was extended by including the approach-avoidance distinction. Specifically, Elliot and Harackiewicz ( 1996 ) proposed the trichotomous model which bifurcates performance-based goals into performance-approach (focused on demonstrating other-based competence) and performance-avoidance (focused on avoiding the demonstration of other-based incompetence). In the 2 × 2 model, Elliot ( 1999 ) extended the trichotomous model by applying the approach-avoidance distinction to mastery-based goals; mastery-avoidance goals were conceptualized in terms of avoiding task-based or intrapersonal incompetence (see also Pintrich, 2000 ). Thus, the 2 × 2 model distinguishes among mastery-approach, mastery-avoidance, performance-approach, and performance-avoidance goals. This model implies that the task and self components of mastery-based goals combine together, when at times they may not. Therefore, in the 3  ×  2 model (Elliot et al., 2011 ), mastery-based goals were split according to the standards used to evaluate competence, resulting in two approach goals (focused on attaining task-based competence and self-based competence) and two avoidance goals (focused on avoiding task-based incompetence and self-based incompetence). Other-based goals – other-approach and other-avoidance – are the same as the original performance-approach and performance-avoidance goals. The 3 × 2 model has not been tested with regard to internalizing problems, thus we focused on the 2 × 2 model in the present research.

The specific conceptualizations of achievement goals vary across studies in the literature, depending on which component (standpoints or standards) of competence is targeted. The standpoints of competence perspective views competence in terms of developing it vs. demonstrating it, whereas the standards of competence perspective evaluates competence with regard to task/self-based vs. other-based standards (Korn & Elliot, 2016 ). For example, in some studies, mastery-based goals focus on the development of ability, whereas in other studies they focus on task- or self-based standards; likewise for performance-based goals, some studies focus on the demonstration of ability (or appearance goals), whereas other studies focus on other-based standards (or normative goals; see Korn et al., 2019 for an overview).

Furthermore, the terminology and specific content of the goal measures varies somewhat across studies (see Hulleman et al., 2010 for an overview of these types of variation). Specifically, mastery-approach goals are conceptualized and operationalized in terms of fulfillment of one’s potential (e.g., mastery-approach goals, Elliot & McGregor, 2001 ), development of ability/competence (e.g., development-approach goals; Elliot et al., 2011 ), interest and curiosity (e.g., task orientation, Nicholls, 1984 ), or doing better than one has done in the past (e.g., learning goals; Dweck, 1986 ). Mastery-avoidance goals are conceptualized and operationalized in terms of being unable to reach one’s potential (mastery-avoidance goals, Elliot & McGregor, 2001 ), avoiding the development of inability/incompetence (development-avoidance goals, Korn & Elliot, 2016 ), or avoiding task-based incompetence (task-avoidance goals, Elliot et al., 2011 ). Performance-approach goals are conceptualized and operationalized in terms of doing better than others (e.g., performance-approach goals Elliot, 1999 ), appearing competent to others (e.g., demonstration-approach goals, Korn & Elliot, 2016 ), demonstrating ability relative to others (e.g., ego orientation, Nicholls, 1984 ), or confirming one’s ability to an audience (e.g., ability goals, Grant & Dweck, 2003 ). Performance-avoidance goals are conceptualized and operationalized in terms of avoiding doing worse than others (e.g., performance-avoidance goals, Elliot & Church, 1997 ), avoiding the demonstration of incompetence relative to others (e.g., demonstration-avoidance goals, Korn & Elliot, 2016 ), or avoiding negative judgments from others (e.g., avoid orientation, VandeWalle, 1997 ). We attend to this variation in terminology in our meta-analysis.

Theory and Empirical Work on Achievement Goals and Internalizing Problems

Achievement goals are about competence, a basic human need that must be satisfied to sustain mental health and well-being (Elliot et al., 2002 ; Ryan & Deci, 2019 ). Achievement goals represent the cognitive-dynamic lens through which people frame and interpret contexts, events, and outcomes (Dweck, 1986 ; Nicholls, 1984 ). As such, it is sensible to posit that these forms of self-regulation are associated with mental health indicators such as anxiety and depression, and several researchers have proposed links accordingly.

The most influential model of achievement goals and internalizing problems is the goal-orientation model of depression vulnerability (Dykman, 1998 ). Essentially all theorizing in this area is directly or indirectly grounded in this model. The model postulates that achievement goals lead to specific cognitive sets and appraisal patterns that have important implications for people’s mental health. Individuals focused on mastery-based goals are oriented toward growth, learning, and improvement, perceive negative outcomes as opportunities for self-development, and their self-worth is not contingent on performance or social comparison; thus, they experience lower levels of depressive symptoms and are more resilient to failure. Individuals focused on performance-based goals, on the other hand, are orientated toward comparing their success/ability to others and their self-worth is contingent on demonstrating competence relative to others. They tend to evaluate challenging or difficult situations as a reflection of their personal traits (e.g., incompetence, unlikability) and as a test of their ability; thus, they report lower self-worth, are less resilient to failure, and are more vulnerable to depression.

Although the goal orientation model is primarily focused on depression, both Dykman ( 1998 ) and others (e.g., Sideridis, 2007 ) have argued that it is also applicable to anxiety, with comparable patterns expected for depression and anxiety. Furthermore, researchers have extended this analysis to include the approach-avoidance distinction, contending that mastery-approach goals are the most beneficial and performance-avoidance goals are the most detrimental to internalizing problems (Duchesne et al., 2014 ; Van Boekel & Martin, 2014 ; Wang et al., 2021 ). In addition, researchers have noted that anxiety and depression are accompanied by impaired cognitive functioning, lack of access to resources, and poor self-regulation which exert an influence on the type of achievement goals that individuals adopt (e.g., less mastery-approach and more performance-avoidance goals; Duchesne et al., 2014 ; Măirean & Diaconu-Gherasim, 2020 ). In other words, the relation between achievement goals and internalizing problems is posited to be reciprocal.

There is a growing body of research examining the relation between achievement goals and internalizing symptoms, and somewhat inconsistent findings have been reported. Some studies have found that mastery-approach goals are related to a low level of anxiety symptoms (e.g., Ariani, 2017 ; Sideridis, 2005 ; Wei, 2018 ) and depressive symptoms (e.g., Madjar et al., 2021 ; Măirean & Diaconu-Gherasim, 2020 ; Sideridis, 2005 ), while others have not found these relations (e.g., Duchesne et al., 2014 ). Performance-approach and performance-avoidance goals have been linked to higher levels of anxiety (Ariani, 2017 ; Madjar et al., 2021 ) and depressive symptoms (Duchesne et al., 2014 ; Măirean & Diaconu-Gherasim, 2020 ), but null effects have also been reported (Duchesne et al., 2014 ; Madjar et al., 2021 ; Sideridis, 2005 ). Very few studies have investigated the relation between mastery-avoidance goals and internalizing symptoms; mixed findings have been reported (e.g., Liu et al., 2019 ; Wang et al., 2021 ).

Moderators of the Relation Between Achievement Goals and Internalizing Problems

The present meta-analysis also addresses potential moderators of the relation between achievement goals and internalizing problems. Several of these moderator variables (e.g., achievement goal model, achievement goal terminology) have been used in prior achievement goal meta-analyses focused on other outcome measures (Huang, 2011 ; Hulleman et al., 2010 ; Senko & Dawson, 2017 ). We examined four broad moderation categories: Conceptualization of achievement goals, conceptualization of internalizing problems, sample characteristics, and methodology- and publication-based characteristics.

Conceptualization of Achievement Goals

Achievement goal models vary not only in the number of goals but also in the way goals are conceptualized, with some models emphasizing the focus of competence alone (i.e., the dichotomous model) and others placing equal emphasis on the focus and the valence of competence (e.g., the 2 × 2 model; Elliot & Hulleman, 2017 ). Further, the terminology used to refer to the different goal categories varies across studies (e.g., development-approach, learning goals, task orientation for mastery-approach goals; see Hulleman et al., 2010 for an overview of terminology). Thus, in our meta-analysis we tested whether the achievement goal model (dichotomous, trichotomous, 2 × 2) and achievement goal terminology (i.e., the labels used for the different goals) moderated the relation between achievement goals and internalizing problems. Several different scales are used in the literature to assess achievement goals (e.g., the Patterns of Adaptive Learning Scale; PALS; Midgley et al., 1993 ; the Achievement Goal Questionnaire [Revised]; AGQ(-R); Elliot & Church, 1997 ) (see Hulleman et al., 2010 for an overview of scales). The different scales may emphasize different conceptual aspects of the goals (e.g., the PALS measures both normative and appearance aspects, whereas the AGQ-R measures only normative aspects), therefore we tested whether achievement goal scale (PALS, AGQ/AGQ-R, TEOSQ, other scales) moderated the relation between achievement goals and internalizing problems. We also tested whether achievement goal setting (general, specific) served as a moderator. Finally, we also assessed whether the specific type of setting (e.g., academic, sports) moderated the achievement goal-internalizing problem relation.

Conceptualization of Internalizing Problems

Both anxiety and depressive disorders vary in their manifestations and consequences (e.g., avoidance behavior and excessive fear in anxiety disorders; persistent sad mood and change in sleep and appetite in depressive disorders; APA, 2013 ). Thus, we evaluated form of anxiety and depression as a moderator of the relation between achievement goals and anxiety and depression. Furthermore, clinical anxiety and depression involve significant functional impairment (Craske et al., 2017 ), so we also evaluated whether the observed findings were more robust at the diagnostic level.

Sample Characteristics

We evaluated whether the participants’ education level (middle school, high school, college), cultural context where the studies were conducted (Western, Eastern), and participants’ gender and age served as moderators.

Methodology- and Publication-based

We evaluated several methodology- and publication-based characteristics as moderators: Measurement approach (self-report, other report), type of design utilized (cross-sectional, longitudinal), direction of the relation between achievement goals and internalizing problems, year of publication, and publication type (peer reviewed, non-peer reviewed).

The Present Research

The present meta-analysis seeks to synthesize the results of studies that have focused on the associations between achievement goals and internalizing symptoms or disorders, and investigates moderators of these relations. We aimed to evaluate the strength of the relations between the four achievement goals of the 2 × 2 model (i.e., mastery-approach, mastery-avoidance, performance-approach, and performance-avoidance) and internalizing symptoms and disorders (i.e., anxiety, depression, and a combination of both). We predicted, based on the aforementioned theorizing, that high levels of mastery-approach goals would be related to lower levels of internalizing symptoms and disorders, and high levels of performance-approach and performance-avoidance goals would be related to higher levels of internalizing symptoms and disorders. We did not make predictions for mastery-avoidance goals, given that they represent a hybrid of adaptive (mastery) and maladaptive (avoidance) components (Elliot & McGregor, 2001 ). By synthesizing data across studies that varied in several different ways, this meta-analysis promises to yield a more clear and thorough understanding of how achievement goals and internalizing problems are related than any individual study can provide. The relations are important for psychological functioning in general, including affect, cognition, and behavior among children, adolescents, and young adults in educational settings.

We examined four sets of moderators regarding the relations between achievement goals and internalizing problems: Conceptualization of achievement goals, conceptualization of internalizing problems, sample characteristics, and methodology- and publication-based characteristics. We had no a priori hypotheses for these moderators; these analyses are exploratory in nature. Evaluation of these moderators is important in order to understand when achievement goals are associated with anxiety and depression, thus furthering the precision and depth of our knowledge regarding these relations.

Our systematic review was conducted according to the PRISMA 2020 (Preferred Reporting Items for Systematic reviews and Meta-Analyses; Page et al., 2021 ) guidelines. The review and meta-analysis protocol was pre-registered in PROSPERO International Prospective Register of Systematic Reviews (protocol number CRD42022298463).

Literature Search

The initial literature search was conducted in July 2021 in six electronic databases: Web of Science, PsycINFO, PubMed, ERIC, Academic Search Premier (EBSCO), and ProQuest. The final literature search for the work reported herein was conducted in December 2022. The same search strategy was applied in all databases, using the following combination of keywords: ("achievement goal" OR "goal orientation" OR "mastery goal" OR "mastery approach goal" OR "mastery orientation" OR mastery-approach OR "task goal" OR "task orientation" OR "learning goal" OR "development-approach goal" OR "task-approach goal" OR "self-approach goal" OR "mastery avoidance goal" OR mastery-avoid OR "development-avoidance goal" OR "task-avoidance goal" OR "self-avoidance goal" OR "performance goal" OR "performance approach goal" OR "ego goal" OR "ego orientation" OR "ability goal" OR "prove goal orientation" OR "performance-prove orientation" OR "self-enhancing goal orientation" OR "demonstration-approach goal" OR "other-approach goal" OR "performance avoidance goal" OR performance-avoidance OR "avoid goal orientation" OR "self-defeating ego orientation" OR "self-defeating orientation" OR "demonstration-avoidance goal" OR "other-avoidance goal") AND (internalizing OR "internalizing symptoms" OR "internalizing problems" OR "internalizing disorders" OR depression OR depressive OR depressed OR "depressive disorders" OR "depressive problems" OR "depressive symptoms" OR "depression symptoms" OR "mood disorders" OR sadness OR anxiety OR "generalized anxiety" OR "social anxiety" OR "anxiety symptoms" OR "anxiety problems" OR "anxiety disorders" OR anxious OR worry OR worries OR fear OR phobia OR phobic OR panic). The search was restricted to title and abstract fields and to the publication period 1980 (when the achievement goal approach emerged) to December, 2022 (when the final search was conducted). No restrictions were applied for publication type. All types of empirical research reports were eligible, including peer reviewed journal articles, conference papers, book chapters, and dissertation theses. Records and studies published in English, French, Spanish, and German were eligible for inclusion, due to the authors’ language competencies. The reference lists of the studies eligible for inclusion were searched for possible additional relevant studies. Unpublished studies were sought by contacting the authors with two or more published studies eligible for inclusion.

Study Selection

After removing duplicates, all of the records identified during the search stage were screened based on title and abstract by applying the eligibility criteria for inclusion and exclusion described below. Further, all retrieved full-text studies were assessed for eligibility and selected based on the inclusion and exclusion criteria. The same three coders were involved in the screening and selection of studies for inclusion in the systematic review. The coders overlapped with each other on a random sample of 50% of the studies and coded the studies independently. Kappa agreement between coders ranged from 0.76 to 0.91 for the abstract screening and 0.89 to 0.93 for the study selection. Disagreements between coders were resolved by discussion and consensus. Before the formal screening, the selection procedure was piloted on a random sample of studies. The flow diagram of the study selection process (following PRISMA 2020) is depicted in Fig.  1 .

The PRISMA 2020 flow diagram

The original quantitative studies that met the eligibility criteria were selected for inclusion. Inclusion criteria: (1) studies must include ratings of achievement goals (mastery-approach, mastery-avoidance, performance-approach, performance-avoidance, or their variants) and internalizing symptoms and disorders (depression, anxiety, or the combination of the two); if a study included an intervention or quasi-experiment, the achievement goals and internalizing measures had to have been collected prior to the intervention or quasi-experiment; (2) the study was written in English, French, Spanish, or German; (3) statistically relevant information was available for the relations between achievement goals and internalizing symptoms or disorders (e.g., correlation, sample size), allowing for the computation of effect size statistics. For the studies with insufficient reported data, the study’s investigators were contacted and requested to provide additional data (e.g., correlation coefficients).

Exclusion criteria: (1) theoretical papers, systematic reviews or meta-analyses, and qualitative studies; (2) studies that measured achievement goals at the group level (or achievement goal structures), and studies that induced achievement goals situationally; (3) studies measuring situational anxiety or anxiety related to specific settings such as educational or sport settings (e.g., test anxiety, sport anxiety, academic anxiety, learning/classroom anxiety, fear of failure, state anxiety, task anxiety, fear of failure, achievement-related emotions); (4) studies for which no full texts were accessible or sent by the authors upon request; (5) studies in which no statistical values for the relations between achievement goals and internalizing problems were reported or sent by the authors upon request. Reported data had to be independent of other studies included in the meta-analysis.

As depicted in Fig.  1 , the database search resulted in 1992 records. After duplicates were removed ( n  = 955), the remaining 1037 unique records were screened based on the abstract, and 777 records were excluded. 260 eligible records were sought for retrieval and the 232 full-text reports that were available were assessed for inclusion, based on the eligibility above criteria. Further, 196 reports were excluded for the following reasons: Did not asses achievement goals or internalizing problems ( n  = 93), were not empirical, quantitative studies ( n  = 4), examined situational or context-specific anxiety or emotions ( n  = 79), assessed achievement goals or goal structures at the group level ( n  = 2), statistical data for calculating the effect size was unavailable ( n  = 14), and were published in other languages ( n  = 4). An additional 21 reports were identified from other sources (i.e., websites, references of included studies) and assessed for eligibility. Overall, 44 reports meeting the eligibility criteria were included in the meta-analysis (see Fig.  1 and Section  7 of the Supplementary Material). The reports include a total of 22,387 participants, in 47 samples of children or adults, between 11 and 60 years old. A summary description of the included studies is presented in Table S1 , Supplementary Material.

Data Extraction and Coding

The following data were extracted and coded from each study: Conceptualization of achievement goals (achievement goal model, achievement goal terminology, achievement goal scale, achievement goal setting, type of informant); conceptualization of internalizing problems (indicators of internalizing problems, type of internalizing problems, form of anxiety, form of depression, type of informant), sample characteristics (education level, cultural context, age, gender); methodological and publication characteristics (type of design, direction of relation, type of publication, year of publication). The data extracted and coded are presented in Table  1 . The correlation coefficients between each achievement goal (mastery-approach, mastery-avoidance, performance-approach, performance-avoidance) and each indicator of internalizing symptoms and disorders (anxiety, depression, their combination) were also extracted. The same three coders from the screening stage extracted and coded the data from the included studies. The coders overlapped on a random sample of 25% of the studies that where coded independently by two coders. Kappa agreement between coders was higher than 94% for all of the categories. Disagreements between the coders were resolved by discussion and consensus coding.

Effect-Size Calculation

Correlation coefficients (i.e., Pearson’s r ) between each type of achievement goal (mastery-approach, mastery-avoidance, performance-approach, and performance-avoidance) and each indicator of internalizing problems (anxiety, depression, their combination) were extracted for effect sizes. When other effect size indicators (e.g., F tests) were reported, we converted them to correlation coefficients.

To ensure the independence of effect sizes, from each study a single effect size derived from a particular sample was included. If a study reported effect size information for different samples (e.g., students), these were considered independent, and the effect sizes from each sample were included. When a study reported more than one effect size from one sample for a particular analysis (e.g., the correlation between performance-avoidance goals and anxiety), several decisions were made to avoid dependency: 1) if a study reported both cross-sectional and longitudinal associations between two indicators, we only included the longitudinal effect size to take advantage of longitudinal research; 2) if a study reported cross-sectional associations from multiple time points between two indicators without reporting longitudinal associations, the coefficients were aggregated into a single effect size; 3) if a study reported the effect sizes for multiple measures of the same indicator (e.g., more than one measure of anxiety) from the same sample and time point, these were aggregated into a single effect size; 4) if a study reported several longitudinal effect sizes between two indicators from multiple time points, these were aggregated into a single effect size.

To test our hypotheses, a random effects meta-analysis model was conducted (Hedges & Olkin, 1985 ). We computed the correlation coefficient for the relation between each type of achievement goal and each indicator of internalizing problems, along with a 95% confidence interval (CI). Egger’s intercept test was used as a publication bias assessment at the global level, testing the funnel plot’s symmetry. Also, a funnel plot analysis was performed for testing publication bias at the moderator level.

Moderator Analysis

Heterogeneity of effect size was computed using Q statistics (Q B; Borenstein, 2009 ) to test whether the relations between the achievement goals and internalizing problems were moderated by the following categorical moderators: 1) conceptualization of achievement goals: Achievement goal model, achievement goal terminology, achievement goal scale, achievement goal setting, and achievement goal informant; 2) conceptualization of internalizing problems: Type of internalizing problems, form of anxiety, form of depression, and internalizing problems informant; 3) sample characteristics: Education level, and cultural context; 4) methodology- and publication-based characteristics of the studies: Type of study design, direction of the relations in longitudinal studies, year of publication, and type of publication. Categorical moderators were evaluated using subgroup analyses. We assessed the significant differences between the categories based on the Q Between test for subgroup differences for a random effect when a moderator had more than two categories. If the moderator had more than two categories, we also conducted a follow-up analysis and compared pairs of categories. For each pair, we assessed the significant differences between categories based on the Q Between test for subgroup differences, as we did in the case of moderators with only two categories. Consistent with previous meta-analytic work (e.g., Brumariu et al., 2022 ), we included a potential moderator only if there were four or more studies available per level. The following moderators were not included in subgroup analyses due to an insufficient number of reports (less than four): Type of informant (all studies used self-reports of achievement goals and internalizing problems), type of internalizing problems (no study assessed disorders), form of depression, and direction of the relation between achievement goals and indicators of internalizing problems. Global internalizing problems (a combination of both depression and anxiety) were examined in a single study, so we were unable to conduct a separate meta-analytic evaluation of the relations between achievement goals and this variable. Not all categories were available for each categorical moderator (e.g., the TEASQ category for the relation between mastery-approach goals and anxiety), thus Tables 2 and 3 present the results of the categorical moderators when each category is at least k = 4. For achievement goal scale the “other scales” category was created, and this category included other validated scales: The Goal Orientation Inventory (Dykman, 1998 ), the Goal Inventory (Roedel et al., 1994 ), the Goal Orientations and Motivational Beliefs scale (Niemivirta, 2002 ), and one author-created scale (Stornes & Bru, 2011 ). The results for follow-up analyses, when a moderator had more than two categories, are presented in the text. Mastery-avoidance goals are not included in any moderator analyses due to an insufficient number of studies. Meta-regression analyses were conducted to evaluate the role of continuous moderators (participant age, percentage of males, and publication year) in the relation between each type of achievement goal and internalizing problems. We present results for these moderators in the text.

The meta-analyses were conducted using R (Version 4.3.2; R Core Team, 2023 ) and the R-packages dmetar (Version 0.1.0; Harrer et al., 2019 ), esc (Version 0.5.1; Lüdecke, 2019 ), MAd (Version 0.8.3; Hoyt, 2014 ), ma ditr (Version 0.8.4; Demin, 2024 ), and meta (Balduzzi et al., 2019 ; Version 6.5.0; Harrer et al., 2019 ).

Associations Between Achievement Goals and Anxiety

As seen in Table  2 , mastery-approach goals were negatively related to anxiety, whereas performance-approach goals were positively related. Performance-avoidance goals were also positively related to anxiety, whereas the relation between mastery-avoidance goals and anxiety was not significant (see Forest plots in Section  2 of Supplementary Material). The significant within-group heterogeneity estimates (Q W values) suggested that there is heterogeneity of the effect sizes.

Moderators of the Relation Between Achievement Goals and Anxiety (see Table  2 )

Mastery-approach goals.

Three of eight categorical variables tested were significant moderators of the negative relation between mastery-approach goals and anxiety. Achievement goal terminology was significant; the negative relation was stronger for development-approach goals than mastery goals/orientation ( Q B  = 7.79, p  = 0.005), or mastery-approach goals ( Q B  = 8.07, p  = 0.004), and did not differ from all other terms (all Q B  > 0.05). Achievement goal setting was significant; the negative relation was stronger in studies in a general setting than in specific settings. Education level was significant; the negative relation was stronger in studies with college samples than in studies with middle school samples (Q B  = 6.89, p  = 0.014) or high school samples (Q B  = 6.17, p  = 0.035). Achievement goal model, achievement goal scale, form of anxiety, type of design and type of publication were not significant. Age ( B  =  − 0.004), percentage of males ( B  =  − 0.002), and year of publication ( B  = 0.006) were not significant.

Performance-Approach Goals

Three of seven categorical variables tested were significant categorical moderators of the positive relation between performance-approach goals and anxiety. Achievement goal terminology was significant; the positive relation was stronger for demonstration-approach goals than for performance-approach goals. Achievement goal setting was significant; the positive relation was stronger in a general setting than in specific settings. Education level was significant; the positive relation was stronger for college students than for high school (Q B  = 6.17, p  = 0.013) or middle school (Q B  = 6.89, p  = 0.009) students. Achievement goal model, achievement goal scale, form of anxiety, and type of publication were not significant moderators. The meta-regression indicated that the percentage of males was a significant moderator; the relation between performance-approach goals and anxiety decreased as the percentage of males increased ( B  =  − 0.003, p  = 0.03). Age ( B  = 0.009) and year of publication ( B  =  − 0.003) were not significant.

Performance-Avoidance Goals

Two of six categorical variables tested were significant categorical moderators of the positive relation between performance-avoidance goals and anxiety. Achievement goal model was significant; the positive relation was stronger in studies using the trichotomous model than in studies using the 2 × 2 model. Achievement goal scale was significant; the positive relation was stronger in studies that used other scales than in studies that used the AGQ/AGQ-R (Q B  = 5.94, p  = 0.015). Form of anxiety, education level, type of design and type of publication were not significant. The meta-regression for age was significant; the relation between the performance-avoidance goals and anxiety increased as participants’ age increased ( B  = 0.008, p  = 0.002). Percentage of males ( B  =  − 0.0003) and year of publication ( B  =  − 0.005) were not significant.

Associations Between the Achievement Goals and Depression

Mastery-approach goals were negatively related, whereas performance-avoidance goals were positively related to depression (see Table 3 ). Performance-approach goals and mastery-avoidance goals were not significantly related to depression, and mastery-avoidance goals was not significantly related to depression (see Forest plots in Section  3 of Supplementary Material). The significant within-group heterogeneity estimates (Q W values) suggested that there is heterogeneity of the effect sizes.

Moderators of the Relations Between the Achievement Goals and Depression (see Table  3 )

None of categorical variables tested (achievement goal model, achievement goal terminology, achievement goal scale, achievement goal setting, education level, cultural context, type of publication) were significant moderators of the negative relation between mastery-approach goals and depression. The meta-regression for year of publication was significant ( B  =  − 0.007, p  = 0.014); the relation between the mastery-approach goals and depression decreased for more recent publications. Age ( B  =  − 0.002) and percentage of males ( B  =  − 0.001) were not significant.

Three of seven categorical variables tested were significant moderators of the relation between performance-approach goals and depression. Achievement goal model was significant; the relation was positive and significant only in studies that used the dichotomous model. Achievement goal scale was significant, but no statistical differences were observed by follow-up analyses (all Q B  > 0.05), however, the relation between performance-approach and depression was negative in studies that used the PALS and positive in studies that used all other scales. Cultural context was significant; the relation was positive and significant only in studies conducted in Western countries. Achievement goal terminology, achievement goal setting, education level and type of publication were not significant. The meta-regressions indicated that year of publication was significant ( B  =  − 0.011, p  = 0.042); the relation between the performance-approach goals and depression decreased for more recent publications. Age ( B  = 0.01) and percentage of males ( B  =  − 0.002) were not significant.

None of five categorical variables (achievement goal model, achievement goal scale, education level, type of publication, cultural context) tested were significant moderators of the positive relation between performance-avoidance goals and depression. Meta-regressions indicated that age ( B  = 0.0001), percentage of males ( B  =  − 0.001), and year of publication ( B  = 0.003) were not signification moderators.

Publication Bias

Egger’s test for studies evaluating anxiety indicated there was no indication of publication bias for mastery-approach goals, t (32)  =  − 1.00, p  = 0.32, and that the probability of publication bias was significant for performance-approach goals, t (33)  = 3.44, p  = 0.002, and performance-avoidance goals, t (22)  = 2.11, p  = 0.04. A trim-and-fill analysis suggested a significant left asymmetry for performance-approach goals, and that an overestimation of the global effect size was plausible (11 powerful studies and one low-powered study should be included to compensate the possible overestimation of the effect; see Funnel plots in Section  4 of Supplementary Material). Similar bias was observed for performance-avoidance goals, suggesting that the effect could be overestimated (2 low-powered studies and 4 medium-powered studies should be included to compensate the possible overestimation of the effect). For depression, there was no indication of publication bias in the studies assessing mastery-approach goals, t (34)  =  − 1.73, performance-approach goals, t (32)  =  − 1.21, and performance-avoidance goals, t (22)  =  − 0.18, all ps  > 0.05 (see Funnel plots in Section  5 of Supplementary Material).

In the present meta-analytic work, we evaluated the strength of the relations of the four goals of the 2 × 2 achievement goal model with anxiety and depression. We found significant effect sizes linking mastery-approach goals and performance-avoidance goals to both anxiety and depression, and performance-approach goals to anxiety but not depression; no significant relations were found for mastery-avoidance goals. We also found significant moderation of these relations, indicating variation as a function of conceptualization of achievement goals, sample characteristics, and methodology- and publication-based characteristics.

Direct Relations Between Achievement Goals and Internalizing Problems

Our findings indicate that achievement goals are related to anxiety and depression. The effect sizes are small to medium in magnitude, with the significant relations ranging from r  = 0.05 (for performance-avoidance goals and depression) to r  = 0.25 (for performance-avoidance goals and anxiety). We found that individuals with higher levels of mastery-approach goals experience lower levels of both anxiety and depression. Individuals with higher levels of performance-approach goals, on the other hand, experience higher levels of anxiety, and exhibit a trend toward higher levels of depression. Those with higher levels of performance-avoidance goals experience higher levels of both anxiety and depression. Mastery-avoidance goals appear to be unrelated to anxiety and depression, although a positive trend is evident for these goals and depression.

These results are in line with and extend the goal-orientation model of depression (Dykman, 1998 ). Individuals who pursue mastery-approach goals are less vulnerable to anxiety and depression, likely because they are focused on growth and self-improvement, and appraise difficult situations/tasks as challenging (Dykman, 1998 ). Individuals who pursue performance-based goals experience higher levels of anxiety and depression, most likely because they are focused on comparing their success/ability to others (Dweck, 1986 ), and interpret difficult situations/tasks as a test of their ability (Dykman, 1998 ). Of note, achievement goals are associated with anxiety as well as depression (Sideridis, 2007 ), and the connection between performance-based goals and internalizing problems is, descriptively, particularly prominent for the avoidance manifestation of such goals (i.e., performance-avoidance). It is also important to note that our findings are mute regarding the causal direction of the relations; it is possible that the relations are bi-direction and feed into one another (Duchesne et al., 2014 ; Măirean & Diaconu-Gherasim, 2020 ).

Although our meta-analysis did not test processes that might explain the observed findings, several cognitive processes seem likely candidates. By encouraging growth and learning, mastery-approach goals might reduce dysfunctional attitudes and negative attributions (e.g., attributing failure to temporary, changeable factors), thus reducing vulnerability to anxiety and depression (Steare et al., 2024 ). Mastery-approach goals might also promote beliefs that ability can be developed through practice (growth mindset) that further encourage adaptive ways of coping with stress and failure that are related to low levels of anxiety and depression (Dykman, 1998 ; Yeager & Dweck, 2023 ). By promoting external standards based on social comparison, performance-based goals might lead to more negative attributions (e.g., attributing difficulty to internal, stable factors) and interpreting failure as a threat to one’s self-worth, facilitating stress and rumination, and elevated levels of anxiety and depression (Steare et al., 2024 ). Performance-based goals might also promote beliefs that ability cannot be developed through practice (fixed mindset), which is related to increased risk of anxiety and depression (Yeager & Dweck, 2023 ). Finally, to the degree that anxiety and depression themselves reduce mastery-approach and promote performance-based goals pursuit, they may do so by impairing cognitive functioning, coping, and self-regulation processes (Duchesne et al., 2014 ).

Mastery-avoidance goals are not significantly related to anxiety and depression which at first glance may suggest that these goals are not relevant to this type of psychopathology. However, these goals are focused on not losing one’s skills and abilities, and thus may be particularly prevalent among and applicable to older people (Elliot & McGregor, 2001 ). The vast majority of studies included in this meta-analysis were conducted on samples of middle school, high school, and college students, and the observed relations may be stronger in samples where age is more evenly distributed. It is also important to take care in interpreting the mastery-avoidance goal results given the small number of existing studies on anxiety (k = 4) and depression (k = 4). Although the results did not reach significance for mastery-avoidance goals and depression, the effect size was similar to those for mastery-approach and performance-avoidance goals and depression; thus, more research is needed to more clearly determine the precise nature of the mastery-avoidance goal-depression relation.

Considering the findings for the four achievement goals together, the pattern of results between achievement goals and internalizing problems is similar to the pattern found for other achievement-relevant processes and outcomes in the literature. Meta-analyses on variables such as achievement, intrinsic motivation, help seeking, etc. have consistently revealed that mastery-approach goals have the most adaptive pattern of relations, performance-avoidance goals have the least adaptive pattern, and the pattern for performance-approach and mastery-avoidance goals lies in between (e.g., Baranik et al., 2010 ; Hulleman et al., 2010 ; Wirthwein et al., 2013 ; see also Butera et al., 2024 for a narrative review). Perhaps most relevant to the current work, prior meta-analytic work has indicated that mastery-approach goals are negatively, and performance-approach, performance-avoidance, and mastery-avoidance goals are positively, related to negative achievement emotions (Huang, 2011 ). Our findings are consistent with this pattern for all but mastery-avoidance goals (which produce null results herein); this is interesting as it suggests that the mastery-avoidance goal pursuit may be less pernicious, with regard to emotional experience, than the two performance-based goals. Critically, our findings extend the prior work on negative affect by linking achievement goals to clinical anxiety and depression marked by persistent and intense emotional experience. Thus, the present work expands the achievement goal nomological network to include broad outcomes beyond the achievement domain and relevant to overall psychological functioning and mental health.

From a theoretical standpoint, our work may be seen as contributing to an understanding of the links between approach and avoidance motivation on one hand and anxiety and depression on the other. Gray’s Reinforcement Sensitivity Theory (Fowles, 1994 ; Gray, 1982 ) focuses on two basic, biologically-based motivational systems, the behavioral inhibition system (BIS; avoidance motivation) and the behavioral activation system (BAS; approach motivation). In this theory, the BIS is sensitive to stimuli representing nonreward, punishment, and novelty and involves moving away from (avoiding) or inhibiting undesirable affective states; high BIS sensitivity is positively associated with both anxiety and depression. The BAS is sensitive to stimuli representing reward and escape from punishment and involves moving toward (approaching) or maintaining desirable affective states; high BAS sensitivity is negatively association with depression, but not anxiety (see Katz et al., 2020 for further elaboration on these constructs and findings). Our findings at the goal level indicate that it is one type of avoidance motivation that is positively associated with anxiety, namely performance-avoidance goals, and that it is one type of approach motivation that is negatively associated with depression, namely mastery-approach goals. In addition, our findings indicate that one type of approach goals, performance-approach, is positively associated with anxiety. Thus, our findings show that the motivation-internalizing symptoms relations become more nuanced and specific as people regulate their basic energization tendencies with more concrete directional aims (Elliot & Thrash, 2002 ; see also Dickson & MacLeod, 2004 for related work on personal goals and internalizing symptoms).

Moderators of the Relations Between Achievement Goals and Internalizing Problems

Several of the tested moderator variable candidates were significant. Below we highlight what we perceive to be the most informative moderator variable findings.

Achievement goal model was a robust moderator across types of goals and internalizing problems. Specifically, the findings were significantly stronger for the dichotomous model than for the trichotomous and 2 × 2 models for the positive relation between performance-approach goals and depression (and they were descriptively stronger for the negative relation between mastery-approach goals and anxiety, and the positive relation performance-approach goals and anxiety). Further, the findings were significantly stronger for the trichotomous model than for the 2 × 2 model for the positive relations between performance-avoidance goals and both anxiety and depression (and they were descriptively stronger for the negative relations between mastery-approach goals and both anxiety and depression, and the positive relation between performance-approach goals and anxiety). Operationally, the achievement goals in the trichotomous model contain content other than goal standards per se, including a preference for challenge (mastery-approach), a desire to impress important others (performance-approach), and worries and fears (performance-avoidance; see Elliot & Church, 1997 ; Middleton & Midgley, 1997 ; Skaalvik, 1997 ; Vandewalle, 1997 ). These added components essentially create goal complexes that encompass both the goal standard and the reasons for pursuing that standard, and these reasons likely add to the predictive power of the standard (Senko & Tropiano, 2016 ; Sommet & Elliot, 2017 ). In essence (that is, heuristically, not technically), one can compare the trichotomous model effect sizes to those for the 2 × 2 model to get a rough estimate of the predictive utility gained by adding reason-based content to the goal standard content (for more on achievement goal complexes, see Sommet et al., 2021 ; Liem & Senko, 2022 ).

Another robust moderator variable was achievement goal terminology. We found that the goals using “development” and “demonstration” terminology showed the strongest relations when there were sufficient sample sizes for this moderator to be tested. Specifically, the negative relation between development-approach goals and anxiety (and, descriptively, depression), as well as the positive relation between demonstration-approach goals and anxiety were stronger than the relations observed for other achievement goal terms. It is interesting to note that the development and demonstration labels aren’t just distinct terms, but they also represent somewhat distinct content. That is, mastery-based goals include both task-based and self-based competence standards, whereas development-based goals focus on self-based standards only (Elliot et al., 2011 ; Korn & Elliot, 2016 ). The stronger associations of development- and demonstration-based goals are mirrored in the significant moderation for achievement goal scale. Studies using the PALS (Midgley et al., 1993 ) and the AGQ/AGQ-R (Elliot & Church, 1997 ) showed weaker associations than studies using other scales (e.g., Dykman, 1998 ). For example, the PALS measured both the normative and appearance aspect of performance-approach goals, the AGQ/AGQ-R measured only normative goals, whereas other scales measured performance goals as demonstration/appearance goals. Our results thus suggest a stronger negative impact of the development or demonstration/appearance component on anxiety and depression. As such, our results suggest that it is appetitive temporal striving – trying to increase one’s competence (development-approach) – that is most strongly linked to lower anxiety and depression. Likewise, performance-approach goals include other-based standards, whereas demonstration-based goals focus more on showing one’s ability to others. As such, our results suggest that a focus on appearance and demonstration may create a propensity for external motivators and a potential reliance on others for validation, which is likely to perpetuate anxious/depressed thoughts (see Hulleman et al., 2010 ; Senko & Dawson, 2017 for related work).

A third robust moderator variable was achievement goal setting. We found that the goals focusing on competence in general relative to those focusing on specific domains showed the strongest relations when there were sufficient sample sizes for this moderator to be tested. That is, domain-general mastery-approach goals had a stronger negative relation with both anxiety and depression than domain-specific mastery-approach goals. Likewise, domain-general performance-approach goals had a stronger positive relation with anxiety than domain-specific performance-approach goals, and domain-general mastery-approach goals had a stronger negative relation with anxiety (and, descriptively, depression) than domain-specific mastery-approach goals. This pattern of findings likely reflects the correspondence principle, which states that the relationship between two variables will be strongest when they are matched in level of generality-specificity (Ajzen & Fishbein, 1977 ). The anxiety and depression variables that were the focus of the present research represent broad, domain-general indicators of mental health, so it is sensible that they would be more strongly related to broad, domain-general indicators of achievement goals than narrow, domain-specific indicators. Achievement goal researchers (and researchers across domains and disciplines) would do well to attend to this often-overlooked principle in their work (for an empirical demonstration of the importance of correspondence in work on other achievement motivation constructs, see Chan et al., 2023 ).

A final point that we would like to highlight concerns the robustness of the performance-avoidance goal findings across moderators. The analyses did show that the performance-avoidance goals links to both anxiety and depression were moderated by a number of difference variables. However, this moderation almost exclusively revealed differences in the relative strength of significant findings, rather than revealing a significant finding under one condition but not another. For example, for each tested moderator of the positive relation between performance-avoidance goals and anxiety, the relation was significant and positive, only varying in magnitude; in fact, all but two of the eleven observed effect sizes for this relation dropped below the r  = 0.30 mark. This robustness across moderators was unique to performance-avoidance goals, testifying to how this type of self-regulation represents a particularly pervasive and pernicious mental health vulnerability (Elliot & Hulleman, 2017 ).

It is important to also note that some other moderators – achievement goal scale, participants’ education level, cultural context, and type of design – were also relevant, even if only for a small number of relations. For example, with regard to cultural context, we found evidence supporting a universalist perspective and evidence supporting cultural differences (see Zusho & Clayton, 2011 ). Findings indicated that mastery-approach goals were positively and performance-avoidance goals negatively related to depression across cultures (a universalist finding), and findings also indicated that performance-approach goals were positively related to depression for Western Cultures but unrelated for Eastern cultures (a cultural difference). There were not enough existing studies to test for mastery-avoidance goal differences. Several things are noteworthy here. First, prior work showing cultural differences regarding achievement goals has tended to find differences for performance-avoidance goals. These goals fit the stronger collectivistic emphasis on avoiding negative outcomes in Eastern, relative to Western cultures, which accounts for why performance-avoidance goals are sometimes not detrimental and can even be beneficial in such contexts (Elliot et al., 2001 ). Importantly, this has been found for performance-based outcomes but not experience-based outcomes such as intrinsic motivation (see Hulleman et al., 2010 ). It may be that performance-avoidance goals afford performance benefits in Eastern contexts, but that the stress of regulating according to a negative normative possibility still exacts a toll on experience and well-being (Roskes et al., 2014 ). Second, our finding that performance-approach goals are detrimental for depression in Western but not Eastern cultures may also be a function of individualistic and collectivistic emphases. In Western cultures, individualistic values emphasize personal achievement and success relative to others, perhaps amplifying the impact of performance-approach goals on mental health. In Eastern cultures, on the other hand, collectivistic values emphasize in-group (e.g. family) achievement and success, and thus performance-approach goals may have fewer implications for how the self is construed (King et al., 2017 ), mitigating the impact of these goals on depression. Third, the fact that there were not enough existing studies to test for cultural differences for the mastery-avoidance goal to depression link nor for cultural differences for any achievement goal and anxiety highlights the clear need for more research in this important area.

In sum, the moderator variable analyses yielded several informative findings that provide a more precise and rich empirical picture than that gleaned from the omnibus relations alone. Nevertheless, two cautions are in order. First, some of the moderator tests should be interpreted with caution given modest numbers of available studies (e.g., mastery-approach goal, task orientation goal, and learning goal orientation variants of mastery-approach goals); furthermore, in some instances there were not enough samples to test for moderation (e.g., ego goal, ability goal, and prove goal variants of performance-approach goals). Thus, some effect sizes may be unstable (note that we highlighted the particularly robust findings above), some differences in associations between moderator levels are practically negligible albeit significantly different (e.g., stronger correlation between performance-avoidance goals and depression measured with other scales than with the AGQ/AGQ-R), and some important moderator information may be missing altogether (e.g., performance-avoidance terminology, form of depression). Second, there is conceptual overlap in some of the moderators tested, especially regarding achievement goal conceptualization (e.g., achievement goal model – dichotomous, trichotomous, 2 × 2 – clearly has some overlap with achievement goal terminology – mastery goal/orientation, mastery-approach goal, etc.). As such, the number of significant moderators may be somewhat misleading, as some of the significant findings may emerge from nonindependent tests. Regardless, it is clear from the present work that there is considerable complexity and nuance underlying the direct relations between achievement goals and internalizing problems documented herein, performance-avoidance goals being the exceptional case.

Limitations, Future Directions, and Implications

Limitations of the present work should be noted; these limitations point to additional avenues for future research. First, we evaluated each achievement goal separately. However, achievement goals are not mutually exclusive and people commonly simultaneously adopt multiple goals (Barron & Harackiewicz, 2001 ; Pintrich, 2000 ). Future studies would do well to evaluate how multiple goals (e.g., high mastery-approach and performance-approach goals) are related to anxiety and depression. Second, all studies in our meta-analytic work used self-reported scales of achievement goals, anxiety, and depression; as such, the results might be affected by common method variance. Future studies would do well to use other informants (e.g., teachers’ reports of students’ achievement goals, clinical interviews to assess depression and anxiety) in order to more definitively document the focal relations. Third, there were not sufficient longitudinal data incorporating multiple measurements of both achievement goals and internalizing problems for us to test direction of causality. Future research would do well to attend to this important issue, particularly given the differing theoretical emphases on this matter. Fourth, although our research documented direct relations between achievement goals and internalizing problems, it did not document the mechanisms responsible for these relations. Future research is needed to test purported mediation of the observed links, such as cognitive appraisals (Dykman, 1998 ), perceived stress (Wang et al., 2021 ), and rumination (Van Boekel & Martin, 2014 ). Finally, the present work revealed minimal or no existing research on the following: Mastery-avoidance goals, different types of anxiety (e.g., generalized anxiety, agoraphobia) and depression (e.g., disruptive mood dysregulation disorder), and older adults. These issues are in clear need of future research attention.

The findings from the present research join the growing corpus of findings indicating that mastery-approach goals are beneficial for psychological functioning, whereas performance-avoidance goals are detrimental. The results reveal a new and promising perspective for prevention efforts, with achievement goals as sensible entry points to prevent poor mental health. Achievement goals are modifiable through interventions targeting school environment by, for example, emphasizing students’ personal growth and learning (evoking mastery-approach goals) (see Elliot & Hulleman, 2017 for a review). School-based interventions are needed to address the structural aspects of the educational system by emphasizing the development of abilities, understanding of material, and promoting mistakes as opportunities for growth (Liu et al., 2024 ; Steare et al., 2024 ). Accordingly, the take-home message from this research is similar to that of other achievement goal research: Teachers, coaches, employers, and parents would do well to structure their instructions, incentives, and feedback to those under their charge in ways that facilitate and support the pursuit of mastery-approach goals, and discourage and disrupt the pursuit of performance-avoidance goals (Bardach et al., 2020 ; Korn et al., 2019 ; Senko, 2016 ). The data are not yet clear enough for emphatic statements about performance-approach or mastery-avoidance goals. Of course, these recommendations must be made while acknowledging the relatively small number of studies conducted and the relatively modest overall effect sizes.

Concluding Thoughts

A clear and unequivocal conclusion that may be drawn from the present meta-analytic work is that achievement goals and internalizing problems are systematically related to each other. This conclusion is of conceptual and applied importance. Conceptually, it means that competence pursuits are relevant to mental health in general, they are not just relevant to competence-specific (e.g., achievement, intrinsic motivation) or domain-specific (e.g., school, work) outcomes, or affective states (e.g., achievement-relevant emotions). Competence is a basic human need, so it makes sense that competence-based pursuits would be linked to broad health and well-being indices (Elliot et al., 2002 ; Ryan & Deci, 2019 ). Further, researchers might consider achievement goals as an explanation for why depression and anxiety are related to individuals’ achievement and adjustment in various settings, including educational, work, and sport settings. In terms of application, it means that greater attention needs to be allocated – in the classroom, the workplace, and the ballfield – to the achievement goal-mental health nexus. Attending to the whole person, not just the individual’s short-term achievements, will likely pay dividends for both long-term accomplishment and overall flourishing and functioning. Accordingly, we believe there is strong reason to sound the call for increased research attention to the relation between achievement goal pursuit and internalizing problems.

Data Availability

The data is available on OSF on the following link: https://osf.io/beu6h/?view_only=b463289949aa413a936f8349ab2a702a

Ajzen, I., & Fishbein, M. (1977). Attitude–behavior relations: A theoretical analysis and review of empirical research. Psychological Bulletin, 84 , 888–918. https://doi.org/10.1037/0033-2909.84.5.888

Article   Google Scholar  

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). https://doi.org/10.1176/appi.books.9780890425596

American Psychiatric Association. (2022). Diagnostic and statistical manual of mental disorders (5th ed., text rev.). https://doi.org/10.1176/appi.books.9780890425787

Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84 , 261–271. https://doi.org/10.1037/0022-0663.84.3.261

Ariani, D. W. (2017). Self-determined motivation, achievement goals and anxiety of economic and business students in Indonesia. Educational Research and Reviews, 12 , 1154–1166. https://doi.org/10.5897/ERR2017.3381

Balduzzi, S., Rücker, G., & Schwarzer, G. (2019). How to perform a meta-analysis with R:43A practical tutorial. Evidence-Based Mental Health, 22 , 153–160. https://doi.org/10.1136/ebmental-2019-300117

Baranik, L. E., Stanley, L. J., Bynum, B. H., & Lance, C. E. (2010). Examining the construct validity of mastery-avoidance achievement goals: A meta-analysis. Human Performance, 23 , 265–282. https://doi.org/10.1080/08959285.2010.488463

Bardach, L., Oczlon, S., Pietschnig, J., & Lüftenegger, M. (2020). Has achievement goal theory been right? A meta-analysis of the relation between goal structures and personal achievement goals. Journal of Educational Psychology, 112 , 1197–1220. https://doi.org/10.1037/edu0000419

Barron, K. E., & Harackiewicz, J. M. (2001). Achievement goals and optimal motivation: Testing multiple goal models. Journal of Educational Psychology, 80 , 706–722. https://doi.org/10.1037/0022-3514.80.5.706

Biswas, T., Scott, J. G., Munir, K., Renzaho, A. M., Rawal, L. B., Baxter, J., & Mamun, A. A. (2020). Global variation in the prevalence of suicidal ideation, anxiety and their correlates among adolescents: A population based study of 82 countries. E-Clinical Medicine, 24 , 100395. https://doi.org/10.1016/j.eclinm.2020.100395

Bitsko, R. H., Holbrook, J. R., Ghandour, R. M., Blumberg, S. J., Visser, S. N., Perou, R., & Walkup, J. T. (2018). Epidemiology and impact of health care provider–diagnosed anxiety and depression among US children. Journal of Developmental and Behavioral Pediatrics, 39 , 395–403. https://doi.org/10.1097/DBP.0000000000000571

Borenstein,. (2009). Effect sizes for continuous data. In H. Cooper, L. Hedges, & J. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 279–293). Sage.

Google Scholar  

Brumariu, L., Waslin, S., Gastelle, M., Kochendorfer, L., & Kerns, K. (2022). Anxiety, academic achievement, and academic self-concept: Meta-analytic syntheses of their relations across developmental periods. Development and Psychopathology, 35 , 1597–1613. https://doi.org/10.1017/S0954579422000323

Butera, F., Dompnier, B., & Darnon, C. (2024). Achievement goals: A social influence cycle. Annual Review of Psychology, 75 , 527–554. https://doi.org/10.1146/annurev-psych-013123-102139

Chan, H. S., Chiu, C. Y., Lee, S. L., Tong, Y. Y., & Leung. (2023). Improving the predictor-criterion consistency of mindset measures: Application of the correspondence principle. Journal of Pacific Rim Psychology, 17 , 1–11. https://doi.org/10.1177/18344909231166964

Christina, S., Magson, N. R., Kakar, V., & Rapee, R. M. (2021). The bidirectional relationships between peer victimization and internalizing problems in school-aged children: An updated systematic review and meta-analysis. Clinical Psychology Review, 85 , 101979. https://doi.org/10.1016/j.cpr.2021.101979

Craske, M. G., Stein, M. B., Eley, T. C., Milad, M. R., Holmes, A., Rapee, R. M., & Wittchen, H. U. (2017). Anxiety Disorders. Nature Reviews Disease Primers, 3 , 17024. https://doi.org/10.1038/nrdp.2017.24

Demin, G. (2024). Maditr: Fast data aggregation, modification, and filtering with pipes and ’data.table’. https://CRAN.R-project.org/package=maditr

Dickson, J. M., & MacLeod, A. K. (2004). Approach and avoidance goals and plans: Their relationship to anxiety and depression. Cognitive Therapy and Research, 28 , 415–432. https://doi.org/10.1023/B:COTR.0000031809.20488.ee

Duchesne, S., Ratelle, C. F., & Feng, B. (2014). Developmental trajectories of achievement goal orientations during the middle school transition: The contribution of emotional and behavioral dispositions. The Journal of Early Adolescence, 34 , 486–517. https://doi.org/10.1177/0272431613495447

Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41 , 1040–1048. https://doi.org/10.1037/0003-066X.41.10.1040

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95 , 256–273. https://doi.org/10.1037/0033-295X.95.2.256

Dykman, B. M. (1998). Integrating cognitive and motivational factors in depression: Initial tests of a goal-orientation approach. Journal of Personality and Social Psychology, 74 , 139–158. https://doi.org/10.1037/0022-3514.74.1.139

Elliot, A. (1997). Integrating the ‘classic’ and ‘contemporary’ approaches to achievement motivation: A hierarchical model of approach and avoidance achievement motivation. In M. Maehr & P. Pintrich (Eds.), Advances in motivation and achievement (Vol. 10, pp. 143–179). JAI Press.

Elliot, A. J., Chirkov, V. I., Kim, Y., & Sheldon, K. M. (2001). A cross-cultural analysis of avoidance (relative to approach) personal goals.  Psychological Science, 12 , 505–510.  https://doi.org/10.1111/1467-9280.00393

Elliot, A. J., & Hulleman, C. S. (2017). Achievement goals. In A. Elliot, C. Dweck, & D. Yeager (Eds.), Handbook of competence and motivation. Theory and application (pp. 43–60). Guilford Press.

Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational Psychologist, 34 , 169–189. https://doi.org/10.1207/s15326985ep3403_3

Elliot, A. J., & Church, M. A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72 , 218–232. https://doi.org/10.1037/0022-3514.72.1.218

Elliot, A. J., & Harackiewicz, J. M. (1996). Approach and avoidance achievement goals and intrinsic motivation: A mediational analysis. Journal of Personality and Social Psychology, 70 , 461–475. https://doi.org/10.1037/0022-0663.100.3.613

Elliot, A. J., & McGregor, H. A. (2001). A 2×2 achievement goal framework. Journal of Personality and Social Psychology, 80 , 501–519. https://doi.org/10.1037/0022-3514.80.3.501

Elliot, A. J., McGregor, H. A., & Thrash, T. M. (2002). The need for competence. In E. Deci & R. Ryan (Eds.), Handbook of self-determination research (pp. 361–387). University of Rochester Press.

Elliot, A. J., Murayama, K., & Pekrun, R. (2011). A 3 × 2 achievement goal model. Journal of Educational Psychology, 103 (3), 632–648. https://doi.org/10.1037/a0023952

Elliot, A. J., & Thrash, T. M. (2002). Approach–avoidance motivation in personality: Approach and avoidance temperaments and goals. Journal of Personality and Social Psychology, 82 , 804–818. https://doi.org/10.1037/0022-3514.82.5.804

Finning, K., Ukoumunne, O. C., Ford, T., Danielsson-Waters, E., Shaw, L., De Jager, I. R., ..., & Moore, D. A. (2019). The association between child and adolescent depression and poor attendance at school: A systematic review and meta-analysis. Journal of Affective Disorders , 245 , 928–938. https://doi.org/10.1016/j.jad.2018.11.055

Fowles, D. C. (1994). A motivational theory of psychopathology. In W. Spaulding (Ed.), Integrative views of motivation, cognition, and emotion (pp. 181–238). University of Nebraska Press.

Ghandour, R. M., Sherman, L. J., Vladutiu, C. J., Ali, M. M., Lynch, S. E., Bitsko, R. H., & Blumberg, S. J. (2019). Prevalence and treatment of depression, anxiety, and conduct problems in US children. The Journal of Pediatrics, 206 , 256–267. https://doi.org/10.1016/j.jpeds.2018.09.021

Grant, H., & Dweck, C. S. (2003). Clarifying achievement goals and their impact. Journal of Personality and Social Psychology, 85 , 541–553. https://doi.org/10.1037/0022-3514.85.3.541

Gray, J. A. (1982). The neuropsychology of anxiety: An enquiry into the functions of the septo-hippocampal system . Oxford University Press.

Harrer, M., Cuijpers, P., Furukawa, T., & Ebert, D. D. (2019). Dmetar: Companion r package for the guide ’doing meta-analysis in r’. http://dmetar.protectlab.org/

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis . Academic Press. https://doi.org/10.1016/C2009-0-03396-0

Book   Google Scholar  

Hoyt, ACDR&WT. (2014). MAd: Meta-analysis with mean differences. https://CRAN.R-project.org/package=MAd

Huang, C. (2011). Achievement goals and achievement emotions: A meta-analysis. Educational Psychology Review, 23 , 359–388. https://doi.org/10.1007/s10648-011-9155-x

Hulleman, C. S., Schrager, S. M., Bodmann, S. M., & Harackiewicz, J. M. (2010). A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? Psychological Bulletin, 136 , 422–449. https://doi.org/10.1037/a0018947

Katz, B. A., Matanky, K., Aviram, G., & Yovel, I. (2020). Reinforcement sensitivity, depression and anxiety: A meta-analysis and meta-analytic structural equation model. Clinical Psychology Review, 77 , 101842. https://doi.org/10.1016/j.cpr.2020.101842

King, R. B., McInerney, D. M., & Nasser, R. (2017). Different goals for different folks: A cross-cultural study of achievement goals across nine cultures. Social Psychology of Education, 20 , 619–642. https://doi.org/10.1007/s11218-017-9381-2

Korn, R. M., & Elliot, A. J. (2016). The 2 x 2 standpoints model of achievement goals. Frontiers in Psychology, 7 , 1–12. https://doi.org/10.3389/fpsyg.2016.00742

Korn, R. M., Elliot, A. J., & Daumiller, M. (2019). Back to the roots: The 2 × 2 standpoints and standards achievement goal model. Learning and Individual Differences, 72 , 92–102. https://doi.org/10.1016/j.lindif.2019.04.009

Koutsimani, P., Montgomery, A., & Georganta, K. (2019). The relationship between burnout, depression, and anxiety: A systematic review and meta-analysis. Frontiers in Psychology, 10 , 284. https://doi.org/10.3389/fpsyg.2019.00284

Liem, G. A. D., & Senko, C. (2022). Goal complexes: A new approach to studying the coordination, consequences, and social contexts of pursuing multiple goals. Educational Psychology Review, 34 , 2167–2195. https://doi.org/10.1007/s10648-022-09701-5

Liu, X., Gao, X., & Ping, S. (2019). Post-1990s college students academic sustainability: the role of negative emotions, achievement goals, and self-efficacy on academic performance. Sustainability, 11 (3), 775. https://doi.org/10.3390/su11030775

Liu, X., Zhang, Y., Cao, X., et al. (2024). Does anxiety consistently affect the achievement goals of college students? A four-wave longitudinal investigation from China. Current Psychology, 43 , 10495–10508. https://doi.org/10.1007/s12144-023-05184-x

Lochbaum, M., & Gottardy, J. (2015). A meta-analytic review of the approach-avoidance achievement goals and performance relationships in the sport psychology literature. Journal of Sport and Health Science, 4 , 164–173. https://doi.org/10.1016/j.jshs.2013.12.004

Lüdecke, D. (2019). Esc: Effect size computation for meta analysis (version 0.5.1). https://doi.org/10.5281/zenodo.1249218

Luttenberger, S., Wimmer, S., & Paechter, M. (2018). Spotlight on math anxiety.  Psychology Research and Behavior Management , 311–322. https://doi.org/10.2147/PRBM.S141421

Madjar, N., Ratelle, C. F., & Duchesne, S. (2021). A longitudinal analysis of the relationships between students’ internalized symptoms and achievement goals. Motivation Science, 7 , 207–218. https://doi.org/10.1037/mot0000195

Măirean, C., & Diaconu-Gherasim, L. R. (2020). Depressive symptoms and achievement goals: Parental rejection as a moderator. The Journal of Early Adolescence, 40 , 1369–1396. https://doi.org/10.1177/0272431619858417

Marx, W., Penninx, B. W., Solmi, M., Furukawa, T. A., Firth, J., Carvalho, A. F., & Berk, M. (2023). Major depressive disorder. Nature Reviews Disease Primers, 9 , 44. https://doi.org/10.1038/s41572-023-00454-1

Melton, T. H., Croarkin, P. E., Strawn, J. R., & McClintock, S. M. (2016). Comorbid anxiety and depressive symptoms in children and adolescents: A systematic review and analysis. Journal of Pychiatric Practice, 22 , 84–98. https://doi.org/10.1097/PRA.0000000000000132

Middleton, M. J., & Midgley, C. (1997). Avoiding the demonstration of lack of ability: An underexplored aspect of goal theory. Journal of Educational Psychology, 89 , 710–718. https://doi.org/10.1037/0022-0663.89.4.710

Midgley, C., Maehr, M. L., & Urdan, T. (1993). Manual for the Patterns of Adaptive Learning Survey (PALS) . University of Michigan.

Nicholls, J. G. (1984). Achievement Motivation: Conceptions of ability, subjective experience, task choice, and performance. Psychological Review, 91 , 328–346. https://doi.org/10.1037/0033-295X.91.3.328

Niemivirta, M. (2002). Motivation and performance in context: The influence of goal orientations and instructional setting on situational appraisals and task performance. Psychologia, 45 , 250–270.

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ..., & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery , 88 , 105906. https://doi.org/10.1016/j.ijsu.2021.105906

Payne, S. C., Youngcourt, S. S., & Beaubien, J. M. (2007). A meta-analytic examination of the goal orientation nomological net. Journal of Applied Psychology, 92 , 128–150. https://doi.org/10.1037/0021-9010.92.1.128

Pintrich, P. R. (2000). An achievement goal theory perspective on issues in motivation terminology, theory, and research. Contemporary Educational Psychology, 25 , 92–104. https://doi.org/10.1006/ceps.1999.1017

R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

Riglin, L., Petrides, K. V., Frederickson, N., & Rice, F. (2014). The relationship between emotional problems and subsequent school attainment: A meta-analysis. Journal of Adolescence, 37 , 335–346. https://doi.org/10.1016/j.adolescence.2014.02.010

Roedel, T. D., Schraw, G., & Plake, B. S. (1994). Validation of a measure of learning and performance goal orientations. Educational and Psychological Measurement, 54 , 1013–1021.

Roskes, M., Elliot, A. J., & De Dreu, C. K. W. (2014). Why is avoidance motivation problematic, and what can be done about it? Current Directions in Psychological Science, 23 , 133–138. https://doi.org/10.1177/0963721414524224

Ryan, R. M., & Deci, E. L. (2019). Brick by brick: The origins, development, and future of self-determination theory. In A. Elliot (Ed.), Advances in motivation science (Vol. 6, pp. 111–156). Elsevier.

Senko, C. (2016). Achievement goal theory: A story of early promises, eventual discords, and future possibilities. In K. Wentzel & D. Miele (Eds), Handbook of motivation at school (2nd ed., pp. 75–95). Routledge.

Senko, C., & Dawson, B. (2017). Performance-approach goal effects depend on how they are defined: Meta-analytic evidence from multiple educational outcomes. Journal of Educational Psychology, 109 , 574–598. https://doi.org/10.1037/edu0000160

Senko, C., & Tropiano, K. L. (2016). Comparing three models of achievement goals: Goal orientations, goal standards, and goal complexes. Journal of Educational Psychology, 108 , 1178–1192. https://doi.org/10.1037/edu0000114

Sheldon, E., Simmonds-Buckley, M., Bone, C., Mascarenhas, T., Chan, N., Wincott, M., ..., & Barkham, M. (2021). Prevalence and risk factors for mental health problems in university undergraduate students: A systematic review with meta-analysis. Journal of Affective Disorders , 287 , 282–292. https://doi.org/10.1016/j.jad.2021.03.054

Sideridis, G. D. (2005). Goal orientation, academic achievement, and depression: Evidence in favor of a revised goal theory framework. Journal of Educational Psychology, 97 , 366–375. https://doi.org/10.1037/0022-0663.97.3.366

Sideridis, G. D. (2007). Why are students with LD depressed? A goal orientation model of depression vulnerability. Journal of Learning Disabilities, 40 (6), 526–539. https://doi.org/10.1177/00222194070400060401

Skaalvik, E. M. (1997). Self-enhancing and self-defeating ego orientation: Relations with task and avoidance orientation, achievement, self-perceptions, and anxiety. Journal of Educational Psychology, 89 , 71–81. https://doi.org/10.1037/P022-0663.89.1.71

Sommet, N., Elliot, A. J., & Sheldon, K. M. (2021). The “what” and “why” of achievement motivation: Conceptualization, operationalization, and consequences of self-determination derived achievement goal complexes. In R. Robbins & O. John (Eds.), Handbook of personality psychology: Theory and research, 4 th Ed (pp, 104–121). Guilford Press.

Sommet, N., & Elliot, A. J. (2017). Achievement goals, reasons for goal pursuit, and achievement goal complexes as predictors of beneficial outcomes: Is the influence of goals reducible to reasons? Journal of Educational Psychology, 109 , 1141–1162. https://doi.org/10.1037/edu0000199

Spoelma, M. J., Sicouri, G. L., Francis, D. A., Songco, A. D., Daniel, E. K., & Hudson, J. L. (2023). Estimated prevalence of depressive disorders in children from 2004 to 2019: A systematic review and meta-analysis. JAMA Pediatrics., 177 , 1017–1027. https://doi.org/10.1001/jamapediatrics.2023.3221

Steare, T., Lewis, G., Lange, K., & Lewis, G. (2024). The association between academic achievement goals and adolescent depressive symptoms: A prospective cohort study in Australia. Lancet Child Adolescent Health, 8 , 413–421. https://doi.org/10.1016/S2352-4642(24)00051-8

Stornes, T., & Bru, E. (2011). Perceived motivational climates and self-reported emotional and behavioural problems among Norwegian secondary school students. School Psychology International, 32 , 425–438. https://doi.org/10.1177/0143034310397280

Van Boekel, M., & Martin, J. M. (2014). Examining the relation between academic rumination and achievement goal orientation. Individual Differences Research, 12 (4-A), 153–169.

Vandewalle, D. (1997). Development and validation of a work domain goal orientation instrument. Educational and Psychological Measurement, 57 , 995–1015. https://doi.org/10.1177/0013164497057006009

Wang, Y., Liu, L., Ding, N., Li, H., & Wen, D. (2021). The mediating role of stress perception in pathways linking achievement goal orientation and depression in Chinese medical students. Frontiers in Psychology, 12 , 614787. https://doi.org/10.3389/fpsyg.2021.614787

Wei, J. (2018). Academic contingent self-worth of adolescents in mainland China: Distinguishing between success and failure as a basis of self-worth. The Chinese College of Hong Kong. ProQuest Dissertations Publishing, 10805400.

Wirthwein, L., Sparfeldt, J. R., Pinquart, M., Wegerer, J., & Steinmayr, R. (2013). Achievement goals and academic achievement: A closer look at moderating factors. Educational Research Review, 10 , 66–89. https://doi.org/10.1016/j.edurev.2013.07.001

Yap, M. B., Morgan, A. J., Cairns, K., Jorm, A. F., Hetrick, S. E., & Merry, S. (2016). Parents in prevention: A meta-analysis of randomized controlled trials of parenting interventions to prevent internalizing problems in children from birth to age 18. Clinical Psychology Review, 50 , 138–158. https://doi.org/10.1016/j.cpr.2016.10.003

Yeager, D. S., & Dweck, C. S. (2023). Mindsets and adolescent mental health. Nature Mental Health, 1 , 79–81. https://doi.org/10.1038/s44220-022-00009-5

Zusho, A., & Clayton, K. (2011). Culturalizing achievement goal theory and research. Educational Psychologist, 46 , 239–260. https://doi.org/10.1080/00461520.2011.614526

Download references

Acknowledgements

The study was supported by a grant of the Romanian Ministry of Education and Research, CNCS-UEFISCDI, project number PN-III-P4-ID-PCE-2020-2963, within PNCDI III.

Author information

Authors and affiliations.

Department of Psychology and Educational Sciences, Alexandru Ioan Cuza University, 3 Toma Cozma, 700554, Iasi, Romania

Loredana R. Diaconu-Gherasim, Alexandra S. Zancu, Cornelia Măirean & Irina Crumpei-Tanasă

Department of Psychology, University of Rochester, Rochester, NY, USA

Andrew J. Elliot

Gordon F. Derner School of Psychology, Adelphi University, Garden City, NY, USA

Laura E. Brumariu

Department of Psychology and Educational Sciences, University of Bucharest, Bucharest, Romania

Cristian Opariuc‑Dan

Ovidius University, Constanta, Romania

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Loredana R. Diaconu-Gherasim .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 295 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Diaconu-Gherasim, L.R., Elliot, A.J., Zancu, A.S. et al. A Meta-Analysis of the Relations Between Achievement Goals and Internalizing Problems. Educ Psychol Rev 36 , 109 (2024). https://doi.org/10.1007/s10648-024-09943-5

Download citation

Accepted : 23 August 2024

Published : 16 September 2024

DOI : https://doi.org/10.1007/s10648-024-09943-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Achievement goals
• Internalizing problems
• Find a journal
• Publish with us
• Track your research