44 Drafting the Methods Section

Drafting the Methods Section

While the substance of the methods section will differ by genre and method chosen, basic components can be derived across genres. Methods sections in the social sciences tend to have five sub-components. They must:

  1. summarize the method used while arguing the value and limitations of the method for your data/context;
  2. discuss the data of your study (participants, artifacts, academic literature)
  3. present the and measures used;
  4. outline the of the data (i.e. how it was collected and ensured of its integrity), and
  5. Analysis: discuss how you analyzed your data.

Splitting these sections into components, however, should not give you the impression that the methods section is merely a list. The methods section is also an argument (Johnson, 2018), meaning that it aims to convince your reader of the value of your method through a narrative that briefly applies your method to the context of your research. In addition, the method must find a way to align with the research question stated in the introduction. Your methodology should build upon the introduction, justifying that the approach you utilized to investigate the problem is suitable (see Table 7.3.1 for common justifications for some established methods). We will refer back to this key point as we overview each step of the methods section.

A simple summary of your method is a good way to begin the method’s section. The length and depth of this summary depends entirely on the method chosen and the audience it is presented to. If the method is commonly known and used within your field, an overly intricate summary of the method is not necessary; a couple sentences overview will work fine. If the method is not commonly used or entirely new, however, more argument will be required for your audience to understand your choice of method. For instance, grounded theory is a commonly used paradigm in many qualitative social science journals. It is therefore unnecessary to sketch the method’s history in detail. Rather, a simple definition, such as grounded theory being an inductive approach which only designates codes after data collection (and how you enacted in your project) will suffice. On the other hand, newer methods such as linked data (methods based on web technologies such as HTTP, RDF, SPARQL, and URIs to enable semantic connections between various databases) would require a more extensive discussion. In addition, while it is good to offer your own summary of the method, the definition used by another researcher (particularly methodological researchers) is a simple way to align your research with a legitimate approach in your field.

Table 7.1 - Justifications for Some Common Methods
Theory Justifications
Grounded Theory Strong for exploratory research (where limited existing work are available), as it is able to be open to new themes and codes that the researcher may not have had in mind before combing through their data

Streamlines and integrates data collection and analysis; flexible to research contexts; produces ‘thick descriptions’ (Charmaz, 2003)

Content Analysis Able to establish the frequency and meaning of particular words, phrases and themes in a larger corpus; it also helps to establish relationships and patterns between them.

Offers objective, systematic and quantitative description

In-Depth Interviews Strong method for unpacking the depth of a participants understanding of a situation; explores issues in great detail

May allow respondents to feel more comfortable to share information based on rapport established in the research process

Ethnography Strong for community research, online research, and other research that requires careful observation of the interactions between participants

Able to capture the behavior of a participant in their environment as opposed to in the staged interview or experimental setting

Discourse Analysis Suitable for understanding a discussion between participants over a specific theme (such as Uber, beard products, or memes)

Appropriate for understanding underlying meanings in sociohistorical contexts and for revealing how language and discourse shape reality

Systematic Literature Review Takes a comprehensive approach to reviewing the literature, drawing on research from multiple theoretical, methodological and disciplinary concerns

Identifies biases and gaps in the literature, and nuances such as whether generalizations can be made across populations, subgroups, settings etc.

Surveys Inexpensive method for aggregating large quantities of data. High level of representability

Flexible across contexts and in design (e.g. online, paper etc.) and can be adapted for anonymity; can overcome interviewer effect.

Panels studies Suitable for analyzing social change, life course and understanding the interrelationships between later outcomes

Enables us to make causal inferences through controlling unobserved heterogeneity (Laurie, 2020)

Quantitative (e.g.,, regression or chi-square analyses) Strong method for establishing relationship between various variables; reliable for determining variables that impact our research topic

It also helps us to identify outliers and anomalies

The key point to keep in mind for summarizing your method is to outline its theory insofar as it explains your procedure, that is, discuss the method’s intentions with respect to how you applied it. We will touch on this again in the procedure section.

As you provide an overview of your method, you must also justify it. Justification of your method must appeal to the method’s ethical, practical and factual utility for the project. Ethical justifications are those that argue that the method is best for reducing the harm of research on your participants (and the communities they inhabit). It must also make the point that your method aligns with the principles of your institution’s ethical values.

Factual issues have to do with the value of the data your method is able to gather. It must consider whether the method is appropriate to your research question and topic. If, for instance, the research question is about “Malaysian immigrants’ conception of justice in comparison to American immigrants,” then you may argue that ‘in-depth interviews’ are the only approach deep enough to unravel a person’s “conception of justice.” Arguments for the factual benefits of a method frequently highlight its novelty for studying a particular method. For instance, there could be a lack of discourse analyses of Uber’s advertising materials (the previous research being content analyses). One could then argue that a discourse analysis approach not only has merits in its own right, but it also may be able to discover data which other approaches miss. Finally, you should outline the strengths of the methods in relation to all aspects of your research process (e.g., alignment with your theory, personal values, practicality etc.).

Practical justifications highlight why the method is suitable given logistics, administrative and everyday concerns. For example, if you are interested in studying how Malaysian migrants’ prepare for their transit to America, an ethnographic approach might be tempting. But practically (financially and time-wise), you might not be able to visit Malaysia to observe their preparations. Hence, you might decide that surveys or interviews (while less desirable) are more practical in that instance.

Again, we emphasize the importance of highlighting the limitations of your method. Every method has weaknesses, it is vital that show an awareness of them. It is important, however, not to have the weaknesses outweigh your method’s benefits. Your reader should be able to understand why you choose the methods, so you need to explain how, in spite of the limitation, your method is the most suitable choice. Hence, you need to justify why you decided to choose the methods over others. To help in the weighing of the costs and benefits of different methodological approaches, we have provided below a list of the potential limitations of some of the more common methods used in the social sciences.

Table 7.2 - Potential Limitations for Common Methods
Theory Limitations
Grounded Theory Lacks a theoretical base to drive the analysis; requires considerable skills from researchers

Reliability and validity might be questionable due to the lack of standard rules to follow; researcher-driven.

Content Analysis Can lack theoretical base; simplistic and reductive

subject to increased error, particularly when relational analysis is used to attain a higher level of interpretation (see Elo et al, 2014)

In-Depth Interviews Not generalizable to the wider population

May be prone to bias: interviewer-effect is always present

Ethnography It may take time to establish trust in order to generate honest data

Too little data may lead to false assumptions about behaviour patterns, while large quantities of data may not be processed effectively (Baral et al, n.d, p.2.)

Discourse Analysis There are multiple methods for doing discourse analysis, making replicability difficult

It focuses primarily on language which often does not tell the entire story; it might need to be supplemented with another method

Systematic Literature Review The results are only as reliable as the method the original authors uses to evaluate the effect of each study i.e., the results are dependent on the study design, interpretation and analysis of the primary authors.

There is some subjectivity in deciding how to pool studies e.g., determining how to treat discordant studies; subjected to methodological flaws

Surveys Various errors might undermine validity and reliability (e.g., measurement, coverage, non-response sampling errors)

In appropriate for getting in depth understandings

Panels Selective panel attrition can be problematic

Panel conditioning: interviews from previous waves might influence subsequent waves

Quantitative (e.g. regression or chi-square analyses) There may be variables other than the ones in the study which influence the response variable

It does not allow us to identify cause and effect; correlation does not imply causation.

Once you have provided an overview of your method and its value for your context, the next step is to discuss the data or population that the method will be used upon. Summarizing the data or population of your method often requires answering (Johnson, 2017): (a) how many participants/cases compose your data? (b) what are some of the common or key attributes of your data? And c) how did you select your data/participants?

Table 7.3 - Johnson's (2017) Three Questions About Participants
Questions About Participants What this Means?
How many participants/cases compose your data? Simple answer of the size of your corpus
What are some of the common or key attributes? Discuss the relevant demographics/variables of your population (if race/ethnicity, gender, age are relevant, list them here).
How did you select your data/participants? Discuss the sampling method you used
Source: Johnson, M. (2017). "Writing a Methods Section" In Allen, M. (2017). The SAGE Encyclopedia of Communication Research Methods (Vols. 1-4). Thousand Oaks, CA: SAGE Publications, Inc doi: 10.4135/9781483381411

The first question means answering the size (e.g., n = 88) of your corpus. Commentary on the size of your sample should also consider the total population that you are attempting to comment on. For instance, if you are writing about “student experiences with online open book examinations,” then it is important to consider how many students participated in the research compared to the students that took the examination (i.e., identify the characteristics of the sample versus the target populations). Answering the common of this case study would mean considering the descriptives (usually sample size, mean, distribution etc.) or characteristics (e.g. gender, age, class, ethnicity) of participants/unit of observation. Building from our previous example, we should ask pertinent questions about the participants that took this exam: what was the average score of the exam? What grade level is this course? Which discipline was this course in, and which discipline do most students in this specific class come from? Finally, we need to also discuss how we recruited these students. Outline your by discussing how you advertised the study, whether a stipend or incentive was offered, and how students finally agreed to join in the study (mainly regarding whether consent forms were required). For online texts or any other data that is not a participant/population, the same questions can be answered. The amount of the texts, components of the texts (their genre and author mainly), and the data collection process is all relevant to your method.

Box 7.1 – Writing About Recruitment

  • Have I discussed how I advertised the study?
  • Have I stated if a stipend was used?
  • Have I stated if a consent form was used?
  • How were participants identified?
  • Where were they recruited?

While the above questions must be asked for all methods, the key concerns will differ depending on whether your research is primary, secondary, content analysis or theoretical research. is research that collects and derives its own raw data by sampling participants/cases, such as selecting and interviewing participants. uses data from other primary research projects, such as a systematic literature review of other research. uses other papers and articles, sometimes even other data (like secondary research), but with the purpose of advancing a new argument in the field. Its method does not have to be as explicit as the other two methods of research. The following table summarizes the distinct tasks of the three approaches when discussing methodology. As theoretical research often uses articles and other to make its argument, its methodological concerns are similar to content analysis. For that reason, we have grouped them together in the following table.

Table 7.4 - Contrasting Primary/Secondary/Theoretical Research Methods
Primary Research Secondary Research Theoretical Essays
Target Sample

Discuss who the target sample was (how many, characteristics). This includes all the points that are discussed above.

Check Method of Original Research

Check the methodology of the original research and justify its association with your project.

Sampling Method

Discuss how you sample your artifacts (e.g., articles, blog posts, social media posts). This includes discussing the search term criteria that you have used to find these artifacts.

Justify Target

Argue the value of selecting and interviewing your target sample with respect to your study.

Alter Method to fit your project

State if you have removed or changed anything about the methodologies of the studies you have sourced data from.

Author and Genre

Discuss the author and audience. Briefly summarize them so you can elaborate further throughout your paper.

Sampling Method

State your recruitment method and also disclose if a stipend or any other material was used to recruit participants.

Summarize Population

If you are focusing on only a particular subpopulation, provide an overview of the sample, but discuss your subsample in detail. Justify why the focus on this subpopulation

Size of Corpus

Discuss how many articles you found initially and how many you ended up upon. Discuss this in relation to the total number of articles that might have met your search criteria (if you found them all then this means your corpus was exhaustive).

It is important to note that merely listing these components of your data/participants does not satisfy the requirements of a good methods section. Remember that you are still making an argument. While discussing the attributes relevant to your study, you are still arguing why you selected these participants/cases for your study, why you are listing that demographic/characteristic(s) as important to your study, and why you chose this sampling method over the other options. Allow yourself to be guided by the argument and the relevant things to include in your sections will follow. The following box provides a checklist of questions to help you to evaluate whether your methods section has successfully addressed all the key questions regarding your ‘data’.

Box 7.2 – Checkist for Summarizing the Data of Your Study

  • Have I stated how many participants/data was in my study?
  • Including in comparison to the relevant groups that they occupy?
  • Have I discussed all the key attributes of my data?
  • Have I discussed the relevant demographics of the populations and groups I am researching?
  • If it is textual data, what is important to know about the author and genre of this text?
  • Have I discussed the sampling method of my study?
  • Have I argued the value and limitations of my sampling method?

Instruments and Measures

Research instruments are the tools that you use to collect, analyze and measure data in your research. These include literature reviews, survey questionnaires, interviews, focus groups, ethnography etc. In your methods section, you must describe how these serve to collect data, for example, was the survey composed of only open-ended questions, how many questions were included in the survey etc. Likewise, if you are using interviews, you need to discuss what kind of interviews were conducted (semi-structured, unstructured, structured)? How were the questions organized (e.g. by theme, no order etc.). Again, if you are doing a meta-analysis (literature review), you need to detail how it was done, what search engines, databases and search criteria were used etc. To reiterate, description of your instruments requires a an outline of how you used the particular measure to collect data.

As you will remember from your methods classes, when we talk about measurement in social research, we are referring to the process of (i.e., what concrete observations are being checked to empirically indicate a concept?). To do this, you must first identify the key variables or issues in your research question and explain how your research instrument captures it. For example, say your research question is “Do international students engage in economic transnationalism while studying on campus?” You will want to indicate how you will determine (a) who is an international student? and (b) what is economic transnationalism? It might seem silly that you need to operationalize “international student” but practically, you must have a system to determine who is an international student. It would be hard to walk around campus and determine who is an international student without having some criteria e.g. a student without Canadian citizenship. But again, how would you determine which student has Canadian citizenship and which ones do not? Operationalization requires that you specify precisely how you determine who is an international student (maybe you simply let participants self-declare their status on the survey or at the beginning of the interview). Likewise, “economic transnationalism” is a concept that needs to be measured (operationalized). You need to specify how in your research you will determine that economic transnationalism was taking place. Maybe you determined that if participants engaged in at least one economic activity (such as remitting money to origin countries, conducting business in origin countries, investing in origin countries etc), then economic transnationalism has taken place. The key to measurement is making it clear (and justifying) to your readers how your key concepts and variables are determined in the study. This requires that you consider how other researchers have measured these variables. If your measures are different from how other researchers measure the same concepts/variables, you need to justify why. On the other hand, the use of existing measures assures measurement consistency and contributes to the reliability of your study.

Measurement in quantitative studies can get complex by recoding, and the creation of composites such as indexes and scales. You must discuss how you recode variables and how the new variables allow you to better measure concepts. Likewise, you need to justify how the scales and indexes that you are using improve your measurements. If you are using established indexes and scales, it is important that you justify why they are applicable in your research. Again, if you are creating new measures, you need to identify why existing ones were inadequate for your research goals. As mentioned before, the methodology is not merely checking boxes, it requires justification of your choices, and engagement with an argument and or the literature (see the following example).

Box 7.3 – Examples – How a Team of Researchers Measured Friendship

Excerpt from Waldsorth et al (2021, p. 138) showing how friendship was operationalized in a research project:
Whenever feasible, we adopted measures already used in the literature to facilitate comparisons. Two aspects of friendship are of interest: the nationality of friends and the strength of the friendships. We asked respondents to rank list their top ten friends and identify the nationality of each friend. This is a modification of a friendship grid used by Hendrickson et al. (2011) to study international students in Hawaii. Hendrickson et al. (2011) used social network analysis to map, for each student, a list of all people with whom they interact, recording if they are from their own country or another country, and a self-determined measure of the strength of each relationship on a scale of 1–10. We felt this was a valuable measurement tool; however, we also felt that asking participants to record an exhaustive list of every person with whom they socialize was an overly onerous task, and not necessary for our purpose.
We therefore modified this grid by asking respondents to rank order the top 10 friends with whom they socialize the most and to identify their friends’ national origin (host-national, co-national or other international). In line with Jindal-Snape and Rienties’s observation that non-university “local-community” can play an important role in supporting international students (2016, p. 6), we do not limit respondents to listing friends who attend the university, which is in keeping with other studies.
Also, like other studies, we only permit participants to list a maximum of ten friends…Respondents ranked their closest friends from 1–10, based on the amount of time spent socializing; friend 1 is the friend with whom they socialize the most. As a result, in our study we operationally define an increase in time spent together as indicative of closeness. According to Hall (2019), the closeness of a friendship is appropriately measured by the time spent together; transitioning from an acquaintance to a friend and to a close friend occurs with increased time spent together. Furthermore, since there are arguably cultural differences in the meanings of close friends as opposed to acquaintances (Gudykunst et al., 1985; Hendrickson et al., 2011; Maeda & Ritchie, 2003), we employ ranked time spent together as a measure of closeness as opposed to a self-determined scale of the strength of the friendship in an effort to control cultural bias.
We created three dummy variables: host-national (Canadian), co-national (from the same country as the respondent), other international (from another country but not the host or the respondent’s home country). From the dummy variables, we constructed measures to determine the proportion of respondents’ friends (all ten friends) that are from the host-nation, from their own country, or from a different country. For each of these three ratios we created additional variables to measure whether or not the designated friendship (host, other, or co-national) was a close or distant friend: we segmented the friendship spectrum by creating measures to identify the nationality of respondents’ closest three friends (friends 1–3), middle friends (friends 4–7) and the last three friends (8–10). The segmented measures of friendship facilitate an examination of the effect of different types of friends on the international student satisfaction measures at different levels of the friendship strength spectrum.

Source: Walsworth, S., Somerville, K., & Robinson, O. (2021). The importance of weak friendships for international student satisfaction: Empirical evidence from Canada. International Journal of Intercultural Relations, 80, 134-146.

Procedure

The procedure is a step-by-step description of what you did to collect and analyze your data (see Figure 7.6.1 and Box 7.6.3). It should be the largest part of your methods section (Johnson, 2017), and it will incorporate explanation of how you collected your data, interpreted your data, and organized your data in your final-write up. As pointed out in the introduction, it is in this part of the methods section that the imitation function of methods – helping others to find our findings – is completed (Johnson, 2017). After providing a walkthrough of each step of your research process, it then discusses how you analyzed that data. The procedure clearly divides and orders the steps of your research chronologically, seeking to present a summative narrative of your research from recruitment/sampling of data to analysis. Documenting your procedure allows another researcher to attempt the exact same process as you with the expectation that they should be able to find (roughly) the same results. It is through this cross-reference system that the method can likewise become generalizable. If other researchers can replicate your procedure across contexts with effective results, then the method proves itself intuitive and effective to be useful for further research. It is an important kind of tedium!

Box 7.4 – Procedure Checklist

Recruitment

  • Did I discuss how I recruited my participants and/or collect my data? Did I mention and outline the type of sampling method I used?
  • Did I discuss how I ensured ethical fairness in gathering my data?
  • Did I discuss how many participants I reached out to, and if any problems occurred in gathering my data?
  • Did I discuss how I solved problems in recruiting my participants?

Data Collection

  • Have I discussed the method for collecting data in my study (interviews, surveys, census data)?
  • Have I discussed procedures for cleaning, recoding, and otherwise ensuring the integrity of my data?

Analysis

The final aim of your methods section will be to discuss how you processed and analyzed the data collected. This will differ significantly depending on the method chosen, but there are a few standard things to do depending on whether your paper is qualitative or quantitative. The following table highlights:

Table 7.5 - Steps in Describing Your Analytical Strategy
Quantitative Qualitative
Describe how you processed data for analysis (e.g., how you cleaned the data set, how you treated missing and extreme values, did you transform or recode variables for analysis?) Describe how you processed the data for analysis (e.g. transcription verbatim or for general ideas) and the coding procedures identified.
Identify what software was used to conduct analyses (e.g. Excel, SPSS, STATA, Python etc) Identify what software if any was used to facilitate analysis (e.g. NViVo, AtlasTi etc)
Describe what statistical test was used (e.g. regression, ANOVA etc.) Describe what kind of analytical strategy was used (e.g. grounded theory, thematic analysis, content analysis, discourse analysis etc.).


The next table provides a demonstration of the analysis portion of Alexander Wilson’s (2021) methodology in his undergraduate honours thesis.

Table 7.6 - Breakdown of Alexander's Analysis
Type Analysis
Analysis program “After the articles and government reports were collected, I uploaded them into NVivo where they were read with attention to the context of the discourse.”
Open Reading into Coding “After an open reading, I divided the issues present in the media into Beckert and Dewey’s (2017, p. 14) two concepts, "externalities and hopes for the future," while paying attention to the time and narrator framing Uber's legitimacy or illegitimacy.”
Example of coding “The externalities voiced in the literature, such as lower wages and safety concerns, were compared alongside the hopes expressed by Uber representatives and finally taken up by the government in legislative decisions.”
Argument for discourse analysis to analyze this data “I used an interpretive method (discourse analysis) to allow room for me to reason and justify my understanding of the context and influence of Uber's frame on the broader debate”
Source: Wilson, A. (2021). Driver’s of Dissidence: A Discourse Analysis of Vancouver’s Road to Ride-Hailing. Undergraduate Thesis. (p. 13).


The above table shows that honours student, Alexander Wilson (2021) began by discussing the instrument used to organize the analysis (NVivo) and then moves onto the steps taken to conduct a discourse analysis (which is overviewed earlier in the paper). It discusses how the media data was analyzed, through an open reading to a codification of the issues according to their being “hopes or externalities” of Uber’s service. The codification is situated within the overall method of discourse analysis, which seeks to interpret the meaning of speech/text with respect to the larger discussion that it is contributing to. For Alexander, this larger discussion leads to a change in legislation, so it is his goal to interpret the significance of Uber’s promises and externalities in the media discourse with respect to the final legislative conclusions.

 

Box 7.5 – Reviewing and Revising the Methods Section

  • Have I summarized the method?
  • Will this summary make sense to someone doing similar research?
  • Have I adequately highlighted the elements of this method which are especially relevant for my research?
  • Have I adequately highlighted elements of this method which are relevant for my argument regarding its value for my research?
  • Have I argued the value of my approach?
  • Have I argued that this methodological approach will be effective for gathering data in an ethical manner?
  • What other approaches can I compare my method with? What are the tradeoffs?
  • Have I summarized the data of my study?
  • Have I clarified the type of my data (participants, text, articles etc.)?
  • Have I stated common attributes of that data?
  • Have I disclosed important ethical concerns regarding interaction with that data or population?
  • Have I outlined the instruments and materials used in my study?
  • Have I discussed the design and procedure of my study?
  • Have I outlined how I analyzed the data?
  • Did I adequately discuss the key steps of my analysis?
  • Did I state how many iterations of my analysis were conducted (how many readings, how many calculations)?

References

Johnson, M. (2017). “Writing a Methods Section” In Allen, M. (2017). The SAGE Encyclopedia of Communication Research Methods (Vols. 1-4). Thousand Oaks, CA: SAGE Publications, Inc doi: 10.4135/9781483381411

Walsworth, S., Somerville, K., & Robinson, O. (2021). The importance of weak friendships for international student satisfaction: Empirical evidence from Canada. International Journal of Intercultural Relations80, 134-146.

Gudykunst, W. B., Yang, S., & Nishida, T. (1985). A Cross-Cultural Test of Uncertainty Reduction Theory: Comparisons of acquaintances, friends, and dating relationships in japan, korea, and the united states. Human Communication Research, 11(3), 407-454. https://doi.org/10.1111/j.1468-2958.1985.tb00054.x

Maeda, E., & Ritchie, L. D. (2003). The concept of shinyuu in japan: A replication of and comparison to cole and Bradac’s study on U.S. friendship. Journal of Social and Personal Relationships, 20(5), 579-598. https://doi.org/10.1177/02654075030205001

Hall, J. A. (2019). How many hours does it take to make a friend? Journal of Social and Personal Relationships, 36(4), 1278-1296. https://doi.org/10.1177/0265407518761225

 

B. Hendrickson, D. Rosen, R.K. Aune. (2011). “An analysis of friendship networks, social connectedness, homesickness, and satisfaction levels of international students.” International Journal of Intercultural Relations, 35 (3) (2011), pp. 281-295

License

Icon for the Creative Commons Attribution-NonCommercial 4.0 International License

Practicing and Presenting Social Research by Oral Robinson and Alexander Wilson is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.

Share This Book