5. RESEARCH METHODS
5.2 Finding and Evaluating Research Sources
In this “information age” when so much information is available at our fingertips on the Internet, it is crucial to be able to critically search through the reams of information in order to select credible sources that can provide reliable and useful data to support your ideas and convince your audience. In the era of “fake news,” deliberate misinformation, and “alternative facts,” developing the skill to evaluate the credibility of sources is critical.
From your previous academic writing course, you are familiar with academic journals and how they differ from popular sources, as in Figure 5.2.1. If you would like to refresh your memory on this, watch UVic library’s video on Scholarly and Popular Sources. These contain peer-reviewed articles written by scholars, often presenting their original research, reviewing the original research of others, or performing a “meta-analysis” (an analysis of multiple studies that analyze a given topic).
Scholarly articles published in academic journals are usually required sources in academic research essays; they are also an integral part of engineering projects and technical reports. Many projects require a literature review, which collects, summarizes and sometimes evaluates the work of researchers in this field whose work has been recognized as a valuable contribution to the “state of the art.” However, they are not the only kind of research you will find useful. Since you are researching in a professional field and preparing for the workplace, there are many credible kinds of sources you will draw on in a professional context. Table 5.2.1 lists several types of sources you may find useful in researching your projects.
| [Skip Table] |
|
| Source Type | Description |
|---|---|
| Academic Journals, Conference Papers, Dissertations, etc. |
Scholarly (peer-reviewed) academic sources publish primary research done by professional researchers and scholars in specialized fields, as well as reviews of that research by other specialists in the same field. For example, the Journal of Computer and System Sciences publishes original research papers in computer science and related subjects in system science; International Journal of Robotics and Animation is one of the most highly ranked journals in the field. |
| Reference Works |
Specialized encyclopaedias, handbooks and dictionaries can provide useful terminology and background information. For example, the Kirk-Othmer Encyclopedia of Chemical Technology is a widely recognized authoritative source. Do not cite Wikipedia or dictionary.com in a technical report. |
| Books
Chapters in Books |
Books written by specialists in a given field and contain a References section can be very helpful in providing in-depth context for your ideas. For example, Designing Engineers by Susan McCahan et al. has an excellent chapter on effective teamwork |
| Trade Magazines and Popular Science Magazines |
Reputable trade magazines contain articles relating to current issues and innovations, and therefore they can be very useful in determining the “state of the art” or what is “cutting edge” at the moment, or finding out what current issues or controversies are affecting the industry. Examples include Computerworld, Wired, and Popular Mechanics. |
| Newspapers (Journalism) |
Newspaper articles and media releases can offer a sense of what journalists and people in industry think the general public should know about a given topic. Journalists report on current events and recent innovations; more in-depth “investigative journalism” explores a current issue in greater detail. Newspapers also contain editorial sections that provide personal opinions on these events and issues. Choose well-known, reputable newspapers such as The New York Times. |
| Industry Websites (.com) |
Commercial websites are generally intended to “sell,” so you have to select information carefully, but these websites can give you insights into a company’s “mission statement,” organization, strategic plan, current or planned projects, archived information, White Papers, technical reports, product details, costs estimates, etc. |
| Organization Websites (.org) |
A vast array of .org sites can be very helpful in supplying data and information. These are often public service sites and are designed to share information with the public. |
| Government Publications and Public Sector Web Sites (.gov/.edu/.ca) |
Government departments often publish reports and other documents that can be very helpful in determining public policy, regulations, and guidelines that should be followed. Statistics Canada,[2] for example, publishes a wide range of data. University web sites also offer a wide array of non-academic information, such as strategic plans, facilities information, etc. |
| Patents |
You may have to distinguish your innovative idea from previously patented ideas; you can look these up and get detailed information on patented or patent-pending ideas. |
| Public Presentations |
Public Consultation meetings and representatives from industry and government speak to various audiences about current issues and proposed projects. These can be live presentations or video presentations available on YouTube or TED talks. |
| Other |
Can you think of some more? (Radio programs, Podcasts, Social Media, etc.) |
The importance of critically evaluating your sources for authority, relevance, timeliness, and credibility cannot be overstated. Anyone can put anything on the internet; and people with strong web and document design skills can make this information look very professional and credible—even if it isn’t. Since much research is currently done online, and many sources are available electronically, developing your critical evaluation skills is crucial to finding valid, credible evidence to support and develop your ideas. In fact, this has become such a challenging issue that there are sites like this List of Predatory Journals that regularly update its online list of journals that subvert the peer review process and simply publish for profit.
Review your understanding of research sources with this H5P interactivity.
Mark Twain, supposedly quoting British Prime Minister Benjamin Disraeli, famously said, “There are three kinds of lies: lies, damned lies, and statistics.” On the other hand, H.G. Wells has been (mis)quoted as stating, “statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.”[3] The fact that the actual sources of both of these “quotations” are unverifiable makes their sentiments no less true. The effective use of statistics can play a critical role in influencing public opinion as well as persuading in the workplace. However, as the fame of the first quotation indicates, statistics can be used to mislead rather than accurately inform—whether intentionally or unintentionally.
When evaluating research sources and presenting your own research, be careful to critically evaluate the authority, content, and purpose of the material, using questions in Table 5.2.2.
| [Skip Table] |
|
| Authority Researchers Authors Creators |
Who are the researchers/authors/creators? Who is their intended audience? What are their credentials/qualifications? What else has this author written? Is this research funded? By whom? Who benefits? Who has intellectual ownership of this idea? How do I cite it? Where is this source published? What kind of publication is it? Authoritative Sources: written by experts for a specialized audience, published in peer-reviewed journals or by respected publishers, and containing well-supported, evidence-based arguments. Popular Sources: written for a general (or possibly niche) public audience, often in an informal or journalistic style , published in newspapers, magazines, and websites with a purpose of entertaining or promoting a product; evidence is often “soft” rather than hard. |
|---|---|
|
Content |
Methodology What is the methodology of their study? Or how has evidence been collected? Is the methodology sound? Can you find obvious flaws? What is its scope? Does it apply to your project? How? How recent and relevant is it? What is the publication date or last update? |
|
Data Is there sufficient data here to support their claims or hypotheses? Do they offer quantitative and/or qualitative data? Are visual representations of the data misleading or distorted in some way? |
|
|
Purpose |
Why has this author presented this information to this audience? Why am I using this source? Will using this source bolster my credibility or undermine it? Am I “cherry picking” – the use of inadequate or unrepresentative data that only supports my position (and ignores substantial amount of data that contradicts it)? Could “cognitive bias” be at work here? Have I only consulted the kinds of sources I know will support my idea? Have I failed to consider alternative kinds of sources? Am I representing the data I have collected accurately? Are the data statistically relevant or significant? |
Given the pie chart in Figure 5.2.2, if you only consulted articles that rejected global warming in a project related to that topic, you would be guilty of cherry picking and cognitive bias.
Beware of Logical Fallacies
There are many logical fallacies that both writers and readers can fall prey to (see Table 5.2.3). It is important to use data ethically and accurately, and to apply logic correctly and validly to support your ideas.
| [Skip Table] | |
| Bandwagon Fallacy |
Argument from popularity – “everyone else is doing it, so we should too!” |
|---|---|
| Hasty Generalization |
Using insufficient data to come to a general conclusion. An Australian stole my wallet; therefore, all Australians are thieves! |
| Unrepresentative Sample |
Using data from a particular subset and generalizing to a larger set that may not share similar characteristics. e.g.: giving a survey to only female students under 20 and generalizing results to all students. |
| False Dilemma |
“Either/or fallacy” – presenting only two options when there are usually more possibilities to consider e.g.: You’re either with us or against us. |
| Slippery Slope |
Claiming that a single cause will lead, eventually, to exaggerated catastrophic results. |
| Slanted Language |
Using language loaded with emotional appeal and either positive or negative connotation to manipulate the reader |
| False Analogy |
Comparing your idea to another that is familiar to the audience but which may not have sufficient similarity to make an accurate comparison e.g.: Governing a country is like running a business. |
| Post hoc, ergo prompter hoc |
“After this; therefore, because of this” e.g.: A happened, then B happened; therefore, A caused B. Just because one thing happened first, does not necessarily mean that the first thing caused the second thing. |
| Begging the Question |
Circular argument – assuming the truth of the conclusion by its premises. e.g.: I never lie; therefore, I must be telling the truth. |
| Ad hominem |
An attack on the person making an argument does not really invalidate that person’s argument. It might make them seem a bit less credible, but it does not dismantle the actual argument or invalidate the data. |
| Straw Man Argument |
Making a “straw man” argument means restating the opposing idea in an inaccurately absurd or simplistic manner to more easily refute or undermine it. |
| Others? |
There are many more… can you think of some? For a bit of fun, check out Spurious Correlations. |
We all have biases when we write or argue; however, when evaluating sources, you want to be on the look out for bias that is unfair, one-sided, or slanted. Consider whether the author has acknowledged and addressed opposing ideas, potential gaps in the research, or limits of the data. Look at the kind of language the author uses: is it slanted, strongly connotative, or emotionally manipulative? Is the supporting evidence presented logically, credibly, and ethically? Has the author cherry-picked or misrepresented sources or ideas? Does the author rely heavily on emotional appeal?
Critical thinking lies at the heart of evaluating sources. You want to be rigorous in your selection of evidence, because once you use it in your paper, it will either bolster your own credibility or undermine it.
- Cover images from journals are used to illustrate difference between popular and scholarly journals, and are for noncommercial, educational use only. ↵
- Government of Canada, Statistics Canada [online]. Available: http://www.statcan.gc.ca/eng/start ↵
- “What is the source of the H.G. Wells quote, ‘Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write/”? Quora n.d.[Online]. Available: https://www.quora.com/What-is-the-source-of-the-H-G-Wells-quote-Statistical-thinking-will-one-day-be-as-necessary-for-efficient-citizenship-as-the-ability-to-read-and-write ↵
- “Why climate deniers have no credibility — in one pie chart,” DeSmog Blog [Online]. Available: https://www.desmogblog.com/2012/11/15/why-climate-deniers-have-no-credibility-science-one-pie-chart ↵
