CRITIQUE OF QUANTITATIVE AND QUALITATIVE RESEARCH1

It is the quality of our work which will please God and not the quantity.--Mahatma Gandhi

OBJECTIVES

The main objective of this handout is to present a framework for critiquing research papers. While the focus of the handout is accounting research, the framework is quite generic and can be amended for use in other fields. I first classify research into two broad categories of quantitative and qualitative and identify their attributes. I then use the proposed framework for critiquing research papers to assess the strength and weaknesses of reports of both qualitative and quantitative research. Upon completion of this handout you will have learned:

1. The definitions of quantitative and qualitative research2. The differences between quantitative and qualitative research methodologies3. A generic framework for critiquing research papers4. A guideline for assessing the research type used in published papers5. A guideline for assessing the details of research in published papers6. A guideline for assessing the writing quality of published papers7. A guideline for assessing the organization of published papers8. A summary statement about the validity and reliability of published papers.

Learning to critique a research report is important for the professional accountant for a number of reasons. The first is that the professional accountant should be able to assess the quality of research to appraise the reliability of its conclusions for use in practice. Like other professionals such as medical doctors or lawyers, the professional accountant should be proficient in assessing the quality of research reports as published in journals or generated through internal research to assess their validity and reliability, and thus their usefulness for practice.

The second reason for assessing the quality of research reports is that professionals are often called upon to provide expert comments on papers that are submitted for publication in professional journal (e.g., the Journal of Accountancy), or a semi-academic, semi-professional journals (e.g., Auditing: A Journal of Practice and Theory). While some may view this service only as overhead with little importance for immediate use in practice, others believe that it actually helps keep the professional accountant abreast of the latest developments in the profession, particularly in one’s area of expertise.

Another benefit from critiquing research papers is that a critique can sharpen one’s skills in conducting professional research in one’s area of specialty. Proficiency in conducting research is a skill that is increasingly important in the complex business environment in which the professional accountant operates. In this environment the accountant may be called upon to use

1 I acknowledge with great appreciation helpful comments from professors Jane Fedorowicz, Jim Hunton and Mahendra Gujarathi on earlier drafts of this manuscript.

1

methodologies such as interviews, surveys, data mining, and experimental methodologies to find answers to contemporary business problems.

QUANTITATIVE AND QUALITATIVE RESEARCH

Definitions

Quantitative research refers to an investigation in which there is a well-developed theory with well-specified variables that can be measured in quantitative terms. As such quantitative research uses data to test the predictions of the theory (i.e., hypotheses) with formal statistical analysis. Data used in quantitative research are expressed in numerical terms and they are collected through various research methods such as empirical or experimental designs or surveys. The results of well-designed quantitative research are said to have high levels of validity and reliability.

Qualitative research refers to a study of a research question for which there is no well-developed theory to predict outcome. Thus, the research is helpful in collecting evidence to develop a new theory or to refine an existing theory. As such qualitative research uses qualitative data to explore and describe a phenomenon. Qualitative data are collected via natural settings such as observations of behavior, interviews, pictures, historical case analysis, literature review, focus groups and the like. Due to concerns about the presence of various intervening variables and weaknesses in researcher’s controls over the research process, scholars believe that the results of qualitative research are tentative with low levels of validity and reliability. If so, these results must be corroborated in more controlled research through quantitative research.

Differences Between Qualitative and Quantitative Research

The definitions provided above suggest fundamental differences between qualitative and quantitative research. Exhibit 1 presents a summary of significant differences between the two methods.2 The nature of reality is one area where qualitative and quantitative researches differ. This difference is sometimes referred to as an ontological assumption. For example, reality is assumed to be generally clear, objective and singular in quantitative research. However, the nature of reality is somewhat subjective and varied with multiple aspects in qualitative research. The reason for this difference is that reality is assumed to be out there in the real world and thus can be measured objectively through quantitative research. However, reality is only the interpretation of the qualitative evidence by the researcher in qualitative research. Thus, depending on researchers’ interactions with the source of evidence (say plant managers in a field study of cost accounting), their own biases and their interpretation, one can come up with different judgments of reality.

[Insert Exhibit 1 About Here]

A related assumption is the relationship between the researcher and that being researched. Also known as the epistemological assumption, it indicates that the researcher is independent of

2 Much of the discussion in this section is adapted from Creswell (1994: 1-7).

2

that being researched in quantitative research, but not in qualitative research. This is because the researcher’s knowledge changes through his or her interactions with that being researched in qualitative research on a dynamic basis. For example, in a focus group the researcher’s knowledge changes constantly resulting in developing follow up questions on the spot based on the nature of the answers to an earlier question. However, the researcher follows a well-planned path for conducting quantitative research, thus staying with the same level of knowledge throughout the research process – no follow up questions, only those that were planned ahead of time.

A third difference between quantitative and qualitative research, also called an axiological assumption, is that while quantitative research is value-free and unbiased, qualitative research is value-laden and biased. Simply put, the researcher in quantitative research focuses on reporting only the facts, controlling for his or her own views or biases. On the other hand, the researcher in qualitative research may interject his or her own biases and interpretations of the qualitative data.

Another difference between quantitative and qualitative research is that while the researcher usually uses formal and impersonal language to report the results of quantitative research, he or she uses more informal and personal language to communicate his/her findings in a qualitative research. This difference is generally known as the rhetoric assumption. For example, the quantitative researcher usually uses the indirect third party style for communication, while the qualitative researcher uses the first (I) or second (you) person to communicate the results of his/her investigation.

The research processes used in quantitative and qualitative research are also different. Known as the methodological assumption, this difference refers to the nature of the process taken by the researcher for investigation. In quantitative research, the study design is static in the sense that the variables being investigated are well isolated based on the theory used and the path of research that is selected. As such, the study is context-free and can result in high validity and reliability. On the other hand, the design in qualitative research is dynamic because relevant variables are identified during the research process based on the context of the study. As such, these variables may be viewed as tentative whose verification for validity and reliability may require future investigation.

Finally, a difference between quantitative and qualitative research is the logic used in the research process to draw conclusions. Generally speaking, deductive logic is used in quantitative research to draw conclusions in a cause and effect relationship that is predicted by the theory used in the study. On the other hand, inductive logic is generally used to try to develop generalizations from the evolution of variables during the study.3

Criteria for Selecting Qualitative and Quantitative Research

The differences between qualitative and quantitative research indicate that the two methods are used for fundamentally different research objectives. While qualitative research is certainly useful for exploratory research questions, concerns about its validity and reliability

3 For definitions of deductive and inductive logic, see Abdolmohammadi and McQuade (2002), Chapter 2.

3

have resulted in it being less popular than the quantitative method for more rigorous investigations. For example, in a study of research published in a journal over 1978-1998, it was found that only 26 percent of the papers could be classified as qualitative.4 A reason for this disparity may be that reviewers apply the same quality criteria to assess the rigor of qualitative research as they do for quantitative research, which may be inappropriate. Proponents of qualitative research contend that there is accepted research protocol for qualitative research that if followed can result in rigorous research papers. These proponents of qualitative research even contend that qualitative research can be just as theory driven as quantitative research.5

From a practical perspective one can actually use both paradigms in the same research in a mixed-method design. For example, using a focus group as a pilot study, a researcher can develop a general understanding of an issue to establish a set of variables and expectations that can then be investigated through follow up quantitative research. In cases where the question is which research paradigm should be selected, one can use a set of broad criteria to choose between the research methods. Exhibit 2 presents several criteria for this selection.6

[Insert Exhibit 2 Here]

The nature of the problem being investigated has a significant effect on the type of research to be selected. For example, if the problem is unstructured for which no holistic theory exists, then investigation of the problem may require exploratory research through the qualitative method. This method is selected simply to develop an understanding of the nature of the problem. On the other hand, if the problem is simple and reasonably well-structured, then much knowledge already exists about the problem that is normally reflected in a theoretical framework and an existing body of literature. As stated earlier, the existence of the body of literature helps identify variables and expectations that can be investigated through quantitative research.

It is desirable to stay with scientific criteria for selecting a research method. However, the researcher’s personal attributes and biases also affect the method of research. For example, if a researcher has a high level of tolerance for ambiguity and is open to undertaking lengthy investigation of a research issue, then perhaps he or she is open to conducting qualitative research. However, if the researcher has a low level of tolerance for ambiguity and feels more comfortable with rigid rules of quantitative research, he or she may be more likely to select the quantitative research for investigation.

Of course, in addition to personal attributes, the researcher’s training and experience with various methods can influence the selection of the research method. For example, if the researcher is well-trained in qualitative research and analysis, has strong writing skills, and is comfortable with qualitative analysis software, then he or she is more likely to select the qualitative method. On the other hand, quantitative research may be more likely to be selected by a researcher who is well-trained in quantitative research and statistical analysis, with technical writing skills, and comfort with quantitative analysis software.

4 See Pardun (2000).5 See Pardun (2000, 532).6 Much of the discussion in this section is adapted from Creswell (1994: 8-12).

4

Finally, the expectations of the audience make a difference in the choice of the research method. The readership of the publication outlets tends to have different interests and expectations. For example, while the readership of the academic publication, Journal of Accounting Research generally expects research papers in the quantitative realm, the professionals who read the Journal of Accountancy prefer qualitative writing, or very descriptive quantitative pieces.

In conclusion, the choice of a research method depends on a number of factors ranging from the personal attributes, training and experience of the researcher to the nature of the problem being investigated and the audience expectations. As discussed in the next section, the method used to communicate the results also depends on audience considerations although a generic format can be developed for such communication regardless of the audience.

Communication of Qualitative and Quantitative Research

The report format used to communicate research results can be different depending on the audience. In general, a research report may contain the following sections:7

Quick reference that includes transmittal letter, if any, title page, table of contents and an executive summary (also called the abstract or synopsis)

Details section that includes an introduction, which specifies the research objectives, a literature review that includes the identification of a theory and the related research hypotheses, the research method used, the results from the study, and concluding statements

Supplements section that provides any illustrations, tables, appendices, list of references, and end-or footnotes.

This report format is generic in nature and thus can be used by both quantitative and qualitative research with some modifications. There can be variations in the sense that some sections may be more applicable to one method than the other. For example, a published paper in an academic journal does not have a transmittal letter although one may have been used to submit the paper to the journal editor for publication. Qualitative reports are more likely to include a transmittal letter especially if the report is targeted for a specific group of readers such as corporate mangers. An author suggests the following sections to communicate the details of a qualitative research report:8

Introduction and general question or topicSignificance of the researchSite and sample selectionsResearcher’s role in management, including entry, reciprocity, and ethicsResearch strategiesData collection techniquesManaging and recording dataData analysis strategies

7 Adapted from Abdolmohammadi and McQuade (2002: 133).8 See for example, Marshal and Rossman (1999)

5

Management plan, timeline, feasibilityAppendices

This list does not have any information that I have included in the Quick reference section as the list focuses on the Details of the report. However, it mentions appendices, which I present as part of the Supplements. Also included in this list is Significance of research, which includes references to the current literature (i.e., literature review).

For papers published in journals, one can assume that the writing quality has met the standards of the journal. In fact, the success of a research report in being published in a prestigious journal not only indicates acceptable writing quality, but also a measure of the quality of the underlying research. This is because before being published, a research paper is normally subjected to a review process. The review process may be limited to an assessment only by the journal editor in some professional journals. Other journals require a system of blind peer review by which at least two scholars who are known to be experts in the field assess the strength and weaknesses of the paper and provide guidance for revisions. This process may render a decision that the paper is not in a shape or quality that is acceptable for publication, or that it requires significant revisions before it can be further assessed for publication.

Does the review process mean that a paper published in a journal is flawless? The answer

is no. There are always limitations to a research paper. Thus, even a paper published in a reputable journal may have significant weaknesses along with strength that had resulted in its publication in the first place. However, the limitations of the research should also be acknowledged in the conclusions of the paper. How does one assess these strength and weaknesses? This is the subject of the next section.

CRITIQUE OF PUBLISHED RESEARCH

A research paper goes through a number of stages and several drafts before a final version is developed and published in a journal. Generally, an author develops a sequence of drafts before he/she is satisfied that a reasonable draft has been developed. To do this, the authors engage in a serious critique of their own work. Upon the development of a reasonable draft, the authors distribute the paper to their colleagues and people familiar with the research topic. If there is more than one author, then the authors collaborate and critique each other’s work to develop a reasonable draft for distribution. Upon receipt of feedback from colleagues, the authors revise the paper and get the paper ready for submission for presentation at scholarly forums. Based on one or more presentations and the comments received from the audience at these meetings, the authors further revise the paper and then submit it to a journal for publication. At this stage if the paper is considered promising, it goes through one or more rounds of reviews and revisions before it is accepted for publication.

Once published, a research paper is out in the public domain and is subject to evaluation by knowledgeable readers and professional evaluators, “and at this stage a scientist’s reputation can be firmly established or irrevocably damaged”9 based on the quality of the publication. Depending on the journal and its emphasis there may be significant differences in the quality of

9 Kuyper (1991): 248.

6

the research. In general, the quality of a published paper depends on its adherence to the characteristics of good research. In this section I provide a summary framework of these characteristics.

A Framework for Critiquing Research Papers

To perform a comprehensive critique of a research paper, there is a need for an organizing framework. A typical critique framework identifies various components of the critique and the issues that each component should address. While these components are presented separately, they are very much interrelated. The framework in Exhibit 3 has five components beginning with the determination of the research type, identification and evaluation of the details of the research design, an assessment of the quality of writing, a commentary on the organization of the paper, and a summary evaluation of the strength and weaknesses of the paper.

[Insert Exhibit 3 About Here]

The framework in Exhibit 3 is generic in nature and can be used for evaluation of pre-publication drafts of research papers as well as published papers. The main difference is that the critique of a pre-publication paper is usually more detailed and is accompanied by constructive suggestions to improve the quality of the research and its presentation before it is published. On the other hand, the critique of a published paper is intended as an evaluation of its merits and whether its findings can be relied upon. Needless to say, before one can critique a research paper, one must read it carefully, perhaps a few times to develop a deep understanding of the research.

Determination of the Type of Research

The first component of the framework in Exhibit 3 is the determination of the research type represented in the paper. This is important because of the implications it has for assessing the remaining components of the framework. For example, while quantitative research is expected to score high in rigor and freedom from bias, qualitative research will be judged knowing that it does not score high on these attributes. Similarly, qualitative research scores low in variable isolation because as stated earlier, this type of research is used in situations where either there is no theory to specify variables and predict outcome, or the theories are viewed as tentative in nature. However, like quantitative research the qualitative research must score high in inferential logic. In research cases where generalizations are made from sample information (e.g., focus groups) to some population, statistical analysis is applicable. The statistical methods may be different from those used in quantitative research. Nevertheless, they should be selected carefully. While powerful statistical packages (e.g., SPSS, SAS, Minitab) have existed for many years for use in quantitative analysis, recent statistical analysis software (e.g., NVivo) have become available for rigorous analysis in qualitative research.

In some research such as case studies where only descriptive information is used, inductive logic is usually used to draw appropriate conclusions. Similarly, analytical modeling papers are based on mathematical formulation and should be judged on the propriety of their inductive logic.

7

Within the qualitative and quantitative realm one can identify other characteristics of research being critiqued. For example, using Exhibit 2-3 in Abdolmohammadi and McQuade (2002), you can identify whether the research is primarily basic or pure, professional, applied, or instructional. Each of these classifications signals a different objective. For example, the main objective of a basic research is to generate new knowledge. Applied research is used to find solutions to contemporary problems, while instructional research is focused on finding the best ways for disseminating current knowledge.

Critique of the Details of the Research Design

This component of the critique framework is the most crucial and detailed. You need to have a clear path to develop your critique. For example, you can use the research process depicted in Abdolmohammadi and McQuade’s (2002) Exhibit 5-1 and the research design types presented in their Exhibit 6-1 as a path to comment on various issues such as those in the following paragraphs.

Identify the specific theory, if any, used in the study. Focus your attention on the literature review section of the paper and assess the adequacy of this review. A reasonable review is one that identifies the key studies from the literature. The paper summarizes the results of the papers in the literature as a means of identifying what is already known about the topic, and what remains to be discovered, or what is controversial or has had mixed results. The critique then identifies and assesses the propriety of the theory, if any that is used in the study to investigate the unknown. While quantitative research is likely to have used a specific theory, there may be no theory in qualitative research. You must identify the theory and comment on both its appropriateness and its proper use for the research question. In rare cases where the paper presents a new theoretical framework or analytical model, you make a comment on whether there is no current theory or clear literature to review.

Identify research hypotheses. This issue is very much tied to the presence or absence of a theory, and as such it is mostly, but not exclusively, a characteristic of quantitative research. Be sure to comment on whether the stated hypotheses are directly identifiable by the theory and whether they are appropriate for the issues being studied. Related to this characteristic is whether variables are identified as dependent and independent and whether they are carefully isolated in the research design. If not, identify unintended variables (also called confounding variables) that might have affected the research outcome (i.e., their effects on dependent variables).

The paper may also have intervening variables. These variables are different from confounding variables in the sense that they are intervening connectors between the dependent and independent variables. Specifically, variable A may be dependent on variable B, which in turn is dependent on variable C. Thus, one can conclude that A is dependent on C, with B being an intervening variable. The intervening variable B may even be unobservable.10 For example, intelligence is an unobservable intervening variable. Let’s assume that a researcher is interested in measuring performance of students on an exam as dependent variable A. The researcher may

10 If so, the intervening variable may also be called a latent variable or a hypothetical construct. See Kerlinger and Lee (2000: 54-55) for details.

8

hypothesize that the exam score is dependent on students’ intelligence as an unobservable intervening variable B, which in turn is dependent on the students’ verbal and mathematical aptitude (variable C) that can be measured via other tests such as the Graduate Record Examination (GRE). The complexities resulting from careful treatment of confounding and intervening variables make critique of research challenging.

Describe the research method used in the study. The study may have used empirical data from various databases, experimental data from laboratory investigations, surveys conducted via the Internet or some other method to collect the data needed for investigation.11 This assessment is relevant to both quantitative and qualitative research. Be sure to comment on the propriety of the research method used to collect data for the purpose of the study.

Assess the analysis used to identify research results. The method used includes analysis, synthesis, and interpretation. For example, in a quantitative research statistical analysis may be used to identify the results. In a professional research the researcher may have only used a synthesis of the professional literature to report his or her findings. On the other hand, interpretation of discovery or descriptive data may be the method used for qualitative research.

Describe the logic used to draw conclusions. Specify whether the inference logic is deductive, which may be a characteristic of quantitative research or inductive reasoning that may be a characteristic of qualitative research. Alternatively, descriptive and normative logic may be used in the paper. For each of the logics used comment on whether the researcher has used the right logic to draw conclusions. Remember that the conclusions must be a direct interpretation of the evidence presented in the paper. One should not mix personal preference statements with those that are directly a consequence of the evidence. However, this does not mean that one should not propose related issues unresolved from the evidence in the paper for future research.

Assess Writing Quality

As stated earlier, there is high variation in the writing quality of research papers and writing quality means different things to different people. For example, a study engaged a sample of 50 business professionals and an equal number of business professors (i.e., accounting, finance, marketing, etc.) to evaluate the writing quality of papers written by marketing students in a Midwestern university.12 The authors observed very little similarity and many significant differences between faculty and professionals in evaluating the papers. In general, both groups emphasized the importance of organization, but their agreement ended there. While professionals emphasized wording, content, and sentence structure more than faculty, faculty emphasized the importance of conciseness, spelling/grammar (i.e., Standard English), appearance, and purpose more than professionals. Also, professionals spend less time than faculty to evaluate papers, focused more on content than style (i.e., formal, informal), and were more lenient in assigning grades than faculty. These differences have implications for the authors. For example, if one is writing a paper for The Accounting Review, one must be cognizant of the way academics evaluate papers because the reviewers are likely to be academicians. On the other hand, if you are writing a paper to submit to the Journal of Accountancy, you should be aware of the interests

11 See Abdolmohammadi and McQuade (2002; Ch. 6) for descriptions of these research methods.12 See Seshardi and Theye (2000).

9

of professional accountants who will likely be the reviewers for the paper.

Indeed published papers go through a review process that results in the writing style preferred by the target journal. Some journals provide specific guidance on how to improve the writing of a research paper. These journals may also have professional editors on their staff to make sure that the paper meets the quality standards of the journal. The Accounting Review that is the flagship publication of the American Accounting Association is one such journal. The Journal of Accountancy published monthly by the American Institute of Certified Public Accountants is another example. Other journals leave the responsibility of the writing completely to the author and do not provide much guidance beyond that suggested by the reviewers. One of the premiere journals in the field of business ethics, Journal of Business Ethics is an example of this group. Other journals fall somewhere in between. For example, once a paper is identified as publishable, the editor of Advances in Accounting sends a Writing Guideline document with detailed directions for final manuscript preparation. The editor may also make additional editorial change suggestions.

Regardless of the level of editorial assistance, a research report is the sole responsibility of the authors and therefore the authors receive the credit for quality of writing or the blame for poor writing. As such one can identify different writing quality and organization even within the same journal issue. Thus, an assessment of the writing quality of a research paper is important. As stated earlier, this assessment can be done for unpublished research reports generated via internal research or through reviewing a research paper for publication. However, you will be surprised at the variation that you will observe even in papers that are published in reputable journals.

To assess the quality of writing of a research paper, comment on the following criteria:13

Coherent Organization. Comment on whether the ideas are arranged logically and the flow of thought is easy to follow and understand. Do the authors use short paragraphs consisting of short sentences or use unnecessary long sentences and paragraphs? Also comment on whether the authors use each paragraph to develop a principal idea or do they mix different ideas in a single paragraph? Furthermore, comment on whether the principal idea is placed in the first sentence, followed by an explanation of the idea in the remaining sentences in the paragraph, or whether the paragraph is constructed in a haphazard way.

Conciseness. The main issue in a paper is to communicate the research in as few words as possible. Comment on whether the authors are concise in communication of ideas in the paper or do they use excess verbiage in communicating. Suggest ways in which the authors could make their writing more concise without loss of content.

Clarity. Use of clear and concise words with precise meaning for sentence construction is preferred over vague terms and long words. Comment on whether or not the authors use clear terms and sentences to communicate their ideas.

Use of Standard English. This writing quality refers to proper punctuation, capitalization,

13 This list is from the AICPA (1990).

10

and spelling. Provide a commentary on the use of Standard English by the paper and provide suggestions to improve it.

Appropriateness for the Target Audience. Finally, comment on whether the writing is appropriate for the target audience. In general, professional and qualitative research reports use informal and the first or second person style for communicating. They also use bullets and summary statements. On the other hand quantitative and academic readers generally expect formal and technical writing and the use of the third party language.

Research Report Format

As stated earlier, the research format may differ between various research types and audiences. You can use the format suggested in Chapter 6 of Abdolmohammadi and McQuade (2002) to comment on the appropriateness of the format used by the paper you are critiquing. The following are the various sections of a research paper, but not every report must have exactly the same organization. For example, an internal report generated for a manager should have a transmittal letter, but a journal article does not have this.

Transmittal Letter. Comment on whether the paper needs a transmittal letter, and if so to whom should it be addressed? What should be the content of such a transmittal letter?

Title Page. A paper published in a journal has a title. Sometimes, there may be multiple titles representing separate, but related issues that some journals place in sidebars. Your critique should include an assessment as to whether there is sufficient information in the title to provide an accurate lead into the issue of investigation. Make sure that the title does not say one thing while the content says a different thing. If so, provide suggestions for improving the wording of the title so that it precisely represents the content. Also, is there sufficient information about the researchers and their credentials so that additional communication can result between interested readers and the researchers?

Table of Contents. Does the paper have a table of contents? If the answer is no, state the reason. Generally, while you will find tables of content in research monographs, books and detailed reports, journal articles eliminate such a list to save space.

Executive Summary. While targeted research reports have an executive summary, many journals publish an abstract or a synopsis of the research. The issue you want to focus on is whether such an abstract or synopsis provides a concise statement about the objectives of the study, the method used to investigate them, and the findings that ensued in a way to get the reader’s attention. This abstract is the gateway into the detailed report on the research and has a significant impact on whether the reader will read the rest of the paper. Of course, the reader may decide not to read the rest of the paper because the topic is not of interest or is not in one’s area of expertise. However, for those who should be interested, the abstract should be of such in formativeness to motivate reading of the whole article. For this reason the abstract should make it clear that the research covers an important issue of concern to professionals or academicians and state the significance of the problem and make a case for the need to investigate the problem.

11

The introduction section should make the following points clear. The objectives of the research listed in order of importance, the overall plan for the research, and the specific achievable goals for the research. This section is where the motivation for the research is made clear. Comment on whether the authors succeed in convincing you, the reader that the issue that they are investigating is an important issue of concern.

The Literature search section provides a review of the most recent research studies as well as old ones that are viewed as influential on the topic in a concise and focused manner. This section must draw a synthesis of the research to date to observe generalizeable results, inconsistencies, mixed or controversial findings. The paper then identifies the theory used in the current paper to address the gaps in the literature and moves on to state research hypotheses that are addressing the specific factors identified for investigation. Your critique must include statements about whether the author has provided a sufficient review of the pertinent literature. The paper does not need to identify every single research published in the area, but it should cover a reasonable amount of literature, particularly those viewed as key articles in the filed. You then make an assessment of the theory used in the paper. Is it the right theory for the study? Is it described well enough to communicate its relevance to the study? Do the hypotheses follow the theory’s predictions? Are the listed hypotheses sufficient, or should the authors include other hypotheses? Conversely, should the authors cut the number of hypotheses in the paper in favor of a more focused set of hypotheses? A loose guideline for the number of hypotheses is four. Less important hypotheses could be listed either as corollaries, or just explained in the paper as additional analysis.

Research method section describes the details of the research design in technical terms with a justification for the appropriateness of the design. Comment on the propriety of the sample selection process, the sample size, and the data collection techniques (e.g., survey instrument, data bases) used in the study. Is the method appropriate for testing the research hypotheses? What other methods would you have used to better test the hypotheses?

The findings (or results) section should have sufficient information about the method used to analyze the data, including statistical or other techniques. As well, the resources used (e.g., a statistical software package) should be identified. Your assessment should focus on whether the analysis used in the paper is directly related to the objectives and hypotheses, if any of the paper. This assessment is perhaps the most difficult for many reviewers because many have little confidence in their abilities to assess the strength and weaknesses of various statistical techniques. It requires both a reasonable level of training, experience as well as confidence in one’s ability to assess statistical analysis.

The Conclusions section should present arguments and conclusions that are based on an appropriate inferential logic (deductive, inductive, descriptive, and normative). Such conclusions must be directly the consequence of the analysis. They should not be speculative in nature and judgmental on the part of the authors. Keep in mind that by the time an author gets to the conclusions section, he/she may have become tired of the whole thing and anxious to finish up the paper. This can be evident if the conclusions are only repeats of the results, not interpretations of the results. A good conclusion section presents authors’ insights and interpretations of the evidence on what we have learned from the results not just a rehash of the results. Are these

12

interpretations counter intuitive, do they run against conventional wisdom? If so, do the authors have strong basis for such interpretations. Also important is that the authors present concluding statements about what they contribute to the literature. In particular, the authors should refer to key studies from the literature that relate to the current research and show precisely how their investigation contributes to the literature.

Also important is that the authors identify and acknowledge the limitations of their research and use these limitations as a means of suggesting future research. Thus, your critique should include statements on whether the major limitations of the study are identified? What other limitations are worthy of mention in the paper?

The Supplements section should include various illustrations such as tables or exhibits. Journals place exhibits and tables within the main body of the paper. However, research reports before they are published place these as supplements at the end of the paper with proper reference in the paper on where these tables or exhibits should be inserted. Appendices, if any are also placed at the end to communicate information of secondary importance. This is followed by a list of references that complies with a standard bibliographical listing such as the American Psychological Association, and the new and evolving internet referencing methods. Finally, if there are endnotes to present additional information of secondary importance, they are placed at the end of the paper. Some journals prefer footnotes that are placed at the bottom of the page where there is a reference to the footnote. Your critique should state whether the paper has appropriately used the Supplement section to present information of secondary importance. Be sure to identify weaknesses and present ideas to improve them.

Summary Statement

The final component of the critique framework is a summary statement about the validity and reliability of the research taken as a whole. The summary statement uses the detailed assessment of the other components of the critique framework to address issues of construct validity, freedom from bias, variable isolation and inferential logic. While I present this summary as the final component of the critique framework, some reviewers might actually prefer to present this as the first component followed by details to back up their conclusions.14

Validity. The validity of a research depends on whether it has:

A purpose where a meaningful reason exists for conducing the research Rigor, which refers to a sound theoretical foundation and research design Testability indicating that the research question can be tested Replicability, which indicates that the research can be performed

numerous times for corroborative evidence Precision that refers to the closeness of the research to reality Confidence that is the probability that the research results are close to

reality Parsimony, which refers to the simplicity with which the research process

14 This section is adapted from Abdolmohammadi and McQuade (2002: 30-33).

13

and its results are communicated

Thus, your critique should address these validity issues and draw an overall conclusion about how the paper ranks on the validity criterion.

Freedom from bias. Freedom from bias refers to the researcher’s impartiality in conducing research. This characteristic is needed to enhance the reliability of the research conclusions. So, your critique should comment on whether the authors have inappropriately mixed personal statements with factual data. Of course the scale of the data used in the paper can introduce subjective interpretation. The ideal and most objective data scale is the ratio scale, where the scale has meaningful ratio denomination. For example, two million dollars of profit is exactly one half of four million dollar profit. The next level is the interval scale that assumes equidistance, but not necessarily the same ratio. For example, measuring temperature in Fahrenheit renders numerical representation of various levels of temperature, each of which corresponding to equidistance in numerical terms, but one temperature (say 50 degrees) does not necessarily mean that it is exactly x-times (say twice) as warm as another temperature (say 25 degrees).

The next scale is ordinal, where even equidistance can not be assumed. For example, a rating of 1, 2, 3 may be used to indicate that 1 is better than 2 and 2 is better than 3, but no equidistance or ratio relationship between the numbers. Finally, a nominal or categorical scale represents only categories. It may not mean anything beyond the category that the number represents. For example, you can not draw any conclusion about any relationship between one 10-digit telephone number and another except that the first three digits (area code) indicates that the phone represents a geographical area, and the next three digits may mean a subset of the geographical area. The remaining four digits may be completely randomized. Thus you can use phone numbers to develop categories (i.e., geographical areas) for the purpose of using other data for analysis. For example, a research study may analyze profitability by geographical areas represented by area codes of a customer base. Thus, to represent categorical data, a researcher may use numbers that don’t mean anything but categories. In binary categorical situations, 0 and 1 may be used for categorical representation (also called dummy variable).

A conclusion from the discussion of the data scale types is that there is an increasing level of subjectivity as one moves from the ratio scale to the nominal scale even if sophisticated statistical methods are used. For example, very little subjectivity may be present in the analysis of a continuous and ratio scale variable such as profitability (say a comparison of profitability over time within companies, or between firms at a given period of time). However, if a scale of 1,2,3 is used to code the data from a survey of customers on the relative importance of one brand vs. another, one can observe that there is subjectivity in the analysis. For example, while a sophisticated statistical analysis may render more importance for brand A vs. B, one can not be sure about the degree with which the analysis really reflects customer preferences.

Variable isolation. Variable isolation is needed for internal validity. It means that the research succeeds in focusing on relevant variables and to control for irrelvant ones. This control is needed so that irrelevant variables are not allowed to contaminate the research process, thus the stated relationship between independent and dependent variables holds as valid. This

14

criterion requires that your critique comments on the independent varuiables investigated and identify those variables that also may have affected the dependent variable.

Appropriate inferential logic. This criterion, also called external validity refers to the use of appropriate statistical methods and inferential logic so that the research results can be generalized from a sample to a general population or to similar situations (i.e., external validity). Thus, overall your assessment addresses the issue of how well the study has adhered to this criterion?

The method you use to assess these criteria and develop a summary statement is completely up to you. You can use narratives and broad statements or use a quantitative scale such as 1-5 (low to high) to indicate the closeness of the research being critiqued to the quality criteria.

AN ILLUSTRATION

In this section I use the critique framework to assess the quality of a research paper in a refereed journal. The paper is reproduced in Appendix A.15 This paper investigates earnings management in a situation where certain securities are selectively sold in order to realize gains or losses to manipulate reported earnings by companies. The study focuses on the classification of debt and equity securities in Statement of Financial Accounting Standards No. 115, “Accounting for Certain Investments in Debt and Equity Securities.”16 Using data from the insurance industry, the study finds that the guidance in SFAS No. 115 security classification results in significant earnings management in this industry. Specifically, SFAS No. 115 classifies securities into three classes of Held-to-maturity (HTM), Trading, and Available-for-sale (AFS). HTMs are debt securities for which management has the intent to hold the security to maturity. These securities are reported at amortized cost where unrealized holding gains and losses are not recognized until the securities are actually sold. Trading securities are both debt and equity securities that are purchased with the intention of holding for a short term to sell and make short-term profit (or loss). Thus, once sold, the company must realize gains and losses from these short term securities.

AFSs are both debt and equity securities that are neither HTM nor Trading. They are reported at fair value, but the unrealized gains or losses are reported only as a separate component of the owners’ equity per SFAS No. 115 and not as an income item.17 Thus, AFS securities can be short-term or long term. This feature of AFS provides flexibility for management in terms of its leeway to hold the security for a short time or a long time. The research study that you are critiquing investigates whether this flexibility can be used by management for gains trading in which only those securities that have unrealized gains are sold to realize income while keeping the securities with unrealized losses in the portfolio.

15 This paper is Jordan, et al. (1997/1998). I have tentatively reproduced it in Appendix A. Depending upon the target outlet for publication of my paper, I will either include the whole article with the permission of the copyright holder, or present only a reference to it for the reader to seek it out. 16 AICPA (1993)17 In a later ruling, the FASB issued SFAS No. 130, “Reporting Comprehensive Income” (FASB, 1997) in which the section on comprehensive income contains these gains and losses.

15

Type of Research

This research is presented in a quasi-quantitative mode because it has the elements of both qualitative and quantitative research. Because there is no strong theoretical foundation for the research to predict causal relationship between independent and dependent variables, one can assume that the research really addresses an exploratory issue of concern. However, while not in a theory-based hypothesis testing mode, the paper nevertheless sets a set of expectations on the basis of a literature review that are test via quantitative methods.

The research also can be assessed to have elements of pure research (creates new knowledge) and professional (deals with an issue of concern to professional accounting community).

Details of the Research Design

Identify the specific theory, if any, used in the study. This paper briefly reviews the literature both on earnings management as well as SFAS No. 115. However, the paper neither develops a formal theory nor does it adopt one from the literature as a means of setting its research direction. The literature reviewed could be expanded to identify various independent variables that have been found to affect earnings management and then carefully identify those that need further investigation in the current study. Also the paper could specifically identify various theories that have been presented in the literature that relate to gains trading.

Identify research hypotheses. There are no theory-based hypotheses in the paper. Earnings management theories such as those related to income smoothing could have been utilized as a basis for setting up research hypotheses. For example, one can identify income smoothing as a theory that predicts that management attempts to show reported income in a monotonically increasing smooth curve. Two hypotheses can be stated based on this theory: (1) gains trading is used by companies in need of a source of additional income to show a long term smooth curve; and (2) companies that do not need gains trading as a source of income for income smoothing do not engage in such trading.

The paper specifically investigates the effects of three independent variables (ROA, LEV and ASSETS) on two dependent variables (GAINSONE and GAINSTWO). There can be at least two concerns about internal validity about the independent variables. The first being that these variables may not be isolated. The ROA is defined in the paper as the ratio of income before taxes and before gains and losses from sales of AFS securities to total assets. We know that ROA can be influnced by a large number of factors both in the numerator (earnings) and the denominator (assets). Thus differences between companies may be due to a number of facors not controlled by the researchers. A similar concern can be raised about LEV, which is defined by the authors as the ratio of year-end debt to the year-end assets. This debt ratio is subject to chnges due to various transactions on both the debt and the assets of the company and thus one can expect variations within and between companies that may not be solely a consequence of AFS trading. For example, a company rich in cash may decide to retire a significant amount of its debt, making the debt ratio lower than a rival that does not have excess cash and yet both

16

companies may still have an incentive (say income smoothing) to engage in gains trading. Finally, the Log of total assets used to define the third independent variable, ASSETS while approporate as a transformation mechanism to reduce the size of the variable. However no theoretical basis or convincing arguement is affered on why the size of a company represented in its assets should have a significant effect on its gains trading? Even if this variable is plausible as an explanatory variable for the changes in the dependent variables, it can nevertheless be subject to many factors other than gains trading.

The second issue of concern is the presence of assets in the formulation of all three independent variables. This fact makes the independent variables not independent of each other. This issue causes a problem that sttisticians call multicolinearity. Multicollinearity refers to a situation where the independent variables are systematically correlated either positively or negatively. For example, if the correlation between ROA and LEV is positive and significant (say with a corrlation coefficient of or higher 0.90), then one of the variables should be dropped from the study because their effect on the dependent variables will be the same. Since the paper does not provide information on the correlation between independent variables, one can not assess the magnitude of this problem. Nevertheless, there is a serious concern about this possibility.

The authors do not formally state any hypotheses. However, in describing the independent variables, they state their expectations in an ad-hoc manner. For example, they state that “Entities with lower ROAs would be more inclined to engage in gains trading to increase their earnings level.”18 Similarly, the authors expect that firms with higher degrees of leverage (LEV) and total assets are more likely to engage in gains trading than those with lower levels of LEV and ASSETS. They do not offer any theory or convincing arguments to back these expectations.

Describe the research method used in the study. The study used empirical data from a database called Compact Disclosure. The authors used the financial information for 1994 and 1995 of 108 insurance companies that disclosed information on investments accounted for under SFAS No. 115. They used selected data for an investigation of gains trading in 1995. This method is certainly appropriate as a source of data for the study. A longitudinal investigation with multiple year data would be preferred but the study was conducted shortly after SFAS No. 115 was promulgated so longitudinal data were not yet available.

Describe the method used to identify research results. The statistical method used in this study is simple and multiple ordinary least square (OLS) linear regressions.19 In a series of six simple linear regressions (see Table 1 of the paper), the authors investigate the effects of each of the three independent variables singularly on each of the two dependent variables. They find that while ROA and LEV have significant effects on GAINSONE and GAINSTWO, ASSETS does not have any significant effect. The authors then use two multiple linear regression models in which each of the two dependent variables are simultaneously regressed against the three independent variables. Although both models are significant at high statistical levels (0.02 for the 18 Jordan, et al. (1997/1998: 52)19 These regression models are quite robust in terms of their low sensitivity to the violations of certain assumptions. For example, low levels of correlation between independent variables should not cause too much concern about multicollinearity.

17

first model and 0.002 for the second model), the only independent variable that is individually significant in affecting the dependent variables is the ROA.

Describe the logic used to draw conclusions. The authors use statistical inferences from the simple and multiple linear regressions to draw a conclusion that ROA is negatively associated with gains trading under SFAS No. 115. They also advocate a conclusion in which the simple regressions indicate that LEV has an effect on gains trading, but this relationship is not indicated in the multiple regression models. The size of the company (i.e., ASSETS) appears to have no significant impact on gains trading.

The statistical inferences used in the study may not be appropriate for the purpose of the study. I focus on two specific issues. The first assumes that the use of an inferential logic is justified in this research and the second one is based on an argument that there is no need for inferential statistics in this research. First, let’s assume that the independent variables are well-isolated and there are causal relationships between them and the dependent variables. Also, let’s assume that there is a need for inference from this sample to a larger population. Then the appropriate inferential statistics would be one that would investigate the main effects of each of the independent variables on the dependent variables as well as the interactions between the independent variables. The statistical model that comes to mind is multivariate analysis of variance or MANOVA, not the simple or multiple linear regressions that are used in this study. The regression models specified have no place for interaction effects that may be present in independent variables due to assets being a part of the definition of each variable. 20

An alternative way of looking at the data set used in this study is that the data set represents the whole population of the insurance companies reporting SFAS No. 115 related information in 1994 and 1995. Why is there a need for inferential statistics? What population is there to infer from this sample? If one accepts this argument, one can reject inferential statistics altogether in favor of descriptive statistics to simply describe the characteristics of the population. Using descriptive and exploratory statistical analysis makes the study more like a qualitative research than its current quantitative presentation.

Writing Quality

Coherent Organization. The ideas in this paper are arranged logically and the flow of thought is fairly easy to follow. However, it would be better to organize the source of the data and the specification of the variables under a caption such as Research Method. Currently the information about these issues is provided in the section titled SFAS No. 15 Research. The authors generally use short paragraphs consisting of short sentences and each paragraph is used to develop a principal idea in which the principal idea is placed in the first sentence, followed by an explanation of the idea in the remaining sentences.

Conciseness. This paper uses as few words as possible to communicate the research 20 The two statistical methods are very much related and they address basically the same issues. In addition to the effects of each of the independent variables on the dependent variables (called main effects), MANOVA also specifies he interactions between independent variables and their effects on the dependent variables. This is a method of addressing the issue of multicollinearity in the data and its effect on the dependent variables that is missing from a simple or multiple regression model.

18

process and its results. There is no evidence of excess verbiage in communicating. Indeed, there may be a need for an expansion of some sections to better communicate content. For example, the final section of the paper can be expanded to better communicate the limitations of the study.

Clarity. The authors use clear terms and sentences to communicate their ideas. However, the coverage of statistical analysis leaves much to the readers’ interpretation. In general, there is a need for justification of the statistical method used and this justification is missing. As stated earlier, other statistical techniques (e.g., MANOVA or descriptive statistics) may be more appropriate for this study than the regression models that have been used for analysis of data.

Use of Standard English. In general this paper has proper punctuation, capitalization, and spelling. However, there are still some typographical errors. For example, a “to” is missing between little and do in the following sentence, “Actual market efficiency or inefficiency, however, may have little do with whether earnings management occurs.”21 Similarly, “relationships” is spelled “rela-tionships” on the first paragraph on page 54 and there is an unnecessary “in” in the “at in least in part” in the first paragraph of Column 2 on page 54. While these typographical errors are minor in nature and inconsequential to the message of the paper, they nevertheless cast doubt in the mind of the reader about the care with which the paper has been prepared.

Appropriateness for the Target Audience. The journal in which this paper is published is Journal of Applied Business Research. The target audience for such a journal is both academicians and business people. The writing style used is primarily formal and quantitative. However, as stated earlier, the paper could easily be presented in a descriptive and informal style.

Research Report Format

This paper does not have a Transmittal Letter, a separate a Title Page or a Table of Contents. These are not needed for a paper published in a journal. However, sufficient information about the researchers and their credentials is desirable so that additional communication can result between interested readers and the researchers. Only the names of the researchers are provided in the paper. The paper does have an Abstract. The abstract provides a concise statement about the objective of the study, the method used to investigate it, and findings that ensued. However, it is a bit deficient in communicating the importance of and the need for the study to the academic and/or professional community.

The introduction section of the paper appropriately discusses the provisions of the SFAS No. 115 and some references from the literature to identify its objectives of the research. However, additional brief statements to cover the overall plan of the research, and the specific achievable goals are missing from this section. The Literature Review section provides a review of the most recent research studies that the authors viewed as influential on the topic in a concise and focused manner. This section also draws a conclusion about the numerous explanatory variables that might influence earnings management and provides selective references for some. However, the section does not carefully identify all of the variables from the literature, nor does it

21 See Jordan et al (1997/1998): 50.

19

identify those that are in need for further investigation. So the reader is left wondering why the variables that this research investigates are important. As a result of this, the paper also fails to identify or develop a theory to predict hypotheses.

As stated earlier, this paper does not have a section called Research Method and it should. It provides the literature on SFAS No. 115, specifies the research objectives, identifies the source of its data, the independent and dependent variables, and the statistical method all in a short section under SFAS No. 115 Research.

The paper has a section called Results in which information about the method used to analyze the data is provided along with the findings from this analysis. However, the paper does not justify the use of the OLS Regression analysis in this section, nor does it identify the statistical software package used to perform the analysis. The analysis is directly related to the objectives of the paper, but as stated earlier the statistical method used may not be appropriate for the analysis due to concerns for likely multicollinearity and interactions between the independent variables.

Finally, the paper presents a Summary and Conclusions section in which a summary of the study is provided, a limitation is identified and suggestions made for additional research. However, as stated earlier, the authors conclude that their study provides strong evidence in support of gains trading. Given the limitations of the study this conclusion may not be well founded. In addition, the authors do not provide much in term of their interpretations and insights about the meaning of the evidence in their paper. This is left to the reader to decide.

The paper does not have any Supplements. As is customary about published paper, tables are included in the body of the paper . Also, a standard bibliographical reference is appropriately provided at the end to list the research studies cited in the paper. Finally, the paper has a few endnotes that are presented right before the reference list.

Summary Statement

Validity. This paper has a purpose because there is a meaningful reason for

determining whether the provisions of SFAS No. 115 result in gains trading and earnings management.

The study is based on a review of the literature indicating that corporate management has incentives to engage in earnings management. However, the paper does not introduce a theory to predict any anticipated outcomes from this research. As stated the expected results appear to be ad hoc in nature.

The testability component of validity is present in this research because the research question may be tested using available and valid data.

Similarly, the research meets the replicability criterion for validity because the research can be performed numerous times in various industries and over different time periods. The focus of this paper is only on the insurance industry in year 1995.

20

The precision criterion is also met because the paper uses the whole population of insurance companies that had classifications of securities under SFAS No. 115. Thus, it precisely describes a part of the reality in this industry in 1995.

The use of inferential statistics in the paper provides information from which one can assess the confidence or the probability that the research results are close to reality. This confidence level provides a mechanism for an inference generalizing from this study to other populations. However, as argued earlier, inferential statistics may not be appropriate for this study.

The paper is generally good in communicating except for the description of the statistical method used and the meaning of the statistical results. Thus, it does not score high in the parsimony criterion.

Freedom from bias. Since the researchers factually report the results of their investigation, one can assess the paper to be free from bias. The data scales used in the study were of ratio type. All three independent variables (ROA, LEV and ASSET) are measured in a ratio scale. Thus, assuming that you can trust the underlying data from the companies in the sample, the scales used for data analysis are of the highest levels of objectivity for data analysis.

Variable isolation. The independent variables in this research are not well-isolated, thus causing concern for internal validity. Simply stated, confounding variables may also have affected the results.

Appropriate inferential logic. The authors use inferential logic through the use of simple and multiple linear regression models. However, there are serious concerns about the theoretical links between independent and dependent variables that make the results dubious. Also, I made a case earlier for the use of either MANOVA if indeed an inferential logic is called for or the use of simple descriptive statistics to describe the characteristics of the population of insurance companies using the provisions of SFAS No. 115 in 1995. Overall, there are serious concerns about the external validity of this study.

In conclusion, there are serious flaws in this study that cast doubt on its validity and reliability. Although the study concludes that it “provides strong evidence that, indeed, gains trading do occur under SFAS No. 115,”22 I would use caution in drawing such a strong conclusion. The authors acknowledge a limitation by stating that their model “did not contain all possible explanatory variables that prior research suggests may be related to earnings management.”23 There are however, many other limitations as identified in this critique that deserve attention in drawing any conclusion from this study.

QUESTIONS AND ASSIGNMENTS

Questions

22 Jordan et al. (1997/1998): 54.23 Jordan et al. (1997/1998): 54.

21

1. List and briefly discus the arguments for professional accountants’ need to acquire the skills for critiquing research papers.

2. Define qualitative research and its attributes.3. Define quantitative research and its attributes.4. Identify the major differences between quantitative and qualitative research.5. What factors influence the selection of quantitative or qualitative research?6. What are the criteria for selecting quantitative or qualitative research?7. Can a research study have both quantitative and qualitative research components?

Explain.8. List and briefly describe the components of a framework to critique a research paper. 9. What are the components of the critique of the details of the research method?10. What considerations do you use to assess the quality of the writing in a research paper?11. List the sections you would expect in the organization of a research paper. Comment on

the differences between organization and writing style of qualitative and quantitative research.

12. Once you have critiqued various components of a research paper, what considerations do you use to provide a summary assessment of the validity and reliability of the research paper?

13. What is rigor in a research study? Can qualitative research have just as much rigor as quantitative research?

14. Validity of a research depends on a number of components. Identify these components and their importance. To have validity, should a research score high on every single component?

15. What is internal validity? How do you identify and isolate independent variables that you are studying in a research?

16. What is external validity? How do you develop cause-effect relationship between dependent and independent variables?

17. How do you assess the propriety of statistical method used in a research paper?

Assignments

A large number of publications can result from a combination of various research fields such as accounting, taxation or mathematics, research types (e.g., qualitative, quantitative, basic), research methods such as survey or empirical, and various statistical analyses such as regression analysis or analysis of variance published in numerous journals. The choice of papers to critique depends on the focus of a particular course. For example, if the focus of the course is on data collection methods in the field of accounting/auditing/AIS/tax, one can select from one’s own readings, current periodicals or electronic sources such as ABI Inform Global or Infotrac research papers to critique with any combination of the following research methods:

1. survey data2. empirical/database data3. experimental/laboratory data4. focus group data5. interview data

22

6. archival data7. case study/field work data

ADDITIONAL READING

This chapter is based on the assumption that the reader has an adequate understanding of research design and methodology. For detailed coverage of these issues see the following books:

Kerlinger, F.N. and H.B. Lee. 2000. Foundations of Behavioral Research (New York, NY: Harcourt College Publishers).

Miller, R.C. and N.J. Salkind. 2002. Handbook of Research Design & Social Measurement 6th Ed. (Thousand Oaks, CA: Sage Publications).

For additional reading on research methods in accounting see:

Ryan, B.; R.W. Scapens and M. Theobold. 2002. Research Method & Methodology in Finance & Accounting. 2nd Ed. (London, UK: Thomson).

For additional reading on qualitative and quantitative and mixed research designs see:

Marshall, C. and G.B. Rossman. 1999. Designing Qualitative Research. 3rd Ed. (Thousand Oaks, CA: Sage Publications).

Tashakkori, A. and C. Teddlie. 1998. Mixed Methodology: Combining Qualitative and Quantitative Approaches (Thousand Oaks, CA: Sage Publications).

REFERENCES

Abdolmohammadi, M.J. and R. J. McQuade 2002. Applied Research in Financial Reporting: Text and Cases (New York, NY: McGraw-Hill/Irwin).

AICPA. 1990. Definition of Writing Skills in Report of the Testing of Writing Skills Subtask Force of the CPA Examination Change Implementation Task Force (New York, NY: American Institute of Certified Public Accountants, September).

FASB. 1993. Statement on Financial Accounting Standards No. 115, “Accounting for Certain Investments in Debt and Equity Securities” (Norwalk, CT: Financial Accounting Standards Board)

FASB. 1997. Statement on Financial Accounting Standards No. 130, “Reporting Comprehensive Income” (Norwalk, CT: Financial Accounting Standards Board)

Creswell, J. W. 1994. Research Design: Qualitative & Quantitative Approaches (Thousand Oaks, CA: Sage Publications).

23

Jordan, C.E.; S.J. Clark and W.R. Smith. 1997/1998. Earnings management Under SFAS No. 115: Evidence From the Insurance Industry. Journal of applied Business Research 14 (1), Winter: 49-56.

Kerlinger, F.N. and H.B. Lee. 2000. Foundations of Behavioral Research (New York, NY: Harcourt College Publishers).

Kuyper, B.J. 1991. Bringing up Scientists in the Art of Critiquing Research. BioScience 14 (4): 248-250.

Marshall, C. and G.B. Rossman. 1999. Designing Qualitative Research. 3rd Ed. (Thousand Oaks, CA: Sage Publications).

Pardun, C.J. 2000. Theory into Practice: An Analysis of Qualitative Research in the Journal of Broadcasting & Electronic Media, 1978-1998. Journal of Broadcasting & Electronic Media (Summer): 529-534.

Seshardi, S. and L. Theye. 2000. Professionals and Professors: Substance or Style? Business Communication Quarterly. 63 (3, September): 9-23.

24

Exhibit 1Characteristics of Qualitative and Quantitative Research

Characteristic Quantitative Research Qualitative ResearchNature of reality Objective and usually singular Subjective and usually multipleRelationship between researcher and that being researched

Researcher is independent of that being researched

Researcher could be biased toward that being researched

Role of values Value-free and unbiased Value-laden and biasedLanguage of research Formal, third party, quantitative Informal, first/second party,

qualitativeResearch process Static design with isolated

variables, context-free, high in validity and reliability

Dynamic design with variables identified during the research process, context-bound, verification needed for validity and reliability

Logic Deductive, cause and effect Inductive with simultaneous shaping of variables,

Adapted from Creswell (1994, p. 5)

25

Exhibit 2Criteria to Select Qualitative and Quantitative Research

Criteria Quantitative Design Qualitative DesignNature of the problem Reasonably structured, supported by

an existing body of literature with known variables and expectations

Unstructured, lack of theory, requires exploratory research to understand the nature of the problem

Researcher attributes Low tolerance for ambiguity, comfort with rigid rules of conducting quantitative research

Tolerance for ambiguity, tolerance and time for lengthy study

Researcher training and experience

Researcher is well-trained in quantitative research and statistical analysis, technical writing skills, comfort with quantitative analysis software

Researcher is well-trained in qualitative research and analysis, literary writing skills, comfort with qualitative analysis software

Audience expectation Readers are accustomed to reading quantitative research

Readers are accustomed to reading qualitative research

Adapted from Creswell (1994, p. 9)

26

Exhibit 3Research Critique Framework

Critique Component Critique Task1. Research Type Determine whether the research is qualitative, quantitative, basic,

professional, applied, instructional2. Research Details Identify and evaluate theory, hypotheses, research method, method of

analysis, and the logic used to draw conclusions3. Writing Quality Assess coherent organization, conciseness, clarity, Standard English,

and style (for the target audience)4. Report Format Comment on various sections such as transmittal letter, title page, table

of contents, executive summary or abstract or synopsis, introduction, literature review, research method, findings or results, conclusions, and supplements

5. Summary Statement Comment on validity (purpose, rigor, testability, replicability, precision, confidence, parsimony), freedom from bias, variable isolation (internal validity), and inferential logic

27

Appendix A:Sample Article

(Next page)

28

29

30

31

32

33

34

35

36