A Note on the Format of Ennis' Multiple-Choice Tests of Deductive Reasoning Competence

E.P. Brandon, 13th October 1992

Archived in ERIC Documentation Service, TM 019 276 

The central notion of deductive logic is that of a valid argument, an argument in which the conclusion really does follow from the premisses.  The standard informal explanation of validity is that it should be impossible for the premisses of the argument to be true and the conclusion false.

Investigation of people's actual competence in matters of deductive logic could use questions of the form "Does r follow from p and q?"  But given the standard informal explanation, one could avoid any uncertainties people might feel about what it is for one statement to follow from another, by framing the question in terms of the notions of truth and falsehood.  So, for instance, in his pioneer investigations of deductive logical reasoning competence, Robert Ennis (Ennis and Paulus, 1965) employed the following question structure:
        Suppose you know that p, q, ....
        Then would it be true that r?
In an adaptation of Ennis' work for use in Jamaica, the three possible answers offered (Yes; No; Maybe) are glossed as "Yes" means "It must be true, given what you are told"; "No" means "It can't be true, given what you are told"; and "Maybe" means "It may be true or it may be false.  You haven't been told enough to be certain whether it is Yes or No."

Given Ennis' question format, and the standard construal of validity, when the sentences constitute a deductively valid argument the correct answer is either "Yes" or "No"; when they do not make up a valid argument the correct answer is "Maybe."

In the course of the Jamaican investigations (reported in Nolan and Brandon, 1986, and Brandon, 1990) a doubt arose concerning the question format.  While the correct answers in the case of valid arguments seem conversationally appropriate, this does not seem so obviously the case for the invalid ones.  Contrast these two dialogues:
    (i) Suppose some vegetarians drink milk;
        would some people who drink milk be vegetarians?
        Yes.
   (ii) Suppose most teachers are women;
        would it be true that most women are teachers?
        Maybe.
Speaking personally, my usual response in cases such as (ii) would be to say "No," or more fully "No, not necessarily."

In Ennis' original investigation, the results were of no consequence for those tested; subjects had no time limit to complete the questions; and they were reminded of the meaning of the answers on every page of the question booklet.  The collection of most of the data in Jamaica has involved three serious departures from this set-up: the questions have formed part of an entrance examination for the Faculty of Education; there has been a time-limit for the examination; and instructions were given only on the first page of the appropriate section of the booklet.  With such an increase in pressure, it is likely that interference from linguistically odd constructions would be more serious than in Ennis' original investigations.

It was decided to investigate the possible effect of the question format by using the same items in two successive years with one small change in question format: to replace "Maybe" with "Not necessarily."

The item analyses in Tables 1 and 2 report the main findings: in general the change of format makes no difference to performance on the valid items (Table 1) but a considerable difference to performance on the invalid questions (Table 2).  The tables give the numbers offering each of the three possible answers or omitting to answer the question, the correct answer in each case, the difficulty index for the item (really a facility index), and finally the chi-square value and its probability for the comparison of the distribution for the two years.

Table 1:  Item analysis 1990/1 - Valid items  

• Note: the items are grouped by the logical form of the argument tested: modpon is if p then q, p, so q; modtol is if p then q, not q, so not p; and dissyl is either p or q, not  p, so q.  The difficulty index above each group of items is the mean for that group.

Table 1 shows that in only one of the 18 valid items in the test was there a significantly different distribution of answers between the two years, though it made no difference to the difficulty of the item.  In general the three forms of reasoning were of similar difficulty for the two groups.

Comparison with Table 2 shows that the overall facility of two of the three invalid types of item increased markedly and that in 13 of the 18 invalid items the distribution of answers was significantly different.  

Table 2: Item analysis 1990/1 - Invalid items   

• Note: see the note to Table 1; unmost is the principle most A are B, all C are A, so some C are B; minval is most A are B, most B are C, so most A are C; nonono is no A are B, no B are C, so no A are C.

These results seem to show that the change from "Maybe" to "Not necessarily" has a marked effect in many of those cases where it is the salient and correct answer.  In most cases, more respondents give the correct answer with "Not necessarily" as the prompt.  They suggest also the possibility that the low scores found by Ennis on invalid items might to some small extent be a product of the test format.

While performance seems to improve on the invalid items taken individually, subjects do not seem to improve quite so much when one looks at the consistency of their performance on groups of items.  Ennis characterises mastery of a principle as the correct answering of at least 5 out of the 6 items for that principle; "borderline" performance involves getting 4 of the 6 right.  Table 3 shows percentage mastery and borderline performance on the six principles tested for the two groups.  As can be seen, there is some improvement in mastery and borderline rates for two of the three principles, but there have been smaller fluctuations between years and there is a trend for performance on minval to improve over the years.  To some extent this failure of better performance on individual items to translate into mastery is due to the time constraints - Tables 1 and 2 reveal increasing omissions towards the end of the test.  But it is also a matter of what is still a hazy grasp of the logic - less than half the subjects get right answers on most of the items.

  Table 3: Percentage mastery/borderline performance       

REFERENCES

Brandon, E.P. (1990). The Deductive Logical Competence of Non-graduate Caribbean Teachers. ERIC Documentation Service, ED 315 330.

Ennis, R. H. and Paulus, D. H. (1965).  Critical Thinking Readiness in Grades 1-12 (Phase I, Deductive Reasoning in Adolescence).  Cornell Critical Thinking Project (ERIC Document Reproduction Service No. ED 003 818).

Nolan, C. A. and Brandon, E. P. (1986).  Conditional reasoning in Jamaica.  Paper given to the Conference on Thinking, Harvard, 1984 (ERIC Document Reproduction Service No. SO 016 755).  

HTML prepared 23rd January, 2000.

URL http://www.uwichill.edu.bb/bnccde/epb/ERIC.htm