Hoping to replicate the U.S. study as closely as we were able, we looked at the correlation between total score on the test and age, sex, and academic achievement. Taking age first, overall at Rusea's, r (204) = .33, p = .001. This, though significant, is not a particularly high correlation, reflecting the considerable range of scores in the different groups. It is quite considerably lower than that reported by Ennis and Paulus (1965, p. IV-28) for a subset of their sample, r (63) = .58. Within each grade, correlations are smaller but negative: for grade 7, r (60) = -.197, not significant; for grade 9, r (63) = -.286, p = .011; and for grade 11, r (79) = -.267, p = .008. None of these grade correlations is very impressive, though they are stronger than in the U.S. study. They may reflect delayed entry to school or the various possibilities for repeating years or examinations that exist in Jamaica.

As in the U.S. study, no significant correlation emerged between total score and sex. As can be seen from Table 2, boys and girls scored virtually the same in all groups. The overall correlation at Rusea's was r (204) = -.011, which given the coding, male = 1, female = 2, indicates a slight male advantage; but that is in fact more the exception than the rule. In most groups the girls scored slightly higher than the boys, though in only one case have we stumbled upon a statistically significant difference: in Hillel, grade 7, r (18) = .552, p = .007. The very general equality between the sexes is perhaps a little surprising in Jamaica since it is well known that girls continually out-perform boys throughout the educational system.

While overall no sex difference in reasoning ability emerges, there is one phenomenon of some interest that might warrant further investigation. The whole sample of 420 has been divided into four ranks (the top rank with a total score of 51 and above, the second rank from 41 up to 51; the third rank from 31 up to 41; and the bottom rank below 31). If one then groups together the valid patterns of argument there are absolutely no sex differences of any interest, and there is a smooth transition from an average score of 37.9 (SD = 5.055) for the top rank (out of a possible 48) to an average of 15.9 (SD = 4. 755) for the bottom. These two very roughly formed groups number 120 and 127, respectively. But when one looks at the groups of invalid principles (likewise now out of 48), not only are there no significant differences between the ranks but in fact the first and second rank girls, and the second rank boys, get worse scores than the bottom two ranks. The figures are given in Table 5. They testify very powerfully to our respondents’ utter inability to recognize invalid patterns of argument, and perhaps to the over-confidence of the brighter students. While the differences between boys and girls in the top rank are barely significant (F (1, 118) = 5.76, p < 05, for what little it may be worth), they may point to an interesting, if subtle, difference in the socialization of academically superior boys and girls.

Table 5

Average Scores on Invalid Principles, by Achievement Ranks and Sex

Unfortunately we have not been able to obtain data on the general intelligence of the pupils tested. They were chosen so as to be fairly typical of students in secondary education in Jamaica, but we cannot demonstrate that they are. Instead of looking at the correlation between the logical reasoning scores and some other measure of intelligence, Nolan obtained school grades in English, Mathematics, and Biology for all the Rusea's students. Many different things can be tested in school examinations in those subjects, and we have found virtually no correlations of any size between scores on the conditional reasoning test and these grades. Since we have no doubt that the test tests some aspects of logical reasoning, we are inclined to conclude that such aspects of reasoning are peripheral to whatever the school is seeking in the subjects chosen.

While we are then unable to report correlations with the other measures of academic ability, we are in a position to report on the very clear differences between educational strata within the school. As noted above, Nolan took forms from the top academic stream and other forms corresponding to the more vocationally oriented new secondary school stratum. In Rusea's the academic stream diverges in grade 11 into a Science and Arts group. Whatever aspect of the data one examines, this bifurcation between academic and vocational is extremely pronounced. Most of the improvements registered in Tables 2 and 3 are due to the academic stream. Thus, if one looks at mean total score, in the academic stream it moves from 35.7 (SD = 10.5) in grade 7, 47.3 (SD = 10.6) in grade 9, to 48.1 (SD = 10.5) in grade 11 Arts and 55.7 (SD = 11.4) in grade 11 Science. On the other hand, the non-academic groups virtually stand still, although they enter not far below the Common Entrance passes. In grade 7, mean total score is 30.8 (SD = 8.5), in grade 9 it is still 30.1 (SD = 9.0), and it ends up in grade 11 Vocational at 38.1 (SD = 8.01), scarcely above the grade 7 academic stream: not much for five years of secondary schooling.

Tables 6 and 7 give a more detailed breakdown of these developments by giving percentage mastery and lack of mastery of the 12 logical principles by forms at Rusea's. The figures should be compared with the figures in Table 4, in which the groups are known not to be representative of their grade levels. The stagnation of the non-academic groups is visible also in the Chapleton figures in Table 2, and is the reason why all grades were combined for Chapleton in Table 4. In fact at Chapleton, the grade 7 had a slight edge over the others, possibly because it still contained a few pupils who could stand a chance at a second competition for entry to some sort of secondary schooling.

Table 6

Percentage Mastery of Logical Principles at Rusea’s, by Forms

Table 7

Percentage Lack of Mastery of Logical Principles at Rusea’s, by Forms

It is also notable that at Rusea's both the grade 11 academic forms seem, in some cases at least, to have regressed from the levels of performance exhibited by the academic grade 9 form. We do not have a clear explanation for this tendency. To some extent, grade 9 pupils may be close to peak academic commitment, before the distractions of adolescence affect them markedly. For these particular students, there is also the fact that the creation of the combined school was not an uncontroversial matter, so that the grade 11 pupils may well have had more distractions than normal. It may also be that motivation to succeed academically is pretty small in what is a comparatively underdeveloped community and parish. But it is worth noting on the other hand that the Science group produced the highest percentages for mastery of the invalid argument principles of any of the groups tested. Perhaps the teaching of science does encourage a critical spirit.

Besides the 12 patterns of argument, the test also examined the ability to handle three different sorts of content. Expressing all scores out of 12, Table 8 gives the mean scores on the three types of content for each of the grades. While it cannot be said that any group can reliably handle the contrast between validity of argument and truth or falsity of the statements in the argument, by grade 9 students are handling suggestive content as well, or as badly, as concrete familiar content, a finding that can be confirmed by reference to the difficulty and discrimination scores for each item in Appendix A. On the other hand, in grade 7 suggestive content is significantly more difficult; compared with symbolic content, correlated t (60) = 3.07, p = .003. By contrast, in grade 9 the greatest gain has been made with symbolic content, which is now significantly easier than the other two; compared with suggestive content, correlated t (63) = 2.44, p = .018.

Table 8

Mean Scores on the Three Types of Content, by Grade at Rusea's

Note: All scores are out of 12.

IV

However one looks at the Rusea's data, it is clear that the Jamaican students are not performing as well on the conditional reasoning test as the U.S. group. This is very striking if one looks at the overall performance by grade, but it remains true when one concentrates on the academic stream. Tables 3, 6 and 7 should be compared with Table 9, which records the percentage mastery and lack of mastery of the U.S. group on each of the principles.

Table 9

Percentage Mastery and Lack of Mastery of Logical Principles, by Grade, in U.S.

Note: Data are taken from Ennis and Paulus (1965), p. V-16 and p. V-18. Age is mean age in months.

A similar picture emerges if one compares the overall mean difficulty of the test. At Rusea's, in grade 7 this is 38.9; in grade 9, 43.2; and in grade 11,, 51.2. In the U.S., grade 5 showed a mean difficulty score of 47.5; grade 7, 55.8; grade 9, 54.6; and grade 11, 61.5 (Ennis and Paulus, 1965, p. IV-30). Yet again, one can prepare mean total scores for the grades. At Rusea's, in grade 7, mean total score is 33.3; in grade 9, 38.4; and in grade 11, 47.2. In the U.S., grade 5 gives 42.4; grade 7, 51.7; grade 9, 55.3; and grade 11, 56.6 (ibid. p. V-16). Both these sets of figures suggest that the U.S. respondents reach a fairly flat plateau by grade 7, although there are some improvements as Table 9 shows. The Rusea's students, on the other hand, are changing rather more rapidly (at least the ones in the academic stream) and considerably narrow the gap that originally separates them from very roughly comparable U.S. grades.

While there is a clear gap between the levels of achievement in the two countries, there are many respects in which the relative picture of logical competencies is very similar. Much could be gleaned from a close comparison of the data presented by Ennis and Paulus (1965) and that presented here; we shall only note a few points now. In both countries the relative difficulty of the 12 principles is similar. The two simple principles using "only if" OMODTOLL and OMODPON, are the simplest, followed by detachment and the two transitivity principles, MODPON, FULLTRAN, and PARTTRAN; the most difficult group of valid argument patterns is BICOND, MODTOLL, and CONTRA. All of these valid patterns are much easier than the four invalid arguments. In Jamaica it is noteworthy that it is only in grade 11 that the invalid principle items begin in general to discriminate positively (see Appendix A).

Again, in both countries we find that suggestive content starts off more difficult but soon becomes as easy as any other material. There are also many close similarities in the relative difficulty and discrimination of items within the different item-groups, which it would be tedious to enumerate now, but which give us strong reason to believe that our Jamaican students were responding in similar ways to the same problems.

More to the educational point, however, is to ask for an explanation of the very noticeable differences in level of performance. This is the more pertinent since we think the differences are symptomatic of a very important contrast. It is well known that schools in Jamaica lack many of the facilities, books, electronic gadgets, etc., etc., that are now taken for granted in developed countries. It is also well known that the general grasp of subject matter on the part of teachers in Jamaican schools is far from ideal — we have already adverted to the problem caused by an inadequate understanding of the intricacies of the language in which instruction is meant to proceed. These varied inadequacies contribute to the explanation of the generally low standard of Jamaican entrants in international examinations, the G.C.E. and C.X.C. examinations in particular. But while a lack of content is no doubt deplorable, it is also something that is usually easily remediable. But one of the educational paradoxes surrounding reasoning is that, while many people laud critical reasoning prowess and try to defend the most diverse curricula in terms of helping pupils to think better, in fact very little attention is ever paid explicitly to reasoning and to how to improve it. A grasp of logical relations is something that is most definitely "caught" rather than "taught" for the vast majority. Students' competence in such matters is, then, an indirect but fairly important measure of the general intellectual quality of their schooling and social context. It is for this reason that we believe our results are of considerable importance for educational planning in Jamaica.

They suggest that the lack of facilities and the inadequate knowledge and skill of many teachers are damaging, not only in the obvious areas of facts and knowledge of theories or practice, but also in as much as they perpetuate an intellectually immature ambience. Many have complained of the deadening experiences of rote learning in the primary school (for a recent sociological survey in Chile, see Filp, Cardemil, Donoso, Torres, Dieguez, and Schiefelbein, 1981; for material directly relevant to reasoning, see Hansen and Pearson (1983)); what we see in the performance of our grade 7 students, and in the whole last phase of the All-Age school, is part of the intellectual cost of this educational neglect.

The most we can say on the basis of the Rusea's data is that the Jamaican school system can still deliver the goods for that very small percentage of secondary pupils it channels into the academic stratum. As we have noted, these forms end up very close to U.S. levels of performance, after they have had four or five Years of "beating book" in secondary school to combat whatever primary school may have done to them (the local phrase reveals perhaps a very great deal of the real social meaning of schooling in Jamaica).

Since we are putting most of the blame on the Jamaican primary schools, it might be worth adding that the extraordinarily high scores found at Hillel may be due, in considerable measure, to the fact that most of the pupils have gone through Hillel's own very efficient, U.S-style, primary school. To go to either institution, pupils need to come from backgrounds wealthy enough to provide a great deal of other intellectual stimulation, but we think that the kind of teaching in primary school counts for much as well. Jamaican parents are often prepared to pay a lot of money in the same belief.

One important aspect of educational failure in Jamaica relates to language, as we noted earlier. For the pupils who did fairly well on the test, language appears not to have made much of a difference. Everything they do in school, all the books they read are in English, so one would not perhaps expect much difference here. What we are at present unable to say is what effect the language of the test has on the academically weaker students. It is planned to translate a simpler and shorter reasoning test into Jamaican creole and to try to estimate if there are any significant differences in performance on the two versions of the test, but that is a task for the future. It is also a task which is not warmly welcomed by many teachers.

One obvious implication of the results is then to reinforce the many calls for all-round improvement in the quality of Jamaican schooling, especially at the primary and non-academic levels (though one should not be too sanguine about the academic either). But in terms of more detailed implications for the teaching of reasoning it is unfortunately still too early to offer much more than hunches. Our explorations in the data have raised one serious question regarding the conceptual framework Ennis and Paulus used. One way of putting the issue is to say that the scoring system, and indeed the whole test, allows one to discover whether someone can obey a rule, whether the person knows what counts as obedience to the rule. They also allow one to judge that someone does not know what counts as obedience to that rule. But if grasping a rule of logic is much like grasping a concept, one might want to insist also that we be able to judge that someone knows when the rule is broken or misapplied. Does it really show much of a grasp of modus ponens (MODPON) if one thinks it equally good as affirming the consequent (AFFCON)?

Another way of putting our point might be to say that it may be premature for empirical investigations of logical reasoning to think continually in terms of logical rules. For persons who have studied logic, the rules often function as "target forms", to borrow Kielkopf's recent label (1984, p.21). They are codifications with which one compares actual arguments; evaluating an argument is not thinking it through according to some set of logical rules but it is rather a matter of holding it up against a formal skeleton. No doubt there are some rules for doing that, but those rules, the ones logicians follow in evaluating arguments, need have nothing much to do with the paradigms of logical reasoning by appeal to which judgements are made. Ordinary people, on the other hand, do not usually have the logician's codifications to hand, and so there is very little reason to expect that their own judgements of what follows or does not follow will neatly mirror the laws of logic. Nor perhaps, though here one is in danger of entering a trackless waste of argument, for thinking of what they do as the following of any rules, at least rules that look anything like the laws of logic.

Perhaps we may illustrate the kind of point we are making on the basis of some very tentative work we have been doing with the reasoning data. In one exploratory factor analysis of the results on the 12 principles for the top rank students, the first factor extracted (by oblique rotation, otherwise using the SPSS default options) seems to involve a very general kind of conversion and inversion and their associated detachment rules. Roughly the "rule" seem to sanction moves from if p then q to if not q then not p (it loads highly on CONTRA) or if not p then not q (a move that is not tested for) and with the addition of q the conclusion p (it loads highly on low scores for AFFCON) or with the addition of not p, the conclusion not q (it loads highly on low scores on DENYANT). The factor also loads highly on BICOND but not on either form of modus tollens. Ordinary, valid, detachment (found in MODPON, FULLTRAN, and PARTTRAN) crops up in the second factor. We do not wish to place any weight upon this particular result, but it illustrates the kind of "rule" that may well be operative, and it suggests why some students may do very well on the valid principles while failing utterly on the invalid ones (cf. Hillel grade 9 in Table 4). It is probable that the tactics needed to pick out invalidity are very different from the inferential procedures used to arrive at conclusions.

The distorting effect of logician's targets can also be illustrated by a contrast between the results on OMODTOLL and OMODPON, both in Jamaica and the U.S., and Ennis' recent reflections on the complexity of only (1981, p. 367). The empirical data reveal that at least some uses of only if are considerably easier than their logical analogues with if (which hardly fits well with the common claim that in ordinary people's usage if incorporates only if) whereas Ennis reports that the word only makes life more difficult for him. We agree with him. But we suspect there is a widespread tendency among students of logic not to think in terms of only: when faced with a premise like p only if q, one translates it first into if p then q before evaluating the argument, because the targets are not usually stated in terms of only if. Ordinary people who do not use the targets may find only if easier because it is, or feels, more restrictive, it seems to leave fewer possibilities open. They certainly seem not to use the translation strategy: the same exploratory factor analysis leaves OMODTOLL and OMODPON out in the cold.

One of the most pervasive components that definitely seems to make for complexity is negation. But, as Ennis says, the evidence is mixed. One way in which the extra complexity of negation appears in the test is in the form of the questions asked each time. Usually the form is "Then would this be true? p". In some cases, however, the form is "Then would this be true? Not p. " (We have marked such items with a "?" in Appendix A.) The most striking case where this appears to make a difference is in PARTTRAN, where in Jamaica and the U.S. item 62 is markedly less difficult than the others. In Jamaica, the one negative question in OMODTOLL also seems consistently harder, as also for grade 7 and 9 in BICOND, but these cases are not found in the U.S. data, nor do negative questions seem noteworthy in the case of the other principles.

There is another set of items in which a possible effect of gratuitous negation is confounded with a possible effect of presenting the premises in a non-standard order (these items are the fourth in each item group in Appendix A). Here the antecedent of one conditional promise is a negated proposition and the premises are presented out of standard logic book order, if that is possible (so that, for example, the form of item 14 is not p, if not p then q, therefore q). In both Jamaica and the U.S. these items were often more difficult or at least anomalous in discriminating power. Thus in both countries, OMODTOLL, OMODPON, MODTOLL, and BICOND showed this result; in Jamaica, MODPON (not so clearly), and in the U.S., PARTTRAN also. Neither group showed any difference here on FULLTRAN or CONTRA. It is not so obvious with the invalid principles but there are signs of an effect in both groups on CONVERSE and AFFCON. If we can carry over Roberge's (1970) finding that reversing the order of premises in a two premise argument makes no difference (and after all, non-standard order is non-standard only for the readers of logic books), these results presumably reflect another effect of a lot of negations in an argument.

Such a finding would suggest the importance of stressing abstract logical form, so that students recognize that as far as validity goes there is absolutely no difference between if p then q, p, therefore q and if (not p or r) then not q, not p or r, therefore not q. But obviously the major pedagogical recommendation coming out of the results is the general one to find ways of making students appreciate when an argument is invalid. Constructing counter-examples is an imaginative exercise, often easiest with rather fanciful claims, so it is perhaps not surprising that it is not the sort of thing that sober and rigid schooling encourages. But the evidence is overwhelming that it is the sort of skill that is most lacking, not only in school children but in the population at large. And it is arguable that it is a serious lack: not realizing that a conclusion validly follows from some premises may prevent one exploiting one's information to the full, but it isn't yet to fall from truth into error (we have not yet analysed the kinds of error students make on the reasoning test so this claim is somewhat tentative). But thinking that one's information does entail a conclusion that in fact does not follow puts one seriously at risk of such a fall from grace. Of course one can construct catastrophic scenarios for the case of not realizing the genuine consequences of a valid argument pattern, but we suspect that on balance it is more damaging to over-invest in apparently safe conclusions than to refuse some real pearls.

We have suggested that certain points are indicated for the promotion of the teaching of logical reasoning. But it is worth noting part of the probable reason for the educational paradox referred to above. While teachers believe in the importance of reasoning, they are in fact on a level with their pupils when it comes to dealing with arguments in general. Very few teachers have ever studied reasoning formally, so they have no more direct access to the logician's targets than their pupils. In addition, once one remembers the tremendous difficulties of translation between the aseptic formalism of logic and the complexities of ordinary language, even the trained logician has to acknowledge a margin of possible error in his evaluations and descriptions of actual arguments. For both these reasons, then, reasoning, either formal or comparatively informal, is not a subject at which school teachers can claim expertise and so one finds not only the neglect already mentioned, but often a violent antagonism to actually trying to tackle questions of argument directly. For these reasons, one should be wary of expecting too much from such expressions of commitment and willingness as one may find for reasoning and critical thinking in the classroom.

V

By way of conclusion, we shall briefly indicate some of the lines of research we hope to pursue. As already mentioned, it is hoped to examine the effects, if any, of English versus Jamaican creole on logical reasoning competence. It is also hoped to take up some of the many loose ends in the study of actual reasoning, some few of which we have briefly mentioned already: the effects of negation; the effects of order of premises; the relation of inversion to denying the antecedent (cf. Fillenbaum, 1978); the relation of a conditional to what Mackie called its "contrary conditional" (1973, p.109: if p then q and if p then not q); etc. And it is also hoped to get similar base-line data on Jamaican students' abilities with other kinds of inference: class-logic, and some other sentential operators.

This work is being conducted on students (school or Teachers' College, for the most part) because it is much easier to get them to take often tiresome tests. As already indicated, we would not expect their teachers to perform very differently, and work is in progress to check on this hunch, although not with purely formal material. So far, we have tried using one of the Illinois Thinking Project's informal reasoning tests (Ennis and Weir, 1983) on the teachers who have enrolled at U.W.I., Mona, in the current academic year. Preliminary results are not very impressive, reflecting minimal skills in picking out and criticizing bad reasoning.

Since we still believe that an appropriate introduction to formal matters can contribute much to a person's battery of argument evaluation strategies, it is hoped that the suggestions coming out of the formal reasoning tests will play a part in our future attempts to find ways of upgrading the general critical reasoning competence of Caribbean teachers.

FOOTNOTES

1. The authors wish to thank the pupils, teachers and principals through whose co-operation the data has been assembled, and in particular Professor R. H. Ennis for his interest and generosity in making available the test we have used and various other material from the Illinois Thinking Project. We hope that this partial replication of his early work will be of some use to the Project. We should also like to thank Graham Webb, Ian Isaacs, and the staff of the U.W.I., Mona, computer centre for varied assistance in the analysis of the findings. None of these persons is in any way responsible for what we have made of them or for any of the other comments we have made in the paper.

2. Figures here come from Government of Jamaica (1982). We have used 1981 figures since the official exchange rate was then stable; the decline since that time makes the sense of comparative figures even more problematic than is usual. For the extreme inequality of Jamaican society, see McLure (1980), rediscovering for the early 1970s the much earlier findings of Ahiram (1964).

REFERENCES

Ahiram, E. Income distribution in Jamaica, 1958. Social and Economic Studies, 1964, 13.2, 333-369.

Brathwaite, E. Contradictory omens. Mona: Savacou Publications, 1974.

Craig, D. Language, society and education in the West Indies. Caribbean Journal of Education, 1980, 7, 1-17.

Ennis, R. H. A conception of deductive logic competence. Teaching Philosophy, 1981, 4, 337-385.

Ennis, R. H., Gardiner, W. L., Guzzetta, J., Morrow, R., Paulus, D., and Ringel, L. The Cornell Conditional Reasoning Test, Form X. Ithaca: Cornell Critical Thinking Project, 1964.

Ennis, R. H. and Paulus, D. H. Critical thinking readiness in grades 1-12 (Phase I. Deductive reasoning in adolescence). (Cooperative Research Project No. 1680) Ithaca: Cornell Critical Thinking Project, 1965. (ERIC Document Reproduction Service No. ED 003 818).

Ennis, R. H. and Weir, E. Ennis-Weir Argumentation Test, Level X: An Essay Test of Critical Thinking Ability. Illinois: Illinois Thinking Project, 1983.

Fillenbaum, S. How to do some things with IF. In J. W. Cotton & R.L. Klatzky (Eds.) Semantic factors in cognition. Hillsdale, N. J.: Lawrence Erlbaum Associates, 1978.

Filp, J., Cardemil, C., Donoso, S., Torres, J., Dieguez, F., Schiefelbein, E. La escuela: Complice del fracaso escolar? Revista de technologia educativa 1981, 340-358.

Government of Jamaica. Economic and social survey, Jamaica, 1981. Kingston: National Planning Agency, 1982

Hansen, J. and Pearson,, P. D. An instructional study: improving the influential comprehension of good and poor fourth-grade readers. Journal of Educational Psychology, 1983, 75, 821-829.

Kielkopf, C.F. Forms for "informal logic". Informal Logic, 1984, 6, 21-25.

Mackie, J. L. Truth, probability and paradox. Oxford: Oxford University Press, 1973.

McLure, Jr., C. The incidence of Jamaican taxes. 1971-72. Social and Economic Studies, 1980, 29, 101-133.

Miller, E. Education and society in Jamaica. In P.M.E. Figueroa and G. Persaud (Eds.) Sociology of education: a Caribbean reader. Oxford: Oxford University Press, 1976.

Nolan, C. An investigation into the logical reasoning competence of selected groups of students in a comprehensive high school in Jamaica. Unpublished B.Ed. study, University of the West Indies, Mona, Jamaica, 1984.

Roberge, J. J. The effect of reversal of premises on children's deductive reasoning ability. Journal of Psychology, 1970, 75, 53-58.

Stone, C. Democracy and clientelism in Jamaica. New Brunswick: Transaction Books, 1980.

UNESCO. Jamaica: development of secondary education. Paris: UNESCO, 1983.

Appendix A

Item and Item-Group Difficulty and discrimination Indices by Grade at Rusea’s