By Professor Simon Crowe FAPS
Professor and Head, School of Psychological Science, La Trobe University

As with many other forms of applied and clinical psychology, neuropsychology has re-defined itself many times in its relatively short history. The original application of neuropsychological techniques was to detect cerebral lesions (Heaton & Pendleton, 1981), usually arising as a consequence of penetrating missile wounds and usually done in repatriation hospital settings. The tests developed to achieve this aim often gave a cut-off score indicating the presence or absence of brain impairment. The theory underlying this view was that people with brain injuries would perform in characteristic ways on these tests, and that these deficits would not be noted in individuals who did not have brain impairment.

As knowledge of brain function grew, it became clear that individuals with brain injury at different sites in the brain produced different patterns of deficit on the tests administered, and that slavish application of a cut-off approach could misdiagnose individuals who had relative preservation of the skills tapped by the test. The logical extension of these observations, given a particular deficit in the performance of a neuropsychological test, is that one could predict the site or sites of injury most likely to underlie the observed deficit (Kolb & Whishaw, 2003). Additional tests were thus developed with a view to tapping into these localised functions within the cortex, allowing more concise localisation statements.

The role of neuropsychology in diagnostic decision-making significantly reduced however with the advent of static imaging techniques, including X-ray, pneumoencephalography, angiography, computed tomography (CT), magnetic resonance imaging (MRI), functional measures of cortical activity including functional MRI (fMRI), positron emission tomography (PET), and single positron emission computed tomography (SPECT). As a result of this diminished role, the interpretation of neuropsychological test performance has shifted in its emphasis from diagnosis and lesion location to determining profiles of cognitive strengths and weaknesses and in estimating the consequences and the implications of brain impairment to the functioning of the individual (Long & Collins, 1997).

The role of neuropsychology in predicting functioning in the real world has often come about as a result of the involvement of neuropsychologists in medico-legal disputation. In a review in the Australian Forensic Psychiatry Bulletin (1989), RJ Stanley, QC noted glowingly that: "Perhaps this is because of the advent of well qualified and experienced neuropsychologists whose evidence is of particular assistance in cases involving brain damaged patients. In such cases, issues such as the capacity of the plaintiff to undertake employment or to marry and live an independent existence are the subject of neuropsychological evidence, whilst for the diagnosis and prognosis the neurologist is called" (p 5).

As a neuropsychologist, I am happy to accept the laudable commentary, although I would perhaps draw the line at endorsing anyone's nuptial bliss!

For neuropsychologists to say useful things about the real world, a distinction needs to be drawn between the implications of brain impairment and the issues of impairment, disability and handicap. Impairment is any loss or abnormality of psychological, physiological or anatomical structure or function. Disability, in contrast, is defined as the restriction or inability to perform an activity in the manner within the range considered normal for a human being as a consequence of the impairment. Handicap is defined as the societal disadvantage to the brain-injured individual which results from the impairment or disability and which limits or prevents the fulfilment of a normal social role for that individual (Unstun, Cooper, Van Duuren, Kennedy, Hendershot & Sartorius, 1995). Thus aphasia (i.e., a form of speech disturbance: the impairment) results in an inability to use the telephone (the disability) preventing work as a receptionist (the handicap).

Disability is usually measured by self report or by measures or observations of activities of daily living, including such things as eating, toileting, walking, dressing or bathing (Williams, 1996). Instrumental activities of daily living refer to more complex activities of the brain-injured individual including such things as shopping, driving, and managing money.

One of the limitations of neuropsychological assessment, applying to measurement of activity in the real world, is that it only measures the deficit at the level of impairment. Most neuropsychological tests have not been empirically validated to predict the impact of impairment on activities of daily living or instrumental activities (Heinrichs, 1990) making the connection between the test result and the target behaviour circumstantial at best.

This sort of evidence has resulted in a movement within contemporary neuropsychological circles (Long & Collins, 1997; Sbordone & Long, 1996) to attempt to develop so-called "ecologically valid" measures of functioning, thought to more closely approximate the real-world situations we are attempting to predict. One of the principal players has been Barbara Wilson who, with her Thames Valley Test Company (now part of Harcourt Assessment), has developed a number of tests aimed at trying to more closely approximate the real world. These tests include the Rivermead Behavioural Memory Test (RBMT), the Test of Everyday Attention (TEA) and the Behavioural Assessment of Dysexecutive Syndromes (BADS), thought to be better predictors of functional disability than either questionnaires or specific neuropsychological tests.

We should however be cautious about abandoning older well researched and validated measures of neuropsychological functioning (such as the Wechsler Scales) until such time as the "ecologically valid measures" have proven their worth. In a study of a Multiple Sclerosis population, Higginson, Arnett and Voss (2000) found that tests including the RBMT and TEA produced a 67 per cent correlation with the functional disability measure as compared to 38 per cent using standard clinical measures of memory and attention. While these correlations are impressive, they are by no means large effect sizes.

Recently my doctoral students Kathryn Hoskin and Kate Mahoney, and my colleague Martin Jackson, have become interested in whether the ability of "traditional" neuropsychological measures is as bad as we have been lead to believe by the "ecologically valid" camp. Much of the evidence presented in favour of the latter view is garnered from limited and narrowly defined samples consisting of only mild to moderately impaired individuals, making it difficult to generalise the results. Heterogenous samples, consisting of individuals with a wide range of cognitive impairment, also have limitations, again making it difficult to draw meaningful conclusions from the results. Some studies would appear to be hindered by small sample sizes, sometimes consisting of as few as five participants, or by use of inadequate predictors of functioning, consisting only of cognitive measures and not functional measures of ability. Others have used only global indices of cognitive impairment, combining the results of multiple specific tests, and therefore fail to address the ecological validity of specific, potentially accurate, individual predictors of performance. Further studies use inadequate functional disability outcome measures that are too global to adequately address the functional abilities of daily activities. One of the most salient methodological problems is that they do not provide details regarding the severity and/or extent of injuries sustained by their participant samples and instead refer non-specifically to "a head injured sample" or "a group of participants with TBI".

The approach we have taken to deal with many of these problems has been to deal with specific tasks in a reasonably well characterised group of participants, to measure both functional performance of the target behaviour, and, at the same time, apply standard neuropsychological measures, client self reports of competence, and the reports of a case worker who knows the client well. The measure we chose in the first of our series of four studies was to look at the competence of brain impaired individuals to perform successfully in their ability to use automated machinery, including automated bank teller machines, automated transport ticketing machines and automated telephone answering devices.

Automated machines and services have rapidly increased as alternatives to personal service delivery and are now a ubiquitous component of modern life. They are marketed to the public as accessible, convenient and efficient service deliverers, and the ease of the user interface is believed to have made their implementation successful without any need for formal customer training (McConaghy, 1999). Financial incentives also contribute to marketing campaigns for automated machine use.

Our first study in this area (Crowe, Mahoney, O'Brien & Jackson, 2003) explored the competence of an acquired brain injured (ABI) sample in using automated service delivery machines. Ninety brain injured participants (30 mild, 30 moderate and 30 severe as classified on the basis of their levels of functional impairment using the Environmental Status Scale) and 30 non-impaired participants completed a questionnaire measure of automated machine usage. The results indicated that the ABI samples consistently endorsed difficulty in dealing with automated transport ticketing machines, with automatic teller machines and with automated telephone answering and responding devices. The major impediment to successful performance identified by the ABI group was difficulty in understanding and remembering the instructions on each task. In the second phase of the study, half of these participants were randomly selected and observed in their actual use of the automated machines. The results indicated that the use of automated machines decreased with severity of functional impairment and that competency with automated machines declined with severity of functional impairment. The results also indicated that self-awareness of competency deteriorated with severity of functional impairment.

The second study in this series (Crowe, Mahoney & Jackson, 2004) set out to explore whether neuropsychological performance on standardised clinical measures could predict functional ability with automated machines and services among people with ABI. Participants were 45 individuals meeting the criteria for mild, moderate or severe ABI and control participants matched on demographic variables including age and education. Each participant was required to complete a standard battery of neuropsychological tests, as well as three tasks involving an automated ticketing machine, an automatic teller machine and an automated telephone service. The results showed consistently high relationships between the neuropsychological measures and competency with automated machines (both as single predictors and in combination). Logistic regressions correctly predicted more than 90 per cent of the overall variance on each of the automated tasks. Performance in using automated machines offer an ecologically valid functional measure of performance that represents a true indication of functional disability. It follows therefore, that functional assessment on automated machines and services is likely to be one of the best measures of learning and adaptive ability, and one that is likely to demonstrate strong correlations with severity of ABI.

With these successful results we next set out to study the money management ability of people with ABI and its relationship to neuropsychological test performance (Hoskin, Jackson & Crowe, in press: a). The case managers of 35 people with ABI, recruited through case management services, plus 15 matched controls completed an experimenter-designed Money Management Survey, a staff-rating measure of the client's money management abilities. Control participants reported difficulties with various aspects of money management, including paying rent and bills late and spending all of their money within a few days. In addition to the problems reported by controls, the group with ABI group had difficulty with spending all their money on things they liked, difficulty with leaving money aside for essentials and in using automatic teller machines.

Neuropsychological assessment demonstrated a modest ability to predict the overall money management score, with prospective and retrospective memory being significant predictors, but executive functions not so. Limited variance in executive abilities was noted within the ABI group. Prediction of specific money management behaviours (problematic impulsive spending and paying the bills or rent late) using specific neuropsychological measures was more successful than prediction of overall money management. These results provided support for the use of the Money Management Survey in an ABI population and indicate that people with ABI have greater problems with money management than people without ABI.

The last study in the series (Hoskin, Jackson & Crowe, in press: b) attempted once again to determine what the implications of neuropsychological performance are to real world performance. This time we investigated the ability of neuropsychological assessment to assist in determining capacity to manage personal finances in an ABI population. It was expected that, in comparison to people with ABI independently managing their personal finances, people with ABI who have administration orders from the Guardianship list would perform significantly worse on tests of executive/attention function and memory. Participants were recruited through community case management services: 28 individuals were managing their money independently (ABI-I) and 15 were judged not competent to manage their personal finances and had had an administrator appointed (ABI-A). As expected, the ABI-A group performed significantly worse than the ABI-I group on measures of executive/attentional abilities (impulse control, planning, flexibility of thinking and working memory), and these measures explained a large proportion of variance (63 per cent) and provided good classification of group membership, with 83.7 per cent of individuals correctly classified. However, contrary to expectations, there were no significant differences between groups on measures of memory. However, the results provide support for the 'ecological validity' of neuropsychological assessment to assist in determining the capacity of an ABI population to manage personal finances.

On the basis of this evidence, it seems fair to suggest that neuropsychological examination has moderately good success in predicting money management ability and performance on an important component of instrumental activity of daily living in the form of automated machine usage. It may, however, fall down on other quality of life type issues.

To provide the most concise predictions of assessment to the real world, the clinician should (as much as possible) establish a detailed description of what the client's target activity involves, allowing insight into what the requirements will be and whether the deficits as observed will create significant complications for the function. Ideally the approach of replicating the target environment in the consulting room is our best hope, but this, I fear, may be unduly optimistic. As Dr Graeme Senior (personal communication, 3rd July, 1999) has proposed, the most appropriate way to assess how someone functions in the real world after an injury is to move in with them for a month and watch how they cope in their daily lives; but how many clinicians have the time, patience or interest to undertake such an assessment?

The clinical neuropsychologist is in an excellent position to provide a detailed description of an examinee's present cognitive functioning. But, to make the leap of faith from deficit to disability may well be unnecessary and unwarranted on the basis of the available evidence, and any such leap must be made with appropriate qualification and hopefully on the basis of what evidence there is available. The benefits associated with well researched and well normed neuropsychological measures such as the Wechsler scales, outweigh the appearance of ecological validity associated with the newer generation of "ecological valid" neuropsychological measures until such time as the new measures are capable of explaining all the variance that traditional measures have explained. At this point in time I do not believe they have delivered on this promise.

References

Crowe, S.F., Mahoney, K., O'Brien, A., & Jackson, M. (2003). An evaluation of the usage patterns and competence in dealing with automated delivery of services in an acquired brain injury sample. Neuropsychological Rehabilitation, 13(5), 497-515.

Crowe, S.F., Mahoney, K., & Jackson, M. (2004). Predicting competency in automated machine use in an acquired brain injury population using neuropsychological measures. Archives of Clinical Neuropsychology, 19, 673-691.

Heaton, R. K., & Pendleton, M. G. (1981). Use of neuropsychological tests to predict adult patients' everyday functioning. Journal of Consulting and Clinical Psychology, 49(6), 807-821.

Heinrichs, R. W. (1990). Current and emergent applications of neuropsychological assessment: Problems of validity and utility. Professional Psychology: Research and Practice, 12(3), 171-176.

Higginson, C.I., Arnett, P.A., & Voss, W.D. (2000). The ecological validity of clinical tests on memory and attention in multiple sclerosis. Archives of Clinical Neuropsychology, 15, 185-204.

Hoskin, K.M., Jackson, M. & Crowe, S.F. (in press: a). Money management after brain injury: The validity of neuropsychological assessment. Rehabilitation Psychology.

Hoskin, K.M., Jackson, M. & Crowe, S.F. (in press: b). Can neuropsychological assessment predict capacity to manage personal finances? A comparison between brain impaired individuals with and without administrators. Psychiatry, Psychology and Law.

Kolb, B., & Whishaw, I.Q. (2003). Fundamentals of Human Neuropsychology (5th ed.). New York, N.Y: Worth Publishers.

Long, C. J., & Collins, L. F. (1997). Ecological validity and forensic neuropsychological assessment. In R. J. McCaffrey (Ed.), Practice of Forensic Neuropsychology: Meeting Challenges in the Courtroom. New York: Plenum.

McConaghy, J. (1999). Trends in e-commerce. Australian Financial Journal, 127, 35-42.
Sbordone, R. J., & Long, C. J. (Eds.). (1996). Ecological Validity of Neuropsychological Testing. FL, USA: Gr Press/St Lucie Press, Inc.

Senior, G. (Personal communication, 3rd July, 1999).

Stanley, R. J. (1989). The psychiatrist as expert witness. Australian Forensic Psychiatry Bulletin, 9, 2-7.

Unstun, T., Cooper, J., Van Duuren, K. S., Kennedy, C., Hendershot, G., & Sartorius, N. (1995). Revision of the ICIDH: Mental health aspects. Disability and Rehabilitation, 17(3/4), 202-209.

Williams, J. M. (1996). A practical model of everyday assessment. In R. J. Sbordone & C. J. Long (Eds.), Ecological Validity of Neuropsychological Testing (pp. 129-145). Florida: GR/St. Lucie.