This podcast is brought to you by PAR. PAR offers the SPECTRA Indices of Psychopathology, a hierarchical-dimensional look at adult psychopathology.
The SPECTRA is available for paper and pencil assessment or administration and scoring via PARiConnect. Learn more at parinc.com\spectra. For a limited time, you can get one free administration and score report for the spectra on PARiConnect by calling PAR at 855-856-4266 just mention promo code S-P-E-C.
[00:01:00] Hey all, glad to have you here. I am happy to share an incredible interview with you today.Dr. Esther Geva is Full Professor at OISE, at the University of Toronto: The Ontario Institute for Studies in Education. Her extensive research, publications, and graduate teaching focus on:
(a) The development of language and literacy skills in students from diverse linguistic backgrounds.
(b) L2 students or English Language Learners with learning difficulties.
(c) Cultural perspectives on children’s psychological problems.
She has presented her work internationally, served on numerous advisory, policy, and review committees in the US and Canada concerned with assessment and policy issues related to culturally and linguistically diverse children and adolescents. In applied practice, she is interested in community-based approaches to prevention and intervention in minority groups, and options in assessment, instruction for English Language Learners and other L2 learners, and culturally sensitive work with families coming from diverse linguistic and cultural backgrounds.
She has a wealth of experience. What is not mentioned in the bio is that she has co-written a book with Judith Weiner called Psychological Assessment of Culturally and Linguistically Diverse Children and Adolescents A Practitioner’s Guide that will be linked in the show notes.
This was such a rich conversation with Esther. We touch on a number of concepts related to assessing English language learners and culturally diverse learners. This is a little bit [00:03:00] different format than typical in that Esther provided me with a few points that she wanted to make sure that I highlighted before we jump into the interview. And I’ll also recap these at the end, just because these are such primary points that we hope that you might take away.
The first is that when assessing second language learners, which can include L2 English language learners, bilingual, multi-lingual, one has to be concerned about both language and cultural differences. And we do talk about this during the interview, but we want to put a fine point on it here.
The second is that language is complex and takes a long time to develop. So, interpreting test norms should be done judiciously. We also discuss this. Assessing language and literacy in the first language is a good idea. One would be looking for confirmation and consistency [00:04:00] across languages. We touch on this in the interview, but I don’t know that it was as clear as Esther has written it here. So I want to certainly highlight that point.
The next is that when interpreting assessment data, one should consider the age of arrival in the immigrant-receiving country, exposure to and opportunities to acquire the second language, school interruption, language typology, positive and negative transfer.
Now, these are all things that we again touch on, but just to pull them together and create one succinct point here, I think is important.
And lastly, we do talk during the interview about the report, but again, Esther just wanted to emphasize that the report should synthesize the various information sources that you gather. And as a psychologist, she really wants to know if the report has taken into account the nuances provided by considering all kinds of complementary data sources with these [00:05:00] linguistically and culturally diverse individuals.
So, those are just a few points to keep in mind, guideposts, if you will, as you’re listening to the interview. My hope is that you will take away not only these points, but many others from a true expert in this area.
So without further ado, let’s get to my conversation with Dr. Esther Geva on Culturally and Linguistically Diverse Assessment.
Hey Esther, welcome to the podcast.
Dr. Geva: Hi there.
Dr. Sharp: How are you this afternoon?
Dr. Geva: I’m very well. I’m [00:06:00] on sabbatical and was lucky enough to be able to make it to Vienna. So, I am a visitor at The University of Vienna. And it’s a good place to be while you wait for this pandemic to disappear.
Dr. Sharp: Yeah, I would imagine. There are many times over the course of the pandemic that I wish I could somehow go on sabbatical myself to somewhere else. I’m glad that you made that work.
Dr. Geva: Yes.
Dr. Sharp: Well, I’m happy to have you. Honestly, I found your work just by looking around online. I’ve been trying to integrate more discussion about the assessment of culturally diverse individuals and marginalized groups and so forth over the last year or so. And this is one of those cases where you have written or co-written a book on that very topic. So, I’m very excited to have you and dive into some of these.
Dr. Geva: I’m glad you discovered us.
[00:07:00] Dr. Sharp: Yes, thanks for being here. Well, let me start. A typical first question here for listeners, they always know I’ll ask, why is this work in particular important to you right now?Dr. Geva: If I may rephrase it right now, it has been important for me for quite a while. When I was in graduate school, I started to become interested academically, intellectually in the topic of second language learning. Part of that reflected my own personal history. I was observing my kids who were being exposed to two languages. So, this was all very nice and dandy.
And then, one of the things that happened shortly after I finished my Ph.D., I started seeing the literature work, which was very important, which talked about [00:08:00] biases, in particular, I’d like to give the one example, which I think is a very powerful one, of one of my colleagues Jim Cummins, who did seminal work in the late 70s early 80s where he showed that children in this particular, it was in Canada, that children who were children of immigrants were diagnosed as being intellectually disabled. At that time, they used a different term but it doesn’t matter. And he showed an effect that all they would do is just give them an IQ test and then say, yeah, this kid has an intellectual disability. In effect, what he showed was that in many of those cases of the immigrants, they didn’t have a chance yet to develop English language skills, which was the group that he was looking at, and so they were being diagnosed as having feelings for disability whereas, in fact, it was very possible that they have [00:09:00] there was still not a good command of English, of the cultural aspects of the testing, and so on and so forth.
He talked about overdiagnosis. Other people at that time also talked about it. People in Sweden. Francis talked about it as well. And so what that get… So at the same time, you sit up from the rise of this idea that there are learning disabilities and then getting better research and then conceptualization of learning disabilities, how we diagnose them, et cetera. People like Jim Cummins talk about the fact that many of those children may not actually have a problem and that it has to do with the fact that there is no consideration of their linguistic proficiency in English.
So when I started to work in this area, I was seeing extra children who were kids of immigrants and who [00:10:00] definitely, for instance, had problems in learning how to decode words, but the system said you cannot diagnose an immigrant child for having a learning disability for five years because first, you have to make sure that the child has had enough opportunities to develop their language skills. So for instance, a same-day immigrant to the US or to Canada, English-speaking Canada, let’s say, you would want to first make sure that they’ve actually had enough opportunities to learn the language before you can then start to say, maybe there’s another problem.
So when I got involved in this area, what I was seeing were kids who actually had decoding problems but the system refused to accept that they were second language learners, but they also had decoding problems. So, that’s what mobilized me to say, well, can we figure out a way of teasing apart those two? The difficulties for instance in doing math [00:11:00] on decoding, can we tease it the part for the fact that somebody is a second language learner?
So that’s what mobilized me to the book, to the course, I’ve been teaching lots of studies, projects.
Dr. Sharp: Sure. I’m always struck by how folks turn these things into a life’s work. And it seems like you have really done that.
Dr. Geva: Definitely.
Dr. Sharp: Then a lot of folks, there’s a personal component as well. I think that’s always a big driver for many folks.
Well, I’m excited to get into some of this with you now. This is a dense, multi-layered complex topic. So, I’ll do my best to touch on the things that are most relevant for folks. But I wonder, could we maybe just start with a little bit of context and talk about… So, [00:12:00] I’ll give some context myself. We’re going to be discussing a lot of aspects of assessment and how to assess English language learners and considerations there, but could we maybe start with a general discussion of the tests that we use and how they are either standardized or normed for individuals who are speaking different languages and how that all relates as we think about assessment.
Dr. Geva: Well, the way I started to tackle this issue was to ask questions. Let’s take something like what we call dyslexia word-based reading disability. The fundamental question I asked myself was, to what extent do you actually have to be completely fluent in English[00:13:00] in order to be able to look at the word and be able to say, that’s the word, high or school or classroom or Spring or whatever? So to what extent do I actually need to know the meaning of the word, in other words, to be proficient in the language in order to look at the page and be able to sound out the word, and maybe if I see the same word a few times, I just remember what it is and I can say spring without having to go spring. Okay?
So that was the fundamental question. And so what we started to do and that ended up being little, did I know, but it was a project, it took us 10 years to collect data. So basically, what we did was a very complex project. We started by tracking children from Grade I to Grade 6. So every year, we gave them the same battery of tests. At [00:14:00] some point we modified the test but we don’t need to get into that now.
And we ask ourselves… and so we had both monolingual children- so English as a first language, and we had children of immigrants to Toronto. Toronto is a highly multilingual place. At the moment, there are about 200 different languages spoken in Toronto as the home language. So highly multilingual. And so we went to the schools, we begged them, can you please allow us to follow those children so we can figure out how to develop their reading, how to develop their writing skills, and how did it also develop on various cognitive processes which we know are associated with learning to read, with problem-solving, et cetera. So it took us 10 years to collect this data. And we ended up with data of about 450 kids who started Grade 1.
So they all start in Grade 1. And [00:15:00] the reason we started with Grade 1 was because we wanted to start with the kids before they started to learn formally to read some other language. So we had the common ground and tested them every year for six years. And because we recruited for 4 consecutive years, additional cohorts of Grade 1 kids, the data collection took 10 years and quite a few dissertations came out of that.
To go back to the topic for today, one of the questions we asked was how long does it take for children to develop English, and in particular, in a moment I’m going to focus on vocabulary? And the other question is, to what extent do second language learners who come from homes where different languages are spoken in their homes and all come to Grade 1 starting to study in English, in this case, in the metropolitan Toronto area. [00:16:00] But it would be different in other places I would say.
How does the development of language skills look like over time? Do they close the gap on various language measures? What does their word reading look like? What does the ability to decode nonwords look like over time? What do the various underlying cognitive processes that we know are related for instance to dyslexia, to what extent do they reflect related to whether they are second language learners or not?
And what we found was that basically, it takes a long time to develop language in particular. And I’m going to talk about vocabulary. So the children get better from year to year on the vocabulary as do most of the monolinguals, of course. But even after six years [00:17:00] of being in an English-speaking environment where for 5 to 6 hours a day, they’re exposed to English, on average, they don’t close the gap on their vocabulary knowledge. So even after six years of being in the educational system, there’s still a gap. And so from that perspective, definitely what Jim Cummins said a few years earlier, was basically replicated. So that’s one finding which we can come back to and talk about what are the implications of that for various assessments.
At the same time, what we found was that on the basic reading skills which don’t require comprehension, for instance, looking at words and being able to decode them being, there’s no difference between monolinguals and second language kids on average. In other words, if I start in Grade 1` and you start in Grade 1, and this kid arrived a year ago from Ghana or Saudi Arabia, they’re all in Grade 1, that on [00:18:00] decoding, on average, they look the same as the monolingual.
Dr. Sharp: That’s fascinating. So can you go back a little bit then and define when you say vocabulary knowledge and how the kids never really closed that gap, what exactly do you mean by vocabulary?
Dr. Geva: One of the measures that we used was the Peabody Picture Vocabulary Test which is a receptive vocabulary measure. And so basically what the kids see… There are versions of the test in many different languages. So basically in this particular test, the PPVT as it’s fondly known, the kids, for each item, they see four different pictures, a pencil kind of a picture, simple pictures.
And you’ll say to the kid, point to the bird. So they have to point to the bird and not to the cat, okay? That means they know what the word bird means. That’s a standardized, [00:19:00] highly reliable measure, and of course, as the items continue on the test, I can set it as a measure, the items become more difficult, less frequent, and so on.
Statistically speaking from Into associates measurement properties, I’d say, it’s a good measure. And I believe many researchers around the world are using versions of that. So that’s an example of a vocabulary measure. We have some other measures, but I don’t think we need to talk about them now.
Dr. Sharp: That’s fair. No, that’s enough. I was just curious how you were gauging vocabulary. That’s helpful.
This might be a leap, but I’m just going to jump into it. So when you say that there were no differences in decoding then, that makes me and I think maybe we’ll be getting to this kind of stuff, but that, of course, opens the door for [00:20:00] characteristics of dyslexia and how we assess those characteristics in English language learners. But at least with decoding, you said that there were no differences between them.
Dr. Geva: No differences.
Dr. Sharp: Oh, okay.
Dr. Geva: So, there’s this one paper, I think one of the first papers that came out of the project was published in 2000. And we actually show profiles where we compare the profiles of monolinguals and second language learners. And remember, those kids are all sitting in the same classrooms. In a given classroom, we may have, let’s say, 10 monolingual kids and 10 ELL kids and we’ll give them the same measures. So that’s the way to think about this. It’s not as though we go to one school where all the ELLs are sitting together, and in another school, at least in the Canadian context, that does not happen that much, though. I believe in some places in the states, it does happen where France, when around the border with Mexico, where actually most people [00:21:00] speak Spanish, but I believe that a similar result from those places as well. Yeah.
Dr. Sharp: Sure. Do you know if that holds true? Have you looked at some of those other variables involved in dyslexia? I mean, so there’s decoding, but then, of course, reading fluency.
Dr. Geva: Right. So, first of all, if you look at reading fluency, we actually looked at that as well. What we see with reading fluency is, first of all, when we talk about reading fluency, I find that it’s useful to distinguish between reading fluency of words versus reading fluency of connected discourse. In other words, if you just get a list of mixed-up words with no meaning, just each word not related to the word that proceeded or the word that comes after it. So there’s no ability to use linguistic grammatical structures to help you know what the next word is going to [00:22:00] be, right? So that’s one way we assess fluency.
And the other one, most tests actually provide information about both. The other one would be the reading fluency of connected texts. So now, here’s a story, read it as fast as you can. Let’s see how many errors you’re making along the way and how much time it took you to read a particular text. These are the typical ways we assess fluency which are probably true in general.
Dr. Sharp: Right.
Dr. Geva: So what we find with the reading fluency is that fluency reading unrelated words, again, it’s very similar from the monolinguals and ELLs, but when it comes to reading in context, so like, just read that story as fast as you can, then we find as expected, the ELLs are less fluent than the monolinguals. And if you think about it in terms of assessment procedures, I suppose [00:23:00] that you say, okay, you have 40 minutes for this test. Well, one can ask, should the ELs get a few more minutes because on average there’ll be reading less fluently because they’re second language learners.
There are all sorts of questions that can be asked. And I would say it even has implications for how we assess for entry to university, et cetera, right?
Dr. Sharp: Yeah. That’s a great point. So you have, do you have data on long-term development?
Dr. Geva: I don’t have it up to the university standards, but there are some other studies that are out there. And definitely, you would expect that if you… I think we kind of drift away from children, but there are studies that show that they will be less fluent than the monolinguals. And then we can ask questions such as should one use the exact same administration procedures for first and second language learners?
Dr. Sharp: Yes.
[00:24:00] Dr. Geva: What we’re doing here is we are comparing the monolinguals with the bilingual or the multilingual, but at the same time, we have to remember that if you look within the bilingual group, there’ll be individual differences, right? And from an assessment perspective, that’s one of the things of interest.So this is a kid who may be reading not as fast as a monolingual kid with the exact same cognitive profile, but they are very fluent in comparison to all the other people in the cohort who are less fluent, who are also second language learners. So it raises questions about the standards and the norms that we’re using.
Dr. Sharp: Sure. Well, that raises a question that I know we were thinking about talking about. So this concept, and this is not exactly the same, but maybe we’re in the same ballpark, hopefully. So the idea of “cultural bias” in our tests, how do you think about [00:25:00] that concept in the context of assessing English language learners?
Dr. Geva: In my work, I have not looked at cultural bias. So the cultural bias, what do we mean by it? We mean by that, I know there are some studies I remember reading when I was a graduate at graduate school where they compared the ability of American undergraduate students to understand a description of an Indian wedding. There are these studies with this comparison to the ability to understand all the customs around Indian weddings versus typical North American weddings. And so they show that there is a difference, right? Yes, so there should be a difference. And if I don’t have any ways of… and I don’t understand what’s the meaning of the custom, and I don’t know how to interpret it, and I don’t know whether the story is about how somebody broke the rules and did not obey what their elders [00:26:00] told them that they should be doing, then I may not understand the story, right? So, definitely cultural background is important.
Bransford, when I was in graduate school, was doing some amazing studies on that. Does it matter? Yes, it does matter. I haven’t been looking at that. It definitely is a relevant thing to consider. But what I’m talking about more is or I’m more engaged in because of my research is, can we use for instance the same, let’s say that you do a GRE and we’re looking at vocabulary, by definition, I would expect somebody who comes from a home and they don’t speak English on average to have a lower vocabulary than a kid with every everything else being equal, right?
[00:27:00] So parental education, exposure to education, everything in the context is the same. I would still expect them to perhaps on average to be less fluent or maybe know fewer words in English, and what will be those words which will make you get into Harvard and not, it will probably be words which are very rare, right? So maybe if I am an English language learner, I may not be familiar with those words. So, then the question is, do we use the same norms or not? And those people in ETS who are studying that topic.Dr: Sharp: Right. Where are we landing at this point with that question? Are we using the same norms or not?
Dr. Geva: I don’t actually know about university students. I am not aware that we are, but I really am not sure about that. I know there’s a discussion about that.
Dr. Sharp: Sure.
Dr. Geva: So the question really here is about, [00:28:00] I know what the current practice is, but the question is, who are you comparing the child to or the student, in the same case when somebody is applying to university? Are you comparing to all the other kids who are like you in terms of their first language background or you’re comparing them to all the kids who are trying to get into university in a given year?
Dr. Sharp: That’s a great question. Well, extending it downward to elementary school or maybe middle school-aged kids too, I wonder if that, or how that might influence say selection for gifted and talented programs depending on what measures they’re using to screen.
Dr. Geva: Yeah, and once you get into talented, again, here, I could just share with you more clinical observations. I haven’t done research on this, but we also have to be aware of cultural [00:29:00] differences that parents from certain cultures may send their kids to private school. So they learn all the possible words in English, and they pass the GRE. So they pay money to practice. And there are companies that make a lot of money out of that. And if you invest in that, then your child, even though you speak at home in another language, may learn all these esoteric words because somebody enabled you to learn them. Is this fair? I don’t know.
Dr. Sharp: Right. That’s a big question. I don’t know if we can answer that question, is it fair or not? Yeah, but it’s definitely happening. That’s true.
Dr. Geva: I can say that a few times, and that’s my experience in the Metro Toronto area that parents from certain cultures come knocking on our door saying, “Can you train my kid? I would [00:30:00] like for my kid to be recognized as gifted. Can you train him to take the IQ test?” And I say, “Sorry, can’t do that.” But it has happened to me and to my colleague.
Dr. Sharp: I believe it. Yeah, it does raise that question. Again, I don’t think we can answer that question, but if it’s not fair, what do you do about it? It seems clear that it is not fair exactly for using the same.
Dr. Geva: It says if you have the money. So it’s not about ability, it’s about whether you have the money. So, we’re talking about gifted but those issues are also relevant when we talk about kids who have learning disabilities, for instance. So how do we assess and there are all sorts of biases built into the system across the board? One can talk about it from that perspective.
Dr. Sharp: Yeah, can you speak to some of those?
Dr. Geva: Yeah. [00:31:00] There’s a variety of topics that one can talk about. We can start with something like parental attributions. We have done some studies and others have done studies where Francis compared the attributions that parents make, I’m talking about immigrant parents here, the occupations that they make about why somebody has difficulties in reading, for instance.
We’ve done a few studies. One of the studies that was particularly interesting to me at the time was a study where we compared immigrants from Iran to Canada with similar middle-class parents in Iran. And we asked them questions about if a child is not doing well at school, they have difficulties with reading, what do you think might be the reasons? [00:32:00] And how do you know that the kid is having difficulties?
The parents in Iran were middle-class parents so they’re educated. They said the clue that the kid is having problems is that they have lower grades, and the reason is that they’re not trying hard enough. And that was very simple. Whereas counterparts in Canada who we interviewed have begun to acculturate.
So, they started to have conceptions of, for instance, dyslexia, or emotional reasons why somebody is not doing so well at school. I remember some even mentioned some problems and the family, for instance, there are some parents who are not getting along, they’re fighting a lot, et cetera. So they started to also bring in some social-emotional issues why a kid is not doing well.
So what we saw is a general shift in terms of attributions for [00:33:00] just work harder, sit harder, sleep less, just work, study more, versus starting to see some individual differences which can then be addressed. So if I have a kid who has difficulty decoding, maybe I can take a tutor or I can send the kid to a private school where they know how to work with kids like my kid.
Dr. Sharp: That’s interesting.
Dr. Geva: So they’re starting to have a broader and more and more nuanced interpretation of what could be the reasons and that leads to what can be done about it. Can I do something about it?
Dr. Sharp: Right. I would guess then that makes them more open to intervention as well.
Dr. Geva: First of all, they have to accept the diagnosis of some kind of assessment and then be open to intervention. Exactly that.
Dr. Sharp: Right. I love that. As we were talking before we started to record, you brought up [00:34:00] this idea of the difference between testing and assessment. That caught my attention. I wonder if we could talk about that a little bit, and then we can start to dive into some of the specific ways that you assess differently.
Let’s take a quick break to hear from our featured partner.
The SPECTRA Indices of Psychopathology provides a hierarchical-dimensional look at adult psychopathology. Decades of research into psychiatric disorders have shown that most diagnoses can be integrated into a few broad dimensions. The SPECTRA measures 12 clinically important constructs of depression, anxiety, social anxiety, post-traumatic stress, alcohol problems, severe aggression, antisocial behavior, drug problems, psychosis, paranoid ideation, manic activation, and grandiose ideation, and organizes them into the three higher-order psychopathology spectra of Internalizing, Externalizing, and Reality-Impairing. These scores provide a quick assessment of the overall burden of an individual psychiatric illness also known as the P factor.
The SPECTRA is available for paper and pencil assessment or administration and scoring via PARiConnect- PAR’s online assessment platform. Learn more at parinc.com\spectra. And for a limited time, get one free administration and score report for the SPECTRA on PARiConnect by calling PAR at 855-856-4266 and mention promo code S-P-E-C.
All right, let’s get back to the podcast.
Dr. Geva: I’m a clinical [00:36:00] psychologist. I’m a professor in the Department of Applied Psychology and Child mental Psychology. We are accredited and there are all these courses we teach, right? So one of the courses it’s kind of the holy grail is how to assess intelligence.
So, lots of effort goes into what is intelligence? What are the different kinds of intelligence? How do we assess it? A lot of focus is on the scoring of tests. So the students are very stressed, of course, for our students to spend many sleepless nights memorizing the criteria, whether an item is a 0, 1, or 2, and how to interpret et cetera, which is why we’re accredited. We do a good job of training students to administer IQ tests. So that to me is on the testing side.
[00:37:00] What we want is that when you administer the test or I administer tests, we should arrive at the same conclusion. So we should apply exactly the same criteria and we should be sensitive to the same issues. So that’s to me on the testing side. And it’s, of course, very important.However, I would argue to be a good clinician, in this case, we’re talking about somebody who deals with learning disabilities as well. I need to know that you are somebody who understands the broader context, the conditions under which one can look for additional information or modify the administration procedures in ways that will give us more reliable and valid information. I would like to emphasize valid information about [00:38:00] what this child’s strengths are, what are their weaknesses? So, we could take some extremes, a kid who comes from, I’m thinking of… I’d like to give an example.
Dr. Sharp: Sure.
Dr. Geva: I think just recently, I’ll let me just backtrack a little bit. So I’m actually teaching a course called Assessment Intervention with Culturally Linguistically Diverse Children. This course actually ended up being the book that you referred to that I published with my colleague Judith Weiner. Each of the students in that course, and these are doctoral students at that point in the clinical program, have to take the course and each of them has to work with a client who comes from some kind of a linguistic and, or culturally different background.
So this is not going to be your English Monolinguals. [00:39:00] It could be a kid who is in a French Immersion program. These tend to be the more privileged kids, or it could be a kid whose parents have just arrived from Syria as refugees. It could be a kid on the border between Mexico and the US. It could be a kid who goes to a private school which is bilingual or trilingual. Any of those contexts.
Dr. Sharp: Okay.
Dr. Geva: And so, those cases in our insights led us to the book that you referred to earlier. So let’s think now of a client as an example, and that’s not the client that made it to the book. I’m thinking of a client whose parents came as refugees from Syria about 3 years ago, maybe 4 years by now, something like that. In this [00:40:00] case, the context is very interesting. We actually have a project where we are following the kids who have come from Syria to see how they are developing in various domains. And one of the testers who herself is coming from an Arabic-speaking country and who is a graduate student said, this kid has difficulties. This kid is not just a typical kid from Syria. He has difficulties. Please, can you take him to the course and assess him? So we did the assessment.
The interesting thing is, first of all, we can look at it from a systemic perspective. So here’s a family from Syria. We all immediately think of PTSD. We think of trauma. It makes sense, of course. And here’s the kid who’s already been in the system for 3 to 4 years, absolutely has zero phonological awareness. He cannot do anything. Games nothing. He [00:41:00] cannot do it. He’s not learning English.
There are some behavioral issues that we observe. And so we start to assess them. So in this case, we give him the WISC? We actually ended up giving him the WISC, but can we just use standard procedures that we overtrained our students to score? Are the norms relevant? No. Would you try to give the instructions in Arabic if you could just to make sure he understands the instructions? Yes. Well, but then you’re breaking the rules, right?
Dr. Sharp: Right.
Dr. Geva: What if you try to teach him to do something like any of the verbal or [00:42:00] verbal tasks? How would you try to teach him and see if he learns as a result of teaching? So this is where we start to bring in some dynamic assessment approaches to see because we are aware that we cannot just use the norms published for the WISC. It’s not relevant for him. So we try to give it instructions in Arabic. We try to do a test semester to see what it is and the sort of intervention is getting better. And so the picture we’re then gathering is much more complex. It’s not just about his Intel IQ as measured by the WISC but rather, what are his problem-solving skills? Is he learning when we teach him and so on and so forth?
So, this is an example where we break out the rules. And so now we break to an assessment because we really found out whether he has the ability to learn to problem solve is a better nonverbal domain, but just using the [00:43:00] norms, he would be flattened everything. But in fact, it’s not flat when you start to use a flat low, I mean, but when you start to look at some alternative approaches where you see if you teach him, he can actually learn and more so in the non-verbal domain.
Dr. Sharp: Sure. I like that distinction. So testing for you is more just black and white rote administration by the rules, scoring by the rules as we were taught, as the manual states. Assessment on the other hand is maybe more holistic, fluid, nuanced.
Dr. Geva: More nuanced, holistic. Let’s look at how a learner, assuming that he’s come from that same background, grew up in a refugee camp so maybe he didn’t go to school. What if we teach him, can he learn? As opposed to we teach them, he can’t learn HIPAA conduct or not for now.
[00:44:00] Dr. Sharp: Right. That’s such an important distinction. It raises so many questions for me. Do you have a definition of dynamic assessment? You used that term just a minute ago. What do you mean by dynamic assessment exactly?Dr. Geva: I actually did not make the term. Other people like Vygotsky and others have been using it. Basically, it has to do with their ability to observe as you try to teach a child a skill to see how they learn. How many repetitions, how much additional information you have to provide, and does the child learn when you do that kind of teaching?
Dr. Sharp: Right, I like that.
Dr. Geva: Yeah, that’s a very simplistic definition.
Dr. Sharp: That’s all right. [00:45:00] I was just curious. That stuck out as you were describing this case.
Dr. Geva: If you look at the kid like the one I just described, he didn’t go to school while they were refugees in Jordan before they came to Canada.
If he didn’t go to school then maybe he didn’t have the opportunity to learn. And not to mention the fact that he would be in an Arabic speaking environment and not in an English speaking environment. So he also has to learn English, the cultural differences, maybe he has never been tested before in a formalized way. So there are all these things that we need to consider, not just administering the test, right?
Dr. Sharp: Of course.
Dr. Geva: And in his case, when we gathered all this information, actually we came to the conclusion that he definitely has serious problems in the language domain. [00:46:00] In fact, we ended up diagnosing also with language impairment, well, now it’s called developmental language disorder, DLD. But in the non-verbal domain, he had some strengths. Will he ever get a Ph.D. in psychology? I don’t think so. Will he be able to be a combat mechanic? I think so. So then that kind of gives us ideas about what is his potential to learn.
Dr. Sharp: Right. That reminds me of a case that I saw fairly recently for a kid, not an English language learner necessarily, but a kid who was truly homeless and just completely off the grid until late elementary school. He finally was discovered by the authorities and had never been to school, had never been in any sort of [00:47:00] formal educational environment. And they wanted an assessment just to get a baseline of what his brain was up to. And there were a lot of parallels to these concerns you’re describing.
Dr. Geva: That reminds me of, there was some very serious affair at night studies that were done on homeless kids, street peddlers in Brazil, I think in a similar kind of thing. So you have these kids who don’t go to school, but some of them really understand math much better than the middle-class kids in Colorado and Ontario. They really understand because they have to make money, right?
So they have a much better idea of how much money they’ve made, how many packages of cigarettes or whatever it is they are selling on the street, they have to sell to actually make some money they can get home much better than that. And again, you would expect the individual differences, so they’re able to learn, right?
And some of them are better at learning [00:48:00] those principles, the math and the other ones who make money. And actually, one of the studies shows that the kids who came from middle-class families didn’t necessarily do better on those measures. So they know the procedures, but they do not have this deep understanding of what it means to handle money and to multiply so many packages of cigarettes by some currencies.
Dr. Sharp: Right. Real-world experience goes a long way.
Dr. Geva: Yes. So the kids are intelligent, right? I’ve said they may not do well on a standardized test. Again, with this distinction between testing and assessment, if you give them a standardized measure of math paper, pencil, they will not do well on that. The smart ones will be able to solve those more sophisticated problems.
Dr. Sharp: Yes, this is probably an obvious point, but I just want to highlight that I think a central theme of our discussion here today is just that test scores do not [00:49:00] necessarily equal intelligence, especially in these kiddos.
Dr. Geva: Yes. It’s exactly what I’m trying to say. And to me, it goes back also to dynamic assessment, which is really what you are to see what your potential to learn.
Dr. Sharp: Yes, so that brings me back. I have so many questions about dynamic assessment. The first question, you do, it sounds like a lot of teaching and training.
So the first question is, how does one teach this? This seems like a very difficult set of skills to master, to know when to break standardization, to know how to break standardization depending on the kid that you’re working with. I mean, how do you even begin to develop that skill of dynamic assessment?
Dr. Geva: One way to think about it is in terms of how many dimensions you’re testing all at the same time. So let’s take something like, let’s take the block design non-verbal [00:50:00] test. What are the skills that are needed in order to solve a block design problem? What are the skills needed as you get better at something?
So usually the way we assess it, we say, okay, how long did it take you to match the pattern, or were you able to match the pattern and how long it took you is basically what we were looking at. But as the items get harder, we get more dimensions, right?
Dr. Sharp: Yes.
Dr. Geva: What is if we take the easiest item and we’d say to the kid, look, this block you see on one side is red, and the other side is white. Can you show me another one like that? How many of those can you show me? So you start from the dept and just started with a simpler one. And talk about what is it that you’re comparing, get them to say that, get them to mimic you.
Now let’s try another one. [00:51:00] Can they now do it on their own? One kid will be able to do it. Another one will need maybe two more trials.
Dr. Sharp: I see. Extending the teaching.
Dr. Geva: It’s teaching. Do you learn when I teach you?
A group in Israel actually, so it started with Feuerstein who was a descendant of people like Vygotsky and some of his students who have developed that and they develop something called the learning potential where they actually start to try to standardize those procedures to be able to kind of say, this kid is learning but it takes them so long before they can learn, et cetera.
[00:52:00] Dr. Sharp: I see.Dr. Geva: Ezuirel is the name of the guy who I believe has written quite a bit about this but kind of creates some standards for how you assess a dynamic assessment.
Dr. Sharp: That’s fantastic. I’ll put that in the show notes and try to find some articles to link to. That sounds great. So then are you building this into the course that you teach? Is this just a matter of procedure?
Dr. Geva: Yes. Well, we do, but we send them to some examples online and to some articles and we talk about it. And when they come back and talk about the cases, they have to describe what procedures they used. And so then the whole group discusses them to see whether there was evidence of [00:53:00] growth or not. But again, you don’t have any standardized norm for that. So it’s more like the observation of can the child… to me, the essence is can the child learn when you teach them?
Dr. Sharp: Yeah. That makes sense to me because that indicates potential. Of course.
Dr. Geva: Yes.
Dr. Sharp: So then I do have a question about that. How do you document these results? Do you even score the measures?
Dr. Geva: We describe them. And it kind of goes back to something you asked me earlier, and it’s sort of how you work with the system. So, maybe I’ll use it as a way of trying to address your question, which is a very important one. I’m thinking of one case [00:54:00] that we had a few years ago of a girl who was born in a refugee camp in an Asian country. And she came to Canada with your family about 2 or 3 years before we were approached at the clinic at the University of Toronto, it’s OISE where I work.
And the social worker kept calling me to say, can you please assess her because we want to put her in a program for kids who are intellectually handicapped. So everybody knew that she’s an intellectual handicap, but the school board needed a number. They needed an IQ number so that she would qualify for the special program, which I’m assuming is the case in many places.
So they actually needed the number. If it’s above [00:55:00] 85, then she doesn’t qualify. She was below 85 IQ, she qualifies depending on how severe the case is. Again, as part of that course, I had a student who worked with her, observed her at school. We tried to teach her how to take the subway from his school to our clinic. She could not learn that. So her brother always had to come and bring her over. A lot of dynamic assessment procedures.
And so now I’m going back to your question. So the report described all the alternative ways that we tried to assess her, observing class, teaching her, using dynamic assessment principles, and how really, she was clearly a kid with intellectual disabilities. And so the report included examples from all these observations of all the different ways we try to assess whether she can learn, things such as…
Another example was, [00:56:00] very close to our department is on campus. There is a survey in a donut and coffee place. The student would take her to see if she could remember how to get there. So every time they met, she tried to teach her to go to that cafe on her own, of course, she would accompany her and see what else she could order. And it was always the same thing. It would be doughnut and chocolate milk and whether she could handle change. So she would always give her a $10 bill to see if she could handle the change. And there was no improvement. She just couldn’t learn.
So that became part of the report. We described how with all these attempts to teach her some small steps in terms of dealing with the spatial information of making it from university to that coffee shop, it was constantly kind of like Dunkin donuts, but a [00:57:00] Canadian version.
Whether she can handle it, it was just not getting better. So all that went into the report actually.
We sent the report to the school, and literally within an hour of sending the report, I get an email saying, can you please give us numbers? So, now we’re back to this testing versus assessment because the policy says we have to establish below the certain cutoff. And I thought we spent so much time on this girl because it was also a learning experience for the doctoral students. I got back to her and I wrote, we’re telling you that she has an intellectual disability. And if you don’t accept that, I’m going to go with that to the human rights commission. [00:58:00] So then they backed off.
Dr. Sharp: Oh goodness. Okay. So you were able to get around that.
Dr. Geva: I was. I just said, look, there’s so much time that we spent on her, but we cannot give her an IQ test. It’s inappropriate to give an IQ test and document so many different ways that this girl, unfortunately, has an intellectual disability.
And by the way, this is part of what we also do. This girl with the same biological parents had a brother who was about to finish high school in Toronto. So, what you get is something we didn’t talk about yet, which is, what is the reference group? So in this case, with this girl, we don’t have a control group, but we have a brother who has the same parents, the same history of being born when in a refugee camp, came to Canada, all came together and he was finishing high school. So [00:59:00] this is a good comparison group.
Dr. Sharp: Right. Gosh, as you talk through this, it makes me realize how flexible we have to be and how adaptive we have to be as clinicians. You think about a reference group of one, but that’s what we’ve got. That’s the closest comparison.
Dr. Geva: Yes. And by the way, the other kid that I talked about earlier, the kid from Syria that we talked about earlier, this kid has 4 siblings. 2 are in high school and they’re doing extremely well in high school. They intend to become doctors. And they probably will.
So again, it’s all this very important thing. What is the relevant comparison group? And in this case, it’s good if you have other siblings. So then, you know, some of them are [01:00:00] doing well, don’t have intellectual disability, don’t have a learning disability, and they are doing well.
Dr. Sharp: Are siblings typically the reference group that you’re using in these cases?
Dr. Geva: We often use a reference group. We use what we can. With a Syrian kid, as an example, because there was this project that’s going on now in Canada that follows the Syrian refugees that Canada took their kids to see how they’re doing and they’re in various domains. So, my colleagues have developed assessment measures that they give to those children. And we can actually compare how our client is doing in comparison to other Syrian refugees on some of the measures. So they are not standardized, but they have good measurement qualities.
So looking at the ability of vocabulary in Arabic, their ability to read [01:01:00] words in Arabic, how they’re learning, to read words in English, how the vocabulary develops in English and so on and so forth, various language and literacy measures. Whenever I can, I do that. I compare the client to the others. So we give them the same measures that we print some and use in our research, and we say, we know how these kids who all came from Punjab background and he was a kid from Punjab who we have as a client. How is this child doing a comparison to other kids from a similar background?
So you have to be open-minded and creative in thinking about what will be a reasonable comparison group.
Dr. Sharp: Sure. That’s the theme here. It seems like it’s just being open-minded and willing to adapt to this work that we’re doing.
Dr. Geva: Yes.
Dr. Sharp: So just going back, I have a specific question with [01:02:00] the little girl who you said you pushed for intellectual disability. In a case like that, do you even attempt a standardized intelligence measure just to say you tried?
Dr. Geva: When we can, we try. We don’t have anything in principle against that. So the two principles here, I think, one is to see whether they can do the standardized procedure. What if you break the rules of the standardized procedures, does that help? That’s good. If it helps, it means that there’s potential for learning, or if it does not help even when you simplified the procedures, we give opportunities to explain the instructions more, to give maybe another example before you start to do the tests. So does that help? So all that becomes relevant.
Dr. Sharp: Right. The way I’m thinking of it in my [01:03:00] mind, you can tell me if this is accurate or not, as it’s like a stepwise model. I mean, you try to administer the test by the book black and white rote administration, that doesn’t work. You do a little more teaching, you provide more support, test limits, extend the time, whatever you might need to do. And then if that doesn’t work, it sounds like at least in this case, y’all move to almost like more of an ecological assessment model where you’re taking her out in the world to see if she can perform real-world tasks. Is that a fair way to conceptualize?
Dr. Geva: I like that. That’s a good scale to think about. Yes.
Dr. Sharp: This is how my brain works. I must organize this information.
Dr. Geva: That’s a good way of describing it.
Dr. Sharp: Sure.
Dr. Geva: I agree.
Dr. Sharp: Yeah. So I wanted to ask as well, let’s see something that you said just [01:04:00] a bit ago that I wanted to follow up on a little bit with this whole principle of dynamic assessment. Does your report then end up being largely, it sounds like what we typically will call behavioral observations in a typical set, does it end up being almost just a long behavioral observations report or is there more standardization to the writeup or what?
Dr. Geva: One of the things that we do is we, actually I don’t have it in front of me here, but actually the same colleague with whom I wrote the book, she has developed an alternative to writing reports that’s more meaningful and accessible, [01:05:00] and that paper that you published a partly, she still has a lot of hits on it on the report writer. It’s called The Report Writer, and it’s organized in terms of questions.
Dr. Sharp: That sounds fascinating.
Dr. Geva: Yeah. We try to keep it, here’s the question, here is what we saw and what the conclusions are, and then we move to the next question. One of the tendencies of our students is to write very long because they document everything. So we try to keep it a bit more succinct to the extent possible and to use appendices for the obsessive-compulsive psychologist at the board who needs to see full data, they can have additional information that’s there.
One thing that we are doing which I think is important to [01:06:00] mention is we also encourage… Well, we get our doctoral students to write a report that’s written to the child, the client. And so it’s sort of a one pager. Here’s what I’m good at. Here’s what I’m poor in. With this Syrian kid we just talked about earlier, actually, most of the information he used was very clever visuals because he cannot read.
And it’s sort of a way for him also to be able to say, this is hard for me. Here’s what I need for you so I can do better. It’s hard for me need to read instructions, but I do well when you read them to me, for instance. So, it’s a self-advocacy kind of a report that we give the clients themselves for them to be able to speak for themselves when they need something.
Dr. Sharp: I love that.
[01:07:00] Dr. Geva: Yeah. So that’s also something that the students do, but definitely, to go back to what you asked in some of those reports, they have both the description, which is along the lines of what you described as well as there’ll be a table with the WISC and here are the scores on working memory and fluid intelligence and so on and so forth.Dr. Sharp: Right.
Dr. Geva: Yeah, it’s there, but it’s supported by observations that were made on the basis of dynamic assessment.
Dr. Sharp: Of course. Great. Well, as we start to wrap up here and end our conversation, are there any other principles or practices of dynamic assessment that we haven’t talked about that you want to make sure of and highlight?
Dr. Geva: So in terms of the report, here are some of the items that we mentioned [01:08:00] in guiding the report writing. There are all sorts of questions that one can ask, should I report on how I modified the tests? That’s one thing. So if you modify the test administration, how did you do that? Or if you conducted a dynamic assessment. Well, another question that we ask our students to consider as they write a report is should I provide qualifications as to why the results should be interpreted with caution? So for instance, here’s what we found, but given these and these reasons, the results may not be reliable.
Another one is, should I report standardized norms? I gave an example of where I refused to provide this and it does have norms but if the board does [01:09:00] require that, then we provide them usually. But we include the warning that this one may not be reliable and valid.
Dr. Sharp: I see.
Dr. Geva: Another one is, should I report descriptive observations? Again, we say, yes, you should provide this part of the dynamic assessment.
Another question that we sometimes are asked and it’s all kind of revolving around the same issue, should they avoid reporting specific scores because I believe that they may be misinterpreted by the school board, right? So that sometimes at the board said, we absolutely need to have the IQ scores and you want to help the child to get the right program that they need given their ability. Sometimes we have to give the scores, but we always then include [01:10:00] some provisors about why those should be interpreted with caution.
So, these are some of the things that we guide our students to do.
Dr. Sharp: That’s great. Well, I just want to say, I know that we have barely scratched the surface with this topic. My hope is that folks are really starting to think about how this idea of dynamic assessment might come into practice and when we might consider an approach like that and how we might do it. I will, of course, have a link to your book in the show notes because it has a ton of information in there about how to do this and many other things.
Dr. Geva: I thought… sorry, go ahead.
Dr. Sharp: No, no, no, [01:11:00] that’s all you.
Dr. Geva: I thought that there might be one more distinction that I like to highlight, which is related to what we talked about, and that’s the distinction between what I call overdiagnosis and underdiagnosis. I don’t think we’ve touched on that today. I think that that’s something, again, the psychologist needs to be aware of.
Overdiagnosis is the example that Jim Cummins talked about, 40-50 years ago, which is too many kids of immigrants who get diagnosed as having intellectual disability. So there’s an overdiagnosis. It couldn’t be so many of them are. I should say at the same time, under-diagnosis and we can talk about it at a systemic level is, and giving an example in Toronto, in particular, a few years ago, and by the [01:12:00] way, the Toronto District school board is I believe it’s the fourth largest in North America. It’s a very large school board. It’s the largest in Canada, but one of the fourth largest in North America.
A few years ago, they were running through various numbers and they realized… So until now, we talked about kids with problems, but what about gifted kids? So, to the other side of the distribution, right?
So, it turned out they did not have any kids who came from African or Caribbean backgrounds who were identified as gifted in the whole school board. So how could it be in a school board that has thousands and thousands of kids who come from that background but nobody is gifted. That’s not possible.
So then you can start to ask some systemic questions such as, [01:13:00] what’s happening? How come the parents from certain groups make sure that the kids get diagnosed, try the best for the kids but other parents don’t know that they should be doing that. So as a result, one of the procedures that’s happening now in the Toronto District School Board is that, in grades 3, all teachers are required to say, I think that this kid and that kid might be gifted. They should be assessed by the system.
So you see a systemic approach to try to correct a clear bias. I suspect that that will be true for the US as well. I don’t know if there’s any work on that, but somewhere there should be.
Dr. Sharp: Sure. I can only speak to some of our local districts, but I know that they are I think doing a pretty comprehensive [01:14:00] job of not only trying to identify English language learners who may be gifted but also assessing those kids as much as possible in their native languages and utilizing measures that are appropriate.
Dr. Geva: Yeah, exactly, because what we’d expect is an in the general population, whether you, your family came as refugees or they are African-American or come from some European descendant, we should see the same percentage of kids who, if they actually have a learning disability in one side, or if they’re gifted, that they should be picked up and receive the appropriate programming. We should expect the same percentages.
Dr. Sharp: Right. And that just relies on the idea that this is normally distributed.
Dr. Geva: [01:15:00] That’s right. Set of cognitive skills, linguistic skills, and that we should find more or less the same distribution. So if it’s tilted in one way or another, from a systemic perspective, we need to ask why. Is this something we’re not doing right? And of course, it has implications, of course, also for what we teach our students when they learn to become clinicians, for instance, or speech and language pathologists, right?
Dr. Sharp: Right. That’s such a good point. This is one of those basic statistical principles, but we miss those things sometimes in real life.
Dr. Geva. Yes.
Dr. Sharp: Well, this has been great. I know there’s so much more that we could talk about, but I hope this has provided at least an introduction and gotten folks thinking about how to do this process differently and how to support English language learners a little better. So, thank you.
Dr. Geva: I hope so.
Dr. Sharp: Yeah. Thank [01:16:00] you so much for coming on and having this conversation with me. It was great.
Dr. Geva: Thank you for inviting me.
Dr. Sharp: Okay, y’all, thank you so much for listening. As you can tell that interview was chock-full of information and case studies and stories. And I really enjoyed how Esther brought these cases to life and illustrated these points through case studies.
Now, again, as I mentioned in the beginning, a few points that we hope that you take away from this interview just to clarify and send you away with some really concrete ideas.
1) When you’re assessing second language learners, both English and multilingual kids, you just have to be concerned about both language and cultural differences and the interplay between the two.
2) Language is complex and takes a long time to develop. So interpreting the test norms should be done judiciously.
3) Assessing language and literacy in the first language is always a good idea because you’d be looking for confirmation and consistency across the languages.
4) When interpreting assessment data, definitely consider age of arrival in the immigrant receiving country, exposure to opportunities to acquire the second language, school interruption, language topology, and positive and negative transfer.
5) The report should synthesize various information sources. You really want to know that the report has taken into account the nuances provided by considering complimentary data sources like some of the ones we mentioned in the interview.
So, again, thank you for listening. I hope you are all again, doing well and enjoying some kind of summer, whatever that looks like for you. If you have not subscribed or followed the podcast, I would be thrilled if you would do so. If you have, I would be [01:18:00] thrilled if you told a friend or two about it and continue to spread the word and increase the listenership, always a good thing and just grateful for you all. Thank you. Stay tuned, more clinical and business episodes coming up.
Take care in the meantime.
The information contained in this podcast and on The Testing Psychologists website are intended for informational and educational purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis or treatment. Please note that no doctor-patient relationship is formed here, and [01:19:00] similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with an expertise that fits your needs.