PAR offers the RIAS-2 and RIST-2 Remote to remotely assess or screen clients for intelligence, and in-person e-Stimulus books for these two tests for in-person administration. Learn more at parinc.com.
All right, everyone. Here we are back with another clinical episode. My guest today, Dr. Joel Schneider has been researching cognitive assessment and cognitive theory for many years, and is truly an expert in this area.
As you will hear during the podcast, I find it fascinating that we haven’t really [00:01:00] tackled cognitive theory on the podcast before. So this is a nice introduction and overview of cognitive theory that transitions later in the episode to directions for the future and specifically, Joel’s work around advancing cognitive theory and moving from more of a hierarchical model to an integrated model. You will understand what I mean by that by listening to our conversation.
This episode is action-packed. Joel gets into some of his family history and why he does this work, does a nice survey of cognitive abilities and our understanding of cognition and then we talk about future directions and where we’re headed from this point. We also dive into how we assess cognitive ability [00:02:00] and whether we are doing that well or not. So it’s a good one. It’s a great one as a matter of fact.
Let me tell you a little bit about Joel before we transition to the conversation. He is an associate professor in the College of Education at Temple University and a member of the School Psychology program. He earned his doctorate in Clinical Psychology from Texas A&M. Along with Nancy Mather, Elizabeth Lichtenberger and Nadeen Kaufman, he wrote the Essentials of Assessment Report Writing, 2nd Edition.
He also writes software to help clinicians make better decisions. We do talk about that and Joel’s current project, which I imagine many of you will probably find fascinating and intriguing. His research interests are concerned with validating and improving upon current assessment practices. In particular, he focuses on understanding the structure and function of intelligence and its assessment.
Joel is fantastic. He’s a [00:03:00] great storyteller. And this was a really enjoyable interview for me, both personally and professionally.
Before we jump to the interview, I want to invite any advanced practice owners who are looking for group coaching and accountability to help take your practice to that next level here in 2021 to check out the Advanced Practice Mastermind group, which starts in June. We are two months away and already have two spots filled. So if you’re interested in some group coaching to try to reach some of those goals in your practice, I’d love to chat with you and see if it’s a good fit. You can get more information at thetestingpsychologist.com/advanced.
I don’t want to keep you waiting any longer. Let’s get to my conversation with Dr. Joel Schneider.
[00:04:00] Hey, Joel. Welcome to the podcast.Dr. Joel: Hi, Jeremy. Thank you for having me.
Dr Sharp: Welcome. Thanks for being here. I got to say, we were talking about before we started to record that I have been reading your blog for a long time off and on and really found it back when I was a very early career psychologist. It was one of those mind blowing experiences because there was a lot of information in there that was completely different than what I’d learned in graduate school. It was that widening the crack in the door like, oh, there’s more. Thanks for that experience. It’s great to have you.
Dr. Joel: I’m glad that people are out there reading. I check in on the blog from time to time, look at the statistics and wow, way more people read the blog than read any of my papers. So it’s a wonderful time in which we live, in which we can reach [00:05:00] people so easily.
Dr. Sharp: It’s so true. Thank you for that. I’ll say that right off the bat. It’s at least affected one psychologist out there. That’s for sure.
First of all, just really thrilled to be able to have this conversation. We have not really talked about the foundations of cognitive assessment or understanding of cognition on the podcast yet, which is wild to say that out loud, but I am curious, you could be doing any number of things, why this? Why is this important to you out of the field of psychology?
Dr. Joel: You can do lots of things with your life. I do lots of things with my life. This is not the only thing that I care about. You can see a guitar behind me. I like to write songs and do things like that and hang out with my family, play games, and all of that, but my [00:06:00] professional time is mostly consumed with trying to find new ways of helping clinicians make better decisions. And that includes me.
The primary beneficiary of my research is that I find out that I was wrong about all sorts of things, and I have to recalibrate my intuitions and also have to come clean from time to time about, hey, the thing I was recommending last year, I can’t stand behind. And so it’s been a wonderful opportunity to contribute to the field in this way. I feel quite grateful that I get to do it.
Dr. Sharp: It’s such a good way to put it. I feel like you give yourself a lot of credit. There is a personality component in there that keeps you open to admitting when you were wrong. I don’t know that everyone [00:07:00] possesses that. Is that something that you had to work on, or does that come naturally to you or what?
Dr. Joel: I grew up in a religious tradition that valorized the idea of admitting fault that this is actually the mark of a good person. I’m not a religious person now, but you can’t get rid of that early influence. And so I think it was something that just the subculture and where I found myself, it was considered a good thing to constantly review your choices. The idea of having no regrets, it makes no sense in that subculture. Regrets are the mark of a good person.
With regret comes the capacity to improve oneself. It’s embedded into the scientific [00:08:00] enterprise. It’s an error correction technology if nothing else. There are cynical quotes about scientific advances, one funeral at a time. That’s not entirely true, that people do change their mind.
I’ve had generous mentors who’ve modeled this over and over again. I had the good fortune of working briefly with Alan Kaufman on a few papers and a book project in 18 Kaufman. They embody that spirit of being open to be admitting fault, even when there is high up on the food chain as they are in this world.
Dr. Sharp: I love that.
Dr. Joel: And so if they can do it, little of me can too.
Dr. Sharp: There you go. It’s inspiring when you [00:09:00] run into folks like that, who we idolize, and then to find out they’re quite humble and model those behaviors, that goes a long way.
Dr. Joel: Yeah. You mentioned, why this? So I gave a general explanation, but there’s a personal component too, that I sometimes have shared in various workshops. I don’t get into the details of it, but I come from a very large family. I have 7 sisters and all of them have many children. And so I have 31 nieces and nephews. I’ve lost count of exactly how many great-nieces and nephews, but they’re quite a few at this point.
We don’t think of ourselves as the learning disability family, but there are a lot of learning disabilities in our family, including and [00:10:00] especially my father. I sometimes tell his story of how he got to the 7th grade without being able to read.
Dr. Sharp: Wow.
Dr. Joel: He was able to hide it because he had this steel trap of a memory. He could memorize extraordinary amounts of material. He was famous in our little world of how he could memorize whole short stories that would take 30 minutes to perform and he would perform them word for word. I check to make sure that he was getting, he did. He would recite poetry start to finish.
He worked at it, but it was a talent that he had. So he gets to 7th grade and his history teacher says, you’re not [00:11:00] passing my class and if you don’t, I’m going to hold you back. He’s terrified. And she says, but I’m willing to give up my lunch break to help you get through this.
She very quickly discovered that it wasn’t a history problem that he had. It was a reading problem. And so she taught him to read in 7th grade. He could do a little bit of reading. It wasn’t completely illiterate, but he was not what you would call your reader.
It took a few months and he got going. He started hanging out in the library and there was a book that he caught sight of. It was Oliver Wiswell. It was historical fiction and he started to read it slowly and was so entranced by it that he couldn’t get enough. He had entered into this new world of reading intransient.
He never [00:12:00] thought of himself as the kid who would go to college, but he did. He was always a fabulous reader; the way he read out loud and he was constantly reading new things, but he was a bad decoder. He mispronounced a lot of words.
I remember as a kid, how strange it was for him to ask me, a 10-year-old, how to read words, what’s this word. And then he would sound it out with me and then he would have it for the rest of his life. He benefited from a teacher who took time to get around his learning disability. My grandfather [00:13:00] taught to the 2nd grade and was not a reader. He could do figures but no text, probably had the same problem.
I have close family members other than my dad who also have had learning difficulties of one kind or another.
My mother and my sisters have benefited greatly from speech-language pathologists, from clinical psychologists, from school psychologists who took the time to figure out what was going on.
My mother and my sisters were able to make use of that information and change the trajectory of their lives in such a way that we don’t think of ourselves as a family with lots of learning disabilities. We [00:14:00] think of ourselves as the Schneiders. It was so important that these professionals took those concerns seriously, that it feels like I’m forever in their debt, that whatever I can do to repay, I’m very happy to do it.
Dr. Sharp: What an incredible story. I can tell just from the way that you tell it, it’s obviously stuck with you. This is important. There’s some purpose there which makes our work all the more fulfilling.
Dr. Joel: Yeah. Part of what I do is nerdy mathematics, but in the end, I always try to find a bridge back to real [00:15:00] people. And so that’s been the focus of my career is trying to find the bridges between the psychometrics and the humanity of each individual.
Dr. Sharp: I think that is a fantastic segue to how you actually do that and how we as a field might be able to do that a little bit more. As we enter this discussion, I have to give the disclaimer, which I’ve given on the podcast many times that I am not a psychometrics person nor even a statistics person or a researchy person, that was not my path. And so these conversations are so fascinating to me to be able to speak with you where you’ve lived in that world and that translating for so long.
Dr. Joel: That was more of an accident than anything else. [00:16:00] My doctoral degree is in clinical psychology. My plan was to become Salvador Minuchin without the accent. Family therapy, that’s what I wanted to do. I wanted to be a practitioner.
The idea of doing research in psychometrics was not on the menu of things I was going to do. I had the experience of Marquis Nada as my assessment instructor, the first one. I worked and slaved over that first report. It came back with more red than I care to admit. I couldn’t believe, I thought I was a decent writer and it was just correction after correction.
The second time around, after I got over myself [00:17:00] and took what she had to say to heart, I turned in a second report and it came back with a lot less red, actually a nice note at the top and she said something like, you’ve really got a feel for this. It was something just completely unexpected. It felt like a gift that she would say that to me. I thought, maybe I do. I went from there and wanted to develop that talent.
I still like family therapy. I still like group therapy. I love all of that. I teach group counseling, but the assessment has become the primary professional identity that I’ve tried to develop. It had that little spark of just that one compliment. I don’t know if she probably doesn’t remember it at all, but it was very important to me.
Dr. Sharp: It’s amazing how those little moments stick with you. [00:18:00] You hear those stories all the time. That’s really cool to know that you had that experience. Thank goodness. It seems like you found your niche.
My working title for this episode, I know listeners will get the final version, but right now it’s still a working title, my working title was the evolution of cognitive assessment. I’m excited to dive into this. I think you used that phrase maybe while we were discussing topics for the podcast. I’m curious what we might mean by that, the evolution of cognitive assessment or understanding of cognition, I suppose. It sounds like you’ve been really working around this conceptualization or maybe reconceptualization of how we think about cognition. Is that right? And if so, can you say more about [00:19:00] that?
Dr. Joel: Sure. The history is really complicated and long, and it’s a fully human story. When I talk about it with students, I always talk about the fights and bars that I imagined that they were getting into. The history of cognitive assessment has been a contentious one. It was an important controversy.
Dr. Sharp: There you go. I was just going to say, can you give us a CliffsNotes version just to provide context here for what we’re going to dive into, or maybe even a non-CliffsNotes version if it’s important?
Dr. Joel: Sure. There has always been a contrast between the needs of institutions and the needs of individual. The person who invented or coined the term intelligence quotient, William Stern, was horrified about what happened to his little invention that [00:20:00] he spoke quite movingly about how the needs of the individual or what he was focused on, and that it pained him to see what was happening, that institutions were using intelligence tests for their own needs at the expense of the individual.
So famously would be things like denying immigration status to people with low scores or institutionalizing people based on scores or in the worst-case scenarios, it happened in the United States all the way up to the early 70s, sterilizing individuals, not for their benefit, but for someone else’s.
There has also been a large number of people who’ve used intelligence research as a method of bolstering their own self-esteem or esteem of their own [00:21:00] group against other people that they perceive as different. And so there’s been an ugly history there rooted in various kinds of oppression. And we in this field have to acknowledge that the tools that we’ve developed can be used for good and for evil.
And so the tradition that I try to follow is that one started by William Stern, which is that we use these tools solely for the benefit of the individuals that we assess and that we do not participate in any action that services that institution at the expense of an individual.
Dr. Sharp: How far do you extend that [00:22:00] to your daily practice? Can you operationalize that for me a little bit?
Dr. Joel: Sure. There are ways of making problems just go away. I remember the very first case that I assessed. There was a little guy. He was in a shelter for abused and neglected children. When I tried to assess him, he refused.
I used a metronome to make sure that my Digit Span was accurate. It was a visual metronome, not an auditory one. One digit per second, so I would see a little light flashing, but he couldn’t see that I could see. He saw it and he picked it up and smashed it down, broke it. At one point he spat at me quite accurately, [00:23:00] which I wrote in the report as an instance of good muscle coordination.
He was not wanting to be assessed, but I kept showing up and tried to negotiate with him. Eventually, we came to an understanding that I was there to help figure out what was helpful for him. He had a previous diagnosis of intellectual disability. It wasn’t called that then, and there was just no way that diagnosis held. You get a feel for this.
I felt that I owed it to him to persist in the assessment so that we would not get that same diagnosis because I knew that he was far more capable than the previous evaluations who probably had the same [00:24:00] experience that I did. He just didn’t want to be tested. And so he found a way around it.
So we took our time and we got through the assessment. It was nowhere near intellectual disability by the time we finished, but it took six times for me to come back. I was in graduate school, so I had the luxury of spending the time. You can’t do that with every case, but the stakes felt pretty high to me that I thought that there was a miscarriage of justice.
I did talk to the teacher. It’s like intellectual disability, really. I didn’t say it in those words, but she said something to the effect of, I was a desperate woman. He was difficult to have in the class. There was a lot going on.
I don’t think she [00:25:00] went into the process thinking I’m going to get this kid diagnosed with intellectual disability, but it did end up having him removed from that classroom. Is that a reasonable example of what I’m talking about?
Dr. Sharp: It is. When I asked that question, I was thinking of schools in particular as an institution versus the needs of the individual students.
Dr. Joel: Schools do have needs. I’m not anti-institutional. I work in them but there’s a hierarchy of needs and the needs of the individual, I hope outweigh the needs of the institution. So try not to misquote Spock from the Wrath of Khan.
Dr. Sharp: We’ll fact-check that and make sure it’s on point.
Dr. Joel: Maybe he said it slightly differently.
Dr. Sharp: Fair enough. [00:26:00] Just for the sake of context as we move forward in this discussion, can you talk through our understanding of cognition? What is cognition? I want to pull in CHC theories, so how does that come about?
Dr. Joel: CHC theory stands for Cattell-Horn-Carroll theory of cognitive abilities. Raymond Cattell, John Horn, and John Carroll were independent researchers who came up with different kinds of cognitive theories in theories of intelligence that happened to align very nicely, that they fit together in the same way that Protestantism and Catholicism are quite similar.
To insiders, the issue of people authority matters quite a lot but if you’re [00:27:00] loo king at it from an anthropologist from Mars’s perspective, there are a lot of like. And so Cattell and Horn had differences of opinion, although they did work together, and then Caroll had a difference of opinion with them but in broad strokes, the theory was very much alike.
The answer that they came to is the same answer that we’ve had roughly since the early 30s when Thurstone did the first multi-battery factor analysis. The answer that we’re getting is that there’s a hierarchical structure to cognitive ability.
There are different kinds of hierarchies in different interpretations. What that might mean, we certainly are not at the end of our journey here, but [00:28:00] in broad strokes, the structure of intelligence hasn’t changed much since the first time that we did a good factor analysis of these feasibilities. And so the names have changed over time, but the underlying similarities are pretty easy to trace.
So what Kevin McGrew did in the late 90s and early 2000s is he recognized that there’s an underlying similarity to all these theories. Let’s just house them in a broad tent like an ecumenical organization that they haven’t worked out their differences of opinion and their theology is different, but they’re going to work together in some way. And so CHC theory is a broad tent under which very similar theories can co-exist without canceling each other out. We’re not solving the differences between them, but there are a lot to [00:29:00] like.
Basically, there is a structure that goes from very specific to very broad abilities, that there are abilities that are very specific to a time, a place and an activity like learning to play a particular video game or being able to move a mouse accurately. That’s very specific to things that are a little bit broader, things like short-term memory, visual-spatial ability, and then even broader than that, of general reasoning.
So that structure has been more or less unchanged since the 1930s. The list of abilities has gotten bigger, but there have been very few [00:30:00] major course corrections that were needed.
Dr. Sharp: Sure. I forget where we are now in terms of top-level but beneath that, where are we at in terms of number of domains that we’re…
Dr. Joel: As most of your listeners would know that the first factor of ability that was identified by Spearman in that in the first 10 years of the 20th century was general intelligence. it’s commonly abbreviated g. Spearman became less enthusiastic about calling it intelligence because intelligence is a false term. It means whatever speakers of English want it to mean. And so he said, well, let’s just give it a symbol and then we’ll build from there.
So general intelligence was the first clearly identified ability. [00:31:00] And then other theorists came after Spearman, and Spearman did some of this work himself, identifying smaller categories of ability. There are probably around 8-12 of them.
There are some that have a tentative status that it’s not exactly clear if we really need to expand the hierarchy. About 8 of them, it’s pretty clear we do need to clearly distinguish between. Maybe I could do a tour of the major landmarks of what these abilities are?
Dr. Sharp: You’re anticipating my questions, Joel. This is great.
Dr. Joel: John Carroll said something to the effect of, to propose a theory of intelligence is to propose a theory of cognition. [00:32:00] So Kevin McGrew and I have worked hard to propose a theory of cognition. It’s in broad strokes consists of, there’s perception. We take sensory information and we have various sensory organs that make sense out of sensation. And that the brain does this via perception, how that happens I don’t pretend to know, but the sensations converted into perceptions give meaning to our experience.
So most of that happens automatically and outside of our awareness, but there are perceptions that are unexpected, and we have a mechanism for directing conscious attention towards that, which is unexpected. [00:33:00] So anything that doesn’t fit our previous mental categories, our expectations, we tend to focus on and in the spotlight of attention, we have additional capacities to elaborate on what doesn’t fit our expectation so we can play with it, we can model it.
So if I am asked to do some mental math in my head, I direct my focus towards the numbers there. I didn’t know what numbers were going to be there, so I have to see what they are. And then I have to translate them into some meaning system and derive an answer.
My mental math might be good enough that I could just do it automatically without thinking like timetables, but it might be a fairly complicated equation that I have to actually work out in my head, and this mental space we call [00:34:00] working memory. We as humans have an extraordinary ability to play with ideas in the theater of the mind and in a way that few other animals do.
There’s some evidence that chimps are really good at working memory and can in some ways outperform us. And there are certain species of birds that have some memory abilities that are a little bit better than ours, but what we can do with it is that we can play with language symbols and abstraction in a way that other animals can’t.
In this little theater of the mind, we can identify new patterns and create new ideas. We call this fluid reasoning. Its fluid in the sense that it can go anywhere. It is general. It’s not confined to a particular set [00:35:00] of skills or sensory modalities. It is a truly domain-general capacity that we can make sense of almost anything that’s abstract, just really quite wonderful that we can do this.
At some point, we acquire new knowledge and we store it away. Our capacity to get that knowledge back usually works pretty well but many of us have that tip of the tongue experience where you know you know it; you can’t quite pull it out. For some reason, I can never remember what the capital of Australia is, even though I know, I know it.
Dr. Sharp: Now you got me thinking about it.
Dr. Joel: Exactly. Right. It’s a famous example because almost everyone has the feeling that they know it or should know it, but very few of us actually do. [00:36:00] Same thing with the capital of Canada, our neighbors to the North. Like it’s not Montreal, it’s not Toronto. What is it? Even now I know that I know it, but pulling it out right now is a little hard. Canberra is the capital of Australia.
Dr. Sharp: That’s what I was thinking. I really was thinking that. Listeners, I really was. I still don’t have the capital of Canada though.
Dr. Joel: The capital of Canada is one thing that I’ve used this example many dozens of times and every time, I can’t seem to pull it out. It takes me a few seconds or a minute or two, and it’ll come to me later. We have memory failures all the time. Some of us have them really often, so much so that we fill our speech with all-in-ones. [00:37:00] This gets more and more common as we age.
There’s something different about this capacity to retrieve knowledge fluently. That’s different from the contents of knowledge. So you can distinguish between the memories that are stored in terms of what’s there, in terms of its breadth, its depth, its connectedness to other kinds of knowledge and then distinguish between that and how fast can you get that information back and use it.
Talk show hosts and comedians are the Olympic athletes of memory retrieval fluency. Their conversational abilities, they can take ideas from their memory, combine them and start new creative ways. This memory retrieval fluency is the precursor for creativity. It’s a necessary [00:38:00] ingredient. People who can pull memories very quickly from their memory banks and combine them have the capacity to be creative. It doesn’t guarantee creativity, but it’s an important component.
Just a backtrack, we have the speed of perception. That’s a thing. We have the complexity of perception that’s possible. Some people can perceive more complex patterns than others. I remember I assessed a man who had had a brain injury and whatever low-level capacity to detect certain gestalts was gone.
I showed him four circles that to me, very clearly arranged like a square, he didn’t see it that way and just forced four small circles. [00:39:00] Making the corners of a square, he saw four circles, but when he reproduced them, they were not square at all. He just didn’t have that capacity to see certain patterns. We have low-level pattern detectors in the individual domain. They have them in the auditory domain.
Kevin McGrew and I suspect that there’s probably a different sensory ability for every sensory modality. The evidence for that is a little less clear. What I mean by that is we give tests of, let’s say, haptic ability or the ability of touch. So we’ll give people objects while blindfolded and say just touch this and see if you can figure out what it is. A factor [00:40:00] analytically, they almost always load on visual-spatial perception.
People in the blind community will tell you, yeah, I see. I just do it a different way. I visualize. Probably, they’re using the same regions of the brain to do it. They’re just seeing with their hands, they’re visualizing listening to sound patterns and the echolocation, all that kind of stuff. They can do it.
So it’s clear that there’s not a one-to-one correspondence with a sensation and a structure of perceptual abilities, but we do think that there are specialized capacities that are specific to different sensory modalities like edge detection, line estimation, pitch discrimination and things like that.
[00:41:00] Dr. Sharp: Sure.Dr. Joel: Then we have a capacity to play with ideas in short-term memory called working memory. We have the capacity to acquire new information and commit it to long-term memory. We have that capacity to pour back into conscious awareness. We have the capacity to reason and we have knowledge banks and skills.
So those are the main features of CHC theory is the speed of perception, the complexity of perception, the distinction between long-term and short-term memory, the ability to reason, and the abilities that are related to knowledge.
Dr. Sharp: Fair enough. And then we have the ways that we try to measure these abilities, right?
[00:42:00] Dr. Joel: Yeah.Dr. Sharp: So this is a can of worms to open. There’s so much to paint around this. And so this question of do our tests actually measure what we think we’re measuring? Because they purport to measure these abilities that you’ve just discussed.
Let’s take a quick break to hear from our featured partner.
PAR has developed new tools to assist clinicians during the COVID-19 pandemic. The RIAS-2 and RIST-2 are our trusted gold standard tests of intelligence and its major components. For clinicians using tele- assessment, PAR now offers the RIAS-2 Remote, which allows you to remotely assess clients for intelligence, and the RIST-2 Remote, which lets you screen clients remotely for general intelligence.
For those assessing clients in office settings, PAR has developed In-Person e-Stimulus [00:43:00] Books for the RIAS-2 and the RIST-2. These are electronic versions of the original paper stimulus books. They are an equivalent, convenient and more hygienic alternative when administering these tests in-person. Learn more at parinc.com/products/RIAS-2.
Let’s get back to the podcast.
Dr. Joel: I think that there’s a little controversy as to do some of these abilities exist? The controversy is where you identified it as, are they ready for prime time with individuals to make high-stakes decisions? So the factor analyses are pretty clear.
Nobody seriously thinks that there’s only general intelligence and nothing else. Even Spearman, he always said that there are specific abilities. And then eventually he said, yeah, broad abilities [00:44:00] exist. I don’t know anyone who seriously objects to the idea these abilities exist.
When I have friendly debates with my friends and colleagues who say we shouldn’t be measuring anything but general intelligence, what they’re saying is I think that these abilities exist. I just don’t think that we can measure them very well. I don’t feel that the tests are good enough to make high-stakes decisions with.
Dr. Sharp: I have two questions with that. One is the question of, okay, there are a lot of folks out there and maybe the evidence to say that, for example, in our IQ tests, we’ll just say the WISC, that it’s most helpful to take the Full Scale IQ, for example, as having [00:45:00] any predictive capacity rather than breaking it down to the index scores and hanging your hat on those, meaning judge in the real world maybe, is that fair?
Dr. Joel: Yeah, I’ve heard that argument.
Dr. Sharp: Okay. I guess the follow-up to that is, do you agree with that, and if not, why not?
Dr. Joel: The thing that people still get in fights in bars about is the general factor of intelligence. So in my world, the CHC theory, Catell and Horn had an explanation as to why the general factor emerges, but they didn’t think of it as a psychological variable.
They believe that fluid intelligence is a domain-general capacity that can be [00:46:00] applied to anything. It’s general in that sense, but the statistical factor g was just that it emerged for reasons that didn’t correspond to a psychological ability.
Caroll, on the other hand says, g is a thing. It’s an ability. When people do the tasks that we asked them to do, they are using an ability called general intelligence. And that distinction is one that we’re going to be fighting about for a long time.
Dr. Sharp: So you would say it’s undetermined. We don’t know. We’re still figuring it out.
Dr. Joel: The only responsible way to go about is to say, I don’t really know. I can tell you what I suspect. There are many theories as to why the general factor emerges. [00:47:00] I’m not going to go over them all here, but there are some really smart people on all sides of this argument.
We’ve had these kinds of arguments for a long time and they don’t seem to be resolving. That’s usually a sign that our data just isn’t good enough and we’ll solve all of it when the data get better. There may be some breakthrough that I don’t pretend to know what it is that will resolve it, but the controversy will live as long as the arguments are unpersuasive and they are.
When I listen to people talk about why this and why that, I was thinking, that sounds reasonable but I listen to the other side too, and their objections sound reasonable to me. So my stance is [00:48:00] I’m happy to wait and see, which is why I’ve aligned myself with CHC theory. It’s this ecumenical broad tent that says, look, we don’t know how we’re going to keep both these ideas alive as possibilities.
That doesn’t mean that I’m entirely neutral. I suspect that fluid intelligence is mostly what we mean by g and the statistical procedures make them so that they’re almost identical, they’re correlated either perfectly at 1 or down close to that, 0.99, 0.96, 0.97.
So whatever that the controversy is between is it fluid intelligence? Is that the general ability or is it g? We’re talking about a difference that is pretty trivial when it comes down to it. We’re talking about things that are so highly [00:49:00] correlated. I’m not sure if it’s worth all of this fighting about.
Dr. Sharp: That’s fair.
Dr. Joel: I haven’t quite answered your question about what about Full Scale IQ? What is that? I was knocked out by David Wechsler’s thoughts on general intelligence. Alan Kaufman does a funny routine in some of his talks and also in his books about how he invited Wechsler to talk to his graduate students.
He’d been preparing them to talk about here’s all the different small factors of ability and you have to work at it like a detective, it’s indulgent testing system. David Wechsler comes in, I’m not good at telling the story as well as he does, but David Wechsler comes in and says, of course, the most important thing to measure is Full Scale IQ. He’s embarrassed by this because that’s not what he was telling them.
So when you read [00:50:00] what David Wechsler has to say about this and why he thought this, it’s pretty interesting to me, at least. Full Scale IQ to him was a measure of what he called global intelligence. What he meant by that was something explicitly different from what Spearman meant.
Spearman thought of general intelligence as this functional unity, Wechsler didn’t. He thought of it as the sum total of all the things that matter, things that were in his tests, things that he did not put in his tests, and it’s this global capacity to solve problems, and that’s what matters to him.
And so if you think about it that way, the Full Scale IQ does a pretty good job of measuring that global capacity. It’s a little bit of this, it’s a little bit of that, put it all together and it makes a [00:51:00] pretty good prediction as to the kinds of problems you’re likely to be able to solve.
I think the Full Scale IQ as a crude but pretty accurate regression equation. You put a bunch of predictors together, you weigh them equally, and it does a pretty good job of predicting a lot of things. A paper in the 70s, Howard Wainer showed how when you have lots of correlated predictors, the specific weighting scheme that you use is not terribly important.
You might be able to increase your ability to predict a little bit better, narrow the confidence a little bit better if you optimally weigh all of your predictors, but if he is doing an equal weighting, you’re going to do a pretty good job of predicting things.
These Full Scale IQs like on the Wechsler have evolved over time. If you look at what the ingredients [00:52:00] of these Full-Scale IQs are, it’s pretty close to what an optimal regression equation would be. So when you do an optimal regression equation, the most heavily weighted will be the knowledge tests, predicts knowledge pretty well. And then a bit second would be the reasoning tests; predicts reasoning pretty well.
And then the smaller factors of ability have smaller weights, they matter but when you’re predicting those outcomes that you care about, they have smaller weights. Full-Scale IQ has 2, 3 or 4 knowledge tests, 2, 3, 4 reasoning tests and a smaller number of working memory tests and a smaller number of processing speed tests.
So in the latest version of the WISC, you’ve got 2 knowledge tests, 2 reasonings tests, 1 visual-special, 1 working memory, and 1 processing speed. Those numbers [00:53:00] mimic what an optimal regression equation would do.
It’s not surprising that Full-Scale IQ in statistical analysis is a remarkably good predictor of a wide variety of outcomes. Can you improve things a little bit by optimizing and tweaking them for different outcomes? Yeah, you can. You can squeeze out 1,2,3, in some cases, 6-10% extra variance, but usually not that big. You get a little bit extra better prediction if you tweak the numbers in a custom way, but your predictions don’t get that much better than the full-scale IQ.
So this is my nod of whichever the hat to the critics who say Full Scale IQ is the only thing we should measure, is like, yeah, it does a pretty good job. What it doesn’t do for me is it doesn’t give a very good explanation. It doesn’t give a [00:54:00] very good handle on the helpful explanations that I like to give in a report like what’s the underlying problem here? How can I help you understand what’s going on? And so if you see specific deficits, talking about them gives people a handle as to why certain things are difficult to forget.
Dr. Sharp: That makes sense.
Dr. Joel: If I put that specific deficit into the Full Scale IQ, it pulls that Full Scale IQ down a bit by about the amount that should to make a prediction, but it doesn’t help a teacher or a parent understand what the problem is. If you just say, their IQs are a little low. That’s why they’re having trouble. What do I do with [00:55:00] that?
Dr Sharp: Not much. It’s hard to actually drill down.
Dr. Joel: A little. If we’re talking about intellectual disabilities or giftedness, that is a pretty good explanation. A lot of the time say there are global deficits or global strengths and that’s what explains this person’s performance, good or bad. But if they’re having specific problems in specific areas, generally, there’s a specific reason for it, and I’d like to be able to measure it.
Do I think that I can? Yes. I’ve developed a number of what I have rediscovered or repurposed solutions made by others to do this in a responsible way and statistically reasonably sophisticated and responsible way. The responsible [00:56:00] way is to admit uncertainty where it is and make your predictions with the right competence intervals.
I wrote a paper in 2013 called What If We Took Our Models Seriously, which I showed how to make the estimates with the right kinds of confidence intervals. It turns out that a lot of our confidence intervals are much wider than we ever taught in graduate school.
Dr. Sharp: What do you mean?
Dr. Joel: If you’re trying to measure some underlying latent ability, that’s a different question from the reliability of that. We’re talking about the validity of the score. So in terms of reliability, those confidence intervals are pretty narrow. In terms of validity like what’s the underlying thing that we’re trying to measure, we’re not as good at measuring those things.
And so our guess is, you need to [00:57:00] have wider confidence intervals around them. So when I say, what’s the confidence interval around someone’s working memory score, I’m talking about a reliability issue. We can measure those abilities with pretty good reliability, but if a working memory score consists of a bunch of things other than working memory, it has impurities in there.
And if I’m trying to get at what the underlying latent variable is, then I have to widen my confidence a little bit, and there are procedures that help you do this. And so I wrote an art package that does this. I think only I use it but I would like to develop software that implements some of these techniques so that someone doesn’t need to know all the nerdy math to get these [00:58:00] responsible confidence intervals so that when you say this person has a working memory problem, you can specify a valid confidence interval about the latent variable, not just the observed score.
What this will mean in practice is that you should identify some specific ability as a problem only when it’s extreme, when the deficit is really different from the other, it’s quite distinct from the others. Small differences of 10 points are just too small to matter. We’re talking 20, 25 point differences before we start paying attention to them and can reliably say, this is a real deficit and I know where it is.
In general, we need to dial back our creative [00:59:00] psychometric interpretations when we’re talking about small differences of ordinary profiles. When you see something extreme, we can be fairly certain that we know what we’re talking about. Those extreme differences can be responsibly measured, and I’m happy to put my arguments up against anybody’s on that point.
Dr. Sharp: I want to make it a little more real for people. So I’ll simplify this just for the sake of giving you something to push back against. So are you saying, for example, that maybe you would not mention a weakness on the WISC or the WIAT or whatever it might be, unless it was said, this individual has average abilities. You wouldn’t even mention anything unless you were seeing scores of like 75, 80?
Dr. Joel: Yeah. I’m happy to include them in the [01:00:00] report. I might mention where they are but if I’m going to say this person has a specific deficit in processing speed and this deficit matters in understanding a person’s reading fluency, I can’t be playing around with 10 point differences as explanations. There are tests that just aren’t reliable enough to get at those explanations.
In order for me to say it with confidence that a person really does have a deficit in processing speed, normally, what I’d need to do is first notice that there’s a deficit there and then verify with follow-up testing, that it really is there.
If I’m going to make a high stakes explanation to say, you know what the problem is, is this, I need to be pretty sure that it’s there for real. I can’t just notice the pattern after the fact and say, oh, look at this, there’s a low score [01:01:00] here. I might notice the score, and then two or three follow-up tests to really nail it down and to investigate how consistent is this deficit, does it show up on one test, two tests, all of them, all the different modalities and testing paradigms? Is it consistent?
If it is, okay, now we’ve got a real deficit and then I have to be equally confident that the person really does have a reading fluency problem or a math calculation fluency problem. And so I’ll need more than just one test to make that determination.
If I get repeatability that reliably gets these deficits then I’m confident in saying, I have confidence that there really is a deficit. It’s large, let’s say, 20, 25 [01:02:00] points below expectations. And this is a reasonable bet that based on the theories and the models that have been developed by me and by others that this is a reasonable explanation for why a person might be struggling in this area.
It might not be a complete explanation. They’re always usually more than one explanation, but it’s a first approximation of why the deficit might be there.
Dr. Sharp: I hear you. There are so many directions we could go with this conversation. My gosh, I want to make sure that we get to the evolution part of our discussion, which is where we’re headed with the conceptualization of cognition and intelligence.
[01:03:00] Dr. Joel: Psychometric theories of intelligence have mostly been about the structure; what are the basic ingredients of ability? And so nailing down what are the basic categories. So CHC theory has been sometimes been criticized. It’s not a theory, it’s a taxonomy. I haven’t liked taxonomy a lot. So I’m okay with that.People mean different things by theory. It is a taxonomy where it needs to go and where it is going is this moving from a taxonomy towards a functionally integrated theory of how these abilities are interrelated. For example, working memory and fluid reasoning are not just two separate abilities just sitting there [01:04:00] like marbles on the ground or something.
They are functionally integrated that working memory is the desktop of the mind and then the fluid reasoning is the ability to make sense of what’s in there. So if I’m dealing with something complicated that I’m learning for the first time, I’ve got to load that information into working memory and then play with it for a bit till it starts to make sense.
If I have a limited working memory capacity, I might still be able to make sense of it, but I’m going to have to load those little pieces of information into long-term memory first in order to make that possible. People with a large working memory capacity, they can load a bunch of things in temporarily and then see the pattern a little easier.
[01:05:00] So there’s a connection there. Processing speed, like coding and symbol search on the WISC. This is a precursor to working memory. You can think of working memory as the ability to juggle information in the short term and processing speed is like the dexterity of your hands. If you’ve got dexterous hands, you can juggle more things in the air.There’s developmental cascade theory by Brian and someone else I can’t remember right now, my retrieval problem, but from processing speed to working memory to fluid reasoning, there’s a chain of associations. So we’re starting to get a structural model of how the abilities are related.
And then the contribution of Cattell is that fluid reasoning is the precursor to crystallize stability or knowledge [01:06:00] language. So you’ve got this core called the simplex structure of an array of abilities leading one to the other in a developmental cascade from processing speed to working memory to fluid reasoning to knowledge skills.
There are other abilities that matter, but that’s like the primary axis of how these things get functionally integrated. And if you were to draw arrows between these abilities, you could call that g if you want to, the connection between them, how they’re functionally integrated or what David Wechsler called global intelligence, this structure as a whole, that’s intelligence.
So this is what Kevin McGrew and I are trying to do. This takes CHC theory from a list to a functionally integrated theory of cognition. [01:07:00] And as a person who does assessments on individuals, the job becomes where in this chain of abilities is the breakdown?
Dr. Sharp: What do you mean to break down?
Dr. Joel: So if there’s a specific deficit in, let’s say, working memory, how does this disrupt reasoning? It doesn’t mean that reasoning seizes. It just means that this person has to find ways around this limited capacity.
Like if I had a very large desk and I needed to sort a bunch of objects into piles, I could do that comfortably. If I have a very small desk, I might still be able to sort all the objects on my desk, but I’m going to have to do it piecemeal. I have to work around the limitations of my small desk.
Dr. Sharp: Yes.
Dr. Joel: And you, as someone who works in neuropsychology, you know that there are people who [01:08:00] have specific deficits that they have to learn to work around.
Dr. Sharp: Sure.
Dr. Joel: Through various skills building that you outsource the working memory deficit to your phone to help you manage some of this information. And by doodling people externalize their working memory. And through language, we did the same thing, we chunk things together in ways that enhance our working memories to get around whatever limitations there are. So that’s the sort of thing that I mean.
Dr. Sharp: That makes sense. I’m going to ask a dumb question that might not even be in the ballpark of what we’re talking about, but when you’re talking about an integrated model like this, it seems like that might be at odds with what we were talking about earlier in terms of seeing these abilities as distinct and [01:09:00] having an impact on functioning somehow. Am I anywhere in the ballpark with that?
Dr. Joel: Yeah. So if it sounds like I’m contradicting myself, I hope that’s not the case. I’m acknowledging that all these things are integrated in the same way that the various systems in your car are integrated, but specific parts can break down. And sometimes that disrupts the entire functioning of the car and sometimes it doesn’t.
There are certain things that are like, well, the AC doesn’t work or I can’t roll the window up with the…, and then there are things like the carburetor doesn’t, the fuel injection doesn’t work and this car isn’t going anywhere. For a little while we can turn on the AC, but that’s about it.
Dr. Sharp: Right. It just makes me think then, as the mechanics, we’ll just continue to go with this metaphor. as the [01:10:00] mechanics that we are, it becomes so important then to figure out where the breakdown in the chain is. I guess that’s where this is leading for me is in the real world, what does this look like in terms of do we need better measures? Do we need to interpret the data in a different way or do we need both? Do we need something totally different?
Dr. Joel: So when I started learning about this field, what I was hoping for would be a really good cookbook as Paul Neil would say, where you get a list of recommendations that if you see this, then you do that. And if that existed, I would love it, but there’s very little in our human experience that works that way.
Imagine if I told you like there’s a friend of [01:11:00] mine that I want you to get to know, I think you guys would be friends. Let me give you their five-factor model of scores and then see if you’re going to like them. If I see this profile, I’m definitely like that person, or what kinds of conversations are we going to have given their NEO profile?
There are very few things that work this way. However, if I did tell you about this potential friend and you conceptualized it in terms of a five-factor model, it might alter how you interact with them, at least at the beginning, but I very much doubt that you would have a series of if then decisions based on someone’s personality profile.
Dr. Sharp: That’s not how you make friends? We have been doing it wrong.
Dr. Joel: Right. The same thing that happens with cognition [01:12:00] is that there are very few things if I see this pattern, then I’m going to do this than the other and if I fault the field for a lack of that good cookbook, I think that’s uncharitable.
What happens instead is that a good assessment gives people an accurate and empathy producing case conceptualization, so that as they’re making those moment to moment decisions about what are we going to do solve this problem, it’s grounded in a reasonably accurate case conceptualization, and you’re going to avoid some basic mistakes, but you’re not going to know in advance exactly what you’re going to do based on some numbers.
Exactly what you’re going to do depends on the ground level interaction with [01:13:00] the child, with the parents, with the teachers. And you’re just never going to get to a cookbook series of interventions based on a profile.
Dr. Sharp: That’s fair. Okay.
Dr. Joel: I still want to know it.
Dr. Sharp: Of course. Before we transition and start to wrap up, I just want to give you any space to talk more about this integrated model and where we’re at with that and the direction you see us going or that you might be going with your research?
Dr. Joel: One of the things that I’ve been thinking about is how all of the different individual differences models need to be integrated. I mentioned the five-factor theory of personality, its academics, think of it as the consensus models. There are other models out there. I’m sure many of [01:14:00] them have valid points to make, but the broad consensus is that the five-factor works pretty well.
And then the theory of interests, Holland RIASEC; realistic, investigative, artistic, social, enterprising, and conventional. Those domains of career interests have a pretty good consensus that it works reasonably well.
Ackerman and colleagues have done a fabulous job of integrating these three domains of personality, interests and ability. I think that we can elaborate on that work and start to think about how does the structure of knowledge relate to the structure of [01:15:00] interests? I think it’s a fairly easy step to say, well, you tend to learn more about things you’re interested in.
Dr. Sharp: That seems reasonable.
Dr. Joel: Sure. You learn things that you’re not interested in too. They’re eat your vegetable courses, like you get your degree, you got to go through the statistics course if you’re not interested, but the things that you learn the most about are the ones you’re interested in. It seemed reasonable to me to think that the structure of knowledge might have an echo of the structure of interests that it might actually have that same axonal shape or circular shape that career just have.
And so in our 2018 chapter, Kevin McGrew and I presented preliminary evidence that actually [01:16:00] does that knowledge domain of having a similar structure to interest domains. So if you have artistic and social interests, you tend to have more knowledge in art and the helping professions, and those things that are more correlated with each other than knowledge in, let’s say, engineering.
So that’s one thing that we can do is to integrate these various individual differences models. In addition, I think that as clinicians, we need to become better at understanding how individual differences in cognition and basic behavioral processes interact. So the role of avoidance and negative reinforcement in general really influence what we master and what we don’t.
And so [01:17:00] what I’ve been trying to emphasize in my writings and when I supervise cases is to talk about like, look, you might have an initial deficit in, let’s say, processing speed, and that slowed your reading down, but that’s probably not the reason that you’re having difficulty reading now. There was probably some other behavioral process that made reading unpleasant and made you avoid it and you never developed full fluency because of that.
That these other kinds of forces of personality and interest and psychopathology play a role in the development of your abilities. And so what I’d like to do is try to develop responsible models of integrating these diverse domains.
Dr. Sharp: I love that idea. [01:18:00] Just thinking about, and hearing you talk about it, it’s exciting to think that we could move in this direction. I think that makes sense.
Dr. Joel: For the past 10 years, I’ve been working on and off on a book that tries to lay out the principles of how you do this. And then I’ve been writing software that helps people implement some of these things and try to do it in a psychometrically defensible way. I think I mentioned before the interview that I’m hoping that saying this publicly gets me to get on this project and finish it.
Dr. Sharp: Yeah, what’s the deadline?
Dr. Joel: It’s my own internal deadline. There’s nobody pushing me. I haven’t decided if I want to have it be published in a traditional manner. I want to just give it away.
[01:19:00] Dr. Sharp: That’s remarkable.Dr. Joel: It seems that giving things away is one way to have a lot more influence, like you’ve given this podcast away and you’re reaching way more people than if you charge them for it. And so if it helps move the field forward, I haven’t quite decided yet, the software, if I do it in the R programming language, it’ll definitely be free. And some of it already is out there.
Dr. Sharp: That’s incredible. I hope that we might be able to talk more about that and many other things at some point in the future. This has been such a rich discussion. I know people are probably walking away with a lot to digest and then their curiosity piqued about the [01:20:00] software.
Anytime I hear anything about software that will help us do our jobs better, there’s a lot of interest around that. So I think you’ve got a lot of unspoken public pressure too.
Dr. Joel: Just to clarify, what things am I talking about? I made an R package, which is a programming language for statistics that calculates how unusual a profile is. So sometimes you can see some WISC profile that looks really weird and it’s like, gosh, this seems too weird to be true. You can calculate how unusual is it for these four numbers to be in this particular configuration.
I’m not one of those people who puts a lot of meaning into the scatter, but if I’m trying to figure out like, is this too weird to be [01:21:00] true? I can get a quick calculation. And then you can also do things like given this person’s cognitive abilities, how unusual are the academic abilities as a whole profile?
For me, the primary benefit is to identify scoring errors on my part like I knew there was something off about that score. It was just too weird to be true, but it may have applications beyond that. So that’s one example of the software that I’m talking about that can identify unusual things or features of your data that you already have. You routinely collect and you could get extra information about it for free.
Dr. Sharp: That’s incredible. I would find that very interesting. [01:22:00] I always talked about my love of technology. I have just enough knowledge in that area to be dangerous and very interested in anything like this. If you come up with something, just promise me and everyone else that whatever that factor is, you will call it the weirdness factor, because I feel like that’s just what you have to do, right?
Dr. Joel: Sure, the naming matters, the weirdness factor.
Dr. Sharp: Oh my gosh. Joel, this has been incredible. I know that we are just scratching the surface with so much of this discussion, but like I said, I hope maybe this is just round one and we might be able to talk again in the future sometimes.
Dr. Joel: I’d love that. I appreciate your thoughtful approach to this interview and the way that you ask questions. As I said before we started [01:23:00] recording, I think this podcast is a wonderful service to the field and the kinds of information that you’ve been able to collect and get from your interviewers is quite spectacular.
Dr. Sharp: I appreciate that and I appreciate you being a part of that process. Thank you.
Dr. Joel: Thank you.
Dr. Sharp: Okay folks, thank you as always for listening. I hope that you enjoyed this one. Like I said, chock- full of information, stories and reflection. I know for me personally, I walked away from this interview with equal parts intellectual understanding of this material but on the flip side, also quite a bit of philosophical reflection about what we’re doing and how we do it. So it was a good one. You can find all of the resources we mentioned in the show notes. There are quite a few articles, books and Joel’s blog, [01:24:00] of course.
Like I said at the beginning, if you’re an advanced practice owner or hoping to be an advanced practice owner and you’re thinking about doing some hiring or expanding or additional streams of income or streamlining your systems, not trading time for money anymore, the Advanced Practice mastermind group might be a good fit for you. We’re going to start in June. You can get more information at thetestingpsychologist.com\advanced. Stay tuned, more episodes coming at you. Take care.
The information contained in this podcast and on The Testing Psychologist website are intended for informational and educational [01:25:00] purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment.
Please note that no doctor-patient relationship is formed here and similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with expertise that fits your needs.