Today’s podcast sponsor is Q-interactive. Once again, Q-interactive is Pearson’s iPad-based system for testing, scoring, and reporting a number of measures that are very widely used. You can experience unheard-of efficiency and client engagement with 20 of the top tests delivered digitally.
That’s the ad script, obviously, but I can say personally that we’ve used Q-interactive in our practice for many years for a variety of tests and it does make things quicker with scoring, interpreting, and little kids like to play on the iPad. So if you test kids, this could be cool. You can learn more at helloq.com/home.
Otherwise, Pearson is doing two webinars to increase your knowledge of Q-interactive. You can find out more again on helloq.com. You can also [00:01:00] look on the Pearson website. I believe the next one is on, let’s see, there’s one on May 9th that I am doing about the costs and the cost-benefit analysis of Q-interactive. And then there’s another one on May 16th- an intro to Q-interactive. So check those out if you’re interested.
Today’s guest is Dr. Ben Lovett. [00:01:22] This is my first repeat guest with good reason. Dr. Lovett and I had a conversation back in episode 44 originally meant to dive into his book Testing Accommodations for Students with Disabilities, but we ended up on a discussion about ADHD assessment and the role of behavior checklist versus neuro-psych tests. It was a great discussion, but we didn’t talk about his book at all. So he has come back today to talk all about his book. I have a link to the book in the show notes.
Just to give a refresher, [00:02:00] if you haven’t heard that past episode, Dr. Lovett is an associate professor of psychology at SUNY at Cortland-State University of New York. His research focuses on the diagnosis of individuals with ADHD, learning disabilities, and related issues, as well as the provision of testing accommodations to students with those disorders. He has published over 70 papers on these topics, and again, he has written a book literally on this topic.
Ben has served as a consultant to numerous testing agencies and schools on disability and assessment issues. So, he is super knowledgeable, and very clearly versed in the research around learning issues, ADHD, psychiatric issues, and test accommodations. So I hope you will enjoy this podcast.
Let’s do it.
Hey’y’all, welcome back to The Testing Psychologist podcast. This is Dr. Jeremy Sharp. Today, my guest is our first repeat guest ever. It’s an illustrious honor. Dr. Ben Lovett is back to talk with us all about many things but we’re going to be talking a lot about testing accommodations for a variety of concerns.
Ben was on the podcast last time and we were supposed to talk about his book, Testing Accommodations for Students with Disabilities, but we got into a really interesting discussion about Assessing ADHD and the roles of behavior checklists vs neuropsychological tests, and how those fit together. Anyway, we had a great discussion back then. If you have not checked out that episode, I would recommend you go back and look at that one.
Today, I think we’re going to focus more on your book, some of those testing accommodations, and the research that you’ve done in that area.
First and foremost, welcome back.
Dr. Lovett: Thank you. I’m very happy to be back. Thanks for having me again. It was a lot of fun last time even though our discussion was about ADHD. I’m always happy to talk about that too.
Dr. Sharp: Sure. I appreciate that you were willing to come back. You’re right, it was fun. We had a good discussion. It generated a lot of talking in the Facebook group. So I would imagine this one will too.
I’m excited to dig into your book. I talked a fair bit about the book last time, but can you maybe give a brief overview of the book and what led you to write it, and then we’ll dig into some of those specifics?
Dr. Lovett: Yeah. It’s been about 12 years or so since I started to do research on accommodations. My doctoral advisor, Larry Lewandowski who became the co-author of the book, we both felt that there wasn’t a resource that based accommodation recommendations and other sorts of accommodations decision theory on actual research.
And as of 12 years ago, there wasn’t that much research out there. And in the interim, we were privileged to be able to do some of that research on certain things, especially extended time accommodations. But the more research that kept coming out, we felt like the decisions really should be based on those empirical results.
That was probably in 2011 or 2012 when we first developed the idea. And then two years later, the book came out. So, even though some research has certainly come out since the book, we feel that it’s still a pretty good review of the research and a lot of the decision theory and other sorts of information. It’s certainly based on empirical research. So, we’re very happy to have that out there.
Dr. Sharp: Absolutely. Sorry, a dumb question, but when you say decision theory, how does that play?
Dr. Lovett: I’m referring to the theory of how accommodations decisions should be made. One of the frameworks that we adopt in the book is actually from 1994. There was an educational measurement professor at Michigan State University at the time. She’s now a full-time consultant, Susan Phillips. She had proposed five questions that she felt were very important in determining whether or not an accommodation was appropriate in any given situation. And so we use that framework throughout the book to talk about those things that might be helpful for our discussion today. I don’t know if it’ll be helpful if I briefly mention them. They might be things that we could come back to.
One of the things that Phillips talked about was, are scores that are obtained with the accommodation comparable in terms of their meaning to scores that are obtained under standard testing conditions. If you give a student extended time to finish a task, do the students who get extended time, do they have scores that are similarly reliable and valid in terms [00:07:00] of being able to predict things, for instance? So, students who take extra time on the SAT, for instance, do their scores just as well predict how they’ll do in college as scores that are obtained under standard conditions? So, that’s one thing.
Another thing that Phillips talks about is, is the test still the same in terms of measuring the same fundamental skills? For instance, if a student receives a read-aloud accommodation, someone reads them the test, then it still may measure say United States history knowledge in a high school, but it wouldn’t be appropriate to measure reading comprehension that way. You’ve changed the constructs to a listening comprehension test. That’s always a question we should be asking. Are we maintaining the ability of the test to measure the skills that it was designed to measure?
Another thing Phillips asks is, are the benefits of the accommodation specific to individuals who have disabilities? If anyone would benefit from the accommodation, and there’s some research to suggest that happens with extended time, if the benefits aren’t specific, then we have to be very careful about assigning that accommodation. Is it fair to give it to some folks but not others who would still benefit from it?
Another thing that she brings up, her fourth question is whether or not students with disabilities can adapt to standard testing conditions. So are we providing accommodations because the student truly can’t access the test under typical conditions, or is it just that they would feel more comfortable or prefer to have a separate room or extended time? Is it based on a need or just a preference?
And then finally she asks whether or not that decision procedures are following some standardized reliable tool, or if we’re basing our accommodations decisions off of what we think would help without any standardized procedure for determining that.
When I encountered those questions, again, that was back in maybe 2005, I felt like they just encompassed everything that we want to know about the accommodations. And so, a lot of my research since that time has been trying to [00:09:00] search for research and sometimes conducting it, exploring those five issues.
Dr. Sharp: Okay. That’s fascinating. I had never heard of her or those criteria, but I could see that seems like a great set of criteria to guide your decision-making process, right?
Dr. Lovett: Um-hum.
Dr. Sharp: In terms of the research that comprises the book, was that original research that you did yourself, or was it more compiling research that was out there or both?
Dr. Lovett: It was almost entirely done by other researchers. Larry Lewandowski, myself, and other collaborators who we’ve worked with, there might be 10 or 20 citations to studies that we had done or other papers that we had published, but certainly, most of it was published by other research teams, some of whom were specifically looking at accommodations and their effects, but other times the researchers were looking at things like the effect of test anxiety on performance or whether or not disability [00:10:00] diagnoses are made accurately because all that plays into whether or not the accommodations decisions are appropriate.
Dr. Sharp: Sure. That’s a good segue to talk about the book a little bit. What is in the book? Let’s pretend. Well, maybe we don’t have to pretend actually. A lot of people probably have not seen your book. So, can you just give an overview of what y’all cover in the book?
Dr. Lovett: Certainly. We have a few preliminary chapters introducing the topic, defining what an accommodation is, talking about the framework of Phillip’s questions, thinking about legal issues; which laws and regulations, and things like special education and disability law, protect individuals and [00:11:00] ensure that appropriate accommodations are provided.
After those few preliminary chapters, we transition to talking about different kinds of disabilities, and what accommodations might be appropriate for them, and we do a detailed review of the literature across a few more chapters looking at different accommodations. What has research shown about timing and scheduling accommodations? What has research shown about presenting information in a different format like a read-aloud accommodation? What has the research shown about setting accommodations, being able to take your test in a different location, and response format accommodations- if you have a scribe to write down your answers, or you don’t need to bubble things into a Scantron sheet, you’ve got to just circle them in the test workbook. So, we review those topics in the center of the block.
And then we have a few more chapters on things like for instance, interventions. When is it appropriate to provide remediation or psychotherapy or some other sort of intervention to help the students so that they may not even need accommodations after the intervention is provided?
[00:12:00] Some later chapters are also on things like post-secondary issues. We’ve found with a lot of, there were quite books, but a lot of articles and book chapters on accommodations focus just on the K to 12 accommodations area. And so, both Larry and I have worked with independent testing agencies that are often trying to help make sure that their exams are accessible to individuals at the college level, graduate professional school level, certification and licensure level, things like that, and beyond. So I wanted to have stuff like that in the book too.
Dr. Sharp: Got you. So you run across the lifespan as much as you can?
Dr. Lovett: Exactly.
Dr. Sharp: That’s fantastic. I think this is such a needed resource. I’m sure I’m going to say that again before we’re done today.
Dr. Lovett: I appreciate it.
Dr. Sharp: It’s nice to pull all that information together. You’re right. Admittedly, I went through grad school. I think a lot of people probably went through grad school where we maybe were [00:13:00] given recommendation banks or templates and maybe learned from a supervisor, but I certainly was never presented with any research behind certain accommodations. It just seemed to make sense anecdotally what we’re recommending.
Dr. Lovett: To be fair, there are certain accommodations where we have little or no research. And there are times when you can use logic and intuition for certain types of disabilities to state it’s likely that this accommodation would benefit the person.
That’s the case for sensory and physical disabilities. You do need to have a lot of expertise in accommodations theory to say that if someone is visually impaired, if they have a visual impairment, then a typical paper and pencil form for an exam would be inappropriate. And so, depending on that particular student’s skills, if that student is failing in Braille, that might be an appropriate accommodation. Depending on the student’s vision level, a large print accommodation may be appropriate, even a read-aloud accommodation depending on the student.
For sensory and physical disabilities, I think accommodations are somewhat different. I don’t want to make too broad a generalization here, but for students who have sensory and physical disabilities, often, it’s very clear that they’re unable to access the test under standard testing conditions. And if you administer the tests under those conditions, that would not be a fair representation of what that student knows and what their skills are.
The problem is we often take that model, the client with learning disabilities, cognitive disabilities, or psychiatric disabilities, and the issues are clear. A student has generalized anxiety disorder and they report that they will have a panic attack let’s say, or a severe anxiety attack if someone else finishes the test before they do when they’re taking the SAT. Is that a basis for a separate room? How to determine this? Is there objective evidence to suggest that that person will be unable to continue taking the test? It’s hard to know. It’s not quite the same thing as a sensory or physical disability.
Dr. Sharp: Sure. Well, I think you’re teasing a lot of topics already. [00:15:00] So, let’s just jump into it. Maybe we could start at the beginning. For me, the beginning is the assessment process because that guides recommendations. Would you agree with that or is there another beginning that we should start?
Dr. Lovett: Yeah. From a psychologist’s point of view, and I think that’s our audience here, the assessment process should be where things start. The referral process is the first step of that. One thing I always say when I present to evaluators or psychologists who are performing assessments, who might recommend accommodations, I always say, it’s very important to be clear about the context of their hurdle.
[00:16:00] There are times when I read reports by psychologists and there’s this nice crystal clear background about what brought the person to you today. There are other times when the reason for referral is something like so-and-so and his or her family was interested in obtaining an updated portrait of their cognitive and academic functioning.
I don’t know how many people do that for fun. I’m always wondering what exactly brought us here because often, what brought someone to your office is some problem that they’re dealing with, a type of impairment, a functional impairment, doing poorly somewhere. That’s helpful to know. And so, I would recommend being very specific about that in detail.
Dr. Sharp: Okay, that’s good to know. So, you got the referral question and making sure that we’re pretty explicit about that in the report right off the bat. So, let’s [00:17:00] say, we jump into the assessment process. We talked about ADHD, certainly last time. I would to chat about some of the other disabilities that you discuss in your book. So from an assessment standpoint, what would you say are the standards for assessing learning disorders, and psychiatric concerns when we’re thinking about accommodations?
Dr. Lovett: Absolutely. So for learning disabilities, very common, the condition is very much just like ADHD, especially when those folks are applying for accommodations.
Again, I’ll go back to the DSM-V which I think has some really helpful information about what they call specific learning disorders, what we tend to think of learning disabilities more generally. So, one thing the DSM-V is very clear about is that we need to see substantially below-average academic skills.
There are older models [00:18:00] of diagnosing learning disabilities. One that was very popular was the IQ achievement discrepancy, but you could almost attract someone’s achievements from their IQ score. If they had an IQ of 120 and the reading score was only 95, that might be a severe discrepancy. And so, that might suggest a learning disability. DSM-V criteria were in part written to ensure that that discrepancy model was gotten rid of because research is not supportive of it in terms of being reliable or valid for diagnosing learning disabilities.
Someone may perform below what our expectations are, but that doesn’t mean that they have a learning disability. And so, we expect to see below-average performance on standardized measures of academic skills and some impact of that in the person’s real role functioning educationally, if it’s an adult, occupationally, where’s the impact of those below-average academic skills. So reliable, validated achievement tests. Measures like the Woodcock-Johnson, the WIAT, the Wechsler Scales, The Kaufman Test of Educational Achievement, and similar sorts of tools are really helpful for measuring those academic skills.
And then for documenting the impact in a real-world setting, [00:19:00] it’s very important to be detailed about exactly how the person is performing. If we’re talking about a student who’s still in school, whether it’s K to 12 or college or graduate school, exactly how are they doing? What are their grades? How are they performing on tests and other sorts of academic assignments? Rather than just, if a person reports that they are struggling or that they’re experiencing difficulty, that certainly may be their honest, subjective perception, but they may be feeling like they’re struggling because they’re getting a B+. That will not generally indicate education on them.
Dr. Sharp: Sure. That’s great.
Dr. Lovett: Those two components, the below-average academic skills as shown by diagnostic achievement tests and [00:20:00] the actual impact in a real-world setting are what we expect to see. And then I would just also note for say high school kids or beyond that college students and adults, the history is very important. Learning disabilities don’t start when someone’s 15 or 20 years old, there’s something present early on. And so we expect to see some trouble with the initial acquisition of academic skills as well.
Dr. Sharp: I have two questions from everything you just said. One, does IQ testing then have any place in the assessment of a learning disorder?
Dr. Lovett: That’s a good question. And it’s certainly a controversial one. One thing that if we just look at the DSM-V or we’re doing a core evaluation of just a learning disability, the real purpose of an IQ test or an IQ screener would be to rule out something like intellectual disability if that’s a concern.
Personally, my opinion is that IQ test results don’t generally show us a lot about whether or not a learning disability is present. They can give a lot of information about a student’s cognitive skills that might help inform interventions and even at times accommodations, but to me, if you’re just trying to check if a learning disability is there, the main purpose would be to rule out at least borderline intellectual functioning, if not an intellectual disability. And in many cases, that’s not a concern. It’s not an issue.
So to me, that’s not an especially important part of a core evaluation, determining if a learning disability is present. I don’t know. I should say there are some testing agencies that expect to see an IQ test as part of the documentation.
Dr. Sharp: Yeah, I’ve seen them.
Dr. Lovett: That’s a different issue. I don’t honestly know if that’s more to rule out general low academic ability or things like that, but I do something, of course, to pay attention to determining whether or not a learning disability is present. To me, it’s academic skills and educational impairments that are much more important than cognitive issues.
Admittedly, I know it is a debated issue. There are certain models for diagnosing learning disabilities, not only the IQ achievement discrepancy model, but one that’s also popular in some settings is the PSW, the pattern of strengths and weaknesses model that requires that there be some below-average academic skill, but also a cognitive deficit that underlines that academic deficit.
I certainly respect researchers, scholars, and practitioners who are trying to make sense of the student’s unique profile using those patterns of strengths and weaknesses models. But in my opinion, the research hasn’t necessarily been all that supportive of them.
There are two ways to apply that pattern of strengths and weaknesses model. Some folks will apply it very rigorously using even software that’s been developed. Dawn Flanagan and her colleagues have one PSW model, the cross-battery assessment approach, a very rigorous software that you can use to determine whether or not there is indeed a pattern.
The other way that some folks apply the pattern of strengths and weaknesses model is just to say, is there a profile? And then, can I find some logical relationship between lower academic scores and some low cognitive scores? I think that capitalizes on chance. It’s really easy after the fact to look at any profile of cognitive and achievement tests and find a pattern of strengths and weaknesses. This is motivated, but it’s very easy to say, well, this score was an 88 on this particular achievement subtest when I gave 12 different subtests. And I guess that connects to working memory, which was also a little bit low.
And so, there are times when I see that being used in the learning disability diagnosis. I think when the PSW model is not applied rigorously, it could lead to a diagnosis with pretty much anyone.
Dr. Sharp: Yeah, I see what you’re saying. Again, anytime we stray from data, that can get you into trouble.
Dr. Lovett: Absolutely. So again, I acknowledged that [00:24:00] the cognitive measures are viewed by some as important.
I don’t necessarily see the research as supporting them, especially as part of a core evaluation to just see if can we define if a learning disability is present. Academics and educational impact are much bigger issues unless we’re trying to rule out more general global low ability.
Dr. Sharp: Okay. That sounds good. Let’s move on to maybe psychiatric concerns. What does the assessment look like there?
Dr. Lovett: Of course, it depends a lot on the nature of a referral concern. Is there suspicion of problems with anxiety/mood? Is it something that’s instead an externalizing problem like oppositional defiant disorder?
The main thing that I would say for any type of concern or any type of disorder is that it’s really helpful to have broadband measures being used that are assessing concerns behind what the perhaps initial referral area is. Let’s say for instance that you have a child who’s referred for anxiety-related concerns, it would still, in my opinion, be very important to do a screening for mood problems, behavior disorder problems, and things like that beyond there.
And there are times when you find that what’s initially a concern about anxiety, the anxiety is related to a desire to… Reports of anxiety or being used to get your way and things like that. So sometimes measuring those other issues turns out to be the bigger problem.
So I recommend the use of norm-reference standardized behavior rating scales from multiple raters just like when ADHD. So using measures like the BASC or CBCL, I think is very helpful as a start. And then using those, have a conversation with the raters to try to find out again, what are some specific examples of these sorts of things that you rated? How was that causing impact to the person’s life? To be able to do a good differential diagnosis of what the underlying problem is.
[00:26:00] So many symptoms can be common to different sorts of disorders. A big one which I often think of related to ADHD is inattention. So, there are reports of inattention. Many folks will lead to ADHD as a possible diagnosis. And that’s not wrong of course, but pretty much every disorder causes inattention. Anxiety causes inattention, depression causes inattention, trouble concentrating things like that. Schizophrenia causes inattention.So, we shouldn’t leap from a particular symptom to a particular diagnosis, except in rare cases where there’s not much of a differential to do.
Dr. Sharp: Yeah, that makes sense. So what about the role of personality assessment [00:27:00] in psychiatric issues for older kids and young adults?
Dr. Lovett: So clinical personality measures like the MMPI and things like that, I think they definitely can be helpful in understanding the person. That’s something that I don’t have as much expertise or experience in using clinically. I certainly was trained with them and I have given them at times, but it’s not something that I…
I again, tend to view a diagnosis as the first step towards accommodations. And so, I tend to think what are the measures that would be most helpful in determining whether DSM criteria are met? And so, personality tests in my experience, aren’t generally as key to the core features of the DSM construct.
Self-reports, I should say can be very helpful, but of course, the behavior and symptom rating scales also usually have self-report versions, but in terms of clinical personality measures, although they might be really helpful in understanding the child, and I should say I’m referring to the object. Personality measures projectives are a whole different kettle of fish entirely. And so, thinking about the objective measures, I just don’t know if that is key to the diagnostic constructs.
[00:28:00] Dr. Sharp: Yeah, that makes sense. I do struggle with that sometimes, especially with these young adults, how much the personality measures contribute above and beyond a BASC and a good interview and maybe some more specific measures.Dr. Lovett: One thing I will say for some of them is that they have very good symptom validity measures compared to some of the rating scales. At least we have more research, I think, on some of them. So for instance, if we’re trying to see if someone’s trying to make a positive or a negative impression, a clinical personality measure like the MMPI can have a lot of different validity indices to see whether or not someone’s taking the measure seriously, to see whether they are trying to present themselves in an unusually favorable or unfavorable way, to see if they’re reporting a number of very rare symptoms that there doesn’t seem to be other evidence for, those sorts of things.
Even though there are validity checks in the behavior rating scales, I have seen very little independent research on them showing them to be all that effective. I can’t say that they’re not, but the main ones that seem to me to be perhaps effective are looking at whether or not the form is filled out consistently.
So some of those validity traps will look for pairs of items that are pretty similar. A rater rates one symptom as extremely often experiencing it, but then the other symptom that’s almost the same thing, it’s almost never happening. I wonder if they’re filling it out carefully, but I don’t know that those rating scales are as good at detecting symptom exaggeration, which may be an issue of someone is trying to demonstrate a need for accommodations, just because of an honest desire to demonstrate, look, I think I am impaired. I do need this.
Dr. Sharp: Oh, sure. So following from that, is there a place for actual symptom validity testing like the TOMM or the MSVT [00:30:00] or something like that? Do y’all take that into account when you’re considering?
Dr. Lovett: I certainly think that. I would usually consider using those performance validity measures. I know that the terms are used inconsistently. I’ll just give a brief overview of how I think about them. The distinction is often made between symptom validity, which is honest reporting of symptoms, and performance validity, which is putting forth good effort during an evaluation on measures of maximal performance, cognitive, academic, and neuropsychological tests.
For SVTs or symptom validity tests, again, the way I tend to use the term, it refers to added indices in personality and behavior rating scales. Things like the F scale on the MMPI and things like that. Whereas performance would be tests that you were mentioning, like the TOMM, the Word Memory Test, and things like that, some of them can be embedded like the Reliable Digit Span (RDS) on the WISC or WAIS, [00:31:00] but a lot of them are, as you mentioned, standalone measures.
And for her learning disabilities like ADHD, I do think research supports the use of them. There seems to be almost a limit of about 50% sensitivity if we want to maintain 90% specificity. So if we only want to make a false accusation of exaggeration or malingering 10% of the time, we seem to be able to detect about 50% of individuals who are exaggerating. And that’s still something. I mean, that’s still a lot. So I do think that they’re helpful for that reason.
Dr. Sharp: Okay. Yeah, I know that’s an ongoing growing area as well research-wise.
Dr. Lovett: I really would encourage clinicians to not think of themselves as neuropsychologists to look at the neuropsychology literature on performance validity tests. There is so much stuff out there validating different measures, both embedded and otherwise.
There was one recent study that just came out last year. A very interesting study on [00:32:00] using the processing speed measures on the WAIS as embedded effort indicators
Dr. Sharp: Oh, that’s interesting.
Dr. Lovett: suggesting that if someone is getting scale scores on coding and symbol search of 5 or below, that could certainly suggest low effort, things like that, or even certainly just low processing speed index scores in folks who don’t have obvious neurologic impairments and it appears to be rather rare, but you find really low processing speed scores if someone is putting forward their full effort.
Dr. Sharp: Got you. Yeah, I know that’s a whole can of worms that we could jump into. I was just curious about your thoughts on that.
Dr. Lovett: Yeah, that’s the one thing I’ve mentioned again, though, the one thing I’d add with [00:33:00] regard to performance measures is you want to ask yourself if the person has a motive for perhaps exaggerating, which could be psychological or psychiatric. It could be that there’s some sort of material benefit, like a student who’s trying to avoid going back to school after having a concussion or something like that. It might not be accommodations-related, but if there is some sort of incentive, does the performance validity test task relate to how the person might strategize to perform poorly?
I think about this a lot with someone who might be working slowly to demonstrate a need for an extended time. If the performance validity test is time, then that’s going to be a better indicator of whether someone is working slowly. So that’s why I was interested in the processing speed index and the processing speed scale scores as a potential validity indicator, whereas a lot of the memory-based PVTs are not necessarily heavily timed. [00:34:00] So if someone’s working slowly, that might not catch them. So, it’s something to consider.
A resource that I would recommend, a book published back, I think it was in 2015 edited by Kirkwood and published by Guilford. […] in Colorado?
Dr. Sharp: Yeah, he’s just down the road.
Dr. Lovett: Okay. He has a wonderful book to be edited with a number of contributors on symptoms and performance validity testing measuring effort and things like that in children and adolescents because so much of the early work was conducted with adults. There was even this myth that children and adolescents would be putting forth adequate effort and would not think to misrepresent symptoms, and that’s certainly not the case. And I think that the book has a lot of wonderful chapters on various topics related to that.
Dr. Sharp: Yeah, I agree. I’ve seen some of those, certainly. It’s very useful. I’ll [00:35:00] list that in the show notes for anybody who might be interested in that.
Let’s dive into some of the actual accommodations. I’m really curious to hear from a research standpoint which accommodations make sense, which ones are supported, and which ones aren’t.
Dr. Lovett: The most common accommodation by far is extended testing time. We see that in requests at the post-secondary level. We see it on the K to 12 level. One of the reasons it’s so common is that students who receive other accommodations often receive extended time just to use the other accommodations. So, it’s sometimes a very common accommodation.
In the United States, we tend to give extended time allotments of either 50% or 100% extended time. I say in the United States because interestingly, there are some other countries where we see testing accommodations but the extended time allotments are not that much.
In the United Kingdom, for instance, in Britain, there tend to be lower levels. Some things like 10 minutes per hour of extended time are more common. Some standardized testing agencies have added 25% extended time as an option here in the United States though. So that’s a recent thing that you’ll sometimes see. I know I’ve seen that being given to the folks on the MCAP, for instance, as well. So 25% extended time is a more recent addition, but we tend to get a 50% or 100% extended time.
So what might be the evidence that would support such an accommodation? One of the big ones is someone who has reading-related problems or whose reading speed is substantially below what is typical. I would say that even more important than reading speed per se would be their time reading comprehension skills. If the student is unable to read and comprehend text and make sense of it within a standard amount of time, and the test is not designed to measure their time reading comprehension, the test that they want accommodations on, then would generally be one piece of evidence, but a sound basis for requesting additional time.
The student generally would be expected to be able to read and understand texts at least as well, or within the average range compared to most other examinees. And so, if someone has substantially below-average time to reading comprehension, that could be a part of an extended time request.
But the one thing that I think we have to be careful about is that we don’t want to extend the time to become an unfair advantage because there is a lot of research showing that when non-disabled students are given extended time on time pressure tests, unsurprisingly, they do better. So, if your accommodations benefits are not necessarily specific and that’s particularly the case on standardized tests as opposed to teacher-made tests in schools.
So there is a myth out there that you’ll sometimes hear that benefited from extended time means that […] I would say that’s very similar to the myth that if your cognition benefits from taking stimulant medication, that means you have ADHD. In the same way, extended time is something that is desired by many students who don’t have disabilities.
And the survey study that Larry Lewandowski, myself and a number of other researchers did, I think it was published back in 2013 or so, we talk about that in the book, we found that out of 600 and some students, over 85% of those with and without disabilities felt that they would improve their score on a standardized test with extended time. And the research shows that there’s a good basis for those expectations. On time-pressure tests, it does appear that most folks will benefit from an extended time.
In our laboratory settings, we actually will give students with and without ADHD or with and without learning disabilities a standard time limit, to see how they’re doing. We ask them to circle where they are, or sometimes we switch what type of color or pencil they’re using so that we can see exactly which items they solved during the standard time limit and then with extended time. And we tend to find that both groups benefit from the extended time.
Now, in some educational settings on teacher-made tests, the tests may not be time pressure. And so, in that case, it’s unlikely that most students with or without disabilities would benefit from an extended time. But on time-pressure standardized tests, we tend to see effects for both groups.
Dr. Sharp: Yeah. I know that a lot of folks recommend extended time for ADHD as well, but I feel like I’ve read some things saying that that’s not helpful or as helpful as we thought it was.
Dr. Lovett: Yeah. One thing to keep in mind, of course, is that ADHD is comorbid to some degree with reading problems. So some portion of students with ADHD will have low timeframe apprehension. So that could still be a very sound basis for the request. If the individual is so distractible that they’re unable to get through a passage without getting distracted and have to go and re-read it many times, then again, there are times on extended time for the appropriate, but the decision has to be made on an individual basis.
So we should never assume that because someone has ADHD, even if it’s validated ADHD, we should never assume that they need extended time on tests. There should always be specific evidence of that access deficit, that deficit in access skills. So again, we would expect to see a time-reading comprehension performance that’s poor. We would expect to see evidence from real-world settings of teachers saying that they’re unable to complete their exams when all of the other students are and things like that.
And so, we need to do it on a very individual basis. There is some basis for it in some students with ADHD, but we should never assume that it usually means extended time. And that’s why we have to get away from this menu or list of accommodations that go with a disability condition. It needs to be made on an individualized basis.
Dr. Sharp: I see. So what are some other common accommodations that are actually supported by research?
Dr. Lovett: All right. So you know one accommodation, the read-aloud accommodation as I mentioned earlier. The research shows that the benefits of read-aloud tend to be specific to individuals who have access skill deficits. Students who are non-disabled don’t generally benefit from read-aloud accommodations. If anything, they don’t like them. And I think that’s easy to understand. If you’re a competent reader, you’re trying to read the test and someone insists on reading it to you, that’s not very pleasant. It’s kind of distracting and things like that. So that would be appropriate.
And the big caveat is we need to make sure the test is not trying to measure reading skills. In my own state of New York, a policy was changed for the state exams that students take at the elementary and middle school levels so that students who have severe reading disabilities could be read the English language access.
Dr. Sharp: That seems problematic.
Dr. Lovett: Yeah, it is. I think that there are reasons why it happened. Students who have disabilities are forced to take these tests, and parents and schools are understandably complaining saying, my students are able to read. The idea that they should have to sit through this reading test is silly.
I agree with that. It is silly. In my opinion, they should in fact take that test, but reading the test to them invalidates their score if that test is supposed to measure their reading skills at all. We can turn a reading comprehension test into a listening comprehension test.
So, if we’re not trying to measure reading skills, then a read-aloud accommodation will generally be appropriate in those cases where someone has documented severe reading problems, especially decoding issues. So poor reading fluency or poor time reading comprehension would, in general, be enough for a read-aloud accommodation, but we would expect that the person has trouble decoding individual words, I can say. I expect to see that.
Dr. Sharp: Okay. Fair enough.
Dr. Lovett: So that’s one accommodation that we see increasingly at the K to 12 level, but sometimes on higher-level exams as well. The SAT for a long time has at times provided audio recordings of the test items. We do see that there too.
Another accommodation that’s very common that unfortunately, we don’t have much research on is separate room accommodations.
Dr. Sharp: Oh, that’s interesting.
Dr. Lovett […]for a wide variety of reasons. The biggest one by far is distractability. ADHD and other conditions may report that they are so distractable that they’re not able to pay attention to the test if they hear noise or they see something in their visual field other than the test, that’s something that would distract them. And it takes them a lot of time to get back to the test mentally. So distractible is one reason we sometimes see. Another one is students who have reading problems may report that they benefit from reading aloud.
Dr. Sharp: Right. Reading aloud to themselves?
Dr. Lovett: Exactly. And so, they’ll say, I need to be in a separate room so I can read the text aloud. I find that helpful.
Another really common reason is anxiety. People will say, as I mentioned, that they get very anxious if other people are taking the test with them. They get upset if someone finishes the test before them. They’re just generally hyper-aroused in terms of anxiety and the extra people in the room add to that. So we have lots of rationales that sound in a sense, superficially reasonable, but we have very, very little research looking at that.
Larry Lewandowski and his colleagues, I wasn’t involved with this study, but they did do one study looking at whether non-disabled college students would benefit from being in a private room. And they did not find any benefit, which is good. I mean, if someone does benefit, then perhaps it is because of their unique disability-related issues. We don’t have research to support that as being the case. I don’t know of any studies that have looked at private room accommodations on a realistic test for students with any of those issues, ADHD, reading problems, or anxiety.
I’m involved in one project now that’s hoping to get that, but we don’t have any basis for that, unfortunately. But we were in a separate room accommodation, I think we always have to look at the specific rationale and say, is there evidence to support that?
One of the pieces of evidence that I like to see is that when the person has had to take exams in the presence of others, they are unable to access the exam. So accommodations are needed. What’s this person’s history of test performance? So it’s often the case that clinicians will recommend accommodations without even referencing the person’s history of test performance.
Often the individual has never had the accommodations in the past. How were they doing? If this is an initial diagnosis of ADHD, for instance, and the person has no history of testing accommodations and the clinician recommends a separate room, well, it sounds easy enough to implement, like why not get someone a separate room, it’s just could be very challenging. There are schools that run out of rooms on state test days.
Dr. Sharp: Yeah, I’ve heard stories about that, certainly.
Dr. Lovett: If you have 20 or 30 students in the school who will each need a separate room, that doesn’t work. There are certain high-stakes exams for certification and licensure where a room has to be rented for the individual. It’s again, very logistically complicated. If it’s actually needed to access the exam, that absolutely should be done, but it’s not an accommodation just to make willingly just to say, it’s not a big deal, why not give them a separate room?
It can be difficult to implement logistically. And so, if you want to take the responsibility as a clinician to say, I’m saying this person requires this to access tests, there really should be evidence of that.
Dr. Sharp: Absolutely. Yeah, I think that’s one theme that’s come through both of our conversations so far is just having as clear a relationship as possible between the history and real life and the test results and the accommodations that you’re requesting. You need to make that explicit.
Dr. Lovett: Yeah. It’s just very easy to have an accommodations list or a menu as I call it and to check them off in your report template, but there’s not always a sound basis for those in the person’s diagnosis. Even assuming that that diagnosis is accurate, it is not always necessarily a sound basis. ADHD doesn’t mean that a person requires a separate room or extended time as we were discussing.
Another thing for ADHD that I would mention is, what’s the person’s test-taking ability when they’re on medication if they are taking medication? Medication could very well change whether or not that person needs accommodations to access tests.
Dr. Sharp: So how do we address that? How do we get at that in making recommendations?
Dr. Lovett: Right. One thing that we would want to do is get a sense, usually from interviews or other sorts of information about what the person’s symptoms are like when they’re on or off medication.
Dr. Sharp: Okay. So it could be history gathering?
Dr. Lovett: Yeah. I do think this is an area that’s under-researched. It’s not uncommon that clinicians will ask me, I’m seeing someone for an evaluation who already has a diagnosis of ADHD. They’ve already been put on medications. At least they have a diagnosis from a physician, and they want to confirm that, but they’re already taking medication. Should they be on medication on the day of the testing?
And that’s a common question. I wish we had more research on that point. There are a lot of things to consider in making that judgment, but what I would say is what are you expecting the evaluation day to do? If you’re hoping to observe, or if you’re hoping to observe symptoms of ADHD, then the person being on their medication will attenuate those symptoms.
So if you’re expecting their test session-like behavior, whether that’s their performance on cognitive measures, or you’re just behavioral observations to be diagnostic of ADHD, then the person being on their medication for the evaluation will be problematic.
If on the other hand, the primary purpose of your evaluation is to determine whether or not the person needs accommodations on an upcoming high-stakes test, and they’re going to be taking their medication when they’re taking that on how many high-stakes tests, in that case, it would seem appropriate for them to be taking their medication on the day of their evaluation, and so on.
Dr. Sharp: Sure. That does make sense.
Dr. Lovett: I would say it depends on what the primary purpose is. I know there are some clinicians who actually do the testing over multiple days, so they’ll have the person be on medication one day and off another day. So there are certain advantages to that. We just need to consider things like if there are any withdrawal effects and how long we’ve waited between those things.
Dr. Sharp: Of course. There are a lot of nuances to ADHD testing. I’m glad you brought that up, actually. I think that’s important. We tend to have people stay off medication, but it’s a lot of initial diagnoses, it’s not so much follow-up for accommodations.
Dr. Lovett: And when you say stay off medication, I would indicate that many of them were already being prescribed that even went out of that diagnosis, right?
Dr. Sharp: Right.
Dr. Lovett: It’s so interesting. The one thing I always say is safety first. If you have concerns that someone who has severe ADHD without medication, that young adult is driving themselves, I always would recommend asking them, are you able to do things without accommodations so that you can safely get here and stuff like that?
Dr. Sharp: Of course, that’s a good reminder. We don’t want anybody getting a wreck on the way to testing.
Dr. Lovett: Exactly, because the clinician told me not to be on medication.
Dr. Sharp: Right. So let’s see. Are there any other maybe more obscure accommodations that you feel like are very well supported that we might not be thinking of?
Dr. Lovett: To go back to the issue of sensory and physical disabilities, there’s not as much research on accommodations for them, things like braille accommodations, things like that. There is some research that’s actually somewhat older. I’m not as familiar with at least recent research on this topic, but those combinations also tend to be less controversial.
They’re not accommodations that are desired by non-disabled individuals. So they tend not to be something that’s highly sought after. It’s not as though someone that’s perceived as giving someone an unfair advantage. And so those are accommodations we tend not to worry as much about. Instead, the difficulty in those accommodations is often finding the right software, the right logistics to implement. And so that’s often a conversation with the school or the testing agency, or whoever’s going to be providing the accommodations.
So some students who would have vision problems, at times hearing impairments, if there are issues related to that and they’re going to be some sort of audio or oral administration of a test, then that would also pose some kind of issue. But those are the sorts of things that can often be worked out on an individual basis with whoever is providing the accommodations.
So, I don’t have to worry as much about those, but also, it’s good to be fair. I just have less expertise with those. I rarely work with testing entities over those sorts of accommodations. In the schools, they are just far less common. So if look at a distribution of students receiving special education, the biggest categories are students who have learning disabilities, speech and language problems, the other health impaired category which has a lot of ADHD, those sorts of things. It’s far, far, far, fewer students who have sensory and orthopedic impairments, according to the special education statistics.
Dr. Sharp: Right. That makes sense.
Dr. Lovett: Yeah, we can have many accommodations that I think are worth a lot of research effort towards are extended time, which is certainly the most controversial one, and some of the difficult ones, like, as I’m saying a separate room, read aloud accommodations, things like that.
Dr. Sharp: Yeah. So I do want to talk about accommodations that are not well-supported. So are there any common requests that you would say just are almost like myths that they’re actually helpful for certain concerns?
Dr. Lovett: Interesting. I wouldn’t tend to think of a particular accommodation as being good or bad or supported or unsupported. It is about in which case it’s supportive and in which case it isn’t. To get back to extended time, something that I think is a myth is that a student who has a low or lower score on processing speed metrics, therefore needs extended time accommodations.
So I see that assumption made all the time and I understand why. We have these diagnostic tests called processing speed measures. It sounds as though the score on those diagnostic tests is getting up the face of the mental speed of the person’s mind. So if that were the case, then low processing speed scores would suggest a need for extended time on everyone as long as we don’t care about how fast someone is.
However, the empirical research does not support that. Low scores on processing speed measures, although they are normed commonly in students who have learning problems, ADHD who may need extended time, a low processing speed scores per se do not indicate a need for extended time on typical academic tests.
In the research that I and my colleagues have done, we have found processing speeds to be a very important predictor of how long students will take on typical reading-based exams where someone has to read test items to answer them. And if you think about what those diagnostic tasks on processing speed measures are, they are nothing like taking a realist academic test.
If a student has low processing speed but their reading fluency is fine, their time reading comprehension is fine, all of those things are fine, their writing fluency, whatever it is that’s relevant to the real world academic tests, that should not be a sound basis for an extended time request unless you can show that the person needs the extent of time because of visual-motor problems. And if that’s the case, then why did those visual-motor problems not impact their reading skills, their writing skills, things like that?
Dr. Sharp: This is great. What is processing speed measuring then? And how does that translate to the real world?
Dr. Lovett: That’s a good question. I don’t know that there’s much research supporting it as measuring anything in particular, given that we use very simple clerical visual-motor tests. I don’t know that processing speed measures tell us very much about how you do things other than processing speed measures.
Dr. Sharp: That’s interesting. So is there any value in using it as a proxy for anxiety or depression?
Dr. Lovett: Yeah, I think there’s certainly a lot of things that can cause low processing speed, low motivation, fatigue, boredom, possibly anxiety, there’s some research to suggest maybe distractability things like that, but the test than measuring any of those things, I can’t say that I would trust processing speed measures to be a strong measure, if anything, other than clerical visual-motor speed.
Dr. Sharp: Okay. And then the translation from that to a real-life task is
Dr. Lovett: very, very weak. Essentially when we have diagnostic tests in our armamentarium that are much closer to a real-world academic task.
And so that’s why I would say, similarly, let’s say that the student has good or better than average processing speed but has poor time reading comprehension, we would never want to deny that student needed accommodations if they need them because of the processing speed score.
Dr. Sharp: That’s a great point.
Dr. Lovett: And for what it’s worth, I don’t see evaluators doing that, but I see all the time folks who are saying, well, the process speed score is low and so they need extra time, and then the evaluator goes on to ignore the average and above-average time reading based diagnostic test scores.
Dr. Sharp: That doesn’t hang together for me anyway. That’d be a tough sell.
Dr. Lovett: I mean, good clerical visual-motor speech should not suggest that the person doesn’t need accommodations on a test and neither with poor visual-motor clerical speed suggest they do.
What I would ask is, the person is preparing to take teacher-made exams, or they’re preparing to take the SAT, they’re not going to have to search words and symbols on the SAT as quickly as possible. I would say maybe they are slow at that. Maybe the low processing speed score is a genuine weakness on those tasks that I tend to take a behavioral approach to interpreting diagnostic test performance in the sense that I view it as that test as a sample of your behavior, it’s a sample of your responses. You appear to be below average of making that particular response.
If it’s a processing speed measure, then okay, you’ll report visual matching or something like that. Is that really what the SAT is measuring? No.
Dr. Sharp: Sure, it’s that question, what are we getting out here? And how does that translate to the test you’re taking? That’s fascinating. Let me know when you figure out what those processing speed tests are measuring.
Dr. Lovett: I will. I wouldn’t hold your breath., I’m not sure if we’ll ever find out exactly that. We know that the processing speed measures don’t load, especially high on general ability either, which is what IQ is supposed to be getting here
Dr. Sharp: Sure. That’s a whole other episode I feel like. Do you know Ellen Braaten? She’s at Mass Gen?
Dr. Lovett: Yeah. I know Ellen but I’ve never personally met her.
Dr. Sharp: She wrote the book, Bright Kids Who Can’t Keep Up. I talked with her a few episodes ago here on the podcast about that.
Dr. Lovett: I have to listen to her.
Dr. Sharp: It was good. It was really good. We talked all about processing speed and how it shows up in real life. There’s a lot to sort through with this how we measure it and how that translates to real life. So, [01:01:00] yeah, it’d be interesting. I’d love to get your take on that if you listen to the episode.
Dr. Lovett: I definitely will. I just think we have to be clear that I think a lot of times we switch back and forth from the operational definition of the score to some much more abstract concept of mental speed.
What I think it’s even coming to be called sluggish cognitive tempo, which is sort of related to inattention and things like that if someone has slow mental speed and we see that in a variety of different contexts, and we see that on different sorts of tasks, the idea that that would be very impactful on the person’s life, that’s an idea I […]. That’s not a claim that I would think to dispute. If someone tends to be generally slow mentally, then that would be a problem. And that could be very functionally limiting. And that could lead to a need for accommodations. But my question would be, are low scores on processing speech sufficient to make that judgment about the person?
Dr. Sharp: Yeah, that’s a great question. That is the question.
Dr. Lovett: If someone can’t keep up to use that phrase, then they can’t keep up on more than coding and civil service.
Dr. Sharp: Right. So we should see that other.
Dr. Lovett: Exactly. So why is there reading fluency 112?
Dr. Sharp: Yeah, that’s a great question. Well, let’s see. This is great. Maybe we’ll have to do a round three.
Dr. Lovett: I appreciate it. That’s all right.
Dr. Sharp: We just have a few more minutes. Let me see. I’m checking to see if there was any other info I wanted to touch on. I don’t know. I’ll turn it over to you. What have we not talked about that feels important to put out there about accommodations and test-taking recommendations?
Dr. Lovett: I think we’ve covered so many different things and I appreciate the opportunity to talk about these issues. I hope it’s beneficial for the community and the audience.
One thing that I might add is that we should always be giving accommodations as part of a general response to someone who has a functional paramedic disability, a disorder, or more than one. And so, accommodations should never be the sole recommendation. One of the things that we should always be thinking about when possible is intervention, especially for learning, cognitive, and psychiatric disorders.
So, thinking that the accommodations are needed right now for someone, especially for a younger child, an elementary school student who has slow reading skills, are we also putting in place something that will allow the individual to improve their reading fluency?
Dr. Sharp: That’s a great question.
Dr. Lovett: If we’re providing accommodations for anxiety, though there are times when that may be warranted, but are we also recommending some evidence-based treatment for anxiety, which is often your responsive to that kind of treatment?
Dr. Sharp: Absolutely.
Dr. Lovett: Especially in educational settings and when the student is still in K-12 schooling, we’re trying to increase their skills. We’re trying to increase their autonomy. And there are times when testing accommodations are a part of that. And there are other times when testing accommodations can impede the development of those skills. And so, I would just ask clinicians to think, what can they do in the best long-term interest of the client in terms of recommending things?
Dr. Sharp: Yeah, I like that you brought that up. And I know that that probably depends a lot on training and philosophy with assessment. Some people tend to lean more heavily on the cognitive recommendations versus the psych recommendations, but I think you’re just emphasizing that point that it’s important to look at the whole picture and recognize all the different pieces that might help someone be successful.
Well, let’s see. And I know that you address that in your book a little bit as well, is that right?
Dr. Lovett: Yes. We do have a whole chapter on accommodations and interventions and how they were laid out. I think that’s a unique aspect of the book. I would say generally those two camps have been sort of kept apart. As someone who was trained in school psychology, I found it unusual that that field, school psychology, has tried over the past few decades to move towards intervention and away from assessment being the sort of sole stereotype of what a school psychologist does. At the same time, I find testing accommodations being used and recommended in an unquestioned way when interventions would be more appropriate.
Dr. Sharp: I see. Just off the top of your head, are there any cases where you could say that’s usually the case where we should go more toward?
Dr. Lovett: Yeah, certainly for anxiety, I would say that that should be the… Our default response to anxiety should be intervention and not accommodations. Accommodations are needed for a time. Often they’re provided because of discomfort rather than genuine need in the case of accommodations.
And the accommodations can provide the message to the child that the testing situation is, in fact, dangerous. And they do provide those messages that in fact, things are much worse than they are, elevating the person’s anxiety and then saying, you can’t do this without accommodations. There are times when that’s the case. Someone can’t do something without accommodations, but in the case of anxiety, that’s often not true.
Dr. Sharp: Yeah, just being cognizant of that and walking that line is important. Well, like I said, this has been another fantastic discussion.
Dr. Lovett: I appreciate it. I’m very happy to be the first return guest.
Dr. Sharp: Yeah. I am really happy to have you back and I’m glad that you agreed. And like I said, maybe we’ll do it again when you write your next book. Or figure out what processing speed tests are measuring, then I’ll have you back on. Thank you.
Dr. Lovett: […]
Dr. Sharp: That sounds great. I appreciate it, Ben. Like I said, this is a great book. I’ll have it in the show notes. I’ll recommend again, that folks check it out. It’s research-driven, which is I think important in what we do, and you have delved into this research thoroughly. So thank you. Thank you for your time and your thoughts.
All right y’all. I hope that you enjoyed that second interview with Dr. Ben Lovett. Again clearly well-steeped in the research around these issues. I certainly learned a lot. I hope you did as well. If you want to check out his book, I’ve got it listed in the show notes, and it’s a good one. So definitely check that out.
Otherwise, another shout-out for Q-interactive, the sponsor this month. Q-interactive is the digital test administration platform through Pearson. You can find out more at helloq.com. You can also sign up for any of their webinars that are coming up in the month of May including one with yours truly on May 9th about the cost-benefit analysis of Q-interactive versus paper and pencil tests.
If you haven’t checked us out in the Facebook group, we’d love to have you there. It’s The Testing Psychologist Community on Facebook. You can answer quick questions and then jump into the discussion about testing, testing batteries, case consultation, and the business side of testing.
So thanks as always for listening. If you have not subscribed to the podcast, I would be so grateful if you did that. It takes about 20 seconds and you can do that wherever you get the podcast from; iTunes or Stitcher or wherever it may be. I would love to have you as a continued listener of the podcast.
All right, y’all. Thanks so much. I look forward to seeing you next time with some more great testing content. Take care.