The BRIEF-2 ADHD form uses BRIEF-2 scores to predict the likelihood of ADHD. It is available on PARiConnect-PAR’s online assessment platform. Learn more at parinc.com.
Hey everybody, here we are back again with another expert clinical interview.
Today, I have two experts co-guesting on the podcast and they are both pretty incredible. Let me tell you about them.
Dr. Allyson Harrison is a clinical neuropsychologist. She’s currently the clinical director of the Regional Assessment and Resource Center at Queen’s University, Canada, which is a government-funded [00:01:00] center mandated to support post-secondary students through the provision of assessments. She also holds an appointment as an associate professor in the department of clinical psychology at Queen’s University. Her research is dedicated to issues of differential diagnosis in LD and ADHD.
Her co-guests, Dr. Julie Suhr is professor and director of Clinical Training of the doctoral program in clinical psychology at Ohio University in Athens, Ohio. She is a neuropsychologist who conducts research on psychological and neuropsychological assessment of many conditions including adult ADHD. Julie is a fellow of the National Academy of Neuropsychology and the society for Clinical Neuropsychology. And she is co-editor in chief of the journal of clinical neuropsychology and the incoming editor for the APA journal psychological assessment.
So, I am very grateful to have gotten a [00:02:00] little over an hour of time with these amazing women. As you can tell, they have a lot of experience around assessment, particularly with ADHD. And we are talking today all about the validity and how to conduct a valid ADHD assessment. This is a question that comes up quite frequently in The Testing Psychologist Facebook Community. And we dive deep on a number of topics yet somehow also managed to only scratch the surface with this topic.
So, if you’re at all interested in honing your ADHD battery and answering some of those difficult questions that we confront when trying to diagnose ADHD in adults and young adults, this is absolutely the episode for you.
So we talk about how clinicians often ignore or just don’t document diagnostic criteria in our reports when we’re diagnosing [00:03:00] ADHD. We talk about Julie and Allyson’s ideal ADHD battery. We talk about the role of PVTs in ADHD assessment. And we dip into some concepts like sensitivity, specificity, and predictive power in the instruments that we use and how that affects our assessments.
So there is so much to take away from this episode. I hope that you stick around and check out the whole thing because it is action-packed all the way to the end.
Now, if you’re an advanced practice owner who has mastered that initial phase of being in private practice and are looking to expand your practice, grow your practice, hire, have some additional streams of income, get more efficient, any of those things, I’d invite you to consider the Advanced Practice Mastermind Group that will be starting in June.
So, this is a cohort of 6 psychologists. It’s a group coaching experience[00:04:00] where we really focus on accountability and support and helping you reach those goals in your practice. If that’s interesting to you, you can check out thetestingpsychologist.com/advanced and get some more information and schedule a phone call to check out the fit.
Okay. Let’s get to my conversation with Dr. Julie Suhr and Dr. Allyson Harrison.
Allyson, Julie. Welcome.
Dr. Allyson: Thank you.
Dr. Julie: Thank you. It’s great to be here.
Dr. Sharp: Yes, thank you so much for being here. I feel like a lucky podcast host being able to land y’all as guests[00:05:00] because this is one of the… well, not one of the cases, this is the only case actually where I have some colleagues at a conference and they were texting me like mid-presentation that you were giving and they were like, “You got to get these ladies on your podcast.” So, I reached out and luckily you’re open to it. So yeah, super grateful for that. Thanks.
I always like to start with the question of why this work is important to you. Why this out of everything in psychology? Julie, do you want to start?
Dr. Julie: Well, actually, I do research on a fair number of things, but this particular area is important to me because of keeping psychology in neuropsychology. I’m a neuropsychologist by training. And I think sometimes those who focus on neurological disorders forget that there’s a lot of psychology in it. But also, I am a scientist and a clinician. And [00:06:00] I see it a lot in the real world. I see it in the referrals to our clinic. And so my research informs my clinical practice, and what I see in the clinic informs what I want to research.
So, this just is a very common presentation and I very commonly see it done in a way I think that’s actually harmful to the students and the clients who come to us and ask questions about ADHD. And so, doing research and informing my own practice by research was really super important to me.
Dr. Sharp: That makes a lot of sense. That’s one thing I miss in private practice. I feel like it’s harder to do research when you’re in private practice to some degree, and yeah, I miss it. I’m kind of jealous of y’all that get to do both and integrate them so well. How about you, Allyson?
Dr. Allyson: She stole my answer.
Dr. Sharp: Of course.
Dr. Allyson: I’m really fortunate because I work at a government-funded assessment center that’s based at a university. And so I get to[00:07:00] do teaching and I get to do research and I get to do clinical work. Like Julie, I research lots of different things, but jeez, I guess about 20 years ago, just from the term was coming out the test of memory malingering, we had to move away for two years when my husband was finishing his training. I worked at an independent medical evaluation center where we were seeing people who had been in motor vehicle accidents and were claiming they had brain injuries or concussions. And I started to see all these people that were exaggerating their symptoms and really all we had at the time were pretty basic tools to identify their exaggeration. And all the work I’d done before that was with students who said they had ADHD and learning disabilities.
And I started thinking, you know what? If it’s so easy for people who’ve been in car accidents and I’d watch the videos[00:08:00] that the insurance company had of these people who claim they’ve been in a horrible car accident and they’d have the video showing that they, in fact, staged at all and they have their friends and relatives being dropped off. And I started to think, you know what? I bet that this could also happen in the type of work we do.
And so that’s how I started researching it. And I was surprised to find out, oh my goodness, there’s a big proportion of students that I’m testing who are failing these tests. That’s not good. And so that’s how I got into it.
Dr. Sharp: That’s a great story making that connection. I know we want to assume the best of people, right?
Dr. Julie: That’s one of the problems. I think this is something that one of our dear colleagues called invalidity shock. It threatens our belief that people are always going to just tell us the honest, accurate truth. And yet we’re scientists and we study human behavior and we know that’s simply not the case, whether it’s deliberate and conscious,[00:09:00] or the M-word malingering or whether it’s not necessarily conscious, but that doesn’t mean it’s accurate and interpretable. So, it’s really challenging because we do want to assume that, or at least assume that we can tell just because we’re such wonderful clinicians that we know when somebody is not being accurate and that’s not true.
Dr. Sharp: That’s not true. I know. That’s terrible.
Dr. Allyson: Yeah, it is. And I know that Malcolm Gladwell’s got a book out now called Talking to Strangers. And then there, he talks about this theory called Truth-default theory which is the same thing. It says that we we have this default to saying, we believe you. And that’s good when you want to be an empathetic clinician and you’re treating someone, but it makes it a lot harder if you’re actually having to determine whether or not this person is really answering accurately or whether the data you’ve got is a valid representation of what they can do.
Dr. Sharp: Right. I’m going to[00:10:00] go off-script right off the bat and ask whether being in this area of research in particular, do you feel like this has affected the way that you relate to people in general in terms of trusting others, looking for times when you might be being fooled or am I reaching there?
Dr. Julie: No, you’re making me chuckle because all my life I’ve been accused of wearing rose-colored glasses about the world. And I still do. I tell myself and my students because I also teach assessment in half a decade is, it is a delicate balance of knowing I need to be empathic and I need to understand how you’re perceiving your world at the same time as I need to wear my science hat and recognize that what you’re telling me might not be accurate.
Again, it doesn’t mean it’s deliberate. And that’s I think one of the keys to help people overcome that barrier is Kyle Boon, another major researcher in [00:11:00] malingering, points out that there’s another deception, which is when people are deliberately trying to fool the other. And once they get out and get back in their car, they’re perfectly functional and drive away and don’t have memory problems or whatever.
And then there are people who are self-deceptive because they’ve become convinced that this is the explanation and that doesn’t make it any more accurate. It’s hard to wear both those hats, but with practice, you can. I need to understand what you’re telling me and I need to hold all these possible diagnostic hypotheses in mind because I know a lot about psychopathology, I know about the causes of all of these problems. And one of them is that this isn’t accurate and that perhaps you are also deliberately not being accurate, but also that you may be aren’t necessarily deliberately being inaccurate, but this is the way you perceive it. It’s just distorted by other elements.
Dr. Allyson: Yep. I agree. And really, to be a good clinician, [00:12:00] you have to know what the differential diagnosis is for any of the conditions you’re diagnosing. And one of the possibilities is that for whatever reason, people aren’t reporting accurately or aren’t behaving accurately. And as Julie said, it doesn’t mean that they’re necessarily doing it deliberately, but it still invalidates the test data.
The story that I always tell is I remember seeing this one poor little woman when I was at this independent medical evaluation center. And she had been working three different jobs. She was of Italian background. Her kids were all. She did everything for the kids and the husband. She cooked and cleaned. And then she’s in this little five mile an hour fender bender in a parking lot. And she doesn’t have to go to any of her jobs.
Her kids start picking up after themselves. The husband leaves her alone. They import a cousin to come home over and do all the housework and cleaning. She’s getting massages three times[00:13:00] a week and having nice interactions with people.
So, where is her motivation? When I’m testing her to see if she has to go back to purgatory, where’s her motivation to try her hardest? And I am sure she didn’t sit there in the morning going, okay. Now, how can I take this? But by the same token, why is she going to say, “Yeah, look, I can do all those things again. send me back.”
And so I think we see that a lot especially the younger students who have been over accommodated all through elementary school and high school, and then they come for an assessment to see what accommodations and get on the SAT or on the MCAT or whatever. Where is the motivation to show that they actually can read that pretty quickly?
Dr. Sharp: That’s a great question.
Dr. Allyson: As opposed to, can I get double time?
Dr. Sharp: Sure. You’re raising so many good points. I know we’re going to dive into all of these things. I’m especially curious to talk about kids being over accommodated and[00:14:00] coddled if you want to use that word or not, but all that can of worms.
Let’s start as close to the beginning as we can. We’re really focusing on the assessment of ADHD, right? We’ll put a little bit of a restriction on this conversation.
And one thing that y’all talk about a lot is how clinicians in general, when they’re assessing ADHD and diagnosing ADHD, just don’t actually pay attention to the diagnostic criteria. I’m very curious about that. Can you talk about that a little bit?
Dr. Allyson: Okay. I’ll start. Because it’s interesting. There’ve been a number of studies that have been published recently, showing that when you look at diagnostic reports that have been done on students where they diagnose ADHD that the majority of clinicians don’t document that all [00:15:00] of the DSM criteria were met. Julie and I both review documentation submitted to the universities and we both review documentation submitted to High-Stakes Testing Agencies.
And what we find is it’s not simply that clinicians just didn’t explain in their report that, oh yeah, I actually documented, I just didn’t write it down. They don’t. That the majority of clinicians, all they do is they document that this person currently says I have symptoms. And that’s great, but they don’t document all the other criteria that DSM says you have to demonstrate in order to make a diagnosis of ADHD.
So we all see reports where the clinician hasn’t actually looked at any objective data. They’ve just taken the client’s self-report. And on the basis of that, they assume that this person has [00:16:00] been impaired in two or more major life areas. They’ve got a history of it.
And then we get all of the data and see that, in fact, this person who said, oh, I struggled to do well in school. I was always the last one to finish. I did poorly on the SAT while they’ve got 1500 on the SAT. They were in the top 10 at their school. They weren’t average. They got a GPA of 4. And when you look at… these are also people who tend to be perfectionistic or they’re anxious, depressed.
And so they haven’t actually looked at those other things to say, what are the other potential reasons that somebody could be complaining of these symptoms now?
And with COVID my clinic and I think yours too, Julie, we’re just overrun right now with students who all think they’ve got ADHD.
Well, they’ve had a year of lockdown[00:17:00] where they’re sitting in front of probably the most addictive device in the whole world and they have to watch all their stuff online. And it’s really easy to get distracted and to get bored and restless. And they’re not sleeping well. They’re stressed. They’re depressed. Sometimes they’ve got lots of other people around them that are distracting them. But they don’t stop and say, oh, I wonder if all those things could be causing my client, they’re distractions. They just hand them a questionnaire and say, oh yeah, Ben, you’ve got the symptoms ADHD.
And I think that doesn’t do these students any service because you’re not really addressing the real causes of their problems. And so you’re not giving them away to self-regulate and to overcome the problems they’re having or to normalize them and say, you know what? There are lots of people who are in this same boat right now, and I’m not denying that you’re having problems, but [00:18:00] we can do things to help those go away.
Dr. Sharp: Right.
Dr. Julie: Which goes back to my very first point that our research is focusing on the accuracy of the data we get in these kinds of evaluations and the fact that there’s a lot of inaccurate data that’s leading to inaccurate diagnosis. But at the core of it is also because that is helpful. And too often, I think we both encounter and our sense in the world is some of these clinicians feel like they’re advocates for their clients.
They think they’re helping because the client came in saying I pretty much, and Ally you can back me on this. The referral question is, I think I have ADHD, or I want to be diagnosed with ADHD. They already are coming in with this from the world, the perception, the commercials on television that tell them, are you distracted? Then this drug is one you should take.
And so they don’t know that there are lots of other questions, but that’s the clinician’s job. But if the clinician[00:19:00] thinks they’re an advocate, oh yes, you’re saying that you’re failing out of school. Well, get the objective records and it turns out that means they got a C in something and expect all A’s. Or they could be objectively failing. But there are so many reasons.
There are so many things that are part of the differential diagnosis, many of which are treatable, many of which could be managed. Helping improving study skills. Addressing the depression. Addressing the anxiety. There are effective treatments for those. And that is helping the person. That is being therapeutic to address all.
Oh my goodness. You went to college and you started drinking. And now you’re drinking all the time. That’s a problem. How about we get your help with that? And maybe that’s why you’re not going to class and not studying. Or nowadays with the legalization of marijuana everywhere, it’s I don’t know why I can’t study. I mean, I smoke in the evening, and the next thing I know it’s morning.
[00:20:00] We’ve landed on a culprit. We’ve landed on a contributor to your academic struggles, but they fill out an ADHD questionnaire. They mark the items. And unfortunately, that often is the result is a diagnosis that’s not actually addressing the contributing factors.I’m going to take issue even with the DSM because it says, to rule out other causes, but they’re all other diagnoses. And a lot of times it’s not a diagnosis. A good developmentally-oriented bio-psycho-social approach means you think, what are all the contributors even though there’s not maybe a diagnostic label to put on them? Therapeutically, we need all of those to know how to best offer resources and interventions, et cetera that could be helpful.
Dr. Sharp: Right. Oh my gosh. So many questions in here. So the first one is, do you all have any[00:21:00] hard and fast rules or rubrics or anything along those lines when someone is smoking a lot of weed because that is… I mean, I’m in Colorado. That’s the direction think we’re all headed. And this just came up in our consult meeting within our practice the other day. One of our adult assessment specialists was like, everybody smokes weed all the time. And what do we do with that? How do y’all think about that in this context of valid assessment?
Dr. Julie: In my US setting, we used to have very high base rates of alcohol use. That’s actually started to dwindle as the rates of pot smoking have gone up. And the students all are convinced that it has no cognitive side effects. Absolutely none. And in fact that it’s an effective treatment for name your disorder.[00:22:00] And there’s no evidence for most of those things. But because they’ve read it or they’ve heard it anecdotally, that horrible internet that they go searching on, they’re very convinced. They’ve made a false correlation between what they did and how they feel.
And so we end up spending a lot of time going over the research evidence for them and saying, well, you know what, actually studies show that probably it’s having an effect on you and it’s not a positive effect on your academic functioning. They’re surprised because that’s not something that they’ve heard. So, trying to take an educational approach when we’re giving feedback is one of the things that we address.
Dr. Sharp: Okay.
Dr. Allyson: And really all the studies of the long-term impact of the use of marijuana, the only thing that they all agree on is it leads to motivation. And so you’ve got someone who’s smoking, probably like you Julie, we see people who admit[00:23:00] I’ll ballpark high. And I say something like, in a week, how much would you smoke? Or in a day, would you go through like 2 grams a day? I don’t think, or what, I think that’s about right.
But they have just been telling me, “I wake up in the morning and I can’t get going. I’m not really motivated to study. So, I go on Instagram and then I make a little cat face on my pictures and the whole day has gone by. I get anxious because I’m not doing my work so that I have a few more. I spoke badly and then I feel better.”
Just see, there’s a pattern right here. And I think that there are a lot of them are using it. And I think this goes hand in hand with the fact that we’re now seeing a generation of kids by and large in general are the most anxious, depressed population of kids that have ever come to college.
And so they’re anxious all the time. They’re depressed all the time. And they used to use alcohol[00:24:00] to self-medicate. Now, they use marijuana to self-medicate. It’s easy. It’s legalized. They can get it anywhere at any time. And because they don’t know how to sell shoes, they don’t know how to regulate their own internal emotions.
Dr. Sharp: Right. Oh my goodness.
Dr. Julie: Especially, anxiety does seem, at least in my clinical impression that the thing that has just spiked over the last 10 years or so in our students.
Dr. Sharp: Of course.
Dr. Allyson: You look at the national college health survey and it shows just that the rates of anxiety and depression, and even the best indicator suicide rates. Suicide rates have just skyrocketed especially in young girls. Young girls, in particular, have been hit really hard. And so again, we know, anxiety, especially chronic anxiety causes you problems with attention, concentration, focus. I always say to people,[00:25:00] that’s a bit like you’ve got six different virus scan programs running in the background of your computer, and it’s taking up a lot of cognitive space. It’s taking up a lot of that brainpower.
So when you try and convince your brain to sit down and focus on some boring task like schoolwork, it says, no, no, no I’ve got all these other things I’m really worried about. The world could end, then you want me to pay attention to this pack.
Dr. Sharp: That’s such a challenge. Yeah.
Dr. Julie: That one’s fascinating too when you think about anxiety because probably the last thing you want an anxious person to do is to stimulate their CNS. And so stimulant medications, especially for those who are having physiological symptoms of anxiety, just don’t really seem like your best option. But again, what they’ve heard is this will help me study and, or they’ve borrowed some from a friend. And it helped them stay up all night to cram in for the test. And so they think, see it’s effective for me. It’ll work.
[00:26:00] And so, sometimes when at feedback, when they’ve really been reluctant to think about their own anxiety or conceptualize their symptoms as anxiety, we talk about what it felt like. And that there’s not a benign substance to be on a stimulant medication.And again, this goes back to being careful, to look at all these differentials and what might be treatable. And so we do a lot of feedback talking about let’s start with the anxiety. Let’s get some good empirically supported treatment for that anxiety and see where you are. We can revisit later, but let’s first tackle something that’s in our expert opinion, clearly contributing to your academic struggles and just your mental health generally, before we throw a med on there that is actually going to ramp up your physiological system.
Dr. Sharp: Exactly. Well, I think that just speaks to the importance of doing a pretty thorough assessment where you’re looking at all these different factors, right? [00:27:00] So, I do want to talk about the assessment process.
Before we totally shift there though, we opened by talking about these diagnostic criteria that aren’t well documented in the reports. A lot of these reports are just missing documentation. Are there any particular criteria that you see that folks are just not documenting? I mean, is it like the individual symptoms or the lifetime history or the rule-outs? What are people not paying attention to and documenting?
Dr. Allyson: The rule-outs.
Dr. Julie: I was going to say yes to all of them, but the rule outs, also impairment. I think that impairment is a big problem as well.
Dr. Allyson: And they’re taking, sorry, Julie. They’re taking self-report, but if you’ve got somebody who’s already highly anxious, their self-report is not necessarily accurate or reliable. And I had someone[00:28:00] I just saw recently, Julie knows this story. And if I’d just gone on her self-report, I’d say, oh, you are really impaired. I mean, everything across the board. She can’t do it. It’s terrible. It’s awful. Well, this is a girl who is extremely perfectionistic and perfectionism and procrastination go hand in hand.
So I’m talking with her. I said, all right, you say here that you’re severely impaired in academic functioning, but I said, I’m really confused because I looked at your transcript and you’ve done really well. And she said, well, I go into an exam with a 100% and I come out with 98%.
But for her, she had a severe impairment in academic performance. And this is the problem with self-report. So even with symptoms, you get people who complain of symptoms, but I would say very rarely, Julie, I don’t know about you, very rarely do I see reports where, in[00:29:00] adults, where they’ve asked someone who knows the person well like their parent or their spouse or their significant other to rate them.
And often when you do, you find that this other person doesn’t really rate them as having a lot of symptoms. And then the psychologist because they’re in this mindset of, well, I have to give the client what they pay for. I’ll see reports where they dance around and stand on their head trying to say, well, it’s because this other person doesn’t really know them anymore or rather than stopping and saying, I wonder if maybe this is a person has some other reason for why they’re reporting all these symptoms.
Dr. Julie: Yeah. And that’s a tricky one because of what I also see though are collaterals who are the parents who continue to snowplow because my child is on academic probation and by gum, I’m going to make sure that they don’t get kicked out of the university. And so they also will endorse. This is the value of records.
[00:30:01] So in our busy time and with at least here in the US billing for our services, it can be hard to wait and get the objective evidence of impairment. But so often I’ve heard clients, as well as their parents, say, oh, they practically failed out of this. Or they were put in the lowest reading group. And then we get their school records and that can take time to get. And what we see is there’s just no evidence of that at all. They were at the 10th percentile in their proficiency exam, whatever that might’ve been. And we get all the records and there’s just no evidence that was the case. My favorite one was they were at the 10th percentile for their very elite boarding school one year, one semester in math.What was told to me was they were severely impaired and behind their peers. But when you looked at the national norms, they were like 80th percentile. [00:31:00] So it had to do with anchors. And again, it goes back to the point that this is not always intentional. It’s not the M-word malingering. They are perceptions.
This perfectionistic person that Ally was talking about was probably very distressed because she thinks she should be 100% all the time. That’s just not realistic. But in her mind, that’s what you’re supposed to be. And that’s normal. Therefore, I’m very abnormal. And so I’ll put myself at the 99th percentile in impairment and on symptoms because there’s no anchor. The anchor you will only have is yourself or your cohort.
And when we look at self-report, that’s the problem. They’re using that as an anchor. And there’s no sense of I have to try harder than all of my friends to get the same grades. I don’t even know how to anchor that. I know after years of research, I have no idea how long it takes them. They might be lying as far as how long it takes them. It’s a perception [00:32:00] and it’s often very inaccurate.
Dr. Sharp: Right. Well, and it’s contingent on a bunch of other people, self-reporting the right, you know, and who knows what that looks like.
Dr. Allyson: And that’s a big gap. Everyone says, “I’m the only one that wasn’t getting As in the life sciences courses.” I’m just like, “Not true.”
Dr. Sharp: Right. It’s funny. It’s like the inverse of a, what’s that statistic? Something like 8 out of 10 people call themselves above-average drivers or something.
Dr. Julie: Yeah. Well, it’s the above-average effect. I mean, we social psychologists have long ago documented that everybody thinks they’re above average and therefore if you’re average, you must be impaired. And that’s just human nature. But why are we then diagnosing it? Why on earth are we making that the diagnosis?
Dr. Sharp: Right.
Dr. Allyson: I was just going to say, I think that that is another thing, I don’t know, Julie, if you would agree, is that I think that that often we’ll see kids who say,[00:33:00] oh, I heard to or their parents too. Oh, they had to work really hard to get As. And then you actually look at what their IQ is, their IQ is 98. So it’s the 48th percentile.
And I sit there going, yeah, you wouldn’t have to work hard to get A’s. But that’s not abnormal. But instead of giving them some realistic feedback about, yeah, there is a reason why you have to really work hard to get A’s and why you’re struggling to keep up with your peers. And there’s really an all-high-powered course that you’re in. I think because clinicians want to be helpful and want to give the client what they came in for, they’ll give them a label which again, doesn’t help them choose appropriately the types of courses into which they should go or the things that they’re going to be better at.
Dr. Sharp: Right. Do y’all know of Dr. Maggie Sibley? She’s in this area as well. She was on the podcast, I don’t know, two years ago. And she said something that still sticks[00:34:00] with me, which is with that impairment piece, theoretically only about 5% of people are going to have ADHD. So you can draw from that. Like they should be more impaired than 95% of the population. And that puts it in very stark terms and really makes you think, yeah, what are we talking about when we say impairment? Like this really it’s got to be the real deal. It’s not just, I have to work harder than my peers. You got to have a solid body of evidence there.
Dr. Julie: Yeah. And yet there was a recent study that surveyed elite, really highly competitive universities in the US colleges and 25% of their student body was getting accommodations for some disability. And often it was a label of ADHD or LD. That’s 25%. That’s a quarter. And it’s an elite university. So how did we get there? I mean, when you go back to the research on psychopathology[00:35:00] generally, and the research on a neural developmental disorder, such as ADHD, that at most should be 5% of the population.
And that we know from longitudinal studies of children carefully diagnosed that it makes it really hard for them to get into the college setting, not impossible but hard because of the nature of the disorder that 25% of your student body would have that impairment? Again, not all of them were for ADHD. That doesn’t make any sense. And so what it suggests along with other data that there’s very likely an over-diagnosis of this condition.
Dr. Sharp: Yes. Oh my God. There is so much to get into. I am just trying to figure out where to go next. Let’s see. So I do want to ask this question, maybe two tangential questions, but I think hopefully we can bring them back. One is that I think there is a relatively strong [00:36:00] movement in our field, especially in private practice but maybe hospitals as well, to shorten our reports and make sure we’re being as efficient as possible. And so I’m curious, how do you reconcile that with what I’m hearing from y’all, which is we need more documentation of these diagnostic criteria and almost like more justification for our clinical decision-making. Do you all have thoughts on that at all?
Let’s take a quick break to hear from our featured partner. BRIEF-2 ADHD form is the latest addition to the BRIEF family of assessment instruments. Using the power of the BRIEF-2, the gold standard rating form for executive function, the BRIEF-2 ADHD form uses BRIEF-2 scores and classification statistics within an evidence-based approach to predict the likelihood of ADHD and to help determine the specific subtype.
It can also help evaluators rule in ADHD and rule out other explanations for observed behaviors. Please note that the BRIEF-2 [00:37:00] parent and or teacher form scores are required to use this form. The BRIEF-2 ADHD form is available on PARiConnect- PAR’s online assessment platform. You can learn more by visiting parinc.com\brief-2_adhd.
All right, let’s get back to the podcast.
Dr. Allyson: Yes. And write an abbreviated report, but still do a very thorough assessment. I mean, when we see people, we want to get all of their elementary school records or high school records. In Canada, we don’t have exams like the SAT which makes it a little tougher. And we ask we have a retrospective questionnaire we send to the parents to have them rate what their memories of their child were before age 12.
So we do that all the time. And it’s really easy to just have[00:38:00] bullet points that sort of say, here’s what we looked at. Here’s what we found. So I don’t think that the idea of having a shorter report is incompatible with showing that you’ve demonstrated all of the diagnostic criteria.
I think the harder part is if you go into the assessment was the mindset of I’m an advocate and I want to give the client what they paid for. Then we know that you’re going to have tunnel vision and bias, and you’re not necessarily going to pay attention to that. So I think the big point is, and Julie was saying, you have to really keep your mind open to all the possibilities, all the other things that could be causing this, and be like a detective.
Dr. Julie: This is a bigger problem, Jeremy, as you say it. It’s a movement generally, which is, I think unfortunate for all of the mental health and maybe not unique to mental health, but that’s, of course, what we’re in because I teach an[00:39:00] integrated primary care class and, of course, the emphasis here is on really super-fast screening or oh, can we get it down to the PHQ-4? Can we have the report?
But the point is those are screeners? It’s supposed to be triaged-level work. Screen, and if it’s positive, it should lead to something else, not a diagnosis because it’s a screener. The whole point of screeners is to be highly sensitive and not miss anyone who might have it but at the major expense of specificity. Unfortunately in that setting, what happens often is here’s our quick little depression screener, and bam, they’re on an antidepressant because people are being forced to move fast.
There are arguments that well, if we didn’t ask it all, we wouldn’t know. And from a public health perspective, that’s true. But from the accuracy of diagnosis perspective, it should be great. If you want to be faster, you triage. It’s a screener. And if someone’s positive on the WHO [00:40:00] instrument for ADHD, you don’t give them a diagnosis. You then say, gosh, this means I should do more of an evaluation. But if they’re negative, okay, we can be done.
It’s really, can we speed up and be more efficient by a triage. But to take screeners and call that your assessment, it’s to do harm. It’s to misdiagnose and to miss correct diagnosis that could actually improve somebody’s quality of life.
Dr. Sharp: Right.
Dr. Allyson: Yeah. And I think people forget that even assessment instruments like the CAARS- The Conners’ Adult ADHD Rating Scale or things like that have a huge false positive rate. We did a study and we found it’s got an almost 80% false-positive rate. So if you score high on that, that’s really nice. There’s only a 20% chance that you really have ADHD. So don’t just rely on those things irregardless of whether or not someone’s misrepresenting or reporting non-credibly.
Dr. Julie: [00:41:00] Yeah. I said the screeners are the world of efficiency and then really cost us in diagnostic accuracy. And unfortunately, psychometrics I think, sometimes I think people learn and then they forget about, and we don’t take the time to do continuing ED and refocus on what does that mean? And how did this work? And what does that really mean when the score is positive versus negative? And what are the accuracy statistics? And those are the CE credits we all should be getting when we’re out there in the field is reminding us how they all are.
Dr. Sharp: Right. Yeah. While we’re on that, can we do just a quick primer on sensitivity versus specificity and base rate? These are all important concepts that we do lose track of. Can we refresh on that a little bit and why that’s important for our assessments?
Dr. Allyson: Yeah. Sensitivity is really saying, I already [00:42:00] know that my patient has this disease. I already what’s the chance to test is going to show that my patient has. And specificity is I already know the patient doesn’t have the disease. What’s the chance that the test is going to show that the patient doesn’t have a disease. Those are lovely questions. Those are not the questions we get asked as clinicians, right? You don’t know until we get our mind-reading in the mail, we can’t do that. So, that’s why you need to look at the positive predictive value and negative predictive value of tests.
And most test manuals don’t tell you that. And most clinicians don’t think about it because that’s saying, that predictive value says, I don’t know what my client has but I just got a positive result on this test. What’s the chance that my patient actually has this problem. And the negative predictive value says, I just got a negative test back. What’s the chance that they really don’t have that problem. And so those are the things you need to know.[00:43:00] Sensitivity and specificity are not sensitive to the base rate, but positive and negative predictive values are.
So the more that a condition is rare or unusual, the harder it is to identify it. The rare it is, the easier it is to usually have a screening test that says, phew, you don’t have it. But it’s a lot harder to accurately identify something when it happens infrequently. So most of the studies that have been done for a lot of the tests that we use, have a group who have ADHD and don’t have ADHD and they’re equal.
Well, it’s pretty darn easy to pick up who’s got ADHD and who doesn’t when A) you’re looking at people who are non-symptomatic and comparing them to people who are, and you already know what their diagnoses are. And B) when you’ve got a 50/50 chance of picking them correctly, just by chance.[00:44:00] So those are the things you really need to be aware of as a clinician. And I certainly see in a lot of the reports I read. I think clinicians have sometimes forgotten that. Julie?
Dr. Julie: Yeah. To emphasize the point that was hopefully not lost in there is that yes, there’s lots of data on the instruments that we use out there in practice. But the vast majority of them, even when it comes to sensitivity and specificity, are relative to asymptomatic groups, which Ally said. But that means the data that’s out there say on most of the ADHD instruments and self-report instruments is, this can identify ADHD versus non-clinical controls.
Well, that’s not particularly hard to do compared to the major, highly comorbid, and high base rate differential diagnoses. That’s a lot harder. Okay. [00:45:00] So it turns out depressed people endorsed tons of those same symptoms and anxious people just to name two of the highest base-rate conditions. And so you can’t even get sensitivity and specificity data that really help us say this disorder versus another.
And we’ve known that for years in psychopathology research. When we think of structured diagnostic interviews, everyone says, oh, they’re wonderful. Well, no, again, the data is all a disorder versus no disorder. It’s not depression versus anxiety versus psychosis versus which type of anxiety. It’s disorder, not disorder.
And so, we barely have good sensitivity, specificity knowledge versus then considering base rates of the disorders. But on the topic of base rates though, and thinking about other aspects of an ADHD evaluation, even though cognitive tests aren’t required at all in the diagnostic criteria, they do speak to impairment. That’s a degenerative way to document current impairment if you didn’t have [00:46:00] say school records. Sometimes you really should try to get the developmental history.
Dr. Sharp: I want to ask you in a bit how you get those school records. But we can pin that for now. But I want to remember that.
Dr. Julie: Not easy. But the base rate issue comes up then because people will say, oh, well, great. Part of my differential might be maybe they just have a generally low cognitive ability, or perhaps they have a learning disorder that we didn’t diagnose yet. And so they might give us a psychoeducational battery with tons and tons of DVs.
So you’ve got tons of sub-test scores. And then you’ve got index scores. And then you’re comparing index scores to each other and sub-test scores to each other and looking at scatter and base rate of impairment. And people seem to have this belief that we’re going to be high average on everything if you’re high average on only one of them. Or because you got one superior score, you’re going to be… it turns out no,[00:47:00] that’s not true. The data actually show that it is absolutely normal for people to score below let’s say the 16th percentile on at least one. So there was a study that Brooks…
Dr. Allyson: Brenda and I resent it.
Dr. Julie: Yeah. They took the normative data from the WAIS and then the WIS and WIMs together because they’re co normed, and just said, let’s look at the actual normative database which people need to remember is a supra, normative database. They excluded a lot of conditions to be allowed to be in the normative database.
So they’re really not a good representation of the general population either in some ways. But they look to say, okay, well, if you give this giant battery with this many DVs, what are the odds that you’re going to score below the 16th percentile? One standard deviation below on something. Over 47% of the standardization of the normative database, [00:48:00] people scored below 16th on at least one of the indices.
So, the problem is we hear base rates and we think of the base rate, and oh, 16th percentile that’s impaired. But that’s isolation. It’s more like you’re doing a hundred different T-tests at the same time. And then you need to be thinking, okay, well, what are the odds that one of them is going to pop?
But again, test manuals don’t publish this information. They give you nice tables. Oh, look, the difference between this sub-test and this is sub-test is rare. But that was considering that two comparisons in total isolation, what are the odds that when you look at all the comparisons you can make that one’s going to show up as different? It’s no longer rare. It turns out it’s very common.
And so a very good thing for clinicians to also remember is scatter is not unusual and high and low comparisons are not unusual. And go look and see what that[00:49:00] is and think beyond the individual unique comparison data, but really that these studies are showing that’s not an unusual thing to happen. What does it look like with regard to the other data you have on them like their grades, et cetera?
Dr. Sharp: Sure. So what would it take, sorry, I just want to jump in and ask, what would it take then for y’all to take notice of scatter or score differences? Is there a threshold where you’re like, “This is more than we would expect. I’m going to actually pay attention to this as a relevant data point?”
Dr. Allyson: Well, this is on the same point that Julie was just saying. Did a study just a year ago where she looked at the normative data set of the Woodcock-Johnson and found that 61% of the general normative sample had at least one cluster score. So not [00:50:00] just one lone sub-test, but one cluster score like public speaking basic reading below the 25th percentile. So more than half of the normative non-disabled sample had at least one cluster score. And she implies that if they just looked at subtests, the number would have been way higher.
So just having 3,4,5 different subtests that are below average if you’re giving a battery of a different test is not unusual. What I certainly look for is, is there consistency. So if you have a neurological reason for your impairment, it should apply every time you’re asked to do that activity. So, if you really have problems with decoding, you should have it everywhere not just on one sub-test. And this is a game. This isn’t even taking into account whether or not you’re trying your hardest.
[00:51:00] Dr. Sharp: Right.Dr. Julie: Yeah, consistency is key for me as well. And what I’m always reminding my students of is I think we put too much weight on when we have numbers. And cognitive tests and even some self-report instruments look great. We’ve got a score. We’ve got a confidence interval. We’ve got, oh, numbers, but we also know what’s in their real world. And so we ask, and we don’t just get, even if we can’t get the school records, which we always try to do in my setting. We ask if they say, oh, I failed this, or I was horrible at it.
I’m like, okay, give me the score. Even if it’s self-reported we do know that could be accurate too, but it’s like, the thing we learned the rule of thumb, when people say, oh, I just drink a little. Like no, no, I’d go back and say how many drinks and how many hours on how many nights of the week and days. But you got to have some numbers to go with that to remind yourself, okay, well, wait, they never got anything,[00:52:00] but As and Bs, and now I’m getting the first percentile on reading achievement? Something doesn’t make sense.
I give that example. I saw you, the audience can’t see the face you made, Jeremy, but we do see that. We see that fairly often in individuals, especially if they’re motivated to get extra time, for example. And processing speeds. So, gosh, here’s, 1st percentiles, 2nd percentile scores on all these psychomotor processing measures, but you’re driving a car. And I know from the data on the ecological validity of the tests that can predict driving accidents. I’m a little worried about you being in a car now because your scores don’t make sense. And yet you also did all these speeded high-stakes tests in your educational record, and you were 75th percentile, which means you’re higher than 75.
That doesn’t mean you’ve got 75%, right? It means you’re 75th percentile relative to the comparison group,[00:53:00] which wasn’t normative either. It’s an elite group. So this doesn’t make sense now. So even if I saw consistently slow performance across a lot of measures of speed, I might question that. I also would go beyond what the label was.
So Woodcock-Johnson might call this one a speed of test, but these other tests have time limits. And so how did you get all the items right and get superior on this thing where each item is timed, when I’ve got the first percentile over here on a task that’s labeled psychomotor processing speed? Again, it doesn’t add up. It doesn’t make sense as a neuropsychologist for those things to happen. So consistency is key in the tests but also with that external real-world data.
Dr. Sharp: Right.
Dr. Allyson: I was just going to say that the other thing too is, what I see a lot is psychologists who did what I call the Texas Sharpshooter approach to assessment, where[00:54:00] instead of having a hypothesis or having a number of hypotheses about what could be causing this, and then searching for information patient to either support or disprove it, they just give every test they ever knew. And then they go, oh, look, there’s a low score there. And there that’s the disability.
And the Texas Sharpshooter Fallacy is where this guy’s in the back 40 shooting on a barn door, and then when he’s finished, he gets out a bucket of paint and he finds the cluster of holes that are closest and he circles and he says, that’s what I was aiming for.
Dr. Julie: Yeah. That’s very often how I’ve seen those reports too when I’ve done expert reviewing is, how on earth did you put all those things together,? But then if that were true, what about this and this and this and this part of it? There’s your disconfirming evidence. And I think too often we do get that confirming. We know that from research on diagnostic decision-making in medicine as well as in psychology that we too often search for that confirming evidence and then we don’t even look at the other[00:55:00] evidence that would have been disconfirming.
So out of a 10 battery of working memory tasks, one score is low. Aha. Like, wait a minute though. What about those others? Like, it’s a whole picture put together that’s necessary. And if that one low score we could expect one isolated low score in most of the population of the normative sample, then how do we put weight on that? That doesn’t make sense.
Dr. Sharp: Right.
Dr. Allyson: And sometimes it can even be an error. I mean, we had one case, we were doing an audit of files and we pulled out one from one of the people who does work for us. And she’d miss scored the Woodcock-Johnson reading fluency. She’d put errors in the number correct column. Yeah. She’d mix them up. So the number of correct, she had zero and the number of errors which was what the number correct was. And so just on the basis of that one score, she diagnosed a reading disability in this kid.[00:56:00] And when we said, you’ve made an error, this is actually a score that’s above average, she didn’t want to admit it because she said, well, she’s already got a disability scholarship and she’s getting accommodations.
But then we also found with the new Woodcock-Johnson, because you can’t score it by hand, it’s all in the cloud. And we discovered because I have a one-person research assistant who enters all the data in, she enters in the raw scores and the actual scale scores into our database. She came to me and she said, there’s something weird going on here because this one person has a score of, I can’t remember 52, let’s say, and it says, it’s at the first percentile, but we’ve got other people in the database who’ve got a score there’s one above or below and it’s average. So we went in and of course, every time you do it, you have to use a test credit up.
It took a year before Woodcock-Johnson would finally even look at it[00:57:00] because we found this a few times and then they wrote us back and said, there was a computer glitch. We fixed it. But they didn’t write to everybody who bought the Woodcock-Johnson (WJ IV) to say, by the way, there was an error and sometimes it was giving you something below the first percentile that was inaccurate.
Dr. Sharp: I have a feeling there are a lot of clinicians out there who just got a lot of validation for something that they probably thought they were crazy if they noticed some of this on their own.
Dr. Julie: And to me, this goes back to you got to look under the hood. I’m always reminding students and myself, wait, go back. Does this make sense? You double-check, triple-check your scoring. I think computerized scoring things make people feel like, oh, well that’s easy. They’ll do the scores for me. But there’s still room for human error. Go back and double-check. And then if you get data that doesn’t make sense, go back and see where that could be? How[00:58:00] did this index score happen if these were the sub-test scores? So something’s not right here.
You got to go back and look under the hood and make sure that your data is accurate. It’s just going back to the issue that you raised actually of this is time-consuming, right? And it means our evaluations are long and nobody wants to pay for them, but we’re not taking blood pressure here. No offense to people take blood pressure, but it does take a long time to measure human behavior. It’s actually really hard.
Dr. Sharp: Certainly.
Dr. Julie: Rather than just caving to people who think we should be more efficient, we need to be advocating we need to do this right because it is harmful to people to give them the wrong diagnosis.
Dr. Sharp: That’s such a good point. Well, and what I hear you saying is we got to: 1) do a comprehensive evaluation with many points of data and we have to be great detectives. That combination is really what’s necessary, right?
So let me ask, if you had[00:59:00] to just lay it out there, what would be your ideal ADHD battery if you had your pick? And you don’t have to name specific tests, but just to like, are we doing cognitive, academic, performance validity, like what’s the ideal battery here? And that includes any records?
Dr. Julie: Well, I wanted to start there because that’s actually something we haven’t gotten to in our time together thus far is research shows that one of the highest base rates differentials that need to be considered when an adult especially is presenting with potential ADHD, especially for the first time is in fact non-credible presentation. And that can be non-credible responding. So something on self-report measures or even structured diagnostic interviews or non-credible behavior which we might see on the actual cognitive tests.
And there are decades of research showing that there are good measures of both [01:00:00] performance validity called PVTs or symptom validity SVTs that have been well validated in the larger literature on malingering, especially with a lot of neurological conditions. And most importantly, showing that they are specific. Meaning, you don’t score low or you don’t score high in the case of self-report and it’s accidental because people with really severe neurological impairments do fine on performance validity tests, for example, and they don’t report these implausible symptoms.
So there is less of a history of that in ADHD. Although as Ally mentioned the way at the beginning, she noticed, “Hey, this is in these other IME cases. Maybe this is an ADHD.” And so there has been a couple of decades of research on that showing again, that these measures are valid for use in adult ADHD, that they are sensitive to it although they are also specific, which again, we can do PPA and MPA. But the key is [01:01:00] they’re not going to be false positives.
And I think that’s something a lot of people worry about. I hear that all the time. Well, if someone can’t pay attention, then they probably can’t do this attention word memory tests. People with a really severe neurologic impairment who live in institutions can do this task. That’s not an explanation.
The research does not show that some explanation. So top on my battery is the use of… there will be PVTs and SVTs in the battery because it’s a high base rate differential that I need to consider and it has implications for what I might recommend. And so I got to go first, Ally. That was part of my battery. So maybe we can go back and forth. What else would be in our batteries?
Dr. Allyson: Well, Julie and I both independently, interestingly enough, came up with embedded measures to put into one self-report ADHD measure, the CAARS. So Julie looked at items on the CAARS[01:02:00] that are endorsed infrequently. It was in the regular population and people who really had ADHD.
And then what I did was I actually got permission from the test developer to do a mock-up where I’ve embedded items that look on the surface like they should be things that people have ADHD endorsed, but in fact, are almost never endorsed by people with ADHD. And so what I always do now is we give that, we call it the ECAARS. And so I can calculate Julie’s impairment index, and I can calculate our impairment index.
And we did a study two years ago where we found that in fact, if somebody fails both of those there’s a very high likelihood that they’re exaggerating or magnifying symptoms for whatever reason.
So we give that. And I also try and get[01:03:00] a parent or a significant other to do a reading of what the person’s like now. And we almost always try and get a parent or an adult who knew them before age 12 to do a retrospective rating of their symptoms, not just what symptoms they had but the severity of symptoms prior to age 12. So that information.
And then I know you were going to come back to this because you want to put a pin in it. We always ask for old school records. And if you have students who say, “Oh, I can’t get them or our parents’ basement was flooded and they’re gone.” then at least in Canada, the school boards are obliged to keep those records until you’re 25.
So we have just a form that we get them to fill out that we fax off to their own high school to say, hey, here’s permission. Send them to us. All we want[01:04:00] is their final report card of each year from kindergarten to grade 8.
Dr. Sharp: That’s amazing.
Dr. Allyson: Yeah, it’s amazing what you get. And you see these people who say, oh, I was always getting in trouble in school. And my teachers always said things about me. And then you get the report cards and the teacher is saying, “Works well without supervision, model student, learning skills are good to excellent,` self-starter.” I can tell you right now, this is not ADHD. I don’t need to do a whole assessment. This is not ADHD.
Dr. Julie: And we get medical records too. So we will ask if they.. well, many students will say, “Well, they thought I might’ve had ADHD but my parents were against medication.” It’s actually one of the things that the internet will tell you, you’re supposed to say if you’re actually trying to malinder. It’s really easy to look this information up. And that’s one of the things they tell you to say.
[01:05:00] But that’s not uncommon. You might’ve gone to a pediatrician. They might’ve raised concerns. And so we went many times I’ve gotten records and we then have parent reports, sometimes teacher report, because the pediatrician has sent forms out. And there’s no evidence that there were really significant symptoms. But now the age of 20 something, even the parents and the child are reporting back that there were these terrible problems.So sometimes we’re not good recallers of our past. And again, research shows that there’s nothing pejorative or value-laden about it. It’s just true. Humans are really great at remembering their past. And they tend to focus on their current lenses, their current framework, and remembering their past. So if schools don’t cooperate, we always fill out authorizations to get records. If they mentioned that they thought they had an IEP or a five or four plan schools are obligated to keep those in the states and they’re supposed to send them to us.
Well, we’ll also ask, well, you went to a community college. Can we get[01:06:00] those records? You went to this college, can we get those records? Because sometimes in the college transcript also shows up things about high school. So, oh my gosh, they took these advanced placement classes and they got credit for them. Well, that suggests they were functioning pretty well because you have to score pretty high on those advanced placement exams too. But they just told us they almost failed out of high school. Well, that’s their subjective report, but again, there’s playing detective and there’s an SAT score will pop up in the college records sometimes.
Attendance often shows up on. Even if you can only get the final high school transcript, it’ll have tardies and attendance and other behavioral indicators. Sometimes it will have the ACT and the SAT scores as well as other proficiency exams that here in the US I’m grateful that we over-test sometimes because I have records. And so that can all be on there besides just grades. And so we just do our darnedest, we call the schools[01:07:00] and say, we just fax this over and this is what we need it for.
The bigger problem we find is the schools will say, “Well, you have to pay for it.” And then we’ll say, this is not for educational purposes. This is clearly coming from a clinic. This is actually considered part of the medical record.
Dr. Sharp: Yes. So you’re just digging really deep for any of these academic records that you can find or medical records. That’s eye opening.
Dr. Allyson: The harder part is when you get the older students who come in and I don’t have them and you can’t get access to those records anymore. And when they say, my parents are dead or I don’t speak to my parents, or I don’t want you to talk to them. Those are the harder ones.
Dr. Julie: That’s tougher. Yes.
Dr. Sharp: Yes. You’re anticipating, that was my next question. What do you do in those cases where… I mean, some are legitimate and some may be less legitimate …where the adult child or whatever, the young adult doesn’t want you to contact anyone else, doesn’t have anyone else.[01:08:00] What do you do in those situations?
Dr. Julie: This is where I go as objective as I can. And I remind myself of the diagnostic criteria which is more than one setting. So yes, very often what brings people in, it’s not the disorder, it’s their perceived impairment.
And so if they’re coming in because they’re struggling academically in their first year of college, or as college advances, and they’re in a tough major, or as they’re planning to go to medical school or graduate school, and are worried that they’re not performing well enough to move into that really highly non-normative competitive environment. Of course, they’re going to perceive I’m not cutting it, right? I’m impaired. I need to go find out what’s wrong with me.
And so asking them as objectively as possible their academic history. So tell me grades in this. And again, report back to me any […] how would you do, what were your weak points? What are your study habits? But then also considering the other settings,[01:09:00] what jobs have you had? Did you ever struggle in those jobs? Were you ever reprimanded?
Because the evidence of symptoms and impairment needs to be not just when you’re in physics class. They need to be pervasive. And so even when they’re not willing to talk to others, it’s being that good detective and saying, wait, let’s go back and really look at this.
Again, a normal human tendency is to compartmentalize. If I were failing some of my last pre-med classes or I took the MCADD and didn’t do as well as I wanted, I would of course be hyper-focused myself on confirmatory evidence that this is what’s keeping me from being successful in what I want to do. And yeah, when I look back this all makes sense now.
And I can feel that deeply in all of my heart and be completely inaccurate about it because I now have not looked for the disconfirming evidence. So I, as the clinician need to put on my detective hat and say, let’s go looking, let’s test all these hypotheses, and let’s see what the evidence is. [01:10:00] And if all I have is to self-report, that’s limited, but try to explore it as thoroughly as you can.
Dr. Allyson: And you get what you were talking about too. Michael Gordon calls it sour cherry picking where they just go back and… But the other thing that I ask and you probably do too is, I asked him about driving. Like, do you drive a car? Have you had any accidents that were your fault? “No. But I have to really work hard to stay focused when I’m driving.” I think. Okay. And you drive back and forth to work or to campus every day. And the drive on the major highways, you’ve never met… you’re telling me that your attention problems are worse than 98% of your peers. That makes no sense.
So things like that. Or have you ever been formally reprimanded at a job? Or have formal warnings? Were you fired? I just had someone recently who came for ADHD screening and this girl worked every summer and all through the school year [01:11:00] as a lifeguard.
I was like, “Okay.” She said, “Well, you know what? They make us move every 15 minutes. We changed position.” I said, “well, they make everybody change positions. Why did they do that?” “Well, it helps us stay focused.” I was like, “Okay.” I really don’t want my lifeguard to be someone who really has ADHD, honestly.
But the other thing that we do, and I know you do typically is, we also try and screen for other conditions that could mimic ADHD. So we’re not just testing this to see if they have ADHD. I’m looking to see, do they have depression, anxiety, perfectionism, or study skills? Do they have a borderline personality disorder? Do they have somatic symptom disorder? All these things. And again, that takes time, but it’s amazing how often you find that somebody in the intake interview doesn’t really tell you about that. It comes in.
[01:12:00] I know you asked about drugs and alcohol and we do too. And lots of students don’t make that connection. They don’t sit there. But they also don’t think about past trauma. So I always ask and in a compassionate way, but I should say, when you were younger, were you ever teased or picked on or bullied or made fun of on a regular basis? Have you ever hit or beaten or hurt on a regular basis? And have you ever had any unwanted sexual experiences?And when I teach my students to do that, they can go, oh, they’re going to be really upset if I asked that. I said, no, they’re not especially if you explain why. And I always at the end, I say I’m not trying to be nosy. The reason I’m asking that is that often people who have a disability are easily victimized and targeted and abused and then that causes a whole bunch of other psychological problems that can interfere with functioning. And that’s why I’m asking.
And so I have never had anyone bulk those questions. But what I have had is students who say,[01:13:00] well, what do you mean by unwanted or, well, I don’t know. I can’t remember anything before I was 16. Or I’ve never told anyone about this and then the flood gates open.
But again, as Julie said, they compartmentalize and they don’t always think this could be contributing to what’s happening to me right now. I mean, we know lots of kids who were victimized and abused as kids. Once they transitioned to college or university and maybe less the environment where they were being abused, there’s a whole bunch of stuff that happens because now all of a sudden it’s safe to start paying attention to and being aware of those traumas and that information. And that can happen.
I remember one young woman I saw. I’m trying to think of how, anyways, it turns out that she had been severely sexually abused. And she came from a culture where she was going to go and have a rape kit done [01:14:00] and her parents said, ” No, no, you’re damaged goods. We’re going to hide this. You are never going to talk to anyone about this. This is a nonstarter, a non-discussion go about your life.”
And this poor kid was dissociating all over the place. And nobody was… again, giving that person a diagnosis of ADHD would have been the biggest disservice to her. And she actually contacted me a few years ago. She called me up out of the blue and she said, you probably don’t remember me. And I said, “Oh no, I absolutely do.” And she said, “I want you to know you saved my life.” But I don’t think she would have had that call if I said, oh, here go on a stimulant.
Dr. Sharp: Right. Oh, that’s so powerful.
Dr. Julie: Allie’s mentioned some great differential diagnoses too. But we also, again, have to remember thinking developmental biopsychosocial model that some of these aren’t necessarily diagnostic, but we need to think that developmental piece. What’s happening when you’re first starting college. You’re managing your own diet.[01:15:00] And so sometimes they may have eating disorders symptoms or it may just be poor eating.
I don’t have any meal until 4 o’clock. Well, how is your brain functioning without glucose most of the day? Not well, right? Sleep. When we know the base rates of sleep problems are enormous generally in the population, but especially for students now off on their own and everyone stayed up till 3:00 A.M. And then they’re living being of their stimulant caffeine all day long. Well, no wonder stimulant medication is effective. It’s keeping them awake. So they’re not micro sleeping through class which feels like inattention.
Those are two big ones. Academic readiness is another one. There’s not a diagnostic label for that. But if you went through school and to keep your grades up high, you didn’t actually take college prep level, or you tried to avoid math to the greatest extent you could, but you had a dream of being a doctor. Well, you didn’t prep yourself well to[01:16:00] do pre-med and now you’re in a really hard major.
And there are also cultural issues. So as Ally alluded to with regard to the individual with severe trauma, it’s also true just when it comes to academic career goals. So as an advisor to hundreds of psychology majors, I see fairly often folks who are loving their psychology courses. They come in to talk to me. And they’re failing their pre-med curriculum. And they’re tentatively thinking about a switch to a psychology major. And it turns out that it’s really a cultural guideline slash. I’m not paying for college if you’re not going to be a doctor or a lawyer. That there are only respectable fields to go into. And this is what leads to their motivation and their poor study skills and their disinterest and thus inattention.
But they somehow also landed in the clinic for ADHD. And really it was about career counseling, [01:17:00] cultural sensitivity, and thinking about that developmental stage of being independent, but yet being financially dependent on parents who say, “I’m paying for this major” and how to help them navigate that discussion.
That’s me being a psychologist and not an assessor or a diagnostician. But that’s absolutely part of that picture of all the hypotheses you want to hold in place for what could be contributing to this presenting problem. And as Ally said, if you just said, “Oh, you think it’s ADHD? Okay, well, here’s an ADHD questionnaire. Yep. Sounds like ADHD.” You’ve actually done a disservice to that individual.
Dr. Allyson: And your point about developmental stages is really relevant too. I just finished reading iGen and Jean Twenge. It says that if you look at the maturity level and the ability to take responsibility and do things, the typical 18 years old now is at about the same stage[01:18:00] developmentally as a 14-year-old was in 1975. But you’d never send a 14-year-old off to college, but we do. And in general, we’ve got kids coming who may not be developmentally ready for the responsibilities of independent learning and studying and scholarship. And so that may be part of why they’re struggling.
Dr. Sharp: Right. I like how y’all phrase that. And just making it clear that we might have these environmental factors or developmental factors that are nondiagnostic. I think we do get wrapped up in differential diagnosis, but there are all kinds of contexts that affect people’s functioning that might not be diagnostic.
Dr. Julie: They require an intervention where you have to have support and resources and intervene that absolutely could improve their presenting concerns.
Dr. Sharp: Yes. So I have[01:19:00] just two things before we wrap up. Our time has flown. This has been fantastic. I want to land the plane on the ideal battery. So far, I’m not going to let you get off the hook. So we’ve got the PVT, we’ve got the CAARS with these embedded indexes that you all have developed which by the way, are those publicly available? Can other people do them or is that just a special thing that only y’all get to use?
Dr. Julie: There are research articles specifically on what items you score for what’s called the CII, which my items are part of the standard CAARS. If people were interested, then I could direct them to the articles and that they’d been turned independently by other researchers and not just myself. Although new CAARS is coming out. So now they probably won’t be valid.
SVTs are the hardest part right now in ADHD. Allyson can describe her measure too and how it could be[01:20:00] available. But there are a few newer studies looking at what we would call standalone symptom validity tests for ADHD, but they just really only have one study. They really need to be cross-validated. But the idea is that they really do need to have those and have them within ADHD instruments. So for decades upon decades scales like the PAI and the MMPI have had self-report validity scales, but they’re really about psychopathology, right?
So while there’ve been some studies using them on ADHD, they’re not ADHD symptom masks for the most part. And so again, you’re going to miss a lot of people if you do focus on that, but some SVT would be crucial.
Dr. Sharp: Okay. And are you doing cognitive as a standard matter of course?
Dr. Julie: I always give a cognitive battery. I am a neuropsychologist. But indeed, most of the time their concern is not a diagnosis but impairment. [01:21:00] And they’re looking for accommodation on a high stakes test or even in college. And so we need to see, well, where are your impairments? And so we will run into folks, again it probably depends on your university in terms of the criterion for entrance, for example, but we’re pretty liberal. And therefore we do have some students who might even below average in general cognition when they first get in. And so they’re going to have some struggles and that’s really important therapeutic feedback.
It doesn’t mean they can’t do college. They just need to be really careful about how they plan college and whether there are certain majors that might just be really hard for them to get through. That’s important therapeutic information.
So we always do a full IQ test, but that also includes working memory and processing speed measures.
And then we use the research data to guide us on other cognitive tests focusing on those that have shown the best differentiation between ADHD and again, asymptomatic[01:22:00] folks because again, that literature is ripe with problems because there’s not ADHD versus depression on a cognitive battery. It’s ADHD versus not ADHD.
So we focus on working memory, processing speed, learning efficiency measures, executive function, inhibition measures. We don’t give continuous performance tests because consensus statements have shown because of the research that they are not diagnostic of ADHD. And they are so state-like, they respond to anything time of day, whether you smoked a cigarette or having a cup of coffee, how much sleep you got, your anxieties. And so we long ago, both our child assessors and adult assessors moved away from using them.
Dr. Sharp: Got you. Can I back up and ask you, which measures you like for some of those other domains, like the working memory, the processing speed, the executive functioning?
Dr. Julie: Yeah, we use the WAIS in my [01:23:00] clinic. We do use the Woodcock-Johnson at times especially if there are academic concerns because they have a full battery. Sometimes we’ll use tests of the cognitive, the TOGRA, because they’re very reliable. They’ve been highly standardized.
We use a lot of tests of the DECAFS battery actually because I get executive functioning battery. Right now we’re doing Tele-psych. So I’m literally thinking now we can’t give some of those in the tele-psych, it’s actually like, what would we do if we could go back to normal? Well, we would give a lot more psychomotor speed tasks than we are currently.
My battery’s heavy in the executive function. And again, we’ll usually typically do a list of learning tasks because learning efficiency and recall are actually helpful. And I use one that has an embedded malingering[01:24:00] measure as well. So that way I have more checks for and then of course PVTs and SATs are in my battery. I usually also give a lot of self-report instruments for other than, but that depends on what the problems are. So I don’t always give the same thing. It really depends on what their concerns are.
Dr. Sharp: Yeah. Are you doing a Broadband checklist or a personality measure of some sort?
Dr. Julie: I was trained on the MMPI. I have used it in much of my research. So I have used the RF for many years and now I used the MMPI-3 that just came out.
Dr. Sharp: Yeah.
Dr. Allyson: We use the PAI. But I think it’s getting at the same thing. You want a broadband screening measure to look at, well, more than a screening measure, you want a broadband measure that’s really evaluating mental health functioning and personality functioning in lots of areas. So we ask people to fill it out. Although again, it [01:25:00] is very subjective, but we have them and do the WAIS functional impairment rating scale, which is a free Impairment checklist. But most of the time, what I think we’re finding is
a) It can be exaggerated.
b) They’re just measuring symptoms again.
But we’ve got that. And we like, Julie, if we’re just trying to say, is this an ADHD diagnosis, we aren’t giving the whole cognitive battery, but I would say, we never get students coming in saying, I just want to know if I have ADHD. Most of the time they’re saying, I think I’m impaired and I want to get accommodations or I want to get stimulant medication.
And so you don’t need to give those tests to make the diagnosis, but you need those tests to help you understand what the impact of their disability is on their academic performance, on their executive functioning performance, and their memory performance. [01:26:00] And so that’s where you need to give those other tests to see how is this causing functional impairment in the academic arena? And I know that Julie and I give a lot of the same PDTs and SVTs. And we spread them out so you don’t just throw them all in one spot. You want to sample as you’re going along how engaged the person is or whether they’re really performing credibly all the time.
And again, I don’t know what you do, Julie, but there certainly are times where we see somebody who’s done all right, everywhere until it’s the very end of the day or the second time they’re coming in. And then they fail on one. And then that I would say is the harder decision about, do you report any of the scores or do you say these scores from this day maybe aren’t as valid or reliable.
But it’s quite interesting. Sometimes we’ll get people [01:27:00] who fail the validity tests and while they’ve exaggerated their symptoms on self-report, their cognitive measures aren’t impaired. So in those situations, we say this is probably like a worst-case scenario of what they could do but regardless there’s no evidence that their disability is causing functional impairment academically.
Dr. Julie: That’s not unusual even in the much longer decades of research on PVTs and SVTs, especially in a mild traumatic brain injury that you don’t always see that suspect scores on a symptom validity test also are highly related to Sussex scores on a performance validity test. And that’s again, we’re measuring the construct we’re measuring is performing invalidly so probably that’s not going to be all that is as reliable in the sense of it’s always going to show up on every measure or that people who self report implausibly are also going to perform implausibly. So the fact that that’s the pattern in ADHD. [01:28:00] is not unusual. That’s also true when we look at other cases where there’s a non-credible report.
I mean, for some people, they very much focus on their subjective impression. And again, if they were deliberately and consciously trying to perform poorly, you might expect a little more consistency. But lots of people report or perform non-crediblly. We’re not psychic. I don’t know that they’re deliberately or consciously doing it. It’s still non-credible. it’s still not necessarily accurate.
And we do also use the self-report impairment indices although we don’t routinely do the WAIS one. But we take that with a grain of salt. They don’t have validity scales on them. And so we fairly, we also give soft report executive function measures mostly for research purposes at this point because it’s probably pretty much everybody who’s scoring. They report at the 99th percentile. That the 99th percentile impairment in all these[01:29:00] executive functions, but they made it to college. They’re juniors in college and doing well.
Again, that’s not possible if that were an accurate indicator that you’re worse than 99% of the normative sample in executive function, you wouldn’t be here. But that’s their perception. And so, again, we’re in those two hats, like, okay, you perceive yourself as that impaired. My feedback on the cognitive measures might be very therapeutic for you. Here’s where you are and that’s okay. And that’s normal. And you are well within normal. And normal is good. Do not pathologize normal.
Dr. Allyson: The one thing that I hope your listeners really take to heart is, if you have somebody who fails validity tests, please don’t just ignore it or try and rationalize it and say, well,[01:30:00] it was their ADHD. That’s what caused them to fail it or they said they didn’t sleep well.
No, that’s not why they failed it. And even if you say, well, people with ADHD have trouble with staying motivated, that’s fine. But what it tells you is your data is invalid. And trust that. My new analogy is I say, it’s sort of like the carbon monoxide detector in your house. You can’t smell it but it’s warning you and saying danger, danger warning, well, Robinson, there’s something wrong here. You may not be able to trust these data. So don’t just explain away.
Dr. Sharp: I like that. Yeah, I think that’s a nice note to end on is don’t second guess that data, trust it. Trust yourself. So we’ve covered so much in this conversation and yet I feel like there’s so much more to cover.[01:31:00] My gosh. So who knows if I haven’t totally burned y’all out. Maybe there’s around two somewhere down the road, but yeah, this is super fun. I really appreciate y’all coming on. I feel like people are going to take a lot away from this, so thanks. Just thanks again for being here.
Dr. Julie: Well, thanks for having us.
Dr. Allyson: Thank you very much. This was a really fun conversation.
Dr. Sharp: Thank you all for listening. Thanks as always. I hope that you learned a lot from this episode. I hope it got you thinking about your standard ADHD battery and how you might document ADHD in the cases where it is a valid diagnosis. Like you could tell, there are so many directions that we could have gone with this conversation. And my only hope is that I can have another round with Julie and Allyson to dig in a little bit more fascinating.
All right. I hope you’re all doing well. Like I said, in the beginning, if you want to jump into a group coaching experience[01:32:00] and expand your practice beyond those initial phases, the Advanced Practice Mastermind Group is starting June 10th. And that’ll be, I don’t know, maybe the 5th cohort, 5th or 6th. I don’t know. Losing track. We’ve had so many good groups over the years. This is a cool experience for anybody looking to get some group coaching and support and accountability and folks who can really help you reach those goals in your practice. So if you’re interested, you can go to thetestingpsychologists.com/advanced and learn more and schedule a pre-group call.
Okay. I think that’s it for this week. I am so excited to be interviewing some amazing folks, continue interviewing some amazing folks over the next few weeks. Hang tight and definitely subscribe if you haven’t already. There are going to be some amazing interviews[01:33:00] coming up that I am very excited about. I hope everyone is doing well, getting vaccinated, staying healthy, and maybe getting some sunshine. We are headed toward summer. And that is very exciting.
All right. You all take care until next time.
The information contained in this podcast and on the Testing Psychologists website is intended for informational and educational purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment.
Please note that no doctor-patient relationship is formed[01:34:00] here. And similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. similarly, if you need supervision on clinical matters, please find a supervisor with expertise that fits your needs.