Today’s conversation is really intense, I’ll be honest. It is a bit rambling largely because I was not well organized and for once in my podcasting history was a little out of my depth in terms of what I should be asking and what I shouldn’t. So bear with us for, I would say the first 30 minutes or so. I think it is interesting but it was just a little rambly, but after about 30 minutes, we really hit our stride.
Today, I’m talking with Dr. Alex Beaujean who is an expert in testing and measurement. He is on the faculty at Baylor University where he works in the Psychology and Neuroscience Department. He basically created the quantitative method specialization for their doctoral program [00:01:00] and he also works and contributes to the school psychology program there as well.
He’s an incredible scholar. He’s published two books on latent variable models, more than 80 articles of book chapters in peer-reviewed scientific outlets, and over 80 posters and papers at professional conferences.
He has a long history of research and scholarship. He teaches, he supervises, and he has a solid grounding in the clinical practice of assessment as well. He is a licensed psychologist in the state of Texas and has been recognized by the American Board of Assessment Psychology for his clinical competence.
Alex has done it all. He has a lot to say and he graciously bears with a lot of my ignorant questions but this is a great conversation and I hope you all really enjoy it.
[00:02:00] Hello everyone and welcome back to another episode of The Testing Psychologist podcast. I’m Dr. Jeremy Sharp. Like you heard in the introduction, I am here today with Dr. Alex Beaujean. I am really excited for this conversation like I am for most but this one in particular, because Alex has spent a lot of time specializing in an area that I am definitely unfamiliar with, but I think will be very relevant for us.So we’ll be talking a lot about measurement test construction, what are we actually doing with our assessments? How does that translate to the real world? Any number of things.
I’m honored to have him here today. He is currently an Associate Professor of Psychology and Neuroscience at Baylor [00:03:00] and also does some consulting and expert testimony on assessment-related matters as well.
So Alex, welcome to the podcast.
Dr. Alex: Thank you for having me.
Dr. Sharp: Yes, absolutely. I of course have to thank you for taking the time. Like I said, I’m so curious to get into some of this material with you.
We were talking before we started to record and I was listening to the School Psyched Podcast probably three or four months ago, I’m not sure. They were interviewing someone else, Ryan McGill, who you’ve done some research with and he mentioned your name. I said, that name sounds familiar. That guy’s in my Facebook group. So I looked your name up and I was like, oh my gosh, how did I miss all this research? We have a real scholar in our midst here in the group. So I’m glad that you agreed to come on and talk through some of this with me.
Dr. Alex: Sure.
[00:04:00] Dr. Sharp: We have a lot to get into, but I would love to hear just how you got to this point in your life, what your training look like and what your day job looks like now and how you spend your time in the psychology field.Dr. Alex: Sure. I had an undergraduate in psychology. Like most people in psychology, an undergraduate didn’t really quite know what wanted to do or didn’t quite know all that was around in the field of psychology. So after I did a postgraduate experience with an engineering-type laboratory.
Looking at the very different aspects of psychology, I said, oh, I think my interest lie in the educational/school psychology world and so I applied to different programs, got into the University of Missouri, and went there. I was initially planning on just focusing on school psychology there and part of the curriculum there [00:05:00] for the doctor program is going to take quite a bit of measurement and statistics courses just as part of the regular curriculum.
And so as I got to maybe start taking the regression and Nova, et cetera type classes, I kept thinking, I like this stuff but I don’t think I have a firm enough grasp on it and I can see why it would be very important to know. And so I kept taking as much as I could.
And then when I took all the courses that you needed to take, I talked with my advisor and said, is it possible I could take some more courses just for my own understanding about that? And he said, go talk with people over an Edcite because they do measurement. And so I went over there, talked with them and very seldom do they get people asking to take more of those type of courses so it was like, sure.
They helped me lay out a sequence and I kept taking more and more courses. The more I was taking, the more I was challenged, the more I really liked it. I was like, oh [00:06:00] I like this. I don’t think I quite understand it all, but I know this is something that is giving me interest.
And so I kept taking more of those courses along with my other required courses. And then at some point, it just dawned on me that these are doing the same thing. In my school psychology curriculum, we’ll be doing a lots of assessment and then in the measurement curriculum we’d be talking about how those assessments are created and what’s all the theoretical properties behind that there and it dawned on me, oh, this is just two sides of the same coin.
As I kept going through with that, eventually, I racked up enough course credits that I was able to get a dual degree there at Missouri and it’s measurement and then school psychology. I had the very fortunate opportunity to do an internship at Applewood Centers in Cleveland. And that at the time was affiliated with [00:07:00] Case Western so I got to work with Eric Youngstrom and his wife, Jen as part of my internship.
Eric is very much also big on assessment but strong in the quantitative measurement aspect as well. So I learned a ton on my internship from Eric and Jen and the other folks there at Applewood Centers. So that prodded me even more like I really liked this field. I don’t know enough about it by any way, shape or form but I really like this.
So I applied for some postdoc and professorships as most people do when they get done with their internship and Baylor offered me a position down here to mostly focus in the quantitative methods stuff. That’s where I started. And then about a year or two in, they started asking me, well, you have a background in assessment, could you start doing stuff in [00:08:00] there as well? And I said, yes, I really like that.
And so then ever since I’ve been in this made-up dual-track type of position where I teach Quantitative Methods or things related to Quantitative Methods but then I also will occasionally teach an assessment course or do supervision of folks doing psychological assessments.
Dr. Sharp: That’s great. So you see it from both sides, you got the theoretical piece and the practical piece.
Dr. Alex: Yes.
Dr. Sharp: I feel like a lot of us and I really want to ask you what you think piqued your interest about measurement because I don’t think that happens for a lot of us. We are pulled more toward the practical side and the measurement classes are important, certainly, and maybe interesting but certainly not the most interesting part of our education. I’m really curious, what do you think it was that pulled you in that direction, [00:09:00] that made measurements so interesting?
Dr. Alex: I think part of it was, I just didn’t understand it. And so whenever I come up across something that I don’t think I quite get and understand, my typical reaction to that as I need to know more. I need to learn more about this because why am I not quite getting all the stuff that’s happening here? Why am I missing some components here?
And so I just started reading more about this, talking to folks about it. And the more I kept diving in to it, the more I realized that, oh, this is such a rich field and it combines some abstract math but as well as some very practical applications with that.
As the more I get into it, the more like, oh, my goodness, I want to know more. And then as you keep doing over and over again, you just start to develop an appreciation for that. [00:10:00] For me anyway, I developed a liking of what I was reading, what I was learning there.
Dr. Sharp: Sure. I love that process. It sounds like there’s a parallel process too, with assessment, where we’re trying to find out more about someone or something that we don’t know a whole lot about and it’s that curiosity that pulled you along and led you further and further down that path with the actual process of measurement.
Dr. Alex: Yes. Very much a parallel there.
Dr. Sharp: Nice. I’m curious these days, you’re a professor, obviously. You have a prolific research history and you’ve written two books. You’ve got a lot going on from the research side. Are you primarily doing research? Are you teaching as well? I know you’re doing some supervising but how much time?
Dr. Alex: It’s a little bit of everything. This particular semester, for example, I’m teaching a course in latent variable models [00:11:00] at the graduate level as well as teaching a course in introductory to personality theories at the undergraduate level. It’s kind of a variety of things as well as I’m supervising, Baylor has a psychology clinic and so I’m a supervisor at that clinic largely with child assessment cases because they don’t have lots of supervisors. I’m sure of that so I get to supervise two or three students at a time over there when they’re doing assessments.
My teaching load tends to be a little bit quantitative and a little bit applied areas. In the spring, I’ll teach a class in introductory psychological assessment as well as a more graduate-level research design causality type of course. I’m very fortunate that there’s not a lot of people vying for those particular courses. It’s like, oh, well, everyone wants to teach this particular course and so you have to [00:12:00] wait there. No one really wants to teach the assessment courses or the advanced design courses. And that’s the stuff I like so I get to teach a variety of those type of courses.
Research-wise, I’m working on another book as well as a handful of articles. That’s something that’s always going on in the background. I’m always working on some scholarly project there.
Dr. Sharp: What’s your current area of interest with research?
Dr. Alex: Well, right now from the assessment perspective, I’ve been doing quite a bit on the patterns of strengths and weaknesses model or method for assessing cognitive functioning, whatever. The terminology is relatively new, the PSW stuff, although the actual process goes back almost [00:13:00] 100 years. The ideas have been around in psychology for a while there but it’s become quite popular, especially for looking at learning disabilities, etcetera.
That has taken up a bit of my time of late. We have a special issue in a journal coming out where I edited that and we’re looking at that particular thing. We have two other articles looking at some of these particular methods and what’s the background behind them.
I’m not necessarily collecting original research but we’re looking at what are the assumptions behind the model, what are the measurement assumptions that go into these particular procedures and do the instruments that are being used to carry out these procedures, do they actually have the properties needed for the procedures to be carried out faithfully? So I’m looking at some of that more [00:14:00] from the theoretical side than the empirical side.
Dr. Sharp: That’s fantastic. When is your book coming out? Are you that far along in the process or no?
Dr. Alex: Well, I have a copy due to the publisher into 2019 so probably maybe 2020 when it’ll be out. It’s on individual differences. It’s looking at the history and modern research dealing with cognitive functioning personality, those type of things as well. It’s going to be a little bit unique in that it has a methodological focus on it which many of the books in this particular area today don’t particularly have that.
Dr. Sharp: It sounds fascinating, my gosh, and a big project.
Dr. Alex: Yes.
Dr. Sharp: Let’s dive into it a little bit. I think that’s probably a good segue to [00:15:00] some of the things that I’d love to talk with you about. I think that one of the things that really got me interested is seeing some of the comments that you have made over the past several months in our Facebook group, trying to think how to phrase it, I don’t know if contrarian is the right word but you have said some things that certainly raise a lot of discussion when people are asking what tests are appropriate to use in certain situations, or how do we diagnose X based on X test results?
What I’ve gathered is that you are very much keyed into, as you should be, how the research actually supports the testing that we do and how that translates to certain diagnoses or certain real life performance and things like that. I’m not even sure where to dive in but I might just start with a really big [00:16:00] question, which is, why do we use these tests that purportedly measure real life skills when that may not actually be the case?
Dr. Alex: I guess there’s two things there; first, why do we use tests? In general, we use tests and I’m using that term very broadly there. It doesn’t just mean paper and pencil test. It could be interviews. It could be questionnaires. So I’m using tests extremely broad there, but we use that to gain information. That’s their only reason. Unless someone has a hobby of taking tests or looking at tests, typically the only reason that we would administer tests are to gain information. There’s really not another reason why you would do that.
If you had unlimited time in the world, you could probably have these very long, extensive interviews with folks and get the information you need but with most assessment cases, you have a very [00:17:00] finite window that you can gather this information from and so tests are an economical way to gather the information that you want to gather in order to make some type of decision. Usually, that’s the resulting of doing the test is not just, oh, I want to get this information just FYI. It’s now we need to make a decision about that, whether it’s a diagnostic decision, whether it’s an intervention decision but typically there’s some type of decision that’s made based on the assessment results. So that’s largely why we do testing as part of the psychological assessment there.
Whether they have usefulness or not, utility or not, that’s a little bit of a contentious thing. You have one school of thought that comes at it from oh, these tests were designed to do X, Y, Z, or they were designed to measure X, Y, and Z therefore, that’s what they’re doing. And then you have another school of thought that [00:18:00] says it really doesn’t matter what they were designed to do, we need empirical evidence before we can say it’s actually doing what it says that it’s doing.
So we really don’t care what it’s designed to do. What we need is utility evidence to say, what are these scores actually doing? Are they actually able to help us make these decisions any better than the coin flip test or other type of less invasive instruments that we could use?
That’s where the contention gets in there because unfortunately, they’re just not a lot of research done with, although we know what should be done, that’s not always the case of what is actually done or at least what’s published by publishers when they bring out instruments that, oh, this test is designed to diagnose learning disabilities. You look at the technical manual, there’s not one lick of information that tells you what the utility is for this particular instrument in [00:19:00] identifying folks with a learning disability.
Dr. Sharp: Yeah. I haven’t done any data gathering on this, but I feel like a lot of the graduate level education that I’m familiar with anyway, was the former of those examples where we use these tests to measure, we just assume that they are measuring what they say they are, it’s not necessarily research driven, if that makes sense.
We just take it at face value that, yes, the WISC is measuring these aspects of intelligence, yes, the academic measures correlate with whatever real world academic performance, and then we can use that to diagnose learning disorders and so forth. Is that actually the case? Are there more programs out there that are more focused on the second example that you gave?
Dr. Alex: I think there’s a variety of program philosophies and how folks are trained. I think [00:20:00] there’s probably a continuum there. Some of them are focused much more on the clinical experience and that we’re going to, based on what these things are designed to do, this is what they’re likely picking up on, then you get probably the other extreme where you have to have 25 publications about something before you can start to think about administering a particular instrument there.
So my guess is that, I’ve only had experience with maybe six or so programs relatively intimately, there’s a continuum about them that some tend to go more one extreme than the other. And then there’s probably quite a few that are in the middle there. It probably just depends on the program philosophy which comes from the faculty that are there.
And I say that, oh, this program always does this well, that program does that only when those faculty are there, when those faculty retire or move on to something else, [00:21:00] then likely that program philosophy changes a bit.
Dr. Sharp: That’s a great point. I guess it raises the question for me of why there are even those diverging perspectives. How can a test make it to market without pretty clear research around what it’s actually doing or what it says that it’s doing?
Dr. Alex: In the U.S. there’s no governing body that says this is what instruments have to do. It’s not like with pharmaceuticals or the FDA is like, you have to go through phase I, phase II, phase III type of trials before you can bring it to market. There’s absolutely nothing like that.
APA, AERA and CME have these standards but there’s no teeth to those in that if a company decides to create and publish an instrument that does not adhere to those standards, there’s nothing that APA or AERA and CME can do about it. [00:22:00] It’s not like they can say, oh, we’re prohibiting that from going to market.
It’s up to the test publishers what they want to publish and how much information they actually want to provide in the technical manuals. And that’s probably going to be largely driven by consumer wants. I guess they probably don’t get lots and lots of communications from consumers asking for, why don’t you put this particular piece of information in the technical manual? Why don’t you give us this particular piece of information? That’s probably not what consumers are asking them when they’re talking to the publishers. So my guess is that’s probably part of the reason why that’s not in those manuals.
At the end of the day, publishers can publish whatever they want to publish and they can put instruments out there that has as much or little information as they want. [00:23:00] At least in the U.S., the onus is on the consumer, which would be psychologist or other related professions when deciding what instrument to purchase that they would say, oh, well, this particular publisher doesn’t give us this particular piece of information, we’re not buying that instrument until the information is available. That would be about the only thing that would cause publishers to say, oh, we need to start including this information in our manuals.
Dr. Sharp: Sure. I would think maybe naively that the conscience of the publishers and authors would make it hard to bring to market a test that’s not maybe measuring what they say it is, but maybe that’s naive. Do you think that’s happening purposefully or is there just some, I’m not sure, misguided understanding of what they’re putting out or how do you see all that?
Dr. Alex: I [00:24:00] can’t speak to the motives of any of the authors. I’m going to assume that the authors are thinking that what they’re doing is best practice in their field. Although I’m mission, I don’t necessarily know test publishers. It’s a really questionable thing. They obviously have financial motives. They have conflicts of interest there that they need to sell those instruments or otherwise they can’t stay buoyant there as a company. And so I think they have to weigh what’s best practice versus what’s going to sell a bit there.
My understanding from people that have written or authored instruments is that they don’t always get final say about what’s going to be included in the instruments and other contracts may differ between organizations, what have you but it’s not like oh, author A [00:25:00] wants all this information included in the manual included and that’s exactly what that says publishers do. It’s here’s what the authors think should happen and then here’s what the publishers do and there’s not always a consensus there.
Having said that test authors, I’m going to assume that they’re doing what they think is best, that they have a good helping motive in mind there. There’s a lot of different trainings from test authors. So if you just take out your catalog from PAR, PRO-ED or Pearson or whoever there, and you look at who the authors are of these tests, there’s a huge variety of backgrounds. Some of them have very strong backgrounds in assessment measurements, some of them have very little. What they think is maybe best practice from their particular perspective, but may not necessarily be best [00:26:00] practice from a larger, more technical perspective.
Dr. Sharp: Sure. Yeah, I suppose that’s true. Let me back way up and just start to maybe dig into this difference. My understanding anyway, is that from a lot of the research that you have done and others have done, there is a difference between what we think we’re measuring with many of these tests and what is actually being measured and how that translates to “real life” and day to day performance is fair to say, or is there a better way to phrase that?
Dr. Alex: I say that that’s an empirical question that needs to be answered on a score-by-score basis. Just because a score says or someone names a score math index or whatever you want to name a particular score from an instrument, that in and of itself does not mean that that particular value is representing what the test authors [00:27:00] say it’s representing.
I would take all the scores on a particular instrument and approach, those are hypotheses about what those values represent. Hopefully, within the technical manual, the authors/publishers have given enough evidence in there to make the case that this particular value actually represents this particular attribute in there, they give a decent argument for that.
I can say that that happens probably far less often than it is desired. If you look at the latest Wechsler scale or Woodcock-Johnson or what have you there, there’s no shortage of scores. I think there’s 30+ scores on the Woodcock-Johnson about that or in the Wechsler scales about that and the Woodcock-Johnson.
In some of them, they devote a lot of space and resources to describing what [00:28:00] it is. And then some of them, they just totally leave it up to the consumer to assume that it’s actually measuring what it’s measuring. The best example I can think of is, on the WIAT or the Woodcock-Johnson achievement, the total achievement score, I have yet to see anybody actually devote space to saying, what do we think this particular attribute is? What is this total achievement that’s comprised of reading, writing, arithmetic or language or whatever it happens to be involved in that particular instrument? What does this particular value represent?
Those total scores are in almost every standardized achievement test that you’ll find but I still have yet to find an instrument would actually give space over say that here’s what this particular attribute is, then here’s where it’s grounded in the literature about what it actually represents.
Dr. Sharp: Before I ask, we talked ahead of time [00:29:00] about, I will definitely ask some dumb questions and this is one of those moments I’m like, this might be a dumb question but what would you hope that it would measure? In a perfect world, what would the answer be for what that number is representing?
Dr. Alex: For the total achievement?
Dr. Sharp: Yeah, let’s just take that as an example.
Dr. Alex: I’m very curious about that. I talked to some folks at the American Psychological Association convention I was at a few months ago and the folks I asked really didn’t quite know. I think we’re going to plan on maybe doing a symposium at the next APA convention trying to discuss what are these numbers actually representing.
If you look at the literature, there’s not a whole lot of literature that describes what a total achievement attribute is. There’s a few things here and there scattered from about a 100 years’ worth of stuff in psychology. There’s a few things published about it, [00:30:00] but it is not a well-defined attribute in the literature. And so within a particular instrument, I really have to throw up my hands. I really don’t know what those numbers represent from a given test taker.
Someone gets 107 on a total achievement on the Woodcock-Johnson or Kaufman or what have you there, I don’t know how I can say that this is what behaviors I should see or skill sets I should see because of that particular score. I don’t necessarily know what that number represents but yet those numbers are there and people interpret them as if they have meaning to them but I don’t know what that number represents.
Dr. Sharp: Thinking about a lot of the other major tests that a lot of folks use, I’m thinking WISC, WIAT, certainly the Kaufman stuff, Woodcock-Johnson, what are some other [00:31:00] examples of numbers that are reported that are unclear as to what they are actually measuring? Are there any big ones?
Dr. Alex: Well, the Wechsler scales are notorious for being atheoretical. I don’t know if you know that the history of the Wechsler scales a lot but when Wechsler originally developed the Bellevue, he basically took a lot of different subtests from other instruments and put them together largely from the Army Alpha and Beta test but also some other tests that were popular at the time and just put them together.
Apparently, intellectual property thing or rights were not a major issue at that time because that was a common thing for folks to do. I think if someone tried to do that today, there would be all sorts of lawsuits, whatever but if you look at the Army Alpha and Beta test, which are published because there were government-funded either published, then you look at the original Wechsler Bellevue items and you can get the, because Wechsler published all those in a book, you can see [00:32:00] that they’re very similar there. He even says that that’s where he pulled it from. He was an examiner for the army at that time.
And so he created that particular instrument just based on, oh, these instruments seem to have worked well but there wasn’t really a whole lot of theory behind that. It’s just like, oh, I want to have some stuff that’s verbal and some stuff that’s “nonverbal” there. I think these are going to work well.
It was only after he put them together where Cohen and some other folks said, what are the attributes actually underlying this instrument and they started doing factor analysis et cetera, trying to come up with attributes that are there. The Wechsler tradition has been, for the Wechsler Bellevue II which moved into the WISC and then the WAIS which came from the original Bellevue there, the idea has been we need to keep our history to this original [00:33:00] Bellevue.
And so many of the subtests that are there are there because of history. They’re not necessarily there because they’re optimal measures of a particular attribute. They’re just there because that was in the original Bellevue or that was in the original WISC or what have you there. So they’re there for historical purposes.
And if you even look at the manual, they say that that’s one of the big components in creating the WISC-V or the WAIS-IV whatever instrument you’re looking at is that it wants to keep the historical tradition going back to the Bellevue. And so that’s a big reason why they chose not all the subtests but many of them is that to keep the Wechsler tradition pass down from instrument to instrument there.
So what that winds up as is many of those subtests are there because of history and so what actually is it that they’re measuring? Like the Processing Speed Index on the Wechsler scales, [00:34:00] they go together because they don’t play well with others. That’s the only reason that they coalesce as far as from on a factor analysis, because they just don’t play well with the other subtests.
They weren’t necessarily selected because they’re such great representation of this attribute, it’s just that they’ve been there throughout multiple editions and they want to keep that link to the past and they don’t work well with other subtests and so they just go together and form this attribute that the psychologist or Pearson says is the Processing Speed Index but I really would be hesitant to say that those values on the PSI are a good representation of processing speed. Processing speed is probably involved in doing those tasks but as far as it being a really good representation of someone’s processing speed, it’s probably not the best representation [00:35:00] of it.
Dr. Sharp: Do you know of any other tests that are maybe a better measure of processing speed?
Dr. Alex: Personally, when we’re looking at things with speed, you probably want to get something where time is the metric there. So looking at things that are going to be computer-based that can actually measure things very accurately down to the milliseconds. I know in many of the cognitive taxonomies, they separate out decision speed from processing speed but things along the lines of reaction time or that type of stuff where it’s actually measuring speed.
Speed is a very nice unit. Seconds are well-defined and they’re actually measuring processing speed in seconds or number of things you can do in a second, something along those lines. They’re not going to be, oh, well, your score is 103 compared to your same age peers, that loses [00:36:00] that particular attribute when you start doing all those conversions there.
So as far as processing speed, things that are going to involve some type of computer, when I was in graduate school, we did a bit of reaction time type of task. We were able to measure people’s ability to respond to things in milliseconds. And so we were able to parse out individual differences relatively well with that.
Dr. Sharp: And then you’re using, if I’m understanding this right, a more concrete, for lack of a better word, measurement unit instead of going through the score conversions you have, whatever, X number of milliseconds versus a standard score that is hard to compare.
Dr. Alex: You hit on a very important point, the units involved in the measures on these scales. Seconds well-defined, meters, all these [00:37:00] base units are very well-defined units, but then you look at the standard scores for many psychological instruments, they’re not in a unit, they’re in standard deviations.
Standard deviation is a statistic and it varies from instrument to instrument what that actually represents, where if you get a ruler, it doesn’t really matter what ruler you use, an inch is defined the same way across instruments, whereas most of these psychological tests, when you look at the norm standard score that they’re on the standard deviation units. So the standard deviation from the WISC-V is not the same as the standard deviation from the Woodcock-Johnson IV, it’s not the same as the KABC-II, it’s not the same as the CAS.
So all these standard deviations, their population are sample-specific. And so you get all these different norming samples and the standard deviations are different across them. And so even though they give you the same IQ unit, those units are not comparable across instruments. [00:38:00] That’s been a problem in psychology and it’s been a known problem.
There’s been articles on the 1910s and 1920s when I started moving toward the standard deviation standard that we’re saying that that’s fine for a short-term stopgap. That’s not a long-term solution to finding units in psychology but that has largely gone on onto somewhat deaf ears there.
Roche tried to somewhat change that when he came up with the Roche model in doing that. Colin Elliott with the DAS tried to grab a hold of that. And Richard Woodcock, when he came up with the first Woodcock-Johnson, that was relatively noble because he tried to do that as well, tried to get a unit that was independent of the norming sample but by and large, those are things that have just not played a big part in psychological assessment.
In fact, you can probably [00:39:00] count on one hand the people that actually interpret the W scores from the Woodcock-Johnson and they just leave them alone or the RPI scores from the Woodcock-Johnson, there’s probably like, oh, well, I don’t know what those are and so I’m just going to leave those alone and just go back and interpret the standard deviation score that I’m accustomed to.
Dr. Sharp: That’s such a good point. I’m just thinking about how so much of at least my training, I think probably a lot of others is based around that standard score, standard deviation interpretation and that’s how you explain test results to people across a battery. So it sounds like that’s pretty, that can be problematic.
Dr. Alex: Again, when you’re looking at trying to represent an attribute, it can be problematic. When you’re looking for more practical diagnostic purposes, those type of units can have meaning to them. When you look [00:40:00] at diagnosing intellectual disability, looking at how far someone is below the typical same age peers, that’s a halfway decent metric to be able to classify those individuals there that, oh, their adaptive skills are quite a bit below what is typical for people their same age or their cognitive functioning is quite a bit low there while from a measurement perspective, it’s probably not the best scale to use, from a clinical perspective or diagnostic perspective, those scores can have clinical use or clinical utility to them.
So I wouldn’t necessarily throw away them altogether but interpret them in the light of here’s what they’re actually doing, here’s what they’re actually representing, and they’re not some absolute unit like inches or meters or seconds or something along those lines. There are very relative unit that aren’t comparable across instruments.
Dr. Sharp: That’s okay. That’s good to hear. I think you touched on [00:41:00] a point that I’ve seen you talk about in the Facebook group and elsewhere in your research is like, just because, I’m trying to think how to explain it, someone who scores, let’s say, a 115 on the WISC, it’s hard to really quantify how much more intelligent they are than someone who has a 96, for example, we don’t have a great idea of what that actually means in real life.
Dr. Alex: Exactly, and that is part of the author/publisher issue there is that they have these numbers but we really don’t have a very good understanding about what that means. You say someone has IQ of 100, well, I don’t know what that necessarily means. It really depends on how old they are, who the normal sample was, what instrument they took. So that 100 doesn’t really have meaning outside of that one particular [00:42:00] instrument there.
There are ways to go around that. I think the Woodcock-Johnson has the means to be able to do that because they scale everything using the Roche model and they could just W score so there actually are items that map onto every level or every person’s W score. So someone gets a W score of, let’s say, 500, there’s actually items that map on to that exact same level. So could say, oh, well, this is all the type of skills that they have in verbal comprehension or math calculation, et cetera there.
So there are ways to get information for that but that typically is not what is done by most popular standard test publishers.
Dr. Sharp: Yeah. I hear that the Woodcock-Johnson is maybe an exception, something that a lot of us should strongly consider using, at least from that perspective [00:43:00] or maybe not.
Dr. Alex: I can say that when Richard Woodcock originally created the Woodcock-Johnson, he had some very novel ideas about that. He made a lot of progress in the test creation work especially the Woodcock-Johnson R and the Woodcock-Johnson III, there were some very novel things involved with that. I think it got lost on the test audience. I think the RPI is very underutilized metric there and the W score is not likewise there.
If you look at some of the stuff on the Woodcock-Johnson IV, I think that there’s some still unanswered questions there. If you look how they did the sampling, they used that bootstrap, so not everyone, actually very few folks in the normal sample actually got all the tests administered and they just got a portion of the tests. And so then they used this resampling to fill in all that missing data.
What the implications are of that, [00:44:00] it’s unknown at this point. Likewise, independent folks that have looked at the factor structure on the Woodcock-Johnson, they haven’t quite mapped on to what the test publishers say the factor structure is. So what those scores are actually getting at or measuring, that’s probably still an unanswered question.
Dr. Sharp: Sure. Okay. The search continues. From that perspective, do you know of tests out there that you would say are “better” than others or more empirically sound? If you were supervising someone, how do you go about constructing a battery based on what you know about measurement?
Dr. Alex: It depends on what the reason for referral is. [00:45:00] So if you’re looking at things like ADHD, whatever, actually, there was a new book that just came out, I think it was this year about assessments that work. They go through most of the areas like depression, bipolar, et cetera, that most clinicians will come across. They review the literature in that.
In some of the things like ADHD, you’re looking at maybe a questionnaire or structured interview but there really isn’t a whole lot of indication that standardized scores are very useful there. So you would probably want to focus in on interviews or maybe ADHD-specific questionnaire for a client or their parents and teachers or what have you there. So it depends on what you’re looking for an assessment.
Dr. Sharp: Can we maybe just take two concrete examples? I think that folks are probably interested in that. Like maybe just starting with [00:46:00] ADHD, is there any utility in administering cognitive measures at all?
Dr. Alex: I’ve not seen literature that has indicated that having scores from an IQ test or some type of neuropsychological instrument gives you any more information about diagnosis, or give me more accuracy in diagnosing ADHD than just doing a focused semi-structured interview on attention issues or some standardized questionnaires to give the parents, teachers or maybe the client there if they’re old enough there.
For diagnostic purposes, I’m not saying that it’s not, won’t ultimately turn to be helpful, it’s just at this particular point in time, I’ve not seen much literature that has supported the use of cognitive instruments as far as [00:47:00] the utility of them goes in making the diagnosis.
A diagnosis is only one component of the assessment. You also need to rule out other things. So if you’re wondering, oh, is this ADHD or is it maybe there’s intellectual disability or whatever. Well, in that case, there would be usefulness in administering a cognitive instrument because you need to rule out, make sure that it’s not intellectual disability that’s causing the behaviors that you’re seeing.
And then as far as interventions go, you probably want to focus largely on functional type of assessments, like what are the particular behaviors in these particular environments that are causing problems? What’s the function of those behaviors and how can they be replaced with other behaviors that’s going to help them succeed more in their particular place in life?
Dr. Sharp: What about a learning disorder [00:48:00] assessment, how are the measures looking with that?
Dr. Alex: I had an email conversation with Jack Fletcher and he has a new edition of his book coming out there. I think that he has a very good model there. It’s time efficient and it largely looks at functional problems there. Basically, they’re looking at some type of the history of academic problems, whether it’s reading or math or whatever there, but there’s a history of they’re not achieving commensurate with the same age peers in a particular academic setting there.
The academic institution has tried to do some type of intervention that tried to do something to help them and those things aren’t necessarily working. Then the question is, well, why aren’t they working? And so then at that point, you can maybe give a standardized achievement test to say, [00:49:00] are there particular skills much lower than their same age peers on a national level? If that’s the case, then maybe look to actually do some more functional academic assessment, but what are the actual functional reading skills? What are they able to do as far as the components of reading go, like identify letters, phonics, et cetera.
So it’s much more of a functional based but you also need some type of normative instrument to be able to say that it not only are they appear to have some functional problems but their current level of achievement is quite a bit lower than you would anticipate for someone in their same age
Dr. Sharp: And are you, I just want to be clear, are you distinguishing functional academic assessment from a standardized achievement battery?
Dr. Alex: I don’t know what you mean by a standardized achievement battery.
[00:50:00] Dr. Sharp: Like a WIAT or a KTEA or …Dr. Alex: WIAT or KTEA are going to be very good at telling you normatively how well is this person’s achievement compared to their same age peers. So they’ll be able to say, oh, this person’s math calculation skills are quite a bit lower than you would expect for someone who’s 12 or whatever years of age, but that doesn’t necessarily tell you why those skills are low.
And so that’s where the functional component goes is to figuring out what is it that’s causing the issue. Is it just a lack of exposure to the material? Is it some type of reversing numbers when they’re looking at them, they don’t quite get this? Is it the caring component that’s causing trouble?
There’s all sorts of reasons why folks can do poorly in an academic area. So the functional aspect gets to, okay, normatively, we know that they’re low but what is it that’s causing them to be low there.
Dr. Sharp: Can you give me an example of a functional academic assessment?
[00:51:00] Dr. Alex: Yeah, a lot of curriculum-based measures like the DIBELS or easyCBM, things along those lines would constitute functional type of assessment because they’re looking at very specific areas and skill set. And so it’s not just how well does someone perform compared to the same age peers but here’s what we expect what the typical level is for a 2nd grader to do, and they’re not able to do this. Here’s what we expect for a 3rd grader to be able to do and they’re not able to do this particular skill with any fidelity.These are skills we expect 2nd or 3rd graders to have, and they don’t have them yet, that’s probably causing their low scores on the WIAT or KTEA, et cetera.
Dr. Sharp: Yeah, I see what you’re saying.
Dr. Alex: And fortunately, a lot of those, from a clinic perspective, I think the DIBELS, whatever, they’re free or they’re relatively [00:52:00] cheap to use. Most of those companies make their money from schools when they’re doing those assessments over multiple time periods and they’re using a computer to do all the scoring, et cetera, there.
I often have students that I’m supervising in the clinic. When students come to the clinic for learning disability assessment, that I’ll have the students do that type of basic evaluation to be able to get some type of indicate, not just interviews and below that the mean, but what are their functional skills that they’re showing some deficits in.
Dr. Sharp: That makes sense from a broad perspective. I could see that. You need to see what’s happening actually in the work that they’re supposed to be doing at school.
We got to delve into the research that you’ve done on PSW and some [00:53:00] of those LD identification models. I get the sense from your research, you’re not a huge fan of that approach or the research maybe is not a huge fan of that approach, if that’s fair?
Dr. Alex: Yes. In training, 10 years ago when I was going through this, a lot of those discrepancy type models were what the norm was. In fact, there wasn’t a whole lot of options then because the DSM at that time said you had to use a discrepancy to make a diagnosis. There really wasn’t a whole lot of options.
The DSM-5 has changed that significantly. The new IDEA legislation a few years ago changed that significantly. So there’s a lot more options available there. And so I think now that there are more approaches, both legally and clinically available to folks to use to make a [00:54:00] diagnosis, that it’s a very legitimate question about, okay, PSW has been around for a very long time. It’s not always been called PSW, but the idea has been around for decades. What is the utility that goes behind that? Are there other methods that have better utility? We’re not just forced to use this one method. We actually can use multiple methods. So let’s see which ones actually have the best utility to use.
In a lot of that PSW stuff that sounds very good, if you read the materials, they sound like, oh, they should work very well but then you say, okay, now what type of studies have they done to demonstrate that? Nor does it sound like it should work well, but it actually does work well. And that’s where the problem arises that they’re just not a whole lot of empirical studies done.
There’s lots of stuff published but a lot of that is opinion pieces or this is why this should work or [00:55:00] case studies. I don’t want to say case studies pejoratively because cases can be very useful, but case studies that are selected for whatever reason, there’s not much background given about why these particular folks were selected to show that the method works, et cetera. So there’s not lots of empirical evidence that would indicate, oh, this method actually has utility in doing that.
Conversely, typically folks that aren’t affiliated with the PSW, they’ve done research with that have shown that it tends not to be very sensitive. It’s very specific, meaning that it can do a decent job at finding folks that don’t have a learning disability, but the sensitivity of those methods tends to be relatively low in that it’s not picking up on the folks that actually do have a learning disability.
Again, granted, there’s not oodles of literature out there but what is out there, I think maybe there’s a dozen or so articles, [00:56:00] and when you think about this has been around for a good, the PSW thing has been around for at least 10 years. Previous iteration has been around for much longer than that but that seems very unusual that something’s been around that long and there’s such little empirical work done on that.
Personally, I think it’s because on the face of it, it looks like this should work. This makes perfect sense why it should actually be picking up on what the people that say it should be picking up on, what they tell actually should work. But then like all good scientists want to say, well, where’s the data that map onto these intuitions and then that’s where the problem arises.
Dr. Sharp: I’ll go back to that question of, if you were supervising someone, how do you coach them through identifying a learning disorder? What’s that process then?
Dr. Alex: If it’s in a clinic which is what I do now, we [00:57:00] try to gather as much people background information as possible. We try to get a lot of work samples. If they come in with a problem with writing or what have you there, you collect as much of homework or in-class writing assignments as they have. Let’s figure out where the problems tend to arises because there’s a motor problem and they just are having very difficult time actually making the letters.
Is it lack of previous exposure to material? So there in 4th grade but they were out half of the 3rd grade and so they haven’t caught up on the concept. Let’s actually get some examples of what the… just saying there’s a writing problem is very broad, it’s kind of narrow this down a little bit there.
And then once you get some work samples, we see, well, I need to rule out things. The problems that we’re seeing, are they due to other issues? Is it a general cognitive functioning problems? Let’s make sure we can rule out intellectual disability or other things that may have a direct [00:58:00] cause on writing issues there.
And if those can be ruled out, then we have to rule out if they had exposure to proper instruction, they actually had enough time in class to be able to learn the concepts. If they haven’t had time in class for what illness or whatever reason, there’s no reason for us to expect them to have the skill set of their same age peers.
So it’s a lot of background information collection or I’m ruling out auxiliary hypotheses about what could be causing that. And then because we’re in a clinic, we can’t necessarily, if schools are in the Florida situation being able to collect lots and lots of data over a very short time period, in clinics we don’t always have that option to do. We’re looking at a standardized achievement test, the WIAT or Woodcock-Johnson.
Woodcock-Johnson tends to have a lot more options in a given academic domain than the many other instruments but getting a standardized instrument about saying, hey, there seems to be a problem in them [00:59:00] doing a particular academic skill. We’ve ruled out that it’s probably not due to some other issues that could be causing this, let’s see if there’s actually a normative deficit that is here or is it just something that they happen to be in a very difficult school and so they’re not achieving well compared to those folks, but normatively across the nation, they seem to be doing relatively well.
So giving a national norm standardizes can tell us, not only are they having difficulties at that particular school but also seems at a national level that they’re quite a bit below their same age peers. And so if that’s the case, then we look to see, well, okay, what are the functional problems or what are the functional issues that they’re having with writing or reading or what have you there that’s causing both the problems in the class, on the standard or the nationally normed assessments, why they’re so low. So that’s what I tend to tell my students to do and look for.
Dr. Sharp: [01:00:00] When you say a normative deficit, I assume you’re still looking at a score above or below a certain point. What would you consider a normative deficit like on a…?
Dr. Alex: I don’t think there’s a particular threshold like, oh, 70, there’s a deficit, 75, there’s not a deficit. I don’t know that there’s necessarily a hard and fast threshold there. Typically, folks are looking at 1, 1.5 or more standard deviations below as at least throwing up red flags if not indicating that there’s definitely a normative deficit.
It could probably depends on how reliable the scores are. If you look at the confidence intervals on these instruments, some of them are quite tight meaning that they’re relatively reliable and some of them are quite large. And so once you start looking at the confidence and you’re like, oh my goodness, [01:01:00] the score says 80 but their confidence interval goes from 72 to 89, well, that’s a huge range of a different to what their actual score could be.
So I don’t want to say that, oh, here’s a particular threshold because I don’t think that that does anybody any good to say that this is the magic threshold. You have to look at what the confidence intervals are on those particular instruments and then what does it mean on that particular scale that they are standard deviation or standard deviation and a half, whatever, below the mean.
Dr. Sharp: Sure. I might be revealing my maybe pathological need for structure here, but it seems like to some degree, it’s a moving target to identify some of these things depending on several factors, right?
Dr. Alex: Yes. [01:02:00] And so far as that most learning disabilities don’t have a homogeneous cause so then it would be a moving target because there can be multiple reasons why someone is struggling in reading or struggling in reading fluency or math calculations or something along those lines. If there is maybe one or two ideology for a given learning disability, it probably wouldn’t necessarily be the case but there can be a pamphlet of reasons why you get five kids that have problems with reading, that can be five extremely different ideologies for those academic deficits there.
So it is somewhat of a moving target there because there’s lots of different reasons why they could be showing the deficits. That’s part of the assessment is you need to rule out some things. If general cognitive delay is causing the reading difficulties, then you wouldn’t want [01:03:00] to diagnose with a learning disorder because there seems to be something a bit more pervasive going on there. Or if they just moved to the country a year ago, in the country where they moved from, English is not the primary language, you need to give them adequate amount of time to be able to catch up and learn the language.
So there is lots of particular reasons why folks can be doing poorly. And so I would say that here’s the A, B, C, D things that you have to look for, for learning disorder. Like, oh, if they have the score of this standard deviation below on this, and they have three or four curriculum-based measures that are below their target line, these are the minimum criteria because having those type of things just can’t account for all the heterogeneity involved in why folks have academic issues.
Dr. Sharp: I like the approach [01:04:00] certainly of ruling out other things, that makes a lot of sense. But then thinking about, when you do get to that point where you say, okay, I don’t think it’s any of these other things that’s getting in the way. Now we have these achievement scores in front of us, how do we determine if it is truly a normative deficit or not and then you get into looking at confidence intervals and maybe some other things too? There’s no strict formula, it sounds like.
Dr. Alex: I think that’s probably a good thing. If you look at the history of LD on specific learning disability legislation across the states, it used to be decades ago that there was very strict. Some states maybe still have them, very strict criteria. If their IQ or if their achievement is 22 points different from the predicted achievement based on the IQ, then that’s an SLD. If [01:05:00] they are 21 points, then it doesn’t meet the criteria.
And then literally that’s what states had codified to be the definite or the operational laws of SLD. I don’t think we want to move back to that stage that, oh, 21 points, you don’t have the diagnosis, 22 points you do, because that just causes a lot more issues than it probably solves.
Dr. Sharp: Sure. I’m totally with you on that. I think this whole conversation is probably making a lot of folks rethink our approach to this process. We’re talking about how tests may not be measuring exactly what they say they are, and then the scores are hard to compare across tests. It’s just got me thinking, how do we do this? What’s the right way to do it?
In one of the [01:06:00] articles of yours that I read, you talked about, I think you called it an evidence-based approach to assessment, which sounds great. Can you speak a little bit about that? If you have any resources around what you call evidence-based assessment and where people might look for that.
Dr. Alex: Yes. What I mean by evidence-based is, and Eric Youngstrom has a lot on this and Hunsley and Mash do as well. And that’s where I got a lot of that terminology is from those authors that have written very prolifically on that. And so if you look at those particular authors, you’ll find that what they’re looking for is empirical data. So it’s not just, oh, this particular instrument or this particular score should actually be able to pick up on this particular disorder but looking at how well [01:07:00] do these scores actually pick up on the classification or the diagnosis that’s needed there.
So they typically will have some type of gold standard type of assessment, then they’ll look at these other questionnaires are shorter forms and see how well can they actually pick up on that. The idea is that it’s not just based on what should work or what folks say does things, but it’s looking at empirical evidence over multiple studies that show that, oh, this particular instrument is able to pick up this particular diagnosis much better than chance.
Dr. Sharp: Yeah. I love that. I’m sorry if I missed what you said, does he have a website or there’s a place to look at that list?
Dr. Alex: Actually a second edition of the book, A Guide to Assessments That Work was edited by Hunsley and Mash, just came out this year as probably one of the better places to start. [01:08:00] I think it was published by Oxford University Press. There’s a few chapters in there at the beginning about what is the evidence-based assessment approach there but then after that, there’s a chapter on ADHD, depression, bipolar disorder, schizophrenia, general anxiety, just about all the major areas in the DSM, there’s a specific chapter on that. The authors basically give you a very nice review of what instruments seem to have evidence that are good for diagnostic purposes.
Dr. Sharp: I love that. I’ll definitely link to that in the show notes. I’m guessing people are probably going to be interested in that, so I’ll make sure to look that up and have a link to it. I like that approach. It’s crazy, I’m just reflecting that this is, I didn’t hear anything about this kind of stuff in graduate school granted it was a while back but it’s [01:09:00] nice to see us moving in that direction.
I know we’re getting to close time wise and we’ve been talking for a while, which I really appreciate. I wanted to see if there’s anything else I wanted to touch on, while I’m thinking about that, we’ve talked about a lot, is there anything that you want to make sure to throw out there, to let people know as we’re thinking about assessment and what works and what we need to be looking for when we’re doing assessment?
Dr. Alex: I think that the biggest thing is to stay abreast of the literature in the field, not just in particular area, but keep abreast of the literature on particular instruments. I did my undergraduate internship in a psychiatric hospital and I never will forget the fact all these drug manufacturers came there and had all sorts of paraphernalia about their drugs.
Paxil was one of the big [01:10:00] things at the time. Its representative came, had Paxil pens, Paxil balloons, Paxil pillows for all the clients. They weren’t talking a whole lot about how well it worked or not, they just had all this nice paraphernalia so everyone’s had their own little Paxil t-shirts and what have you there.
I thought that’s odd. They’re not talking about the drug per se, they’re giving us all this stuff that makes us think positive things about the particular drug, maybe with psychiatrists, they were doing that a bit more but I just thought, why do they have all these other stuff that just seems a bit odd to me.
Fast forward now to 2017, 2018, the same kind of thing is happening with psychological assessments. When the Wechsler comes out, there’s all sorts of paraphernalia, go to this webinar, get this t-shirt, get this free coaster that has WISC-V on it, what have you there. So there’s lots and lots of hype behind a lot of these instruments, especially from the big testing companies there.
I guess the best thing or the biggest thing I [01:11:00] could suggest to folks in assessment is to do your due diligence behind before you adopt an issue. Just because it’s the 7th edition of an instrument doesn’t necessarily make it an appropriate instrument to use, every new edition is actually a new instrument that’s coming out there. So do your due diligence and figure out does the test author or test publishers actually provide you enough information in the manual for you to decide that, yes, I would feel comfortable defending my decision to use this particular instrument there.
And likewise with the instruments from the manual because most of these instruments now give you lots and lots of scores to use. So just because the score is there doesn’t necessarily mean that it should be used or so do the due diligence behind that and figure out oh, does this score make sense to interpret, just because it’s printed out in a nice [01:12:00] document does actually make sense for me to use it or to interpret it.
So to do that due diligence behind the instruments and you’re probably going to have to rely on things other than the technical manual. So unless folks feel very comfortable delving deep into the technical manual, that’s hard to do. Some of them are manuals a bit, they obfuscate a bit more than others do and so it’s hard to terse out the wheat from the chaff there.
There are other resources like tests or mental measurement yearbooks. That’s probably one of the best underutilized resources out there. Kurt Geisinger is the head of that in Nebraska, and they just published reviews of all these commercial instruments by folks that have no affiliations at all with the instruments so there’s no financial conflict of interest at all.
And with the big instruments, they often get two or three reviewers there and they just work through [01:13:00] the technical manual and the ancillary materials very detailed and let you know, hey, this instrument seems to be doing what it’s supposed to be doing, this instrument that says it’s doing one thing but it’s actually seems to be doing something else.
What’s really nice is that those reviews are relatively affordable. I think they’re maybe like $15, $20. You can go to mental measurement yearbooks online and you can purchase them but they’re relatively affordable. Especially, we look at the cost to review was $2000 for a cognitive instrument by time you get the kits and all the protocols and everything and there’s $20, $30 for a review, they’re quite economical there.
I’d ask folks to make use of those resources. If you don’t feel comfortable going through a technical manual and figuring out, oh, this seems to be on the up and up or this seems to be a little bit shady, make use of those other resources that are out there. There are a lot of them out there, it’s just that they don’t get utilized a whole lot.
Dr. Sharp: Got you. I was just looking [01:14:00] through that list and it looks like a very comprehensive manual. That’s great. I will link to that as well.
Let me ask you two random questions here in the last two minutes that we have knowing full these might be really big answers but I’m just going to go for it. Do you have thoughts on Q-interactive?
Dr. Alex: Yes, my thoughts are the exact same as with the others which the Pearson has touted them as being fully equivalent to the paper and pencil versions. They have maybe one technical report for each of them as their sole line of evidence that the scores are actually equivalent. From my perspective, that is not enough.
I think at best what we could say with Q-interactive is that you have the WISC-V paper and pencil and the WISC-V Q version and those are two separate instruments. I would not say that the scores are interchangeable at all
[01:15:00] until such a time that Pearson can provide evidence that, oh, 100 on this instrument is equal to 100 on this instrument and heretofore they’ve just not provided that information.Dr. Sharp: Okay. That’s a definitive statement. I appreciate that. Second random question, since you worked with Eric Youngstrom, I feel like I have to ask you because I cannot seem to get him on the podcast. He’s so busy. Do you have a sense of the best way to assess bipolar disorder in kids?
Dr. Alex: When I did my internship, the gold standard was this full diagnostic interview that we had to do was maybe five, six-hour interview. It was extremely comprehensive to do, that’s probably not going to be efficient for most clinical purposes. I know Eric has two articles [01:16:00] relatively recent that he goes through this model that he uses looking at prior probabilities and posterior probabilities there, and looking at two different screening type of of instruments there.
He walks through some instruments. Off the top of my head, probably I am not going to be able to tell you the names of them accurately, so I won’t try. I know he has a few very explicit articles that these things have a lot of utility and they don’t cost you four or five hours of a large interview with the client and 25 of the client’s family members et cetera. That’s just probably not feasible for most clinicians there.
At least with juvenile bipolar, he has published a lot with that. The names aren’t, I don’t have off the top of my head, without being able to look them up, I won’t be able to [01:17:00] give them to you accurately, but I know that he has quite a few publications on that.
Dr. Sharp: Okay. I think I’m familiar with two of those articles so I will go back and link to those as well. Man, thank you so much. I feel like we’ve talked about a lot of pretty important topics. It just leaves me with more of this question of what are we actually doing here with our assessments and just wanting to be extra diligent that we’re using the best measures we can and that we’re not becoming overconfident, I suppose, with what we’re measuring and how to translate that to clients.
Dr. Alex: That’s probably the best stage of advice you could give, not to be overconfident and to treat every instrument as a hypothesis, is this instrument actually doing what it says it’s doing? Are the scores actually able to diagnose accurately? Those type of things. Come at it with an inquisitive [01:18:00] mindset, it’s probably the best advice for any clinician.
Dr. Sharp: Sure. Thank you one last time for being here with us and talking through all of these pieces. For folks listening, Alex has tons of articles, two books, you can easily do a search and find any number of research publications from him. If people do happen to have questions and might want to get in touch with you, what’s the best way to find you?
Dr. Alex: They can email me, that’s probably the best way to get ahold of me. My last name is Beaujean and I’m available. If you type that into Google, all my stuff will come out. My actual email is alex_beaujean@baylor.edu. If they want to remember that to type it in. If they email [01:19:00] me, I’ll be happy to do what I can over email to answer questions or point them in the right direction.
Dr. Sharp: Absolutely. It’s very generous and I will throw in a plug as well for the Facebook group. You’ve been really active in there over the last several months and I see you answering questions and offering perspectives in The Testing Psychologist Community too, which I’m thankful for.
Dr. Alex: And I learn about as much as I contribute so I’m glad to be part of it.
Dr. Sharp: Oh, that’s fantastic. We’re glad to have you. Well, thank you again. This has been fantastic. I hope our paths cross again in the future.
Dr. Alex: Definitely.
Dr. Sharp: All right, bye-bye.
Dr. Alex: Bye.
Dr. Sharp: Okay, folks. There you have it, Dr. Alex Beaujean, and an information-filled episode on testing and measurement. I would love to have him back on and focus myself a little bit but my hope is that you took a good bit away from this and at least, if nothing else, [01:20:00] maybe you’re starting to think about checking out the grounding of the tests that we use and the validity and their measurement statistics and the technical manuals and a lot of the other things that he mentioned and just becoming open to the idea that not all tests are created equal. It was very eye-opening for me.
I did not mention anything at the beginning of the podcast in terms of “calls to action” so here you go. I’ve got paperwork packets for you. If you need some paperwork in your testing practice, there’s a clinical packet, an administrative packet, and a psychometrist training manual. You can get one or two or three, all of those, whatever you would like at thetestingpsychologist.com/paperwork. You can get 20% off your entire order if you use the code “podcast”.
If you’re not a member of the Facebook group, [01:21:00] The Testing Psychologist Community, please come join us. You can search for that on Facebook. That’s where you can find it, is on Facebook. So just search for it. It’s a group and we have tons of discussion about everything testing. We’re up to about 1400 members now, and there’s a lot of information flying back and forth.
All right, y’all take care. We’ll talk to you next time. Bye bye.