Hey everyone. Welcome back to another episode of The Testing Psychologist.
I’ve done episodes in the past on field research and data collection. I’m going to continue that theme here today with another episode focused specifically on field research and our roles as practitioners in this whole process.
My guests today are Kristina Breaux and Jan Williams. They are both employees at Pearson which I’m sure you recognize as a major test publisher. Let me tell you a little bit about them and then I’ll talk a little bit more about what we [00:01:00] cover.
Dr. Kristina Breaux is a licensed special educator, learning disability specialist, researcher, and published author. She is the Principal Research Director at Pearson, where she leads the development of clinical assessments for measuring academic achievement especially those used for dyslexia, dysgraphia, and dyscalculia evaluations. She also serves as the Chair of the Scientific Council at Pearson, a group that oversees the scientific standards and methods utilized during product development.
Since completing her clinical training at Northwestern University, Dr. Breaux has taught students with learning disabilities and other neurodevelopmental disorders across the grade range, also age range, and conducted numerous psychoeducational evaluations for children, adolescents, and adults.
My other guest, Jan Williams is a licensed Speech-Language Pathologist with [00:02:00] clinical experience conducting assessments and interventions with the school-age population as well as with adults in inpatient rehab treatment facilities.
After she left her clinical practice, she joined Pearson back when it was Harcourt as a Research Director and spent several years developing assessments for the Speech and Language markets. Jan now serves in dual roles as the Director of the Project Management Office, as well as the Director of the Field Research team for Pearson Clinical.
In the field research role, she’s responsible for collecting research data needed to develop Pearson’s Clinical Assessments. The Field Research team contracts with hundreds of professionals: Psychologists, Speech-Language Pathologists, Occupational Therapists, and so forth to administer research versions of all of the tests in support of the Test Development process.
So, fantastic guests today. What I love about this episode is that we focused [00:03:00] on the practitioner experience of field research. first of all, just to establish some context for field research, why it is so important to get good data, and what an important and valuable role practitioners play in that process in gathering that data.
Then we dive into the logistics of field research; what it’s like for practitioners, how to apply, how to get selected, how to recruit, and some of the practicalities around field research so that you can get a really good idea of how to do it if you would like to. We do touch on some of the differences in administration when you’re doing so for research purposes versus in your everyday practice. And we touch on a number of other things that I think are quite relevant for us as practitioners.
If you’ve ever wanted to play a more important and [00:04:00] active role in the Test Development process, this is a good episode for you. As always, I think there’s a lot to gather from our interview. So let’s go ahead and jump to my conversation with Dr. Kristina Breaux and Jan Williams.
Kristina, Jan, welcome to the podcast.
Jan: Hi, Jeremy. Glad to be with you.
Dr. Kristina: Thanks for having us.
Dr. Sharp: I’m very glad to have you all. I’m excited to talk about this topic of field research and how clinicians can participate in this whole test development process. I know that you too have been doing this kind of stuff for a long time. So, I’m thrilled to have you here.
I would love to start and have y’all talk a little bit about what you [00:05:00] do, what your roles are, and that’ll also give folks a chance to orient to each of your voices so that we have a better idea of who’s jumping in throughout our interview. Jan, do you want to go first?
Jan: Sure. Thank you. I’m Jan Williams and I am the Director of the Field Research group within Pearson Clinical. My role and my team’s role is to collect all of the data that we need to develop our clinical products and the assessments that are used by many clinicians. That’s my role. That’s what I do.
Dr. Sharp: Thanks. Kristina.
Dr. Kristina: Yes, I’m Kristina Breaux. I have two different roles at Pearson. I’m a Principal Research Director and an R&D manager. I’ve been developing clinical assessments for 17 years. My background is in learning disabilities and special education. So I specialize in academic learning and [00:06:00] achievement assessments. I led the development of the WIAT-4 most recently as well as the KTEA-3 and the RAT-5 just to name a few.
The Clinical Assessment Group, by the way, covers about 8 different domains. So, in addition to academic learning and achievement, we develop assessments in the areas of neuropsychology, cognition, personality and mental health, speech and language, early childhood, occupational therapy, behavior, and socio-emotional learning. So we have experts on staff in each of those areas.
I’m also the current Chair of the Scientific Council on Clinical Assessment. Just to say a little bit about what that is. We have an international group of colleagues from our offices here in the US and other parts of the world. So, currently, we have Australia and the UK represented, but we’ve also had Canada and India on [00:07:00] council as well. And our group advocates for the science of clinical assessment. So we establish standards for quality and best practices in test development. We want to make sure that the standards that we’re following are in accordance with the standards for educational and psychological testing and the standards put forth by various professional organizations and governing bodies.
When practitioners hear the name Pearson or some of our brands like Wechsler and Kaufman, those names tend to be associated with trusted high-quality assessments. And so we want that to always be the case. We don’t want to get complacent and take that reputation for granted. We want to make sure that our products are defensible. That requires evidence, and the foundation for providing that evidence to support the reliability and validity of our products is in this data set that we collect for each of our products.
Many of us who work [00:08:00] at Pearson we’re former clinicians and practitioners ourselves, and so, we understand how important it is that the results from our clinical assessments are trustworthy and that they lead to improved outcomes for the individuals being assessed. So, we have a scientific council whose job is to ensure that we deliver on that promise.
Dr. Sharp: I appreciate you diving into that a little bit. I know I asked that question in our pre-podcast chat, what is the scientific council exactly? I think it’s important for folks to know that that’s out there and it’s a pretty big part of this whole process.
Let see. This will be an interesting question with the both of you, but I always begin by asking why this kind of thing is important to you. When they presumably asked, who wants to talk about this topic on a podcast, why did you volunteer? Why is this important? [00:09:00] Let me see. I’ll go in reverse. Kristina, you can go first.
Dr. Kristina: Well, as I said, we’re committed to maintaining the quality of our assessments and the foundation for that is in the data that we collect. It’s such a big part of the test development process and such an important one that we’d love to just engage with your listeners on this topic to see if any of them would be interested in working with us if they have that same commitment to quality.
We’re always looking for partners in this work. It’s difficult work. It’s challenging. And so, we need good people to partner with in that work. But also, even if that’s not a possibility for some of your listeners, I think it’s always a good idea to demystify the test development process a little bit with people and let people into what goes [00:10:00] into these decisions and the work that we do because none of us went to school for developing clinical assessments. And so, it’s not something that many practitioners are very familiar with. And so, I’d also like to share a bit more about that process.
Dr. Sharp: Wonderful. Jan, how about you? Why spend your time and energy on this?
Jan: I am also a Speech-Language Pathologist by training. And so I happen to play these different roles within Pearson. Way back in the day, I was a practicing professional out there serving children and adults in rehab settings and so I used many of these products as I was in my clinical practice. To be truthful, I scanned the technical information as I was going to the norms to write up my reports.
Over the years, [00:11:00] Pearson had sent out a note about, “Do you want to be a part of this process?” And I had no idea what had gone into those tests. I just thought if I bought it from Pearson, I trusted it and that was it. But there’s so much more to it than that. And so I realized that if you don’t have a large enough sample or a representative enough sample, you don’t know that the test results are telling you what you think they are.
And so, many years later, decades later, I had the opportunity to work for Pearson Harcourt before that, if any of you are familiar with that. And then recently, around the time of COVID actually, I was asked to take on the field research group. That’s the group that has the job of partnering with professionals such as you all who are listening to help us collect that data.
Now [00:12:00] that I’m on the other side of that, it is a hard task. We’re looking for very specific kinds of candidates and there’s lots of inclusion and exclusionary criteria, but it’s what we need to go into the data that makes up our test. So because I started as a clinician and now I’ve been with the development effort with Pearson for many decades, I’m very passionate about it, and I do think, as Kristina said, I don’t think not enough people maybe, or know about this process and know how they can get involved. And we’re always looking for more professionals to be involved in this with us. It just helps us get better and better representativeness of our data.
Dr. Sharp: Yeah, that makes sense. That’s great. It’s cool that you started out on one side and moved to the other, [00:13:00] get it from both perspectives. I don’t feel like a lot of us get to do that. And like you said, it’s a little bit of a mysterious process, test development. Even for me, I’ve had conversations with folks about this process and it’s still a little bit mysterious. I’m not exactly sure what happens. So, I’m excited to talk more about it.
I would love to double click real fast on something that you said, Kristina at the beginning, this idea of good data and how important that is in this whole test development process. Can you say any more about that just to put the rest of our discussion in context as we dive into what field research is and how we can play a role?
Dr. Kristina: Yeah, it’s a great question. When we talk about the data that we collect, we collect different types of data [00:14:00] that provide evidence for quality. It’s evidence that the test is reliable, it’s valid, it’s doing what you think it’s doing.
So some of the data is in developing the norms and that’s I think what we think of first, right? And so the normative sample is that reference group that we collect that’s designed to be representative of a particular population and to provide a full range of abilities that we can use to make meaningful comparisons of performance. So that’s one type of data that we need.
We collect also clinical data that provides clinical validity data, and so, there are specific requirements around collecting that sort of data. And so we can talk maybe a little bit more about that.
We have reliability and validity studies. Those are very specific types of data that we need to support the product. And so, in [00:15:00] each of those studies, we have to go through all of that data that examiners collect for us and make sure that it’s of high quality, that it’s making sense, that there’s nothing… We look for outliers and data that might need a careful look. And so it’s just really important that all of the data in each of those studies is of the highest quality so that we report results that are accurate and trustworthy.
Dr. Sharp: Can you say more about the difference between the normative data and the clinical data?
Dr. Kristina: Yes. The normative sample sometimes includes some clinical cases. So sometimes we do seed in some clinical as well. For other projects, they don’t include clinical cases in the normative sample, but basically for the normative sample, our inclusion and exclusion criteria typically excludes a lot of [00:16:00] clinical conditions. So we want to see that normative sample being typically achieving or typically performing in a broad sense. And so we’ll have the inclusion-exclusion criteria specific to that. So we exclude people that are taking certain medications or have conditions that would impact their performance. So that’s more of our typical sample.
And then the clinical studies are really focused on particular clinical conditions that are relevant for that product. And so what we’re trying to do with those studies is establish validity. So we want a group of individuals who have a particular clinical diagnosis, and we try to exclude a lot of other comorbid conditions that might typically be seen with that clinical condition.
So, even though we know that comorbidity is the rule rather than the exception, for validation studies, it’s most helpful if we can [00:17:00] isolate a particular clinical condition because what we want to show is the impact of that clinical condition on test performance. So that’s why it’s helpful to isolate. And then we compare that clinical group with a matched control group to show how well the test differentiates between them on that one clinical variable.
Dr. Sharp: How hard is it to isolate one diagnosis?
Dr. Kristina: Well, Jan can tell you.
Jan: It depends on the diagnosis. For example, ADHD often will have many comorbid kinds of diagnoses with them. Others lesser. So it depends on the diagnosis, but it can be very hard to find someone who’s that pure, whatever it is, right? TBI or pure kind of epilepsy or [00:18:00] pure kind of learning disability. For example, we often have reading and math and not just one and we do require evidence of those particular diagnostic conditions to the extent that we can. We don’t require MRIs or anything like that, but testing scores, we might want to look at those to ensure that you’re performing in a range that we agree. Like, yes, that is a language disorder, or this is a intellectual disability, or so on.
Sometimes that might require that you take what we call a qualifying test. So we might give a very short form, maybe of an IQ test just to double check to make sure that you do meet the qualifications for that criteria. So, for examiners who participate, sometimes it’s a little bit of extra work to help us find those particular clinicals that meet our criteria.
Dr. Sharp: Yeah, you touched on [00:19:00] this just a bit, but my question is, how do you find those individuals with these diagnoses? Assuming we’re doing a lot of assessment for diagnostic clarity, are we recruiting folks that already have the diagnosis or is it a retroactive classification after we do the assessment? How does that work?
Jan: Kristina, do you want to take that one?
Dr. Kristina: Yeah. So the question is how we qualify a clinical case for the study?
Dr. Sharp: Yeah.
Dr. Kristina: Typically, we look for existing scores and documentation as a first step. A lot of times an individual will have already had a lot of really great testing. And as long as it’s fairly recent and covers the constructs that we need to consider, sometimes that’s enough. We can just base the qualification on existing test scores [00:20:00] and information that’s already been collected. If there are some gaps or we need a little bit more verification, then we can administer some additional tests or ask for a little bit more to help support those cases.
And again, this is for clinical studies. We have rigorous standards for those, but that’s not to say that we can’t still utilize some other cases that you might have access to. So sometimes you do have someone that has dyslexia and a bunch of other comorbid conditions like ADHD and so forth. Sometimes we can still utilize that case as part of the normative sample. We see it in some of those. So it’s not to say that we are only interested in pure clinical cases, that’s not the case, but some of the studies are more rigorous than others.
Dr. Sharp: Got you. Great. Thanks for doing a little background there. I don’t want to get super into the weeds with qualifying cases, but I appreciate it. I think we could bridge [00:21:00] though to this broader area of field research and what that looks like, how we participate as practitioners. And so I wonder if we might start with a basic definition. When we say field research, what does that mean?
Jan: Sure. And that’s probably a term that we made up internally for our business. It may not resonate to other folks, but really what we think about field research, it’s doing research out in the field or having research conducted out in the field, whatever the field is. Sometimes the particular project or test that we’re working on is for psychologists that work in the schools or with the school-age population, right? Sometimes it’s within speech and language. Sometimes it’s in emotional health and well-being. So, it’s that population.
We want to go out to the field, and usually, we’re looking [00:22:00] for examiners who also function in that field. So we would look for school psychologist, for example, if we’re working on a project for a school; speech pathologist, the same thing, if they’re in a school, or they’re in a rehab unit, we try to, first and foremost, it would be great if we could always get the examiners to participate on a project for which that’s their area of expertise. But that’s not always possible, and it may not even be practical or required. We look for professionals who know how to administer tests in a standardized way. And that’s the main gist, but it does help, certainly.
If you are used to testing the aging population, you understand the nuances of working with those individuals. Likewise, if you’re used to working with preschoolers, you know that [00:23:00] population, and if you try to go from a preschooler to an 85-year-old, that’s a really big difference. So there are some examiners who do a good job switching that way, but a lot of examiners function in the area that they’re most comfortable because that’s where they function each and every day.
Dr. Sharp: Of course, that makes sense. Great. And so we are the field.
Jan: You’re the field.
Dr. Sharp: Yes. Many fields.
Jan: We’ll try.
Dr. Sharp: Let’s talk about how this actually gets going. We can start at the beginning. Just to couch this in the whole test development process, when is field research happening in this process?
Jan: That’s a great question. So throughout, I’ll chime in a little bit and then Kristina can jump in a bit, but broadly speaking, in the test development process, [00:24:00] we first of all, if you don’t know, it might take 3 to 5 years for a product to go from its beginning to the time that you see it and are able to use it on the market.
The reason for that time period is because we are doing research all throughout to develop a good, appropriate, reliable, and valid test. And so we typically go through several rounds of research. We name them different things. We might talk about assessment labs or cog labs, which might be really small, and test item directions or maybe the item art.
Then we’ll do a larger study might be a pilot and that might be several hundred. Maybe we’re testing something different. We have our sub-test together. We’re looking at how the items fit together. Kristina will give more information about that.
And then finally, when we feel like we have the right sub-test and the right items, then we go out for that final phase of [00:25:00] research called standardization. As Kristina said, that’s the one that all of the test’s technical properties, the norms, the reliability, the validity are based on. So it’s multiple rounds of research over time. And each round of research, we do field testing. We go out to the field and ask them to collect data for us.
Kristina, do you want to add anything to that?
Dr. Kristina: Yeah, that’s a nice way to think about it. We have different research additions. And so, if you’re an examiner, you’ll see, oh, I’m using the pilot edition right now. It’ll say that on the forms and so forth. And so there’s definitely differences that you’ll notice in the pilot edition versus the standardization edition versus the final edition. There’s some really interesting differences along the way.
In early research, sometimes that’s when we can take risks. We can try new things. And so it might just be a few new sub-test ideas that we have. That’s [00:26:00] an interesting phase to participate in for that reason. You get to see some new ideas. We’re collecting smaller samples, but the discontinue rules may be really long. We may need you to administer all of the items in an early research phase because we need data on everything at that point. So things like that might be different.
The standardization edition is going to be most similar to final. So we try not to change things too dramatically between standardization and final. That’s a really good one to participate in if you’re interested in getting a sneak peek at the latest and greatest assessment that’s coming out before anybody else has seen it because it’s going to be pretty similar to the standardization edition.
Dr. Sharp: Got you. Okay. Well, then that makes me want to ask about how we can actually get involved and the logistics around that process. So is it an application process? [00:27:00] I don’t even know what else it might be. Like, we just submit a form on the website. Do you email people? How does this work if we want to do this?
Jan: Let me jump into all those fun things. That’s my area for sure.
There are two ways to get involved. Jeremy, I will send you the email links. I think if you can share those out with the transcript after this so folks can have those email links, but there’s two different ways.
On pearsonassessment.com, the Pearson website, there is a landing page with information about participating in field research. Lots of good information about how to do it, who can do, it all of those things. I’ll send the email to you there. And then you could also email directly my team at frexaminersupport@pearson.com and just say, hey, I want to be an examiner and we will take care of you.
But [00:28:00], when you talk about, is there an application process? How do you get to sign up? There is an application process. It gets you into our home or our space, get you into our space so we know who you are and we start working with you through the process.
The first piece, when we think about applying as an examiner, for the most part, if you are interested in doing this work, and you are a licensed professional for the products that you are going to participate in, we want you to come and work with us. There’s not a screening process and so on. We bring you in and we want to work with you. And so we do get your information about your credentials and your licensure and all that kind of stuff.
Then what happens is practically, you will go and create an account for yourself in our [00:29:00] examiner portal. That gets you registered with us, get some information about you so that we can pay you down the road and we know what kind of populations you have access to. It gives us some information about you.
And then from there, that’s when the real fun starts. As an examiner, we ask you to start recruiting candidates in your network. Now, depending on which projects we have active at the time or coming up, you may choose to recruit lots of children or maybe older adults. That’s up to you.
Recruiting comes from all the places you might think- all the places you interact in your world. We don’t let examiners test their immediate family just so you know that, but you can test your neighbors and others for good reasons. We don’t want you to tell the answers to your husband or your child or what have you, but you can recruit people, recruit all kinds of places: their kids’ soccer teams, their church environment, [00:30:00] their volunteer space, all those different places.
Some professionals recruit at their workplace. Now, there’s lots of things around there you need to be thoughtful of. They don’t necessarily recruit to test on working hours, but sometimes you might find a candidate that you have in speech and language therapy and if the parent is agreeable, and you’re agreeable, and you want to test them on a Saturday or outside of your work, it’s totally fine. So that’s the way some individuals find those clinicals.
But then oftentimes those clinicals have siblings and parents and so on. And so the network just grows from there. They have neighbors. Lots of examiners that we talked to say they recruit by word of mouth and social media these days. You can put it on and people say that they’re interested.
So, you start entering candidates in your portal, some [00:31:00] basic information that lets Pearson know, I have a 5-year-old named whatever, and here’s the birth date and they’re African American and they’re in the West or what have you. You start building that candidate pool.
When Pearson has a project and you said, oh, I’d like to participate in that project, first of all, you get the choice. You can decide which projects you want to participate in based on your work life, what you have time to do, and what you’re interested in. But if you have a project that you’re interested in, you’re putting candidates in your portal, Pearson will go in and make a match between individuals you have in your portal and what we need to fill our study.
We function with a matrix of needs, if you will. And in standardization, as you know, those are 1000, 1500, and then our total samples on some projects are 3000 to 4000 individuals once you put all the special [00:32:00] studies in. So Pearson does that match and then we make those assignments to you. This is all done through the portal. So you can keep track of that.
Now, one thing we do is we have the examiner submit a first test. So, once we make an assignment, you work with that candidate to schedule the testing at a time and a location that is mutually agreeable for you. Pearson doesn’t work with that. You do that. Then Pearson sends you all of the materials that you will need for testing the candidates. So we’ll ship that to you and we’ll give you labels to ship things back to us as you’re testing. So we send that.
Most projects have some sort of training, whether it’s a video we want you to watch or read the instructions and submit a fake test just to make sure you are really picking up on all the things that we need you to pay attention to. And then [00:33:00] we ask you to submit a first test. And that will come into Pearson and our research team like Kristina is in, they review that test for quality of the administration. And we give you feedback on that test. And so your feedback might be this looks great. Keep going. Thank you. It may be, don’t forget to let us know when you queried, or don’t forget to write the students’ answers verbatim or whatever it might be; just giving you little tips to make sure you’re going to be successful.
And then once you’ve tested your first test, we’ll start. That process just keeps going. We’ll make more assignments as you have candidates that fit our needs and your schedule and you test and you send those tests in as soon as you’ve got each one tested in the envelopes we send you with the label on it.
And then, our [00:34:00] content teams as well as field research, we do quality reviews on tests all the time. You’re just not… We may send you feedback along the way, but we don’t hold you up from testing that next one unless there’s some issue and we need to say, oh, you’re not understanding this. Let us explain to you how this sub test works and then you’re on your merry way. There’s a lot of support during that process.
Now, I talked a lot, Jeremy. I’m going to let you interrupt me or Kristina, what else would you add there?
Dr. Kristina: That was a great overview of the logistics along the way. There’s definitely a lot of benefits to participating along the way. You’re getting paid. Your name is going to go into our final manual, which is pretty nice. So we recognize you in that way. You’re learning the assessments and getting high-quality, free training on these [00:35:00] assessments from the developers, which is not easily accessed. So you’re going to have to pay for that in the field usually. So that’s another, I think, perk that I would highlight.
I think the one that most of our examiners would say is the draw for them is that they want to contribute directly to the development of these products. And so it feels really rewarding to be part of that feedback loop. So not only are we giving feedback to you as the examiner on how you’re doing, but you’re giving us feedback on how you think the test went. So you can tell us, I don’t like this item, those administration instructions are clunky, or I think you should improve these specific areas. And we listen. So, you’re directly involved in the development of these products. And so that can be rewarding.
Dr. Sharp: Sure. Can I go back and ask a clarifying question about the logistics? I’m curious. It [00:36:00] sounds like, Jan, you were saying, and maybe I misunderstood, feel free to correct me, that we do the recruitment before we might necessarily have a measure to try. Is that right? We decide we want to do some field research and then recruit as many people as possible, and just preemptively gather their names and information, put them in a database, and then we wait until there’s a project that might be relevant.
Jan: Yes, you can, or you hear we send out information like, we will be starting the, I’m just going to say WAIS, the cat’s out of the bag. We’re almost finished with WAIS now, but we will be standardizing the WAIS starting in September. We’ll tell you that and we’ll tell examiners on our, we have a portal that examiners can access once you sign up, it’s different than your portal, it’s for field research. It’s our hub actually. [00:37:00] And so we’ll have information about WAIS is starting this time and we’ll give you a brief description of the kinds of candidates we’re looking for, and the needs and so on. So you can start that process of recruiting.
Now, what we find with examiner, Jeremy, you might be leading to well, what if I go out and I find 40 candidates and I take the time to put their name in and you don’t pick any of them? That is true, right? And that could happen, I suppose.
One of the things that’s very important that we should mention here because sometimes people don’t realize this is, when we’re talking about a nicely representative sample, we talk about it as a stratified sample. So it’s got all the various demographics, various education levels, maybe 1 through 5, there’s different 5 different levels of education, right? There’s all the different ethnicities. There’s all the [00:38:00] different regions. There’s gender.
If you go and recruit all females that have a college degree from the South and they’re all white, we probably won’t use all 40 of those because we wouldn’t want that piece of the sample to come from one person. We probably don’t need 40 of that demographic. So the more broad your demographic base is in your candidate pool, the more candidates you are likely to get selected for.
Dr. Sharp: That makes sense.
Jan: And so over time, examiners who participate with us year after year, and we have many that have been with us years and years and years, decades, some that come on for a particular project and then they leave us. It just depends. But examiners who are with us for a while, they get that. They understand that. So they’re really [00:39:00] looking for a lower pad, other ethnicity, what have you.
There are certain populations for sure that we refer to as hard to find. They’re not only harder to find physically, but to convince to do this kind of testing, like the older populations, 80 and older, especially men, for whatever reason, is a particularly hard population for us to find. And so, we do support our examiners with some recruiting materials, some talking points, and sometimes some bonus opportunities to try to get that population to come on board. So we work in partnership with our examiners.
Dr. Sharp: I was going to ask about these hard-to-find populations. I’ve heard other developers allude to that. That seems like a challenging phenomenon to [00:40:00] tackle and it sounds like y’all… So you’re providing recruitment materials, and talking points. Do you get to the point where you’re actively, I’m not sure what the word would be, soliciting maybe, field examiners in a certain geographic region, any number of other demographic factors? Does it get that active on y’all’s part?
Jan: It depends on, I’m pleased to say, we have a pretty broad base of examiners who are with us project after project, and we’re always looking to bring on more and we are bringing on.
But, for example, when we have a certain population like we needed a significant number of Asian candidates, and we just weren’t getting them, we do some work with saying, where are the pockets of [00:41:00] Asians in the US? And then we look for examiners or we look through all of our networks at Pearson to try to find, do we have some examiners in that area that would be willing to cultivate those relationships and come on board with us just to get that good population. We cover 80% of the sample pretty easily, but then it’s those last ones that are hard. And then we’re working all of our connections and reaching out to examiners.
One of the reasons we wanted to be on the podcast today Jeremy is to get the word out to more examiners out there. You all can help us. You’re in those communities where individuals are and you can help us make sure your populations are represented in the products.
Dr. Sharp: Yeah, that’s great. It might be nice to transition to, like you said, some of the [00:42:00] benefits to participating. Kristina, you jumped into that and then I backtracked us to something else, but I want to highlight that because I think clinicians, of course, people are like, do I have the time or the energy? What’s the compensation? For better or for worse, people are always like, what’s the benefit to me for doing this outside of the transcendent, moral victory of creating a better test? What does that look like for participants?
Jan: For participants or examiners?
Dr. Sharp: Sorry, for examiners.
Jan: For examiners. As Kristina said, and this is what we hear from examiners, we ask them periodically, why do you do this? And I’ll be honest. I’ll start with the compensation. Not that it’s always the highest, but some examiners say, this is how my family goes on vacation in the summer. I use this money because [00:43:00] it’s this extra benefit. Great. We’re happy to do that or for that to happen.
As Jeremy has alluded to before, we do pay both you as the examiner and the professional. We also pay the candidate. Now, the pay varies depending on the length and complexity of the test and to a certain extent on the difficulty of the sample, right? So there’s all those kinds of things. But if you’re interested and you want to get more information, we can share all of that with you. There’s some ranges and things.
To your point, you’re helping your profession, and maybe that’s too altruistic, but we do have examiners that say that is why I do it because I want to make sure the tests are great, revision after revision after revision.
The ability to influence the assessment, like Kristina said, and that happens much more on the early phases when we’re in [00:44:00] pilot or CogLab or even try out. That’s when we have time to change things if we want. And so if examiner say, none of my children knew what that picture was. That’s great feedback, right? We want to fix that before I get to the market. So being able to contribute.
Getting the training and the connection with our research and development team. Along the way, while they’re giving feedback to examiners, examiners periodically will ask another question.
We do also hold panels for examiners where we would do a call, like a Zoom call and we will say, we’re going to hold a Q&A session for examiners that are working on the WAIS. So, if you have questions about administering the test or about why are we testing that, you can come on and ask that of the developer. Kristina might be on a call and she can explain like, what’s the rationale for that construct? So examiners feel like they get a lot of inside [00:45:00] information that way, and they understand the test a little bit better from that.
Another aside from the compensation, some examiners say, I love administering the test, but I don’t have to write a report and I’m thrilled about that. I just capture the responses and it’s interesting to me to see how my students or my clients do and then I just mail it off to Pearson.
Those are the main benefits that we hear. Kristina, anything else comes to mind?
Dr. Kristina: Yeah, I guess I might add that it can be flexible. If you’re somebody who feels like, I could probably do a few cases here and there, but it’s not going to be a large amount of my time. That’s okay. We need a large number of examiners doing cases. Even small numbers is helpful.[00:46:00] We also have examiners who want to be more involved, and we have more of a partnership with them. We have long-term relationships that we develop, and we sometimes involve them in other opportunities and so forth. So it’s really up to you how involved with us you want to be. I would just highlight that as well.
Jan: And to Kristina this point, you can choose to participate at whatever level and the time commitment that makes sense for your world. So that’s a nice thing as well.
Dr. Sharp: That is cool. I could personally see, that’s interesting to me to be involved early on and see some of those early prototypes and be able to say, yeah, this picture is weird or that instruction doesn’t make sense. I like the super practical aspects of it like that. And I imagine a lot [00:47:00] of other folks do, too.
Lord knows. We have enough complaints about measures in our field. I see these discussions almost daily about why don’t we have this kind of measure. Why is this so long? Or why do I have to repeat this so many times? There’s so many examples of that. I’m just highlighting that to say like, okay, folks, here’s an opportunity to actually jump in and maybe make a difference in some of these things that…
Jan: Jeremy, that just reminds me, we periodically, as Kristina said, especially in early phases of development, that gives us an opportunity to try things. I think this is a classic example that we’ve done a couple of times where we might want to test something at a lower age than we’ve ever tested it before.
We want to see if the 2-year-olds can do this. And we’re so certain because the market has been asking for that. They want that sub-test to go down to 2 years old and [00:48:00] we will try it. And then examiners who are involved in that will say, they know if it works or not. And so then they feel like, okay, I understand now why that sub-test doesn’t go down to 2 years old because 2-year-olds just can’t do it. And so in that way, to Kristina’s point, examiners and professionals are more a partner with us. And that makes a better test.
Dr. Sharp: This is great. I think it’s important to highlight the benefits and I would guess that people do it for a variety of reasons. It’s got me thinking to a D. It’s so fascinating to talk with two people who have been field researchers for years. I’m so curious what that’s like from a practitioner’s perspective having not worked with anyone.
Jan: Do that as a podcast.
Dr. Sharp: Well, that’s what I’m thinking. That’d be a great topic to explore. So if there’s anybody out there who’s been doing field [00:49:00] research for a number of years through a number of projects, get in touch with me. I’d love to chat with you and get the other side of things.
Jan: For sure.
Dr. Sharp: Let’s see. I’m trying to think what else might be important about this process for folks. Are there any major differences in administering as a field researcher versus just a practitioner that we might want to highlight?
Jan: Yes, very important question. Kristina, please take that one.
Dr. Kristina: Yes, this is an important one. One of the things I would say right up front is that when we’re testing for research, the stakes are lower for the people involved. So the examiner is not going to be using the data collected to make a clinical or educational decision for the examinee. So, for that reason, that’s why we really [00:50:00] emphasize this commitment to quality, because we still need that administration to be of the highest quality. We need that examiner to establish a good rapport with the examinee, encourage them, and motivate them to try their best.
Things that we’re all trained to do as practitioners, we still need that to happen in that session or we’re not going to get quality data. The norms are not going to reflect people’s best ability. So, even though the stakes are lower, we still need strong administrations. So that’s one of the big differences I would say is it’s just the stakes that are at hand.
And then getting down more into the test itself. We mentioned a little bit before that the research additions tend to be longer sometimes because we have new items that we’re trying out that may not make it into the final edition. So we need to have some extra ones in there. The administration rules are a [00:51:00] little bit more lenient. So we have longer discontinue rules and we’ll tighten those up before final, but sometimes that can be a little frustrating, right? We hear from our examiners like, it’s taking me forever. I’m not discontinuing. It’s going on forever.
So there are reasons for that. We need that extra data so that we can reorder items and tighten up those rules later. So that’s something that you’ll find in the research editions.
We might have more requirements for recording like recording verbatim, tell us if you gave that query or not. All of those nitty gritty details are super important when we’re doing research. And so we need people that have that eye for detail and very meticulous in the recording department. That’s super helpful.
Dr. Sharp: Could I jump in and ask a related question to that? It reminds me of consent for clients or for the participants. Do you provide the field researchers with consent [00:52:00] forms for the participants or are we responsible for those? How does that work?
Jan: Absolutely. I’m glad you mentioned that Jeremy. If a candidate is selected for a particular test, we send them a consent form that Pearson legal team has developed with us and has all of the language that protects us protects you that we have the authority to use the data for research purposes and so on. So, absolutely. We do not use any test if we do not have a consent form. Absolutely. For sure.
The other thing as you mentioned that Jeremy is we take the security of PII extremely seriously. Pearson has many policies and procedures that we follow and they train us all the time. Our systems are checked all the [00:53:00] time to make sure that any PII data is secure.
And quite frankly, we’re finding that, especially with older populations of candidates, we get questioned about that. How are you using my data? I’m nervous about telling you this information because of all the scams that are around. And so, we do have to provide information that gives people a comfort level that Pearson takes the security of their data seriously. I am glad you mentioned.
Dr. Sharp: Sure. I think I cut in when we were talking about the differences between practitioner administration, and research administration. Were there other points that we wanted to highlight there?
Jan: The test won’t look as pretty in earlier phases. In a pilot phase or maybe even an Assessment Lab or CogLab, [00:54:00] everything may be black and white. It might come in a 3-ring binder because it’s early phases. We don’t do that final design and mock-up until it releases to the market. And so, sometimes people are a little surprised by that, but that’s just internal efficiency. There’s no need to do that until the final. So it does look different. It is longer. They are longer. Research versions are always longer.
Kristina, any thoughts with that?
Dr. Kristina: I think those were probably the main ones. Not really.
Dr. Sharp: Great. Well, I know we’ve covered a lot of ground and this is so valuable. I feel like I have a nice working understanding of what this might be like from the practitioner side and some super concrete steps of, I want them to get involved. Are there other things, other aspects about this process that we might want to highlight before we wrap up or have we hit the high points [00:55:00] here?
Jan: I think we’ve hit the high points. I would just say, if you’ve not been involved in this kind of work and you’re interested, even if you’re not willing to commit, if you’re interested, I would encourage you to contact us through one of the emails or websites that Jeremy will provide. Talk to us. You might jump in and it works beautifully for you. I would encourage you to do that. There’s not a commitment. We’d love to work with you. We have hundreds of examiners. Many of them stay with us year after year after year. And so we do look at it as a partnership and we’re always looking to make our process better. So we look at that as a supportive, collaborative working environment. So come and join us. We’d love to have you.
Dr. Sharp: I love that. It’s a very welcoming invite. I will [00:56:00] include all of those links and ways to explore this process in the show notes so that folks can just go there and click on those links and see what that might look like.
I really appreciate y’all’s time. This is a fun conversation. I’m not just saying this, it’s really got me thinking about what this might look like, and it seems like it could be fun. Thanks for being here. It’s so great to connect with both of you and have you share your knowledge.
Jan: Well, thank you so much.
Dr. Kristina: Yes. Thanks for having us, Jeremy. I’m glad we piqued your interest. So our work here is done.
Dr. Sharp: You absolutely did. Yes. Nice job. Thanks y’all.
Dr. Kristina: Thank you.
Dr. Sharp: All right, y’all. Thank you so much for tuning into this episode. Always grateful to have you here. I hope that you take away some information that you can implement in your practice and in your life. Any resources that we mentioned during the episode will be listed in the show notes. [00:57:00] So make sure to check those out.
If you like what you hear on the podcast, I would be so grateful if you left a review on iTunes or Spotify or wherever you listen to your podcasts.
And if you’re a practice owner or aspiring practice owner, I’d invite you to check out The Testing Psychologist mastermind groups. I have mastermind groups at every stage of practice development, beginner, intermediate, and advanced. We have homework, we have accountability, we have support, we have resources. These groups are amazing. We do a lot of work and a lot of connecting. If that sounds interesting to you, you can check out the details at thetestingpsychologist.com/consulting. You can sign up for a pre-group phone call and we will chat and figure out if a group could be a good fit for you. Thanks so much.
The information contained in this podcast and on The Testing Psychologist website are intended for informational and educational purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric or medical advice, diagnosis, or treatment.
Please note that no doctor-patient relationship is formed here, and similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with an expertise that [00:59:00] fits your needs.