Hey everyone. Welcome back. Glad to be here with you. And very fortunate to have two return guests today. I’ve got Dr. Dustin Wahlstrom, who’s the Vice President of Technology Strategy at Pearson, where he leads the development teams for Q-global and Q-interactive. He has a Ph.D. in Clinical Psychology from the University of Minnesota and has held various roles within Pearson such as Product Owner for Q-interactive and a Research Director for the Wechsler Intelligence Scales.
I also have Kristen Getz. She is the Product Owner of Q-interactive. Her role is to define customer needs and business priorities to the Q-interactive [00:01:00] development team. She’s been with Pearson for 10 years, first as a Research Director and then as a Product Owner. Prior to joining Pearson, Kristen worked as a speech-language pathologist primarily in educational settings.
So Kristen and Dustin are here to talk with me about the major update to Q-interactive that is coming very soon. We talk about lots of aspects of the Q-interactive update and plenty of questions and details that many of you likely want to know if you are using Q-interactive and plan to update to the new app. We also talk about the evolution of digital assessment, both specific to Q-interactive and to the digital assessment field in general. And then we close with some ideas around the future of digital assessment- trends that we [00:02:00] are seeing. We talk about ChatGPT, of course, and many other things. So stay tuned and learn a little bit about digital assessment and Q-interactive.
Before we totally transition to the episode, you may have heard about Crafted Practice: A Retreat for Psychologists happening this summer, August 9th-13th. This is a small group retreat where we’ll be talking about the business and clinical aspects of running your practice. It’s here in Colorado. It’s going to be beautiful and it’s going to be an amazing experience. You can get more information at thetestingpsychologist.com/retreat and sign up for one of those spots. If you have questions, you can also schedule a call to talk with me about the retreat and whether it’s a good fit. I hope to see you there.
All right. Let’s get to my conversation with Dustin and [00:03:00] Kristen about Q-interactive and digital assessment.
Hey, Dustin, Kristen, welcome back to The Testing Psychologist podcast.
Dr. Dustin: Thanks for having us again.
Kristen: Thank you. It’s great to be here.
Dr. Sharp: Yeah. It’s good to see y’all again. I think back. I think we were talking before we started to record about the last time and back then it was Dustin and James but this is maybe, I don’t know, 6 or 7 years when we did the original Q-interactive podcast. So I’m excited to be back here with you. I know there’s tons of updates and lots that we get to chat about today. so, thanks again for being here.
Dr. Dustin: It’s gone fast. [00:04:00] I thought it was like a year and a half ago. I was shocked when you said six.
Dr. Sharp: Yeah. You’re telling me. It’s wild. That’s a theme of my life right now but we’re not going to get into my midlife crisis on this podcast. I would love to hear…Before we totally dive into Q-interactive updates and just talk about digital assessment, I always ask folks why spend time on this. Why is this particular topic or set of ideas important to y’all? Of all the things that you could do in our field, why spend time on this? Dustin, do you want to go first?
Dr. Dustin: Sure I can start. I’m a clinical psychologist by training. I came to Pearson straight out of graduate school and started within our research and development group. And then eventually was brought over to help with some of these new technology projects that we had like Q-interactive.
For me, as a psychologist really [00:05:00] believe in the value that assessment can provide to our field and the people that we serve. I think technology just offers so many ways to really enhance the effectiveness of the assessment. So for me, it’s a really exciting time and it’s a great way to have an impact on the field in a way that I never would’ve thought I had. When I was back in graduate school, I would’ve never thought I’d be sitting here with you talking about technology and assessment but here we are.
Dr. Sharp: Here we are.
Dr. Dustin: Yeah.
Dr. Sharp: Did you have any background in technology or software development or anything like that? Would you have considered yourself a techie person before getting into all of this?
Dr. Dustin: Not necessarily. I liked technology. I didn’t have an engineering background. In graduate school, I worked in a lab that did a lot of like [00:06:00] neuroscience or neuropsychology type testing, and all of the measures that we use, well, not all the measures, the majority of the measures that we used were digital. So when we started talking about doing some of that with our classic measures like the Wechsler, I found it really exciting. So I had a natural interest but not necessarily any specific background.
Dr. Sharp: Sure. Kristen, what about you? Why spend your time on this kind of stuff?
Kristen: Well, like Dustin, my background is that I’m a speech-language pathologist; so again, I want to do what’s best for my field and for others that are assessing individuals. I just see the benefits as far as efficiency, accessibility, data, and all of those things. It’s just real positive.
Dr. Sharp: Absolutely. Yeah, there’s so much potential which we’re going to talk about. That might be a nice transition just to get into some [00:07:00] of the reflections on digital assessment from the past, I don’t know, 10 years or so. I think there have been a lot of companies working on digital assessment but y’all changed the game, so to speak, with Q-interactive and you’ve been in this now for about 10 years, right? So I’m curious, think back to when you started with Q-interactive to where we are now, what are some of the biggest changes that you’ve seen just over the past 10 years or so in digital assessment?
Dr. Dustin: You mentioned the last 10 years, this is good timing, this podcast because this is the 10-year anniversary of Q-interactive. We launched it in 2013. So it’s been 10 years exactly. I think there’s been a lot of things that have changed. One of the biggest ones [00:08:00] and Kristen, maybe you can speak about this, is just how differently our customers and the field at large feel about digital testing. When we first started, there was a fair amount of hesitance and a lot of questions and things that people wanted Q-interactive to do and not do. I think a lot of those attitudes over the past 10 years, technology, in general, has just become more ubiquitous, have changed.
Kristen: Yeah. I would say it was a skepticism around digital testing that, like Dustin said, originally people said, I still want to be 100% in charge like I am with my paper administrations. And then what we’ve seen over the last 10 years is people are more and more [00:09:00] accepting. They see the benefit so they’re more about what else can you do for me? What else can digital test development provide?
Dr. Dustin: You see that in Q-interactive, so there are parts of Q-interactive that feel very manual and that’s the reason why. What I mean by that is an examinee may touch something on the screen or something like that but then the psychologist is still responsible for scoring it, even though it’s obvious that the computer should be able to score it based on what the examinee touched. That was intentional on our part based on early feedback where people were saying no, I don’t want the iPads to be doing that for me. I need to be in control.
I think as soon as people started using it, almost immediately what they felt was a certain level of inefficiency and like, well, why am I having to take this extra step and touch this extra button? So in part, I think just their experience with the app over time changed their perceptions of what they [00:10:00] wanted it to do. I think in general, as technology does more and more for us, people get more comfortable with the notion that they don’t always have to be in control of every single detail like that.
Dr. Sharp: Yeah. I’m curious if y’all have seen, just in talking with folks and doing this for the past several years, have you been able to isolate anything or identify anything that helped people adopt this new technology- things that they’ve said or you’ve seen? Because it’s a big shift and I really think of it from a behavior change standpoint. It’s hard to change people’s behavior and this is a major shift in testing behavior. I’m curious if you have any thoughts about what helped that happen to move from skepticism to acceptance and now I guess, desire for even more.
Dr. Dustin: I’ll jump [00:11:00] in, Kristen, and then if you have anything else to add, go ahead. I think part of is just peer influence. I think when you have something brand new like Q-interactive, there’s a level of skepticism and then there’s a level of people not necessarily being the first ones out of the gate to try it and share in some of the growing pains that come along with brand new technology like that.
And so, I think as people started to use it and then as others started to see that it was effective for them and that they were able to do some of the things that they were always able to do and do some of that even better, then over time people start to get more comfortable with it just because they’re seeing their peers and people that they trust use it and use it effectively. I don’t think there’s anything we necessarily did on the Pearson side other than make the software work and make the software reliable, right? That’s the most important thing we really could have done from our end.
[00:12:00] Kristen: I think people very quickly see the benefits once they started using it. For example, when we offer trial subscriptions, some people sign up and start to use it. They really see, okay, what you’re telling me regarding efficiency and time savings and not having to carry kits around, all of those things become immediately apparent. So the adoption just happened naturally for people. And I agree with Dustin, the trust factor, when you see the numbers, how many people are administering Q-interactive now, 10 years later.Dr. Sharp: Right. Think about it, I don’t know if y’all did this deliberately or if it was just a really nice coincidence to just the parallel of the popularity of tablets in general, the iPad and Q-interactive were pretty close in terms of when they came out. [00:13:00] I wonder if that’s driven some of these. People have just gotten more comfortable with tablets in general. Kids especially are using tablets a lot more and that’s a big draw for a lot of the pediatric folks out there. So just being in the right place at the right time with the technology too, all came together.
Dr. Dustin: Yeah, for us, tablets are the piece of technology that enabled Q-interactive to be. We were looking at things like laptops and things like that, they really didn’t allow you to replicate that one-on-one experience that we wanted to preserve. Having a screen up in front of you gets in the way and seems to create a disconnect between you and the examinee. So having these tablets allowed us to have something that was the right size to show stimuli and allowed us to [00:14:00] create that nice flow between clinician and examinee. So it was a necessary invention.
Dr. Sharp: Sure. It’s funny, I was thinking about the, I guess, I was just feeling grateful for this technology. Over the past few months, we’ve gotten this contract to evaluate incarcerated individuals. In one of the jails that we go to, it’s that very classic, there’s the glass barrier between you and the individual and there’s the slot that you pass things back and forth. I was so grateful to be able to just pass the iPad through the slot and do what we needed to do. Paper assessment would’ve been really challenging in that setting. It’s nice to have an option for that.
Dr. Dustin: That’s a great anecdote. I had never heard of anyone doing that before. It’s been interesting going [00:15:00] through this whole experience. We’ve heard of amazing use cases from people that we never would’ve thought about, that technology really helps address.
The most interesting one was someone really early on who told us that they do remote testing in Alaska. So they get on these tiny little prop planes and then fly out to all of these remote locations and they’ve got a weight limit on these planes. The woman was basically saying to me like it’s, this is going to sound a little dramatic, but it’s what she said. She was it like, before, if I had to pack a WISC and a WIAT and a CELF and whatever it might be. That’s a lot of weight. And she was like, now I just have two iPads and my set of blocks or whatever it is. She was like, I have more room to pack survival gear in case the plane goes down. I was like, wow, I never really thought of that one.
Dr. Sharp: Right. Q-interactive saving lives.
Dr. Dustin: That’s right. Yeah.
Dr. Sharp: It’s really cool. That makes me think about [00:16:00] COVID too. So obviously we had this time over the past three years or so where digital assessment and remote assessment took a big turn and came right into the spotlight for a lot of us, right? I’m curious how that impacted things for y’all and how you see that maybe influencing digital assessment moving forward. I’d love to hear any thoughts just on the impact of COVID.
Dr. Dustin: Kristen, you want to take that one?
Kristen: Oh, sure. Obviously, we had to pivot immediately to meet our customers where their needs were, and obviously, testing didn’t stop. One of the very first things we did was make the digital assets available on Q-global because we had a lot of people that needed the STEM books and everything. We made those available to our customers with Q-interactive.
Q-interactive was a bonus for us in [00:17:00] that you can administer Q- interactive with the examiner and the examinee iPad at a distance. So people that were still testing in clean rooms were still with the individual, semi–close to them. They could sit at a distance and we offered the picture and picture so they didn’t have to be looking very closely at the examining screen.
Q-interactive, like you were talking about the barrier in the prison when people were administering through plexiglass and things like that, Q-interactive works just fine.
Another benefit to it during COVID was it was easily cleanable. You can’t wipe down a STEM book without damaging it and those sorts of things, but with the iPads, you could sanitize them between clients. It just became a very useful medium during that time period.
Dr. Sharp: I totally agree. They’ve really become [00:18:00] an integral part of our workflow. This is a separate story and we don’t have to spend time on this, but for reasons beyond Pearson’s control, this is all Apple, we are locked out of our 20 iPads that we use for testing. We have had to go back to paper administration over the last two weeks and it’s been a nightmare, to be honest. We realize how much we’re missing when we can’t use iPads in our workflow.
Kristen: I would say, that’s something that we hear frequently, especially people who have access to Q-interactive at wherever they’re working and they switch, they go somewhere else, and all of a sudden that facility doesn’t use Q-interactive and they have to go back to paper. They’ll come up to us in a booth or wherever they see us and say, oh my goodness, how much I miss Q-interactive once I go back to paper
Dr. Sharp: Sure. Well, I know, oh yeah, go ahead, Dustin.
Dr. Dustin: Well, I was just going to say, like a lot of those [00:19:00] COVID protocols, we’ve gone back to normal in a lot of ways and in-person testing and all of that stuff, but I think just like we’ve probably all seen in other facets of our life, there are certain aspects of what changed during that time that I think are here to stay because people find them convenient or whatever it is. Remote testing, for example. I don’t think we’re probably seeing like the level of remote testing that we would’ve in the summer or fall of 2020 but there’s certainly more of it now than there was in 2019 because I think it serves a purpose even outside of COVID.
So when you think about the ability to mail questionnaire-type behaviors on Q-global or share simple tests and administer those remotely, I think all of that’s here to here to stay and you couldn’t really do that without technology. [00:20:00] I think technology, just in general, allows you to react more quickly and pivot to these types of needs. Kristen described how we were able to put all these manuals and stimulus books up very quickly online for people. In traditional publishing models, you can’t really move that fast but technology allows you to meet customer needs really quickly.
I remember right away, that first week after stuff started to get shut down, people were stuck at home and just needed manuals because they needed to look up a score. They weren’t even trying to do testing, they just had a very practical concern of just like, I can’t go to my school to go grab that paper manual and look up the Full Scale IQ, what do I do? We were able to help very quickly in those scenarios because of technology.
Dr. Sharp: Yeah. It allows for a lot of flexibility and access. I think, at least for our practice, it’s opened the door to alternative [00:21:00] models of testing. We’re much more willing to say, well, maybe just travel. Maybe that’s the easiest way to put it, like travel outside the office to do testing because the materials are more portable and we can be more flexible. It’s great.
Dr. Dustin: You mentioned access; one of the big changes over the past 10 years has been around regulation and compliance, so things like security, which is an obvious one, I think everyone’s much more attuned to security than they were 10 years ago. There’s a lot more in place now, things like FedRAMP, which is the federal government’s standardized approach to assessing cloud vendors and their security and risk. Things like GDPR in Europe. The other big one is accessibility which is Kristen our resident expert in. That’s a huge change. Something that gets a lot of attention now and really didn’t probably in 2013 when we first launched Q-interactive.
[00:22:00] Kristen: Yeah. More and more customers are asking for proof- for documentation of ADA compliance in order to adopt our platforms. Basically, ADA compliance now is saying that all websites should be accessible and inclusive to everyone. So obviously, 10 years ago, those same compliance standards for accessibility weren’t in place and obviously when we’re going to be talking about this a little bit more, but our legacy application wasn’t necessarily 100% compliant. That wasn’t intentional. It’s just it required refurbishing to meet those standards and it’s difficult to do that on an older legacy platform but when we talk more about it, our new platform was built from the ground up with accessibility in mind.Dr. Dustin: And that’s a compliance issue but also a huge opportunity for us to, again, leverage technology to really make our tests more accessible and useful for people of various special [00:23:00] needs.
Dr. Sharp: Really important. Yes, absolutely.
Well, before we totally transition to the massive update coming to Q-interactive, I’m curious, are there any other reflections, surprises, anything from the last 10 years or so that is was worth talking about in the digital assessment world?
Dr. Dustin: One of the things that we were talking about before we got on the recording here was just, I think we met shortly after Q-interactive came out in 2013. I was saying, had you and I sat down for a podcast in 2013 and had vision casted what we thought 2023 would look like; I think both of us would’ve said it would be much different than it actually is. [00:24:00] And different in a way where we would’ve said, and I don’t even know, in some ways technology would’ve moved perhaps further than it actually has.
For example, I may have told you, we won’t even be on iPads anymore in 2023. Like, who knows what’s going to be out there, right? We’re going to have to move to this brand new thing that none of us even know exists yet but here we are, we’re still on iPads and they’re still doing the job just fine. So that might be a surprise as we think back about the past 10 years and what’s transpired.
Dr. Sharp: Sure. Yeah, it seems like technology moves very quickly and very slowly at the same time, it like these leaps, and then it’s just incremental iterations and that’s totally okay. But yeah, I use the flying car analogy, right? Like by 2023 we’ll have flying cars and testing but it’s…
Dr. Dustin: We may be in the middle of one of those leaps with AI right now, which we [00:25:00] can talk about later if you want. That may be a perfect example of what you’re talking about.
Dr. Sharp: Sure. I’m going to pin that for the reflections on the future. I do want to talk about this new Q-interactive update because this is a big deal. I wonder where we might start with that. There’s so many questions with that. Maybe could you just talk a little bit about the rationale and why this massive update, what it’s going to look like? Let’s start with some general information and go from there.
Kristen: You want to take that one, Dustin?
Dr. Dustin: No, go ahead, Kristen. This is all you; you’re living and breathing it every second of the day.
Kristen: Well, it’s a massive update. Obviously, the legacy application is 10 years old and it was heavily customized and maintenance heavy and prohibitive to some of the things that we wanted to do next. So [00:26:00] we went back to our development team and said, what do we need to do? What does the next version look like? Is it something that we can just update what we currently have or do we need to start with something from the ground up? The decision was made. We need to start from the ground up, migrate all of our content over and just address a lot of the things over the last 10 years for customers that were pain points.
As we said earlier, our customers have moved with us and they’re asking for more things, and we have 10 years’ worth of feedback from them. We just knew that we needed to make the move to provide better performance, improved customer service or improved customer experience, I should say, better workflow and that sort of thing.
Dr. Sharp: So we talked about this and I asked this question, why start from the ground up with a brand new app versus just updating the existing one, right? It seems like [00:27:00] obviously the pros outweigh the cons, but I imagine that was quite a decision to make. Could you just share a little bit of information of why you made that choice to start from the ground up with this new one?
Kristen: Dustin, want to take that one?
Dr. Dustin: We’re both a little bit over our skis here since we’re not engineers. The upshot is the change is so big. It required a brand new app because it’s not really incremental. In general, apps like this get built on technology frameworks that then allow the app to not just be iOS but used in Android and other places like that. The original framework that we used for Q-interactive is something called PhoneGap that I don’t think has been supported for many years, Kristen.
Kristen: 2014.
Dr. Dustin: 2014. So what that means is that every single time something breaks that’s related to PhoneGap, we would have to write something custom to [00:28:00] fix that. You can imagine that’s very labor intensive. So we had to go from PhoneGap to something that’s brand new called Ionic. The change was so significant in terms of how the framework interacts with the iPad itself if you think about how we store data down to the iPad while you have data on the app, things like that simply doing an app upgrade, it really wasn’t tenable. It just required us to start all the way over.
Dr. Sharp: Yeah. It makes me think, again, about the parallel growth of this technology with tablets and the App Store in general. I’m also not a developer and I’m going to say something dumb in the next 30 seconds I’m sure, but just the coding languages and how you even [00:29:00] develop apps for the App Store has massively changed over the past 10 years. Being on the front end of that, it seems inevitable that you would have to rewrite the app at some point to match the updated technology and the new architecture and coding that’s available, right?
Dr. Dustin: Yeah, that’s right. We spend a decent amount of time all the time rewriting things, updating software, libraries, all of that stuff. It’s a never-ending thing trying to stay on top of all of that technology. But you have to do it for, Kristen mentioned performance reasons. You have to do it for security reasons and all of that stuff.
Dr. Sharp: Sure. What are some of the big changes that are coming with this new app?
Kristen: So there’s lots, probably too many for us to talk about in one call. I’ve mentioned the improved workflow flow in the brand new look and feel. When customers sign in, they’re going to notice right away that this is a different app for sure.[00:30:00] The performance updates, accessibility upgrades, all of those things. We’ve made so many changes within the app. Some of them big and some of them just like little Easter eggs that are throughout, I would say.
Some of the features that I would mention just off the top of my head would be changes to the dashboard when customers sign in. The entire dashboard has been reimagined and that was based on, again, customer feedback. The old dashboard, people got used to using it, but nobody really thought that it met their needs as far as workflow. We went through several iterations of calls and beta testing and things like that with customers. I think we actually came to you, Jeremy, at one point and asked you to try some things out for us and reimagine the dashboard so that it actually met people’s needs where they can order things, [00:31:00] they can search for things so that it’s very easy to find what they’re looking for and organize their day and how their testing schedule looks like.
We’ve made massive updates to the assessment details page, so it’s more user-friendly. Some of those little changes, like I mentioned, was adding grade onto the assessment details page. We got lots of complaints from customers who said, I have a student that comes to me, I think they’re in 6th grade. The minute they sit down, I start talking about their day. All of a sudden they say they’re in 5th grade and then at that point they’re stuck because they’ve already piped down this assessment that’s got the incorrect grade to it. So we’re giving you the ability to change the grade- make more of those edits within the assessment details page.
The results pages have been totally reimagined so that they’re very clean and they can be used to, actually as a visual to interpret results. If you want to actually show something to a [00:32:00] parent or to another colleague or whoever you need to, you have new graphs and all sorts of things. I could go on and on and I’m happy to talk about this. There’s things with the don’t know and no response buttons. We have dynamic corrective feedback. We try to eliminate everything.
One of our biggest complaints was so many swipes to get through content. We’re trying to make it as clean and concise as possible to keep all content on one page. So lots of changes. And so far as we’ve showed it to customers and we’ve done several customer forums and webinars and spoke to people at conferences, all of the feedback that we’ve gotten has been overwhelmingly positive.
Dr. Sharp: That’s great. That sounds like a massive update.
Kristen: Yeah. I think it’s the sort of thing and we wanted to make the changes so that our current [00:33:00] customers didn’t sign in all of a sudden say, okay, I’m used to using the current version of Assess and now you’ve changed it so much that I don’t know what I’m doing. So we feel like that’s not going to happen. Everybody who’s a current user of Assess when they sign in, it’s going to be intuitive. It’s going to even be, we hope, more intuitive for people. You’re not going to need to go to a training to relearn how to administer assessments on the new application. We think that you should come and view one of our webinars or watch our training videos that we’re going to send out just so that you know what to expect but it’s not going to require a huge learning curve at all.
Dr. Sharp: That’s good to hear. Going back to the behavior change thing, people have trouble, things change, right? So are there any major changes that people will miss that as far as you can tell or anything that [00:34:00] might be more difficult for folks if you had to guess?
Kristen: I don’t think so. I think we’ve changed a few of the interactions. A lot of those were due to accessibility reasons. So some of the ways in which, let’s say, you would edit the battery instead of dragging and dropping. So it’s a tap. I think that it may be a second where somebody says, wait a minute, I used to drag and drop this and now I only need to tap it, but I can’t think of anything huge that somebody would miss. We definitely did not remove anything from the app. We didn’t take any functionality away. We just enhanced the functionality that was there. For example,…
Dr. Dustin: The biggest…
Kristen: Go ahead.
Dr. Dustin: Sorry, I was going to say, maybe the biggest hangup for people, Kristen, might just be the actual act of transitioning from the old app to the new app and what that actually looks like. Because it’s a new app, what’s going to happen is on [00:35:00] the day we release the new version of Assess, like nothing’s going to happen to your old app if you didn’t do anything that day. If you don’t go into the App Store, you won’t even notice a change. So people are going to have to go into the App Store, download that new app. And then for a time period probably they’re going to have both of the apps on their iPad at the same time. We’re doing that intentionally because we want to give people a period to transition practice with a new app. I don’t think we’re advocating that you go in and test someone without practicing first even though we think it’s going to be a pretty natural transition.
So there will be a period where you have both apps on the iPad at once. What we want people to do is go into that old app and start deleting off all of their client data that’s on that. So remove it, sync it back to the website so that you’ve done everything there. And then once it’s all empty, you can delete that one and permanently move over to the new one. In that interim period, you could see, and we’re going to do a lot of communication about this to people, [00:36:00] but you could see there being maybe some confusion with people.
So if you create Susie and hit send to iPad and send Susie to the iPad when you’re on the website, what’s going to happen is the first app that you log into after that is going to be where Susie goes. So you could see people wanting Susie to go to the new app but accidentally logging into the other old one first and then it goes to the wrong place. So people are just going to have to be cognizant of that during that transition period. I think that’s probably the most likely place people get tripped up.
Dr. Sharp: Sure. So that’s an important thing to highlight. That process of generating a battery on the website and sending it to the iPad is going to stay in place.
Kristen: No changes at all. Your data between the old app, the legacy app and the new app, they’re all syncing back to that same central website and they’re all on the same repository. So if you need to go back and view results, [00:37:00] same process.
Dr. Sharp: Great. Are you overhauling the website itself or just the app?
Kristen: Just the application. The website, we made a few changes that were necessary just because of the application, but they were very minor. So for right now, the central website’s going to remain the same.
Dr. Sharp: Sure. Okay. This may not be a question you can answer necessarily, was there any consideration of being able to just do everything on the iPad versus separating the website and the iPad, that whole process?
Dr. Dustin: Kristen.
Kristen: Yes.
Dr. Sharp: You’re like, who’s going to take this?
Dr. Dustin: Who’s going to take that one?
Kristen: We do get frequent requests for people to actually create clients on the iPad and that is one of the features and updates that we have in our backlog that we are still considering. So more to come perhaps.
Dr. Sharp: Okay. [00:38:00] I’ll take it. Let’s see. So the app is going to be released pretty soon, especially, I’m thinking ahead to when this episode will be airing. Can you nail down the release date for folks?
Kristen: Our release date right now is mid-April. It’s April 13th. As long as everything goes as planned as and as expected, then people should be able to download the app on that day. The new one.
Dr. Sharp: All right. I like how you highlighted that this is not a change that people are going to have to make right off the bat. You can keep the old app if you want and then you make the transition slowly. Is it just basic information like, are the same tests going to be available? Is the login going to be the same? Just questions like that. Very logistical, practical questions.
Kristen: Yes. So your login is going to remain the same, so whatever login you’re [00:39:00] using for your current application, it’s the same login for the new application. So nothing changing there. As far as will the same suite of tests be available? Yes, in most cases, you’ll still have access to all of the tests in your current licenses. The one change that we did make is there are a few of our older tests that not everyone still has access to but we had a few people that had requested to keep some of our older tests. A lot of them for research reasons, WISC-IV, PPVT-4, CVLT®-II, WIAT-III, all of these are older tests that have had revisions done and the revisions are all available.
Some of those older tasks will no longer be available when we sunset the old legacy application. So the sunset of the legacy application is going to be December 15th. So we’re giving everyone a fairly large window to make the transition between April and December. So those older tests that will only remain on the legacy application, so when that goes away [00:40:00] they will no longer be available.
On the flip side, there’s going to be some new tests that will be available only in the new application. So moving forward, all of our development effort will go to the new Assess app. So when Assess launches in April, customers who have standard license and an SLP license will have access to the Bracken test. So ABCS and BSRA, we’ll both be available on the new Q-interactive Assess app. Then lots more are coming down the pipeline now that we’ve lift our barriers with the delays from COVID and we’re finishing standardization on some huge flagship products. Lots of new tests will be released on the new Assess application.
Dr. Sharp: That’s great to hear. I know that COVID was a huge disruption for data gathering and [00:41:00] standardization and development of some of these measures. So that’s exciting to hear that they’re back on track.
So what about other… I’m sure people are wondering if any of the data is going to get lost or like what if you’re in the middle of an administration when the new app comes out, can you finish it? Little things like that. Can y’all address those two questions as well?
Kristen: Yeah, no changes to data. Obviously our secure storage is going to remain. We recommend, Dustin touched on this, that you really want to finish out your assessment on your old application before starting anything new on the new application. We think the cleanest way to do this is for you to finish all your end progress administrations, sync them back to central, delete off the old app, so there’s no confusion. And then [00:42:00] start with a new application.
Like Dustin said, you can have both on your iPad at the same time. And we’ve come up with use cases why people may want to do that. So you just have to be sure, like Dustin said, that you know when you open up and you sync your assessment information down, that you’ve opened the correct application. And they do look different. They’re both named Assess. There will be additional verbiage in the app store to let you know that what is the legacy app and what is the new app. And you can see the difference. The new icon is a teal light blue but you really need to be sure that you’re opening the correct app so you can get your data synced to the right place.
Dr. Sharp: Sure.
Dr. Dustin: The nice thing about having it be a brand new app is that there is no impact to the current app. So whether you got an assessment in flight on that app or whatever it is, you are downloading the new app and using the new application is [00:43:00] totally independent of that. So there really is no worry about data loss or what to do about assessments that are halfway done or anything like that.
Dr. Sharp: That’s great. I have to imagine that y’all are putting together all sorts of training material on how to do this. So what’s that look like for folks who want to access information training materials? Tell me about them.
Kristen: Maybe several different webinars that are being offered right now. Listeners can find that information on our website. We’ve completed four. We still have two more that are planned but we may be doing more after that as well. If customers are interested, they can register for the two that we’re doing in April. And then we also have lots of different training materials being produced.
We have an hour long whole workflow video that shows the new application and just [00:44:00] how to use Q-interactive in general. That is going to be sent out to all Q-interactive users in an email once we launch the app. So once you get your launch email, there’ll be a link in there to a video. You can click on that and it’s in modules. You don’t have to watch it all at once. You can go back and forth. It’ll be, we think, a good tool for customers.
We’ll be also be updating a lot of our support tab videos that sit in the Q-interactive website. Those will be available shortly after our launch but customers can go there as well and look for additional information. And then lot of information on our website. We’ve produced frequent an FAQ document. We have transition documents. We’re trying to make sure we have everything in place to make this change because we know it’s a big one, and as smooth as possible.
Dr. Sharp: Yeah, I can only imagine the work that goes into a project like this and making sure [00:45:00] things are in place for folks. I’m sure there will be questions that maybe you haven’t thought of but I know you’re doing your best. So plenty of information to access for folks who are curious.
I would love to shift to talking about more the future and looking ahead with digital assessment. I know y’all have been working hard on getting this launched and a big part of it is having an eye to the future and what this is going to give us the capability to do on the digital side. So maybe we just start, what are you looking toward in the future with digital assessment in terms of, I don’t know, maybe we call it like trends or things, directions you see the field heading that are worth tracking?
Dr. Dustin: We should definitely come back to this AI issue in one second but I think [00:46:00] one thing we should touch on right away, which is just practical thing is, Kristen mentioned that the new Assess app and the framework that it’s built on provides us a lot of enhancements around performance and the speed at which we can develop tests and things like that. One of the implications of that is it really allows us to take the next step in terms of making digitally native tests that have a lot more functionality that you may expect to see through digital games and stuff like that.
So one of the products that’ll be coming up, and I don’t think it’s any big secret we could talk about this, Kristen, is just the D-KEFS. So the next version of the D-KEFS will be all digital. It had to be built on the new version of [00:47:00] Assess. It would’ve not worked on the first one, I don’t believe. That one’s going to be completely digital, so it’ll be no paper with that D-KEFS at all. There’s a ton of really new and exciting and engaging tasks that measure things in different ways than we’ve ever measured them before. I don’t know if you want to talk about two of them, Kristen.
Kristen: Yeah. I think everything you said is correct. All of the different things, the way that we’re able to remove the manipulatives from that test in in particular, so if you think of something like the Tower test with the eczema discs and all those things, we’ve been able to reimagine that in a completely digital format. And by doing that, then all of a sudden it opens up all of these different ways to collect variables that we weren’t able to do on paper. We can do things with timing and errors and all sorts of things [00:48:00] that we can collect information on. So it’s really exciting for us when once you’re not tied to a paper equivalency, it really opens up what the possibilities are for our completely digital psychological assessment.
Dr. Dustin: And it’s time for us to be doing that. I mentioned how I used mostly digital tests back when I was in graduate school. One of those was the Tower of London, which I think is the cant tab maybe. And surely it didn’t look all that great. It looked like an Atari game from the 80s, but it was all digital. So it’s high time we start doing some of that on Q-interactive. So we’re super excited about that. For us, that’s really the next logical extension of what we’re doing on that platform.
Dr. Sharp: I think that makes sense. That was one of the biggest points of hesitation for folks, I think, in the beginning especially, and there’s still discussion around this was the equivalency between paper and digital [00:49:00] and leapfrogging that obstacle is, I would imagine very helpful as we move forward for a lot of reasons.
Dr. Dustin: Yeah.
Kristen: Our answer a lot of times when people say like, why can’t you do this? And the answer now is, we can do that but we have to do what makes the most sense. One of our biggest, like you said, hurdles is the equivalency to the paper counterpart. So unless we’re going to norms on a completely digital product or a completely paper product and keep those separate, then we’re very mindful of that. It’s something that our research directors are constantly discussing.
Dr. Sharp: Yeah. Are there any other tests that y’all can talk about right now that are being developed as digitally native?
Kristen: We don’t have any 100% digitally native but [00:50:00] we’re are just starting up again on our standardization of WAIS and WMS. So I think you’re going to see some very interesting things. For example, in WMS where we’ve been able to eliminate some of the heavy manipulative subtests, for example, spatial edition and designs with the cards and the boards and things like that. In the new version those are completely digital. Drag and drop and things like that, or tap and drop.
So more updates. Definitely, more good things coming. Even when we still have to be equivalent or do additional data collection, we’re trying to push the envelope as much as possible.
Dr. Dustin: And that’s important from a testing experience perspective, right? I think it’s a better way to take the test, but there’s also just practical benefits of not having to call in and order record forms or wait for them in [00:51:00] the mail or not record forms response booklets or the grids or whatever, trying to find a place to store them. It just simplifies everything for people. So there’s just like a real practical benefit in terms of usability and all of that by making these things all digital.
Dr. Sharp: Yes. Can I go back to the D-KEFS and ask the question that everybody wants to know and that is, when is it going to be released?
Dr. Dustin: Well, can we give a timeframe on that, Kristen, confidently?
Kristen: Well, things can always change but right now it looks like it’ll be towards the end of next year.
Dr. Sharp: The end of next year. So like end of 2024?
Kristen: Yes.
Dr. Sharp: Okay.
Dr. Dustin: We’re in standardization now. Got people out there.
Kristen: That test will require a heavy work from our psychometrics team and our author and our content development group. So we’re trying [00:52:00] to give them a big window to get everything done.
Dr. Sharp: Yeah, that totally makes sense. That’s a nice concrete answer. That’s very helpful. Thinking about the future, let’s talk about AI for a little bit, that’s hot right now and we’ll probably continue to be hot. So I’m curious what y’all are thinking as far as integration of AI with assessment.
Dr. Dustin: Well, probably a lot of the same things. It’s a very hot topic. You’ve seen advancements now. I was just reading something the other day about using AI to detect certain types of cancer super early. And they’re doing this in retrospective studies but you’re starting to see all these areas now in medicine where they’re using AI to be able to identify issues earlier. And I think there’s a direct application to mental health. [00:53:00] I know there’s similar things going on with AI and being able to identify mental health issues earlier, and there’s obviously an opportunity to take test data and derive better insights out of it than we are currently. So I think there’s just a lot of opportunity from just the data per perspective to utilize that technology.
The big one right now is chatGPT, of course. From my perspective, well, we probably shouldn’t get Kristen going on this because the podcast will end up being three hours long. But the thing that’s been so exciting about that is just how big of a productivity booster it is for everyone, right? I would encourage anyone listening to this podcast. If you’re not already signed up and using it, you [00:54:00] should.
I’d like to get your thoughts on this, Jeremy, because my wife’s in private practice. She’s a psychologist. She’s has a therapy practice. I remember when she started it, how much time we spent creating her website and writing all the content for the website. We agonized over it and it was hours and hours work. So the other day, just for fun, I went in there and I said, create me an HTML homepage for a private practitioner in Austin, Texas who does therapy with adults. I put in two areas of specialty and it derived the entire website and it was pretty darn good. I just thought, wow. For psychologists who are out there on their own trying to get all this stuff set up, it’s enabled so much for people to move so much quicker and be so much more self-sufficient when it comes to [00:55:00] running your own practice or managing your own business. Have you guys used it at all?
Dr. Sharp: Yeah. I’m right with you. We use it a lot for marketing right now to fine tune the SEO on our website or the copy on pages that we’re writing; we’re doing a lot of that. I’ve absolutely worked with it as far as making recommendations for certain diagnoses and searching the literature to find research-supported intervention for different age groups and different presenting concerns. I’m really waiting for the time when it can generate images. That would be amazing to be able to graph results and create visual representation of our data. That would be incredible. Gosh, there’s all sorts of things.
Dr. Dustin: The recommendations one was interesting. I thought about this last night as I [00:56:00] was thinking about today’s conversation. So I, I plugged in something around recommendations for a kid with ADHD and tried to put in as much information as I could. It was helpful. Sometimes you think, oh my gosh, this thing’s going to take over the world in the next six months, then other times it’s just so general that you’re like, ah, I still got a ways to go. But the recommendations felt more like the latter to me but it was like this great way to get you started.
I was imagining myself writing a neuropsychogical report as a graduate student even, and just like needing to get started on an intervention section. And then I started to go way deep, like give me like examples of specific classroom interventions and it generated a ton, some I hadn’t thought of before. Certainly you take that stuff and then you have to customize it to the person that you’re talking about but it’s great for sparking creative thinking and just really getting you going.
Dr. Sharp: Yes. [00:57:00] I think that this will be like a lot of things, kind of a love-hate relationship. I’ll be curious to see how many folks actually adopt it, because I would include myself in this. I feel like the work that we do in data synthesis and conceptualization is really the magic of what we do as psychologists. And if there gets to be the technology to do that for us, I wonder if people will be hesitant to adopt it because it’s sort of shooting ourselves in the foot or something or acknowledging that maybe we’re not that special and that’s a hard thing to do.
Dr. Dustin: Well, I think there’s an interesting parallel here between some of what we were talking about in terms of customer perceptions of Q- interactive when we first launched it. Brand new technology, we launched it. People were wary like, well, I don’t want you to do the scoring for [00:58:00] me. I don’t want to have you do the discontinue and stop points for me because that’s what we do as psychologists. And then once they started using it was like, no, nevermind. I don’t want to do this anymore. I’m going to move my value up the value chain and do this stuff over here.
And there may be a similar thing with AI where people are initially wary like, well, are they doing my job? And then like, as it starts to evolve and people adapt, they maybe start to say like, well, no, maybe it can’t do certain things here but then I’m really able to add more value as a psychologist over there. I don’t know.
I put in some test scores together last night to see if it could interpret it. I don’t think we’re in any danger anytime soon of chatGPT doing. I wasn’t super impressed, but maybe GPT5 will change that. Who knows?
Dr. Sharp: We’ll see. I would love, maybe y’all are already thinking about this, I would love to somehow hook you [00:59:00] in and use that pattern recognition to do real-time almost embedded performance assessment. So based on a person’s performance on Q-interactive stuff and their pattern of scores, could we get a notification that says, hey, this looks like suboptimal effort or something like that where it’s doing real-time analysis of performance. I don’t know how far away that is or if that’s even reasonable, but that’d be a cool thing.
Dr. Dustin: So that’s interesting. No, that’s a great application. And I would guess a lot of this is going to start coming out of like universities and research departments. We’ve got a really cool partnership going on right now with NNN: the National Neuropsychology Network. So this is a group of universities that are basically collecting neuropsychological data as they normally would within their clinics but a lot of it is flowing through Q- interactive and then it’s going to a centralized database. They’re compiling a large amount of detailed [01:00:00] clinical data that I think can be used to facilitate a lot of the types of analysis that we’re talking about here.
Dr. Sharp: I love that. Yeah, big data is valuable. I’m glad that we’re getting in on that. So what else? I could talk about AI technology forever, but other things that y’all are seeing just for the future with Q-interactive or even just digital assessment in general, maybe it would be interesting to talk about just what is not possible right now and what people think we should be able to do but we can’t. You mentioned that earlier or boundaries around digital assessment, at least for the next few years. I don’t know if there’s anything to dig into there.
Kristen: I think one of the requests we get a lot is around the complex figure drawing. So drawing on an iPad as opposed to drawing on paper is one of those [01:01:00] difficult, I would just say it’s challenging. Obviously, we understand that it’s different. Everything about it from the stylist to the actual sensitivity of the iPad, it just feels different. Everything is different about it and plus being able to detect and properly score something that complex is something we hope to do someday. We’re not quite there yet.
Dr. Sharp: Sure. Oh my gosh, if I could do the Rey on an iPad, that would be incredible. Score it, actually just score, that’d be incredible.
Well, it’s an exciting time. It’s really exciting to be talking about all these changes, the updates, the future. I 100% see us moving in this direction and with, I think you said this earlier, Dustin, we should be doing this more, right? And if [01:02:00] we were to make that 10 year prediction now, maybe the 10 year prediction is, in 10 years no tests will be on paper. I don’t know if y’all would agree with that but that’s my hope.
Dr. Dustin: Yeah. Throw that one out there. We’ll come back in 2033.
Dr. Sharp: We’ll put it on the record now; Jeremy says all paper are all digital.
Dr. Dustin: See how that turns out.
Dr. Sharp: Yeah. I really appreciate it y’all. Thanks for coming on and having this conversation. As we talked about, there are a lot of resources for folks as we make this transition to the new Q-interactive app, but it’s just fun to talk about this realm.
Dr. Dustin: Thanks for having us. It’s always fun.
Kristen: Thank you.
Dr. Sharp: All right, y’all, thank you so much for tuning into this episode. Always grateful to have you here. I hope that you take away some information that you can implement in your practice and in your life. Any resources that we mentioned during the episode will be listed in the show [01:03:00] notes, so make sure to check those out.
If you like what you hear on the podcast, I would be so grateful if you left a review on iTunes or Spotify or wherever you listen to your podcast.
And if you’re a practice owner or aspiring practice owner, I’d invite you to check out The Testing Psychologist mastermind groups. I have mastermind groups at every stage of practice development; beginner, intermediate, and advanced. We have homework, we have accountability, we have support, we have resources. These groups are amazing. We do a lot of work and a lot of connecting. If that sounds interesting to you, you can check out the details at thetestingpsychologist.com/consulting. You can sign up for a pre-group phone call and we will chat and figure out if a group could be a good fit for you.
Thanks so much. [00:04:00]
The information contained in this podcast and on The Testing Psychologist website is intended for informational and educational purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment. Please note that no doctor-patient relationship is formed here, and similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with expertise that [01:05:00] fits your needs.