Relativity aiR: Real-world results – accelerating review and case strategy with GenAI

Transcript:

Intro: Hello, and welcome to Tech Law Talks, a podcast brought to you by Reed Smith's Emerging Technologies Group. In each episode of this podcast, we will discuss cutting-edge issues on technology, data, and the law. We will provide practical observations on a wide variety of technology and data topics to give you quick and actionable tips to address the issues you are dealing with every day. 

Anthony: Hello, this is Anthony and Diana from Reed Smith's Emerging Technologies Group. I'm a partner here in the New York office of Reed Smith. And today, we are doing another of our podcast series on AI-enabled e-discovery, which hopefully will help legal departments determine when and if to use AI-enabled e-discovery. Our podcast series is really focused on practical and legal issues when considering using AI-enabled e-discovery. And we are going to focus on actual use cases and not just on the theoretical. And with that in mind, today we're going to be focusing on Relativity Air, obviously one of the big AI tools out there that was available to a lot of people. And joining me today are Marcin Krieger, who's counsel here at Reed Smith, part of the RED team and Emerging Technologies Group, and Sharri Wilner, who's part of Gravity Stack, our technology sub of Reed Smith. So thank you guys for joining me.

Marcin: Yeah, thanks for having us, Anthony. Kind of cool thing to have Sharri here with us. You know, we are now part of Reed Smith Legal Solutions, which is really combining the technology and the legal side of things. And so the two of us are pretty excited to provide some perspectives on Relativity Air and our use of it.

Sharri: Yeah, I'm excited to join you guys today. Thanks for having me. 

Anthony: Terrific. So Sharri, let's start with you. Since you're the technologist here, what is Relativity Air? Everybody hears about it. What is it? What does it do?

Sharri: Yeah, absolutely. Relativity Air is Relativity's generative AI framework. It's built directly into Relativity One, and it brings the power of these large language models into the e-discovery and legal workflow environment. But because it's within Relativity's secure framework, it helps that it has that more defensible air to it, if you will, since it's in that particular cloud. But what makes it special or unique is that it's not a plugin or a third-party add-on. It's embedded right into the platform. So this means teams can work in there faster and hopefully smarter without sacrificing any control or concerns of compliance issues. 

Anthony: And what does it do? Like, how do you use it?

Marcin: Yeah, so what's really cool about Relativity Air is that it doesn't really rely on the traditional technologies that we use in discovery. Unlike traditional technology-assisted review, which is sometimes called TAR or CAL, Relativity Air is leveraging the latest and greatest generative AI technologies, primarily but but not entirely built on top of OpenAI's GPT protocols. So this is a really new way of approaching discovery. It's a completely novel approach. Now it's being adopted broadly in the industry, and there are a lot of products out there that are leveraging the same technology, but it's novel in the sense that this is doing new things in new ways that we've never been able to do before.

Sharri: And there are two key components to it right now. There's error for review, which assists in the document reviewers by suggesting coding decisions. And then there's error for case strategy, which is focused on deposition transcripts and summarizing testimony, you know, extracting issues and building factual narratives faster. 

Anthony: And then let's just focus on the relativity error of the review part. You mentioned that it suggests basically, I'm assuming it suggests whether it's responsive or not. Is that fair? Is that what the concept is or relevant to the litigation?

Marcin: Yeah, it's interesting. So with things like Cal technology, when documents are reviewed, the algorithms look at other documents in the review universe, they assign a score. So you always hear about like a TAR or a Cal score of like zero to 100 or some other integer value, and it pulls similar documents based upon what human reviewers are doing. But what air does is completely different what air does is you tell air what makes a document responsive it's almost like you are giving it a document review protocol like what you would give to a junior associate or an e-discovery attorney aiR then reads that protocol and it looks at each document independently and it says based upon what I read up in this document i tell you it's highly likely to be relevant it's relevant it's maybe relevant it's not relevant or you know maybe there's an error or something else but rather than scoring documents zero to a hundred based upon how they are similar to other documents each document is looked at independently on its own based entirely upon the instructions that are provided to it through the the prompting that you know that like e-discovery protocol thing I provide but essentially it's a very complicated prompted. You hear the term prompt writing. This is prompt writing, God-tier level difficulty. You write really amazing prompts and you get really amazing results from the AI. 

Anthony: We're going to talk a little bit more on that because that seems like a challenge to me. But first, before we get there, based on your experience, and this is something that we talk about in some of the other podcasts where you have to look at a tool and evaluate it and determine what the best use cases are because they're not going to be good for everything. So for a relativity aiR review, what are the use cases that you're seeing where it's most effective?

Sharri: Yeah, there are a few standout areas, Marcin, if you want to. Yeah, well, so I'll give a couple. Sharri has a lot more experience because the work that she does touches a lot of different practice groups. But myself being in litigation, where we see aiR being used the most frequently right now is in accelerated document review, where you have a fairly large data footprint, maybe not massive. But like, you know, data footprints, 50,000 to 100,000 documents where the cost of a human review makes it unfeasible, usually based upon things like the amounts of controversy of being able to get human reviewers to get through 100,000 documents may not be financially feasible. These tools are lower cost. And with a small investment of time up front, we can then use this technology to go through these documents, identify the ones that are responsive or relevant. But also, interestingly, aiR allows you to do issue coding. So another place that we see this being used is in environments where we already know the documents are highly likely to be relevant. Let's say it's an opposing party's production. And rather than reviewing them for relevance, because they're presumptively relevant because they're in response to our discovery requests, we can use aiR to identify which requests they are responsive to. So this would be used really well for categorization. In that way, let's say an opposing party produces 20,000 documents, presumptively all of them are going to be relevant. So traditional TAR technology doesn't really apply because TAR is going to say all of these are relevant. Human review team might go through these and categorize these documents into five different buckets. But then you have to take one of those five buckets, get an associate or a senior counsel to review that bucket. Now we can use aiR to categorize documents. And we tell aiR, here are our five buckets and our prompting defines them. That way aiR does that first pass. And then you could say, okay, great. Documents in bucket one go to our damages expert. Documents in bucket two go to senior counsel. Three might go to a junior associate that's just going to confirm that nothing was overlooked. But using aiR to as a culling method for very rich document populations is something that you couldn't do with traditional TAR without basically having to review the whole universe. Now we can use aiR to really cut things down. Another useful place, and Sharri, I'll let you provide some insights that you've worked with aiR on, are things like internal investigations. So we're outside of discovery, now we're dealing with very narrow issues, but very broad populations of documents, sort of the inverse. In my last example, we were talking about a universe where everything was relevant. Here, we're talking about a universe where less than 1% of documents are relevant. If we were using traditional TAR algorithms, we probably would still have to review 25% of the population and then still have to go through a second or a third round to really get to those super key documents, with appropriate prompting, we can conduct an internal investigation in hours instead of months. And we can identify the 100 most important documents out of a population of tens of thousands. Instead of the traditional method of tens of thousands, you still have to review 10,000, which leaves you with 5,000. And then an associate has to get those down to those couple hundred. Now aiR can really get you to those couple hundred documents, not literally instantaneously, but in terms of the timescale of a traditional internal investigation, the equivalent of nearly instantaneously. 

Anthony: Yeah. The speed to knowledge is the big call. 

Marcin: Exactly. 

Anthony: Internal investigations and the use of AI. So Sharri, what else are you thinking in terms of use cases that you've seen that it's been particularly helpful?

Sharri: Transcript summarization and deposition preparation. So using error for case strategy application to help summarize key points and align particular issues within transcripts to help prepare for additional depositions and or really just intelligence driving. So instead of worrying about taking a set of data to just do review, now instead you're driving the intelligence piece of it. So reviewing those, having it help you review those transcripts to make smarter questioning for future depositions and, you know, certainly upcoming trials. So it's that knowledge capture piece of it, I think, is really where we're seeing good use cases on that. I mean, Marcin, you've used it, I think, a few times in that particular respect. And soon what we're hoping to be able to do with aiR Assist is being able to surface those ‘hot’ documents, if you will, Marcin, the ones that you're referring to in investigations, to bring those really up to the surface quicker with respect to, you know, understanding very particular issues and themes within a investigation. 

Marcin: Yeah, that's right. And Sharri, you mentioned something pretty important, which maybe we didn't clue in on. Within aiR, we mentioned that there is aiR for review and aiR for case strategy, but Relativity just recently this month at Relativity Fest announced that they are opening an entire new suite of aiR products. One of them being, they call it aiR Assist, which is taking these workflows, this, Anthony, you call it speed to knowledge, our quest to find key documents faster, and they are making new tools around it. So I described how we could use aiR for review to find key documents, but Relativity is really listening to its clients, and they're now coming out with completely different feature sets on how to do early case analysis, how to do early key document and issue spotting, and taking a lot of those early case assessment tools that already may have existed on the market from other providers, but approaching them in a new way using generative AI instead of traditional, algorithms and programmatic tools for things like identifying the right custodians identifying the right date ranges identifying the right hot spots in your data set. 

Anthony: All right then that's great. Let's talk a little bit about the challenges and obviously every use case is going to be you know there's going to be certain challenges so Sharri why don't we talk first with you in terms of some of the technical challenges.

Sharri: Yeah, absolutely. From a technical standpoint, some of the challenges that I'm seeing for reviewers happens to be something as easy as change management, right? Understanding that we are now otherwise needing reviewers to more clearly communicate the background and issues within the case, right? And being able to get them to think about those at the forefront before any review starts, which also then brings to one of the most challenging from a technical standpoint that we see, and we alluded to earlier, is the prompting skills. aiR responds best to specific contextual prompts, as Marcin had indicated. Teams really need to practice phrasing questions that are going to get them the meaningful answers that they're looking for. 

Marcin: So building on that, you're absolutely right, Sharri. Some of the challenges that we see from the legal side, I'll start off with what you said, which is getting a case team to articulate what it needs out of the documents at the start of the review is a lot harder than people think. Most document review workflows are you start with your protocol with what you think your issues are, you start working through the documents, you're three days in, you're changing course. You're a week in, you're changing course. With aiR, changing course isn't really an option. You really have to have that roadmap set early on, and you have to know how to articulate that roadmap. Like Sharri said, prompt writing. From the legal perspective, we have a lot of novel challenges. We are over a decade past the cases that said that TAR is, you know, accepted, right? Judge Peck's early opinions. Now we are still fighting over the use of TAR, and now we're introducing a whole new technology, a whole new way of approaching document review. So we have challenges in terms of. Client buy-in, getting clients past their reluctance about the use of new technology. When we're not even in a litigation context, you have clients who just don't trust it yet, who are concerned that they're going to miss a needle in a haystack. And the challenge is effectively communicating to them that if you had humans review all of your documents, you're still going to have a high risk of losing the needle in a haystack. It's called a needle in a haystack for a reason. The idea that someone's using secret code words and a human review team looking at every document and spending a million dollars somehow is going to find that code word better than an AI which is designed to pick up on those things and say, oh, you're using code words. So that's our first challenge. Next challenge is in the actual litigation context. We don't yet have judicial acceptance of the use of generative AI technology. So if clients are deploying it in the litigation context, we have to make sure that what we are doing conforms to the existing rules, the existing case law. There is a framework of case law that allows for the use of these types of tools, even without judicial acceptance. But we have to make sure that we are conducting ourselves in an ethical way so that should questions arise, our work is defensible. And if it ever gets in front of the judge, we want to make sure that we are the law firm that gets a judge to say, I've looked at what you're doing and I agree, and not the law firm that says, I looked at what you're doing and I have concerns. 

Anthony: Yeah, Marcin, on that side, I'm obviously with TAR, there was a lot of discussion about validation, right? How do I know I'm getting everything? And there was lots of sampling after the fact and whatever. We all developed really good processes, which is why judicially acceptanced. Are we using the same type of thing for Relatively aiR in gen AI tools? I mean, it's not the same way of looking at it. So I'm just wondering, How do we validate? What is the validation?

Marcin: Right. So validation is the name, you know, that's the real name of the game. It doesn't matter what technology you're using. If you're not validating the technology, you're not using it correctly. Another way of saying validating, validating is just a nice way of saying you're using it the right way. Now, the traditional way to validate a TAR review is through statistical sampling. Very simple, but very well accepted and understood mathematical principles. You have the universe of documents you've identified that are responsive. You have the universe of documents that the AI says are not responsive. It doesn't matter if you're using TAR or Gen AI or something completely different. There is a very basic way that you can validate, which is called illusion testing, and that's a whole separate podcast. But you basically take statistically significant samples out of the stuff that the AI says are not relevant, and you can calculate how many relevant documents did you miss? The challenge with generative AI workflows is that with TAR, we knew that the algorithms pulled together documents that are similar to the ones you've coded responsive. So if you code a document that talks about combs as responsive, other docs about combs get pulled in. And so there's like a conceptual understanding that, yeah, like this is all sort of like homogenized and separated and we can sample. With validating generative AI, that validation process starts on day one it starts with validating the prompts themselves you have to make sure that you start your document review very carefully and slowly with a very small universe of documents let's say 50 to 100 you test aka validate your prompts against them you make sure that the results are incredibly correct very accurate as in the 50 responsive docs you put into your seed set, the AI correctly identified them as responsive and the rationale it gave makes sense. And the documents that you put in there that are not responsive also make sense. Then you expand that to 500 documents, then a thousand. Then you can turn it on on your entire document review population. So you're validating for errors before you run it on the whole. Now after you've run it on the whole, I think the same workflows that you would implement in a human review kick in, you do things like look for file names that should be relevant, look for concepts that should be relevant. But these are validation steps that you would take in a human review as well to make sure that your human reviewers aren't missing things. And as the last step, you can do that random sampling and you can do that illusion test but the main difference is we spent a decade with TAR and CAL making algorithms that are resistant to errors early in the process so you can validate later generative AI has no tolerance for errors at the start of the process a lot of the validation work happens before you even kick off the full project, 

Anthony: And that leads us to sort of the best practices, which is like, that seems like a lot of work. We'll see. And obviously, I think as you alluded to, Sharri, is you have to know what you're doing, right? So I think best practices, don't do this on your own without having someone who knows what they're doing. Because the way you explained it, I couldn't do this. If I started, I wouldn't be able to do this. I assume I would learn every time. But Sharri, what other best practices are you seeing? Because we have to close up here. But what other best practices besides what Marcin said in validation?

Sharri: It actually sums up everything that Marcin was saying. Start small, pilot an error in one workspace with very specific issues, validate often, so you're comparing your AI results even with human review samples, document the process so that you are, you know, documenting what your findings are. So you're keeping records of your prompts, your validation results, your workflows, and all that's going to help strengthen your defensibility. Train your team, right? Make sure that, you know, your technical reviewers are on standby with you to help you craft those good prompts and how to interpret the results. And then iterate continuously. Treat error like it's a teammate. Refine your process and, you know, capture your lessons learned so that you can take these lessons onto larger, more complex use cases. 

Marcin: So my best practice, going back to what you said, Anthony, don't try to do it on your own and partner with people who know what they're doing. Partner with people like Sharri, who understand the technology and the process, who can detect when things are going wrong before things really go wrong. Have the right experts in the room with you, and then you don't have to be an expert, but trust your experts and listen to them. 

Anthony: And as we know, Marcin, you have an ethical obligation to do that as well. And I think, look, I think there is certainly risk here, and I'll conclude on this. So I think making sure you know what you're doing, you have an ethical obligation to know what you're doing when using these tools. I think this is the wave of the future, but I think it's going to take some time. And my fear is that we're going to have that bad case that leads to disaster, right? Where someone doesn't know what they're doing and the court's like, why are you using this tool at all? So hopefully that won't happen. So best practice, make sure you have good people who know what they're doing. We're going to have many other podcasts to talk about this. We may have another one just on Relativity aiR if we have other use cases that we can think of. But thanks, everybody, for joining. And thank you both.

Marcin: Thank you. 

Sharri: Thanks again, Anthony. 

Outro: Tech Law Talks is a Reed Smith production. Our producers are Ali McCardell and Shannon Ryan. For more information about Reed Smith's Emerging Technologies Practice, please email [email protected]. You can find our podcast on Spotify, Apple Podcasts, Google Podcasts, Readsmith.com, and our social media accounts. 

Disclaimer: This podcast is provided for educational purposes. It does not constitute legal advice and is not intended to establish an attorney-client relationship, nor is it intended to suggest or establish standards of care applicable to particular lawyers in any given situation. Prior results do not guarantee a similar outcome. Any views, opinions, or comments made by any external guest speaker are not to be attributed to Reed Smith LLP or its individual lawyers. 

All rights reserved.

Transcript is auto-generated.