Beyond the hype: Real-world challenges of AI-enabled e-discovery

Transcript:

Anthony: Hello, this is Anthony and Diana from Reed Smith and welcome to Tech Law Talks. Today we are continuing a podcast series on AI-enabled e-discovery. This podcast series will focus on practical and legal issues when considering using AI-enabled e-discovery with a focus on actual use cases, not just the theoretical. Today I am joined by my friend Dera Nevin from FTI. Welcome, Dera.

Dera: Thank you. Thank you for having me.

Anthony: And today we're gonna focus, we had an earlier episode today, we're gonna be focusing on the challenges of GenAI. Last time we looked at sort some of the opportunities of GenAI. And today I wanna focus a little bit on some of the challenges. And obviously there are challenges here. So Dera let's start with, obviously I think you're seeing people starting to use GenAI. I mean, this is not theoretical, right? People are using it now. And I think now, as opposed to a year ago, the real challenges are showing up because you're actually using it. So what are you seeing in terms of some of the challenges that you see from trying to use GenAI?

Dera: Yeah, so I'm going to start by just talking about challenges associated with the approach to using GenAI and then we'll get into specific things you need look at. So I think there's still a bit of a misconception that generative AI can do more than it can. And what I mean by that is that a lot of people think that it's still a general purpose tool, good at all things. Now, generative AI is remarkable. It has some remarkable capabilities and capacities. But a single generative AI isn't going to be able to do everything out of the box in every context. And so if it's been good at summarizing one document, that doesn't necessarily mean it's going to be good at helping you in a privileged classification. So I think just being realistic about it and then understanding that the major foundational models have different strengths and capabilities, and you probably need a few of them in your toolkit, is very helpful. with all of the caveats around privacy and security and making sure you're not using an open model. So once you still educate people about that, is that they are tools and techniques with capabilities instead of all purpose replacements for junior lawyers, then you're going to have a more productive conversation about which tool for which use case. And then you can get into specific challenges you might have with those.

Anthony: Yeah. And I think, I think that is going to be one of the major challenges. And I totally agree with you. I think one of the challenges that I think a lot of organizations are going to have is everybody wants to operationalize, right? Like this is what we do. This is the type of thing we use. And I think it's still very hard to do that because you can't, like you said, you can't say, I'm going with this vendor in this tool and I'm going to use it for everything. It's I have, you know, probably several vendors and several different tools for each vendor. and you're gonna be using different things for different business processes and the like. And I think that is, as you said, that's gonna be one of the big challenges. It's not like TAR, like when we did technologies review, there were what? Maybe two models, right? But it was really two models, right at the time. And you figured out which one was best, whatever, but this is very different. It's very different tools. And I agree with you. It is not, when we say GenAI I think we'll probably be... Next year, I don't think we talk about GenAI. We're going to be talking about a different type of tool and saying, oh, this tool is really good for whatever. Yeah. Yeah. So I think that's a challenge. I think one of the challenges is you have to have people who are fluent in all these tools. And I think that's one of the challenges that we have as a law firm. And I'm sure every organization there is going to have a challenge is you get really good at one tool and then you have to learn again on a different tool depending on the workflow. You know, it's that ramp up is gonna be a challenge for the whole industry as we start ramping up, developing tools, they're getting better. We figure out what use cases they are. So totally agree with you.

Dera: Yeah. Yeah. I'm sort of taken back to, you know, I don't want to date myself to when I was in kindergarten doing e-discovery but I remember how there were different data types. We had different processing engines for different types of data. You'd put group wise in one, you do your pop threes in a different, and we used to refer to those as a processing workbench. Like I got a processing workbench and we didn't really have the viewer problem because there wasn't relativity. It hadn't solved that issue yet. So we were different things. We'd tiff it out. So I refer to a lot of these now as it's we're back to that workbench stage where different tools have really great capabilities to help us with discrete tasks that can meaningfully drive value and savings. And I think that's the important thing is these tools can be deployed. They can have a meaningful impact in operational process, efficiency, cost saving. And so it's helpful. possibly even an important responsibility that we have to continue to understand these tools, but understand them as a workbench, right? Different tools for different capabilities.

Anthony: And I think just in terms of giving people advice, I think similarly, like as we're talking with clients about like, where do I use AI, whatever, obviously we can talk about where it's strong and weak and stuff like that, where we're seeing, and I think we did that a little bit on the last episode, like internal investigation stuff, and that's the easy one. But I think one of the things that I think clients also have to think about is like, where do I get the best ROI, right? Like, and I think that's, that is going to be like, again, we can talk about it and I think we talked about it last time. GenAI for like just traditional review, it's probably not as strong, but there's other places where you have a problem. Like I'm always like, don't use the technology and find a problem. Find the problem and say, can I use AI here to make it better? Right? And that's what I think it's like, and I think you talked about, like, I think privilege is one of them. There's others too, but I think that's one of things that people should do in terms of. You're not going to boil the ocean and try to say, I'm going to use AI and all my GenAI and all my different workflows. How am I going to use it? It's probably more sit down, think through and say, where, one, as you said, where is it most effective? Where GenAI is most effective? And then, two, where is it going to have the biggest impact on cost, efficiency, whatever? When you marry those two, then you have a plan, right? And then you can slowly do it and sort of overcome the challenge, right? Because it's overwhelming if you said, I've got to evaluate AI for all of my different workflows that you're going to take forever, right? Because there's so many different products and stuff like that.

Dera: Yeah. I also think breaking it down, what I see as another major sort of what I'm going to call top level challenges, people want to go to the outcome and use AI to get to the outcome instead of tackling a piece of the process, which might be the step before the outcome where it can meaningfully make a contribution. Right. So I see a lot of people saying, well, I want to run AI across all my documents and then develop a timeline. Well, why don't we categorize the documents in terms of what's likely to be responsive to the timeline before we actually do it? Or maybe have them group stuff or find patterns that maybe we didn't know or interrogate the documents or look at what we're not looking for. So there's other ways to approach the use of AI than trying to get at your final output. And your final output, you're going to need human review anyway. So if you use it for these more discrete pieces, you have better human oversight. You have better chances of catching if there is a problem, like a hallucination or a dropped citation or something like that. You can catch it earlier and you're still involved in the creation of the output. So as a lawyer or as an advisor, you can continue to have a meaningful contribution in the process rather than substituting the AI. Use the AI to do stuff that the humans have a really hard time doing that it's going to be good at, right?

Anthony: Totally agree. Alright so that's, obviously we talked about that's one big challenge. What other challenges are you seeing out there when trying to use GenAI?

Dera: So prompting, right? I think, you know, you can't really talk about GenAI without talking about prompting and the quality of prompting. So as I continue to work with people, there's still always the realization that prompting is quite different than a definitive search. So when you're using a search and you've got an index that's static and you're using Boolean operators, it will always return the same documents as long as the index isn't corrupted and there's no additional content in the corpus. Prompting does not work the same way. You do not always get the exact same outputs. And so it's interesting as I'm working with people and working on with prompting and I'm not even convinced I'm an expert level prompter. I think I'm good. I'm solidly, solidly good and I'm learning. You know, just the refining the prompts, right? And I actually think prompting isn't a single step. I think like prompting is an activity. we're starting to implement a whole methodology around prompting, prompting five times, specific ways of adjusting prompting to see how the results change and measuring the changes to determine the efficacy of the prompting. There's a whole methodology involved in prompting and what you change and how you structure the prompting. And so I do think that much the same way that lawyers have templates for certain types of outcomes or certain transactional documents or LOIs. we're gonna start to develop templates around prompting. And people think that prompting is magic words, like if I just have this defined prompt, but actually prompting is a methodology that involves wordsmithing. And just that iterative process of refining the prompting is so helpful. I also think, again, people use prompting to get to an outcome. I find my most effective uses of generative AI when it's truly impressed me and helped me. is when I've asked it to find the counterfactuals, when I've asked it to disprove what I'm doing, when I've asked it to do the opposite of what I'm trying to accomplish, that can be very helpful. Also asking AI to improve your prompting and explaining why can also be very effective. So a challenge is around prompting, but there are strategies available to people to improve it in a quick and effective way. If they can.

Anthony: Yeah. I look at it, I refer to this as I remember we started, you know, again, dating ourselves search terms, right? Search terms were, you know, e-discovery 1.0 search terms were the thing. And everyone was just coming up with search terms and everybody knows like they're bad search terms and there's good search terms. And it was a science, right? Exactly. You would test it, you test the search term, you'd look at the precision, the recall, you go back and forth, you negotiate with the other side. So I think we're sort of in the same way. Like we're now starting with prompts. It's not like it's going to be non-achievable. I have a problem with it just because it's not like, it's not as intuitive, right? Like as you said, it's not as intuitive as, if I add this, if I add this word, I know I'm going to get these documents that have that word. and now I'm comfortable and I can look at it and look at the documents, whatever. And again, same thing with TAR, when you said, okay, I found a document, this wasn't getting deemed responsive, I could just add that document and I know I'm gonna get similar documents. It's not the same, right? And I think that's gonna be the challenge is getting people to think about prompting in a new way, because it's not the same as those two which were used to.

Dera: Yeah, and I also want to just emphasize that the thing about searching is searching is always connected to retrieval, right? Searching and retrieval go together. The thing about prompting is that you can have a variety of different outputs. It can involve retrieval, it can involve summarization, it can involve the research. So there are different things that you're prompting things to do and the way you structure your prompt and how you structure your prompt really does impact the outcome. And if you're doing retrieval, it's quite different than if you're doing summarization, if you're doing classification, if you're doing a variety of other things. So AI is more powerful in that way, but how we approach each of those tasks needs to be adapted.

Anthony: So again, it's similar to we just talking about. It's a little more complicated so that for each use case, every business process you're saying, I'm going to use GenAI you're going to have to have probably a slightly different prompt methodology to sort of get to where you're going to be. And then obviously over time, I think we'll all be comfortable that, for this methodology, for this use case, for this tool, here's the methodology I've used to do the prompting to get a better result.

Dera: Yeah. Use case. Yeah. And by methodology, the methodology may be an iterative process, but the content structure quality of the prompting is going to be different. It's also different generative AI tools perform differently in different kinds of prompting. So I've given the same prompt to and GPT and a variety of things, they pointing at the same corpus and they behave differently. And so just understanding how those foundational models will perform in a context also is going to be important for people to understand.

Anthony: And I think, again, we talk about one of the challenges, this is going to be, just because all of this is, say, somewhat subjective and we're all learning, this is why I think a lot of people are nervous about using GenAI where you have to basically negotiate with the other side, right? Like anytime you're dealing with the other side and they have to buy into it, it's going to be incredibly difficult because you're not going to give them the prompts, right? It's obvious, in my mind, a lot of these prompts and the methodology is very focused on what I know about the case, right? And so I think that's gonna be a challenge as we develop, like where can we use this? And I think it's one of the things you have to think about when you're thinking about, you know, is this an appropriate use case to use GenAI and his prompting is, do I have to explain this to anybody, right? If I have to explain to the court, if I have to explain it to the other side, then it gets difficult. And that may be an area where you say, I may not get the bank for the buck. Maybe I should go somewhere else, you know, use a different. process because I still think because of all these things we're talking about, it's going to be like I remember in beginning of the day, like negotiating search terms used to be hard when they didn't understand search terms, right? And they didn't understand precision recall. They're like, I don't understand. But of course the name of the company is a search term. Like that's going to get every document. Like they didn't understand that. And I think it's going to be, I mean, we spend weeks negotiating search terms. I think it's going to be even harder for this. So I think Keep that in mind when you're thinking about, is this the right process to start using GenAI? Am I going to have to explain it the other side? Because I may want to wait, because all of this has to be played out. You have to get the firm methodology that you can go to defend and stuff.

Dera: Yeah. And that sort of speaks to don't always use it to try and get to the outcome, but you might be able to really refine a process. Right.

Anthony: One of the things we talked about is, which is obviously everyone's concerned with is cost, right? All the things we just talked about, not free, right? It's not like searching where you'd say, I can just use search terms and I can test search terms over and over again, and there's no real cost to it. Could you explain some of the cost implications of using GenAI? I know everything is different. The models are changing, but yeah.

Dera: Yeah, the models are different. And I think also it's really important to understand that a lot of the pricing is based on tokens and people don't really understand what tokens are, but tokens are basically characters. This is a very poor analogy for how it actually works, but it's the way that I've ended up describing it. So everybody knows that digital information is made up of bits and bytes. And a token is a certain number of bits and bytes. And it may correspond to a word. It may not correspond to a word. It really depends on how that technology divides up the number of bits and bytes into tokens. So you're paying per token. So the larger the data set, the more tokens it requires, the more expensive it's going to be. And then there are other things like context window and the size of the memory or the context in which that prompt is operating. So there's a lot of variables which make it less effective sometimes with larger sets and price. in Elastic with larger sets. And just sometimes the runtime just doesn't make any sense. Like it just does not make sense economically in terms of runtime to send an AI to summarize a million documents. You know, your case will just stall out and that kind of thing. So again, it's picking the use case and not going to the end and understanding the cost implications of what you're trying to do and pointing it to places. where it can be effective in saving time or saving money, often through a different process that you are displacing. So if there's a human-based process that is very time-consuming and hourly-based, and you can replace it with a technological process, it may be the same amount of money, but just a totally shorter time, which might be a real reason to do it, right?

Anthony: Yeah, and I think yeah, I think that's and I think the cost thing again It's a challenge because you have to understand that I know some some advocate and I think every vendor is slightly different I mean the costs are changing crazy right now again. It's not like early on it was by terabyte or whatever It's all over the place. So every time I talk to a vendor, they have completely different pricing models. Yeah

Dera: It could be per prompt, it could be per page, it could be for everything. And people are sorting it out because they're trying to figure out how to recover on the token. So it's very much like early processing when you're like, am I paying per page or per gig or what am I doing? It hasn't normalized yet. And I do think that the price is going to go down in time. It'll come down probably faster than we think as more capability and capacity goes online as the models get smaller and able to run on smaller. devices and contacts. So the price will come down. And I do think that there will be an incumbent's advantage to people who are using it now and determining good use cases. You'll be able to transfer all of that knowledge into the new.

Anthony: And I think one of the things that I did on the case where the vendor basically said, because of the cost issues and for some of these other issues, they did a sample, right? represented a sample and you ran it on a thousand documents of the hundreds of thousands that we had. And then we did the testing and the redo of everything on that thousand sample set. When we went overall, it wasn't great, right? Because it's a sample, you try, but. It's never gonna be as right. So that's one of the challenges I think we have. And again, before you start jumping in, make sure you understand the cost. Because what I don't want, I think this is the fear we all have is somebody does it, they do it wrong, it ends up being ridiculously expensive, and then everyone throws up their hands like, why would I ever use GenAI? We can't be in that situation. So just be careful when you're dealing with the costs, because it can run up very high if you make mistakes, very much so. Well, I think we're, why don't we stop here? Is there any other challenge? I know we were talking about use cases, but I think we'll do that in another podcast. So any other challenges that you can think of or is that sort of high level where we are?

Dera: No, I mean, we can also talk about just sorting out which ones to use and all of that kind of stuff. I think we've touched on that. People are aware that that's a problem. And I do think that as use cases solidify and we start to understand winners, it's going to be easier to navigate this market. But certainly in this area, partner with organizations and people that know what's going on. There are pathways through this. And so just continue to learn, educate yourself, experiment. not on client documents. But you know, it's actually, it can be fun. It can be really fun to you.

Anthony: Yeah, no, I agree. Although it's exciting time. It's certainly an exciting time, e-discovery hopefully you're all going to join us on this journey. So thank you, everybody. And like I said, there'll be more coming even with FTI and Dara. So thank you all.

Outro: Tech Law Talks is a Reed Smith production. Our producer is Shannon Ryan. For more information about Reed Smith's Emerging Technologies Practice, please email [email protected]. You can find our podcast on all streaming platforms, reedsmith.com and our social media accounts at Reed Smith LLP.

Disclaimer: This podcast is provided for educational purposes. It does not constitute legal advice and is not intended to establish an attorney-client relationship, nor is it intended to suggest or establish standards of care applicable to particular lawyers in any given situation. Prior results do not guarantee a similar outcome. Any views, opinions, or comments made by any external guest speaker are not to be attributed to Reed Smith LLP or its individual lawyers. 

All rights reserved.

Transcript is auto-generated.