AI compliance: AI in e-discovery, a new category of evidence

Transcript:

Anthony: Hello, this is Anthony Diana, and welcome to Tech Law Talks. Today we're going to be talking about AI in e discovery, a new category of evidence. and this is all part of our AI compliance series with SMARSH. And from SMARSH today is Robert Cruz joining us, along with Therese Caparo and Michael Rubayo from Reed Smith. Welcome all. Robert, why don't you take it over?

Robert: Awesome. Thanks again for the invitation. Hey, so you know, there's a lot to talk about regarding AI and and both e-discovery as well as investigation. You know, and I think that as we approach some of these topics on on the regulatory side, some of them are are fairly novel, but it feels like we've we've got a little bit more, you know, underfoot within discovery as far as how firms are using advanced technologies, machine learning and the like in the e-discovery process court decisions to lean upon, plenty of writings and speeches from Jud Judge Peck. but Anthony, what what are some of the key the key things that have already been established as far as the use of AI because of what we've been through with predictive analytics and TAR. Kind of what where do we stand right now?

Anthony: Yeah, look, I think, you know, more it was more than a decade ago, now 15 years ago when when people were really this was the big issue, right? Using TAR, predictive analytics, machine learning, and there was going to replace every document reviewer, which obviously didn't happen. there was a lot of discussion of like how do we make this work, right? There's a lot of dispute about it. I think it pr became pretty well settled that yes, you can use it. disclosure, I think, generally is high level disclosure is probably fine. There was a lot of discussion about validation sets and all this kind of stuff. But basically, I think the industry as a whole, with case law, developed really sound processes on how to use TAR, right? And validation was the most important thing, right? It's you have your seed set, there was a good process in place. It was well documented. And then you always did validation. And validation was critical to making sure that everyone knew that the tool was working, right? And cutoffs and all that kind of stuff. So that was pretty well established. I think people got very comfortable with it. You know, lots of best practices about continuous active learning and the like. And that was better. So it was sort of developed over time. And I think everyone in SL pretty much comfortable with the use of use of TAR, right? And we knew what the challenges were and whatnot. so that I think is what the expectation is going in with Gen AI and the use of Gen AI. Some challenges, I think, is it's not exactly the same. I know there's you know, recent case law that sort of said, well, you know, it was part of the disclosure. They said they were using AI. They used AI. There was a lot of dispute about, well, you know, we want to know more what you did about using the was relativity air and gen AI and how you did it. And the court basically said, look, it's just like TAR, it was disclosed. Generally they were going to be using it. you know, it's fine. And unless you have a dispute with something that wasn't produced or whatever, I'm not gonna have discovery on discovery, which was fine, except that. I'm not sure it's exactly right. I don't think, you know, Gen AI and the use of Gen AI for document review is very different than than TAR just as a technology. I don't, you know, there's still validation is going to be critical. I don't think we know exactly what the best practices are. When I talk to vendors, frankly they got they're all over the place in terms of how to use Gen AI and what the best practices are. we're all still learning. So we're sort of in that phase, I think. People are using it in the background, maybe not using it for final decisions on production. so there's a lot more sort of human in the loop document review still happening and maybe combining it with using TAR, using it for early case assessment, all of that. I think people are a little hesitant to sort of go in and say, We're gonna just rely on Gen AI, do a good prompt and send the documents out. so I think, you know, we'll see how it plays out. One of the concerns I have, again, maybe it doesn't matter, is that with Gen AI, if you run a prompt, right, and you think it's right, and it's the same data set, nothing changes, and you run that same prompt the next day or the next week, you're not going to get the exact results. That scares me. I don't know why, but that scares me as a it doesn't seem like Debbie's working. Everyone tells me I'm wrong, whatever. You still do the validation. So if it's validating, it's fine. But I think that is going to be an issue that people are going to try to grapple with, along with you know what what prompt did you use? And are are we gonna really get into I wanna see the prompt. So

Robert: Yeah. I think that's the key challenge is that non deterministic nature of the way the technology works. It's not it's not able to answer the question, you know, give me every individual document that has these keywords from these individuals. It may give you more or less, and that may change the next time you run it. So so Therese, y you know, we're not talking about in reviewing and making decisions on individual documents. You're talking about, you know, using artificial intelligence to potentially surface an item from a corpus or to use it with an ECA just to be able to pull back a a relevant subset of items. I mean I mean is that raise more risk, more concerns, more challenges, or is it back to ensuring that you can defend the method?

Therese: Look, I I think it goes back to ensuring that you can defend the method. Like if if we take away anything from, you know, the TAR and predictive analytics lessons for AI, what I would say is this. One, yes, it's okay to use technology to do your document review. Right? This the courts have been pretty clear that there is no block or prohibition on using technology to do your document review. Two, if you are going to do that, you need to have a process and be able to explain it. This this this is, you know, 99% of the time when we have an issue with discovery, it's because that the, you know, the folks involved couldn't really explain it very well, what they did and why they did it. And I think at first lessons learned from TAR we did have that at first where people couldn't or refused to explain how they were doing it and what the procedures were. So what I'd say is be prepared to be able to explain your process, right, if it gets to that point, right? Understand what the process is, have one, be able to explain it. As Anthony said, validation is the number one thing. You have to be able to say why you know it's working. Why do you know it's working? That can be because you look at what you didn't produce. It can be because you have processes in place to QC. There's a lot of different ways to do that, but you have to be able to explain why you did that, right? And you know what? Know the rules. Some courts, you know, have rules that require disclosure. If it requires disclosure, be prepared to disclose it, right? Understand the court that you're in, and the like. So I think that if you're taking away, you know, the learnings, it's really about, as we all know, if you're using technology as a lawyer, you are obligated to understand how that technology works. And if you are pushed, you have to be able to explain why you know it is working, why the bottom line, and even in the TAR cases. A lot of what came out is that it, yes, it is the obligation of the producing party to comply with the rules and to produce responsive documents. Your obligation doesn't change because you're using technology, right? You have to be able to demonstrate how you are meeting those obligations. And I think the bottom line is be prepared to do that. Understand the technology, know what you're doing, make sure you are complying with your obligation to produce those responsive documents, and understand that if you may get questioned, and if you get questioned, you just need to be able to explain it, right? And that's how you defend the work that you're doing, is that you are able to demonstrate you're complying with the rules.

Robert: Right. So what is case law telling us so far? 'Cause I know we've seen a cluster of cases that either related to citing fictitious sources, which I think Darwin will take care of that problem, otherwise privilege and you know, I'd imagine that AI washing is not too far behind in terms of false and misleading claims regarding how AI is being used by a system. That's regulatory in nature and also, you know, contractual as well. Any themes or patterns that you're observing to this point in some of the the prominent cases regarding the use of AI?

Therese: Yeah, I think that what I would say is, you know, th there's so many pieces to this when you're looking at, you know, AI in in in discovery. And I think that common themes that we're seeing at this point in time is one, AI is discoverable. Now when I'm talking about discoverable, we're focused on do you need to produce, you know, AI created information in, you know, a legal matter, you know, whether it's to a regulator or you know, in litigation. I mean, I think that number one is, and we'll go back to if y'all haven't heard our earlier podcasts, you should listen to them. is that the rules still apply to AI, right? The rules haven't changed. So if AI is creating relevant information, it is discoverable, right? We can make arguments around you know privacy concerns or you know, burden or things like that. All of which still apply. But the bottom line is one, courts have been pretty clear that if the AI creates relevant information, right, it those same rules apply and you need to produce that information. We've seen a lot coming out, right, privilege, right? What is what is the law saying, you know, about privilege? And again, the common theme from the courts is the privilege rules still apply. Now how they apply with AI can right it it can be tricky and it really depends on the usage. And the like of that. But I think the answer is be mindful when you're using AI that you are taking into account the privilege rules. What does that mean? That means that, okay, if I'm using AI in for in a legal matter, that I make sure that the use of AI by people supporting the lawyers are directed by the lawyers, right? So that if you want report product protection, it is clearly being done at the direction of the lawyers and for the purpose of the litigation, right? Similarly, if they're using it to get information to give to the lawyer for advice that it's clear that what they are doing and how they are doing it. Right. So you are keeping in mind if you are using tools, right, that the use of it is clear, right, under a privileged purpose, right? The the courts have not held that all use of public AI is not privileged, right? That you lose all privilege. But they are saying it's are you following, are you keeping it confidential? Right? Are you taking steps to make sure that that information is kept confidential, that it isn't exposed to other parties and the like. So I think they have been clear that look, when you're using AI, privilege rules apply. Make sure you have the same kind of controls in place when you are using AI that you would have if you were working with a human to make sure that you are demonstrating the intent to keep it confidential and the intent to have it protected by the privilege rules and the like. And I think that the other very clear rule that has come out, which we all can say should be obvious and apparent to all of us, but that, you know, if you use AI to draft a brief or to a complaint or for anything that you are going to use in the litigation, your obligation is to check it and to make sure that it is correct. You cannot solely rely on the AI and then add you know, walk away from your responsibilities to, you know, actually review the information as a lawyer and make sure it is correct and make sure it is a good faith argument. I mean you are not submitting something to the court that either contains you know false or misleading facts or law, right? So I think that, you know, as we're looking at the law around AI usage in the legal industry and in litigation and in discovery start to evolve, I think those are some of the clear themes that we're seeing.

Anthony: I would say I do think that there's gonna be a lot of litigation on proportionality. And do I really have to produce AI? I think there's gonna be a lot of pushback. we'll see where the courts go, but you know, meeting summaries is probably the one that I'm most interested in because we know it's not accurate. It's we know it's not you know, particularly helpful, particularly when you have notes about what happened in the meeting and the like. So I still think and prompts may not be relevant. So I think we're all preserving it. I think it's clear you have to preserve all of this or try to preserve it. But I think production is still there's going to be a lot of fights about it. Cause is it really, you know, you don't have to produce everything that's relevant, right? It should be material. And I think there'll be a lot of fights about is it really worth doing all this work to find prompts, responses, meeting summaries or whatever that could be potentially relevant when in the end they're not revealing anything that matters to the case. So I still think there's gonna be a lot of discussion about that. We'll see.

Robert: Hmm. Yeah.

Therese: Good luck to you.

Robert: Interesting on that front, just to examine how playbooks might evolve here, you know, it seems like it it may come into question how your reactive versus proactive strategy evolves. You know, do you retain it all in source systems? And if you do, how difficult is it going to be to extract it? Or how long is it going to take for that to take place? Or maybe it's not even within the SLA. it it's a choice, but maybe that affects the proportionality argument. But Michael, wh what do you see as the biggest changes to the kind of the way the companies are thinking about playbooks here and the fact that, you know, the original source could be a multitude of different locations, some of which y you you may have easy access to and others not not so easy. So what do you see the big changes here as far as playbooks go?

Michael: Yeah, I mean I think, you know, regarding e-discovery playbooks and what e-discovery teams need to do, both with the advent of AI just generally and then also with their adoption of AI tools, is is is you know the same as with any tool or with any new channel, right? If you want to, you know, we we know that AI is discoverable. We know that AI is producible in some some instances, so e-discovery teams need to be aware of where those AI prompts, where those AI outputs are being stored. They need to be aware of whether or not those prompts and outputs and that storage location are currently being stored indefinitely, or if there's you know disposition turned on. Because if disposition is turned on, you need to be able to stop it. You need to be able to place a legal hold if it's you know determined to be relevant to some active matter. that's sort of you know the discoveral aspect. But also as e-discovery teams start to adopt the usage of AI in their own practices, they need to make sure, as Therese was sort of getting to earlier that they are, you know, addressing and incorporating, you know, these new tools and new AI approaches into their processes and procedures so they can explain it, you know, defend it, make sure they QC it, validate it, and and you know, the same steps that they do with TAR and predictive analysis, but they now need to make sure they incorporate that with AI tools as well.

Robert: Yeah, great great comments are and I and I the point you raised regarding how long the data is retained is an important one, because we've already seen a case where one of the providers to no one's understanding or advanced warning change their retention policy from three months to five years. At some point they're gonna realize how many more data centers they need to build in order to maintain data for five years and they may switch it back. So that kind of leads to the last point of like how do legal and compliance teams deal with just the additional exposure and risk that might be created because of, you know, all the locations that may be leveraged, you know, in a in a needing to retrieve information and place it on hold and ensure it's available for review. So Anthony, what what are some key considerations there?

Anthony: Yeah, I mean, look, I think as as as Michael said, and and this is where it's daunting because there's so many different tools and so many different AI, you know, there's internal tools, external tools. It is daunting for e-discovery teams to figure this all out, right? I mean, it should be a requirement for with the enterprise that that retention and the like and and legal holds are considered when they're saying we're gonna launch a tool inside the business but then similarly, eDiscovery also has to be thinking about downstream. What tools are we using? How are they operating? I would hope that a lot of the downstream AI tools understand the risks associated with it. but I do think it's it's sort of both ways. I think, you know, resourcing is hard, right? Like everything else, you have to triage. So I would start with let's make sure the major enterprise tools. I know, understand all the issues and retention and make sure it's there and it doesn't change. because if you have, you know, hundreds of AI tools within your organization, you're not going to be able to police all of them. You would love to be able to say, look, we want it shouldn't be longer than 90 days. And if it is, you have to tell us or something like that, so you can sort of give a guidance and then at least it's it's brought to you. it's hard. I mean, I think the biggest challenge right now that legal departments and compliance and any discovery halves, there's so many. policing all of that is really difficult. So I think you just have to triage, give guidance, and then really focus on the ones that are most likely to show up in litigation. That's what we've been telling clients. Like you know generally the business processes that are going to be litigated and what comes up, HR, all these things that you know, like employment, if they're using it, you know what's going to happen. You know there's going to be litigation about it. Make sure you have those processes in place. So I think that's gonna be the key. 'cause you can't do everything, but you should definitely be trying, most importantly.

Therese: Yeah. I think that to build on that, I think that's never say this about Anthony, that that's exactly right. I think that what we're talking when you're looking at what you you know need to do, I think it is a couple of things. I think one, you're right. If you know, if there are enterprise wide, you know, specific tools that are general use, think of like everyone uses email, so we need to know what email is, right? Focus on those because you know they're gonna come up in litigation. Two, monitor. Ask about this in your cases. If you're doing custodian interviews, you're doing that. What AI tools do you use? Do you use any AI tools with this to make sure that you're identifying them at the outset and you can take steps for that case to preserve, right? Three, monitor your cases. If you have the same things coming up over and over again, put your time into those to make sure you understand how they work, you understand how that you can preserve and the like. Because again, like Anthony said, you can't look at all the but look at the ones that come up all the time and spend your time there to make sure you have standard processes, right, for those that are in place. So I mean, I think those are just some practical things that you can do to at least monitor. And I think, like Anthony said, don't try to preserve everything and find every tool, because you can't. You need to focus your resources on what is important to litigation, not on boiling the ocean.

Robert: Right. And I'll just add one final thing on that, which is it's it's also an important governance step to make sure your acceptable use policies are are appropriate for the way that the tools are being used. 'Cause one of the problems that starting to see from a lot of firms is these tools are expensive and you run out of tokens. And if you run out of tokens, you're driving people off the reservation. And maybe they pick up hugging face or granola or something else, and who knows what their capabilities are. So we're kind of unfortunately creating another off channel situation where you need to make sure you've got policies with enough latitude so people can do their jobs and aren't pushing them toward unsupported tools. Because that ultimately leads you to the problems of, you know, having to worry about those all those other infinite number of tools.

Anthony: Yeah. On that note, I think we could end there, Robert. Thank you for joining us. thanks everybody for joining us and listen in for new tech law talks. Thanks.

Outro: Tech Law Talks is a Reed Smith production. Our producers are Shannon Ryan, Amanda Saunders, and Mason Kautz. For more information about Reed Smith's Emerging Technologies Practice, please email [email protected]. You can find our podcast on all streaming platforms, reedsmith.com and our social media accounts at Reed Smith LLP.

Disclaimer: This podcast is provided for educational purposes. It does not constitute legal advice and is not intended to establish an attorney-client relationship, nor is it intended to suggest or establish standards of care applicable to particular lawyers in any given situation. Prior results do not guarantee a similar outcome. Any views, opinions, or comments made by any external guest speaker are not to be attributed to Reed Smith LLP or its individual lawyers. 

All rights reserved.

Transcript is auto-generated.