The missing foundation of AI governance: Getting data governance right

Transcript:

Anthony: Hello, this is Anthony Diana from Reed Smith, and welcome to Tech Law Talks and our podcast series on the importance of data governance to AI governance. Today, we will be discussing generally how data governance helps with AI governance. In future episodes, we will be focusing on more practical steps you can take to effectively implement AI governance using some of the elements we'll be talking about today on data governance. With me today are Therese Craparo, Kiriaki Tourikis, and Kristen Pologruto. Welcome, guys. Why don't we get started? This is an important issue. I think we've dealt a lot with AI governance and talking to clients about AI governance. And oftentimes, this is sort of the missing foundation because they're thinking, I want to do AI governance. And if you don't have data governance, it makes it very difficult. But let's start generally with, for those who don't know, what is data governance and why is it important just generally? Kristen, why don't you start with this?

Kristen: Sure thing, Anthony. Thanks. So data governance is really the programmatic framework that enables an organization to manage its data assets properly. Data governance coordinates across all data disciplines and functions within an organization. It really pulls together the professionals who are responsible for records management, data security, information technology, legal compliance, privacy and cybersecurity. All of these functions work together to ensure that the information assets are properly aligned to meet and contribute to the organization's overall business goals as well as making sure it meets its legal and regulatory obligations. And data governance really addresses the entire life cycle of data, from its creation to its classification through its disposal. And really, we accomplish this from a a lot of different ways, but organizations create policies, they create and they update retention schedules and record inventories, they engage in employee training, as well as creating and managing programs that allow for the proper disposal once the data life cycle has been met. And really all of these factors sort of are what we overarching call data governance.

Anthony: Yeah. And I think as I said, mentioned, in future episodes, we'll start again with a little of the some of the elements that you mentioned in terms of a data governance program, how, policies, record management policies and some of the data governance policies help. But I think particularly, record retention schedules and record inventories. And generally it's like basically like just identifying where's your data? Where's your most valuable data? What is your most valuable data? How long you should be keeping it? And where is it, which is frankly one of the challenges when people are using AI and they're sort of pointing AI to your data in your organization, you want it to work.

Therese: I think, Anthony, though, one of the the things that I think people often don't realize is there is a presumption that, well, the company must be managing their data well. We must have data governance in place. And we often hear that as a kind of misconception. And the reality is that for a very long time, a lot of companies don't have a complete data governance program in place. They don't have complete procedures and policies supporting their management of their data, identifying where their good data is, identifying where their data is that's not their records, that maybe not - may be less reliable. And I think that what we're seeing is that in the the rush to use AI and the desire to expand the use of AI, companies are starting to realize just how much their data governance programs may not be as robust as they would like them to be and starting to realize how important that will is as they try to roll out these AI, you know, tools and processes.

Anthony: And I think you're probably sugarcoating it. I think most organizations have very bad data governance policies, right? And a program. it's been something that has been neglected often for like twenty, thirty years. and we saw a little bit of that change with privacy and GDPR where people started saying, we need to know where our our personal information is and starting deleting stuff. But I don't think there's really any organization that I think we've dealt with that you would say has a mature data governance program. And I think Therese, as you said, AI is sort of exposing how bad this is. And you know, as we've often said, data governance is corporate governance, right? It's part of your corporate governance, but is often neglected, partly because it's not a priority. There's no budget. We hear that all the time. But that is now changing. So Kiriaki, why don't we talk a little bit how data governance, good and bad, may impact AI governance and the use and the risk of AI just generally.

Kiriaki: Yeah, So I think one of the most important points is that you really can't have, I think, effective AI governance without effective governance of the data that sits underneath the AI. When folks talk about AI governance, I think the conversation understandably tends to focus on the AI tool or algorithm itself. Is the model reliable? Could it hallucinate? Is there bias? What's the human oversight? And those are important questions, but the layer that gets overlooked is the data layer. What data is the AI actually interacting with? AI systems obviously depend on data. They interact with enormous amounts of enterprise information very quickly. And so that's where traditional data governance becomes incredibly important. Because when we're talking about AI governance, we're also thinking about what data is the AI allowed to have in the pipeline, under what conditions, what should be excluded, what are the controls we have. Sounds like super easy questions, but as Anthony's referring, most organizations don't have mature or robust data governance programs. So they have accumulated decades worth of data across disparate systems and before AI, those weaknesses in the environment, of course, still created risk, but AI magnifies that risk because of the speed and the scale at which it can find, combine, summarize, and analyze information, right? So if you had decades of information on a share drive, an individual employee might not ever really actually sift through all of that information when answering a question, but you connect the AI tool to that repository and suddenly the technology is capable of searching and synthesizing all of it. So you may be asking a very specific question about something, you know, a process that is more current. but it's really looking across the corpus of of information that it has access to. So it also impacts classification and data access, right? All enterprise data is not the same. You're gonna have public information, but you're gonna have internal business information, confidential information, privileged material, personal information. For some organizations, you'll have material non-public information. The AI system isn't inherently going to know the rules around those distinctions, right? It understands access and those technical boundaries. So if you, as part of your data governance, don't have data classification, and access controls, you're starting, I think, your AI governance program with significant weaknesses. It is important to consider that AI doesn't just consume information, it creates information. So again, as you're generating prompts, you're also generating outputs that could be summaries, analyses, transcripts, recommendations. And that's going to raise traditional records management questions in a new context. When is that going to be a business record? How should that be stored? What kind of disposal or attention should be around all of that? All of these questions are part of a broader data and records governance framework, so I think good data governance should be viewed as enabler of AI. And I want to stress that I think AI governance is not just governance around the model. It's governance of the entire information life cycle around the model. So the data going in and what it can access and the data coming back out.

Anthony: Therese, let's talk a little bit. There's a lot to unpack there, but I like to make that real, right? And we've seen this with tons of clients with Copilot, right? Copilot, it's enterprise, and I think that's one of the - the challenges here is, there's obviously use cases where AI is very specific in terms of what it's going after, and maybe you want you're comfortable with that data set, but enterprise AI, which is, you know, things like Copilot, really have exposed a lot of bad data governance. So why don't you put a little bit more meat to the bones in terms of how some of the risks that we're seeing with Copilot and other enterprise tools.

Therese: Sure. And and I think Anthony, this goes back to what you were mentioning earlier about kind of decades of a lack of data governance around systems. Because, look, what we're seeing in these enterprise systems is that somebody goes to use Copilot as an example, and it surfaces data they didn't even know they had access to, and that maybe they should not have access to. And so it's surfacing a lack of access controls around the underlying data that, like Kiriaki said, you use these systems, and it's finding data the users didn't even know was there that they had access to, that they would not have looked at. And in some cases, it's exposing access to data that they shouldn't have access to because the access controls are not so good. Or we've seen a lot of feedback around some of these general enterprise tools saying, this isn't so good. We're not really getting great results from this. results are inaccurate, the tool isn't really valuable. And sometimes it's not that it's the tool that isn't valuable or working correctly, but they look at the underlying data and realize it's pulling from data sources that are not reliable data sources, that are not data sources that are going to help inform the model to give you a more accurate result. And again, that's a result of looking at everything, and all of that data is being treated as data that is valuable data or accurate data to help ground an informed response when really, that isn't the case. Other examples we have seen is concerns about data that are like privileged or confidential data where somebody is using a tool to create a document or to get an answer, and it's pulling in information that may be privileged, that can't be shared with a third party, yet they're using it to create, say, a PowerPoint that they want to present to a large group. But those documents, the security or confidentiality levels on the documents, the privileged nature of the documents isn't built into the identification of those documents and the security of those documents, so the system is going to pull those in regardless, right? And we've seen this happen over and over and over again with the use of these tools. Again, it's looking underneath the hood and making people realize, particularly with these broad systems, where you're not targeting the data, that that underlying lack of governance is impacting the value of the output and the ability to really leverage the AI tools to their greatest capacity.

Anthony: Yeah. And I think that's the one of two big things when we're talking to clients about data governance, and particularly with Copilot and the like. And they're like, well, we want to keep everything. One is the effectiveness, which I think is obviously - that goes directly to the benefit, right? ROI, which is a big focus right now; am I getting really good stuff? And oftentimes the problem is if you don't have good data governance, you're not getting good results, which is a problem. You're not getting the value from the tool. But then I think the risk aspect, which, you know, we've been talking to lots of legal departments about this, is really high. Whether it's violating NDAs, privilege issues, like the legal department doesn't know that you know, so many people have access suddenly to privileged information, that's gonna wipe it. I fully expect there's gonna be litigation about this and waiving of privilege because of AI exposing the fact that too many people have access to it. Access to a privilege document suddenly breaks the privilege. So I think there's a lot of risk there. And when everybody says, well, how do we solve it? The easiest thing to do is just data governance. Get rid of stuff that you don't need, making sure you have good access controls. All of that is really going to be important. But it's it's still not quite there yet, which is why we're having this podcast to make sure that people understand that. So next I want to talk about, Kristen and Kiriaki, one of the issues that we're seeing in terms of AI use is you're gonna have to justify what you're doing with AI to third parties. How can data governance sort of help with that?

Kiriaki: I can jump in first. So I think Anthony, one the most important benefits in the context of AI is defensibility. So as organizations move beyond just experimenting with AI, they're using it for substantive business activities, I think they're gonna increasingly be asked to explain the uses to third parties, to regulators, to auditors, to customers, litigation adversaries, somebody even conducting due diligence in connection with with a transaction or acquisition. And so I think the broader question becomes: if somebody asks you to justify how you're using AI, can you show your work? I mean to me that's what defensibility really means in this context. And so it's not demonstrating AI is perfect, but it's, I think, demonstrating that the organization understood what it was doing, understood the information involved, made reasoned decisions about the use, put controls around that, and documented the support for those decisions, right? So I think three concepts come into play: explainability, data provenance, and life cycle accountability. So explainability: really, can you explain what you're doing? So I think when people hear AI explainability, they focus again just on the technical mechanics of the model, which is obviously important, but from a broader data governance perspective, there's also a more practical level of explainability. Can you say what we're using the AI tool for? But what information is supposed to be used? What information is it not permitted to be used? What are the controls you have in place? And what happens to the information the AI generates, right? So can you explain those governance decisions around that use? For data provenance, where did this information come from? Right? It sounds technical, but it could be a simple where did you pull from? If you're relying on that AI output, you really want to understand where did it get its information? So what systems supply the information, what businesses own that, is that, you know, information the user should have access to? Is that current information or is it stale? It's important if somebody ever challenges your AI use or the the output to know Again, what is the quality of the data? And I think that's where some of the more traditional data governance tools can also become AI governance tools, right? A data inventory, a record inventory is a basic exercise in terms of governance, but it's what systems do we have, what information's in there, who owns it, right? But it's gonna help you demonstrate maybe access controls or what information was used. Same thing with the record retention schedule as you alluded to, Anthony, right? It's telling us what data is important at the organization and how long are we keeping it. And the last is life cycle accountability, right? I think this probably feels more like traditional data governance, right? But you're thinking about the entire life cycle. I think an organization has to think through all of these questions because if you don't, you're gonna end up with having massive amounts of of AI generated data without a clear reason for keeping it. And the last point I wanna make is it's important to have data governance for defensibility because it's really contemporaneous documentation, right? There's a meaningful difference between documenting a decision when you make it and trying to reconstruct your rationale years later, after somebody asks a difficult question. So if a third party, a regulator, a customer, litigation opponent asks, why did the AI have access to this system? Why did you retain the system? What controls in place? If you're trying to answer these questions for the first time years later, it's a significant reconstruction exercise. Practically speaking, people's roles have changed, systems changed, configurations change, memories fade, right? And so this becomes a very time intensive exercise and may not be fully complete. On the other hand, if you have data governance, you already have a policy, both for records management or information management and AI policy, you're gonna have a record retention schedule, you're gonna have a record inventory or maybe some data maps, you'll have documented access controls, right? So you're gonna have all of these things that tell the story in real time, and that's gonna show that we were using AI responsibly. We were thinking about it. So for me, AI defensibility is really documentation plus life cycle control. So you have the the policies, the schedule, the inventories, access controls, things like that, and they're just the contemporaneous evidence that the organization really made reasoned decisions. And so I think that puts the organization in in a much stronger position. It's important to also think about the underlying data itself, right? We've been talking about that. And so I think Kristen can kind of take us into how understanding your data environment and putting the right controls around the data affects both the reliability and ultimately, therefore, the defensibility of the AI.

Kristen: Yeah, I totally agree, Kiriaki. I think just a couple of points on access control, because I think that's one of the more important aspects when we think about AI governance is really controlling the data that is fed into the tool. So, you know, we we worry about prompts, we worry about inputs, outputs, and that sort of thing, but we need to think about: what's the data that this tool is accessing? You can set up a tool where it can, go through your entire data infrastructure. It can go structured, unstructured data, custodial, non-custodial data sources, active, legacy repositories. So without those proper access controls that's really directing the AI to the proper data sources, you can really come back with some not desirable output. And you know, it's sort of that old adage, you know, garbage in, garbage out. I think that's really true when you start thinking about these AI tools when you're not really controlling the access to where the tool is and not directing the tool to the proper data set sets that you want your tool to access so that when you are putting in those prompts, you're getting out the outputs that you're desiring to get out. So really having a good understanding of your data framework overall, talking to the right people within your organizations, making sure that you're not leaving, you know, stones unturned over when it comes to potential data sources that your tool can access. You need to really focus on those controls.

Anthony: Yeah, and I think we see, and Therese, like we see this all the time. People have policies, AI policies that say, Well, you know, if it has personal information or sensitive personal information, you can't use the tool against that, right? So the some approval committee says, Yeah, you can use the tool as long as you don't do that. The problem is if you don't know where that stuff is, you could be violating that. And I think this is where, from a regulatory perspective, right, this is what the regulator is going to be focused on, if you have privacy regulators where it's state AGs or and if you're global, the DPAs, they're gonna ask: what controls do you have in place to make sure you don't - you're not using the tool on personal information or sensitive personal information? And if you don't know where that is, you can't justify it to the regulator. What are some other thoughts on where whether it's courts or regulators maybe really pushing on AI governance? But it really ends up being a question about data governance.

Therese: We all know that the regulators are heavily focused on AI use across the board and across industries. And a lot of what they are scrutinizing to echo Kiriaki's emphasis on being able to defend the use of it is, how are you managing it? Do you know what data you are using in these tools? What are your controls? Not just who's using the AI, but what are your controls around the data that is being used? I mean, frankly, the regulatory framework that we're starting to look at with AI is not just about the models. It is about what is happening with the data. What is the output? How is that output managed? What's going in? How is that being managed? The fear is less around the technology, but around the controls and what it is doing to the data. And I think that's something that, as Kiriaki was saying, we have to think about it now. The regulatory framework is really just starting to be developed in this area. The regulators are looking about what's the appropriate way to regulate AI usage. There are greater calls for more controls around the use of AI that we see, you know, in the public generally coming from our politicians around the world out of a concern about appropriate use. And appropriate use really comes down to the data. What is happening with it and how is it being managed? The more diligent you can be now, in terms of being able to explain that you have controls, that you know what those controls are, that you know what data is being used, that you know how it is being used, that you have these different controls in place to manage issues like privacy and all of those things, the better off you will be. we see this all the time with the regulators and with the courts. People get tagged the worst when they cannot explain what happened. They cannot explain what happened or why it happened, and they cannot explain the controls they had in place that would have helped to stop whatever the negative thing is that happened. When you are able to explain, because you have diligently documented your controls and your process and your management of your data, you are much less likely to face a severe result than if you say, I don't know. And I think that's something that's really important to keep in mind is that that is what they are looking for. They are looking for diligence. They are looking for a sincere effort to implement controls. They're looking for awareness around what data is being used and how it's being used with the AI tools so that you can explain and understand the outputs and the best way to use them. And the more that you have that in place, frankly, not only will you be able to explain your use, but it puts the company in the position for greater use, expanded use, more complex use, because you've already built that foundation that will support.

Anthony: Yeah. and I think it's only a matter of time where there's going to be some regulatory, some big fines, some big investigations that some regulator is going to do, whether it's on the privacy side, it could be financial services, it could be in the healthcare place where this is going to be exposed and everyone's going to be freaking out. But anyway, thank you all for for listening. Listen for some new podcasts. Like I said, we're gonna have on this series some more practical advice on sort of how to get the governance done, how to get data governance to really help your AI governance. So talk to you then. Thanks.

Outro: Tech Law Talks is a Reed Smith production. Our producers are Shannon Ryan, Amanda Saunders, and Mason Kautz. For more information about Reed Smith's Emerging Technologies Practice, please email [email protected]. You can find our podcast on all streaming platforms, reedsmith.com and our social media accounts at Reed Smith LLP.

Disclaimer: This podcast is provided for educational purposes. It does not constitute legal advice and is not intended to establish an attorney-client relationship, nor is it intended to suggest or establish standards of care applicable to particular lawyers in any given situation. Prior results do not guarantee a similar outcome. Any views, opinions, or comments made by any external guest speaker are not to be attributed to Reed Smith LLP or its individual lawyers. 

All rights reserved.

Transcript is auto-generated.