Blog

Navigating three privacy pitfalls of AI adoption

Trustworthy AI extends well beyond just privacy, spanning security and ethical considerations as well. But in order to get AI right, you need to first get data privacy right. Learn more about three common privacy pitfalls in AI adoption, and how you can avoid them

Bex Evans
Senior Product Marketing Manager
June 10, 2024

3 privacy pitfalls in AI adoption
Purpose limitation in the context of AI
Proportionality and balance in the context of AI
Business continuity and future-proofing data

Navigating responsible AI: A privacy professional's guide

Download our white paper to learn more about the role of privacy professionals in responsible AI

Download now

Gartner® predicts that “By 2026, more than 80% of enterprises will have used generative artificial intelligence (GenAI) application programming interfaces (APIs) or models, and/or deployed GenAI-enabled applications in production environments, up from less than 5% in 2023”¹

At the same time, according to Gartner Hype Cycle Methodology, “Interest wanes as experiments and implementations fail to deliver. Producers of the technology shake out or fail. Investments continue only if the surviving providers improve their products to the satisfaction of early adopters.”²

Research from Forrester identifies data privacy and security concerns as the top barrier to generative AI adoption. The promotion of trustworthy AI extends well beyond the privacy domain, spanning security and ethical considerations as well. But in order to get AI right, you first need to get data privacy right.

3 privacy pitfalls in AI adoption

That’s because AI is an amplifier of existing privacy gaps – a single misconfigured access point can get exponentially more problematic when its exposed to an AI system. When navigating the role of data privacy for AI systems, there are three privacy pitfalls to be aware of:

Purpose limitation: AI incentivizes secondary uses of data, which means using data for a different purpose than it was originally intended for.
Proportionality: AI requires robust datasets to ensure accuracy, fairness, and quality which may conflict with data minimization practices
Business continuity: Data comes in, but it doesn’t go out – if you don’t put the right data in then someday that system won’t be available for you.

Here, we’ll explore each pitfall in more detail and look at some common practices for addressing the risks associated with it.

Purpose limitation in the context of AI

A common scenario that illustrates the importance of responsible data usage is the collection of birth dates for identity verification. In contexts such as two-factor authentication or banking, verifying a person's identity is crucial. This process often involves collecting sensitive information – like their birth date.

However, possessing this data for identity verification does not automatically grant permission to use it for other purposes. For instance, if a marketing team wants to use birth dates to send out birthday promotions, they must first obtain explicit consent from the individuals involved. Without such consent, using birth dates for marketing is a violation of data privacy principles.

Challenges with generative AI and consent

The advent of large language models (LLMs) and generative AI adds another layer of complexity to this issue.

Providing clear and detailed information upfront is essential for obtaining informed consent, meaning individuals must understand in plain language how their data will be used to provide informed consent.

One significant challenge organizations face is striking a balance between offering enough context and avoiding overwhelming individuals with lengthy terms and conditions that they might – and often do – simply skip. Effective communication in the audience's language is vital to ensure that consent is truly informed and not just a formality.

Proportionality and balancing data robustness and minimization in the context of AI

Consider the use of a resume scanning tool designed to streamline hiring practices. Historically, organizations might exclude sensitive information such as race, gender, and ethnicity from resumes to minimize privacy risks and reduce overall risk if the application were to experience a breach. However, excluding these data points can also prevent the identification and mitigation of bias within the hiring process.

Bias can persist even when sensitive data is omitted, as other indirect factors might contribute to biased outcomes. To accurately analyze and ensure fair representation in a dataset, it’s necessary to document and record sensitive data points. This allows for proactive monitoring of the system for fairness and the detection of potential biases.

A common challenge faced by organizations today is that they may not have collected race, gender, or ethnicity information initially due to privacy concerns or the potential discomfort it might cause applicants. Consequently, they lack the data needed to perform thorough fairness assessments.

Privacy-enhancing technologies (PETs)

To address these challenges, privacy-enhancing technologies (PETs) can be employed. PETs such as differential privacy, synthetic data, homomorphic encryption, and multi-party computation help protect sensitive inputs while enabling the necessary analysis. These technologies allow for the safeguarding of individual privacy during data processing and model training.

However, it is important to recognize that there is no one-size-fits-all solution. The choice of PET depends on the specific use case and the infrastructure in place. In many scenarios, a combination of multiple PETs may be required to adequately protect privacy while maintaining the utility of the data.

Business continuity and future-proofing data in the context of AI

For consent to be truly lawful, it must be freely given and able to be withdrawn at any time. This principle poses a significant challenge when a consumer requests the deletion of their data from a business-critical system. If these processes rely on AI systems trained on personal data, the removal of such data can disrupt business continuity.

AI models, much like human brains, can’t simply forget information once it has been learned. The only solution is to roll back to a previous version of the model, trained before the data in question was included, and then retrain the model without it.

This necessitates robust documentation on model versioning, dataset versioning, and detailed tracking of data categories and identifiers to ensure the data can be accurately removed.

Data governance and model retraining

The complexities associated with enforcing data governance and model retraining highlight the importance of thorough documentation and precise version control. This involves maintaining detailed logs of model versions, datasets, and the identifiers used to track individual data points. When a data subject revokes consent, these records allow for the targeted rollback and retraining of models.

Retrieval-augmented generation (RAG)

Given these data governance challenges, there is a compelling case for employing retrieval-augmented generation (RAG). RAG involves retrieving facts from an external knowledge base to ground LLMs in the most accurate and up-to-date information. This approach offers several benefits:

Flexibility: External data sources used in RAG can be updated without the need to retrain the entire model.
Accuracy: By retrieving information from trusted sources, RAG enhances factual accuracy and significantly reduces the chances of AI-generated hallucinations.
Accessibility: RAG lowers the barrier to adopting customizable generative AI services as it doesn’t require advanced data science techniques.
Specificity: RAG is well-suited for retrieving specific information to provide accurate answers.
Cost: Using external knowledge as a prompt input is more cost-effective than fine-tuning the entire LLM.

By utilizing RAG, organizations can maintain control over the data input at the point of prompt rather than continuously retraining models. This approach helps ensure business continuity and compliance with data privacy regulations, even when individual data points are removed due to withdrawn consent.

¹ Gartner Press Release, “Gartner Says More Than 80% of Enterprises Will Have Used Generative AI APIs or Deployed Generative AI-Enabled Applications by 2026,” October 11, 2023. https://www.gartner.com/en/newsroom/press-releases/2023-10-11-gartner-says-more-than-80-percent-of-enterprises-will-have-used-generative-ai-apis-or-deployed-generative-ai-enabled-applications-by-2026

² Gartner Research Methodologies, Gartner Hype Cycle, June 4th, 2024, https://www.gartner.com/en/research/methodologies/gartner-hype-cycle

Blog

Navigating three privacy pitfalls of AI adoption

Table of contents

Navigating responsible AI: A privacy professional's guide

3 privacy pitfalls in AI adoption

Purpose limitation in the context of AI

Challenges with generative AI and consent

Proportionality and balancing data robustness and minimization in the context of AI

Privacy-enhancing technologies (PETs)

Business continuity and future-proofing data in the context of AI

Data governance and model retraining

Retrieval-augmented generation (RAG)

You may also like

AI Governance

Operationalizing the EU AI Act

In this webinar, we’ll explore how OneTrust helps organizations meet EU AI Act compliance by operationalizing AI governance frameworks.

January 28, 2025

AI Governance

Automating metadata capture: Future-proofing data management for AI

This webinar will explore how automating metadata capture can streamline the management of unstructured data, making it AI-ready while ensuring data quality and security.

January 14, 2025

AI Governance

Overcoming the privacy pitfalls of GenAI (APAC)

Join us and learn about the data privacy risks of adopting GenAI and practical strategies on avoiding them.

January 08, 2025

AI Governance

ISMG hosted panel discussion: Generative AI in 2024 – Navigating business rewards and security risks

In this recorded panel discussion, experts from OneTrust, Forcepoint, Optiv, and Protiviti explore findings from ISMG’s recent global survey of over 400 business and cybersecurity professionals.

December 19, 2024

Responsible AI

Overcoming the privacy pitfalls of GenAI

This webinar will explore the key privacy pitfalls organizations face when implementing GenAI, focusing on purpose limitation, data proportionality, and business continuity.

December 10, 2024

Data Discovery & Security

The new data landscape: Navigating the shift to AI-ready data

This webinar will explore the how AI is affecting the data landscape, focusing on how data teams can extend common data practices to support AI’s unique use of data.

November 12, 2024

AI Governance

How to build an AI project intake workflow that balances risk and efficiency

Download our guide to building an AI project intake workflow that balances risk and efficiency, complete with a checklist for thorough, informed assessments.

November 06, 2024

AI Governance

AI project intake workflow checklist

Download our AI Project Intake Checklist to guide thorough assessments and ensure secure, compliant, and effective AI project planning from start to finish.

November 06, 2024

Navigating the top 5 data sharing challenges

This webinar will uncover the top 5 data sharing challenges organizations face and demonstrate how advanced data governance solutions can streamline processes, improve data quality, and enhance compliance, allowing organizations to discover the full potential of their data assets.

October 31, 2024

AI Governance

How the EU AI Act and recent FTC enforcements for AI shape data governance

Download this white paper to learn how to adapt your data governance program, by defining AI-specific policies, monitoring data usage, and centralizing enforcement.

October 30, 2024

Responsible AI

Getting Ready for the EU AI Act, Phase 1: Discover & Catalog, The Gartner® Report​

Getting Ready for the EU AI Act, Phase 1: Discover & Catalog, The Gartner® Report​

October 16, 2024

AI Governance

California's approach to AI: Unpacking new legislation

This webinar unpacks California’s approach to AI and emerging legislations, including legislation on defining AI, AI transparency disclosures, the use of deepfakes, generative AI, and AI models.

October 15, 2024

AI Governance

Establishing a scalable AI governance framework: Key steps and tech for success

Download this coauthored eBook by OneTrust and Protiviti to learn how organizations are building scalable AI governance models and managing AI risks.

October 01, 2024

Privacy Automation

Discover the economic benefits of OneTrust

Download this 2024 Forrester Consulting Total Economic Impact™ study to see how OneTrust has helped organizations navigate data management complexities, generate significant ROI, and enable the responsible use of data and AI.

September 24, 2024

Privacy Automation

Global regulatory update: Recent privacy developments and compliance trends

Join us for a webinar on the latest updates and emerging trends in global privacy regulations.

September 12, 2024

AI Governance

Securing reliable AI solutions: Strategies for trustworthy procurement

Download this eBook to explore strategies for trustworthy AI procurement and learn how to evaluate vendors, manage risks, and ensure transparency in AI adoption.

September 12, 2024

AI Governance

Data and AI governance for responsible use of data

Learn why discovering, classifying, and using data responsibly is the only way to ensure your AI is governed properly.

September 12, 2024

Getting Ready for the EU AI Act, Phase 1: Discover & Catalog, The Gartner® Report

Getting Ready for the EU AI Act, Phase 1: Discover & Catalog, The Gartner® Report

Scaling to new heights with AI Governance