Detecting and masking personal or secret data in prompts (and outputs) so it isn't sent to an external LLM, logged in plaintext, or leaked back to the wrong user — data-loss prevention adapted to the prompt path.
How it works
A detection layer is crucial for identifying sensitive information in prompts and outputs. Tools like Microsoft Presidio and AWS Comprehend, or even custom solutions using Regex and Named Entity Recognition (NER), scan the text to tag entities such as names, social security numbers, credit card details, API keys, and health data. This tagging process allows the system to recognize what needs to be redacted or masked before any data leaves your controlled environment. Once the sensitive information is identified, the next step is tokenization or pseudonymization. For example, the name John Smith can be replaced with a token like [PERSON_1]. This step ensures that the model can still function without exposing sensitive data, as the real values can be re-hydrated in the response after processing.
In addition to scanning prompts before they reach the model, the system also scans the outbound output. This is important because the model might inadvertently echo back sensitive information it has learned during training or leak context from another tenant's data. By implementing a two-way scanning mechanism, you ensure that both incoming prompts and outgoing responses are free from sensitive information. This comprehensive approach to data loss prevention (DLP) is essential for maintaining compliance and protecting user privacy.
For example, consider a healthcare application that sends patient data to a hosted language model for analysis. The detection layer identifies patient names and health conditions, replacing them with tokens before sending the request. After the model generates a response, the system replaces the tokens with the actual patient data for internal use, ensuring that no sensitive information was exposed during the process. Without this mechanism, you risk exposing personally identifiable information (PII) to external systems, leading to potential breaches and compliance violations.
- Detect PII in user input.
- Tokenize sensitive data before sending.
- Rehydrate PII in the output.
- Scan model output for leaks.
- Use regex or NER for detection.
The software-engineering analogue
This process is similar to log scrubbing and field-level encryption used in traditional software engineering. When data crosses a trust boundary, like sending it to a third-party service, organizations often scrub logs and encrypt sensitive fields to prevent unauthorized access. However, the key difference here is that the payload being handled is unstructured free text. This means that the detection of sensitive information relies on probabilistic NER methods rather than a straightforward column-based mask. As a result, the recall rate for detecting PII is never going to be 100%, which adds complexity to the implementation.
Why not the obvious alternative
A common misconception is that you can simply trust the provider's no-training policy for handling sensitive data. This approach overlooks critical issues like data residency, subprocessor exposure, and the potential for your own logs to contain sensitive information. By relying solely on the provider's assurances, you expose yourself to risks that could lead to data breaches or compliance failures.
Tokenization provides a more strong solution by ensuring that raw PII remains within your control, regardless of what the provider does. This method aligns with the requirements set by auditors and regulations like GDPR, which mandate that sensitive data is not exposed during processing. By keeping the PII tokenized, you create a barrier that protects against accidental leaks and unauthorized access.
When to use which
When dealing with regulated data such as health records, financial information, or personal data from EU residents, it is essential to implement redaction and tokenization when sending data to a hosted API. This ensures compliance and protects sensitive information. Conversely, if you are using a self-hosted model within your Virtual Private Cloud (VPC), while DLP still matters for logs and preventing cross-tenant leakage, the urgency for egress protection is somewhat reduced since the data remains within your controlled environment.
In production: when it breaks
Why it happens. PII redaction failures can occur due to misconfigured regex patterns or incomplete coverage of data types. For instance, if the regex used to detect SSNs is too permissive or not applied consistently across all input and output paths, it may allow sensitive data to slip through the redaction process.
- Step 1: Review the current regex patterns used for PII detection to ensure they are comprehensive and correctly configured.
- Step 2: Check the logs for instances of unmasked PII to identify the specific patterns that failed to trigger redaction.
- Step 3: Conduct an A/B test with different regex configurations to measure the effectiveness of PII detection in a controlled environment.
What it delivers
By implementing effective PII redaction and prompt DLP, you can take advantage of best-in-class hosted models without the risk of shipping regulated data outside your control. This not only enhances your security posture but also provides a paper trail that demonstrates compliance with data protection regulations, giving you peace of mind and protecting your organization from potential liabilities.