Securely storing, rotating, scoping, and monitoring the API keys and secrets for LLM providers and vector/data services — where a leaked key means runaway spend and data exposure, not just an outage.
How it works
To manage API keys and secrets securely, you start by storing them in a Secrets manager like HashiCorp Vault or AWS Secrets Manager. This approach prevents hardcoding sensitive information directly into your code or client bundles, which is a major security risk. The secrets manager keeps your keys safe and allows you to implement features like rotation and scoping. For example, you can set up project-specific keys that limit access to only the necessary resources, ensuring that even if one key is compromised, the damage is contained.
Next, you route all API calls through a Gateway or proxy, such as LiteLLM or Cloudflare AI Gateway. This setup ensures that the keys remain on the server side, away from client exposure. The gateway can also manage virtual keys for different teams, applying rate limits and logging usage. This means you can monitor how much each team is spending and prevent runaway costs. For instance, if a team exceeds their budget, the gateway can automatically throttle their access, protecting your overall spending.
you can implement per-key spend limits and budget alerts. These features notify you when a key approaches its spending cap, allowing you to take action before costs spiral out of control. The rotation of keys is also crucial; it ensures that even if a key is leaked, its lifespan is limited. By regularly rotating keys, you minimize the risk of long-term exploitation, creating a more secure environment for your applications.
- Store keys in a secrets manager.
- Use scoped keys for projects.
- Rotate keys regularly.
- Route API calls through a gateway.
- Monitor usage and set alerts.
The software-engineering analogue
This concept parallels standard secrets management practices in software engineering, where sensitive information is stored securely and accessed when needed. However, the key difference here is that these API keys directly correlate to financial costs. A leak doesn't just lead to unauthorized access; it can result in real-time financial exploitation. For example, if a malicious actor gains access to a key, they can make unlimited API calls, leading to significant unexpected charges.
the rise of AI introduces new attack vectors, such as prompt injection, where an agent might attempt to exfiltrate keys through cleverly crafted inputs. This adds a layer of complexity to the security model, as traditional secrets management strategies may not account for these AI-specific threats.
Why not the obvious alternative
A common initial approach might involve hardcoding a single shared key in an environment variable. This method seems straightforward but comes with severe limitations. Without the ability to rotate keys without redeployment, you face a scenario where any leak—such as a committed .env file or a client-side API call—can drain your account. There’s no per-team attribution, no spend cap, and no way to contain the fallout from a compromised key.
In contrast, using a gateway with scoped keys mitigates these risks. It allows for individual team management, enabling you to set specific limits and monitor usage effectively. This containment strategy not only enhances security but also provides financial oversight, which is critical in environments where costs can escalate quickly.
When to use which
As a baseline, always implement a secrets manager for storing API keys and secrets securely. This is essential for any application that interacts with external services. Once your architecture expands to include multiple teams or applications, or if you require per-tenant cost attribution, it’s time to add an LLM gateway. This gateway will help you manage the complexity of multiple access points and ensure that your spending is controlled and monitored. Importantly, avoid calling provider APIs directly from browsers or mobile clients to prevent exposing your keys.
In production: when it breaks
Why it happens. The sudden cost increase may result from an exposed or leaked API key, which is being used by unauthorized parties to make excessive requests. This can happen if the key is hardcoded in public repositories or shared in unsecured environments, allowing external actors to exploit it.