Secrets & Authentication¶

CONCEPT:AU-OS.config.secrets-authentication — Secrets & Authentication

This document covers how agent-utilities manages secrets, credentials, and authentication across the agent ecosystem.

Overview¶

The SecretsClient provides a unified, pluggable interface for storing and retrieving sensitive values (API keys, tokens, SSH credentials, etc.). It ships with two live backends and supports URI-style references for maximum flexibility (CONCEPT:AU-OS.identity.encrypted-secret-store).

┌─────────────────────────────┐
│        SecretsClient        │  ← High-level API
│  get_or_env() / resolve_ref │
└────────┬────────────────────┘
         │  (pluggable)
   ┌─────┴───────────────────┬──────────────┐
   │ InEpistemicGraphBackend │  Vault (hvac) │
   │ (engine-encrypted,      │  (enterprise) │
   │  default everywhere)    │               │
   └─────────────────────────┴──────────────┘

The default InEpistemicGraphBackend is a durable, engine-backed store: secrets live as :Secret nodes in a dedicated __secrets__ epistemic-graph graph, the secret value held as an encrypted node property sealed by the engine's encryption-at-rest (ChaCha20-Poly1305 over the redb value blobs, keyed by EPISTEMIC_GRAPH_ENCRYPTION_KEY + the KMS seam — CONCEPT:EG-KG.sharding.row-level-security), while the key name + metadata stay queryable plaintext. It is the store in every profile — even zero-infra tiny, because GraphComputeEngine auto-starts the pre-bundled pi-tier engine on demand (the OS-5.63 resolver). The master key now lives in EPISTEMIC_GRAPH_ENCRYPTION_KEY/KMS, no longer in a sibling .key file co-located with the ciphertext.

SQLiteBackend is retained ONLY as the read source for a one-time migration off the legacy ~/.agent-utilities/secrets.db (read-old → write-new → delete-old, run automatically on first engine-backed boot). It is never selected as a live backend.

Quick Start¶

from agent_utilities import create_secrets_client

# Zero-config (in-memory, encrypted)
client = create_secrets_client()

# Store and retrieve
client.set("gitlab/token", "glpat-xxx")
token = client.get("gitlab/token")  # "glpat-xxx"

# Fallback to environment variable
token = client.get_or_env("gitlab/token", "GITLAB_TOKEN")

# URI resolution
token = client.resolve_ref("vault://agents/mcp/gitlab/token")
token = client.resolve_ref("env://GITLAB_TOKEN")

Secret Manager CLI¶

agent-utilities provides a built-in CLI to easily populate and manage your local secrets (such as the SQLite database) before running your agent.

# Set a secret
secret-manager set gitlab/token glpat-my-token

# Set a secret explicitly using the sqlite backend (overrides env var)
secret-manager --backend sqlite set my-service/api-key 12345

# Retrieve a secret
secret-manager get gitlab/token

# List all stored keys
secret-manager list

# Delete a secret
secret-manager delete gitlab/token

Backends¶

Engine-encrypted `secrets` store (Default everywhere)¶

Zero config — works out of the box; durable across restart
Secrets are :Secret nodes in a dedicated __secrets__ engine graph
The secret value is an encrypted node property (engine encryption-at-rest, ChaCha20-Poly1305, keyed by EPISTEMIC_GRAPH_ENCRYPTION_KEY/KMS); the key name + metadata stay queryable plaintext
Works on every profile (the OS-5.63 resolver auto-starts the pi-tier engine for tiny); there is no local-disk / RAM fallback
One-time migration off the legacy ~/.agent-utilities/secrets.db runs on first engine-backed boot, then deletes the old db + .key file

HashiCorp Vault & OpenBao (Enterprise / Open Source)¶

Requires pip install agent-utilities[vault] (installs hvac)
Uses KV v2 secrets engine
Best for: production, multi-tenant, corporate deployments
OpenBao Support: OpenBao (an open-source fork of HashiCorp Vault initiated at Vault 1.14.7) is fully compatible out-of-the-box. Because OpenBao maintains complete API compatibility with HashiCorp Vault, the hvac Python client and all authentication methods (Static Token, AppRole, Kubernetes, OIDC/JWT) work seamlessly. No code or configuration changes are needed.

To configure your agent to use Vault or OpenBao, export:

export SECRETS_BACKEND=vault
export SECRETS_VAULT_URL=https://openbao.example.com  # Points directly to your OpenBao or Vault server
export SECRETS_VAULT_MOUNT=secret
export VAULT_TOKEN=hvs.xxx

Configuration¶

Environment Variable	Default	Description
`SECRETS_BACKEND`	`inmemory`	`inmemory` → the durable engine-encrypted `__secrets__` store (default everywhere); `vault` → enterprise OpenBao/Vault
`EPISTEMIC_GRAPH_ENCRYPTION_KEY`	(unset → encryption off)	Engine encryption-at-rest key material / KMS seam that seals the `__secrets__` value blobs (CONCEPT:EG-KG.sharding.row-level-security)
`SECRETS_VAULT_URL`	`http://127.0.0.1:8200`	Vault server URL
`SECRETS_VAULT_MOUNT`	`secret`	Vault KV v2 mount point

The inmemory backend id is retained for config compatibility but now resolves to the engine-backed __secrets__ store. The secret value is sealed by the engine (EPISTEMIC_GRAPH_ENCRYPTION_KEY), so the master key and the ciphertext no longer share a directory.

Two surfaces — `graph_secret` MCP tool + `/graph/secret` REST route¶

Secrets are reachable from the gateway (REST) and the MCP server, not just a Python import. The graph_secret MCP tool and the /graph/secret REST twin both dispatch into the one SecretsClient core. Actions: set / get / list / delete. Mutations (set / delete) are governed by the ActionPolicy gate (secret.set / secret.delete, approval_required); get / list are reads and not gated. list returns key names only — never values.

How to Load Secrets¶

Current: Environment Variables (Still Supported)¶

The existing pattern continues to work. All os.environ / .env files are loaded by python-dotenv at startup:

# .env
LLM_API_KEY=sk-xxx
GITLAB_TOKEN=glpat-xxx

New: SecretsClient (Recommended for Sensitive Values)¶

from agent_utilities import create_secrets_client

client = create_secrets_client()

# 1. Programmatic storage
client.set("gitlab/token", "glpat-xxx")

# 2. Retrieve with env-var fallback
token = client.get_or_env("gitlab/token", "GITLAB_TOKEN")

# 3. URI references (for config files)
token = client.resolve_ref("vault://agents/mcp/gitlab/token")
token = client.resolve_ref("env://GITLAB_TOKEN")
token = client.resolve_ref("sqlite://gitlab/token")

URI Schemes¶

Scheme	Example	Behavior
`vault://`	`vault://agents/mcp/github/token`	Looks up key in backend
`secret://`	`secret://path/to/secret`	Alias for vault://
`env://`	`env://GITLAB_TOKEN`	Reads `os.environ["GITLAB_TOKEN"]`
`sqlite://`	`sqlite://gitlab/token`	Looks up key in backend
(plain)	`gitlab/token`	Direct backend lookup

Migration Path¶

Phase 0 (current):  .env + os.environ → plaintext
Phase 1 (this PR):  SecretsClient with encrypted storage + env fallback
Phase 2 (planned):  JWT session tokens + OIDC delegation + per-user isolation

Integration with GraphDeps¶

The SecretsClient is available on GraphDeps.secrets_client during graph execution. Specialist nodes and MCP tools can resolve credentials from the execution context:

# In a graph step or tool
if ctx.deps.secrets_client:
    token = ctx.deps.secrets_client.get_or_env("gitlab/token", "GITLAB_TOKEN")

MCP Token Delegation (Existing)¶

The ecosystem already supports OAuth2 token delegation for MCP servers:

UserTokenMiddleware captures incoming Bearer tokens into thread-local storage
MCP servers like gitlab-api read the token via threading.local().user_token
The token is exchanged via RFC 8693 (Token Exchange) for a scoped service token

See middlewares.py and mcp_utilities.py for the full auth stack (--auth-type jwt|oidc-proxy|oauth-proxy|remote-oauth).

Endpoint Authentication (auth.py)¶

The agent server supports two authentication mechanisms that can be used independently or combined (logical OR):

API Key (Legacy)¶

Static shared secret via the X-API-Key header. Enable with:

export ENABLE_API_AUTH=true
export AGENT_API_KEY=your-secret-key

JWT Bearer Token (Recommended)¶

Validates tokens against a JWKS endpoint from any OIDC provider (Azure AD, Okta, Keycloak, Auth0, etc.). Requires pip install agent-utilities[auth].

export AUTH_JWT_JWKS_URI=https://login.microsoftonline.com/.../discovery/v2.0/keys
export AUTH_JWT_ISSUER=https://login.microsoftonline.com/.../v2.0
export AUTH_JWT_AUDIENCE=api://my-agent-api

The server will accept either a valid API key OR a valid JWT Bearer token. When no auth mechanism is configured, the server operates in open mode.

Config Variable	Description
`AUTH_JWT_JWKS_URI`	JWKS endpoint for token verification
`AUTH_JWT_ISSUER`	Expected `iss` claim
`AUTH_JWT_AUDIENCE`	Expected `aud` claim

OIDC Flows by Client Type¶

Interface	OAuth2 Flow	Library
WebUI (React/Next.js)	Authorization Code + PKCE	NextAuth.js v5 / Auth.js
CLI/TUI (Textual)	Device Authorization Grant	`authlib` + custom CLI flow
Service-to-Service	Client Credentials	`authlib`

CORS & Host Restriction¶

CORS and trusted host policies are configurable via environment variables:

# Restrict to specific origins
export ALLOWED_ORIGINS=https://app.example.com,https://admin.example.com

# Restrict trusted hosts
export ALLOWED_HOSTS=api.example.com,*.example.com

Both default to * (allow all) when unset. In production, always set explicit origins.

MCP Token Forwarding¶

When the agent server invokes MCP tools via subprocess (MCPToolset stdio transport), the user's session token is automatically forwarded:

The server stores the user's token in SecretsClient or AGENT_USER_TOKEN
mcp_utilities.py injects AGENT_USER_TOKEN into the subprocess env
MCP tools read os.environ["AGENT_USER_TOKEN"] for delegated auth

# In an MCP tool:
token = os.environ.get("AGENT_USER_TOKEN")
if token:
    headers["Authorization"] = f"Bearer {token}"

Security Best Practices¶

Never commit secrets to version control
Use short-lived tokens where possible (rotate every 30–90 days)
Audit logging: Every SecretsClient.get() / .set() call is logged at INFO level (key name only, never values)
Least privilege: Only request the scopes needed for the current graph node
Encrypted at rest: All backends encrypt values before storage
Key file permissions: SQLite backend creates .key files with 0o600
JWT over API keys: Prefer JWT Bearer auth for production — tokens expire, carry claims, and can be revoked at the IdP
Restrict CORS: Set ALLOWED_ORIGINS to specific trusted origins

Local Secret Storage (Vault, OpenBao, & SQLite)¶

The ecosystem provides a unified SecretsClient designed to replace static .env files, supporting inmemory, sqlite, and vault (HashiCorp Vault & OpenBao) backends.

Light Configuration Example (SQLite):

export SECRETS_BACKEND=sqlite
export SECRETS_SQLITE_PATH=~/.agent-utilities/secrets.db

Usage in Code & URI Schemes: Secrets can be resolved securely in Python via the context, or directly in mcp_config.json via URI schemes:

# Direct code resolution without os.environ
token = ctx.deps.secrets_client.get_or_env("gitlab/token", "GITLAB_TOKEN")

# URI Scheme support for configuration files
"env_vars": { "GITLAB_TOKEN": "secret://gitlab/token" }

Secret Manager CLI: Use the built-in CLI to easily populate your local database before running your agent:

secret-manager set gitlab/token glpat-xxx
secret-manager list

Native xAI OAuth Integration¶

agent-utilities supports native xAI OAuth 2.0 PKCE authentication to access the X / xAI API and search X posts or browse individual posts without hitting static API key limitations.

Architecture¶

The authentication flow utilizes the OAuth 2.0 Authorization Code Flow with Proof Key for Code Exchange (PKCE) (RFC 7636).

┌──────────────┐          1. Click link          ┌──────────────┐
│ Agent / CLI  ├────────────────────────────────►│ x.com Auth   │
│              │◄────────────────────────────────┤ Login Page   │
│ (Spin Server)│     2. Callback with Code       └──────┬───────┘
└──────┬───────┘    (or manual CLI input)               │
       │                                                │
       │ 3. Exchange Auth Code + Verifier               │
       ▼                                                │
┌──────────────┐                                        │
│  xAI OAuth   │◄───────────────────────────────────────┘
│  Token Endpt │
└──────┬───────┘
       │ 4. Store encrypted tokens in SecretsClient
       ▼
┌──────────────┐
│SecretsClient │
└──────────────┘

Flow Options¶

Auto-Callback Server: Launches a temporary local web server (defaults to http://localhost:8000) to catch the callback and automatically parse the authorization code.
Manual CLI Fallback: If a port is occupied or a server cannot be started, prints the authorization URL to the terminal and prompts the user to paste the callback URL or code directly.

Usage in Python¶

from agent_utilities.security.xai_auth import XaiAuthManager
from agent_utilities.secrets_client import create_secrets_client

secrets = create_secrets_client()
manager = XaiAuthManager(secrets_client=secrets)

# Perform authentication (launches loopback server or CLI paste fallback)
tokens = manager.login()
print("Access Token:", tokens.get("access_token"))

Auto Token Refresh¶

The XaiAuthManager automatically handles token expiration and token refresh using OAuth 2.0 refresh token rotation:

# Get a valid, fresh token (auto-refreshes if expired; pass auto_login=True
# to trigger the interactive flow when no cached tokens exist)
valid_token = manager.resolve_credentials(auto_login=True)

Loopback & Headless Authentication Support¶

How Loopback Works Remotely The OIDC callback server runs inside the workspace environment at http://127.0.0.1:56121/callback. Because graph-os runs as an MCP server, standard standard-input prompts (input()) cannot be used (since the IDE uses standard input/output for JSON-RPC communication, reading from stdin would hang the MCP server). Therefore, the callback server is the exclusive way to exchange tokens without crashing the MCP channel.
How to Authenticate in a Headless/Remote Environment To authenticate using your local browser while the MCP server runs on the remote container/VM:
Forward the Callback Port: Set up a local port forward for port 56121 in your IDE (or via SSH using ssh -L 56121:127.0.0.1:56121).
Authorize: Click the xAI auth link in your browser and log in.
Seamless Redirect: When the browser redirects to http://127.0.0.1:56121/callback, the traffic will be forwarded back to your remote workspace. The OIDC loopback server will instantly capture the authorization code, exchange it, and save the token securely—completing the setup automatically with zero manual copy-pasting!

Configuration¶

The OAuth client ID, issuer, scope, and loopback redirect URI are built-in constants in agent_utilities/security/xai_auth.py (XAI_OAUTH_CLIENT_ID, XAI_OAUTH_REDIRECT_PORT = 56121, XAI_OAUTH_REDIRECT_URI); the public PKCE client requires no client secret, so there are no XAI_CLIENT_ID / XAI_REDIRECT_URI environment variables to set for the auth flow.

For the X search tools (separate from the auth flow), the following environment variables are honored:

Environment Variable	Default	Description
`XAI_BASE_URL`	xAI API base URL	Override the xAI API endpoint used by the search tool
`XAI_SEARCH_MODEL`	built-in default	Model used for X search/browse

For more details on X search tools, see the Tools Guide.

Full Documentation: HashiCorp Vault & OpenBao setup, encryption details, and API references are covered in the sections above (see HashiCorp Vault & OpenBao and Local Secret Storage).

Secrets & Authentication¶

Overview¶

Quick Start¶

Secret Manager CLI¶

Backends¶

Engine-encrypted __secrets__ store (Default everywhere)¶

HashiCorp Vault & OpenBao (Enterprise / Open Source)¶

Configuration¶

Two surfaces — graph_secret MCP tool + /graph/secret REST route¶

How to Load Secrets¶

Current: Environment Variables (Still Supported)¶

New: SecretsClient (Recommended for Sensitive Values)¶

URI Schemes¶

Migration Path¶

Integration with GraphDeps¶

MCP Token Delegation (Existing)¶

Endpoint Authentication (auth.py)¶

API Key (Legacy)¶

JWT Bearer Token (Recommended)¶

OIDC Flows by Client Type¶

CORS & Host Restriction¶

MCP Token Forwarding¶

Security Best Practices¶

Local Secret Storage (Vault, OpenBao, & SQLite)¶

Native xAI OAuth Integration¶

Architecture¶

Flow Options¶

Usage in Python¶

Auto Token Refresh¶

Loopback & Headless Authentication Support¶

Configuration¶

Engine-encrypted `secrets` store (Default everywhere)¶

Two surfaces — `graph_secret` MCP tool + `/graph/secret` REST route¶