diff --git a/AGENTS.md b/AGENTS.md index 1de238e..1cedd15 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,20 +9,24 @@ AOC is a FastAPI microservice that ingests Microsoft Entra (Azure AD) audit logs - **Runtime**: Python 3.11 (3.14 for tests) - **Web Framework**: FastAPI + Uvicorn (Gunicorn in production) - **Database**: MongoDB (PyMongo) +- **Cache/Queue**: Valkey/Redis 8 (caching + arq async job queue) - **Frontend**: Alpine.js + HTML/CSS (served as static files from `backend/frontend/`) - **Authentication**: Optional OIDC Bearer token validation against Microsoft Entra (using `python-jose` and MSAL.js on the frontend) - **External APIs**: Microsoft Graph API, Office 365 Management Activity API, Azure OpenAI / MS Foundry - **Deployment**: Docker Compose (dev), Docker Compose + nginx (prod) - **CI/CD**: Gitea Actions (lint + test + Docker build + release) +- **Secrets Storage**: Environment variables (`.env`) or optional Azure Key Vault ## Project Structure ``` backend/ main.py # FastAPI app, router registration, background periodic fetch - config.py # Pydantic Settings configuration (loads .env) + config.py # Pydantic Settings configuration (loads .env + optional Key Vault) database.py # MongoClient setup (db = micro_soc, collection = events) auth.py # OIDC Bearer token validation, JWKS caching, role/group checks + secrets_manager.py # Optional Azure Key Vault integration for secrets + rate_limiter.py # Redis-backed fixed-window rate limiter (fail-closed) requirements.txt # Python dependencies Dockerfile # python:3.11-slim image, non-root user, version baked at build mcp_server.py # Standalone MCP server for Claude Desktop / Cursor integration @@ -34,6 +38,9 @@ backend/ health.py # GET /health, GET /metrics rules.py # Rule-based alerting endpoints webhooks.py # Microsoft Graph change notification webhooks + alerts.py # Alert management endpoints + saved_searches.py # Saved filter presets + jobs.py # Async job status polling graph/ auth.py # Client credentials token acquisition for Graph audit_logs.py # Fetch and enrich directory audit logs from Graph @@ -59,16 +66,42 @@ Copy `.env.example` to `.env` at the repo root and fill in values: cp .env.example .env ``` -Key variables: +### Core variables - `TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET` — Microsoft app registration credentials (application permissions) - `AUTH_ENABLED` — set `true` to protect API/UI with OIDC Bearer tokens - `AUTH_TENANT_ID`, `AUTH_CLIENT_ID` — token validation audience/issuer - `AUTH_ALLOWED_ROLES`, `AUTH_ALLOWED_GROUPS` — comma-separated access control lists - `ENABLE_PERIODIC_FETCH`, `FETCH_INTERVAL_MINUTES` — background ingestion scheduler - `MONGO_ROOT_USERNAME`, `MONGO_ROOT_PASSWORD`, `MONGO_PORT` — used by Docker Compose for MongoDB + +### AI / LLM variables - `AI_FEATURES_ENABLED` — set `false` to completely disable AI endpoints and UI (default `true`) - `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, `LLM_MAX_EVENTS`, `LLM_TIMEOUT_SECONDS` — LLM provider settings - `LLM_API_VERSION` — required for Azure OpenAI / MS Foundry endpoints +- `LLM_ALLOWED_DOMAINS` — comma-separated domain allowlist for LLM endpoints (e.g. `api.openai.com,*.openai.azure.com`) + +### Security variables +- `CORS_ORIGINS` — comma-separated allowed origins (default `*`; set explicit origins in production) +- `DOCS_ENABLED` — set `true` to expose `/docs`, `/redoc`, `/openapi.json` (default `false`) +- `METRICS_ALLOWED_IPS` — comma-separated CIDRs allowed to access `/metrics` (default: private networks + loopback) +- `WEBHOOK_CLIENT_SECRET` — secret for validating Graph webhook `clientState` +- `SIEM_ENABLED`, `SIEM_WEBHOOK_URL` — optional SIEM forwarding +- `SIEM_ALLOWED_DOMAINS` — comma-separated domain allowlist for SIEM webhook URLs +- `RATE_LIMIT_ENABLED`, `RATE_LIMIT_REQUESTS`, `RATE_LIMIT_WINDOW_SECONDS` — Redis-backed rate limiting + +### Optional Azure Key Vault +- `AZURE_KEY_VAULT_NAME` — name of the Azure Key Vault to load secrets from +- When set, AOC fetches these secrets at startup: + - `aoc-client-secret` → `CLIENT_SECRET` + - `aoc-llm-api-key` → `LLM_API_KEY` + - `aoc-mongo-uri` → `MONGO_URI` + - `aoc-webhook-client-secret` → `WEBHOOK_CLIENT_SECRET` +- Requires `azure-identity` and `azure-keyvault-secrets` (uncomment in `requirements.txt`) + +### Privacy / access control +- `PRIVACY_SERVICES` — comma-separated services to hide from non-privileged users (e.g. `Exchange,Teams`) +- `PRIVACY_SENSITIVE_OPERATIONS` — comma-separated operations to gate +- `PRIVACY_SERVICE_ROLES` — comma-separated Entra roles that grant access to privacy data ## Build and Run Commands @@ -102,7 +135,9 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000 - `GET /api/config/features` — feature flags (`ai_features_enabled`) - `POST /api/ask` — natural language query; returns LLM narrative + referenced events (only when `AI_FEATURES_ENABLED=true`) - `GET /health` — liveness probe with DB connectivity -- `GET /metrics` — Prometheus metrics +- `GET /metrics` — Prometheus metrics (IP-restricted by default) +- `GET /api/source-health` — last fetch status per ingestion source +- `GET /api/version` — running version ## MCP Server @@ -162,16 +197,30 @@ When adding new features or bug fixes, add or update tests in `backend/tests/`. - Auth middleware and token validation - API endpoints (`/api/events`, `/api/fetch-audit-logs`, `/api/ask`) - NLQ time range extraction, entity extraction, query building +- Rate limiting behavior ## Security Considerations -- **Secrets**: `CLIENT_SECRET`, `LLM_API_KEY`, and other credentials come from `.env`. Never commit `.env`. -- **Auth validation**: When `AUTH_ENABLED=true`, the backend fetches JWKS from `https://login.microsoftonline.com/{AUTH_TENANT_ID}/v2.0/.well-known/openid-configuration`, caches keys for 1 hour, and validates tenant/issuer claims. Tokens are decoded without strict signature verification (`jwt.get_unverified_claims`), so the tenant and issuer checks are the primary gate. -- **Role/Group gating**: Access is allowed if the token’s `roles` intersect `AUTH_ALLOWED_ROLES` or `groups` intersect `AUTH_ALLOWED_GROUPS`. If neither list is configured, all authenticated users are allowed. +- **Secrets**: `CLIENT_SECRET`, `LLM_API_KEY`, and other credentials come from `.env` or Azure Key Vault. Never commit `.env`. +- **Auth validation**: When `AUTH_ENABLED=true`, the backend fetches JWKS from `https://login.microsoftonline.com/{AUTH_TENANT_ID}/v2.0/.well-known/openid-configuration`, caches keys for 1 hour, and validates tenant/issuer/audience claims. Tokens are decoded with RS256 signature verification. +- **Role/Group gating**: Access is allowed if the token's `roles` intersect `AUTH_ALLOWED_ROLES` or `groups` intersect `AUTH_ALLOWED_GROUPS`. If neither list is configured, all authenticated users are allowed — a startup warning is logged in this case. +- **CORS**: When `AUTH_ENABLED=true` and `CORS_ORIGINS="*"`, `allow_credentials` is forced to `false` to prevent cross-origin token leakage. +- **Rate limiting**: Redis-backed fixed-window rate limiting with per-category limits (fetch=10/hr, ask=30/min, write=20/min, default=120/min). Fails closed (returns 429) when Redis is unavailable. - **Pagination limits**: `page_size` is clamped to a maximum of 500 to prevent large queries. - **Fetch window cap**: `hours` is clamped to 720 (30 days) to avoid runaway API calls. +- **LLM SSRF guard**: `LLM_BASE_URL` must be HTTPS and cannot point to private IPs. Optional `LLM_ALLOWED_DOMAINS` restricts to specific domains. +- **SIEM SSRF guard**: `SIEM_WEBHOOK_URL` has the same validation as LLM URLs, plus optional `SIEM_ALLOWED_DOMAINS`. +- **Metrics IP gating**: `/metrics` is restricted to private/loopback IPs by default via `METRICS_ALLOWED_IPS`. +- **OpenAPI docs**: Disabled by default (`DOCS_ENABLED=false`). Enable only in development. +- **CSP**: Content-Security-Policy headers are set on all responses. `unsafe-eval` is required for Alpine.js v3 expression evaluation. +- **SRI**: CDN scripts (Alpine.js, MSAL.js) include Subresource Integrity hashes to prevent supply chain compromise. - **MCP server**: The MCP server bypasses auth entirely. Only run it in trusted environments or behind a VPN. +### Security Documentation + +- `PEN_TEST_REPORT_v1.7.11.md` — Internal soft penetration test findings and remediation +- `THREAT_MODEL_v1.7.13.md` — Comprehensive threat model covering Entra/token abuse vectors + ## Maintenance and Operations The `backend/maintenance.py` script provides two CLI commands useful for backfilling or correcting stored data: diff --git a/DEPLOY.md b/DEPLOY.md index cf2fd0d..5000691 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -7,6 +7,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose: - **nginx** — reverse proxy, TLS termination, static file serving - **backend** — FastAPI application (Gunicorn + Uvicorn workers) - **mongo** — MongoDB data store (not exposed externally) +- **valkey** — Redis-compatible cache and async job queue (not exposed externally) ## Prerequisites @@ -20,7 +21,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose: 1. **Clone / pull the latest release** ```bash - git checkout v1.1.0 + git checkout v1.7.14 ``` 2. **Copy and edit environment variables** @@ -33,7 +34,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose: 3. **Set the release version** ```bash - export AOC_VERSION=v1.1.0 + export AOC_VERSION=v1.7.14 ``` 4. **Deploy** @@ -53,7 +54,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose: ## Updating to a new release ```bash -export AOC_VERSION=v1.2.0 +export AOC_VERSION=v1.7.14 docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d ``` @@ -75,24 +76,56 @@ docker compose -f docker-compose.prod.yml up -d Replace the `nginx` service in `docker-compose.prod.yml` with a Certbot-friendly setup (e.g., use the `nginx-proxy` + `acme-companion` stack) or mount the Certbot certificates into `nginx/ssl/`. -## Security hardening +## Security Hardening - MongoDB is **not exposed** to the host — only the backend container can reach it. +- Valkey/Redis is **not exposed** to the host — only the backend container can reach it. - The backend runs as a non-root (`aoc`) user inside the container. - nginx adds security headers (`X-Frame-Options`, `X-Content-Type-Options`, etc.). - Keep `.env` out of version control — it is listed in `.gitignore`. +- Set `AUTH_ENABLED=true` and configure `AUTH_ALLOWED_ROLES` or `AUTH_ALLOWED_GROUPS` to restrict access to admin/security roles. +- Set explicit `CORS_ORIGINS` — do not use `*` in production when auth is enabled. +- Set `DOCS_ENABLED=false` to hide OpenAPI docs (`/docs`, `/openapi.json`). +- Configure `WEBHOOK_CLIENT_SECRET` to validate Graph webhook notifications. +- Set `LLM_ALLOWED_DOMAINS` if using AI features (e.g. `api.openai.com,*.openai.azure.com`). +- Set `SIEM_ALLOWED_DOMAINS` if using SIEM forwarding. +- Review `METRICS_ALLOWED_IPS` — defaults to private networks + loopback. + +## Azure Key Vault (Optional) + +To eliminate long-lived secrets from `.env`: + +1. Create an Azure Key Vault and add these secrets: + - `aoc-client-secret` — your Graph app `CLIENT_SECRET` + - `aoc-llm-api-key` — your `LLM_API_KEY` (if using AI) + - `aoc-mongo-uri` — your `MONGO_URI` + - `aoc-webhook-client-secret` — your `WEBHOOK_CLIENT_SECRET` + +2. Uncomment `azure-identity` and `azure-keyvault-secrets` in `backend/requirements.txt` + +3. Set `AZURE_KEY_VAULT_NAME=your-keyvault-name` in `.env` + +4. Grant the container identity `Get` permission on secrets: + - If using Azure Container Instances / AKS: assign a managed identity + - If using VM: assign a managed identity or use a service principal + - If using local Docker: authenticate via `az login` on the host + +5. Rebuild and redeploy: + ```bash + docker compose -f docker-compose.prod.yml up -d --build + ``` ## Rollback ```bash -export AOC_VERSION=v1.0.3 +export AOC_VERSION=v1.7.13 docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d ``` ## Monitoring -- Prometheus metrics: `http://your-host/metrics` +- Prometheus metrics: `http://your-host/metrics` (IP-restricted by default) - Health check: `http://your-host/health` - Container logs: @@ -100,4 +133,13 @@ docker compose -f docker-compose.prod.yml up -d docker compose -f docker-compose.prod.yml logs -f backend docker compose -f docker-compose.prod.yml logs -f nginx docker compose -f docker-compose.prod.yml logs -f mongo + docker compose -f docker-compose.prod.yml logs -f valkey ``` + +## Troubleshooting + +- **Auth warning in logs**: "AUTH_ENABLED is true but no AUTH_ALLOWED_ROLES or AUTH_ALLOWED_GROUPS are configured" — set these to restrict access. +- **CORS issues**: Set `CORS_ORIGINS` to your exact frontend origin(s). Wildcard with auth enabled disables credentials. +- **Rate limiting 429s**: Check Redis/Valkey connectivity. The rate limiter fails closed (returns 429) when Redis is down. +- **LLM errors**: Verify `LLM_BASE_URL` is in `LLM_ALLOWED_DOMAINS` if the allowlist is configured. +- **SIEM not forwarding**: Verify `SIEM_WEBHOOK_URL` uses HTTPS and is in `SIEM_ALLOWED_DOMAINS`. diff --git a/README.md b/README.md index fa9c370..6a89f02 100644 --- a/README.md +++ b/README.md @@ -11,13 +11,14 @@ FastAPI microservice that ingests Microsoft Entra (Azure AD) and other admin aud - Optional OIDC bearer auth (Entra) to protect the API/UI and gate access by roles/groups. - Natural language query (`/api/ask`) powered by LLM (OpenAI, Azure OpenAI, or any compatible API). - MCP server for Claude Desktop / Cursor integration. +- Optional Azure Key Vault integration for secrets storage. ## Prerequisites (macOS) - Python 3.11 - Docker Desktop (for the quickest start) or a local MongoDB instance - An Entra app registration with **Application** permission `AuditLog.Read.All` and admin consent granted - Also required to fetch other sources: - - `https://manage.office.com/.default` (Audit API) with `ActivityFeed.Read`/`ActivityFeed.ReadDlp` (built into the app registration’s API permissions for Office 365 Management APIs) + - `https://manage.office.com/.default` (Audit API) with `ActivityFeed.Read`/`ActivityFeed.ReadDlp` (built into the app registration's API permissions for Office 365 Management APIs) - Intune audit: `DeviceManagementConfiguration.Read.All` (or broader) for `/deviceManagement/auditEvents` - Optional API protection: configure `AUTH_ENABLED=true` and set `AUTH_TENANT_ID`/`AUTH_CLIENT_ID` (the audience) plus allowed roles/groups. @@ -49,8 +50,43 @@ cp .env.example .env # LLM_BASE_URL=https://api.openai.com/v1 # LLM_MODEL=gpt-4o-mini # LLM_TIMEOUT_SECONDS=30 +# LLM_ALLOWED_DOMAINS=api.openai.com,*.openai.azure.com + +# Optional: SIEM forwarding +# SIEM_ENABLED=true +# SIEM_WEBHOOK_URL=https://your-siem.com/webhook +# SIEM_ALLOWED_DOMAINS=your-siem.com + +# Optional: Azure Key Vault for secrets storage +# AZURE_KEY_VAULT_NAME=your-keyvault-name ``` +### Using Azure Key Vault for secrets +Instead of storing `CLIENT_SECRET`, `LLM_API_KEY`, `MONGO_URI`, and `WEBHOOK_CLIENT_SECRET` in `.env`, you can store them in Azure Key Vault: + +1. Create a Key Vault and add secrets with these names: + - `aoc-client-secret` → your Graph app `CLIENT_SECRET` + - `aoc-llm-api-key` → your `LLM_API_KEY` + - `aoc-mongo-uri` → your `MONGO_URI` + - `aoc-webhook-client-secret` → your `WEBHOOK_CLIENT_SECRET` +2. Uncomment `azure-identity` and `azure-keyvault-secrets` in `backend/requirements.txt` +3. Set `AZURE_KEY_VAULT_NAME=your-keyvault-name` in `.env` +4. Ensure the container has Azure identity credentials (managed identity, service principal, or Azure CLI auth) + +## Security Hardening Checklist + +Before deploying to production: + +- [ ] Set `AUTH_ENABLED=true` and configure `AUTH_ALLOWED_ROLES` or `AUTH_ALLOWED_GROUPS` to restrict access +- [ ] Set explicit `CORS_ORIGINS` (do not use `*` in production with auth enabled) +- [ ] Set `DOCS_ENABLED=false` (default) to hide OpenAPI docs +- [ ] Configure `WEBHOOK_CLIENT_SECRET` to validate Graph webhook notifications +- [ ] Set `LLM_ALLOWED_DOMAINS` if using AI features to prevent data exfiltration +- [ ] Set `SIEM_ALLOWED_DOMAINS` if using SIEM forwarding +- [ ] Review `METRICS_ALLOWED_IPS` — defaults to private networks only +- [ ] Consider Azure Key Vault instead of `.env` for secrets +- [ ] Review the threat model: `THREAT_MODEL_v1.7.13.md` + ## Run with Docker Compose (recommended) ```bash docker compose up --build @@ -76,7 +112,7 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000 ## API - `GET /health` — health check with MongoDB connectivity status. -- `GET /metrics` — Prometheus metrics for request latency, fetch volume, and errors. +- `GET /metrics` — Prometheus metrics for request latency, fetch volume, and errors (IP-restricted). - `GET /api/version` — running version (baked into the Docker image at build time). - `GET /api/fetch-audit-logs` — pulls the last 7 days by default (override with `?hours=N`, capped to 30 days) of: - Entra directory audit logs (`/auditLogs/directoryAudits`) @@ -171,7 +207,7 @@ curl http://localhost:8000/api/fetch-audit-logs - Visit the UI at http://localhost:8000 to filter by user/service/action/result/time, search raw text, paginate, and view raw events. ## Maintenance (Dockerized) -Use the backend image so you don’t need a local venv: +Use the backend image so you don't need a local venv: ```bash # ensure Mongo + backend network are up docker compose up -d mongo @@ -182,10 +218,15 @@ docker compose run --rm backend python maintenance.py dedupe ``` Omit `--limit` to process all events. You can also run commands inside a running backend container with `docker compose exec backend ...`. +## Security Documentation +- `PEN_TEST_REPORT_v1.7.11.md` — Penetration test findings and remediation +- `THREAT_MODEL_v1.7.13.md` — Comprehensive threat model covering Entra application abuse, token handling, data exfiltration vectors + ## Notes / Troubleshooting - Ensure `TENANT_ID`, `CLIENT_ID`, and `CLIENT_SECRET` match an app registration with `AuditLog.Read.All` (application) permission and admin consent. - Additional permissions: Office 365 Management Activity (`ActivityFeed.Read`), and Intune audit (`DeviceManagementConfiguration.Read.All`). -- Auth: if `AUTH_ENABLED=true`, issued tokens must be from `AUTH_TENANT_ID`, audience = `AUTH_CLIENT_ID`; access is granted if roles or groups overlap `AUTH_ALLOWED_ROLES`/`AUTH_ALLOWED_GROUPS` (if set). +- Auth: if `AUTH_ENABLED=true`, issued tokens must be from `AUTH_TENANT_ID`, audience = `AUTH_CLIENT_ID`; access is granted if roles or groups overlap `AUTH_ALLOWED_ROLES`/`AUTH_ALLOWED_GROUPS` (if set). A startup warning is logged if auth is enabled but no roles/groups are configured. - Backfill limits: Management Activity API typically exposes ~7 days of history via API (longer if your tenant has extended/Advanced Audit retention). Directory/Intune audit retention follows your tenant policy (commonly 30–90 days, longer with Advanced Audit). - If you change Mongo credentials/ports, update `MONGO_URI` in `.env` (Docker Compose passes it through to the backend). - The service uses the `micro_soc` database and `events` collection by default; adjust in `backend/config.py` if needed. +- If using Azure Key Vault, ensure the runtime identity (managed identity, service principal, or local Azure CLI) has `Get` permission on secrets. diff --git a/ROADMAP.md b/ROADMAP.md index 85c7d4a..4878fb2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -59,7 +59,7 @@ Goal: evolve from a polling dashboard into a full security operations tool. --- -## Phase 5: Intelligence +## Phase 5: Intelligence ✅ Goal: add AI-powered analysis and external tool integration. - [x] AI feature flag (`AI_FEATURES_ENABLED`) to gate LLM-dependent features @@ -76,7 +76,26 @@ UI polish (topbar, footer, clickable pills) in v1.6.1–v1.6.4. --- -## Phase 6: Multi-Tenancy (Premium) ⏸️ +## Phase 6: Security Hardening ✅ +Goal: address penetration test findings and threat model gaps. + +- [x] Fix CORS credentials leak (v1.7.12) +- [x] Add security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy) (v1.7.12) +- [x] Make rate limiter fail-closed on Redis failure (v1.7.12) +- [x] Disable OpenAPI docs by default (v1.7.12) +- [x] Hide tenant_id/client_id from config endpoint when auth disabled (v1.7.12) +- [x] Validate webhook validationToken before echo (v1.7.12) +- [x] Gate `/metrics` behind IP allowlist (v1.7.12) +- [x] Add LLM domain allowlist (`LLM_ALLOWED_DOMAINS`) (v1.7.14) +- [x] Add SIEM webhook SSRF guard + domain allowlist (v1.7.14) +- [x] Add SRI hashes to CDN scripts (v1.7.14) +- [x] Add startup warning for auth misconfiguration (v1.7.14) +- [x] Add Azure Key Vault integration for secrets storage (v1.7.14) +- [x] Internal penetration test + threat model documentation + +--- + +## Phase 7: Multi-Tenancy (Premium) ⏸️ Goal: allow MSPs to manage multiple client tenants from a single deployment. Status: **Planned — not started**. Architecture designed, pending validation of core features (SIEM export, alerting) in production first. @@ -88,10 +107,10 @@ Status: **Planned — not started**. Architecture designed, pending validation o - Super-admin role for MSP staff to access all tenants ### Implementation phases -- **Phase 6.1** (2–3 days): Tenant model & registry, tenant-aware data layer, per-tenant Graph API auth -- **Phase 6.2** (1 day): Tenant-scoped API routes, tenant-specific config endpoints -- **Phase 6.3** (2 days): Frontend tenant switcher, tenant name display, admin page -- **Phase 6.4** (1 day): License gating — signed JWT `LICENSE_KEY` gates multi-tenant mode +- **Phase 7.1** (2–3 days): Tenant model & registry, tenant-aware data layer, per-tenant Graph API auth +- **Phase 7.2** (1 day): Tenant-scoped API routes, tenant-specific config endpoints +- **Phase 7.3** (2 days): Frontend tenant switcher, tenant name display, admin page +- **Phase 7.4** (1 day): License gating — signed JWT `LICENSE_KEY` gates multi-tenant mode ### Licensing model - Single-tenant: remains MIT/free