docs: update AGENTS.md, README.md, DEPLOY.md, ROADMAP.md for v1.7.14 security features
This commit is contained in:
61
AGENTS.md
61
AGENTS.md
@@ -9,20 +9,24 @@ AOC is a FastAPI microservice that ingests Microsoft Entra (Azure AD) audit logs
|
|||||||
- **Runtime**: Python 3.11 (3.14 for tests)
|
- **Runtime**: Python 3.11 (3.14 for tests)
|
||||||
- **Web Framework**: FastAPI + Uvicorn (Gunicorn in production)
|
- **Web Framework**: FastAPI + Uvicorn (Gunicorn in production)
|
||||||
- **Database**: MongoDB (PyMongo)
|
- **Database**: MongoDB (PyMongo)
|
||||||
|
- **Cache/Queue**: Valkey/Redis 8 (caching + arq async job queue)
|
||||||
- **Frontend**: Alpine.js + HTML/CSS (served as static files from `backend/frontend/`)
|
- **Frontend**: Alpine.js + HTML/CSS (served as static files from `backend/frontend/`)
|
||||||
- **Authentication**: Optional OIDC Bearer token validation against Microsoft Entra (using `python-jose` and MSAL.js on the frontend)
|
- **Authentication**: Optional OIDC Bearer token validation against Microsoft Entra (using `python-jose` and MSAL.js on the frontend)
|
||||||
- **External APIs**: Microsoft Graph API, Office 365 Management Activity API, Azure OpenAI / MS Foundry
|
- **External APIs**: Microsoft Graph API, Office 365 Management Activity API, Azure OpenAI / MS Foundry
|
||||||
- **Deployment**: Docker Compose (dev), Docker Compose + nginx (prod)
|
- **Deployment**: Docker Compose (dev), Docker Compose + nginx (prod)
|
||||||
- **CI/CD**: Gitea Actions (lint + test + Docker build + release)
|
- **CI/CD**: Gitea Actions (lint + test + Docker build + release)
|
||||||
|
- **Secrets Storage**: Environment variables (`.env`) or optional Azure Key Vault
|
||||||
|
|
||||||
## Project Structure
|
## Project Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
backend/
|
backend/
|
||||||
main.py # FastAPI app, router registration, background periodic fetch
|
main.py # FastAPI app, router registration, background periodic fetch
|
||||||
config.py # Pydantic Settings configuration (loads .env)
|
config.py # Pydantic Settings configuration (loads .env + optional Key Vault)
|
||||||
database.py # MongoClient setup (db = micro_soc, collection = events)
|
database.py # MongoClient setup (db = micro_soc, collection = events)
|
||||||
auth.py # OIDC Bearer token validation, JWKS caching, role/group checks
|
auth.py # OIDC Bearer token validation, JWKS caching, role/group checks
|
||||||
|
secrets_manager.py # Optional Azure Key Vault integration for secrets
|
||||||
|
rate_limiter.py # Redis-backed fixed-window rate limiter (fail-closed)
|
||||||
requirements.txt # Python dependencies
|
requirements.txt # Python dependencies
|
||||||
Dockerfile # python:3.11-slim image, non-root user, version baked at build
|
Dockerfile # python:3.11-slim image, non-root user, version baked at build
|
||||||
mcp_server.py # Standalone MCP server for Claude Desktop / Cursor integration
|
mcp_server.py # Standalone MCP server for Claude Desktop / Cursor integration
|
||||||
@@ -34,6 +38,9 @@ backend/
|
|||||||
health.py # GET /health, GET /metrics
|
health.py # GET /health, GET /metrics
|
||||||
rules.py # Rule-based alerting endpoints
|
rules.py # Rule-based alerting endpoints
|
||||||
webhooks.py # Microsoft Graph change notification webhooks
|
webhooks.py # Microsoft Graph change notification webhooks
|
||||||
|
alerts.py # Alert management endpoints
|
||||||
|
saved_searches.py # Saved filter presets
|
||||||
|
jobs.py # Async job status polling
|
||||||
graph/
|
graph/
|
||||||
auth.py # Client credentials token acquisition for Graph
|
auth.py # Client credentials token acquisition for Graph
|
||||||
audit_logs.py # Fetch and enrich directory audit logs from Graph
|
audit_logs.py # Fetch and enrich directory audit logs from Graph
|
||||||
@@ -59,16 +66,42 @@ Copy `.env.example` to `.env` at the repo root and fill in values:
|
|||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
```
|
```
|
||||||
|
|
||||||
Key variables:
|
### Core variables
|
||||||
- `TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET` — Microsoft app registration credentials (application permissions)
|
- `TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET` — Microsoft app registration credentials (application permissions)
|
||||||
- `AUTH_ENABLED` — set `true` to protect API/UI with OIDC Bearer tokens
|
- `AUTH_ENABLED` — set `true` to protect API/UI with OIDC Bearer tokens
|
||||||
- `AUTH_TENANT_ID`, `AUTH_CLIENT_ID` — token validation audience/issuer
|
- `AUTH_TENANT_ID`, `AUTH_CLIENT_ID` — token validation audience/issuer
|
||||||
- `AUTH_ALLOWED_ROLES`, `AUTH_ALLOWED_GROUPS` — comma-separated access control lists
|
- `AUTH_ALLOWED_ROLES`, `AUTH_ALLOWED_GROUPS` — comma-separated access control lists
|
||||||
- `ENABLE_PERIODIC_FETCH`, `FETCH_INTERVAL_MINUTES` — background ingestion scheduler
|
- `ENABLE_PERIODIC_FETCH`, `FETCH_INTERVAL_MINUTES` — background ingestion scheduler
|
||||||
- `MONGO_ROOT_USERNAME`, `MONGO_ROOT_PASSWORD`, `MONGO_PORT` — used by Docker Compose for MongoDB
|
- `MONGO_ROOT_USERNAME`, `MONGO_ROOT_PASSWORD`, `MONGO_PORT` — used by Docker Compose for MongoDB
|
||||||
|
|
||||||
|
### AI / LLM variables
|
||||||
- `AI_FEATURES_ENABLED` — set `false` to completely disable AI endpoints and UI (default `true`)
|
- `AI_FEATURES_ENABLED` — set `false` to completely disable AI endpoints and UI (default `true`)
|
||||||
- `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, `LLM_MAX_EVENTS`, `LLM_TIMEOUT_SECONDS` — LLM provider settings
|
- `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, `LLM_MAX_EVENTS`, `LLM_TIMEOUT_SECONDS` — LLM provider settings
|
||||||
- `LLM_API_VERSION` — required for Azure OpenAI / MS Foundry endpoints
|
- `LLM_API_VERSION` — required for Azure OpenAI / MS Foundry endpoints
|
||||||
|
- `LLM_ALLOWED_DOMAINS` — comma-separated domain allowlist for LLM endpoints (e.g. `api.openai.com,*.openai.azure.com`)
|
||||||
|
|
||||||
|
### Security variables
|
||||||
|
- `CORS_ORIGINS` — comma-separated allowed origins (default `*`; set explicit origins in production)
|
||||||
|
- `DOCS_ENABLED` — set `true` to expose `/docs`, `/redoc`, `/openapi.json` (default `false`)
|
||||||
|
- `METRICS_ALLOWED_IPS` — comma-separated CIDRs allowed to access `/metrics` (default: private networks + loopback)
|
||||||
|
- `WEBHOOK_CLIENT_SECRET` — secret for validating Graph webhook `clientState`
|
||||||
|
- `SIEM_ENABLED`, `SIEM_WEBHOOK_URL` — optional SIEM forwarding
|
||||||
|
- `SIEM_ALLOWED_DOMAINS` — comma-separated domain allowlist for SIEM webhook URLs
|
||||||
|
- `RATE_LIMIT_ENABLED`, `RATE_LIMIT_REQUESTS`, `RATE_LIMIT_WINDOW_SECONDS` — Redis-backed rate limiting
|
||||||
|
|
||||||
|
### Optional Azure Key Vault
|
||||||
|
- `AZURE_KEY_VAULT_NAME` — name of the Azure Key Vault to load secrets from
|
||||||
|
- When set, AOC fetches these secrets at startup:
|
||||||
|
- `aoc-client-secret` → `CLIENT_SECRET`
|
||||||
|
- `aoc-llm-api-key` → `LLM_API_KEY`
|
||||||
|
- `aoc-mongo-uri` → `MONGO_URI`
|
||||||
|
- `aoc-webhook-client-secret` → `WEBHOOK_CLIENT_SECRET`
|
||||||
|
- Requires `azure-identity` and `azure-keyvault-secrets` (uncomment in `requirements.txt`)
|
||||||
|
|
||||||
|
### Privacy / access control
|
||||||
|
- `PRIVACY_SERVICES` — comma-separated services to hide from non-privileged users (e.g. `Exchange,Teams`)
|
||||||
|
- `PRIVACY_SENSITIVE_OPERATIONS` — comma-separated operations to gate
|
||||||
|
- `PRIVACY_SERVICE_ROLES` — comma-separated Entra roles that grant access to privacy data
|
||||||
|
|
||||||
## Build and Run Commands
|
## Build and Run Commands
|
||||||
|
|
||||||
@@ -102,7 +135,9 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
|||||||
- `GET /api/config/features` — feature flags (`ai_features_enabled`)
|
- `GET /api/config/features` — feature flags (`ai_features_enabled`)
|
||||||
- `POST /api/ask` — natural language query; returns LLM narrative + referenced events (only when `AI_FEATURES_ENABLED=true`)
|
- `POST /api/ask` — natural language query; returns LLM narrative + referenced events (only when `AI_FEATURES_ENABLED=true`)
|
||||||
- `GET /health` — liveness probe with DB connectivity
|
- `GET /health` — liveness probe with DB connectivity
|
||||||
- `GET /metrics` — Prometheus metrics
|
- `GET /metrics` — Prometheus metrics (IP-restricted by default)
|
||||||
|
- `GET /api/source-health` — last fetch status per ingestion source
|
||||||
|
- `GET /api/version` — running version
|
||||||
|
|
||||||
## MCP Server
|
## MCP Server
|
||||||
|
|
||||||
@@ -162,16 +197,30 @@ When adding new features or bug fixes, add or update tests in `backend/tests/`.
|
|||||||
- Auth middleware and token validation
|
- Auth middleware and token validation
|
||||||
- API endpoints (`/api/events`, `/api/fetch-audit-logs`, `/api/ask`)
|
- API endpoints (`/api/events`, `/api/fetch-audit-logs`, `/api/ask`)
|
||||||
- NLQ time range extraction, entity extraction, query building
|
- NLQ time range extraction, entity extraction, query building
|
||||||
|
- Rate limiting behavior
|
||||||
|
|
||||||
## Security Considerations
|
## Security Considerations
|
||||||
|
|
||||||
- **Secrets**: `CLIENT_SECRET`, `LLM_API_KEY`, and other credentials come from `.env`. Never commit `.env`.
|
- **Secrets**: `CLIENT_SECRET`, `LLM_API_KEY`, and other credentials come from `.env` or Azure Key Vault. Never commit `.env`.
|
||||||
- **Auth validation**: When `AUTH_ENABLED=true`, the backend fetches JWKS from `https://login.microsoftonline.com/{AUTH_TENANT_ID}/v2.0/.well-known/openid-configuration`, caches keys for 1 hour, and validates tenant/issuer claims. Tokens are decoded without strict signature verification (`jwt.get_unverified_claims`), so the tenant and issuer checks are the primary gate.
|
- **Auth validation**: When `AUTH_ENABLED=true`, the backend fetches JWKS from `https://login.microsoftonline.com/{AUTH_TENANT_ID}/v2.0/.well-known/openid-configuration`, caches keys for 1 hour, and validates tenant/issuer/audience claims. Tokens are decoded with RS256 signature verification.
|
||||||
- **Role/Group gating**: Access is allowed if the token’s `roles` intersect `AUTH_ALLOWED_ROLES` or `groups` intersect `AUTH_ALLOWED_GROUPS`. If neither list is configured, all authenticated users are allowed.
|
- **Role/Group gating**: Access is allowed if the token's `roles` intersect `AUTH_ALLOWED_ROLES` or `groups` intersect `AUTH_ALLOWED_GROUPS`. If neither list is configured, all authenticated users are allowed — a startup warning is logged in this case.
|
||||||
|
- **CORS**: When `AUTH_ENABLED=true` and `CORS_ORIGINS="*"`, `allow_credentials` is forced to `false` to prevent cross-origin token leakage.
|
||||||
|
- **Rate limiting**: Redis-backed fixed-window rate limiting with per-category limits (fetch=10/hr, ask=30/min, write=20/min, default=120/min). Fails closed (returns 429) when Redis is unavailable.
|
||||||
- **Pagination limits**: `page_size` is clamped to a maximum of 500 to prevent large queries.
|
- **Pagination limits**: `page_size` is clamped to a maximum of 500 to prevent large queries.
|
||||||
- **Fetch window cap**: `hours` is clamped to 720 (30 days) to avoid runaway API calls.
|
- **Fetch window cap**: `hours` is clamped to 720 (30 days) to avoid runaway API calls.
|
||||||
|
- **LLM SSRF guard**: `LLM_BASE_URL` must be HTTPS and cannot point to private IPs. Optional `LLM_ALLOWED_DOMAINS` restricts to specific domains.
|
||||||
|
- **SIEM SSRF guard**: `SIEM_WEBHOOK_URL` has the same validation as LLM URLs, plus optional `SIEM_ALLOWED_DOMAINS`.
|
||||||
|
- **Metrics IP gating**: `/metrics` is restricted to private/loopback IPs by default via `METRICS_ALLOWED_IPS`.
|
||||||
|
- **OpenAPI docs**: Disabled by default (`DOCS_ENABLED=false`). Enable only in development.
|
||||||
|
- **CSP**: Content-Security-Policy headers are set on all responses. `unsafe-eval` is required for Alpine.js v3 expression evaluation.
|
||||||
|
- **SRI**: CDN scripts (Alpine.js, MSAL.js) include Subresource Integrity hashes to prevent supply chain compromise.
|
||||||
- **MCP server**: The MCP server bypasses auth entirely. Only run it in trusted environments or behind a VPN.
|
- **MCP server**: The MCP server bypasses auth entirely. Only run it in trusted environments or behind a VPN.
|
||||||
|
|
||||||
|
### Security Documentation
|
||||||
|
|
||||||
|
- `PEN_TEST_REPORT_v1.7.11.md` — Internal soft penetration test findings and remediation
|
||||||
|
- `THREAT_MODEL_v1.7.13.md` — Comprehensive threat model covering Entra/token abuse vectors
|
||||||
|
|
||||||
## Maintenance and Operations
|
## Maintenance and Operations
|
||||||
|
|
||||||
The `backend/maintenance.py` script provides two CLI commands useful for backfilling or correcting stored data:
|
The `backend/maintenance.py` script provides two CLI commands useful for backfilling or correcting stored data:
|
||||||
|
|||||||
54
DEPLOY.md
54
DEPLOY.md
@@ -7,6 +7,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose:
|
|||||||
- **nginx** — reverse proxy, TLS termination, static file serving
|
- **nginx** — reverse proxy, TLS termination, static file serving
|
||||||
- **backend** — FastAPI application (Gunicorn + Uvicorn workers)
|
- **backend** — FastAPI application (Gunicorn + Uvicorn workers)
|
||||||
- **mongo** — MongoDB data store (not exposed externally)
|
- **mongo** — MongoDB data store (not exposed externally)
|
||||||
|
- **valkey** — Redis-compatible cache and async job queue (not exposed externally)
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
@@ -20,7 +21,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose:
|
|||||||
1. **Clone / pull the latest release**
|
1. **Clone / pull the latest release**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git checkout v1.1.0
|
git checkout v1.7.14
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Copy and edit environment variables**
|
2. **Copy and edit environment variables**
|
||||||
@@ -33,7 +34,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose:
|
|||||||
3. **Set the release version**
|
3. **Set the release version**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export AOC_VERSION=v1.1.0
|
export AOC_VERSION=v1.7.14
|
||||||
```
|
```
|
||||||
|
|
||||||
4. **Deploy**
|
4. **Deploy**
|
||||||
@@ -53,7 +54,7 @@ AOC runs as a set of Docker containers orchestrated by Docker Compose:
|
|||||||
## Updating to a new release
|
## Updating to a new release
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export AOC_VERSION=v1.2.0
|
export AOC_VERSION=v1.7.14
|
||||||
docker compose -f docker-compose.prod.yml pull
|
docker compose -f docker-compose.prod.yml pull
|
||||||
docker compose -f docker-compose.prod.yml up -d
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
```
|
```
|
||||||
@@ -75,24 +76,56 @@ docker compose -f docker-compose.prod.yml up -d
|
|||||||
|
|
||||||
Replace the `nginx` service in `docker-compose.prod.yml` with a Certbot-friendly setup (e.g., use the `nginx-proxy` + `acme-companion` stack) or mount the Certbot certificates into `nginx/ssl/`.
|
Replace the `nginx` service in `docker-compose.prod.yml` with a Certbot-friendly setup (e.g., use the `nginx-proxy` + `acme-companion` stack) or mount the Certbot certificates into `nginx/ssl/`.
|
||||||
|
|
||||||
## Security hardening
|
## Security Hardening
|
||||||
|
|
||||||
- MongoDB is **not exposed** to the host — only the backend container can reach it.
|
- MongoDB is **not exposed** to the host — only the backend container can reach it.
|
||||||
|
- Valkey/Redis is **not exposed** to the host — only the backend container can reach it.
|
||||||
- The backend runs as a non-root (`aoc`) user inside the container.
|
- The backend runs as a non-root (`aoc`) user inside the container.
|
||||||
- nginx adds security headers (`X-Frame-Options`, `X-Content-Type-Options`, etc.).
|
- nginx adds security headers (`X-Frame-Options`, `X-Content-Type-Options`, etc.).
|
||||||
- Keep `.env` out of version control — it is listed in `.gitignore`.
|
- Keep `.env` out of version control — it is listed in `.gitignore`.
|
||||||
|
- Set `AUTH_ENABLED=true` and configure `AUTH_ALLOWED_ROLES` or `AUTH_ALLOWED_GROUPS` to restrict access to admin/security roles.
|
||||||
|
- Set explicit `CORS_ORIGINS` — do not use `*` in production when auth is enabled.
|
||||||
|
- Set `DOCS_ENABLED=false` to hide OpenAPI docs (`/docs`, `/openapi.json`).
|
||||||
|
- Configure `WEBHOOK_CLIENT_SECRET` to validate Graph webhook notifications.
|
||||||
|
- Set `LLM_ALLOWED_DOMAINS` if using AI features (e.g. `api.openai.com,*.openai.azure.com`).
|
||||||
|
- Set `SIEM_ALLOWED_DOMAINS` if using SIEM forwarding.
|
||||||
|
- Review `METRICS_ALLOWED_IPS` — defaults to private networks + loopback.
|
||||||
|
|
||||||
|
## Azure Key Vault (Optional)
|
||||||
|
|
||||||
|
To eliminate long-lived secrets from `.env`:
|
||||||
|
|
||||||
|
1. Create an Azure Key Vault and add these secrets:
|
||||||
|
- `aoc-client-secret` — your Graph app `CLIENT_SECRET`
|
||||||
|
- `aoc-llm-api-key` — your `LLM_API_KEY` (if using AI)
|
||||||
|
- `aoc-mongo-uri` — your `MONGO_URI`
|
||||||
|
- `aoc-webhook-client-secret` — your `WEBHOOK_CLIENT_SECRET`
|
||||||
|
|
||||||
|
2. Uncomment `azure-identity` and `azure-keyvault-secrets` in `backend/requirements.txt`
|
||||||
|
|
||||||
|
3. Set `AZURE_KEY_VAULT_NAME=your-keyvault-name` in `.env`
|
||||||
|
|
||||||
|
4. Grant the container identity `Get` permission on secrets:
|
||||||
|
- If using Azure Container Instances / AKS: assign a managed identity
|
||||||
|
- If using VM: assign a managed identity or use a service principal
|
||||||
|
- If using local Docker: authenticate via `az login` on the host
|
||||||
|
|
||||||
|
5. Rebuild and redeploy:
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export AOC_VERSION=v1.0.3
|
export AOC_VERSION=v1.7.13
|
||||||
docker compose -f docker-compose.prod.yml pull
|
docker compose -f docker-compose.prod.yml pull
|
||||||
docker compose -f docker-compose.prod.yml up -d
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
## Monitoring
|
## Monitoring
|
||||||
|
|
||||||
- Prometheus metrics: `http://your-host/metrics`
|
- Prometheus metrics: `http://your-host/metrics` (IP-restricted by default)
|
||||||
- Health check: `http://your-host/health`
|
- Health check: `http://your-host/health`
|
||||||
- Container logs:
|
- Container logs:
|
||||||
|
|
||||||
@@ -100,4 +133,13 @@ docker compose -f docker-compose.prod.yml up -d
|
|||||||
docker compose -f docker-compose.prod.yml logs -f backend
|
docker compose -f docker-compose.prod.yml logs -f backend
|
||||||
docker compose -f docker-compose.prod.yml logs -f nginx
|
docker compose -f docker-compose.prod.yml logs -f nginx
|
||||||
docker compose -f docker-compose.prod.yml logs -f mongo
|
docker compose -f docker-compose.prod.yml logs -f mongo
|
||||||
|
docker compose -f docker-compose.prod.yml logs -f valkey
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **Auth warning in logs**: "AUTH_ENABLED is true but no AUTH_ALLOWED_ROLES or AUTH_ALLOWED_GROUPS are configured" — set these to restrict access.
|
||||||
|
- **CORS issues**: Set `CORS_ORIGINS` to your exact frontend origin(s). Wildcard with auth enabled disables credentials.
|
||||||
|
- **Rate limiting 429s**: Check Redis/Valkey connectivity. The rate limiter fails closed (returns 429) when Redis is down.
|
||||||
|
- **LLM errors**: Verify `LLM_BASE_URL` is in `LLM_ALLOWED_DOMAINS` if the allowlist is configured.
|
||||||
|
- **SIEM not forwarding**: Verify `SIEM_WEBHOOK_URL` uses HTTPS and is in `SIEM_ALLOWED_DOMAINS`.
|
||||||
|
|||||||
49
README.md
49
README.md
@@ -11,13 +11,14 @@ FastAPI microservice that ingests Microsoft Entra (Azure AD) and other admin aud
|
|||||||
- Optional OIDC bearer auth (Entra) to protect the API/UI and gate access by roles/groups.
|
- Optional OIDC bearer auth (Entra) to protect the API/UI and gate access by roles/groups.
|
||||||
- Natural language query (`/api/ask`) powered by LLM (OpenAI, Azure OpenAI, or any compatible API).
|
- Natural language query (`/api/ask`) powered by LLM (OpenAI, Azure OpenAI, or any compatible API).
|
||||||
- MCP server for Claude Desktop / Cursor integration.
|
- MCP server for Claude Desktop / Cursor integration.
|
||||||
|
- Optional Azure Key Vault integration for secrets storage.
|
||||||
|
|
||||||
## Prerequisites (macOS)
|
## Prerequisites (macOS)
|
||||||
- Python 3.11
|
- Python 3.11
|
||||||
- Docker Desktop (for the quickest start) or a local MongoDB instance
|
- Docker Desktop (for the quickest start) or a local MongoDB instance
|
||||||
- An Entra app registration with **Application** permission `AuditLog.Read.All` and admin consent granted
|
- An Entra app registration with **Application** permission `AuditLog.Read.All` and admin consent granted
|
||||||
- Also required to fetch other sources:
|
- Also required to fetch other sources:
|
||||||
- `https://manage.office.com/.default` (Audit API) with `ActivityFeed.Read`/`ActivityFeed.ReadDlp` (built into the app registration’s API permissions for Office 365 Management APIs)
|
- `https://manage.office.com/.default` (Audit API) with `ActivityFeed.Read`/`ActivityFeed.ReadDlp` (built into the app registration's API permissions for Office 365 Management APIs)
|
||||||
- Intune audit: `DeviceManagementConfiguration.Read.All` (or broader) for `/deviceManagement/auditEvents`
|
- Intune audit: `DeviceManagementConfiguration.Read.All` (or broader) for `/deviceManagement/auditEvents`
|
||||||
- Optional API protection: configure `AUTH_ENABLED=true` and set `AUTH_TENANT_ID`/`AUTH_CLIENT_ID` (the audience) plus allowed roles/groups.
|
- Optional API protection: configure `AUTH_ENABLED=true` and set `AUTH_TENANT_ID`/`AUTH_CLIENT_ID` (the audience) plus allowed roles/groups.
|
||||||
|
|
||||||
@@ -49,8 +50,43 @@ cp .env.example .env
|
|||||||
# LLM_BASE_URL=https://api.openai.com/v1
|
# LLM_BASE_URL=https://api.openai.com/v1
|
||||||
# LLM_MODEL=gpt-4o-mini
|
# LLM_MODEL=gpt-4o-mini
|
||||||
# LLM_TIMEOUT_SECONDS=30
|
# LLM_TIMEOUT_SECONDS=30
|
||||||
|
# LLM_ALLOWED_DOMAINS=api.openai.com,*.openai.azure.com
|
||||||
|
|
||||||
|
# Optional: SIEM forwarding
|
||||||
|
# SIEM_ENABLED=true
|
||||||
|
# SIEM_WEBHOOK_URL=https://your-siem.com/webhook
|
||||||
|
# SIEM_ALLOWED_DOMAINS=your-siem.com
|
||||||
|
|
||||||
|
# Optional: Azure Key Vault for secrets storage
|
||||||
|
# AZURE_KEY_VAULT_NAME=your-keyvault-name
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Using Azure Key Vault for secrets
|
||||||
|
Instead of storing `CLIENT_SECRET`, `LLM_API_KEY`, `MONGO_URI`, and `WEBHOOK_CLIENT_SECRET` in `.env`, you can store them in Azure Key Vault:
|
||||||
|
|
||||||
|
1. Create a Key Vault and add secrets with these names:
|
||||||
|
- `aoc-client-secret` → your Graph app `CLIENT_SECRET`
|
||||||
|
- `aoc-llm-api-key` → your `LLM_API_KEY`
|
||||||
|
- `aoc-mongo-uri` → your `MONGO_URI`
|
||||||
|
- `aoc-webhook-client-secret` → your `WEBHOOK_CLIENT_SECRET`
|
||||||
|
2. Uncomment `azure-identity` and `azure-keyvault-secrets` in `backend/requirements.txt`
|
||||||
|
3. Set `AZURE_KEY_VAULT_NAME=your-keyvault-name` in `.env`
|
||||||
|
4. Ensure the container has Azure identity credentials (managed identity, service principal, or Azure CLI auth)
|
||||||
|
|
||||||
|
## Security Hardening Checklist
|
||||||
|
|
||||||
|
Before deploying to production:
|
||||||
|
|
||||||
|
- [ ] Set `AUTH_ENABLED=true` and configure `AUTH_ALLOWED_ROLES` or `AUTH_ALLOWED_GROUPS` to restrict access
|
||||||
|
- [ ] Set explicit `CORS_ORIGINS` (do not use `*` in production with auth enabled)
|
||||||
|
- [ ] Set `DOCS_ENABLED=false` (default) to hide OpenAPI docs
|
||||||
|
- [ ] Configure `WEBHOOK_CLIENT_SECRET` to validate Graph webhook notifications
|
||||||
|
- [ ] Set `LLM_ALLOWED_DOMAINS` if using AI features to prevent data exfiltration
|
||||||
|
- [ ] Set `SIEM_ALLOWED_DOMAINS` if using SIEM forwarding
|
||||||
|
- [ ] Review `METRICS_ALLOWED_IPS` — defaults to private networks only
|
||||||
|
- [ ] Consider Azure Key Vault instead of `.env` for secrets
|
||||||
|
- [ ] Review the threat model: `THREAT_MODEL_v1.7.13.md`
|
||||||
|
|
||||||
## Run with Docker Compose (recommended)
|
## Run with Docker Compose (recommended)
|
||||||
```bash
|
```bash
|
||||||
docker compose up --build
|
docker compose up --build
|
||||||
@@ -76,7 +112,7 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
|||||||
|
|
||||||
## API
|
## API
|
||||||
- `GET /health` — health check with MongoDB connectivity status.
|
- `GET /health` — health check with MongoDB connectivity status.
|
||||||
- `GET /metrics` — Prometheus metrics for request latency, fetch volume, and errors.
|
- `GET /metrics` — Prometheus metrics for request latency, fetch volume, and errors (IP-restricted).
|
||||||
- `GET /api/version` — running version (baked into the Docker image at build time).
|
- `GET /api/version` — running version (baked into the Docker image at build time).
|
||||||
- `GET /api/fetch-audit-logs` — pulls the last 7 days by default (override with `?hours=N`, capped to 30 days) of:
|
- `GET /api/fetch-audit-logs` — pulls the last 7 days by default (override with `?hours=N`, capped to 30 days) of:
|
||||||
- Entra directory audit logs (`/auditLogs/directoryAudits`)
|
- Entra directory audit logs (`/auditLogs/directoryAudits`)
|
||||||
@@ -171,7 +207,7 @@ curl http://localhost:8000/api/fetch-audit-logs
|
|||||||
- Visit the UI at http://localhost:8000 to filter by user/service/action/result/time, search raw text, paginate, and view raw events.
|
- Visit the UI at http://localhost:8000 to filter by user/service/action/result/time, search raw text, paginate, and view raw events.
|
||||||
|
|
||||||
## Maintenance (Dockerized)
|
## Maintenance (Dockerized)
|
||||||
Use the backend image so you don’t need a local venv:
|
Use the backend image so you don't need a local venv:
|
||||||
```bash
|
```bash
|
||||||
# ensure Mongo + backend network are up
|
# ensure Mongo + backend network are up
|
||||||
docker compose up -d mongo
|
docker compose up -d mongo
|
||||||
@@ -182,10 +218,15 @@ docker compose run --rm backend python maintenance.py dedupe
|
|||||||
```
|
```
|
||||||
Omit `--limit` to process all events. You can also run commands inside a running backend container with `docker compose exec backend ...`.
|
Omit `--limit` to process all events. You can also run commands inside a running backend container with `docker compose exec backend ...`.
|
||||||
|
|
||||||
|
## Security Documentation
|
||||||
|
- `PEN_TEST_REPORT_v1.7.11.md` — Penetration test findings and remediation
|
||||||
|
- `THREAT_MODEL_v1.7.13.md` — Comprehensive threat model covering Entra application abuse, token handling, data exfiltration vectors
|
||||||
|
|
||||||
## Notes / Troubleshooting
|
## Notes / Troubleshooting
|
||||||
- Ensure `TENANT_ID`, `CLIENT_ID`, and `CLIENT_SECRET` match an app registration with `AuditLog.Read.All` (application) permission and admin consent.
|
- Ensure `TENANT_ID`, `CLIENT_ID`, and `CLIENT_SECRET` match an app registration with `AuditLog.Read.All` (application) permission and admin consent.
|
||||||
- Additional permissions: Office 365 Management Activity (`ActivityFeed.Read`), and Intune audit (`DeviceManagementConfiguration.Read.All`).
|
- Additional permissions: Office 365 Management Activity (`ActivityFeed.Read`), and Intune audit (`DeviceManagementConfiguration.Read.All`).
|
||||||
- Auth: if `AUTH_ENABLED=true`, issued tokens must be from `AUTH_TENANT_ID`, audience = `AUTH_CLIENT_ID`; access is granted if roles or groups overlap `AUTH_ALLOWED_ROLES`/`AUTH_ALLOWED_GROUPS` (if set).
|
- Auth: if `AUTH_ENABLED=true`, issued tokens must be from `AUTH_TENANT_ID`, audience = `AUTH_CLIENT_ID`; access is granted if roles or groups overlap `AUTH_ALLOWED_ROLES`/`AUTH_ALLOWED_GROUPS` (if set). A startup warning is logged if auth is enabled but no roles/groups are configured.
|
||||||
- Backfill limits: Management Activity API typically exposes ~7 days of history via API (longer if your tenant has extended/Advanced Audit retention). Directory/Intune audit retention follows your tenant policy (commonly 30–90 days, longer with Advanced Audit).
|
- Backfill limits: Management Activity API typically exposes ~7 days of history via API (longer if your tenant has extended/Advanced Audit retention). Directory/Intune audit retention follows your tenant policy (commonly 30–90 days, longer with Advanced Audit).
|
||||||
- If you change Mongo credentials/ports, update `MONGO_URI` in `.env` (Docker Compose passes it through to the backend).
|
- If you change Mongo credentials/ports, update `MONGO_URI` in `.env` (Docker Compose passes it through to the backend).
|
||||||
- The service uses the `micro_soc` database and `events` collection by default; adjust in `backend/config.py` if needed.
|
- The service uses the `micro_soc` database and `events` collection by default; adjust in `backend/config.py` if needed.
|
||||||
|
- If using Azure Key Vault, ensure the runtime identity (managed identity, service principal, or local Azure CLI) has `Get` permission on secrets.
|
||||||
|
|||||||
31
ROADMAP.md
31
ROADMAP.md
@@ -59,7 +59,7 @@ Goal: evolve from a polling dashboard into a full security operations tool.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 5: Intelligence
|
## Phase 5: Intelligence ✅
|
||||||
Goal: add AI-powered analysis and external tool integration.
|
Goal: add AI-powered analysis and external tool integration.
|
||||||
|
|
||||||
- [x] AI feature flag (`AI_FEATURES_ENABLED`) to gate LLM-dependent features
|
- [x] AI feature flag (`AI_FEATURES_ENABLED`) to gate LLM-dependent features
|
||||||
@@ -76,7 +76,26 @@ UI polish (topbar, footer, clickable pills) in v1.6.1–v1.6.4.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 6: Multi-Tenancy (Premium) ⏸️
|
## Phase 6: Security Hardening ✅
|
||||||
|
Goal: address penetration test findings and threat model gaps.
|
||||||
|
|
||||||
|
- [x] Fix CORS credentials leak (v1.7.12)
|
||||||
|
- [x] Add security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy) (v1.7.12)
|
||||||
|
- [x] Make rate limiter fail-closed on Redis failure (v1.7.12)
|
||||||
|
- [x] Disable OpenAPI docs by default (v1.7.12)
|
||||||
|
- [x] Hide tenant_id/client_id from config endpoint when auth disabled (v1.7.12)
|
||||||
|
- [x] Validate webhook validationToken before echo (v1.7.12)
|
||||||
|
- [x] Gate `/metrics` behind IP allowlist (v1.7.12)
|
||||||
|
- [x] Add LLM domain allowlist (`LLM_ALLOWED_DOMAINS`) (v1.7.14)
|
||||||
|
- [x] Add SIEM webhook SSRF guard + domain allowlist (v1.7.14)
|
||||||
|
- [x] Add SRI hashes to CDN scripts (v1.7.14)
|
||||||
|
- [x] Add startup warning for auth misconfiguration (v1.7.14)
|
||||||
|
- [x] Add Azure Key Vault integration for secrets storage (v1.7.14)
|
||||||
|
- [x] Internal penetration test + threat model documentation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 7: Multi-Tenancy (Premium) ⏸️
|
||||||
Goal: allow MSPs to manage multiple client tenants from a single deployment.
|
Goal: allow MSPs to manage multiple client tenants from a single deployment.
|
||||||
|
|
||||||
Status: **Planned — not started**. Architecture designed, pending validation of core features (SIEM export, alerting) in production first.
|
Status: **Planned — not started**. Architecture designed, pending validation of core features (SIEM export, alerting) in production first.
|
||||||
@@ -88,10 +107,10 @@ Status: **Planned — not started**. Architecture designed, pending validation o
|
|||||||
- Super-admin role for MSP staff to access all tenants
|
- Super-admin role for MSP staff to access all tenants
|
||||||
|
|
||||||
### Implementation phases
|
### Implementation phases
|
||||||
- **Phase 6.1** (2–3 days): Tenant model & registry, tenant-aware data layer, per-tenant Graph API auth
|
- **Phase 7.1** (2–3 days): Tenant model & registry, tenant-aware data layer, per-tenant Graph API auth
|
||||||
- **Phase 6.2** (1 day): Tenant-scoped API routes, tenant-specific config endpoints
|
- **Phase 7.2** (1 day): Tenant-scoped API routes, tenant-specific config endpoints
|
||||||
- **Phase 6.3** (2 days): Frontend tenant switcher, tenant name display, admin page
|
- **Phase 7.3** (2 days): Frontend tenant switcher, tenant name display, admin page
|
||||||
- **Phase 6.4** (1 day): License gating — signed JWT `LICENSE_KEY` gates multi-tenant mode
|
- **Phase 7.4** (1 day): License gating — signed JWT `LICENSE_KEY` gates multi-tenant mode
|
||||||
|
|
||||||
### Licensing model
|
### Licensing model
|
||||||
- Single-tenant: remains MIT/free
|
- Single-tenant: remains MIT/free
|
||||||
|
|||||||
Reference in New Issue
Block a user