15 Commits

Author SHA1 Message Date
47e0dfc2ca chore: bump version to 1.5.0
All checks were successful
CI / lint-and-test (push) Successful in 37s
Release / build-and-push (push) Successful in 1m51s
2026-04-22 08:30:20 +02:00
2fffe3aec2 feat: operation-level privacy gating instead of broad service-level
All checks were successful
CI / lint-and-test (push) Successful in 21s
- Replace broad service-level hiding with fine-grained operation-level gating
- PRIVACY_SENSITIVE_OPERATIONS config: hide specific operations across ALL services
- PRIVACY_SERVICES still works for broad service-level blocking (optional)
- Users without PRIVACY_SERVICE_ROLES:
  * Don't see sensitive operations in /api/filter-options
  * Can't query sensitive operations via /api/events or /api/ask
  * Get 403 on /api/events/{id}/explain for sensitive events
- Exchange/Teams services remain visible; only privacy ops are hidden
- Update .env.example with new operation-level config docs
2026-04-22 08:23:46 +02:00
b2f4cabef4 feat: service-level role gating for privacy-sensitive services (Option A)
All checks were successful
CI / lint-and-test (push) Successful in 25s
- Add PRIVACY_SERVICES and PRIVACY_SERVICE_ROLES config variables
- Add user_can_access_privacy_services(claims) helper in auth.py
- /api/events filters out privacy services for users without required roles
- /api/filter-options excludes privacy services from dropdown options
- /api/ask excludes privacy services from NLQ queries
- /api/events/{id}/explain returns 403 for privacy events if unauthorized
- Teams added to default noisy service exclusion (frontend + backend)
- Update .env.example with privacy config documentation
- Add tests for event filtering, filter-options exclusion, and explain 403
2026-04-22 07:26:21 +02:00
e069869a94 feat: exclude Teams from defaults + GUID resolution in explain
All checks were successful
CI / lint-and-test (push) Successful in 26s
- Add Teams to noisy services excluded by default (frontend + backend ask)
- Exchange, SharePoint, and Teams now unchecked by default in filters
- Enhance explain endpoint with GUID resolution:
  * Extract UUIDs from raw event JSON recursively
  * Resolve directory objects via Graph API (user, group, SP, device)
  * Include resolved names in LLM prompt so explanations reference
    human-readable names instead of raw GUIDs
- Add asyncio import for to_thread wrapper around sync Graph calls
2026-04-22 07:12:10 +02:00
fb2386e190 feat: saved searches (bookmarks)
All checks were successful
CI / lint-and-test (push) Successful in 23s
- Add saved_searches_collection to database.py with index on created_by+created_at
- New routes/saved_searches.py: GET /api/saved-searches, POST, DELETE
- Saved searches are scoped per user (created_by = token sub)
- Mount router in main.py
- Frontend: Save filters button, saved search pills with load/delete
- loadSavedSearches called on initApp
- applySavedSearch restores filters and validates services against current options
- Add CSS for saved-searches row
- Add tests for CRUD, delete 404, and name validation
2026-04-22 07:04:07 +02:00
05f5f07e7b chore: bump version to 1.4.0
All checks were successful
CI / lint-and-test (push) Successful in 30s
Release / build-and-push (push) Successful in 1m24s
2026-04-22 06:48:47 +02:00
681f7d468a ui: persist filters, default exclude noisy services, true=green
All checks were successful
CI / lint-and-test (push) Successful in 32s
- Add 'true' to result pill success keywords (pill--ok)
- Persist filter state to localStorage (loadSavedFilters/saveFilters)
- Default service selection excludes Exchange and SharePoint
- clearFilters resets to default selection (excluding noisy services)
- Restore saved services only if they still exist in current options
2026-04-22 06:41:33 +02:00
fb5d45dfb3 chore: bump version to 1.3.2
All checks were successful
CI / lint-and-test (push) Successful in 23s
Release / build-and-push (push) Successful in 1m38s
2026-04-21 22:28:52 +02:00
658ddd0aac feat: copy raw event and AI explain in modal
All checks were successful
CI / lint-and-test (push) Successful in 32s
- Add POST /api/events/{id}/explain endpoint that fetches event + related events
  and asks the LLM for a plain-language explanation with security context
- Add 'Copy' button to raw event modal (uses navigator.clipboard)
- Add 'Explain' button to raw event modal (only when AI_FEATURES_ENABLED)
- Show explanation in modal with markdown rendering
- Add CSS for modal actions and explanation panel
- Add tests for explain endpoint (404, no LLM key, mocked LLM success)
2026-04-21 22:26:26 +02:00
a5db0d363d chore: bump version to 1.3.1
All checks were successful
Release / build-and-push (push) Successful in 1m16s
CI / lint-and-test (push) Successful in 25s
2026-04-21 11:28:32 +02:00
43582692ba ui: fix page title and hero text to match product name
All checks were successful
CI / lint-and-test (push) Successful in 38s
2026-04-21 07:41:41 +02:00
5122739c01 feat: MCP server over SSE with OIDC auth
All checks were successful
CI / lint-and-test (push) Successful in 36s
- Extract shared MCP tool handlers to mcp_common.py
- mcp_server.py now uses shared handlers (stdio transport for local dev)
- New routes/mcp.py: SSE transport behind existing OIDC Bearer auth
- Mount MCP ASGI app at /mcp in main.py when AI_FEATURES_ENABLED
- /mcp/sse  -> establishes SSE stream (requires valid token when auth enabled)
- /mcp/messages/ -> receives MCP client messages
- Update README with SSE MCP docs
- Add tests for mount existence, auth, and message routing
2026-04-21 07:38:12 +02:00
6cf5c0a28b ui: move filters section before ask section
All checks were successful
CI / lint-and-test (push) Successful in 27s
2026-04-20 18:17:09 +02:00
6aa47e9b1e docs: update README and ROADMAP for v1.3.0
All checks were successful
CI / lint-and-test (push) Successful in 27s
2026-04-20 18:14:28 +02:00
60b6ad15c4 Release v1.3.0: AI feature flag and MCP server
All checks were successful
CI / lint-and-test (push) Successful in 45s
Release / build-and-push (push) Successful in 1m34s
- Add AI_FEATURES_ENABLED config flag to gate AI/natural-language features
- Conditionally register /api/ask router based on AI_FEATURES_ENABLED
- Add GET /api/config/features endpoint for frontend feature detection
- Update frontend to hide Ask panel when AI features are disabled
- Implement standalone MCP server (backend/mcp_server.py) with tools:
  * search_events, get_event, get_summary, ask
- Add mcp dependency to requirements.txt
- Update .env.example, AGENTS.md, and ROADMAP.md
- Bump VERSION to 1.3.0
2026-04-20 18:11:26 +02:00
21 changed files with 1372 additions and 84 deletions

View File

@@ -34,6 +34,10 @@ SIEM_WEBHOOK_URL=
# Optional: enable rule-based alerting during ingestion # Optional: enable rule-based alerting during ingestion
ALERTS_ENABLED=false ALERTS_ENABLED=false
# Optional: enable AI/natural-language features (/api/ask, MCP server)
# Set to false to completely disable AI endpoints and UI elements
AI_FEATURES_ENABLED=true
# Optional: LLM configuration for natural language querying (/api/ask) # Optional: LLM configuration for natural language querying (/api/ask)
# Supports any OpenAI-compatible API (OpenAI, Azure OpenAI, Ollama, etc.) # Supports any OpenAI-compatible API (OpenAI, Azure OpenAI, Ollama, etc.)
# For Azure OpenAI / MS Foundry, set BASE_URL to your deployment endpoint # For Azure OpenAI / MS Foundry, set BASE_URL to your deployment endpoint
@@ -45,3 +49,11 @@ LLM_MODEL=gpt-4o-mini
LLM_MAX_EVENTS=200 LLM_MAX_EVENTS=200
LLM_TIMEOUT_SECONDS=30 LLM_TIMEOUT_SECONDS=30
LLM_API_VERSION= LLM_API_VERSION=
# Optional: privacy / access control
# Hide entire services from users without PRIVACY_SERVICE_ROLES
# PRIVACY_SERVICES=Exchange,Teams
# Hide specific operations across all services from users without PRIVACY_SERVICE_ROLES
# PRIVACY_SENSITIVE_OPERATIONS=MailItemsAccessed,Search-Mailbox,Send,ChatMessageRead
# Comma-separated list of Entra roles that can access privacy-sensitive data
# PRIVACY_SERVICE_ROLES=SecurityAdministrator,ComplianceAdministrator

View File

@@ -6,28 +6,34 @@ AOC is a FastAPI microservice that ingests Microsoft Entra (Azure AD) audit logs
## Technology Stack ## Technology Stack
- **Runtime**: Python 3.11 - **Runtime**: Python 3.11 (3.14 for tests)
- **Web Framework**: FastAPI + Uvicorn - **Web Framework**: FastAPI + Uvicorn (Gunicorn in production)
- **Database**: MongoDB (PyMongo) - **Database**: MongoDB (PyMongo)
- **Frontend**: Vanilla HTML/CSS/JS (served as static files from `backend/frontend/`) - **Frontend**: Alpine.js + HTML/CSS (served as static files from `backend/frontend/`)
- **Authentication**: Optional OIDC Bearer token validation against Microsoft Entra (using `python-jose` and MSAL.js on the frontend) - **Authentication**: Optional OIDC Bearer token validation against Microsoft Entra (using `python-jose` and MSAL.js on the frontend)
- **External APIs**: Microsoft Graph API, Office 365 Management Activity API - **External APIs**: Microsoft Graph API, Office 365 Management Activity API, Azure OpenAI / MS Foundry
- **Deployment**: Docker Compose - **Deployment**: Docker Compose (dev), Docker Compose + nginx (prod)
- **CI/CD**: Gitea Actions (lint + test + Docker build + release)
## Project Structure ## Project Structure
``` ```
backend/ backend/
main.py # FastAPI app, router registration, background periodic fetch main.py # FastAPI app, router registration, background periodic fetch
config.py # Environment-based configuration (loads .env) config.py # Pydantic Settings configuration (loads .env)
database.py # MongoClient setup (db = micro_soc, collection = events) database.py # MongoClient setup (db = micro_soc, collection = events)
auth.py # OIDC Bearer token validation, JWKS caching, role/group checks auth.py # OIDC Bearer token validation, JWKS caching, role/group checks
requirements.txt # Python dependencies requirements.txt # Python dependencies
Dockerfile # python:3.11-slim image Dockerfile # python:3.11-slim image, non-root user, version baked at build
mcp_server.py # Standalone MCP server for Claude Desktop / Cursor integration
routes/ routes/
fetch.py # GET /api/fetch-audit-logs, run_fetch() fetch.py # GET /api/fetch-audit-logs, run_fetch()
events.py # GET /api/events, GET /api/filter-options events.py # GET /api/events, GET /api/filter-options, PATCH tags, POST comments
config.py # GET /api/config/auth config.py # GET /api/config/auth, GET /api/config/features
ask.py # POST /api/ask — natural language query with LLM
health.py # GET /health, GET /metrics
rules.py # Rule-based alerting endpoints
webhooks.py # Microsoft Graph change notification webhooks
graph/ graph/
auth.py # Client credentials token acquisition for Graph auth.py # Client credentials token acquisition for Graph
audit_logs.py # Fetch and enrich directory audit logs from Graph audit_logs.py # Fetch and enrich directory audit logs from Graph
@@ -41,7 +47,7 @@ backend/
mappings.yml # User-editable category labels and summary templates mappings.yml # User-editable category labels and summary templates
maintenance.py # CLI for re-normalization and deduplication of stored events maintenance.py # CLI for re-normalization and deduplication of stored events
frontend/ frontend/
index.html # Single-page UI with filters, pagination, raw-event modal index.html # Single-page UI with filters, pagination, ask panel, raw-event modal
style.css # Dark-themed stylesheet style.css # Dark-themed stylesheet
``` ```
@@ -60,6 +66,9 @@ Key variables:
- `AUTH_ALLOWED_ROLES`, `AUTH_ALLOWED_GROUPS` — comma-separated access control lists - `AUTH_ALLOWED_ROLES`, `AUTH_ALLOWED_GROUPS` — comma-separated access control lists
- `ENABLE_PERIODIC_FETCH`, `FETCH_INTERVAL_MINUTES` — background ingestion scheduler - `ENABLE_PERIODIC_FETCH`, `FETCH_INTERVAL_MINUTES` — background ingestion scheduler
- `MONGO_ROOT_USERNAME`, `MONGO_ROOT_PASSWORD`, `MONGO_PORT` — used by Docker Compose for MongoDB - `MONGO_ROOT_USERNAME`, `MONGO_ROOT_PASSWORD`, `MONGO_PORT` — used by Docker Compose for MongoDB
- `AI_FEATURES_ENABLED` — set `false` to completely disable AI endpoints and UI (default `true`)
- `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL`, `LLM_MAX_EVENTS`, `LLM_TIMEOUT_SECONDS` — LLM provider settings
- `LLM_API_VERSION` — required for Azure OpenAI / MS Foundry endpoints
## Build and Run Commands ## Build and Run Commands
@@ -87,35 +96,81 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000
## API Endpoints ## API Endpoints
- `GET /api/fetch-audit-logs?hours=168` — pulls last N hours (capped at 720 / 30 days) from all sources, normalizes, dedupes, and upserts into MongoDB - `GET /api/fetch-audit-logs?hours=168` — pulls last N hours (capped at 720 / 30 days) from all sources, normalizes, dedupes, and upserts into MongoDB
- `GET /api/events` — list stored events with filters (`service`, `actor`, `operation`, `result`, `start`, `end`, `search`) and pagination (`page`, `page_size`) - `GET /api/events` — list stored events with filters (`service`, `actor`, `operation`, `result`, `start`, `end`, `search`) and cursor-based pagination
- `GET /api/filter-options` — best-effort distinct values for UI dropdowns - `GET /api/filter-options` — best-effort distinct values for UI dropdowns
- `GET /api/config/auth` — auth configuration exposed to the frontend - `GET /api/config/auth` — auth configuration exposed to the frontend
- `GET /api/config/features` — feature flags (`ai_features_enabled`)
- `POST /api/ask` — natural language query; returns LLM narrative + referenced events (only when `AI_FEATURES_ENABLED=true`)
- `GET /health` — liveness probe with DB connectivity
- `GET /metrics` — Prometheus metrics
## MCP Server
A standalone MCP server (`backend/mcp_server.py`) exposes audit log tools for Claude Desktop, Cursor, and other MCP clients.
Available tools:
- `search_events` — Search by entity, service, operation, result, time range
- `get_event` — Retrieve a single event by ID (raw JSON)
- `get_summary` — Aggregated counts by service, operation, result, actor
- `ask` — Natural language question (returns recent events + guidance)
**Claude Desktop config** (`~/.config/claude/claude_desktop_config.json`):
```json
{
"mcpServers": {
"aoc": {
"command": "python",
"args": ["/path/to/aoc/backend/mcp_server.py"],
"env": {"MONGO_URI": "mongodb://root:example@localhost:27017/"}
}
}
}
```
The MCP server imports `database.py` directly and does not go through the FastAPI layer, so it shares the same MongoDB connection but bypasses auth.
## AI Feature Flag
Set `AI_FEATURES_ENABLED=false` in `.env` to:
- Prevent the `ask` router from being registered in FastAPI
- Hide the "Ask a question" panel in the frontend
- Return `ai_features_enabled: false` from `/api/config/features`
This is intended for the open-core monetization split: core features (ingestion, filtering, search, export) are always available; premium AI features (NLQ, MCP) can be disabled.
## Code Conventions ## Code Conventions
- Python modules use absolute imports within the `backend/` package (e.g., `from graph.auth import get_access_token`). When running locally, ensure the working directory is `backend/` so these resolve correctly. - Python modules use absolute imports within the `backend/` package (e.g., `from graph.auth import get_access_token`). When running locally, ensure the working directory is `backend/` so these resolve correctly.
- No formal formatter or linter is configured. Keep changes consistent with the existing style: simple functions, explicit exception handling, and informative docstrings. - The project uses `ruff` for linting and formatting. Run `ruff check . && ruff format .` before committing.
- The frontend is a single HTML file with inline JavaScript. It relies on the MSAL.js CDN (`https://alcdn.msauth.net/browser/2.37.0/js/msal-browser.min.js`). - Keep changes consistent with the existing style: simple functions, explicit exception handling, and informative docstrings.
- The frontend is a single HTML file with inline JavaScript and Alpine.js.
## Testing ## Testing
There are currently **no automated tests** in this repository. When adding new features or bug fixes, verify behavior manually: Tests run with pytest and mongomock (no real MongoDB required):
1. Start the server (Docker Compose or local uvicorn). ```bash
2. Run a smoke test: cd backend
```bash python -m venv .venv_test
curl http://localhost:8000/api/events source .venv_test/bin/activate
curl http://localhost:8000/api/fetch-audit-logs pip install -r requirements.txt
``` pytest tests/ -q
3. Open http://localhost:8000 in a browser, apply filters, paginate, and click "View raw event". ```
When adding new features or bug fixes, add or update tests in `backend/tests/`. The test suite covers:
- Event normalization and deduplication
- Auth middleware and token validation
- API endpoints (`/api/events`, `/api/fetch-audit-logs`, `/api/ask`)
- NLQ time range extraction, entity extraction, query building
## Security Considerations ## Security Considerations
- **Secrets**: `CLIENT_SECRET` and other credentials come from `.env`. Never commit `.env`. - **Secrets**: `CLIENT_SECRET`, `LLM_API_KEY`, and other credentials come from `.env`. Never commit `.env`.
- **Auth validation**: When `AUTH_ENABLED=true`, the backend fetches JWKS from `https://login.microsoftonline.com/{AUTH_TENANT_ID}/v2.0/.well-known/openid-configuration`, caches keys for 1 hour, and validates tenant/issuer claims. Tokens are decoded without strict signature verification (`jwt.get_unverified_claims`), so the tenant and issuer checks are the primary gate. - **Auth validation**: When `AUTH_ENABLED=true`, the backend fetches JWKS from `https://login.microsoftonline.com/{AUTH_TENANT_ID}/v2.0/.well-known/openid-configuration`, caches keys for 1 hour, and validates tenant/issuer claims. Tokens are decoded without strict signature verification (`jwt.get_unverified_claims`), so the tenant and issuer checks are the primary gate.
- **Role/Group gating**: Access is allowed if the tokens `roles` intersect `AUTH_ALLOWED_ROLES` or `groups` intersect `AUTH_ALLOWED_GROUPS`. If neither list is configured, all authenticated users are allowed. - **Role/Group gating**: Access is allowed if the tokens `roles` intersect `AUTH_ALLOWED_ROLES` or `groups` intersect `AUTH_ALLOWED_GROUPS`. If neither list is configured, all authenticated users are allowed.
- **Pagination limits**: `page_size` is clamped to a maximum of 500 to prevent large queries. - **Pagination limits**: `page_size` is clamped to a maximum of 500 to prevent large queries.
- **Fetch window cap**: `hours` is clamped to 720 (30 days) to avoid runaway API calls. - **Fetch window cap**: `hours` is clamped to 720 (30 days) to avoid runaway API calls.
- **MCP server**: The MCP server bypasses auth entirely. Only run it in trusted environments or behind a VPN.
## Maintenance and Operations ## Maintenance and Operations

View File

@@ -9,6 +9,8 @@ FastAPI microservice that ingests Microsoft Entra (Azure AD) and other admin aud
- Office 365 Management Activity API client for Exchange/SharePoint/Teams admin audit logs. - Office 365 Management Activity API client for Exchange/SharePoint/Teams admin audit logs.
- Frontend served from the backend for filtering/searching events and viewing raw entries. - Frontend served from the backend for filtering/searching events and viewing raw entries.
- Optional OIDC bearer auth (Entra) to protect the API/UI and gate access by roles/groups. - Optional OIDC bearer auth (Entra) to protect the API/UI and gate access by roles/groups.
- Natural language query (`/api/ask`) powered by LLM (OpenAI, Azure OpenAI, or any compatible API).
- MCP server for Claude Desktop / Cursor integration.
## Prerequisites (macOS) ## Prerequisites (macOS)
- Python 3.11 - Python 3.11
@@ -38,6 +40,15 @@ cp .env.example .env
# Optional: CORS origins if the frontend is served separately # Optional: CORS origins if the frontend is served separately
# CORS_ORIGINS=http://localhost:3000,https://app.example.com # CORS_ORIGINS=http://localhost:3000,https://app.example.com
# Optional: enable AI/natural-language features (/api/ask, MCP server)
# AI_FEATURES_ENABLED=true
# Optional: LLM configuration for natural language querying
# LLM_API_KEY=...
# LLM_BASE_URL=https://api.openai.com/v1
# LLM_MODEL=gpt-4o-mini
# LLM_TIMEOUT_SECONDS=30
``` ```
## Run with Docker Compose (recommended) ## Run with Docker Compose (recommended)
@@ -66,6 +77,7 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000
## API ## API
- `GET /health` — health check with MongoDB connectivity status. - `GET /health` — health check with MongoDB connectivity status.
- `GET /metrics` — Prometheus metrics for request latency, fetch volume, and errors. - `GET /metrics` — Prometheus metrics for request latency, fetch volume, and errors.
- `GET /api/version` — running version (baked into the Docker image at build time).
- `GET /api/fetch-audit-logs` — pulls the last 7 days by default (override with `?hours=N`, capped to 30 days) of: - `GET /api/fetch-audit-logs` — pulls the last 7 days by default (override with `?hours=N`, capped to 30 days) of:
- Entra directory audit logs (`/auditLogs/directoryAudits`) - Entra directory audit logs (`/auditLogs/directoryAudits`)
- Exchange/SharePoint/Teams admin audits (via Office 365 Management Activity API) - Exchange/SharePoint/Teams admin audits (via Office 365 Management Activity API)
@@ -82,11 +94,34 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000
- `GET /api/source-health` — last fetch status for each ingestion source (`directory`, `unified`, `intune`). - `GET /api/source-health` — last fetch status for each ingestion source (`directory`, `unified`, `intune`).
- `PATCH /api/events/{id}/tags` — update tags on an event (e.g., `investigating`, `false_positive`). - `PATCH /api/events/{id}/tags` — update tags on an event (e.g., `investigating`, `false_positive`).
- `POST /api/events/{id}/comments` — add a comment to an event. - `POST /api/events/{id}/comments` — add a comment to an event.
- `POST /api/events/{id}/explain` — AI explanation of a single audit event with security context (requires `LLM_API_KEY`).
- `POST /api/ask` — natural language query. Returns a narrative answer + referenced events. Supports time ranges, entity names, and respects active UI filters. Only available when `AI_FEATURES_ENABLED=true`.
- `GET /api/config/features` — feature flags (`ai_features_enabled`).
- `GET /api/rules` — list alert rules. - `GET /api/rules` — list alert rules.
- `POST /api/rules` — create an alert rule. - `POST /api/rules` — create an alert rule.
- `PUT /api/rules/{id}` — update an alert rule. - `PUT /api/rules/{id}` — update an alert rule.
- `DELETE /api/rules/{id}` — delete an alert rule. - `DELETE /api/rules/{id}` — delete an alert rule.
### MCP Server
AOC exposes an MCP interface in two forms:
**1. HTTP/SSE (production)** — mounted at `/mcp` inside the FastAPI app, behind OIDC auth:
- `GET /mcp/sse` — establish SSE stream (requires Bearer token if `AUTH_ENABLED=true`)
- `POST /mcp/messages/?session_id=...` — send tool calls
This is the recommended way to use MCP against a remote deployment like `aoc.cqre.net`. Any MCP client that supports SSE transport (e.g. Cursor, Claude Desktop with an SSE bridge, or custom scripts) can connect using the same Entra token as the web UI.
**2. stdio (local development)**`python backend/mcp_server.py`:
- Runs as a local subprocess for Claude Desktop
- Connects directly to MongoDB (bypasses FastAPI auth)
- Useful for local development when you have the repo cloned and MongoDB running locally
Available tools (both transports):
- `search_events` — filter by entity, service, operation, result, time range.
- `get_event` — retrieve raw event JSON by ID.
- `get_summary` — aggregated summary (service, operation, result, actor counts) for the last N days.
- `ask` — natural language query returning recent events.
Stored document shape (collection `micro_soc.events`): Stored document shape (collection `micro_soc.events`):
```json ```json
{ {

View File

@@ -59,5 +59,15 @@ Goal: evolve from a polling dashboard into a full security operations tool.
--- ---
## Phase 5: Intelligence
Goal: add AI-powered analysis and external tool integration.
- [x] AI feature flag (`AI_FEATURES_ENABLED`) to gate LLM-dependent features
- [x] Natural language query endpoint (`/api/ask`) with intent extraction and smart sampling
- [x] MCP (Model Context Protocol) server for Claude Desktop / Cursor integration
- [ ] Advanced analytics dashboard (trending operations, anomaly detection)
- [ ] Redis caching for LLM responses and frequent queries
- [ ] Async queue for LLM requests to prevent timeout/cost explosions at scale
## Completed in this PR ## Completed in this PR
All Phase 1 items were implemented in the latest changes. All Phase 5 items marked done were implemented in v1.3.0.

View File

@@ -1 +1 @@
1.2.7 1.5.0

View File

@@ -8,6 +8,8 @@ from config import (
AUTH_CLIENT_ID, AUTH_CLIENT_ID,
AUTH_ENABLED, AUTH_ENABLED,
AUTH_TENANT_ID, AUTH_TENANT_ID,
PRIVACY_SERVICE_ROLES,
PRIVACY_SERVICES,
) )
from fastapi import Header, HTTPException from fastapi import Header, HTTPException
from jwt import ExpiredSignatureError, InvalidTokenError, decode from jwt import ExpiredSignatureError, InvalidTokenError, decode
@@ -82,6 +84,14 @@ def _decode_token(token: str, jwks):
raise HTTPException(status_code=401, detail=f"Invalid token ({type(exc).__name__})") from None raise HTTPException(status_code=401, detail=f"Invalid token ({type(exc).__name__})") from None
def user_can_access_privacy_services(claims: dict) -> bool:
"""Check if the user has roles that grant access to privacy-sensitive services."""
if not PRIVACY_SERVICES or not PRIVACY_SERVICE_ROLES:
return True
user_roles = set(claims.get("roles", []) or claims.get("role", []) or [])
return bool(user_roles.intersection(PRIVACY_SERVICE_ROLES))
def require_auth(authorization: str | None = Header(None)): def require_auth(authorization: str | None = Header(None)):
if not AUTH_ENABLED: if not AUTH_ENABLED:
return {"sub": "anonymous"} return {"sub": "anonymous"}

View File

@@ -42,7 +42,8 @@ class Settings(BaseSettings):
# Alerting # Alerting
ALERTS_ENABLED: bool = False ALERTS_ENABLED: bool = False
# LLM / Natural Language Query # AI / Natural Language Query
AI_FEATURES_ENABLED: bool = True
LLM_API_KEY: str = "" LLM_API_KEY: str = ""
LLM_BASE_URL: str = "https://api.openai.com/v1" LLM_BASE_URL: str = "https://api.openai.com/v1"
LLM_MODEL: str = "gpt-4o-mini" LLM_MODEL: str = "gpt-4o-mini"
@@ -50,6 +51,12 @@ class Settings(BaseSettings):
LLM_TIMEOUT_SECONDS: int = 30 LLM_TIMEOUT_SECONDS: int = 30
LLM_API_VERSION: str = "" # e.g. 2025-01-01-preview for Azure OpenAI LLM_API_VERSION: str = "" # e.g. 2025-01-01-preview for Azure OpenAI
# Privacy / access control
# Entire services can be hidden, or specific operations can be gated.
PRIVACY_SERVICES: str = "" # comma-separated, e.g. "Exchange,Teams"
PRIVACY_SENSITIVE_OPERATIONS: str = "" # comma-separated, e.g. "MailItemsAccessed,Search-Mailbox,Send"
PRIVACY_SERVICE_ROLES: str = "" # comma-separated, e.g. "SecurityAdministrator,ComplianceAdministrator"
_settings = Settings() _settings = Settings()
@@ -77,9 +84,14 @@ SIEM_ENABLED = _settings.SIEM_ENABLED
SIEM_WEBHOOK_URL = _settings.SIEM_WEBHOOK_URL SIEM_WEBHOOK_URL = _settings.SIEM_WEBHOOK_URL
ALERTS_ENABLED = _settings.ALERTS_ENABLED ALERTS_ENABLED = _settings.ALERTS_ENABLED
AI_FEATURES_ENABLED = _settings.AI_FEATURES_ENABLED
LLM_API_KEY = _settings.LLM_API_KEY LLM_API_KEY = _settings.LLM_API_KEY
LLM_BASE_URL = _settings.LLM_BASE_URL LLM_BASE_URL = _settings.LLM_BASE_URL
LLM_MODEL = _settings.LLM_MODEL LLM_MODEL = _settings.LLM_MODEL
LLM_MAX_EVENTS = _settings.LLM_MAX_EVENTS LLM_MAX_EVENTS = _settings.LLM_MAX_EVENTS
LLM_TIMEOUT_SECONDS = _settings.LLM_TIMEOUT_SECONDS LLM_TIMEOUT_SECONDS = _settings.LLM_TIMEOUT_SECONDS
LLM_API_VERSION = _settings.LLM_API_VERSION LLM_API_VERSION = _settings.LLM_API_VERSION
PRIVACY_SERVICES = {s.strip() for s in _settings.PRIVACY_SERVICES.split(",") if s.strip()}
PRIVACY_SENSITIVE_OPERATIONS = {o.strip() for o in _settings.PRIVACY_SENSITIVE_OPERATIONS.split(",") if o.strip()}
PRIVACY_SERVICE_ROLES = {r.strip() for r in _settings.PRIVACY_SERVICE_ROLES.split(",") if r.strip()}

View File

@@ -7,6 +7,7 @@ from pymongo import ASCENDING, DESCENDING, TEXT, MongoClient
client = MongoClient(MONGO_URI or "mongodb://localhost:27017") client = MongoClient(MONGO_URI or "mongodb://localhost:27017")
db = client[DB_NAME] db = client[DB_NAME]
events_collection = db["events"] events_collection = db["events"]
saved_searches_collection = db["saved_searches"]
logger = structlog.get_logger("aoc.database") logger = structlog.get_logger("aoc.database")
@@ -20,6 +21,7 @@ def setup_indexes(max_retries: int = 5, delay: float = 2.0):
events_collection.create_index([("timestamp", DESCENDING)]) events_collection.create_index([("timestamp", DESCENDING)])
events_collection.create_index([("service", ASCENDING), ("timestamp", DESCENDING)]) events_collection.create_index([("service", ASCENDING), ("timestamp", DESCENDING)])
events_collection.create_index("id") events_collection.create_index("id")
saved_searches_collection.create_index([("created_by", ASCENDING), ("created_at", DESCENDING)])
events_collection.create_index( events_collection.create_index(
[("actor_display", TEXT), ("raw_text", TEXT), ("operation", TEXT)], [("actor_display", TEXT), ("raw_text", TEXT), ("operation", TEXT)],
name="text_search_index", name="text_search_index",

View File

@@ -3,7 +3,7 @@
<head> <head>
<meta charset="UTF-8" /> <meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>AOC Events</title> <title>Admin Operations Center</title>
<link rel="stylesheet" href="/style.css?v=8" /> <link rel="stylesheet" href="/style.css?v=8" />
<script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script> <script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
<script src="https://alcdn.msauth.net/browser/2.37.0/js/msal-browser.min.js" crossorigin="anonymous"></script> <script src="https://alcdn.msauth.net/browser/2.37.0/js/msal-browser.min.js" crossorigin="anonymous"></script>
@@ -13,8 +13,8 @@
<header class="hero"> <header class="hero">
<div> <div>
<p class="eyebrow">Admin Operations Center <span class="version-badge" x-text="appVersion"></span></p> <p class="eyebrow">Admin Operations Center <span class="version-badge" x-text="appVersion"></span></p>
<h1>Directory Audit Explorer</h1> <h1>Audit Log Explorer</h1>
<p class="lede">Filter Microsoft Entra audit events by user, app, time, action, and action type.</p> <p class="lede">Search and review Microsoft audit events from Entra, Intune, Exchange, SharePoint, and Teams.</p>
</div> </div>
<div class="cta"> <div class="cta">
<button id="authBtn" class="ghost" aria-label="Login" x-text="authBtnText" @click="toggleAuth()"></button> <button id="authBtn" class="ghost" aria-label="Login" x-text="authBtnText" @click="toggleAuth()"></button>
@@ -38,49 +38,6 @@
</div> </div>
</section> </section>
<section class="panel">
<h3>Ask a question</h3>
<form class="ask-form" @submit.prevent="askQuestion()">
<div class="ask-row">
<input
type="text"
placeholder="What happened to device ABC123 in the last 3 days?"
x-model="askQuestionText"
class="ask-input"
/>
<button type="submit" :disabled="askLoading" x-text="askLoading ? 'Thinking…' : 'Ask'">Ask</button>
</div>
<div x-show="hasActiveFilters()" class="ask-filter-hint">
<small>Respecting active filters: <span x-text="activeFilterSummary()"></span></small>
</div>
</form>
<template x-if="askAnswer">
<div class="ask-result">
<div x-show="askLlmError" class="ask-error" x-text="askLlmError"></div>
<div class="ask-answer" x-html="askAnswerHtml"></div>
<template x-if="askEvents.length">
<div class="ask-events">
<h4>Referenced events</h4>
<template x-for="(evt, idx) in askEvents" :key="evt.id || idx">
<article class="event event--compact">
<div class="event__meta">
<span class="pill" x-text="evt.display_category || evt.service || '—'"></span>
<span class="pill" :class="['success','succeeded','ok','passed'].includes((evt.result || '').toLowerCase()) ? 'pill--ok' : 'pill--warn'" x-text="evt.result || '—'"></span>
</div>
<h3 x-text="evt.operation || '—'"></h3>
<p class="event__detail" x-show="evt.display_summary"><strong>Summary:</strong> <span x-text="evt.display_summary"></span></p>
<p class="event__detail"><strong>Actor:</strong> <span x-text="evt.actor_display || '—'"></span></p>
<p class="event__detail"><strong>Target:</strong> <span x-text="Array.isArray(evt.target_displays) ? evt.target_displays.join(', ') : '—'"></span></p>
<p class="event__detail"><strong>When:</strong> <span x-text="evt.timestamp ? new Date(evt.timestamp).toLocaleString() : '—'"></span></p>
</article>
</template>
</div>
</template>
<button type="button" class="ghost" @click="clearAsk()">Clear</button>
</div>
</template>
</section>
<section class="panel"> <section class="panel">
<form id="filters" class="filters" @submit.prevent="resetPagination(); loadEvents()"> <form id="filters" class="filters" @submit.prevent="resetPagination(); loadEvents()">
<div class="filter-row"> <div class="filter-row">
@@ -155,14 +112,69 @@
<div class="actions"> <div class="actions">
<button type="submit">Apply filters</button> <button type="submit">Apply filters</button>
<button type="button" id="clearBtn" class="ghost" @click="clearFilters()">Clear</button> <button type="button" id="clearBtn" class="ghost" @click="clearFilters()">Clear</button>
<button type="button" class="ghost" @click="saveCurrentFilters()">Save filters</button>
<button type="button" class="ghost" @click="bulkTagMatching()">Bulk tag matching</button> <button type="button" class="ghost" @click="bulkTagMatching()">Bulk tag matching</button>
<button type="button" class="ghost" @click="exportJSON()">Export JSON</button> <button type="button" class="ghost" @click="exportJSON()">Export JSON</button>
<button type="button" class="ghost" @click="exportCSV()">Export CSV</button> <button type="button" class="ghost" @click="exportCSV()">Export CSV</button>
</div> </div>
</div> </div>
<div class="filter-row" x-show="savedSearches.length">
<div class="saved-searches">
<span>Saved:</span>
<template x-for="ss in savedSearches" :key="ss.id">
<span class="pill pill--tag" style="cursor:pointer;" @click="applySavedSearch(ss)">
<span x-text="ss.name"></span>
<button type="button" class="link" style="margin-left:4px;" @click.stop="deleteSavedSearch(ss.id)">×</button>
</span>
</template>
</div>
</div>
</form> </form>
</section> </section>
<section class="panel" x-show="aiFeaturesEnabled">
<h3>Ask a question</h3>
<form class="ask-form" @submit.prevent="askQuestion()">
<div class="ask-row">
<input
type="text"
placeholder="What happened to device ABC123 in the last 3 days?"
x-model="askQuestionText"
class="ask-input"
/>
<button type="submit" :disabled="askLoading" x-text="askLoading ? 'Thinking…' : 'Ask'">Ask</button>
</div>
<div x-show="hasActiveFilters()" class="ask-filter-hint">
<small>Respecting active filters: <span x-text="activeFilterSummary()"></span></small>
</div>
</form>
<template x-if="askAnswer">
<div class="ask-result">
<div x-show="askLlmError" class="ask-error" x-text="askLlmError"></div>
<div class="ask-answer" x-html="askAnswerHtml"></div>
<template x-if="askEvents.length">
<div class="ask-events">
<h4>Referenced events</h4>
<template x-for="(evt, idx) in askEvents" :key="evt.id || idx">
<article class="event event--compact">
<div class="event__meta">
<span class="pill" x-text="evt.display_category || evt.service || '—'"></span>
<span class="pill" :class="['success','succeeded','ok','passed','true'].includes((evt.result || '').toLowerCase()) ? 'pill--ok' : 'pill--warn'" x-text="evt.result || '—'"></span>
</div>
<h3 x-text="evt.operation || '—'"></h3>
<p class="event__detail" x-show="evt.display_summary"><strong>Summary:</strong> <span x-text="evt.display_summary"></span></p>
<p class="event__detail"><strong>Actor:</strong> <span x-text="evt.actor_display || '—'"></span></p>
<p class="event__detail"><strong>Target:</strong> <span x-text="Array.isArray(evt.target_displays) ? evt.target_displays.join(', ') : '—'"></span></p>
<p class="event__detail"><strong>When:</strong> <span x-text="evt.timestamp ? new Date(evt.timestamp).toLocaleString() : '—'"></span></p>
</article>
</template>
</div>
</template>
<button type="button" class="ghost" @click="clearAsk()">Clear</button>
</div>
</template>
</section>
<section class="panel"> <section class="panel">
<div class="panel-header"> <div class="panel-header">
<h2>Events</h2> <h2>Events</h2>
@@ -174,7 +186,7 @@
<article class="event"> <article class="event">
<div class="event__meta"> <div class="event__meta">
<span class="pill" x-text="evt.display_category || evt.service || '—'"></span> <span class="pill" x-text="evt.display_category || evt.service || '—'"></span>
<span class="pill" :class="['success','succeeded','ok','passed'].includes((evt.result || '').toLowerCase()) ? 'pill--ok' : 'pill--warn'" x-text="evt.result || '—'"></span> <span class="pill" :class="['success','succeeded','ok','passed','true'].includes((evt.result || '').toLowerCase()) ? 'pill--ok' : 'pill--warn'" x-text="evt.result || '—'"></span>
</div> </div>
<h3 x-text="evt.operation || '—'"></h3> <h3 x-text="evt.operation || '—'"></h3>
<p class="event__detail" x-show="evt.display_summary"><strong>Summary:</strong> <span x-text="evt.display_summary"></span></p> <p class="event__detail" x-show="evt.display_summary"><strong>Summary:</strong> <span x-text="evt.display_summary"></span></p>
@@ -214,7 +226,15 @@
<div class="modal__content"> <div class="modal__content">
<div class="modal__header"> <div class="modal__header">
<h3 id="modalTitle">Raw Event</h3> <h3 id="modalTitle">Raw Event</h3>
<button type="button" id="closeModal" class="ghost" @click="modalOpen = false">Close</button> <div class="modal__actions">
<button type="button" class="ghost" @click="copyRawEvent()">Copy</button>
<button type="button" class="ghost" x-show="aiFeaturesEnabled" :disabled="modalExplainLoading" @click="explainEvent()" x-text="modalExplainLoading ? 'Explaining…' : 'Explain'">Explain</button>
<button type="button" id="closeModal" class="ghost" @click="modalOpen = false">Close</button>
</div>
</div>
<div x-show="modalExplanation || modalExplainError" class="modal__explanation">
<div x-show="modalExplainError" class="ask-error" x-text="modalExplainError"></div>
<div x-show="modalExplanation" class="ask-answer" x-html="_mdToHtml(modalExplanation)"></div>
</div> </div>
<pre id="modalBody" x-text="modalBody"></pre> <pre id="modalBody" x-text="modalBody"></pre>
</div> </div>
@@ -233,6 +253,10 @@
currentCursor: null, currentCursor: null,
modalOpen: false, modalOpen: false,
modalBody: '', modalBody: '',
modalEventId: '',
modalExplanation: '',
modalExplainLoading: false,
modalExplainError: '',
authBtnText: 'Login', authBtnText: 'Login',
authConfig: null, authConfig: null,
msalInstance: null, msalInstance: null,
@@ -243,7 +267,9 @@
actor: '', selectedServices: [], search: '', operation: '', result: '', start: '', end: '', limit: 100, includeTags: '', excludeTags: '', actor: '', selectedServices: [], search: '', operation: '', result: '', start: '', end: '', limit: 100, includeTags: '', excludeTags: '',
}, },
options: { actors: [], services: [], operations: [], results: [] }, options: { actors: [], services: [], operations: [], results: [] },
savedSearches: [],
appVersion: '', appVersion: '',
aiFeaturesEnabled: true,
askQuestionText: '', askQuestionText: '',
askLoading: false, askLoading: false,
askAnswer: '', askAnswer: '',
@@ -255,13 +281,33 @@
async initApp() { async initApp() {
await this.loadVersion(); await this.loadVersion();
await this.initAuth(); await this.initAuth();
this.loadSavedFilters();
if (!this.authConfig?.auth_enabled || this.accessToken) { if (!this.authConfig?.auth_enabled || this.accessToken) {
await this.loadFilterOptions(); await this.loadFilterOptions();
await this.loadSavedSearches();
await this.loadSourceHealth(); await this.loadSourceHealth();
await this.loadEvents(); await this.loadEvents();
} }
}, },
loadSavedFilters() {
try {
const saved = localStorage.getItem('aoc_filters');
if (!saved) return;
const parsed = JSON.parse(saved);
const fields = ['actor', 'selectedServices', 'search', 'operation', 'result', 'start', 'end', 'limit', 'includeTags', 'excludeTags'];
fields.forEach((f) => {
if (parsed[f] !== undefined) this.filters[f] = parsed[f];
});
} catch {}
},
saveFilters() {
try {
localStorage.setItem('aoc_filters', JSON.stringify(this.filters));
} catch {}
},
async loadVersion() { async loadVersion() {
try { try {
const res = await fetch('/api/version'); const res = await fetch('/api/version');
@@ -302,6 +348,18 @@
this.authConfig = { auth_enabled: false }; this.authConfig = { auth_enabled: false };
} }
try {
const featRes = await fetch('/api/config/features');
if (featRes.ok) {
const featBody = await featRes.json();
this.aiFeaturesEnabled = featBody.ai_features_enabled !== false;
} else {
this.aiFeaturesEnabled = true;
}
} catch {
this.aiFeaturesEnabled = true;
}
if (!this.authConfig?.auth_enabled) { if (!this.authConfig?.auth_enabled) {
this.authBtnText = ''; this.authBtnText = '';
return; return;
@@ -424,6 +482,7 @@
this.nextCursor = body.next_cursor || null; this.nextCursor = body.next_cursor || null;
this.countText = body.total >= 0 ? `${body.total} event${body.total === 1 ? '' : 's'}` : ''; this.countText = body.total >= 0 ? `${body.total} event${body.total === 1 ? '' : 's'}` : '';
this.statusText = this.events.length ? '' : 'No events found for these filters.'; this.statusText = this.events.length ? '' : 'No events found for these filters.';
this.saveFilters();
} catch (err) { } catch (err) {
this.statusText = err.message || 'Failed to load events.'; this.statusText = err.message || 'Failed to load events.';
} }
@@ -459,8 +518,19 @@
this.options.services = (opts.services || []).slice(0, 200); this.options.services = (opts.services || []).slice(0, 200);
this.options.operations = (opts.operations || []).slice(0, 200); this.options.operations = (opts.operations || []).slice(0, 200);
this.options.results = (opts.results || []).slice(0, 200); this.options.results = (opts.results || []).slice(0, 200);
if (!this.filters.selectedServices.length && this.options.services.length) {
this.filters.selectedServices = [...this.options.services]; const saved = localStorage.getItem('aoc_filters');
if (!saved && this.options.services.length) {
// Default: exclude noisy high-volume services
const noisy = ['Exchange', 'SharePoint', 'Teams'];
this.filters.selectedServices = this.options.services.filter((s) => !noisy.includes(s));
} else if (saved) {
try {
const parsed = JSON.parse(saved);
if (parsed.selectedServices) {
this.filters.selectedServices = parsed.selectedServices.filter((s) => this.options.services.includes(s));
}
} catch {}
} }
} catch {} } catch {}
}, },
@@ -473,6 +543,59 @@
} catch {} } catch {}
}, },
async loadSavedSearches() {
try {
const res = await fetch('/api/saved-searches', { headers: this.authHeader() });
if (!res.ok) return;
this.savedSearches = await res.json();
} catch {}
},
async saveCurrentFilters() {
const name = prompt('Name this saved filter:');
if (!name || !name.trim()) return;
try {
const res = await fetch('/api/saved-searches', {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...this.authHeader() },
body: JSON.stringify({ name: name.trim(), filters: { ...this.filters } }),
});
if (!res.ok) throw new Error(await res.text());
const created = await res.json();
this.savedSearches.unshift(created);
this.statusText = 'Filters saved.';
setTimeout(() => { if (this.statusText === 'Filters saved.') this.statusText = ''; }, 2000);
} catch (err) {
this.statusText = err.message || 'Failed to save filters.';
}
},
applySavedSearch(ss) {
if (!ss || !ss.filters) return;
const fields = ['actor', 'selectedServices', 'search', 'operation', 'result', 'start', 'end', 'limit', 'includeTags', 'excludeTags'];
fields.forEach((f) => {
if (ss.filters[f] !== undefined) this.filters[f] = ss.filters[f];
});
// Validate selectedServices against current options
this.filters.selectedServices = this.filters.selectedServices.filter((s) => this.options.services.includes(s));
this.resetPagination();
this.loadEvents();
},
async deleteSavedSearch(id) {
if (!confirm('Delete this saved search?')) return;
try {
const res = await fetch(`/api/saved-searches/${id}`, {
method: 'DELETE',
headers: this.authHeader(),
});
if (!res.ok) throw new Error(await res.text());
this.savedSearches = this.savedSearches.filter((s) => s.id !== id);
} catch (err) {
this.statusText = err.message || 'Failed to delete saved search.';
}
},
resetPagination() { resetPagination() {
this.cursorStack = []; this.cursorStack = [];
this.nextCursor = null; this.nextCursor = null;
@@ -494,7 +617,9 @@
}, },
clearFilters() { clearFilters() {
this.filters = { actor: '', selectedServices: [...this.options.services], search: '', operation: '', result: '', start: '', end: '', limit: 100, includeTags: '', excludeTags: '' }; const noisy = ['Exchange', 'SharePoint', 'Teams'];
this.filters = { actor: '', selectedServices: this.options.services.filter((s) => !noisy.includes(s)), search: '', operation: '', result: '', start: '', end: '', limit: 100, includeTags: '', excludeTags: '' };
this.saveFilters();
this.resetPagination(); this.resetPagination();
this.loadEvents(); this.loadEvents();
}, },
@@ -659,9 +784,44 @@
} catch (err) { } catch (err) {
this.modalBody = `Error serializing event:\n${err.message}\n\nEvent ID: ${e.id || 'N/A'}`; this.modalBody = `Error serializing event:\n${err.message}\n\nEvent ID: ${e.id || 'N/A'}`;
} }
this.modalEventId = e.id || '';
this.modalExplanation = '';
this.modalExplainError = '';
this.modalOpen = true; this.modalOpen = true;
}, },
async copyRawEvent() {
if (!this.modalBody) return;
try {
await navigator.clipboard.writeText(this.modalBody);
this.statusText = 'Raw event copied to clipboard.';
setTimeout(() => { if (this.statusText === 'Raw event copied to clipboard.') this.statusText = ''; }, 2000);
} catch (err) {
this.statusText = 'Failed to copy to clipboard.';
}
},
async explainEvent() {
if (!this.modalEventId) return;
this.modalExplainLoading = true;
this.modalExplanation = '';
this.modalExplainError = '';
try {
const res = await fetch(`/api/events/${this.modalEventId}/explain`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...this.authHeader() },
});
if (!res.ok) throw new Error(await res.text());
const body = await res.json();
this.modalExplanation = body.explanation;
this.modalExplainError = body.llm_error || '';
} catch (err) {
this.modalExplainError = err.message || 'Failed to explain event.';
} finally {
this.modalExplainLoading = false;
}
},
async addTag(e, tag) { async addTag(e, tag) {
if (!tag.trim()) return; if (!tag.trim()) return;
const tags = [...(e.tags || []), tag.trim()]; const tags = [...(e.tags || []), tag.trim()];

View File

@@ -364,6 +364,30 @@ input {
margin-bottom: 10px; margin-bottom: 10px;
} }
.modal__actions {
display: flex;
gap: 8px;
align-items: center;
}
.saved-searches {
display: flex;
flex-wrap: wrap;
gap: 8px;
align-items: center;
font-size: 13px;
}
.modal__explanation {
background: rgba(255, 255, 255, 0.03);
border: 1px solid var(--border);
border-radius: 10px;
padding: 12px;
margin-bottom: 10px;
font-size: 14px;
line-height: 1.6;
}
.modal pre { .modal pre {
background: rgba(255, 255, 255, 0.02); background: rgba(255, 255, 255, 0.02);
color: var(--text); color: var(--text);

View File

@@ -6,7 +6,7 @@ from pathlib import Path
import structlog import structlog
from audit_trail import log_action from audit_trail import log_action
from config import CORS_ORIGINS, ENABLE_PERIODIC_FETCH, FETCH_INTERVAL_MINUTES from config import AI_FEATURES_ENABLED, CORS_ORIGINS, ENABLE_PERIODIC_FETCH, FETCH_INTERVAL_MINUTES
from database import setup_indexes from database import setup_indexes
from fastapi import FastAPI, HTTPException, Request from fastapi import FastAPI, HTTPException, Request
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
@@ -14,13 +14,13 @@ from fastapi.responses import Response
from fastapi.staticfiles import StaticFiles from fastapi.staticfiles import StaticFiles
from metrics import observe_request, prometheus_metrics from metrics import observe_request, prometheus_metrics
from middleware import CorrelationIdMiddleware from middleware import CorrelationIdMiddleware
from routes.ask import router as ask_router
from routes.config import router as config_router from routes.config import router as config_router
from routes.events import router as events_router from routes.events import router as events_router
from routes.fetch import router as fetch_router from routes.fetch import router as fetch_router
from routes.fetch import run_fetch from routes.fetch import run_fetch
from routes.health import router as health_router from routes.health import router as health_router
from routes.rules import router as rules_router from routes.rules import router as rules_router
from routes.saved_searches import router as saved_searches_router
from routes.webhooks import router as webhooks_router from routes.webhooks import router as webhooks_router
@@ -113,7 +113,14 @@ app.include_router(events_router, prefix="/api")
app.include_router(config_router, prefix="/api") app.include_router(config_router, prefix="/api")
app.include_router(webhooks_router, prefix="/api") app.include_router(webhooks_router, prefix="/api")
app.include_router(health_router, prefix="/api") app.include_router(health_router, prefix="/api")
app.include_router(ask_router, prefix="/api") if AI_FEATURES_ENABLED:
from routes.ask import router as ask_router
app.include_router(ask_router, prefix="/api")
from routes.mcp import mcp_asgi
app.mount("/mcp", mcp_asgi)
app.include_router(saved_searches_router, prefix="/api")
app.include_router(rules_router, prefix="/api") app.include_router(rules_router, prefix="/api")

187
backend/mcp_common.py Normal file
View File

@@ -0,0 +1,187 @@
"""Shared MCP tool handlers used by both stdio and SSE transports."""
import json
from datetime import UTC, datetime, timedelta
from database import events_collection
from mcp.types import TextContent
async def handle_search_events(arguments: dict) -> list[TextContent]:
days = arguments.get("days", 7)
limit = min(arguments.get("limit", 20), 100)
since = (datetime.now(UTC) - timedelta(days=days)).isoformat().replace("+00:00", "Z")
filters = [{"timestamp": {"$gte": since}}]
services = arguments.get("services")
if services:
filters.append({"service": {"$in": services}})
operation = arguments.get("operation")
if operation:
filters.append({"operation": {"$regex": operation, "$options": "i"}})
result = arguments.get("result")
if result:
filters.append({"result": {"$regex": result, "$options": "i"}})
entity = arguments.get("entity")
if entity:
entity_safe = entity.replace(".", "\\.").replace("(", "\\(").replace(")", "\\)")
filters.append(
{
"$or": [
{"target_displays": {"$elemMatch": {"$regex": entity_safe, "$options": "i"}}},
{"actor_display": {"$regex": entity_safe, "$options": "i"}},
{"actor_upn": {"$regex": entity_safe, "$options": "i"}},
{"raw_text": {"$regex": entity_safe, "$options": "i"}},
]
}
)
query = {"$and": filters}
cursor = events_collection.find(query).sort("timestamp", -1).limit(limit)
events = list(cursor)
if not events:
return [TextContent(type="text", text="No matching events found.")]
lines = [f"Found {len(events)} event(s):\n"]
for e in events:
ts = e.get("timestamp", "?")[:16].replace("T", " ")
svc = e.get("service", "?")
op = e.get("operation", "?")
actor = e.get("actor_display", "?")
result_str = e.get("result", "?")
lines.append(f"{ts} | {svc} | {op} | {actor} | {result_str}")
return [TextContent(type="text", text="\n".join(lines))]
async def handle_get_event(arguments: dict) -> list[TextContent]:
event_id = arguments["event_id"]
event = events_collection.find_one({"id": event_id})
if not event:
return [TextContent(type="text", text=f"Event {event_id} not found.")]
event.pop("_id", None)
return [TextContent(type="text", text=json.dumps(event, indent=2, default=str))]
async def handle_get_summary(arguments: dict) -> list[TextContent]:
days = arguments.get("days", 7)
since = (datetime.now(UTC) - timedelta(days=days)).isoformat().replace("+00:00", "Z")
query = {"timestamp": {"$gte": since}}
total = events_collection.count_documents(query)
if total == 0:
return [TextContent(type="text", text="No events in the specified period.")]
svc_pipeline = [
{"$match": query},
{"$group": {"_id": "$service", "count": {"$sum": 1}}},
{"$sort": {"count": -1}},
{"$limit": 10},
]
op_pipeline = [
{"$match": query},
{"$group": {"_id": "$operation", "count": {"$sum": 1}}},
{"$sort": {"count": -1}},
{"$limit": 10},
]
result_pipeline = [
{"$match": query},
{"$group": {"_id": "$result", "count": {"$sum": 1}}},
{"$sort": {"count": -1}},
]
actor_pipeline = [
{"$match": query},
{"$group": {"_id": "$actor_display", "count": {"$sum": 1}}},
{"$sort": {"count": -1}},
{"$limit": 10},
]
svc_counts = list(events_collection.aggregate(svc_pipeline))
op_counts = list(events_collection.aggregate(op_pipeline))
result_counts = list(events_collection.aggregate(result_pipeline))
actor_counts = list(events_collection.aggregate(actor_pipeline))
lines = [f"Summary for the last {days} days ({total} total events)\n"]
lines.append("By service:")
for row in svc_counts:
lines.append(f" {row['_id'] or 'Unknown'}: {row['count']}")
lines.append("\nBy action:")
for row in op_counts:
lines.append(f" {row['_id'] or 'Unknown'}: {row['count']}")
lines.append("\nBy result:")
for row in result_counts:
lines.append(f" {row['_id'] or 'Unknown'}: {row['count']}")
lines.append("\nTop actors:")
for row in actor_counts:
lines.append(f" {row['_id'] or 'Unknown'}: {row['count']}")
return [TextContent(type="text", text="\n".join(lines))]
async def handle_ask(arguments: dict) -> list[TextContent]:
"""For now, returns recent events + guidance. In the future this could call the LLM backend."""
question = arguments["question"]
days = arguments.get("days", 7)
result = await handle_search_events({"entity": "", "days": days, "limit": 50})
base_text = result[0].text if result else ""
text = (
f"You asked: '{question}'\n\n"
f"Here are the most recent events from the last {days} days:\n\n"
f"{base_text}\n\n"
f"Tip: Use the 'search_events' tool with specific filters "
f"to narrow down the dataset before asking follow-up questions."
)
return [TextContent(type="text", text=text)]
# JSON schemas for tool definitions
SEARCH_EVENTS_SCHEMA = {
"type": "object",
"properties": {
"entity": {"type": "string", "description": "Device name, user UPN, or email to search for"},
"services": {
"type": "array",
"items": {"type": "string"},
"description": "Filter by service (e.g. Intune, Directory, Exchange)",
},
"operation": {"type": "string", "description": "Filter by operation name"},
"result": {"type": "string", "description": "Filter by result (success, failure)"},
"days": {"type": "integer", "description": "Number of days to look back (default 7)"},
"limit": {"type": "integer", "description": "Max events to return (default 20)"},
},
}
GET_EVENT_SCHEMA = {
"type": "object",
"properties": {
"event_id": {"type": "string", "description": "The event ID to retrieve"},
},
"required": ["event_id"],
}
GET_SUMMARY_SCHEMA = {
"type": "object",
"properties": {
"days": {"type": "integer", "description": "Number of days to summarise (default 7)"},
},
}
ASK_SCHEMA = {
"type": "object",
"properties": {
"question": {"type": "string", "description": "Natural language question about audit logs"},
"days": {"type": "integer", "description": "Number of days to look back (default 7)"},
},
"required": ["question"],
}

88
backend/mcp_server.py Normal file
View File

@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""
AOC MCP Server — stdio transport
Standalone MCP server for local use (Claude Desktop, Cursor, etc.).
For the HTTP/SSE version (production, behind auth), see routes/mcp.py.
Usage:
python mcp_server.py
Claude Desktop config (~/.config/claude/claude_desktop_config.json):
{
"mcpServers": {
"aoc": {
"command": "python",
"args": ["/path/to/aoc/backend/mcp_server.py"],
"env": {"MONGO_URI": "mongodb://..."}
}
}
}
"""
import asyncio
import os
import sys
# Ensure backend modules are importable when run standalone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import TextContent, Tool
from mcp_common import (
ASK_SCHEMA,
GET_EVENT_SCHEMA,
GET_SUMMARY_SCHEMA,
SEARCH_EVENTS_SCHEMA,
handle_ask,
handle_get_event,
handle_get_summary,
handle_search_events,
)
app = Server("aoc")
@app.list_tools()
async def list_tools() -> list[Tool]:
return [
Tool(
name="search_events",
description="Search audit events by entity, service, operation, or result.",
inputSchema=SEARCH_EVENTS_SCHEMA,
),
Tool(name="get_event", description="Retrieve a single audit event by its ID.", inputSchema=GET_EVENT_SCHEMA),
Tool(
name="get_summary",
description="Get an aggregated summary of audit activity for the last N days.",
inputSchema=GET_SUMMARY_SCHEMA,
),
Tool(
name="ask",
description="Ask a natural language question about audit logs. Returns a narrative answer.",
inputSchema=ASK_SCHEMA,
),
]
@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if name == "search_events":
return await handle_search_events(arguments)
if name == "get_event":
return await handle_get_event(arguments)
if name == "get_summary":
return await handle_get_summary(arguments)
if name == "ask":
return await handle_ask(arguments)
raise ValueError(f"Unknown tool: {name}")
async def main():
async with stdio_server() as (read_stream, write_stream):
await app.run(read_stream, write_stream, app.create_initialization_options())
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -13,3 +13,4 @@ tenacity
prometheus-client prometheus-client
httpx httpx
gunicorn gunicorn
mcp

View File

@@ -1,11 +1,21 @@
import asyncio
import json import json
import re import re
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
import httpx import httpx
import structlog import structlog
from auth import require_auth from auth import require_auth, user_can_access_privacy_services
from config import LLM_API_KEY, LLM_API_VERSION, LLM_BASE_URL, LLM_MAX_EVENTS, LLM_MODEL, LLM_TIMEOUT_SECONDS from config import (
LLM_API_KEY,
LLM_API_VERSION,
LLM_BASE_URL,
LLM_MAX_EVENTS,
LLM_MODEL,
LLM_TIMEOUT_SECONDS,
PRIVACY_SENSITIVE_OPERATIONS,
PRIVACY_SERVICES,
)
from database import events_collection from database import events_collection
from fastapi import APIRouter, Depends, HTTPException from fastapi import APIRouter, Depends, HTTPException
from models.api import AskRequest, AskResponse from models.api import AskRequest, AskResponse
@@ -49,7 +59,7 @@ _SERVICE_INTENTS = {
# Services that are extremely noisy for typical admin questions. # Services that are extremely noisy for typical admin questions.
# We exclude them by default on broad questions unless the user explicitly mentions them. # We exclude them by default on broad questions unless the user explicitly mentions them.
_NOISY_SERVICES = {"Exchange", "SharePoint"} _NOISY_SERVICES = {"Exchange", "SharePoint", "Teams"}
# Services that are generally admin-relevant and kept by default. # Services that are generally admin-relevant and kept by default.
_DEFAULT_ADMIN_SERVICES = { _DEFAULT_ADMIN_SERVICES = {
@@ -456,6 +466,198 @@ def _to_event_ref(e: dict) -> dict:
} }
_EXPLAIN_SYSTEM_PROMPT = """You are a Microsoft 365 security and compliance expert.
An administrator needs help understanding an audit event.
Your task:
1. Explain what happened in plain language (1-2 sentences).
2. Identify who performed the action and what was the target.
3. Assess whether this is typical admin activity or something to investigate.
4. Highlight any security implications (privilege escalation, unusual actor, after-hours activity, etc.).
5. Suggest what the admin should do next, if anything.
Keep the answer under 200 words. Use bullet points for readability.
Do not invent facts that are not in the data.
"""
_GUID_RE = re.compile(r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$")
def _extract_guids(obj: dict | list | str) -> set[str]:
"""Recursively extract UUID-like strings from a JSON structure."""
guids = set()
if isinstance(obj, dict):
for k, v in obj.items():
if k.lower() in ("id", "groupid", "userid", "targetid") and isinstance(v, str) and _GUID_RE.match(v):
guids.add(v)
guids.update(_extract_guids(v))
elif isinstance(obj, list):
for item in obj:
guids.update(_extract_guids(item))
elif isinstance(obj, str) and _GUID_RE.match(obj):
guids.add(obj)
return guids
async def _resolve_guids_for_event(event: dict) -> dict[str, str]:
"""Try to resolve GUIDs in an event to human-readable names via Graph API."""
raw = event.get("raw") or {}
guids = _extract_guids(raw)
# Also include any GUIDs in targetResources that might not have displayName
for tr in raw.get("targetResources") or []:
tid = tr.get("id")
if tid and _GUID_RE.match(tid):
guids.add(tid)
for tr in raw.get("modifiedProperties") or []:
for key in ("oldValue", "newValue"):
val = tr.get(key)
if val and _GUID_RE.match(val):
guids.add(val)
if not guids:
return {}
try:
from graph.auth import get_access_token
from graph.resolve import resolve_directory_object
token = await asyncio.to_thread(get_access_token)
cache: dict[str, dict] = {}
resolved = {}
for gid in guids:
result = await asyncio.to_thread(resolve_directory_object, gid, token, cache)
if result:
resolved[gid] = result["name"]
return resolved
except Exception as exc:
logger.warning("GUID resolution failed", error=str(exc))
return {}
async def _explain_event(event: dict, related: list[dict]) -> str:
if not LLM_API_KEY:
raise RuntimeError("LLM_API_KEY not configured")
# Resolve GUIDs to names before sending to LLM
resolved = await _resolve_guids_for_event(event)
event_text = json.dumps(event, indent=2, default=str)
resolution_text = ""
if resolved:
resolution_text = "\nResolved GUIDs:\n"
for gid, name in resolved.items():
resolution_text += f" {gid}{name}\n"
related_text = ""
if related:
related_text = "\n\nRelated events in the last 24 hours:\n"
for i, e in enumerate(related[:10], 1):
ts = e.get("timestamp", "?")[:16].replace("T", " ")
op = e.get("operation", "unknown")
actor = e.get("actor_display", "unknown")
targets = ", ".join(e.get("target_displays") or []) or ""
result = e.get("result", "")
related_text += f"{i}. {ts}{op} by {actor} on {targets} ({result})\n"
messages = [
{"role": "system", "content": _EXPLAIN_SYSTEM_PROMPT},
{
"role": "user",
"content": f"Audit event:\n{event_text}{resolution_text}{related_text}\n\nPlease explain this event.",
},
]
url = _build_chat_url(LLM_BASE_URL, LLM_API_VERSION)
headers = {"Content-Type": "application/json"}
if "azure" in LLM_BASE_URL.lower() or "cognitiveservices" in LLM_BASE_URL.lower():
headers["api-key"] = LLM_API_KEY
else:
headers["Authorization"] = f"Bearer {LLM_API_KEY}"
payload = {
"model": LLM_MODEL,
"messages": messages,
"max_completion_tokens": 600,
}
async with httpx.AsyncClient(timeout=LLM_TIMEOUT_SECONDS) as client:
resp = await client.post(url, headers=headers, json=payload)
if resp.status_code >= 400:
body = resp.text
logger.error("LLM API error", status_code=resp.status_code, url=url, response_body=body)
raise RuntimeError(f"LLM API error {resp.status_code}: {body[:500]}")
data = resp.json()
return data["choices"][0]["message"]["content"].strip()
@router.post("/events/{event_id}/explain")
async def explain_event(event_id: str, user: dict = Depends(require_auth)):
event = events_collection.find_one({"id": event_id})
if not event:
raise HTTPException(status_code=404, detail="Event not found")
if (
event.get("service") in PRIVACY_SERVICES or event.get("operation") in PRIVACY_SENSITIVE_OPERATIONS
) and not user_can_access_privacy_services(user):
raise HTTPException(status_code=403, detail="Access to this event is restricted")
event.pop("_id", None)
# Fetch related events for context (same actor or target in last 24h)
related = []
since = (datetime.now(UTC) - timedelta(hours=24)).isoformat().replace("+00:00", "Z")
actor = event.get("actor_upn") or event.get("actor_display")
target = event.get("target_displays", [None])[0] if event.get("target_displays") else None
or_filters = [{"timestamp": {"$gte": since}}, {"id": {"$ne": event_id}}]
if actor:
or_filters.append(
{
"$or": [
{"actor_upn": actor},
{"actor_display": actor},
]
}
)
if target:
or_filters.append({"target_displays": target})
if len(or_filters) > 2:
try:
rel_cursor = events_collection.find({"$and": or_filters}).sort("timestamp", -1).limit(10)
related = list(rel_cursor)
for r in related:
r.pop("_id", None)
r.pop("raw", None)
except Exception as exc:
logger.warning("Failed to fetch related events", error=str(exc))
if not LLM_API_KEY:
return {
"explanation": "LLM is not configured. Set LLM_API_KEY in your environment to enable event explanations.",
"llm_used": False,
"llm_error": "LLM_API_KEY not configured",
}
try:
explanation = await _explain_event(event, related)
return {
"explanation": explanation,
"llm_used": True,
"llm_error": None,
"related_count": len(related),
}
except Exception as exc:
logger.warning("Event explanation failed", error=str(exc))
return {
"explanation": "Unable to generate an explanation at this time. Please check the raw event details.",
"llm_used": False,
"llm_error": str(exc),
"related_count": len(related),
}
@router.post("/ask", response_model=AskResponse) @router.post("/ask", response_model=AskResponse)
async def ask_question(body: AskRequest, user: dict = Depends(require_auth)): async def ask_question(body: AskRequest, user: dict = Depends(require_auth)):
question = body.question.strip() question = body.question.strip()
@@ -490,6 +692,8 @@ async def ask_question(body: AskRequest, user: dict = Depends(require_auth)):
# ----------------------------------------------------------------------- # -----------------------------------------------------------------------
# Build and run query # Build and run query
# ----------------------------------------------------------------------- # -----------------------------------------------------------------------
privacy_excluded_services = [] if user_can_access_privacy_services(user) else list(PRIVACY_SERVICES)
privacy_excluded_ops = [] if user_can_access_privacy_services(user) else list(PRIVACY_SENSITIVE_OPERATIONS)
query = _build_event_query( query = _build_event_query(
entity, entity,
start, start,
@@ -501,6 +705,13 @@ async def ask_question(body: AskRequest, user: dict = Depends(require_auth)):
include_tags=body.include_tags, include_tags=body.include_tags,
exclude_tags=body.exclude_tags, exclude_tags=body.exclude_tags,
) )
extra_filters = []
if privacy_excluded_services:
extra_filters.append({"service": {"$nin": privacy_excluded_services}})
if privacy_excluded_ops:
extra_filters.append({"operation": {"$nin": privacy_excluded_ops}})
if extra_filters:
query["$and"] = query.get("$and", []) + extra_filters
try: try:
total = events_collection.count_documents(query) total = events_collection.count_documents(query)

View File

@@ -1,4 +1,5 @@
from config import ( from config import (
AI_FEATURES_ENABLED,
AUTH_CLIENT_ID, AUTH_CLIENT_ID,
AUTH_ENABLED, AUTH_ENABLED,
AUTH_SCOPE, AUTH_SCOPE,
@@ -18,3 +19,10 @@ def auth_config():
"scope": AUTH_SCOPE, "scope": AUTH_SCOPE,
"redirect_uri": None, # frontend uses window.location.origin by default "redirect_uri": None, # frontend uses window.location.origin by default
} }
@router.get("/config/features")
def features_config():
return {
"ai_features_enabled": AI_FEATURES_ENABLED,
}

View File

@@ -3,8 +3,9 @@ import re
from datetime import UTC, datetime from datetime import UTC, datetime
from audit_trail import log_action from audit_trail import log_action
from auth import require_auth from auth import require_auth, user_can_access_privacy_services
from bson import ObjectId from bson import ObjectId
from config import PRIVACY_SENSITIVE_OPERATIONS, PRIVACY_SERVICES
from database import events_collection from database import events_collection
from fastapi import APIRouter, Depends, HTTPException, Query from fastapi import APIRouter, Depends, HTTPException, Query
from models.api import ( from models.api import (
@@ -44,6 +45,7 @@ def _build_query(
cursor: str | None = None, cursor: str | None = None,
include_tags: list[str] | None = None, include_tags: list[str] | None = None,
exclude_tags: list[str] | None = None, exclude_tags: list[str] | None = None,
exclude_operations: list[str] | None = None,
) -> dict: ) -> dict:
filters = [] filters = []
@@ -51,6 +53,8 @@ def _build_query(
filters.append({"service": service}) filters.append({"service": service})
if services: if services:
filters.append({"service": {"$in": services}}) filters.append({"service": {"$in": services}})
if exclude_operations:
filters.append({"operation": {"$nin": exclude_operations}})
if actor: if actor:
actor_safe = re.escape(actor) actor_safe = re.escape(actor)
filters.append( filters.append(
@@ -125,6 +129,8 @@ def list_events(
exclude_tags: list[str] | None = Query(default=None), exclude_tags: list[str] | None = Query(default=None),
user: dict = Depends(require_auth), user: dict = Depends(require_auth),
): ):
privacy_excluded_services = [] if user_can_access_privacy_services(user) else list(PRIVACY_SERVICES)
privacy_excluded_ops = [] if user_can_access_privacy_services(user) else list(PRIVACY_SENSITIVE_OPERATIONS)
query = _build_query( query = _build_query(
service=service, service=service,
services=services, services=services,
@@ -137,7 +143,13 @@ def list_events(
cursor=cursor, cursor=cursor,
include_tags=include_tags, include_tags=include_tags,
exclude_tags=exclude_tags, exclude_tags=exclude_tags,
exclude_operations=privacy_excluded_ops,
) )
if privacy_excluded_services:
query = query if query else {}
if "$and" not in query:
query = {"$and": [query]} if query else {"$and": []}
query["$and"].append({"service": {"$nin": privacy_excluded_services}})
safe_page_size = max(1, min(page_size, 500)) safe_page_size = max(1, min(page_size, 500))
@@ -202,6 +214,8 @@ def bulk_tags(
exclude_tags: list[str] | None = Query(default=None), exclude_tags: list[str] | None = Query(default=None),
user: dict = Depends(require_auth), user: dict = Depends(require_auth),
): ):
privacy_excluded_services = [] if user_can_access_privacy_services(user) else list(PRIVACY_SERVICES)
privacy_excluded_ops = [] if user_can_access_privacy_services(user) else list(PRIVACY_SENSITIVE_OPERATIONS)
query = _build_query( query = _build_query(
service=service, service=service,
services=services, services=services,
@@ -213,7 +227,13 @@ def bulk_tags(
search=search, search=search,
include_tags=include_tags, include_tags=include_tags,
exclude_tags=exclude_tags, exclude_tags=exclude_tags,
exclude_operations=privacy_excluded_ops,
) )
if privacy_excluded_services:
query = query if query else {}
if "$and" not in query:
query = {"$and": [query]} if query else {"$and": []}
query["$and"].append({"service": {"$nin": privacy_excluded_services}})
tags = [t.strip() for t in body.tags if t.strip()] tags = [t.strip() for t in body.tags if t.strip()]
if not tags: if not tags:
raise HTTPException(status_code=400, detail="No tags provided") raise HTTPException(status_code=400, detail="No tags provided")
@@ -235,7 +255,10 @@ def bulk_tags(
@router.get("/filter-options", response_model=FilterOptionsResponse) @router.get("/filter-options", response_model=FilterOptionsResponse)
def filter_options(limit: int = Query(default=200, ge=1, le=1000)): def filter_options(
limit: int = Query(default=200, ge=1, le=1000),
user: dict = Depends(require_auth),
):
safe_limit = max(1, min(limit, 1000)) safe_limit = max(1, min(limit, 1000))
try: try:
services = sorted(events_collection.distinct("service"))[:safe_limit] services = sorted(events_collection.distinct("service"))[:safe_limit]
@@ -247,6 +270,10 @@ def filter_options(limit: int = Query(default=200, ge=1, le=1000)):
except Exception as exc: except Exception as exc:
raise HTTPException(status_code=500, detail=f"Failed to load filter options: {exc}") from exc raise HTTPException(status_code=500, detail=f"Failed to load filter options: {exc}") from exc
if not user_can_access_privacy_services(user):
services = [s for s in services if s not in PRIVACY_SERVICES]
operations = [o for o in operations if o not in PRIVACY_SENSITIVE_OPERATIONS]
return { return {
"services": services, "services": services,
"operations": operations, "operations": operations,

124
backend/routes/mcp.py Normal file
View File

@@ -0,0 +1,124 @@
"""MCP server over SSE (HTTP) transport, mounted inside FastAPI with OIDC auth."""
import structlog
from auth import (
AUTH_ALLOWED_GROUPS,
AUTH_ALLOWED_ROLES,
AUTH_ENABLED,
_allowed,
_decode_token,
_get_jwks,
)
from mcp.server import Server
from mcp.server.sse import SseServerTransport
from mcp.types import TextContent, Tool
from mcp_common import (
ASK_SCHEMA,
GET_EVENT_SCHEMA,
GET_SUMMARY_SCHEMA,
SEARCH_EVENTS_SCHEMA,
handle_ask,
handle_get_event,
handle_get_summary,
handle_search_events,
)
from starlette.requests import Request
from starlette.responses import Response
logger = structlog.get_logger("aoc.mcp")
mcp_app = Server("aoc")
transport = SseServerTransport("/messages/")
@mcp_app.list_tools()
async def list_tools() -> list[Tool]:
return [
Tool(
name="search_events",
description="Search audit events by entity, service, operation, or result.",
inputSchema=SEARCH_EVENTS_SCHEMA,
),
Tool(name="get_event", description="Retrieve a single audit event by its ID.", inputSchema=GET_EVENT_SCHEMA),
Tool(
name="get_summary",
description="Get an aggregated summary of audit activity for the last N days.",
inputSchema=GET_SUMMARY_SCHEMA,
),
Tool(
name="ask",
description="Ask a natural language question about audit logs. Returns a narrative answer.",
inputSchema=ASK_SCHEMA,
),
]
@mcp_app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if name == "search_events":
return await handle_search_events(arguments)
if name == "get_event":
return await handle_get_event(arguments)
if name == "get_summary":
return await handle_get_summary(arguments)
if name == "ask":
return await handle_ask(arguments)
raise ValueError(f"Unknown tool: {name}")
async def _validate_auth(request: Request) -> dict | None:
"""Validate Bearer token. Returns claims dict or None on failure."""
if not AUTH_ENABLED:
return {"sub": "anonymous"}
auth_header = request.headers.get("authorization", "")
if not auth_header or not auth_header.lower().startswith("bearer "):
return None
token = auth_header.split(" ", 1)[1]
try:
jwks = _get_jwks()
claims = _decode_token(token, jwks)
except Exception as exc:
logger.warning("MCP auth failed", error=str(exc))
return None
if not _allowed(claims, AUTH_ALLOWED_ROLES, AUTH_ALLOWED_GROUPS):
logger.warning("MCP auth forbidden", sub=claims.get("sub"))
return None
return claims
async def mcp_asgi(scope: dict, receive, send):
"""ASGI application for MCP over SSE, mounted under /mcp in FastAPI."""
if scope["type"] != "http":
return
request = Request(scope, receive)
# Auth check
claims = await _validate_auth(request)
if claims is None:
response = Response("Unauthorized", status_code=401)
await response(scope, receive, send)
return
path = scope.get("path", "")
root_path = scope.get("root_path", "")
relative_path = path[len(root_path) :] if path.startswith(root_path) else path
method = scope.get("method", "")
if relative_path == "/sse" and method == "GET":
logger.info("MCP SSE connection established", sub=claims.get("sub", "unknown"))
async with transport.connect_sse(scope, receive, send) as (read_stream, write_stream):
await mcp_app.run(
read_stream,
write_stream,
mcp_app.create_initialization_options(),
)
elif relative_path == "/messages/" and method == "POST":
await transport.handle_post_message(scope, receive, send)
else:
response = Response("Not found", status_code=404)
await response(scope, receive, send)

View File

@@ -0,0 +1,60 @@
"""CRUD for saved filter searches (bookmarks)."""
import uuid
from datetime import UTC, datetime
import structlog
from auth import require_auth
from database import saved_searches_collection
from fastapi import APIRouter, Depends, HTTPException
router = APIRouter(dependencies=[Depends(require_auth)])
logger = structlog.get_logger("aoc.saved_searches")
def _user_sub(user: dict) -> str:
return user.get("sub", "anonymous")
@router.get("/saved-searches")
async def list_saved_searches(user: dict = Depends(require_auth)):
"""Return saved searches for the current user."""
sub = _user_sub(user)
cursor = saved_searches_collection.find({"created_by": sub}).sort("created_at", -1)
items = []
for doc in cursor:
doc["id"] = doc.pop("_id")
items.append(doc)
return items
@router.post("/saved-searches")
async def create_saved_search(body: dict, user: dict = Depends(require_auth)):
"""Save the current filter set."""
name = (body.get("name") or "").strip()
if not name:
raise HTTPException(status_code=400, detail="Name is required")
filters = body.get("filters") or {}
doc = {
"_id": str(uuid.uuid4()),
"name": name,
"filters": filters,
"created_at": datetime.now(UTC).isoformat().replace("+00:00", "Z"),
"created_by": _user_sub(user),
}
saved_searches_collection.insert_one(doc)
logger.info("Saved search created", name=name, user=doc["created_by"])
doc["id"] = doc.pop("_id")
return doc
@router.delete("/saved-searches/{search_id}")
async def delete_saved_search(search_id: str, user: dict = Depends(require_auth)):
"""Delete a saved search (only if owned by current user)."""
sub = _user_sub(user)
result = saved_searches_collection.delete_one({"_id": search_id, "created_by": sub})
if result.deleted_count == 0:
raise HTTPException(status_code=404, detail="Saved search not found")
logger.info("Saved search deleted", search_id=search_id, user=sub)
return {"status": "deleted"}

View File

@@ -22,14 +22,23 @@ def mock_watermarks_collection():
@pytest.fixture(scope="function") @pytest.fixture(scope="function")
def client(mock_events_collection, mock_watermarks_collection, monkeypatch): def client(mock_events_collection, mock_watermarks_collection, monkeypatch):
monkeypatch.setattr("database.events_collection", mock_events_collection) monkeypatch.setattr("database.events_collection", mock_events_collection)
monkeypatch.setattr("database.saved_searches_collection", mock_events_collection)
monkeypatch.setattr("routes.fetch.events_collection", mock_events_collection) monkeypatch.setattr("routes.fetch.events_collection", mock_events_collection)
monkeypatch.setattr("routes.events.events_collection", mock_events_collection) monkeypatch.setattr("routes.events.events_collection", mock_events_collection)
monkeypatch.setattr("routes.ask.events_collection", mock_events_collection) monkeypatch.setattr("routes.ask.events_collection", mock_events_collection)
monkeypatch.setattr("routes.saved_searches.saved_searches_collection", mock_events_collection)
monkeypatch.setattr("watermark.watermarks_collection", mock_watermarks_collection) monkeypatch.setattr("watermark.watermarks_collection", mock_watermarks_collection)
monkeypatch.setattr("routes.health.watermarks_collection", mock_watermarks_collection) monkeypatch.setattr("routes.health.watermarks_collection", mock_watermarks_collection)
monkeypatch.setattr("routes.fetch.get_watermark", lambda source: None) monkeypatch.setattr("routes.fetch.get_watermark", lambda source: None)
monkeypatch.setattr("routes.fetch.set_watermark", lambda source, ts: None) monkeypatch.setattr("routes.fetch.set_watermark", lambda source, ts: None)
monkeypatch.setattr("auth.AUTH_ENABLED", False) monkeypatch.setattr("auth.AUTH_ENABLED", False)
monkeypatch.setattr("routes.mcp.AUTH_ENABLED", False)
monkeypatch.setattr("config.PRIVACY_SERVICES", set())
monkeypatch.setattr("config.PRIVACY_SENSITIVE_OPERATIONS", set())
monkeypatch.setattr("routes.events.PRIVACY_SERVICES", set())
monkeypatch.setattr("routes.events.PRIVACY_SENSITIVE_OPERATIONS", set())
monkeypatch.setattr("routes.ask.PRIVACY_SERVICES", set())
monkeypatch.setattr("routes.ask.PRIVACY_SENSITIVE_OPERATIONS", set())
monkeypatch.setattr("database.db.command", lambda cmd: {"ok": 1} if cmd == "ping" else {}) monkeypatch.setattr("database.db.command", lambda cmd: {"ok": 1} if cmd == "ping" else {})
# Mock audit trail and rules collections so tests don't wait on real MongoDB # Mock audit trail and rules collections so tests don't wait on real MongoDB

View File

@@ -1,6 +1,252 @@
from datetime import UTC, datetime from datetime import UTC, datetime
def test_config_features(client):
response = client.get("/api/config/features")
assert response.status_code == 200
data = response.json()
assert "ai_features_enabled" in data
assert isinstance(data["ai_features_enabled"], bool)
def test_ask_disabled_when_ai_features_off():
import subprocess
import sys
code = """
import sys
sys.path.insert(0, '.')
import os
os.environ['AI_FEATURES_ENABLED'] = 'false'
# Re-import config with the env override
import importlib
import config
importlib.reload(config)
# Now import main; it will pick up the new AI_FEATURES_ENABLED
import main
ask_paths = [r.path for r in main.app.routes if hasattr(r, 'path') and 'ask' in r.path]
print('ASK_PATHS:', ask_paths)
assert len(ask_paths) == 0, f"Expected no ask routes, found: {ask_paths}"
print('OK')
"""
result = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, cwd=".")
assert result.returncode == 0, f"Subprocess failed: {result.stdout}\n{result.stderr}"
assert "OK" in result.stdout
def test_mcp_sse_mount_exists():
from main import app
mcp_mounts = [r for r in app.routes if getattr(r, "path", "") == "/mcp"]
assert len(mcp_mounts) == 1, "MCP mount not found in app routes"
def test_mcp_messages_no_session(client):
response = client.post("/mcp/messages/")
# MCP transport returns 400 when session_id is missing, 404 when session not found
assert response.status_code in (400, 404)
def test_mcp_sse_auth_required_when_enabled(client, monkeypatch):
monkeypatch.setattr("routes.mcp.AUTH_ENABLED", True)
response = client.get("/mcp/sse")
assert response.status_code == 401
def test_explain_event_not_found(client):
response = client.post("/api/events/nonexistent/explain")
assert response.status_code == 404
def test_explain_event_no_llm_key(client, mock_events_collection, monkeypatch):
monkeypatch.setattr("routes.ask.LLM_API_KEY", "")
mock_events_collection.insert_one(
{
"id": "evt-explain",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Directory",
"operation": "Add user",
"result": "success",
"actor_display": "Alice",
"raw_text": "",
}
)
response = client.post("/api/events/evt-explain/explain")
assert response.status_code == 200
data = response.json()
assert "explanation" in data
assert data["llm_used"] is False
assert "LLM_API_KEY" in (data.get("llm_error") or "")
def test_explain_event_with_llm_mock(client, mock_events_collection, monkeypatch):
monkeypatch.setattr("routes.ask.LLM_API_KEY", "test-key")
async def fake_explain(event, related):
return "This is a test explanation."
monkeypatch.setattr("routes.ask._explain_event", fake_explain)
mock_events_collection.insert_one(
{
"id": "evt-explain2",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Directory",
"operation": "Add user",
"result": "success",
"actor_display": "Alice",
"raw_text": "",
}
)
response = client.post("/api/events/evt-explain2/explain")
assert response.status_code == 200
data = response.json()
assert data["explanation"] == "This is a test explanation."
assert data["llm_used"] is True
def test_saved_searches_crud(client, monkeypatch):
monkeypatch.setattr("auth.AUTH_ENABLED", False)
# Create
response = client.post(
"/api/saved-searches", json={"name": "Test search", "filters": {"actor": "alice", "result": "success"}}
)
assert response.status_code == 200
created = response.json()
assert created["name"] == "Test search"
assert created["filters"]["actor"] == "alice"
search_id = created["id"]
# List
response2 = client.get("/api/saved-searches")
assert response2.status_code == 200
items = response2.json()
assert len(items) == 1
assert items[0]["name"] == "Test search"
# Delete
response3 = client.delete(f"/api/saved-searches/{search_id}")
assert response3.status_code == 200
# List empty
response4 = client.get("/api/saved-searches")
assert response4.status_code == 200
assert len(response4.json()) == 0
def test_saved_searches_delete_not_found(client, monkeypatch):
monkeypatch.setattr("auth.AUTH_ENABLED", False)
response = client.delete("/api/saved-searches/nonexistent")
assert response.status_code == 404
def test_saved_searches_create_validation(client, monkeypatch):
monkeypatch.setattr("auth.AUTH_ENABLED", False)
response = client.post("/api/saved-searches", json={"name": " ", "filters": {}})
assert response.status_code == 400
def test_privacy_filtering_events_by_operation(client, mock_events_collection, monkeypatch):
monkeypatch.setattr("config.PRIVACY_SENSITIVE_OPERATIONS", {"MailItemsAccessed", "Send"})
monkeypatch.setattr("routes.events.PRIVACY_SENSITIVE_OPERATIONS", {"MailItemsAccessed", "Send"})
monkeypatch.setattr("auth.PRIVACY_SERVICE_ROLES", {"SecurityAdmin"})
monkeypatch.setattr("auth.user_can_access_privacy_services", lambda claims: False)
monkeypatch.setattr("routes.events.user_can_access_privacy_services", lambda claims: False)
mock_events_collection.insert_one(
{
"id": "evt-safe",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Exchange",
"operation": "Add-MailboxPermission",
"result": "success",
"actor_display": "Alice",
"raw_text": "",
}
)
mock_events_collection.insert_one(
{
"id": "evt-priv",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Exchange",
"operation": "Send",
"result": "success",
"actor_display": "Bob",
"raw_text": "",
}
)
response = client.get("/api/events")
assert response.status_code == 200
data = response.json()
ids = [e["id"] for e in data["items"]]
assert "evt-safe" in ids
assert "evt-priv" not in ids
def test_privacy_filter_options_shows_service_hides_ops(client, mock_events_collection, monkeypatch):
monkeypatch.setattr("config.PRIVACY_SENSITIVE_OPERATIONS", {"MailItemsAccessed"})
monkeypatch.setattr("routes.events.PRIVACY_SENSITIVE_OPERATIONS", {"MailItemsAccessed"})
monkeypatch.setattr("auth.PRIVACY_SERVICE_ROLES", {"SecurityAdmin"})
monkeypatch.setattr("auth.user_can_access_privacy_services", lambda claims: False)
monkeypatch.setattr("routes.events.user_can_access_privacy_services", lambda claims: False)
mock_events_collection.insert_one(
{
"id": "evt-1",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Exchange",
"operation": "MailItemsAccessed",
"result": "success",
"actor_display": "Alice",
"raw_text": "",
}
)
mock_events_collection.insert_one(
{
"id": "evt-2",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Exchange",
"operation": "Add-MailboxPermission",
"result": "success",
"actor_display": "Bob",
"raw_text": "",
}
)
response = client.get("/api/filter-options")
assert response.status_code == 200
data = response.json()
assert "Exchange" in data["services"]
assert "MailItemsAccessed" not in data["operations"]
assert "Add-MailboxPermission" in data["operations"]
def test_privacy_explain_forbidden_by_operation(client, mock_events_collection, monkeypatch):
monkeypatch.setattr("config.PRIVACY_SENSITIVE_OPERATIONS", {"Send"})
monkeypatch.setattr("routes.ask.PRIVACY_SENSITIVE_OPERATIONS", {"Send"})
monkeypatch.setattr("auth.PRIVACY_SERVICE_ROLES", {"SecurityAdmin"})
monkeypatch.setattr("auth.user_can_access_privacy_services", lambda claims: False)
monkeypatch.setattr("routes.ask.user_can_access_privacy_services", lambda claims: False)
mock_events_collection.insert_one(
{
"id": "evt-send",
"timestamp": datetime.now(UTC).isoformat(),
"service": "Exchange",
"operation": "Send",
"result": "success",
"actor_display": "Bob",
"raw_text": "",
}
)
response = client.post("/api/events/evt-send/explain")
assert response.status_code == 403
def test_health(client): def test_health(client):
response = client.get("/health") response = client.get("/health")
assert response.status_code == 200 assert response.status_code == 200