Commit Graph

16 Commits

Author SHA1 Message Date
d01e7801ed security: v1.7.7 hardening release
All checks were successful
CI / lint-and-test (push) Successful in 51s
Release / build-and-push (push) Successful in 1m57s
- Add WEBHOOK_CLIENT_SECRET validation for Graph webhooks
- Add Redis-backed rate limiting (fetch/ask/write/default tiers)
- Validate LLM_BASE_URL to prevent SSRF (HTTPS only, block private IPs)
- Enforce non-wildcard CORS when AUTH_ENABLED=true
- Add Content-Security-Policy headers
- Fix audit middleware to use verified JWT claims via contextvars
- Cap bulk_tags updates to 10,000 documents
- Return generic error messages to clients (no internal detail leakage)
- Strict AlertCondition Pydantic model for alert rules
- Security warning on MCP stdio server startup
- Remove MongoDB/Redis host ports from docker-compose
- Remove mongo_query from /ask API response
2026-04-27 09:16:57 +02:00
cbd46adaa6 style: ruff format
All checks were successful
CI / lint-and-test (push) Successful in 25s
2026-04-22 10:08:32 +02:00
f75f165911 feat: Redis caching + async queue for LLM scaling (v1.6.0)
Some checks failed
Release / build-and-push (push) Successful in 1m24s
CI / lint-and-test (push) Failing after 29s
- Add async Redis client singleton (redis_client.py) for caching and arq pool
- Add arq job functions (jobs.py) for background LLM processing
- Cache ask/explain LLM responses with TTL (1h ask, 24h explain)
- Add async mode to /api/ask: enqueue job, return job_id, poll /api/jobs/{id}
- Add GET /api/jobs/{job_id} endpoint for job status polling
- Add arq worker service to docker-compose (dev + prod)
- Switch from Redis to Valkey (BSD fork) in Docker Compose
- Add REDIS_URL config setting
- Add tests for cache hit, async mode, and job status
2026-04-22 09:55:05 +02:00
2fffe3aec2 feat: operation-level privacy gating instead of broad service-level
All checks were successful
CI / lint-and-test (push) Successful in 21s
- Replace broad service-level hiding with fine-grained operation-level gating
- PRIVACY_SENSITIVE_OPERATIONS config: hide specific operations across ALL services
- PRIVACY_SERVICES still works for broad service-level blocking (optional)
- Users without PRIVACY_SERVICE_ROLES:
  * Don't see sensitive operations in /api/filter-options
  * Can't query sensitive operations via /api/events or /api/ask
  * Get 403 on /api/events/{id}/explain for sensitive events
- Exchange/Teams services remain visible; only privacy ops are hidden
- Update .env.example with new operation-level config docs
2026-04-22 08:23:46 +02:00
b2f4cabef4 feat: service-level role gating for privacy-sensitive services (Option A)
All checks were successful
CI / lint-and-test (push) Successful in 25s
- Add PRIVACY_SERVICES and PRIVACY_SERVICE_ROLES config variables
- Add user_can_access_privacy_services(claims) helper in auth.py
- /api/events filters out privacy services for users without required roles
- /api/filter-options excludes privacy services from dropdown options
- /api/ask excludes privacy services from NLQ queries
- /api/events/{id}/explain returns 403 for privacy events if unauthorized
- Teams added to default noisy service exclusion (frontend + backend)
- Update .env.example with privacy config documentation
- Add tests for event filtering, filter-options exclusion, and explain 403
2026-04-22 07:26:21 +02:00
e069869a94 feat: exclude Teams from defaults + GUID resolution in explain
All checks were successful
CI / lint-and-test (push) Successful in 26s
- Add Teams to noisy services excluded by default (frontend + backend ask)
- Exchange, SharePoint, and Teams now unchecked by default in filters
- Enhance explain endpoint with GUID resolution:
  * Extract UUIDs from raw event JSON recursively
  * Resolve directory objects via Graph API (user, group, SP, device)
  * Include resolved names in LLM prompt so explanations reference
    human-readable names instead of raw GUIDs
- Add asyncio import for to_thread wrapper around sync Graph calls
2026-04-22 07:12:10 +02:00
658ddd0aac feat: copy raw event and AI explain in modal
All checks were successful
CI / lint-and-test (push) Successful in 32s
- Add POST /api/events/{id}/explain endpoint that fetches event + related events
  and asks the LLM for a plain-language explanation with security context
- Add 'Copy' button to raw event modal (uses navigator.clipboard)
- Add 'Explain' button to raw event modal (only when AI_FEATURES_ENABLED)
- Show explanation in modal with markdown rendering
- Add CSS for modal actions and explanation panel
- Add tests for explain endpoint (404, no LLM key, mocked LLM success)
2026-04-21 22:26:26 +02:00
b4e504a87b feat: intent-aware querying + smart sampling for large audit datasets
All checks were successful
Release / build-and-push (push) Successful in 1m31s
CI / lint-and-test (push) Successful in 34s
- Add keyword-based intent extraction: 'device' → Intune, 'user' → Directory, etc.
- Broad questions without intent auto-exclude noisy services (Exchange, SharePoint)
- Smart stratified sampling: failures always included, high-value services prioritised
- Fetch up to 1000 events from MongoDB, then curate best 200 for the LLM
- Excluded services noted in LLM prompt and query_info so the admin knows the scope
2026-04-20 17:41:21 +02:00
a255be93fe feat: aggregate large event sets before sending to LLM
All checks were successful
CI / lint-and-test (push) Successful in 18s
Release / build-and-push (push) Successful in 29s
When a query matches >50 events, the LLM now receives:
- Aggregated counts by service, operation, result, and actor
- A list of failures (up to 10)
- The 50 most recent raw events as samples

This scales to thousands of events without blowing the token budget
or losing signal. The LLM gets a bird's-eye view plus concrete examples.

Also updates the system prompt to handle both individual event lists
and aggregated overviews correctly.
2026-04-20 16:23:55 +02:00
cfe9397cc5 feat: raise LLM event limit to 200 and show total count awareness
All checks were successful
CI / lint-and-test (push) Successful in 23s
Release / build-and-push (push) Successful in 27s
- Bump LLM_MAX_EVENTS default from 50 to 200
- Add total_matched count to /api/ask response
- Include 'Showing X of Y total' header in LLM prompt so the model
  knows when its view is a subset and avoids false certainty
- Update system prompt to instruct acknowledging scale when truncated
- Update test mocks to accept new total parameter
2026-04-20 16:13:52 +02:00
cf0283b20b feat: natural language queries respect UI filters (v1.2.0)
All checks were successful
CI / lint-and-test (push) Successful in 22s
Release / build-and-push (push) Successful in 36s
- AskRequest now accepts optional filter fields: services, actor, operation,
  result, start, end, include_tags, exclude_tags
- ask_question merges NL-extracted constraints with explicit UI filters
- Frontend sends active filter state with every ask request
- Show filter hint below ask input when filters are active
- Add tests for service+result filtering and actor filtering in /api/ask

Bump version to 1.2.0
2026-04-20 16:07:35 +02:00
4303b8f02c fix: use max_completion_tokens and remove temperature for Azure OpenAI compat
All checks were successful
CI / lint-and-test (push) Successful in 35s
Release / build-and-push (push) Successful in 40s
- Replace max_tokens with max_completion_tokens (required by newer Azure models)
- Remove hardcoded temperature (not supported by all model types)
- Add response body logging on LLM API errors for easier debugging
2026-04-20 15:55:00 +02:00
9ec193ea13 feat: expose LLM error reason in /api/ask response and UI
All checks were successful
CI / lint-and-test (push) Successful in 21s
Release / build-and-push (push) Successful in 28s
- Add llm_error field to AskResponse so users know why AI summarisation was skipped
- Show orange warning banner in frontend when LLM is not configured or call fails
- Update AskEndpoint tests to assert llm_error presence
2026-04-20 15:45:32 +02:00
be319688f6 feat: add Azure OpenAI / MS Foundry support for /api/ask
All checks were successful
CI / lint-and-test (push) Successful in 24s
Release / build-and-push (push) Successful in 43s
- Add LLM_API_VERSION config for Azure api-version query param
- Detect Azure endpoints and use api-key header instead of Bearer
- Handle base URLs that already include /chat/completions path
- Update .env.example with Azure OpenAI guidance
2026-04-20 15:28:12 +02:00
22d237fbfb style: apply ruff fixes
All checks were successful
CI / lint-and-test (push) Successful in 33s
Release / build-and-push (push) Successful in 37s
2026-04-20 15:21:34 +02:00
0ef50c91f7 feat: natural language query + production hardening
Some checks failed
CI / lint-and-test (push) Failing after 41s
Release / build-and-push (push) Successful in 1m33s
Features:
- Add /api/ask endpoint for plain-language audit log queries
- Regex-based time/entity extraction (no LLM required for parsing)
- LLM-powered narrative summarisation with OpenAI-compatible APIs
- Graceful fallback to structured bullet lists when LLM is unavailable
- Frontend ask panel with markdown rendering and cited events

Production:
- Harden Dockerfile: non-root user, gunicorn+uvicorn workers
- Add docker-compose.prod.yml with internal networks and health checks
- Add nginx reverse proxy with security headers
- MongoDB no longer exposed externally in production

Tests:
- 29 new tests for ask parsing, query building, and endpoint behaviour
- Fix conftest monkeypatch for routes.ask events collection

Bump version to 1.1.0
2026-04-20 15:10:55 +02:00