25 Commits

Author SHA1 Message Date
e348881083 feat: Admin Operations SIEM — alerts, notifications, pre-built rules
- Add pluggable notification system (webhook, Slack, Teams) with retry
- Add alert deduplication: same rule + actor within 15 min = one alert
- Add 10 pre-built admin-ops rule templates seeded on startup:
  - Failed Conditional Access, After-Hours Admin Activity
  - New Application Registration, Admin Role Assignment
  - License Change, Bulk User Deletion
  - Device Compliance Failure, Exchange Transport Rule Change
  - Service Principal Credential Added, External Sharing Enabled
- Add /api/alerts, /api/alerts/{id}/status, /api/alerts/summary endpoints
- Add alert dashboard to frontend with status filters and ack/resolve buttons
- Add alert summary badge in hero header (high/medium/low counts)
- New env vars: ALERT_WEBHOOK_URL, ALERT_WEBHOOK_FORMAT, ALERT_DEDUPE_MINUTES
2026-04-22 14:12:36 +02:00
cbd46adaa6 style: ruff format
All checks were successful
CI / lint-and-test (push) Successful in 25s
2026-04-22 10:08:32 +02:00
e4bafbc4b0 chore: fix ruff import order in test_ask.py
Some checks failed
CI / lint-and-test (push) Failing after 19s
2026-04-22 10:06:07 +02:00
f75f165911 feat: Redis caching + async queue for LLM scaling (v1.6.0)
Some checks failed
Release / build-and-push (push) Successful in 1m24s
CI / lint-and-test (push) Failing after 29s
- Add async Redis client singleton (redis_client.py) for caching and arq pool
- Add arq job functions (jobs.py) for background LLM processing
- Cache ask/explain LLM responses with TTL (1h ask, 24h explain)
- Add async mode to /api/ask: enqueue job, return job_id, poll /api/jobs/{id}
- Add GET /api/jobs/{job_id} endpoint for job status polling
- Add arq worker service to docker-compose (dev + prod)
- Switch from Redis to Valkey (BSD fork) in Docker Compose
- Add REDIS_URL config setting
- Add tests for cache hit, async mode, and job status
2026-04-22 09:55:05 +02:00
2fffe3aec2 feat: operation-level privacy gating instead of broad service-level
All checks were successful
CI / lint-and-test (push) Successful in 21s
- Replace broad service-level hiding with fine-grained operation-level gating
- PRIVACY_SENSITIVE_OPERATIONS config: hide specific operations across ALL services
- PRIVACY_SERVICES still works for broad service-level blocking (optional)
- Users without PRIVACY_SERVICE_ROLES:
  * Don't see sensitive operations in /api/filter-options
  * Can't query sensitive operations via /api/events or /api/ask
  * Get 403 on /api/events/{id}/explain for sensitive events
- Exchange/Teams services remain visible; only privacy ops are hidden
- Update .env.example with new operation-level config docs
2026-04-22 08:23:46 +02:00
b2f4cabef4 feat: service-level role gating for privacy-sensitive services (Option A)
All checks were successful
CI / lint-and-test (push) Successful in 25s
- Add PRIVACY_SERVICES and PRIVACY_SERVICE_ROLES config variables
- Add user_can_access_privacy_services(claims) helper in auth.py
- /api/events filters out privacy services for users without required roles
- /api/filter-options excludes privacy services from dropdown options
- /api/ask excludes privacy services from NLQ queries
- /api/events/{id}/explain returns 403 for privacy events if unauthorized
- Teams added to default noisy service exclusion (frontend + backend)
- Update .env.example with privacy config documentation
- Add tests for event filtering, filter-options exclusion, and explain 403
2026-04-22 07:26:21 +02:00
fb2386e190 feat: saved searches (bookmarks)
All checks were successful
CI / lint-and-test (push) Successful in 23s
- Add saved_searches_collection to database.py with index on created_by+created_at
- New routes/saved_searches.py: GET /api/saved-searches, POST, DELETE
- Saved searches are scoped per user (created_by = token sub)
- Mount router in main.py
- Frontend: Save filters button, saved search pills with load/delete
- loadSavedSearches called on initApp
- applySavedSearch restores filters and validates services against current options
- Add CSS for saved-searches row
- Add tests for CRUD, delete 404, and name validation
2026-04-22 07:04:07 +02:00
658ddd0aac feat: copy raw event and AI explain in modal
All checks were successful
CI / lint-and-test (push) Successful in 32s
- Add POST /api/events/{id}/explain endpoint that fetches event + related events
  and asks the LLM for a plain-language explanation with security context
- Add 'Copy' button to raw event modal (uses navigator.clipboard)
- Add 'Explain' button to raw event modal (only when AI_FEATURES_ENABLED)
- Show explanation in modal with markdown rendering
- Add CSS for modal actions and explanation panel
- Add tests for explain endpoint (404, no LLM key, mocked LLM success)
2026-04-21 22:26:26 +02:00
5122739c01 feat: MCP server over SSE with OIDC auth
All checks were successful
CI / lint-and-test (push) Successful in 36s
- Extract shared MCP tool handlers to mcp_common.py
- mcp_server.py now uses shared handlers (stdio transport for local dev)
- New routes/mcp.py: SSE transport behind existing OIDC Bearer auth
- Mount MCP ASGI app at /mcp in main.py when AI_FEATURES_ENABLED
- /mcp/sse  -> establishes SSE stream (requires valid token when auth enabled)
- /mcp/messages/ -> receives MCP client messages
- Update README with SSE MCP docs
- Add tests for mount existence, auth, and message routing
2026-04-21 07:38:12 +02:00
60b6ad15c4 Release v1.3.0: AI feature flag and MCP server
All checks were successful
CI / lint-and-test (push) Successful in 45s
Release / build-and-push (push) Successful in 1m34s
- Add AI_FEATURES_ENABLED config flag to gate AI/natural-language features
- Conditionally register /api/ask router based on AI_FEATURES_ENABLED
- Add GET /api/config/features endpoint for frontend feature detection
- Update frontend to hide Ask panel when AI features are disabled
- Implement standalone MCP server (backend/mcp_server.py) with tools:
  * search_events, get_event, get_summary, ask
- Add mcp dependency to requirements.txt
- Update .env.example, AGENTS.md, and ROADMAP.md
- Bump VERSION to 1.3.0
2026-04-20 18:11:26 +02:00
b4e504a87b feat: intent-aware querying + smart sampling for large audit datasets
All checks were successful
Release / build-and-push (push) Successful in 1m31s
CI / lint-and-test (push) Successful in 34s
- Add keyword-based intent extraction: 'device' → Intune, 'user' → Directory, etc.
- Broad questions without intent auto-exclude noisy services (Exchange, SharePoint)
- Smart stratified sampling: failures always included, high-value services prioritised
- Fetch up to 1000 events from MongoDB, then curate best 200 for the LLM
- Excluded services noted in LLM prompt and query_info so the admin knows the scope
2026-04-20 17:41:21 +02:00
cfe9397cc5 feat: raise LLM event limit to 200 and show total count awareness
All checks were successful
CI / lint-and-test (push) Successful in 23s
Release / build-and-push (push) Successful in 27s
- Bump LLM_MAX_EVENTS default from 50 to 200
- Add total_matched count to /api/ask response
- Include 'Showing X of Y total' header in LLM prompt so the model
  knows when its view is a subset and avoids false certainty
- Update system prompt to instruct acknowledging scale when truncated
- Update test mocks to accept new total parameter
2026-04-20 16:13:52 +02:00
cf0283b20b feat: natural language queries respect UI filters (v1.2.0)
All checks were successful
CI / lint-and-test (push) Successful in 22s
Release / build-and-push (push) Successful in 36s
- AskRequest now accepts optional filter fields: services, actor, operation,
  result, start, end, include_tags, exclude_tags
- ask_question merges NL-extracted constraints with explicit UI filters
- Frontend sends active filter state with every ask request
- Show filter hint below ask input when filters are active
- Add tests for service+result filtering and actor filtering in /api/ask

Bump version to 1.2.0
2026-04-20 16:07:35 +02:00
22d237fbfb style: apply ruff fixes
All checks were successful
CI / lint-and-test (push) Successful in 33s
Release / build-and-push (push) Successful in 37s
2026-04-20 15:21:34 +02:00
0ef50c91f7 feat: natural language query + production hardening
Some checks failed
CI / lint-and-test (push) Failing after 41s
Release / build-and-push (push) Successful in 1m33s
Features:
- Add /api/ask endpoint for plain-language audit log queries
- Regex-based time/entity extraction (no LLM required for parsing)
- LLM-powered narrative summarisation with OpenAI-compatible APIs
- Graceful fallback to structured bullet lists when LLM is unavailable
- Frontend ask panel with markdown rendering and cited events

Production:
- Harden Dockerfile: non-root user, gunicorn+uvicorn workers
- Add docker-compose.prod.yml with internal networks and health checks
- Add nginx reverse proxy with security headers
- MongoDB no longer exposed externally in production

Tests:
- 29 new tests for ask parsing, query building, and endpoint behaviour
- Fix conftest monkeypatch for routes.ask events collection

Bump version to 1.1.0
2026-04-20 15:10:55 +02:00
196e1b7781 fix(tests): use services query param for multi-service filter test
Some checks failed
CI / lint-and-test (push) Successful in 23s
Release / build-and-push (push) Failing after 22s
2026-04-17 15:57:48 +02:00
52f565b647 style: apply ruff formatting to tests/test_rules.py
Some checks failed
CI / lint-and-test (push) Failing after 24s
2026-04-16 19:01:24 +02:00
9774277bd0 fix(tests): defer rules import in test_rules.py to avoid CI db init error
Some checks failed
CI / lint-and-test (push) Failing after 29s
2026-04-16 19:00:20 +02:00
4713b43afe style: apply ruff formatting to all backend files
Some checks failed
CI / lint-and-test (push) Failing after 38s
2026-04-16 18:58:41 +02:00
3761aa6d74 feat(tags): add bulk tagging and tag-based filtering
Some checks failed
CI / lint-and-test (push) Failing after 1m24s
- Add include_tags/exclude_tags query params to /api/events
- Add POST /api/events/bulk-tags endpoint with append/replace modes
- Frontend: add Include tags / Exclude tags filter inputs
- Frontend: add Bulk tag matching button with prompt for tag and mode
- Update filter layout to accommodate new tag fields
- Add tests for tag filtering and bulk tag append/replace
2026-04-16 18:50:57 +02:00
82bafc06c9 fix(auth): resolve JWT InvalidSignatureError and improve frontend UX
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Fix auth by using idToken fallback when accessToken audience mismatches
- Add PyJWT verification with audience-aware token selection in frontend
- Source health: track last_attempt_time and error status per source
- Frontend: fix modal outside x-data scope, add circular-safe JSON stringify
- Frontend: support multi-select service filter with All/None toggles
- Frontend: improve filter layout into organized rows
- Frontend: fix text overflow and result pill colors (success/succeeded)
- Intune: normalize application actors (auditActorType=Application)
- Add cache-control middleware for HTML/API responses
- Update tests for multi-service filtering and source health
2026-04-16 11:32:45 +02:00
ed310a06de fix: replace python-jose with PyJWT for robust JWKS signature verification
Some checks failed
CI / lint-and-test (push) Has been cancelled
python-jose failed to correctly construct RSA public keys from Microsoft
JWKS entries lacking an explicit alg field, causing signature verification
failures. Switch auth.py to PyJWT + jwt.algorithms.RSAAlgorithm.from_jwk()
which handles Entra JWKS correctly. Add cryptography explicitly to deps.
Update auth tests to remove unused python-jose fixture code.
2026-04-14 16:47:54 +02:00
b35cac42e0 feat: implement Phase 4 enhancements
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Migrate frontend to Alpine.js for reactive state management
- Add source health dashboard in UI and /api/source-health endpoint
- Add event tagging (PATCH /api/events/{id}/tags) and commenting (POST /api/events/{id}/comments)
- Add CSV/JSON export from the UI
- Add rule-based alerting engine (rules.py) with CRUD endpoints (/api/rules)
- Add SIEM export via webhook (siem.py)
- Add AOC audit trail middleware logging all mutations to aoc_audit collection
- Update config with SIEM_ENABLED, SIEM_WEBHOOK_URL, ALERTS_ENABLED
- Add tests for rules engine, tags, comments, and source health
2026-04-14 15:38:39 +02:00
b0198012eb feat: implement Phase 3 scaling
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Replace skip-based pagination with cursor-based pagination (timestamp|_id cursors)
- Add Prometheus /metrics endpoint with request latency, fetch volume, and error counters
- Implement incremental fetch watermarking per source (watermarks collection in MongoDB)
- Add Graph change notification webhook endpoint (/api/webhooks/graph)
- Add correlation ID middleware for distributed tracing (x-request-id header)
- Update frontend to use cursor-based pagination with Prev/Next navigation
- Update tests for cursor pagination, metrics, webhooks, and watermark mocking
2026-04-14 14:58:50 +02:00
9271b4e461 feat: implement Phase 2 stabilization
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Cache Graph API tokens with expiry-aware reuse in graph/auth.py
- Add tenacity-based retry/backoff wrapper (utils/http.py) and apply to all Graph/source API calls
- Add Pydantic request/response models (models/api.py) and FastAPI query constraints
- Add unit tests for event_model, auth and integration tests for API endpoints
- Configure ruff linter/formatter in pyproject.toml
- Add GitHub Actions CI pipeline (.github/workflows/ci.yml)
- Add requirements-dev.txt with pytest, mongomock, httpx, ruff
- Clean up typing imports and fix ruff linting across codebase
2026-04-14 12:02:28 +02:00