Compare commits
32 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d100388c7d | |||
| 11fd87411d | |||
| 6a80bf4eb9 | |||
| 5e02f5a402 | |||
| 0c3e5ec57b | |||
| a255be93fe | |||
| cfe9397cc5 | |||
| cf0283b20b | |||
| 28542f7b80 | |||
| 4303b8f02c | |||
| 9ec193ea13 | |||
| be319688f6 | |||
| 22d237fbfb | |||
| 0ef50c91f7 | |||
| b0eba09f0f | |||
| 91a4c6dccf | |||
| 196e1b7781 | |||
| 30dc75d0e5 | |||
| b45d9bb8a3 | |||
| 52f565b647 | |||
| 9774277bd0 | |||
| 4713b43afe | |||
| b86539399b | |||
| 86966bb57f | |||
| 3761aa6d74 | |||
| 6d00d7cf32 | |||
| de9ea45e1e | |||
| bade860fd4 | |||
| 9f4601c4d9 | |||
| 194858490d | |||
| 153ae05114 | |||
| 9dba33aa9f |
12
.env.example
12
.env.example
@@ -33,3 +33,15 @@ SIEM_WEBHOOK_URL=
|
|||||||
|
|
||||||
# Optional: enable rule-based alerting during ingestion
|
# Optional: enable rule-based alerting during ingestion
|
||||||
ALERTS_ENABLED=false
|
ALERTS_ENABLED=false
|
||||||
|
|
||||||
|
# Optional: LLM configuration for natural language querying (/api/ask)
|
||||||
|
# Supports any OpenAI-compatible API (OpenAI, Azure OpenAI, Ollama, etc.)
|
||||||
|
# For Azure OpenAI / MS Foundry, set BASE_URL to your deployment endpoint
|
||||||
|
# (e.g. https://your-resource.openai.azure.com/openai/deployments/your-deployment)
|
||||||
|
# and set API_VERSION to something like 2025-01-01-preview
|
||||||
|
LLM_API_KEY=
|
||||||
|
LLM_BASE_URL=https://api.openai.com/v1
|
||||||
|
LLM_MODEL=gpt-4o-mini
|
||||||
|
LLM_MAX_EVENTS=200
|
||||||
|
LLM_TIMEOUT_SECONDS=30
|
||||||
|
LLM_API_VERSION=
|
||||||
|
|||||||
22
.gitea/workflows/release.yml
Normal file
22
.gitea/workflows/release.yml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
name: Release
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- "v*"
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build-and-push:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout repository
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Log in to Gitea Container Registry
|
||||||
|
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login git.cqre.net -u ${{ github.actor }} --password-stdin 2>&1 | grep -v "WARNING! Your credentials are stored unencrypted"
|
||||||
|
|
||||||
|
- name: Build Docker image
|
||||||
|
run: docker build ./backend --build-arg VERSION=${{ gitea.ref_name }} --tag git.cqre.net/cqrenet/aoc-backend:${{ gitea.ref_name }}
|
||||||
|
|
||||||
|
- name: Push Docker image
|
||||||
|
run: docker push git.cqre.net/cqrenet/aoc-backend:${{ gitea.ref_name }}
|
||||||
103
DEPLOY.md
Normal file
103
DEPLOY.md
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
# Production Deployment Guide
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
AOC runs as a set of Docker containers orchestrated by Docker Compose:
|
||||||
|
|
||||||
|
- **nginx** — reverse proxy, TLS termination, static file serving
|
||||||
|
- **backend** — FastAPI application (Gunicorn + Uvicorn workers)
|
||||||
|
- **mongo** — MongoDB data store (not exposed externally)
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Docker Engine 24+ and Docker Compose plugin
|
||||||
|
- A server with ports 80/443 reachable from your users
|
||||||
|
- TLS certificates (place in `nginx/ssl/` or use Let's Encrypt)
|
||||||
|
- A valid `.env` file at the repo root (see `.env.example`)
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
1. **Clone / pull the latest release**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git checkout v1.1.0
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Copy and edit environment variables**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env and fill in real credentials
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Set the release version**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export AOC_VERSION=v1.1.0
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Deploy**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml pull
|
||||||
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Verify**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost/health
|
||||||
|
curl http://localhost/api/events
|
||||||
|
```
|
||||||
|
|
||||||
|
## Updating to a new release
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export AOC_VERSION=v1.2.0
|
||||||
|
docker compose -f docker-compose.prod.yml pull
|
||||||
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
## Enabling HTTPS
|
||||||
|
|
||||||
|
### Option A: Use your own certificates
|
||||||
|
|
||||||
|
1. Place `cert.pem` and `key.pem` in `nginx/ssl/`
|
||||||
|
2. Uncomment the HTTPS server block in `nginx/nginx.conf`
|
||||||
|
3. Uncomment the HTTP → HTTPS redirect server block
|
||||||
|
4. Reload nginx:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml exec nginx nginx -s reload
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option B: Let's Encrypt with Certbot
|
||||||
|
|
||||||
|
Replace the `nginx` service in `docker-compose.prod.yml` with a Certbot-friendly setup (e.g., use the `nginx-proxy` + `acme-companion` stack) or mount the Certbot certificates into `nginx/ssl/`.
|
||||||
|
|
||||||
|
## Security hardening
|
||||||
|
|
||||||
|
- MongoDB is **not exposed** to the host — only the backend container can reach it.
|
||||||
|
- The backend runs as a non-root (`aoc`) user inside the container.
|
||||||
|
- nginx adds security headers (`X-Frame-Options`, `X-Content-Type-Options`, etc.).
|
||||||
|
- Keep `.env` out of version control — it is listed in `.gitignore`.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export AOC_VERSION=v1.0.3
|
||||||
|
docker compose -f docker-compose.prod.yml pull
|
||||||
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
- Prometheus metrics: `http://your-host/metrics`
|
||||||
|
- Health check: `http://your-host/health`
|
||||||
|
- Container logs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml logs -f backend
|
||||||
|
docker compose -f docker-compose.prod.yml logs -f nginx
|
||||||
|
docker compose -f docker-compose.prod.yml logs -f mongo
|
||||||
|
```
|
||||||
56
RELEASE_NOTES_v1.1.0.md
Normal file
56
RELEASE_NOTES_v1.1.0.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
# AOC v1.1.0 Release Notes
|
||||||
|
|
||||||
|
**Release date:** 2026-04-20
|
||||||
|
|
||||||
|
## What's new
|
||||||
|
|
||||||
|
### Natural language query (`/api/ask`)
|
||||||
|
Ask questions in plain English and get AI-generated answers backed by your audit logs.
|
||||||
|
|
||||||
|
- **Regex-based parsing** extracts time ranges (`last 3 days`, `yesterday`, `today`) and entities (`device ABC123`, `user bob@example.com`) without calling an LLM — fast and deterministic.
|
||||||
|
- **AI narrative summarisation** via any OpenAI-compatible API (OpenAI, Azure OpenAI, MS Foundry, Ollama). The LLM reads the matching events and writes a concise story for non-expert admins.
|
||||||
|
- **Graceful fallback** when no LLM is configured — returns a structured bullet list instead of a narrative.
|
||||||
|
- **Cited evidence** — every answer includes the raw events that back it up, so admins can verify claims.
|
||||||
|
|
||||||
|
### Azure OpenAI / MS Foundry support
|
||||||
|
- Automatic `api-key` header detection for Azure endpoints.
|
||||||
|
- `LLM_API_VERSION` config for Azure `api-version` query parameters.
|
||||||
|
- `max_completion_tokens` support for newer model deployments.
|
||||||
|
|
||||||
|
### Production hardening
|
||||||
|
- **Dockerfile:** runs as non-root user, uses Gunicorn + Uvicorn workers.
|
||||||
|
- **docker-compose.prod.yml:** MongoDB is internal-only (no host port exposure), health checks on all services, nginx reverse proxy with security headers.
|
||||||
|
- **nginx config:** gzip, security headers (`X-Frame-Options`, `X-Content-Type-Options`), ready for TLS.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- New **"Ask a question"** panel at the top of the page.
|
||||||
|
- Markdown rendering for LLM answers (bold, italic, code).
|
||||||
|
- Orange warning banner when LLM is not configured or fails.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
- 29 new tests covering ask parsing, query building, and endpoint behaviour.
|
||||||
|
- 62 tests total, all passing.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Add to your `.env`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Required for AI narrative summarisation
|
||||||
|
LLM_API_KEY=your-key
|
||||||
|
LLM_BASE_URL=https://api.openai.com/v1
|
||||||
|
LLM_MODEL=gpt-4o-mini
|
||||||
|
LLM_MAX_EVENTS=50
|
||||||
|
LLM_TIMEOUT_SECONDS=30
|
||||||
|
LLM_API_VERSION= # set for Azure OpenAI, e.g. 2024-12-01-preview
|
||||||
|
```
|
||||||
|
|
||||||
|
## Upgrade notes
|
||||||
|
|
||||||
|
No breaking changes. Existing `/api/events`, filters, pagination, tags, and comments work unchanged.
|
||||||
|
|
||||||
|
## Docker image
|
||||||
|
|
||||||
|
```
|
||||||
|
git.cqre.net/cqrenet/aoc-backend:v1.1.0
|
||||||
|
```
|
||||||
78
RELEASE_NOTES_v1.2.5.md
Normal file
78
RELEASE_NOTES_v1.2.5.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
# AOC v1.2.5 Release Notes
|
||||||
|
|
||||||
|
**Release date:** 2026-04-20
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's new
|
||||||
|
|
||||||
|
### Natural language query (`/api/ask`)
|
||||||
|
Ask questions in plain English and get AI-generated answers backed by your audit logs.
|
||||||
|
|
||||||
|
- **Regex-based parsing** extracts time ranges (`last 3 days`, `yesterday`, `today`) and entities (`device ABC123`, `user bob@example.com`) without calling an LLM.
|
||||||
|
- **AI narrative summarisation** via any OpenAI-compatible API (OpenAI, Azure OpenAI, MS Foundry, Ollama).
|
||||||
|
- **Graceful fallback** when no LLM is configured — returns a structured bullet list with a clear error banner.
|
||||||
|
- **Cited evidence** — every answer includes the raw events that back it up.
|
||||||
|
|
||||||
|
### Filter-aware queries
|
||||||
|
The ask endpoint now respects the filter panel. When you set **Service = Exchange**, **Result = failure** and ask *"What happened to device X?"*, the LLM only sees failed Exchange events for that device.
|
||||||
|
|
||||||
|
### Scales to thousands of events
|
||||||
|
For large result sets (>50 events), the LLM receives an **aggregated overview** instead of a raw dump:
|
||||||
|
- Counts by service, action, result, and actor
|
||||||
|
- Failure highlights
|
||||||
|
- The 50 most recent raw events as samples
|
||||||
|
|
||||||
|
This keeps token usage low while preserving accuracy.
|
||||||
|
|
||||||
|
### Azure OpenAI / MS Foundry support
|
||||||
|
- Automatic `api-key` header detection for Azure endpoints.
|
||||||
|
- `LLM_API_VERSION` config for Azure `api-version` query parameters.
|
||||||
|
- `max_completion_tokens` support for newer model deployments.
|
||||||
|
|
||||||
|
### Version display
|
||||||
|
- `GET /api/version` endpoint reads the `VERSION` file.
|
||||||
|
- Frontend shows a version badge in the header (e.g., **1.2.5**).
|
||||||
|
|
||||||
|
### Production hardening (from v1.1.0)
|
||||||
|
- Dockerfile runs as non-root user with Gunicorn + Uvicorn workers.
|
||||||
|
- `docker-compose.prod.yml` with internal-only MongoDB, health checks, and nginx reverse proxy.
|
||||||
|
- Security headers (`X-Frame-Options`, `X-Content-Type-Options`, etc.).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Add to your `.env`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Required for AI narrative summarisation
|
||||||
|
LLM_API_KEY=your-key
|
||||||
|
LLM_BASE_URL=https://api.openai.com/v1
|
||||||
|
LLM_MODEL=gpt-4o-mini
|
||||||
|
LLM_MAX_EVENTS=200
|
||||||
|
LLM_TIMEOUT_SECONDS=30
|
||||||
|
LLM_API_VERSION= # set for Azure OpenAI, e.g. 2024-12-01-preview
|
||||||
|
```
|
||||||
|
|
||||||
|
For Azure OpenAI / MS Foundry:
|
||||||
|
```bash
|
||||||
|
LLM_BASE_URL=https://your-resource.openai.azure.com/openai/deployments/your-deployment
|
||||||
|
LLM_API_KEY=your-azure-key
|
||||||
|
LLM_API_VERSION=2024-12-01-preview
|
||||||
|
LLM_MODEL=your-deployment-name
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Upgrade notes
|
||||||
|
|
||||||
|
No breaking changes. Existing `/api/events`, filters, pagination, tags, and comments work unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Docker image
|
||||||
|
|
||||||
|
```
|
||||||
|
git.cqre.net/cqrenet/aoc-backend:v1.2.5
|
||||||
|
```
|
||||||
@@ -1,6 +1,31 @@
|
|||||||
FROM python:3.11-slim
|
FROM python:3.11-slim
|
||||||
|
|
||||||
|
# Bake the version into the image at build time
|
||||||
|
ARG VERSION=unknown
|
||||||
|
ENV VERSION=${VERSION}
|
||||||
|
|
||||||
|
# Security: run as non-root
|
||||||
|
RUN groupadd -r aoc && useradd -r -g aoc aoc
|
||||||
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Install dependencies first for layer caching
|
||||||
COPY requirements.txt .
|
COPY requirements.txt .
|
||||||
RUN pip install --no-cache-dir -r requirements.txt
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
# Copy application code
|
||||||
COPY . .
|
COPY . .
|
||||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
|
||||||
|
# Create directories for potential volume mounts and fix permissions
|
||||||
|
RUN mkdir -p /app/data && chown -R aoc:aoc /app
|
||||||
|
|
||||||
|
USER aoc
|
||||||
|
|
||||||
|
# Production: use gunicorn with uvicorn workers
|
||||||
|
# Workers = 2-4 x $NUM_CORES; keep it conservative for containerised workloads
|
||||||
|
ENV PYTHONUNBUFFERED=1
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1
|
||||||
|
|
||||||
|
EXPOSE 8000
|
||||||
|
|
||||||
|
CMD ["gunicorn", "main:app", "-k", "uvicorn.workers.UvicornWorker", "--bind", "0.0.0.0:8000", "--workers", "2", "--timeout", "120", "--access-logfile", "-", "--error-logfile", "-"]
|
||||||
|
|||||||
@@ -42,6 +42,14 @@ class Settings(BaseSettings):
|
|||||||
# Alerting
|
# Alerting
|
||||||
ALERTS_ENABLED: bool = False
|
ALERTS_ENABLED: bool = False
|
||||||
|
|
||||||
|
# LLM / Natural Language Query
|
||||||
|
LLM_API_KEY: str = ""
|
||||||
|
LLM_BASE_URL: str = "https://api.openai.com/v1"
|
||||||
|
LLM_MODEL: str = "gpt-4o-mini"
|
||||||
|
LLM_MAX_EVENTS: int = 200
|
||||||
|
LLM_TIMEOUT_SECONDS: int = 30
|
||||||
|
LLM_API_VERSION: str = "" # e.g. 2025-01-01-preview for Azure OpenAI
|
||||||
|
|
||||||
|
|
||||||
_settings = Settings()
|
_settings = Settings()
|
||||||
|
|
||||||
@@ -68,3 +76,10 @@ CORS_ORIGINS = [o.strip() for o in _settings.CORS_ORIGINS.split(",") if o.strip(
|
|||||||
SIEM_ENABLED = _settings.SIEM_ENABLED
|
SIEM_ENABLED = _settings.SIEM_ENABLED
|
||||||
SIEM_WEBHOOK_URL = _settings.SIEM_WEBHOOK_URL
|
SIEM_WEBHOOK_URL = _settings.SIEM_WEBHOOK_URL
|
||||||
ALERTS_ENABLED = _settings.ALERTS_ENABLED
|
ALERTS_ENABLED = _settings.ALERTS_ENABLED
|
||||||
|
|
||||||
|
LLM_API_KEY = _settings.LLM_API_KEY
|
||||||
|
LLM_BASE_URL = _settings.LLM_BASE_URL
|
||||||
|
LLM_MODEL = _settings.LLM_MODEL
|
||||||
|
LLM_MAX_EVENTS = _settings.LLM_MAX_EVENTS
|
||||||
|
LLM_TIMEOUT_SECONDS = _settings.LLM_TIMEOUT_SECONDS
|
||||||
|
LLM_API_VERSION = _settings.LLM_API_VERSION
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import structlog
|
|||||||
from config import DB_NAME, MONGO_URI, RETENTION_DAYS
|
from config import DB_NAME, MONGO_URI, RETENTION_DAYS
|
||||||
from pymongo import ASCENDING, DESCENDING, TEXT, MongoClient
|
from pymongo import ASCENDING, DESCENDING, TEXT, MongoClient
|
||||||
|
|
||||||
client = MongoClient(MONGO_URI)
|
client = MongoClient(MONGO_URI or "mongodb://localhost:27017")
|
||||||
db = client[DB_NAME]
|
db = client[DB_NAME]
|
||||||
events_collection = db["events"]
|
events_collection = db["events"]
|
||||||
logger = structlog.get_logger("aoc.database")
|
logger = structlog.get_logger("aoc.database")
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
<meta charset="UTF-8" />
|
<meta charset="UTF-8" />
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
<title>AOC Events</title>
|
<title>AOC Events</title>
|
||||||
<link rel="stylesheet" href="/style.css?v=7" />
|
<link rel="stylesheet" href="/style.css?v=8" />
|
||||||
<script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
|
<script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
|
||||||
<script src="https://alcdn.msauth.net/browser/2.37.0/js/msal-browser.min.js" crossorigin="anonymous"></script>
|
<script src="https://alcdn.msauth.net/browser/2.37.0/js/msal-browser.min.js" crossorigin="anonymous"></script>
|
||||||
</head>
|
</head>
|
||||||
@@ -12,7 +12,7 @@
|
|||||||
<div class="page" x-data="aocApp()" x-init="initApp()">
|
<div class="page" x-data="aocApp()" x-init="initApp()">
|
||||||
<header class="hero">
|
<header class="hero">
|
||||||
<div>
|
<div>
|
||||||
<p class="eyebrow">Admin Operations Center</p>
|
<p class="eyebrow">Admin Operations Center <span class="version-badge" x-text="appVersion"></span></p>
|
||||||
<h1>Directory Audit Explorer</h1>
|
<h1>Directory Audit Explorer</h1>
|
||||||
<p class="lede">Filter Microsoft Entra audit events by user, app, time, action, and action type.</p>
|
<p class="lede">Filter Microsoft Entra audit events by user, app, time, action, and action type.</p>
|
||||||
</div>
|
</div>
|
||||||
@@ -38,6 +38,49 @@
|
|||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<section class="panel">
|
||||||
|
<h3>Ask a question</h3>
|
||||||
|
<form class="ask-form" @submit.prevent="askQuestion()">
|
||||||
|
<div class="ask-row">
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
placeholder="What happened to device ABC123 in the last 3 days?"
|
||||||
|
x-model="askQuestionText"
|
||||||
|
class="ask-input"
|
||||||
|
/>
|
||||||
|
<button type="submit" :disabled="askLoading" x-text="askLoading ? 'Thinking…' : 'Ask'">Ask</button>
|
||||||
|
</div>
|
||||||
|
<div x-show="hasActiveFilters()" class="ask-filter-hint">
|
||||||
|
<small>Respecting active filters: <span x-text="activeFilterSummary()"></span></small>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
<template x-if="askAnswer">
|
||||||
|
<div class="ask-result">
|
||||||
|
<div x-show="askLlmError" class="ask-error" x-text="askLlmError"></div>
|
||||||
|
<div class="ask-answer" x-html="askAnswerHtml"></div>
|
||||||
|
<template x-if="askEvents.length">
|
||||||
|
<div class="ask-events">
|
||||||
|
<h4>Referenced events</h4>
|
||||||
|
<template x-for="(evt, idx) in askEvents" :key="evt.id || idx">
|
||||||
|
<article class="event event--compact">
|
||||||
|
<div class="event__meta">
|
||||||
|
<span class="pill" x-text="evt.display_category || evt.service || '—'"></span>
|
||||||
|
<span class="pill" :class="['success','succeeded','ok','passed'].includes((evt.result || '').toLowerCase()) ? 'pill--ok' : 'pill--warn'" x-text="evt.result || '—'"></span>
|
||||||
|
</div>
|
||||||
|
<h3 x-text="evt.operation || '—'"></h3>
|
||||||
|
<p class="event__detail" x-show="evt.display_summary"><strong>Summary:</strong> <span x-text="evt.display_summary"></span></p>
|
||||||
|
<p class="event__detail"><strong>Actor:</strong> <span x-text="evt.actor_display || '—'"></span></p>
|
||||||
|
<p class="event__detail"><strong>Target:</strong> <span x-text="Array.isArray(evt.target_displays) ? evt.target_displays.join(', ') : '—'"></span></p>
|
||||||
|
<p class="event__detail"><strong>When:</strong> <span x-text="evt.timestamp ? new Date(evt.timestamp).toLocaleString() : '—'"></span></p>
|
||||||
|
</article>
|
||||||
|
</template>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
<button type="button" class="ghost" @click="clearAsk()">Clear</button>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
</section>
|
||||||
|
|
||||||
<section class="panel">
|
<section class="panel">
|
||||||
<form id="filters" class="filters" @submit.prevent="resetPagination(); loadEvents()">
|
<form id="filters" class="filters" @submit.prevent="resetPagination(); loadEvents()">
|
||||||
<div class="filter-row">
|
<div class="filter-row">
|
||||||
@@ -76,12 +119,20 @@
|
|||||||
To
|
To
|
||||||
<input name="end" type="datetime-local" x-model="filters.end" />
|
<input name="end" type="datetime-local" x-model="filters.end" />
|
||||||
</label>
|
</label>
|
||||||
|
<label>
|
||||||
|
Include tags
|
||||||
|
<input name="includeTags" type="text" placeholder="backup, critical" x-model="filters.includeTags" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Exclude tags
|
||||||
|
<input name="excludeTags" type="text" placeholder="noise, auto" x-model="filters.excludeTags" />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div class="filter-row">
|
||||||
<label class="span-2">
|
<label class="span-2">
|
||||||
Search (raw/full-text)
|
Search (raw/full-text)
|
||||||
<input name="search" type="text" placeholder="Any text to search in raw/summary" x-model="filters.search" />
|
<input name="search" type="text" placeholder="Any text to search in raw/summary" x-model="filters.search" />
|
||||||
</label>
|
</label>
|
||||||
</div>
|
|
||||||
<div class="filter-row filter-row--tall">
|
|
||||||
<div class="filter-group span-2">
|
<div class="filter-group span-2">
|
||||||
<span>App / Service</span>
|
<span>App / Service</span>
|
||||||
<div class="multi-select">
|
<div class="multi-select">
|
||||||
@@ -104,6 +155,7 @@
|
|||||||
<div class="actions">
|
<div class="actions">
|
||||||
<button type="submit">Apply filters</button>
|
<button type="submit">Apply filters</button>
|
||||||
<button type="button" id="clearBtn" class="ghost" @click="clearFilters()">Clear</button>
|
<button type="button" id="clearBtn" class="ghost" @click="clearFilters()">Clear</button>
|
||||||
|
<button type="button" class="ghost" @click="bulkTagMatching()">Bulk tag matching</button>
|
||||||
<button type="button" class="ghost" @click="exportJSON()">Export JSON</button>
|
<button type="button" class="ghost" @click="exportJSON()">Export JSON</button>
|
||||||
<button type="button" class="ghost" @click="exportCSV()">Export CSV</button>
|
<button type="button" class="ghost" @click="exportCSV()">Export CSV</button>
|
||||||
</div>
|
</div>
|
||||||
@@ -188,11 +240,20 @@
|
|||||||
accessToken: null,
|
accessToken: null,
|
||||||
authScopes: [],
|
authScopes: [],
|
||||||
filters: {
|
filters: {
|
||||||
actor: '', selectedServices: [], search: '', operation: '', result: '', start: '', end: '', limit: 100,
|
actor: '', selectedServices: [], search: '', operation: '', result: '', start: '', end: '', limit: 100, includeTags: '', excludeTags: '',
|
||||||
},
|
},
|
||||||
options: { actors: [], services: [], operations: [], results: [] },
|
options: { actors: [], services: [], operations: [], results: [] },
|
||||||
|
appVersion: '',
|
||||||
|
askQuestionText: '',
|
||||||
|
askLoading: false,
|
||||||
|
askAnswer: '',
|
||||||
|
askAnswerHtml: '',
|
||||||
|
askEvents: [],
|
||||||
|
askLlmUsed: false,
|
||||||
|
askLlmError: '',
|
||||||
|
|
||||||
async initApp() {
|
async initApp() {
|
||||||
|
await this.loadVersion();
|
||||||
await this.initAuth();
|
await this.initAuth();
|
||||||
if (!this.authConfig?.auth_enabled || this.accessToken) {
|
if (!this.authConfig?.auth_enabled || this.accessToken) {
|
||||||
await this.loadFilterOptions();
|
await this.loadFilterOptions();
|
||||||
@@ -201,6 +262,16 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async loadVersion() {
|
||||||
|
try {
|
||||||
|
const res = await fetch('/api/version');
|
||||||
|
if (res.ok) {
|
||||||
|
const body = await res.json();
|
||||||
|
this.appVersion = body.version || '';
|
||||||
|
}
|
||||||
|
} catch {}
|
||||||
|
},
|
||||||
|
|
||||||
authHeader() {
|
authHeader() {
|
||||||
return this.accessToken ? { Authorization: `Bearer ${this.accessToken}` } : {};
|
return this.accessToken ? { Authorization: `Bearer ${this.accessToken}` } : {};
|
||||||
},
|
},
|
||||||
@@ -320,6 +391,12 @@
|
|||||||
if (this.filters.selectedServices && this.filters.selectedServices.length) {
|
if (this.filters.selectedServices && this.filters.selectedServices.length) {
|
||||||
this.filters.selectedServices.forEach((s) => params.append('services', s));
|
this.filters.selectedServices.forEach((s) => params.append('services', s));
|
||||||
}
|
}
|
||||||
|
if (this.filters.includeTags) {
|
||||||
|
this.filters.includeTags.split(/[,;]+/).map((t) => t.trim()).filter(Boolean).forEach((t) => params.append('include_tags', t));
|
||||||
|
}
|
||||||
|
if (this.filters.excludeTags) {
|
||||||
|
this.filters.excludeTags.split(/[,;]+/).map((t) => t.trim()).filter(Boolean).forEach((t) => params.append('exclude_tags', t));
|
||||||
|
}
|
||||||
if (this.filters.start) {
|
if (this.filters.start) {
|
||||||
const d = new Date(this.filters.start);
|
const d = new Date(this.filters.start);
|
||||||
if (!isNaN(d.getTime())) params.append('start', d.toISOString());
|
if (!isNaN(d.getTime())) params.append('start', d.toISOString());
|
||||||
@@ -417,11 +494,140 @@
|
|||||||
},
|
},
|
||||||
|
|
||||||
clearFilters() {
|
clearFilters() {
|
||||||
this.filters = { actor: '', selectedServices: [...this.options.services], search: '', operation: '', result: '', start: '', end: '', limit: 100 };
|
this.filters = { actor: '', selectedServices: [...this.options.services], search: '', operation: '', result: '', start: '', end: '', limit: 100, includeTags: '', excludeTags: '' };
|
||||||
this.resetPagination();
|
this.resetPagination();
|
||||||
this.loadEvents();
|
this.loadEvents();
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async askQuestion() {
|
||||||
|
const q = this.askQuestionText.trim();
|
||||||
|
if (!q) return;
|
||||||
|
this.askLoading = true;
|
||||||
|
this.askAnswer = '';
|
||||||
|
this.askAnswerHtml = '';
|
||||||
|
this.askEvents = [];
|
||||||
|
this.askLlmError = '';
|
||||||
|
|
||||||
|
const payload = { question: q };
|
||||||
|
if (this.filters.selectedServices && this.filters.selectedServices.length) {
|
||||||
|
payload.services = this.filters.selectedServices;
|
||||||
|
}
|
||||||
|
if (this.filters.actor) payload.actor = this.filters.actor;
|
||||||
|
if (this.filters.operation) payload.operation = this.filters.operation;
|
||||||
|
if (this.filters.result) payload.result = this.filters.result;
|
||||||
|
if (this.filters.start) payload.start = new Date(this.filters.start).toISOString();
|
||||||
|
if (this.filters.end) payload.end = new Date(this.filters.end).toISOString();
|
||||||
|
if (this.filters.includeTags) {
|
||||||
|
payload.include_tags = this.filters.includeTags.split(/[,;]+/).map(t => t.trim()).filter(Boolean);
|
||||||
|
}
|
||||||
|
if (this.filters.excludeTags) {
|
||||||
|
payload.exclude_tags = this.filters.excludeTags.split(/[,;]+/).map(t => t.trim()).filter(Boolean);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const res = await fetch('/api/ask', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', ...this.authHeader() },
|
||||||
|
body: JSON.stringify(payload),
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error(await res.text());
|
||||||
|
const body = await res.json();
|
||||||
|
this.askAnswer = body.answer;
|
||||||
|
this.askAnswerHtml = this._mdToHtml(body.answer);
|
||||||
|
this.askEvents = body.events || [];
|
||||||
|
this.askLlmUsed = body.llm_used;
|
||||||
|
this.askLlmError = body.llm_error || '';
|
||||||
|
} catch (err) {
|
||||||
|
this.askAnswer = 'Sorry, something went wrong: ' + (err.message || 'Unknown error');
|
||||||
|
this.askAnswerHtml = this.askAnswer;
|
||||||
|
} finally {
|
||||||
|
this.askLoading = false;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
clearAsk() {
|
||||||
|
this.askQuestionText = '';
|
||||||
|
this.askAnswer = '';
|
||||||
|
this.askAnswerHtml = '';
|
||||||
|
this.askEvents = [];
|
||||||
|
this.askLlmUsed = false;
|
||||||
|
this.askLlmError = '';
|
||||||
|
},
|
||||||
|
|
||||||
|
_mdToHtml(text) {
|
||||||
|
// Very lightweight markdown-to-HTML for LLM answers
|
||||||
|
return text
|
||||||
|
.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>')
|
||||||
|
.replace(/\*\*(.+?)\*\*/g, '<strong>$1</strong>')
|
||||||
|
.replace(/\*(.+?)\*/g, '<em>$1</em>')
|
||||||
|
.replace(/`([^`]+)`/g, '<code>$1</code>')
|
||||||
|
.replace(/Event #(\d+)/g, '<strong>Event #$1</strong>')
|
||||||
|
.replace(/\n/g, '<br>');
|
||||||
|
},
|
||||||
|
|
||||||
|
hasActiveFilters() {
|
||||||
|
return this.filters.actor || this.filters.operation || this.filters.result ||
|
||||||
|
this.filters.start || this.filters.end || this.filters.includeTags ||
|
||||||
|
this.filters.excludeTags ||
|
||||||
|
(this.filters.selectedServices && this.filters.selectedServices.length &&
|
||||||
|
this.filters.selectedServices.length < this.options.services.length);
|
||||||
|
},
|
||||||
|
|
||||||
|
activeFilterSummary() {
|
||||||
|
const parts = [];
|
||||||
|
if (this.filters.actor) parts.push('actor');
|
||||||
|
if (this.filters.operation) parts.push('action');
|
||||||
|
if (this.filters.result) parts.push('result');
|
||||||
|
if (this.filters.start || this.filters.end) parts.push('time');
|
||||||
|
if (this.filters.includeTags || this.filters.excludeTags) parts.push('tags');
|
||||||
|
const svcCount = this.filters.selectedServices?.length || 0;
|
||||||
|
const allCount = this.options.services?.length || 0;
|
||||||
|
if (svcCount && svcCount < allCount) parts.push(`${svcCount} service${svcCount === 1 ? '' : 's'}`);
|
||||||
|
return parts.join(', ') || 'none';
|
||||||
|
},
|
||||||
|
|
||||||
|
async bulkTagMatching() {
|
||||||
|
const tag = prompt('Enter tag to apply to all matching events:');
|
||||||
|
if (!tag || !tag.trim()) return;
|
||||||
|
const mode = confirm('Click OK to REPLACE existing tags.\nClick Cancel to APPEND the new tag.') ? 'replace' : 'append';
|
||||||
|
const params = new URLSearchParams();
|
||||||
|
['actor', 'operation', 'result', 'search'].forEach((key) => {
|
||||||
|
const val = this.filters[key];
|
||||||
|
if (val) params.append(key, val);
|
||||||
|
});
|
||||||
|
if (this.filters.selectedServices && this.filters.selectedServices.length) {
|
||||||
|
this.filters.selectedServices.forEach((s) => params.append('services', s));
|
||||||
|
}
|
||||||
|
if (this.filters.includeTags) {
|
||||||
|
this.filters.includeTags.split(/[,;]+/).map((t) => t.trim()).filter(Boolean).forEach((t) => params.append('include_tags', t));
|
||||||
|
}
|
||||||
|
if (this.filters.excludeTags) {
|
||||||
|
this.filters.excludeTags.split(/[,;]+/).map((t) => t.trim()).filter(Boolean).forEach((t) => params.append('exclude_tags', t));
|
||||||
|
}
|
||||||
|
if (this.filters.start) {
|
||||||
|
const d = new Date(this.filters.start);
|
||||||
|
if (!isNaN(d.getTime())) params.append('start', d.toISOString());
|
||||||
|
}
|
||||||
|
if (this.filters.end) {
|
||||||
|
const d = new Date(this.filters.end);
|
||||||
|
if (!isNaN(d.getTime())) params.append('end', d.toISOString());
|
||||||
|
}
|
||||||
|
this.statusText = 'Applying bulk tag…';
|
||||||
|
try {
|
||||||
|
const res = await fetch(`/api/events/bulk-tags?${params.toString()}`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', ...this.authHeader() },
|
||||||
|
body: JSON.stringify({ tags: [tag.trim()], mode }),
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error(await res.text());
|
||||||
|
const body = await res.json();
|
||||||
|
this.statusText = `Tagged ${body.matched} events (${body.modified} modified).`;
|
||||||
|
await this.loadEvents();
|
||||||
|
} catch (err) {
|
||||||
|
this.statusText = err.message || 'Failed to apply bulk tag.';
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
displayActor(e) {
|
displayActor(e) {
|
||||||
const app = e.actor?.application || e.actor?.app;
|
const app = e.actor?.application || e.actor?.app;
|
||||||
if (app?.displayName) return app.displayName;
|
if (app?.displayName) return app.displayName;
|
||||||
|
|||||||
@@ -377,6 +377,113 @@ input {
|
|||||||
margin: 0;
|
margin: 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Ask / Natural Language Query */
|
||||||
|
.ask-form {
|
||||||
|
margin-top: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-row {
|
||||||
|
display: flex;
|
||||||
|
gap: 10px;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-input {
|
||||||
|
flex: 1;
|
||||||
|
padding: 12px 14px;
|
||||||
|
border-radius: 10px;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
background: rgba(255, 255, 255, 0.02);
|
||||||
|
color: var(--text);
|
||||||
|
font-size: 15px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-result {
|
||||||
|
margin-top: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-answer {
|
||||||
|
background: rgba(125, 211, 252, 0.06);
|
||||||
|
border: 1px solid rgba(125, 211, 252, 0.2);
|
||||||
|
border-radius: 12px;
|
||||||
|
padding: 16px;
|
||||||
|
line-height: 1.55;
|
||||||
|
margin-bottom: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-answer code {
|
||||||
|
background: rgba(255,255,255,0.06);
|
||||||
|
padding: 2px 6px;
|
||||||
|
border-radius: 6px;
|
||||||
|
font-size: 13px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-error {
|
||||||
|
background: rgba(249, 115, 22, 0.1);
|
||||||
|
border: 1px solid rgba(249, 115, 22, 0.3);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 10px 14px;
|
||||||
|
color: #fdba74;
|
||||||
|
font-size: 14px;
|
||||||
|
margin-bottom: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-filter-hint {
|
||||||
|
margin-top: 6px;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
.version-badge {
|
||||||
|
display: inline-block;
|
||||||
|
margin-left: 8px;
|
||||||
|
padding: 2px 8px;
|
||||||
|
border-radius: 999px;
|
||||||
|
background: rgba(125, 211, 252, 0.15);
|
||||||
|
border: 1px solid rgba(125, 211, 252, 0.3);
|
||||||
|
color: var(--accent-strong);
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 600;
|
||||||
|
letter-spacing: 0.05em;
|
||||||
|
vertical-align: middle;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-events {
|
||||||
|
margin-bottom: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ask-events h4 {
|
||||||
|
margin: 0 0 10px;
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 14px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.06em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.event--compact {
|
||||||
|
padding: 12px;
|
||||||
|
margin-bottom: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.event--compact h3 {
|
||||||
|
font-size: 15px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.source-health {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(min(200px, 100%), 1fr));
|
||||||
|
gap: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.health-card {
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 10px;
|
||||||
|
padding: 10px 12px;
|
||||||
|
background: rgba(255, 255, 255, 0.02);
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
@media (max-width: 640px) {
|
@media (max-width: 640px) {
|
||||||
.hero {
|
.hero {
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
@@ -386,4 +493,9 @@ input {
|
|||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
align-items: stretch;
|
align-items: stretch;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.ask-row {
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: stretch;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,10 +9,7 @@ def fetch_audit_logs(hours: int = 24, since: str | None = None, max_pages: int =
|
|||||||
"""Fetch paginated directory audit logs from Microsoft Graph and enrich with resolved names."""
|
"""Fetch paginated directory audit logs from Microsoft Graph and enrich with resolved names."""
|
||||||
token = get_access_token()
|
token = get_access_token()
|
||||||
start_time = since or (datetime.utcnow() - timedelta(hours=hours)).isoformat() + "Z"
|
start_time = since or (datetime.utcnow() - timedelta(hours=hours)).isoformat() + "Z"
|
||||||
next_url = (
|
next_url = f"https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$filter=activityDateTime ge {start_time}"
|
||||||
"https://graph.microsoft.com/v1.0/"
|
|
||||||
f"auditLogs/directoryAudits?$filter=activityDateTime ge {start_time}"
|
|
||||||
)
|
|
||||||
headers = {"Authorization": f"Bearer {token}"}
|
headers = {"Authorization": f"Bearer {token}"}
|
||||||
|
|
||||||
events = []
|
events = []
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
|
|
||||||
from utils.http import get_with_retry
|
from utils.http import get_with_retry
|
||||||
|
|
||||||
|
|
||||||
@@ -48,7 +47,10 @@ def resolve_directory_object(object_id: str, token: str, cache: dict[str, dict])
|
|||||||
|
|
||||||
probes = [
|
probes = [
|
||||||
("user", f"https://graph.microsoft.com/v1.0/users/{object_id}?$select=id,displayName,userPrincipalName,mail"),
|
("user", f"https://graph.microsoft.com/v1.0/users/{object_id}?$select=id,displayName,userPrincipalName,mail"),
|
||||||
("servicePrincipal", f"https://graph.microsoft.com/v1.0/servicePrincipals/{object_id}?$select=id,displayName,appId,appDisplayName"),
|
(
|
||||||
|
"servicePrincipal",
|
||||||
|
f"https://graph.microsoft.com/v1.0/servicePrincipals/{object_id}?$select=id,displayName,appId,appDisplayName",
|
||||||
|
),
|
||||||
("group", f"https://graph.microsoft.com/v1.0/groups/{object_id}?$select=id,displayName,mail"),
|
("group", f"https://graph.microsoft.com/v1.0/groups/{object_id}?$select=id,displayName,mail"),
|
||||||
("device", f"https://graph.microsoft.com/v1.0/devices/{object_id}?$select=id,displayName"),
|
("device", f"https://graph.microsoft.com/v1.0/devices/{object_id}?$select=id,displayName"),
|
||||||
]
|
]
|
||||||
@@ -82,12 +84,7 @@ def resolve_service_principal_owners(sp_id: str, token: str, cache: dict[str, li
|
|||||||
)
|
)
|
||||||
payload = _request_json(url, token)
|
payload = _request_json(url, token)
|
||||||
for owner in (payload or {}).get("value", []):
|
for owner in (payload or {}).get("value", []):
|
||||||
name = (
|
name = owner.get("displayName") or owner.get("userPrincipalName") or owner.get("mail") or owner.get("id")
|
||||||
owner.get("displayName")
|
|
||||||
or owner.get("userPrincipalName")
|
|
||||||
or owner.get("mail")
|
|
||||||
or owner.get("id")
|
|
||||||
)
|
|
||||||
if name:
|
if name:
|
||||||
owners.append(name)
|
owners.append(name)
|
||||||
|
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ from fastapi.responses import Response
|
|||||||
from fastapi.staticfiles import StaticFiles
|
from fastapi.staticfiles import StaticFiles
|
||||||
from metrics import observe_request, prometheus_metrics
|
from metrics import observe_request, prometheus_metrics
|
||||||
from middleware import CorrelationIdMiddleware
|
from middleware import CorrelationIdMiddleware
|
||||||
|
from routes.ask import router as ask_router
|
||||||
from routes.config import router as config_router
|
from routes.config import router as config_router
|
||||||
from routes.events import router as events_router
|
from routes.events import router as events_router
|
||||||
from routes.fetch import router as fetch_router
|
from routes.fetch import router as fetch_router
|
||||||
@@ -85,12 +86,14 @@ async def audit_middleware(request: Request, call_next):
|
|||||||
response = await call_next(request)
|
response = await call_next(request)
|
||||||
if request.url.path.startswith("/api/") and request.method in ("POST", "PATCH", "PUT", "DELETE"):
|
if request.url.path.startswith("/api/") and request.method in ("POST", "PATCH", "PUT", "DELETE"):
|
||||||
from auth import AUTH_ENABLED
|
from auth import AUTH_ENABLED
|
||||||
|
|
||||||
user = "anonymous"
|
user = "anonymous"
|
||||||
if AUTH_ENABLED:
|
if AUTH_ENABLED:
|
||||||
auth_header = request.headers.get("authorization", "")
|
auth_header = request.headers.get("authorization", "")
|
||||||
if auth_header.lower().startswith("bearer "):
|
if auth_header.lower().startswith("bearer "):
|
||||||
try:
|
try:
|
||||||
from jose import jwt
|
from jose import jwt
|
||||||
|
|
||||||
token = auth_header.split(" ", 1)[1]
|
token = auth_header.split(" ", 1)[1]
|
||||||
claims = jwt.get_unverified_claims(token)
|
claims = jwt.get_unverified_claims(token)
|
||||||
user = claims.get("sub", "unknown")
|
user = claims.get("sub", "unknown")
|
||||||
@@ -110,12 +113,14 @@ app.include_router(events_router, prefix="/api")
|
|||||||
app.include_router(config_router, prefix="/api")
|
app.include_router(config_router, prefix="/api")
|
||||||
app.include_router(webhooks_router, prefix="/api")
|
app.include_router(webhooks_router, prefix="/api")
|
||||||
app.include_router(health_router, prefix="/api")
|
app.include_router(health_router, prefix="/api")
|
||||||
|
app.include_router(ask_router, prefix="/api")
|
||||||
app.include_router(rules_router, prefix="/api")
|
app.include_router(rules_router, prefix="/api")
|
||||||
|
|
||||||
|
|
||||||
@app.get("/health")
|
@app.get("/health")
|
||||||
async def health_check():
|
async def health_check():
|
||||||
from database import db
|
from database import db
|
||||||
|
|
||||||
try:
|
try:
|
||||||
db.command("ping")
|
db.command("ping")
|
||||||
return {"status": "ok", "database": "connected"}
|
return {"status": "ok", "database": "connected"}
|
||||||
@@ -129,6 +134,13 @@ async def metrics():
|
|||||||
return Response(content=prometheus_metrics(), media_type="text/plain")
|
return Response(content=prometheus_metrics(), media_type="text/plain")
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/version")
|
||||||
|
async def version():
|
||||||
|
import os
|
||||||
|
|
||||||
|
return {"version": os.environ.get("VERSION", "unknown")}
|
||||||
|
|
||||||
|
|
||||||
frontend_dir = Path(__file__).parent / "frontend"
|
frontend_dir = Path(__file__).parent / "frontend"
|
||||||
app.mount("/", StaticFiles(directory=frontend_dir, html=True), name="frontend")
|
app.mount("/", StaticFiles(directory=frontend_dir, html=True), name="frontend")
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ new display fields. Example:
|
|||||||
|
|
||||||
python maintenance.py renormalize --limit 500
|
python maintenance.py renormalize --limit 500
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
|
||||||
from database import events_collection
|
from database import events_collection
|
||||||
@@ -53,7 +54,9 @@ def dedupe(limit: int = None, batch_size: int = 500) -> int:
|
|||||||
"""
|
"""
|
||||||
Remove duplicate events based on dedupe_key. Keeps the first occurrence encountered.
|
Remove duplicate events based on dedupe_key. Keeps the first occurrence encountered.
|
||||||
"""
|
"""
|
||||||
cursor = events_collection.find({}, projection={"_id": 1, "dedupe_key": 1, "raw": 1, "id": 1, "timestamp": 1}).sort("timestamp", 1)
|
cursor = events_collection.find({}, projection={"_id": 1, "dedupe_key": 1, "raw": 1, "id": 1, "timestamp": 1}).sort(
|
||||||
|
"timestamp", 1
|
||||||
|
)
|
||||||
if limit:
|
if limit:
|
||||||
cursor = cursor.limit(int(limit))
|
cursor = cursor.limit(int(limit))
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
|
|
||||||
from prometheus_client import Counter, Histogram, generate_latest
|
from prometheus_client import Counter, Histogram, generate_latest
|
||||||
|
|
||||||
REQUEST_DURATION = Histogram(
|
REQUEST_DURATION = Histogram(
|
||||||
|
|||||||
@@ -54,6 +54,11 @@ class TagsUpdateRequest(BaseModel):
|
|||||||
tags: list[str]
|
tags: list[str]
|
||||||
|
|
||||||
|
|
||||||
|
class BulkTagsRequest(BaseModel):
|
||||||
|
tags: list[str]
|
||||||
|
mode: str = "append" # "append" or "replace"
|
||||||
|
|
||||||
|
|
||||||
class CommentAddRequest(BaseModel):
|
class CommentAddRequest(BaseModel):
|
||||||
text: str
|
text: str
|
||||||
|
|
||||||
@@ -65,3 +70,34 @@ class AlertRuleResponse(BaseModel):
|
|||||||
severity: str
|
severity: str
|
||||||
conditions: list[dict]
|
conditions: list[dict]
|
||||||
message: str
|
message: str
|
||||||
|
|
||||||
|
|
||||||
|
class AskRequest(BaseModel):
|
||||||
|
question: str
|
||||||
|
services: list[str] | None = None
|
||||||
|
actor: str | None = None
|
||||||
|
operation: str | None = None
|
||||||
|
result: str | None = None
|
||||||
|
start: str | None = None
|
||||||
|
end: str | None = None
|
||||||
|
include_tags: list[str] | None = None
|
||||||
|
exclude_tags: list[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class AskEventRef(BaseModel):
|
||||||
|
id: str | None = None
|
||||||
|
timestamp: str | None = None
|
||||||
|
operation: str | None = None
|
||||||
|
actor_display: str | None = None
|
||||||
|
target_displays: list[str] | None = None
|
||||||
|
display_summary: str | None = None
|
||||||
|
service: str | None = None
|
||||||
|
result: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class AskResponse(BaseModel):
|
||||||
|
answer: str
|
||||||
|
events: list[AskEventRef]
|
||||||
|
query_info: dict
|
||||||
|
llm_used: bool
|
||||||
|
llm_error: str | None = None
|
||||||
|
|||||||
@@ -75,10 +75,7 @@ def _target_types(targets: list) -> list:
|
|||||||
types = []
|
types = []
|
||||||
for t in targets or []:
|
for t in targets or []:
|
||||||
resolved = t.get("_resolved") or {}
|
resolved = t.get("_resolved") or {}
|
||||||
t_type = (
|
t_type = resolved.get("type") or t.get("type")
|
||||||
resolved.get("type")
|
|
||||||
or t.get("type")
|
|
||||||
)
|
|
||||||
if t_type:
|
if t_type:
|
||||||
types.append(t_type)
|
types.append(t_type)
|
||||||
return types
|
return types
|
||||||
@@ -101,7 +98,9 @@ def _display_summary(operation: str, target_labels: list, actor_label: str, targ
|
|||||||
return " | ".join(pieces)
|
return " | ".join(pieces)
|
||||||
|
|
||||||
|
|
||||||
def _render_summary(template: str, operation: str, actor: str, target: str, category: str, result: str, service: str) -> str:
|
def _render_summary(
|
||||||
|
template: str, operation: str, actor: str, target: str, category: str, result: str, service: str
|
||||||
|
) -> str:
|
||||||
try:
|
try:
|
||||||
return template.format(
|
return template.format(
|
||||||
operation=operation or category or "Event",
|
operation=operation or category or "Event",
|
||||||
@@ -177,13 +176,16 @@ def normalize_event(e):
|
|||||||
else:
|
else:
|
||||||
display_actor_value = actor_label
|
display_actor_value = actor_label
|
||||||
|
|
||||||
dedupe_key = _make_dedupe_key(e, {
|
dedupe_key = _make_dedupe_key(
|
||||||
"id": e.get("id"),
|
e,
|
||||||
"timestamp": e.get("activityDateTime"),
|
{
|
||||||
"service": e.get("category"),
|
"id": e.get("id"),
|
||||||
"operation": e.get("activityDisplayName"),
|
"timestamp": e.get("activityDateTime"),
|
||||||
"target_displays": target_labels,
|
"service": e.get("category"),
|
||||||
})
|
"operation": e.get("activityDisplayName"),
|
||||||
|
"target_displays": target_labels,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"id": e.get("id"),
|
"id": e.get("id"),
|
||||||
|
|||||||
@@ -11,3 +11,5 @@ pydantic-settings
|
|||||||
structlog
|
structlog
|
||||||
tenacity
|
tenacity
|
||||||
prometheus-client
|
prometheus-client
|
||||||
|
httpx
|
||||||
|
gunicorn
|
||||||
|
|||||||
422
backend/routes/ask.py
Normal file
422
backend/routes/ask.py
Normal file
@@ -0,0 +1,422 @@
|
|||||||
|
import json
|
||||||
|
import re
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import structlog
|
||||||
|
from auth import require_auth
|
||||||
|
from config import LLM_API_KEY, LLM_API_VERSION, LLM_BASE_URL, LLM_MAX_EVENTS, LLM_MODEL, LLM_TIMEOUT_SECONDS
|
||||||
|
from database import events_collection
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from models.api import AskRequest, AskResponse
|
||||||
|
|
||||||
|
router = APIRouter(dependencies=[Depends(require_auth)])
|
||||||
|
logger = structlog.get_logger("aoc.ask")
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Time-range extraction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_TIME_PATTERNS = [
|
||||||
|
(r"\blast\s+(\d+)\s+days?\b", "days"),
|
||||||
|
(r"\blast\s+(\d+)\s+hours?\b", "hours"),
|
||||||
|
(r"\blast\s+(\d+)\s+minutes?\b", "minutes"),
|
||||||
|
(r"\blast\s+week\b", "week"),
|
||||||
|
(r"\byesterday\b", "yesterday"),
|
||||||
|
(r"\btoday\b", "today"),
|
||||||
|
(r"\bin\s+the\s+last\s+(\d+)\s+days?\b", "days"),
|
||||||
|
(r"\bin\s+the\s+last\s+(\d+)\s+hours?\b", "hours"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_time_range(question: str) -> tuple[str | None, str | None]:
|
||||||
|
"""Return (start_iso, end_iso) or (None, None) if no time detected."""
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
q_lower = question.lower()
|
||||||
|
|
||||||
|
for pattern, unit in _TIME_PATTERNS:
|
||||||
|
m = re.search(pattern, q_lower)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
if unit == "week":
|
||||||
|
start = now - timedelta(days=7)
|
||||||
|
elif unit == "yesterday":
|
||||||
|
start = now - timedelta(days=1)
|
||||||
|
elif unit == "today":
|
||||||
|
start = now.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||||
|
else:
|
||||||
|
num = int(m.group(1))
|
||||||
|
delta = {"days": timedelta(days=num), "hours": timedelta(hours=num), "minutes": timedelta(minutes=num)}[
|
||||||
|
unit
|
||||||
|
]
|
||||||
|
start = now - delta
|
||||||
|
return start.isoformat().replace("+00:00", "Z"), now.isoformat().replace("+00:00", "Z")
|
||||||
|
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Entity extraction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_ENTITY_HINTS = [
|
||||||
|
r"device\s+['\"]?([^'\"\s]+)['\"]?",
|
||||||
|
r"user\s+['\"]?([^'\"\s]+)['\"]?",
|
||||||
|
r"laptop\s+['\"]?([^'\"\s]+)['\"]?",
|
||||||
|
r"vm\s+['\"]?([^'\"\s]+)['\"]?",
|
||||||
|
r"server\s+['\"]?([^'\"\s]+)['\"]?",
|
||||||
|
r"computer\s+['\"]?([^'\"\s]+)['\"]?",
|
||||||
|
]
|
||||||
|
|
||||||
|
_EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_entity(question: str) -> str | None:
|
||||||
|
"""Best-effort extraction of the device / user / entity name."""
|
||||||
|
q_lower = question.lower()
|
||||||
|
|
||||||
|
# Look for explicit hints: "device ABC123"
|
||||||
|
for pattern in _ENTITY_HINTS:
|
||||||
|
m = re.search(pattern, q_lower)
|
||||||
|
if m:
|
||||||
|
# Extract from the original question to preserve case
|
||||||
|
start, end = m.span(1)
|
||||||
|
return question[start:end].strip().rstrip("?.!,;:")
|
||||||
|
|
||||||
|
# Look for quoted strings
|
||||||
|
m = re.search(r'"([^"]{2,50})"', question)
|
||||||
|
if m:
|
||||||
|
return m.group(1).strip()
|
||||||
|
m = re.search(r"'([^']{2,50})'", question)
|
||||||
|
if m:
|
||||||
|
return m.group(1).strip()
|
||||||
|
|
||||||
|
# Look for email addresses
|
||||||
|
m = _EMAIL_RE.search(question)
|
||||||
|
if m:
|
||||||
|
return m.group(0)
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# MongoDB query builder
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _build_event_query(
|
||||||
|
entity: str | None,
|
||||||
|
start: str | None,
|
||||||
|
end: str | None,
|
||||||
|
services: list[str] | None = None,
|
||||||
|
actor: str | None = None,
|
||||||
|
operation: str | None = None,
|
||||||
|
result: str | None = None,
|
||||||
|
include_tags: list[str] | None = None,
|
||||||
|
exclude_tags: list[str] | None = None,
|
||||||
|
) -> dict:
|
||||||
|
filters = []
|
||||||
|
|
||||||
|
if start or end:
|
||||||
|
time_filter = {}
|
||||||
|
if start:
|
||||||
|
time_filter["$gte"] = start
|
||||||
|
if end:
|
||||||
|
time_filter["$lte"] = end
|
||||||
|
filters.append({"timestamp": time_filter})
|
||||||
|
|
||||||
|
if entity:
|
||||||
|
entity_safe = re.escape(entity)
|
||||||
|
filters.append(
|
||||||
|
{
|
||||||
|
"$or": [
|
||||||
|
{"target_displays": {"$elemMatch": {"$regex": entity_safe, "$options": "i"}}},
|
||||||
|
{"actor_display": {"$regex": entity_safe, "$options": "i"}},
|
||||||
|
{"actor_upn": {"$regex": entity_safe, "$options": "i"}},
|
||||||
|
{"raw_text": {"$regex": entity_safe, "$options": "i"}},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
if services:
|
||||||
|
filters.append({"service": {"$in": services}})
|
||||||
|
if actor:
|
||||||
|
actor_safe = re.escape(actor)
|
||||||
|
filters.append(
|
||||||
|
{
|
||||||
|
"$or": [
|
||||||
|
{"actor_display": {"$regex": actor_safe, "$options": "i"}},
|
||||||
|
{"actor_upn": {"$regex": actor_safe, "$options": "i"}},
|
||||||
|
{"actor.user.userPrincipalName": {"$regex": actor_safe, "$options": "i"}},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if operation:
|
||||||
|
filters.append({"operation": {"$regex": re.escape(operation), "$options": "i"}})
|
||||||
|
if result:
|
||||||
|
filters.append({"result": {"$regex": re.escape(result), "$options": "i"}})
|
||||||
|
if include_tags:
|
||||||
|
filters.append({"tags": {"$all": include_tags}})
|
||||||
|
if exclude_tags:
|
||||||
|
filters.append({"tags": {"$not": {"$all": exclude_tags}}})
|
||||||
|
|
||||||
|
return {"$and": filters} if filters else {}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# LLM summarisation
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_SYSTEM_PROMPT = """You are an IT operations assistant. An administrator has asked a question about audit logs.
|
||||||
|
Your job is to read the data below and write a concise, plain-language answer.
|
||||||
|
|
||||||
|
The input may be either:
|
||||||
|
- A small list of individual audit events (numbered Event #1, #2, etc.), or
|
||||||
|
- An aggregated overview with counts by service, action, result, and actor, plus sample events.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Assume the reader is a non-expert admin.
|
||||||
|
- For aggregated overviews: summarise the scale, top patterns, and highlight anomalies or failures.
|
||||||
|
- For small event lists: group related events together and tell a coherent story.
|
||||||
|
- Highlight anything unusual, failed actions, or privilege escalations.
|
||||||
|
- Reference specific event numbers (e.g., "Event #3") when making claims so the user can verify.
|
||||||
|
- If the data is an aggregated subset of a larger result set, acknowledge the scale (e.g., "847 events occurred — the top pattern was...").
|
||||||
|
- If there are no events, say so clearly.
|
||||||
|
- Keep the answer under 300 words.
|
||||||
|
- Do not invent events or patterns that are not supported by the data.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_counts(events: list[dict]) -> dict:
|
||||||
|
"""Build lightweight aggregation tables for large result sets."""
|
||||||
|
from collections import Counter
|
||||||
|
|
||||||
|
svc_counts = Counter(e.get("service") or "Unknown" for e in events)
|
||||||
|
op_counts = Counter(e.get("operation") or "Unknown" for e in events)
|
||||||
|
result_counts = Counter(e.get("result") or "Unknown" for e in events)
|
||||||
|
actor_counts = Counter(e.get("actor_display") or "Unknown" for e in events)
|
||||||
|
return {
|
||||||
|
"services": svc_counts.most_common(10),
|
||||||
|
"operations": op_counts.most_common(10),
|
||||||
|
"results": result_counts.most_common(5),
|
||||||
|
"actors": actor_counts.most_common(10),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _format_events_for_llm(events: list[dict], total: int | None = None) -> str:
|
||||||
|
lines = []
|
||||||
|
|
||||||
|
# If we have a large result set, send aggregation + samples instead of raw dump
|
||||||
|
if total is not None and total > len(events) and len(events) >= 50:
|
||||||
|
lines.append(f"Result set overview: {total} total events (showing the {len(events)} most recent).\n")
|
||||||
|
agg = _aggregate_counts(events)
|
||||||
|
lines.append("Breakdown by service:")
|
||||||
|
for svc, cnt in agg["services"]:
|
||||||
|
lines.append(f" {svc}: {cnt}")
|
||||||
|
lines.append("\nBreakdown by action:")
|
||||||
|
for op, cnt in agg["operations"]:
|
||||||
|
lines.append(f" {op}: {cnt}")
|
||||||
|
lines.append("\nBreakdown by result:")
|
||||||
|
for res, cnt in agg["results"]:
|
||||||
|
lines.append(f" {res}: {cnt}")
|
||||||
|
lines.append("\nTop actors:")
|
||||||
|
for actor, cnt in agg["actors"]:
|
||||||
|
lines.append(f" {actor}: {cnt}")
|
||||||
|
# Include failures and a few recent samples
|
||||||
|
failures = [e for e in events if str(e.get("result") or "").lower() in ("failure", "failed")]
|
||||||
|
if failures:
|
||||||
|
lines.append(f"\nFailures ({len(failures)}):")
|
||||||
|
for e in failures[:10]:
|
||||||
|
ts = e.get("timestamp", "?")[:16].replace("T", " ")
|
||||||
|
op = e.get("operation", "unknown")
|
||||||
|
actor = e.get("actor_display", "unknown")
|
||||||
|
lines.append(f" {ts} — {op} by {actor}")
|
||||||
|
lines.append("\nMost recent sample events:")
|
||||||
|
else:
|
||||||
|
if total is not None and total > len(events):
|
||||||
|
lines.append(f"Showing {len(events)} of {total} total matching events (most recent first):\n")
|
||||||
|
|
||||||
|
# Always include the first N raw events as detail (up to 50)
|
||||||
|
for i, e in enumerate(events[:50], 1):
|
||||||
|
ts = e.get("timestamp") or "unknown time"
|
||||||
|
op = e.get("operation") or "unknown action"
|
||||||
|
actor = e.get("actor_display") or "unknown actor"
|
||||||
|
targets = ", ".join(e.get("target_displays") or []) or "unknown target"
|
||||||
|
svc = e.get("service") or "unknown service"
|
||||||
|
result = e.get("result") or "unknown result"
|
||||||
|
summary = e.get("display_summary") or ""
|
||||||
|
lines.append(
|
||||||
|
f"Event #{i}\n"
|
||||||
|
f" Time: {ts}\n"
|
||||||
|
f" Service: {svc}\n"
|
||||||
|
f" Action: {op}\n"
|
||||||
|
f" Actor: {actor}\n"
|
||||||
|
f" Target: {targets}\n"
|
||||||
|
f" Result: {result}\n"
|
||||||
|
f" Summary: {summary}\n"
|
||||||
|
)
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_chat_url(base_url: str, api_version: str) -> str:
|
||||||
|
"""Construct the chat completions URL, handling Azure OpenAI endpoints."""
|
||||||
|
base = base_url.rstrip("/")
|
||||||
|
url = base if base.endswith("/chat/completions") else f"{base}/chat/completions"
|
||||||
|
if api_version:
|
||||||
|
url = f"{url}?api-version={api_version}"
|
||||||
|
return url
|
||||||
|
|
||||||
|
|
||||||
|
async def _call_llm(question: str, events: list[dict], total: int | None = None) -> str:
|
||||||
|
if not LLM_API_KEY:
|
||||||
|
raise RuntimeError("LLM_API_KEY not configured")
|
||||||
|
|
||||||
|
context = _format_events_for_llm(events, total=total)
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": _SYSTEM_PROMPT},
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": f"Question: {question}\n\nAudit events:\n{context}\n\nPlease answer the question based only on the events above.",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
url = _build_chat_url(LLM_BASE_URL, LLM_API_VERSION)
|
||||||
|
headers = {
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
}
|
||||||
|
# Azure OpenAI uses api-key header; standard OpenAI uses Bearer token
|
||||||
|
if "azure" in LLM_BASE_URL.lower() or "cognitiveservices" in LLM_BASE_URL.lower():
|
||||||
|
headers["api-key"] = LLM_API_KEY
|
||||||
|
else:
|
||||||
|
headers["Authorization"] = f"Bearer {LLM_API_KEY}"
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"model": LLM_MODEL,
|
||||||
|
"messages": messages,
|
||||||
|
"max_completion_tokens": 800,
|
||||||
|
}
|
||||||
|
|
||||||
|
async with httpx.AsyncClient(timeout=LLM_TIMEOUT_SECONDS) as client:
|
||||||
|
resp = await client.post(url, headers=headers, json=payload)
|
||||||
|
if resp.status_code >= 400:
|
||||||
|
body = resp.text
|
||||||
|
logger.error("LLM API error", status_code=resp.status_code, url=url, response_body=body)
|
||||||
|
raise RuntimeError(f"LLM API error {resp.status_code}: {body[:500]}")
|
||||||
|
data = resp.json()
|
||||||
|
return data["choices"][0]["message"]["content"].strip()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# API endpoint
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _to_event_ref(e: dict) -> dict:
|
||||||
|
return {
|
||||||
|
"id": e.get("id"),
|
||||||
|
"timestamp": e.get("timestamp"),
|
||||||
|
"operation": e.get("operation"),
|
||||||
|
"actor_display": e.get("actor_display"),
|
||||||
|
"target_displays": e.get("target_displays"),
|
||||||
|
"display_summary": e.get("display_summary"),
|
||||||
|
"service": e.get("service"),
|
||||||
|
"result": e.get("result"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/ask", response_model=AskResponse)
|
||||||
|
async def ask_question(body: AskRequest, user: dict = Depends(require_auth)):
|
||||||
|
question = body.question.strip()
|
||||||
|
if not question:
|
||||||
|
raise HTTPException(status_code=400, detail="Question is required")
|
||||||
|
|
||||||
|
start, end = _extract_time_range(question)
|
||||||
|
entity = _extract_entity(question)
|
||||||
|
|
||||||
|
# Default to last 7 days if no time range detected
|
||||||
|
if not start:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
start = (now - timedelta(days=7)).isoformat().replace("+00:00", "Z")
|
||||||
|
end = now.isoformat().replace("+00:00", "Z")
|
||||||
|
|
||||||
|
query = _build_event_query(
|
||||||
|
entity,
|
||||||
|
start,
|
||||||
|
end,
|
||||||
|
services=body.services,
|
||||||
|
actor=body.actor,
|
||||||
|
operation=body.operation,
|
||||||
|
result=body.result,
|
||||||
|
include_tags=body.include_tags,
|
||||||
|
exclude_tags=body.exclude_tags,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
total = events_collection.count_documents(query)
|
||||||
|
cursor = events_collection.find(query).sort([("timestamp", -1)]).limit(LLM_MAX_EVENTS)
|
||||||
|
events = list(cursor)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.error("Failed to query events for ask", error=str(exc))
|
||||||
|
raise HTTPException(status_code=500, detail=f"Database query failed: {exc}") from exc
|
||||||
|
|
||||||
|
for e in events:
|
||||||
|
e["_id"] = str(e.get("_id", ""))
|
||||||
|
|
||||||
|
# If no events, return early
|
||||||
|
if not events:
|
||||||
|
return AskResponse(
|
||||||
|
answer="I couldn't find any audit events matching your question. Try broadening the time range or checking the spelling of the device/user name.",
|
||||||
|
events=[],
|
||||||
|
query_info={"entity": entity, "start": start, "end": end, "event_count": 0},
|
||||||
|
llm_used=False,
|
||||||
|
llm_error="LLM not used — no events found." if not LLM_API_KEY else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Try LLM summarisation
|
||||||
|
answer = ""
|
||||||
|
llm_used = False
|
||||||
|
llm_error = None
|
||||||
|
if not LLM_API_KEY:
|
||||||
|
llm_error = "LLM_API_KEY is not configured. Set it in your .env to enable AI narrative summarisation."
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
answer = await _call_llm(question, events, total=total)
|
||||||
|
llm_used = True
|
||||||
|
except Exception as exc:
|
||||||
|
llm_error = f"LLM call failed: {exc}"
|
||||||
|
logger.warning("LLM call failed, falling back to structured summary", error=str(exc))
|
||||||
|
|
||||||
|
# Fallback: structured summary if LLM unavailable or failed
|
||||||
|
if not answer:
|
||||||
|
parts = [f"Found {len(events)} event(s)"]
|
||||||
|
if entity:
|
||||||
|
parts.append(f"related to **{entity}**")
|
||||||
|
parts.append(f"between {start[:10]} and {end[:10]}.\n")
|
||||||
|
|
||||||
|
for i, e in enumerate(events[:10], 1):
|
||||||
|
ts = e.get("timestamp", "?")[:16].replace("T", " ")
|
||||||
|
op = e.get("operation", "unknown action")
|
||||||
|
actor = e.get("actor_display", "unknown")
|
||||||
|
targets = ", ".join(e.get("target_displays") or []) or "—"
|
||||||
|
result = e.get("result", "—")
|
||||||
|
parts.append(f"{i}. **{ts}** — {op} by {actor} on {targets} ({result})")
|
||||||
|
|
||||||
|
if len(events) > 10:
|
||||||
|
parts.append(f"\n...and {len(events) - 10} more events.")
|
||||||
|
|
||||||
|
answer = "\n".join(parts)
|
||||||
|
|
||||||
|
return AskResponse(
|
||||||
|
answer=answer,
|
||||||
|
events=[_to_event_ref(e) for e in events],
|
||||||
|
query_info={
|
||||||
|
"entity": entity,
|
||||||
|
"start": start,
|
||||||
|
"end": end,
|
||||||
|
"event_count": len(events),
|
||||||
|
"total_matched": total,
|
||||||
|
"mongo_query": json.dumps(query, default=str),
|
||||||
|
},
|
||||||
|
llm_used=llm_used,
|
||||||
|
llm_error=llm_error,
|
||||||
|
)
|
||||||
@@ -8,6 +8,7 @@ from bson import ObjectId
|
|||||||
from database import events_collection
|
from database import events_collection
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||||
from models.api import (
|
from models.api import (
|
||||||
|
BulkTagsRequest,
|
||||||
CommentAddRequest,
|
CommentAddRequest,
|
||||||
FilterOptionsResponse,
|
FilterOptionsResponse,
|
||||||
PaginatedEventResponse,
|
PaginatedEventResponse,
|
||||||
@@ -31,10 +32,9 @@ def _decode_cursor(cursor: str) -> tuple[str, str]:
|
|||||||
raise HTTPException(status_code=400, detail="Invalid cursor") from exc
|
raise HTTPException(status_code=400, detail="Invalid cursor") from exc
|
||||||
|
|
||||||
|
|
||||||
@router.get("/events", response_model=PaginatedEventResponse)
|
def _build_query(
|
||||||
def list_events(
|
|
||||||
service: str | None = None,
|
service: str | None = None,
|
||||||
services: list[str] | None = Query(default=None),
|
services: list[str] | None = None,
|
||||||
actor: str | None = None,
|
actor: str | None = None,
|
||||||
operation: str | None = None,
|
operation: str | None = None,
|
||||||
result: str | None = None,
|
result: str | None = None,
|
||||||
@@ -42,9 +42,9 @@ def list_events(
|
|||||||
end: str | None = None,
|
end: str | None = None,
|
||||||
search: str | None = None,
|
search: str | None = None,
|
||||||
cursor: str | None = None,
|
cursor: str | None = None,
|
||||||
page_size: int = Query(default=50, ge=1, le=500),
|
include_tags: list[str] | None = None,
|
||||||
user: dict = Depends(require_auth),
|
exclude_tags: list[str] | None = None,
|
||||||
):
|
) -> dict:
|
||||||
filters = []
|
filters = []
|
||||||
|
|
||||||
if service:
|
if service:
|
||||||
@@ -87,6 +87,10 @@ def list_events(
|
|||||||
]
|
]
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
if include_tags:
|
||||||
|
filters.append({"tags": {"$all": include_tags}})
|
||||||
|
if exclude_tags:
|
||||||
|
filters.append({"tags": {"$not": {"$all": exclude_tags}}})
|
||||||
|
|
||||||
if cursor:
|
if cursor:
|
||||||
try:
|
try:
|
||||||
@@ -102,17 +106,44 @@ def list_events(
|
|||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
query = {"$and": filters} if filters else {}
|
return {"$and": filters} if filters else {}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/events", response_model=PaginatedEventResponse)
|
||||||
|
def list_events(
|
||||||
|
service: str | None = None,
|
||||||
|
services: list[str] | None = Query(default=None),
|
||||||
|
actor: str | None = None,
|
||||||
|
operation: str | None = None,
|
||||||
|
result: str | None = None,
|
||||||
|
start: str | None = None,
|
||||||
|
end: str | None = None,
|
||||||
|
search: str | None = None,
|
||||||
|
cursor: str | None = None,
|
||||||
|
page_size: int = Query(default=50, ge=1, le=500),
|
||||||
|
include_tags: list[str] | None = Query(default=None),
|
||||||
|
exclude_tags: list[str] | None = Query(default=None),
|
||||||
|
user: dict = Depends(require_auth),
|
||||||
|
):
|
||||||
|
query = _build_query(
|
||||||
|
service=service,
|
||||||
|
services=services,
|
||||||
|
actor=actor,
|
||||||
|
operation=operation,
|
||||||
|
result=result,
|
||||||
|
start=start,
|
||||||
|
end=end,
|
||||||
|
search=search,
|
||||||
|
cursor=cursor,
|
||||||
|
include_tags=include_tags,
|
||||||
|
exclude_tags=exclude_tags,
|
||||||
|
)
|
||||||
|
|
||||||
safe_page_size = max(1, min(page_size, 500))
|
safe_page_size = max(1, min(page_size, 500))
|
||||||
|
|
||||||
try:
|
try:
|
||||||
total = events_collection.count_documents(query) if not cursor else -1
|
total = events_collection.count_documents(query) if not cursor else -1
|
||||||
cursor_query = (
|
cursor_query = events_collection.find(query).sort([("timestamp", -1), ("_id", -1)]).limit(safe_page_size)
|
||||||
events_collection.find(query)
|
|
||||||
.sort([("timestamp", -1), ("_id", -1)])
|
|
||||||
.limit(safe_page_size)
|
|
||||||
)
|
|
||||||
events = list(cursor_query)
|
events = list(cursor_query)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise HTTPException(status_code=500, detail=f"Failed to query events: {exc}") from exc
|
raise HTTPException(status_code=500, detail=f"Failed to query events: {exc}") from exc
|
||||||
@@ -125,10 +156,28 @@ def list_events(
|
|||||||
for e in events:
|
for e in events:
|
||||||
e["_id"] = str(e["_id"])
|
e["_id"] = str(e["_id"])
|
||||||
|
|
||||||
log_action("list_events", "/api/events", {"filters": {k: v for k, v in {
|
log_action(
|
||||||
"service": service, "actor": actor, "operation": operation, "result": result,
|
"list_events",
|
||||||
"start": start, "end": end, "search": search, "cursor": cursor, "page_size": page_size,
|
"/api/events",
|
||||||
}.items() if v is not None}}, user.get("sub", "anonymous"))
|
{
|
||||||
|
"filters": {
|
||||||
|
k: v
|
||||||
|
for k, v in {
|
||||||
|
"service": service,
|
||||||
|
"actor": actor,
|
||||||
|
"operation": operation,
|
||||||
|
"result": result,
|
||||||
|
"start": start,
|
||||||
|
"end": end,
|
||||||
|
"search": search,
|
||||||
|
"cursor": cursor,
|
||||||
|
"page_size": page_size,
|
||||||
|
}.items()
|
||||||
|
if v is not None
|
||||||
|
}
|
||||||
|
},
|
||||||
|
user.get("sub", "anonymous"),
|
||||||
|
)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"items": events,
|
"items": events,
|
||||||
@@ -138,6 +187,53 @@ def list_events(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/events/bulk-tags")
|
||||||
|
def bulk_tags(
|
||||||
|
body: BulkTagsRequest,
|
||||||
|
service: str | None = None,
|
||||||
|
services: list[str] | None = Query(default=None),
|
||||||
|
actor: str | None = None,
|
||||||
|
operation: str | None = None,
|
||||||
|
result: str | None = None,
|
||||||
|
start: str | None = None,
|
||||||
|
end: str | None = None,
|
||||||
|
search: str | None = None,
|
||||||
|
include_tags: list[str] | None = Query(default=None),
|
||||||
|
exclude_tags: list[str] | None = Query(default=None),
|
||||||
|
user: dict = Depends(require_auth),
|
||||||
|
):
|
||||||
|
query = _build_query(
|
||||||
|
service=service,
|
||||||
|
services=services,
|
||||||
|
actor=actor,
|
||||||
|
operation=operation,
|
||||||
|
result=result,
|
||||||
|
start=start,
|
||||||
|
end=end,
|
||||||
|
search=search,
|
||||||
|
include_tags=include_tags,
|
||||||
|
exclude_tags=exclude_tags,
|
||||||
|
)
|
||||||
|
tags = [t.strip() for t in body.tags if t.strip()]
|
||||||
|
if not tags:
|
||||||
|
raise HTTPException(status_code=400, detail="No tags provided")
|
||||||
|
|
||||||
|
update = {"$set": {"tags": tags}} if body.mode == "replace" else {"$addToSet": {"tags": {"$each": tags}}}
|
||||||
|
|
||||||
|
try:
|
||||||
|
result_obj = events_collection.update_many(query, update)
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=500, detail=f"Failed to update tags: {exc}") from exc
|
||||||
|
|
||||||
|
log_action(
|
||||||
|
"bulk_tags",
|
||||||
|
"/api/events/bulk-tags",
|
||||||
|
{"tags": tags, "mode": body.mode, "matched": result_obj.matched_count},
|
||||||
|
user.get("sub", "anonymous"),
|
||||||
|
)
|
||||||
|
return {"matched": result_obj.matched_count, "modified": result_obj.modified_count}
|
||||||
|
|
||||||
|
|
||||||
@router.get("/filter-options", response_model=FilterOptionsResponse)
|
@router.get("/filter-options", response_model=FilterOptionsResponse)
|
||||||
def filter_options(limit: int = Query(default=200, ge=1, le=1000)):
|
def filter_options(limit: int = Query(default=200, ge=1, le=1000)):
|
||||||
safe_limit = max(1, min(limit, 1000))
|
safe_limit = max(1, min(limit, 1000))
|
||||||
|
|||||||
@@ -54,11 +54,14 @@ def run_fetch(hours: int = 168):
|
|||||||
if key:
|
if key:
|
||||||
ops.append(UpdateOne({"dedupe_key": key}, {"$set": doc}, upsert=True))
|
ops.append(UpdateOne({"dedupe_key": key}, {"$set": doc}, upsert=True))
|
||||||
else:
|
else:
|
||||||
ops.append(UpdateOne({"id": doc.get("id"), "timestamp": doc.get("timestamp")}, {"$set": doc}, upsert=True))
|
ops.append(
|
||||||
|
UpdateOne({"id": doc.get("id"), "timestamp": doc.get("timestamp")}, {"$set": doc}, upsert=True)
|
||||||
|
)
|
||||||
events_collection.bulk_write(ops, ordered=False)
|
events_collection.bulk_write(ops, ordered=False)
|
||||||
|
|
||||||
if ALERTS_ENABLED:
|
if ALERTS_ENABLED:
|
||||||
from rules import evaluate_event
|
from rules import evaluate_event
|
||||||
|
|
||||||
for doc in normalized:
|
for doc in normalized:
|
||||||
evaluate_event(doc)
|
evaluate_event(doc)
|
||||||
|
|
||||||
@@ -75,7 +78,12 @@ def fetch_logs(
|
|||||||
):
|
):
|
||||||
try:
|
try:
|
||||||
result = run_fetch(hours=hours)
|
result = run_fetch(hours=hours)
|
||||||
log_action("fetch_audit_logs", "/api/fetch-audit-logs", {"hours": hours, "stored": result["stored_events"]}, user.get("sub", "anonymous"))
|
log_action(
|
||||||
|
"fetch_audit_logs",
|
||||||
|
"/api/fetch-audit-logs",
|
||||||
|
{"hours": hours, "stored": result["stored_events"]},
|
||||||
|
user.get("sub", "anonymous"),
|
||||||
|
)
|
||||||
return result
|
return result
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise HTTPException(status_code=502, detail=str(exc)) from exc
|
raise HTTPException(status_code=502, detail=str(exc)) from exc
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
|
|
||||||
from auth import require_auth
|
from auth import require_auth
|
||||||
from fastapi import APIRouter, Depends
|
from fastapi import APIRouter, Depends
|
||||||
from models.api import SourceHealthResponse
|
from models.api import SourceHealthResponse
|
||||||
@@ -19,17 +18,21 @@ def source_health():
|
|||||||
status = doc.get("status")
|
status = doc.get("status")
|
||||||
if not status:
|
if not status:
|
||||||
status = "healthy" if doc.get("last_fetch_time") else "unknown"
|
status = "healthy" if doc.get("last_fetch_time") else "unknown"
|
||||||
results.append({
|
results.append(
|
||||||
"source": source,
|
{
|
||||||
"last_fetch_time": doc.get("last_fetch_time"),
|
"source": source,
|
||||||
"last_attempt_time": doc.get("last_attempt_time"),
|
"last_fetch_time": doc.get("last_fetch_time"),
|
||||||
"status": status,
|
"last_attempt_time": doc.get("last_attempt_time"),
|
||||||
})
|
"status": status,
|
||||||
|
}
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
results.append({
|
results.append(
|
||||||
"source": source,
|
{
|
||||||
"last_fetch_time": None,
|
"source": source,
|
||||||
"last_attempt_time": None,
|
"last_fetch_time": None,
|
||||||
"status": "unknown",
|
"last_attempt_time": None,
|
||||||
})
|
"status": "unknown",
|
||||||
|
}
|
||||||
|
)
|
||||||
return results
|
return results
|
||||||
|
|||||||
@@ -11,10 +11,7 @@ def fetch_intune_audit(hours: int = 24, since: str | None = None, max_pages: int
|
|||||||
"""
|
"""
|
||||||
token = get_access_token()
|
token = get_access_token()
|
||||||
start_time = since or (datetime.utcnow() - timedelta(hours=hours)).isoformat() + "Z"
|
start_time = since or (datetime.utcnow() - timedelta(hours=hours)).isoformat() + "Z"
|
||||||
url = (
|
url = f"https://graph.microsoft.com/v1.0/deviceManagement/auditEvents?$filter=activityDateTime ge {start_time}"
|
||||||
"https://graph.microsoft.com/v1.0/deviceManagement/auditEvents"
|
|
||||||
f"?$filter=activityDateTime ge {start_time}"
|
|
||||||
)
|
|
||||||
headers = {"Authorization": f"Bearer {token}"}
|
headers = {"Authorization": f"Bearer {token}"}
|
||||||
|
|
||||||
events = []
|
events = []
|
||||||
@@ -69,7 +66,8 @@ def _normalize_intune(e: dict) -> dict:
|
|||||||
"targetResources": [
|
"targetResources": [
|
||||||
{
|
{
|
||||||
"id": target.get("id"),
|
"id": target.get("id"),
|
||||||
"displayName": target.get("displayName") or target.get("modifiedProperties", [{}])[0].get("displayName"),
|
"displayName": target.get("displayName")
|
||||||
|
or target.get("modifiedProperties", [{}])[0].get("displayName"),
|
||||||
"type": target.get("type"),
|
"type": target.get("type"),
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ def client(mock_events_collection, mock_watermarks_collection, monkeypatch):
|
|||||||
monkeypatch.setattr("database.events_collection", mock_events_collection)
|
monkeypatch.setattr("database.events_collection", mock_events_collection)
|
||||||
monkeypatch.setattr("routes.fetch.events_collection", mock_events_collection)
|
monkeypatch.setattr("routes.fetch.events_collection", mock_events_collection)
|
||||||
monkeypatch.setattr("routes.events.events_collection", mock_events_collection)
|
monkeypatch.setattr("routes.events.events_collection", mock_events_collection)
|
||||||
|
monkeypatch.setattr("routes.ask.events_collection", mock_events_collection)
|
||||||
monkeypatch.setattr("watermark.watermarks_collection", mock_watermarks_collection)
|
monkeypatch.setattr("watermark.watermarks_collection", mock_watermarks_collection)
|
||||||
monkeypatch.setattr("routes.health.watermarks_collection", mock_watermarks_collection)
|
monkeypatch.setattr("routes.health.watermarks_collection", mock_watermarks_collection)
|
||||||
monkeypatch.setattr("routes.fetch.get_watermark", lambda source: None)
|
monkeypatch.setattr("routes.fetch.get_watermark", lambda source: None)
|
||||||
|
|||||||
@@ -25,15 +25,17 @@ def test_list_events_empty(client):
|
|||||||
|
|
||||||
def test_list_events_cursor_pagination(client, mock_events_collection):
|
def test_list_events_cursor_pagination(client, mock_events_collection):
|
||||||
for i in range(5):
|
for i in range(5):
|
||||||
mock_events_collection.insert_one({
|
mock_events_collection.insert_one(
|
||||||
"id": f"evt-{i}",
|
{
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
"id": f"evt-{i}",
|
||||||
"service": "Directory",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"operation": "Add user",
|
"service": "Directory",
|
||||||
"result": "success",
|
"operation": "Add user",
|
||||||
"actor_display": f"Actor {i}",
|
"result": "success",
|
||||||
"raw_text": "",
|
"actor_display": f"Actor {i}",
|
||||||
})
|
"raw_text": "",
|
||||||
|
}
|
||||||
|
)
|
||||||
response = client.get("/api/events?page_size=2")
|
response = client.get("/api/events?page_size=2")
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
data = response.json()
|
data = response.json()
|
||||||
@@ -48,24 +50,28 @@ def test_list_events_cursor_pagination(client, mock_events_collection):
|
|||||||
|
|
||||||
|
|
||||||
def test_list_events_filter_by_service(client, mock_events_collection):
|
def test_list_events_filter_by_service(client, mock_events_collection):
|
||||||
mock_events_collection.insert_one({
|
mock_events_collection.insert_one(
|
||||||
"id": "evt-1",
|
{
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
"id": "evt-1",
|
||||||
"service": "Exchange",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"operation": "Update",
|
"service": "Exchange",
|
||||||
"result": "success",
|
"operation": "Update",
|
||||||
"actor_display": "Alice",
|
"result": "success",
|
||||||
"raw_text": "",
|
"actor_display": "Alice",
|
||||||
})
|
"raw_text": "",
|
||||||
mock_events_collection.insert_one({
|
}
|
||||||
"id": "evt-2",
|
)
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
mock_events_collection.insert_one(
|
||||||
"service": "Directory",
|
{
|
||||||
"operation": "Add",
|
"id": "evt-2",
|
||||||
"result": "success",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"actor_display": "Bob",
|
"service": "Directory",
|
||||||
"raw_text": "",
|
"operation": "Add",
|
||||||
})
|
"result": "success",
|
||||||
|
"actor_display": "Bob",
|
||||||
|
"raw_text": "",
|
||||||
|
}
|
||||||
|
)
|
||||||
response = client.get("/api/events?service=Exchange")
|
response = client.get("/api/events?service=Exchange")
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
data = response.json()
|
data = response.json()
|
||||||
@@ -74,34 +80,40 @@ def test_list_events_filter_by_service(client, mock_events_collection):
|
|||||||
|
|
||||||
|
|
||||||
def test_list_events_filter_by_services(client, mock_events_collection):
|
def test_list_events_filter_by_services(client, mock_events_collection):
|
||||||
mock_events_collection.insert_one({
|
mock_events_collection.insert_one(
|
||||||
"id": "evt-1",
|
{
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
"id": "evt-1",
|
||||||
"service": "Exchange",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"operation": "Update",
|
"service": "Exchange",
|
||||||
"result": "success",
|
"operation": "Update",
|
||||||
"actor_display": "Alice",
|
"result": "success",
|
||||||
"raw_text": "",
|
"actor_display": "Alice",
|
||||||
})
|
"raw_text": "",
|
||||||
mock_events_collection.insert_one({
|
}
|
||||||
"id": "evt-2",
|
)
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
mock_events_collection.insert_one(
|
||||||
"service": "Directory",
|
{
|
||||||
"operation": "Add",
|
"id": "evt-2",
|
||||||
"result": "success",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"actor_display": "Bob",
|
"service": "Directory",
|
||||||
"raw_text": "",
|
"operation": "Add",
|
||||||
})
|
"result": "success",
|
||||||
mock_events_collection.insert_one({
|
"actor_display": "Bob",
|
||||||
"id": "evt-3",
|
"raw_text": "",
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
}
|
||||||
"service": "Teams",
|
)
|
||||||
"operation": "Delete",
|
mock_events_collection.insert_one(
|
||||||
"result": "success",
|
{
|
||||||
"actor_display": "Charlie",
|
"id": "evt-3",
|
||||||
"raw_text": "",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
})
|
"service": "Teams",
|
||||||
response = client.get("/api/events?service=Exchange&service=Directory")
|
"operation": "Delete",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Charlie",
|
||||||
|
"raw_text": "",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.get("/api/events?services=Exchange&services=Directory")
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
data = response.json()
|
data = response.json()
|
||||||
assert len(data["items"]) == 2
|
assert len(data["items"]) == 2
|
||||||
@@ -117,16 +129,18 @@ def test_list_events_page_size_validation(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_filter_options(client, mock_events_collection):
|
def test_filter_options(client, mock_events_collection):
|
||||||
mock_events_collection.insert_one({
|
mock_events_collection.insert_one(
|
||||||
"id": "evt-1",
|
{
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
"id": "evt-1",
|
||||||
"service": "Intune",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"operation": "Assign",
|
"service": "Intune",
|
||||||
"result": "failure",
|
"operation": "Assign",
|
||||||
"actor_display": "Charlie",
|
"result": "failure",
|
||||||
"actor_upn": "charlie@example.com",
|
"actor_display": "Charlie",
|
||||||
"raw_text": "",
|
"actor_upn": "charlie@example.com",
|
||||||
})
|
"raw_text": "",
|
||||||
|
}
|
||||||
|
)
|
||||||
response = client.get("/api/filter-options")
|
response = client.get("/api/filter-options")
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
data = response.json()
|
data = response.json()
|
||||||
@@ -168,15 +182,17 @@ def test_graph_webhook_notification(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_update_tags(client, mock_events_collection):
|
def test_update_tags(client, mock_events_collection):
|
||||||
mock_events_collection.insert_one({
|
mock_events_collection.insert_one(
|
||||||
"id": "evt-tags",
|
{
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
"id": "evt-tags",
|
||||||
"service": "Directory",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"operation": "Add user",
|
"service": "Directory",
|
||||||
"result": "success",
|
"operation": "Add user",
|
||||||
"actor_display": "Alice",
|
"result": "success",
|
||||||
"raw_text": "",
|
"actor_display": "Alice",
|
||||||
})
|
"raw_text": "",
|
||||||
|
}
|
||||||
|
)
|
||||||
response = client.patch("/api/events/evt-tags/tags", json={"tags": ["investigating", "urgent"]})
|
response = client.patch("/api/events/evt-tags/tags", json={"tags": ["investigating", "urgent"]})
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert response.json()["tags"] == ["investigating", "urgent"]
|
assert response.json()["tags"] == ["investigating", "urgent"]
|
||||||
@@ -185,15 +201,17 @@ def test_update_tags(client, mock_events_collection):
|
|||||||
|
|
||||||
|
|
||||||
def test_add_comment(client, mock_events_collection):
|
def test_add_comment(client, mock_events_collection):
|
||||||
mock_events_collection.insert_one({
|
mock_events_collection.insert_one(
|
||||||
"id": "evt-comment",
|
{
|
||||||
"timestamp": datetime.now(UTC).isoformat(),
|
"id": "evt-comment",
|
||||||
"service": "Directory",
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
"operation": "Add user",
|
"service": "Directory",
|
||||||
"result": "success",
|
"operation": "Add user",
|
||||||
"actor_display": "Alice",
|
"result": "success",
|
||||||
"raw_text": "",
|
"actor_display": "Alice",
|
||||||
})
|
"raw_text": "",
|
||||||
|
}
|
||||||
|
)
|
||||||
response = client.post("/api/events/evt-comment/comments", json={"text": "Looks suspicious"})
|
response = client.post("/api/events/evt-comment/comments", json={"text": "Looks suspicious"})
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
data = response.json()
|
data = response.json()
|
||||||
@@ -241,3 +259,76 @@ def test_rules_crud(client):
|
|||||||
res5 = client.get("/api/rules")
|
res5 = client.get("/api/rules")
|
||||||
assert res5.status_code == 200
|
assert res5.status_code == 200
|
||||||
assert len(res5.json()) == 0
|
assert len(res5.json()) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_events_filter_by_include_tags(client, mock_events_collection):
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-tagged",
|
||||||
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Add user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"raw_text": "",
|
||||||
|
"tags": ["backup", "auto"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-untagged",
|
||||||
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Remove user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Bob",
|
||||||
|
"raw_text": "",
|
||||||
|
"tags": [],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.get("/api/events?include_tags=backup")
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert len(data["items"]) == 1
|
||||||
|
assert data["items"][0]["id"] == "evt-tagged"
|
||||||
|
|
||||||
|
|
||||||
|
def test_bulk_tags_append(client, mock_events_collection):
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-bulk",
|
||||||
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
|
"service": "Exchange",
|
||||||
|
"operation": "Update",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"raw_text": "",
|
||||||
|
"tags": ["existing"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post("/api/events/bulk-tags?service=Exchange", json={"tags": ["backup"], "mode": "append"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["matched"] == 1
|
||||||
|
doc = mock_events_collection.find_one({"id": "evt-bulk"})
|
||||||
|
assert "backup" in doc["tags"]
|
||||||
|
assert "existing" in doc["tags"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_bulk_tags_replace(client, mock_events_collection):
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-bulk2",
|
||||||
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
|
"service": "Exchange",
|
||||||
|
"operation": "Update",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"raw_text": "",
|
||||||
|
"tags": ["old"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post("/api/events/bulk-tags?service=Exchange", json={"tags": ["backup"], "mode": "replace"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
doc = mock_events_collection.find_one({"id": "evt-bulk2"})
|
||||||
|
assert doc["tags"] == ["backup"]
|
||||||
|
|||||||
352
backend/tests/test_ask.py
Normal file
352
backend/tests/test_ask.py
Normal file
@@ -0,0 +1,352 @@
|
|||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
from routes.ask import _build_event_query, _extract_entity, _extract_time_range
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Unit tests: time-range extraction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestExtractTimeRange:
|
||||||
|
def test_last_n_days(self):
|
||||||
|
start, end = _extract_time_range("What happened in the last 3 days?")
|
||||||
|
assert start is not None
|
||||||
|
assert end is not None
|
||||||
|
# Start should be roughly 3 days before end
|
||||||
|
start_dt = datetime.fromisoformat(start.replace("Z", "+00:00"))
|
||||||
|
end_dt = datetime.fromisoformat(end.replace("Z", "+00:00"))
|
||||||
|
delta = end_dt - start_dt
|
||||||
|
assert delta.days == 3
|
||||||
|
|
||||||
|
def test_last_n_hours(self):
|
||||||
|
start, end = _extract_time_range("Show me events in the last 24 hours")
|
||||||
|
start_dt = datetime.fromisoformat(start.replace("Z", "+00:00"))
|
||||||
|
end_dt = datetime.fromisoformat(end.replace("Z", "+00:00"))
|
||||||
|
delta = end_dt - start_dt
|
||||||
|
assert delta.total_seconds() == 24 * 3600
|
||||||
|
|
||||||
|
def test_last_week(self):
|
||||||
|
start, end = _extract_time_range("What happened last week?")
|
||||||
|
start_dt = datetime.fromisoformat(start.replace("Z", "+00:00"))
|
||||||
|
end_dt = datetime.fromisoformat(end.replace("Z", "+00:00"))
|
||||||
|
assert (end_dt - start_dt).days == 7
|
||||||
|
|
||||||
|
def test_yesterday(self):
|
||||||
|
start, end = _extract_time_range("Show me yesterday's events")
|
||||||
|
start_dt = datetime.fromisoformat(start.replace("Z", "+00:00"))
|
||||||
|
end_dt = datetime.fromisoformat(end.replace("Z", "+00:00"))
|
||||||
|
assert (end_dt - start_dt).days == 1
|
||||||
|
|
||||||
|
def test_today(self):
|
||||||
|
start, end = _extract_time_range("What happened today?")
|
||||||
|
start_dt = datetime.fromisoformat(start.replace("Z", "+00:00"))
|
||||||
|
# end_dt is not needed for this assertion
|
||||||
|
# Should be from midnight today to now
|
||||||
|
assert start_dt.hour == 0
|
||||||
|
assert start_dt.minute == 0
|
||||||
|
assert start_dt.second == 0
|
||||||
|
|
||||||
|
def test_no_time_pattern_returns_none(self):
|
||||||
|
start, end = _extract_time_range("What happened to device ABC?")
|
||||||
|
assert start is None
|
||||||
|
assert end is None
|
||||||
|
|
||||||
|
def test_last_n_minutes(self):
|
||||||
|
start, end = _extract_time_range("Show me events in the last 15 minutes")
|
||||||
|
start_dt = datetime.fromisoformat(start.replace("Z", "+00:00"))
|
||||||
|
end_dt = datetime.fromisoformat(end.replace("Z", "+00:00"))
|
||||||
|
assert (end_dt - start_dt).total_seconds() == 15 * 60
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Unit tests: entity extraction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestExtractEntity:
|
||||||
|
def test_device_hint(self):
|
||||||
|
assert _extract_entity("What happened to device LAPTOP-001?") == "LAPTOP-001"
|
||||||
|
|
||||||
|
def test_user_hint(self):
|
||||||
|
assert _extract_entity("Show me user alice@example.com") == "alice@example.com"
|
||||||
|
|
||||||
|
def test_laptop_hint(self):
|
||||||
|
assert _extract_entity("What did laptop HR-Desk-04 do?") == "HR-Desk-04"
|
||||||
|
|
||||||
|
def test_server_hint(self):
|
||||||
|
assert _extract_entity("Check server WEB-01") == "WEB-01"
|
||||||
|
|
||||||
|
def test_quoted_string(self):
|
||||||
|
assert _extract_entity('What happened to "Surface-Pro-7"?') == "Surface-Pro-7"
|
||||||
|
|
||||||
|
def test_single_quoted_string(self):
|
||||||
|
assert _extract_entity("What happened to 'VM-WEB-01' today?") == "VM-WEB-01"
|
||||||
|
|
||||||
|
def test_email_address(self):
|
||||||
|
assert _extract_entity("What did tomas.svensson@contoso.com do?") == "tomas.svensson@contoso.com"
|
||||||
|
|
||||||
|
def test_no_entity_returns_none(self):
|
||||||
|
assert _extract_entity("What happened in the last 3 days?") is None
|
||||||
|
|
||||||
|
def test_vm_hint(self):
|
||||||
|
assert _extract_entity("Show me vm APP-SERVER-02") == "APP-SERVER-02"
|
||||||
|
|
||||||
|
def test_computer_hint(self):
|
||||||
|
assert _extract_entity("What happened to computer DESK-123?") == "DESK-123"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Unit tests: query builder
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestBuildEventQuery:
|
||||||
|
def test_entity_only(self):
|
||||||
|
q = _build_event_query("ABC123", None, None)
|
||||||
|
assert "$and" in q
|
||||||
|
or_clause = q["$and"][0]["$or"]
|
||||||
|
assert any("target_displays" in c for c in or_clause)
|
||||||
|
assert any("actor_display" in c for c in or_clause)
|
||||||
|
assert any("raw_text" in c for c in or_clause)
|
||||||
|
|
||||||
|
def test_time_only(self):
|
||||||
|
q = _build_event_query(None, "2024-01-01T00:00:00Z", "2024-01-02T00:00:00Z")
|
||||||
|
assert q["$and"][0]["timestamp"]["$gte"] == "2024-01-01T00:00:00Z"
|
||||||
|
assert q["$and"][0]["timestamp"]["$lte"] == "2024-01-02T00:00:00Z"
|
||||||
|
|
||||||
|
def test_entity_and_time(self):
|
||||||
|
q = _build_event_query("DEV-01", "2024-01-01T00:00:00Z", "2024-01-02T00:00:00Z")
|
||||||
|
assert len(q["$and"]) == 2
|
||||||
|
assert "timestamp" in q["$and"][0] or "timestamp" in q["$and"][1]
|
||||||
|
|
||||||
|
def test_empty_returns_empty(self):
|
||||||
|
q = _build_event_query(None, None, None)
|
||||||
|
assert q == {}
|
||||||
|
|
||||||
|
def test_entity_is_escaped_for_regex(self):
|
||||||
|
q = _build_event_query("DEV.01", None, None)
|
||||||
|
# The dot should be escaped in the regex
|
||||||
|
or_clause = q["$and"][0]["$or"]
|
||||||
|
raw_regex = or_clause[-1]["raw_text"]["$regex"]
|
||||||
|
assert raw_regex == "DEV\\.01"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Integration tests: /api/ask endpoint
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestAskEndpoint:
|
||||||
|
def test_ask_empty_question(self, client):
|
||||||
|
response = client.post("/api/ask", json={"question": ""})
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
def test_ask_no_events(self, client):
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to device NONEXISTENT in the last 3 days?"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["answer"] != ""
|
||||||
|
assert data["events"] == []
|
||||||
|
assert data["llm_used"] is False
|
||||||
|
assert data["query_info"]["entity"] == "NONEXISTENT"
|
||||||
|
|
||||||
|
def test_ask_with_events_fallback(self, client, mock_events_collection):
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-ask-1",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Device",
|
||||||
|
"operation": "Update device",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Admin Bob",
|
||||||
|
"actor_upn": "bob@example.com",
|
||||||
|
"target_displays": ["LAPTOP-001"],
|
||||||
|
"display_summary": "Update device | device: LAPTOP-001 by Admin Bob",
|
||||||
|
"raw_text": "LAPTOP-001 something",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to device LAPTOP-001 in the last 3 days?"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["llm_used"] is False
|
||||||
|
assert len(data["events"]) == 1
|
||||||
|
assert data["events"][0]["id"] == "evt-ask-1"
|
||||||
|
assert "LAPTOP-001" in data["answer"]
|
||||||
|
assert data["query_info"]["entity"] == "LAPTOP-001"
|
||||||
|
assert data["query_info"]["event_count"] == 1
|
||||||
|
|
||||||
|
def test_ask_defaults_to_7_days_when_no_time(self, client, mock_events_collection):
|
||||||
|
# Insert an event from 5 days ago
|
||||||
|
five_days_ago = datetime.now(UTC) - timedelta(days=5)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-ask-old",
|
||||||
|
"timestamp": five_days_ago.isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Add user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"target_displays": ["DESKTOP-999"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to DESKTOP-999?"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["query_info"]["event_count"] == 1
|
||||||
|
assert data["events"][0]["id"] == "evt-ask-old"
|
||||||
|
|
||||||
|
def test_ask_event_outside_time_window(self, client, mock_events_collection):
|
||||||
|
# Event from 10 days ago — outside default 7-day window
|
||||||
|
old = datetime.now(UTC) - timedelta(days=10)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-too-old",
|
||||||
|
"timestamp": old.isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Add user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"target_displays": ["OLD-DEVICE"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to OLD-DEVICE?"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
# Default is 7 days, so 10-day-old event should not match
|
||||||
|
assert data["query_info"]["event_count"] == 0
|
||||||
|
|
||||||
|
def test_ask_with_llm(self, client, mock_events_collection, monkeypatch):
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-llm",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Device",
|
||||||
|
"operation": "Wipe device",
|
||||||
|
"result": "failure",
|
||||||
|
"actor_display": "System",
|
||||||
|
"target_displays": ["PHONE-001"],
|
||||||
|
"display_summary": "Wipe device | device: PHONE-001 by System",
|
||||||
|
"raw_text": "PHONE-001 wipe failed",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def fake_llm(question, events, total=None):
|
||||||
|
return "The device had a failed wipe attempt."
|
||||||
|
|
||||||
|
monkeypatch.setattr("routes.ask.LLM_API_KEY", "fake-key")
|
||||||
|
monkeypatch.setattr("routes.ask._call_llm", fake_llm)
|
||||||
|
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to PHONE-001 in the last day?"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["llm_used"] is True
|
||||||
|
assert data["answer"] == "The device had a failed wipe attempt."
|
||||||
|
assert len(data["events"]) == 1
|
||||||
|
|
||||||
|
def test_ask_falls_back_when_llm_errors(self, client, mock_events_collection, monkeypatch):
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-fallback",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Add user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"target_displays": ["USER-001"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def failing_llm(question, events, total=None):
|
||||||
|
raise RuntimeError("LLM service down")
|
||||||
|
|
||||||
|
monkeypatch.setattr("routes.ask.LLM_API_KEY", "fake-key")
|
||||||
|
monkeypatch.setattr("routes.ask._call_llm", failing_llm)
|
||||||
|
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to USER-001?"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["llm_used"] is False # Falls back
|
||||||
|
assert len(data["events"]) == 1
|
||||||
|
assert "Found 1 event" in data["answer"]
|
||||||
|
|
||||||
|
def test_ask_with_explicit_filters(self, client, mock_events_collection):
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-exchange",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Exchange",
|
||||||
|
"operation": "Update",
|
||||||
|
"result": "failure",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"target_displays": ["LAPTOP-001"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-directory",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Add user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"target_displays": ["LAPTOP-001"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post(
|
||||||
|
"/api/ask",
|
||||||
|
json={"question": "What happened to LAPTOP-001?", "services": ["Exchange"], "result": "failure"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["query_info"]["event_count"] == 1
|
||||||
|
assert data["events"][0]["id"] == "evt-exchange"
|
||||||
|
|
||||||
|
def test_ask_with_explicit_actor_filter(self, client, mock_events_collection):
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-bob",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Add user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Bob",
|
||||||
|
"actor_upn": "bob@example.com",
|
||||||
|
"target_displays": ["USER-001"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
mock_events_collection.insert_one(
|
||||||
|
{
|
||||||
|
"id": "evt-alice",
|
||||||
|
"timestamp": now.isoformat(),
|
||||||
|
"service": "Directory",
|
||||||
|
"operation": "Remove user",
|
||||||
|
"result": "success",
|
||||||
|
"actor_display": "Alice",
|
||||||
|
"actor_upn": "alice@example.com",
|
||||||
|
"target_displays": ["USER-001"],
|
||||||
|
"display_summary": "summary",
|
||||||
|
"raw_text": "raw",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response = client.post("/api/ask", json={"question": "What happened to USER-001?", "actor": "bob"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["query_info"]["event_count"] == 1
|
||||||
|
assert data["events"][0]["id"] == "evt-bob"
|
||||||
@@ -30,9 +30,7 @@ def test_normalize_event_basic():
|
|||||||
"userPrincipalName": "alice@example.com",
|
"userPrincipalName": "alice@example.com",
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"targetResources": [
|
"targetResources": [{"id": "t1", "displayName": "Bob", "type": "User"}],
|
||||||
{"id": "t1", "displayName": "Bob", "type": "User"}
|
|
||||||
],
|
|
||||||
}
|
}
|
||||||
out = normalize_event(e)
|
out = normalize_event(e)
|
||||||
assert out["id"] == "abc"
|
assert out["id"] == "abc"
|
||||||
|
|||||||
@@ -1,29 +1,35 @@
|
|||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
from rules import _matches, evaluate_event
|
|
||||||
|
|
||||||
|
|
||||||
def test_matches_equals():
|
def test_matches_equals():
|
||||||
rule = {"conditions": [{"field": "operation", "op": "eq", "value": "Add user"}]}
|
rule = {"conditions": [{"field": "operation", "op": "eq", "value": "Add user"}]}
|
||||||
event = {"operation": "Add user", "timestamp": datetime.now(UTC).isoformat()}
|
event = {"operation": "Add user", "timestamp": datetime.now(UTC).isoformat()}
|
||||||
|
from rules import _matches
|
||||||
|
|
||||||
assert _matches(rule, event) is True
|
assert _matches(rule, event) is True
|
||||||
|
|
||||||
|
|
||||||
def test_matches_not_equals():
|
def test_matches_not_equals():
|
||||||
rule = {"conditions": [{"field": "operation", "op": "neq", "value": "Delete user"}]}
|
rule = {"conditions": [{"field": "operation", "op": "neq", "value": "Delete user"}]}
|
||||||
event = {"operation": "Add user", "timestamp": datetime.now(UTC).isoformat()}
|
event = {"operation": "Add user", "timestamp": datetime.now(UTC).isoformat()}
|
||||||
|
from rules import _matches
|
||||||
|
|
||||||
assert _matches(rule, event) is True
|
assert _matches(rule, event) is True
|
||||||
|
|
||||||
|
|
||||||
def test_matches_contains():
|
def test_matches_contains():
|
||||||
rule = {"conditions": [{"field": "actor_display", "op": "contains", "value": "Admin"}]}
|
rule = {"conditions": [{"field": "actor_display", "op": "contains", "value": "Admin"}]}
|
||||||
event = {"actor_display": "Admin (admin@example.com)", "timestamp": datetime.now(UTC).isoformat()}
|
event = {"actor_display": "Admin (admin@example.com)", "timestamp": datetime.now(UTC).isoformat()}
|
||||||
|
from rules import _matches
|
||||||
|
|
||||||
assert _matches(rule, event) is True
|
assert _matches(rule, event) is True
|
||||||
|
|
||||||
|
|
||||||
def test_matches_after_hours():
|
def test_matches_after_hours():
|
||||||
rule = {"conditions": [{"field": "timestamp", "op": "after_hours", "value": None}]}
|
rule = {"conditions": [{"field": "timestamp", "op": "after_hours", "value": None}]}
|
||||||
event = {"timestamp": "2024-01-01T22:00:00Z"}
|
event = {"timestamp": "2024-01-01T22:00:00Z"}
|
||||||
|
from rules import _matches
|
||||||
|
|
||||||
assert _matches(rule, event) is True
|
assert _matches(rule, event) is True
|
||||||
|
|
||||||
event2 = {"timestamp": "2024-01-01T10:00:00Z"}
|
event2 = {"timestamp": "2024-01-01T10:00:00Z"}
|
||||||
@@ -31,13 +37,24 @@ def test_matches_after_hours():
|
|||||||
|
|
||||||
|
|
||||||
def test_evaluate_event_creates_alert(monkeypatch):
|
def test_evaluate_event_creates_alert(monkeypatch):
|
||||||
from rules import alerts_collection
|
from rules import alerts_collection, evaluate_event
|
||||||
|
|
||||||
monkeypatch.setattr("rules.load_rules", lambda: [
|
monkeypatch.setattr(
|
||||||
{"_id": "r1", "name": "Test rule", "enabled": True, "severity": "high", "conditions": [{"field": "operation", "op": "eq", "value": "Add user"}], "message": "Alert!"}
|
"rules.load_rules",
|
||||||
])
|
lambda: [
|
||||||
|
{
|
||||||
|
"_id": "r1",
|
||||||
|
"name": "Test rule",
|
||||||
|
"enabled": True,
|
||||||
|
"severity": "high",
|
||||||
|
"conditions": [{"field": "operation", "op": "eq", "value": "Add user"}],
|
||||||
|
"message": "Alert!",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
inserted = {}
|
inserted = {}
|
||||||
|
|
||||||
def mock_insert(doc):
|
def mock_insert(doc):
|
||||||
inserted["doc"] = doc
|
inserted["doc"] = doc
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
|
|
||||||
import requests
|
import requests
|
||||||
import structlog
|
import structlog
|
||||||
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
|
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
|
||||||
@@ -18,12 +17,16 @@ RETRY_CONFIG = {
|
|||||||
|
|
||||||
|
|
||||||
@retry(**RETRY_CONFIG)
|
@retry(**RETRY_CONFIG)
|
||||||
def get_with_retry(url: str, headers: dict | None = None, params: dict | None = None, timeout: float = 20) -> requests.Response:
|
def get_with_retry(
|
||||||
|
url: str, headers: dict | None = None, params: dict | None = None, timeout: float = 20
|
||||||
|
) -> requests.Response:
|
||||||
res = requests.get(url, headers=headers, params=params, timeout=timeout)
|
res = requests.get(url, headers=headers, params=params, timeout=timeout)
|
||||||
return res
|
return res
|
||||||
|
|
||||||
|
|
||||||
@retry(**RETRY_CONFIG)
|
@retry(**RETRY_CONFIG)
|
||||||
def post_with_retry(url: str, headers: dict | None = None, data: dict | None = None, params: dict | None = None, timeout: float = 15) -> requests.Response:
|
def post_with_retry(
|
||||||
|
url: str, headers: dict | None = None, data: dict | None = None, params: dict | None = None, timeout: float = 15
|
||||||
|
) -> requests.Response:
|
||||||
res = requests.post(url, headers=headers, data=data, params=params, timeout=timeout)
|
res = requests.post(url, headers=headers, data=data, params=params, timeout=timeout)
|
||||||
return res
|
return res
|
||||||
|
|||||||
65
docker-compose.prod.yml
Normal file
65
docker-compose.prod.yml
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
services:
|
||||||
|
mongo:
|
||||||
|
image: mongo:7
|
||||||
|
container_name: aoc-mongo
|
||||||
|
restart: always
|
||||||
|
# Do NOT expose MongoDB port to the host in production
|
||||||
|
# Only backend can reach it via the internal Docker network
|
||||||
|
environment:
|
||||||
|
MONGO_INITDB_ROOT_USERNAME: ${MONGO_ROOT_USERNAME}
|
||||||
|
MONGO_INITDB_ROOT_PASSWORD: ${MONGO_ROOT_PASSWORD}
|
||||||
|
volumes:
|
||||||
|
- mongo_data:/data/db
|
||||||
|
networks:
|
||||||
|
- aoc-internal
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
|
backend:
|
||||||
|
image: git.cqre.net/cqrenet/aoc-backend:${AOC_VERSION:-latest}
|
||||||
|
container_name: aoc-backend
|
||||||
|
restart: always
|
||||||
|
env_file:
|
||||||
|
- .env
|
||||||
|
environment:
|
||||||
|
MONGO_URI: mongodb://${MONGO_ROOT_USERNAME}:${MONGO_ROOT_PASSWORD}@mongo:27017/
|
||||||
|
depends_on:
|
||||||
|
mongo:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- aoc-internal
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
|
||||||
|
interval: 15s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 3
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
|
nginx:
|
||||||
|
image: nginx:alpine
|
||||||
|
container_name: aoc-nginx
|
||||||
|
restart: always
|
||||||
|
ports:
|
||||||
|
- "80:80"
|
||||||
|
- "443:443"
|
||||||
|
volumes:
|
||||||
|
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||||
|
- ./nginx/ssl:/etc/nginx/ssl:ro
|
||||||
|
depends_on:
|
||||||
|
backend:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- aoc-internal
|
||||||
|
- aoc-public
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
mongo_data:
|
||||||
|
|
||||||
|
networks:
|
||||||
|
aoc-internal:
|
||||||
|
internal: true
|
||||||
|
aoc-public:
|
||||||
@@ -13,6 +13,8 @@ services:
|
|||||||
|
|
||||||
backend:
|
backend:
|
||||||
build: ./backend
|
build: ./backend
|
||||||
|
# For production, use the pre-built image instead:
|
||||||
|
# image: git.cqre.net/cqrenet/aoc-backend:v1.2.5
|
||||||
container_name: aoc-backend
|
container_name: aoc-backend
|
||||||
restart: always
|
restart: always
|
||||||
env_file:
|
env_file:
|
||||||
|
|||||||
94
nginx/nginx.conf
Normal file
94
nginx/nginx.conf
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
user nginx;
|
||||||
|
worker_processes auto;
|
||||||
|
error_log /var/log/nginx/error.log warn;
|
||||||
|
pid /var/run/nginx.pid;
|
||||||
|
|
||||||
|
events {
|
||||||
|
worker_connections 1024;
|
||||||
|
}
|
||||||
|
|
||||||
|
http {
|
||||||
|
include /etc/nginx/mime.types;
|
||||||
|
default_type application/octet-stream;
|
||||||
|
|
||||||
|
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
||||||
|
'$status $body_bytes_sent "$http_referer" '
|
||||||
|
'"$http_user_agent" "$http_x_forwarded_for"';
|
||||||
|
|
||||||
|
access_log /var/log/nginx/access.log main;
|
||||||
|
|
||||||
|
sendfile on;
|
||||||
|
tcp_nopush on;
|
||||||
|
tcp_nodelay on;
|
||||||
|
keepalive_timeout 65;
|
||||||
|
types_hash_max_size 2048;
|
||||||
|
|
||||||
|
# Gzip compression
|
||||||
|
gzip on;
|
||||||
|
gzip_vary on;
|
||||||
|
gzip_proxied any;
|
||||||
|
gzip_comp_level 6;
|
||||||
|
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
|
||||||
|
|
||||||
|
# Security headers
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-XSS-Protection "1; mode=block" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
# Upstream backend
|
||||||
|
upstream aoc_backend {
|
||||||
|
server backend:8000;
|
||||||
|
}
|
||||||
|
|
||||||
|
# HTTP → HTTPS redirect (optional; enable once TLS is configured)
|
||||||
|
# server {
|
||||||
|
# listen 80;
|
||||||
|
# server_name _;
|
||||||
|
# return 301 https://$host$request_uri;
|
||||||
|
# }
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
server_name _;
|
||||||
|
|
||||||
|
client_max_body_size 50M;
|
||||||
|
proxy_connect_timeout 60s;
|
||||||
|
proxy_send_timeout 60s;
|
||||||
|
proxy_read_timeout 60s;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://aoc_backend;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_buffering off;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# HTTPS server (uncomment and configure once you have certificates)
|
||||||
|
# server {
|
||||||
|
# listen 443 ssl http2;
|
||||||
|
# server_name _;
|
||||||
|
#
|
||||||
|
# ssl_certificate /etc/nginx/ssl/cert.pem;
|
||||||
|
# ssl_certificate_key /etc/nginx/ssl/key.pem;
|
||||||
|
# ssl_protocols TLSv1.2 TLSv1.3;
|
||||||
|
# ssl_ciphers HIGH:!aNULL:!MD5;
|
||||||
|
# ssl_prefer_server_ciphers on;
|
||||||
|
#
|
||||||
|
# client_max_body_size 50M;
|
||||||
|
#
|
||||||
|
# location / {
|
||||||
|
# proxy_pass http://aoc_backend;
|
||||||
|
# proxy_http_version 1.1;
|
||||||
|
# proxy_set_header Host $host;
|
||||||
|
# proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
# proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
# proxy_buffering off;
|
||||||
|
# }
|
||||||
|
# }
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user