Tomas Kracmar
b0198012eb
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Replace skip-based pagination with cursor-based pagination (timestamp|_id cursors) - Add Prometheus /metrics endpoint with request latency, fetch volume, and error counters - Implement incremental fetch watermarking per source (watermarks collection in MongoDB) - Add Graph change notification webhook endpoint (/api/webhooks/graph) - Add correlation ID middleware for distributed tracing (x-request-id header) - Update frontend to use cursor-based pagination with Prev/Next navigation - Update tests for cursor pagination, metrics, webhooks, and watermark mocking
19 lines
571 B
Python
19 lines
571 B
Python
from database import db
|
|
|
|
watermarks_collection = db["watermarks"]
|
|
|
|
|
|
def get_watermark(source: str) -> str | None:
|
|
"""Return the ISO timestamp of the last successful fetch for a source."""
|
|
doc = watermarks_collection.find_one({"source": source})
|
|
return doc.get("last_fetch_time") if doc else None
|
|
|
|
|
|
def set_watermark(source: str, timestamp: str):
|
|
"""Persist the latest successful fetch timestamp for a source."""
|
|
watermarks_collection.update_one(
|
|
{"source": source},
|
|
{"$set": {"last_fetch_time": timestamp}},
|
|
upsert=True,
|
|
)
|