Files
elysium/tests
tomas.kracmar 59e187e9f0 perf(khdb): compiled merge and validation; skip rebuild when unchanged
The combined khdb.txt was built and validated with PowerShell code per
hash line. Each line paid a function call, a regex, a split and an
uppercase conversion, then a second regex pass validated the output.
A full KHDB is hundreds of millions of lines, so the rebuild took
hours on Windows PowerShell 5.1 - and on the VM it ran overnight and
had not finished. The step printed nothing, so it looked hung.

- KhdbMerge.ps1 (new): Elysium.KhdbMerger appends shards in prefix
  order with the same rules as before (32 hex, optional ":digits"
  suffix dropped, uppercase output, global sort enforced, duplicates
  removed and counted) and Elysium.KhdbText.Validate checks a finished
  file in one pass. Compiled with Add-Type, so it runs unchanged on
  Windows PowerShell 5.1 and PowerShell 7 with no extra install.
- Merge-ShardsToFile and Validate-KHDBFile keep their signatures and
  error messages; the per-line Convert-KHDBLineToHash is removed.
  Merge-ShardsToFile prints progress per shard and a duplicate count.
- Update-ElysiumKhdb skips the rebuild when the manifest hash equals
  the one recorded by the last successful build, nothing was downloaded
  or removed, and khdb.txt exists. The khdb-build.sha256 marker is
  written only after the new file is moved into place, so a failed run
  cannot leave a state that skips a needed rebuild.

Measured on pwsh 7 (macOS): 5,000,000 hashes in 256 shards merged in
3.6 s and validated in 1.2 s. The Windows PowerShell 5.1 timing still
needs measuring on the VM.

Tests: 39/39 pass. New: lowercase and HASH:count normalization with
cross-shard duplicates removed and counted; malformed content named
by shard and line; bare or non-digit count suffixes rejected;
duplicate warning on validation; a 300,000-hash, 256-shard merge
checked against a sorted reference. Update: unchanged manifest keeps
khdb.txt (same write time), a missing khdb.txt is rebuilt with no
download needed.
2026-11-05 08:20:41 +01:00
..