The combined khdb.txt was built and validated with PowerShell code per
hash line. Each line paid a function call, a regex, a split and an
uppercase conversion, then a second regex pass validated the output.
A full KHDB is hundreds of millions of lines, so the rebuild took
hours on Windows PowerShell 5.1 - and on the VM it ran overnight and
had not finished. The step printed nothing, so it looked hung.
- KhdbMerge.ps1 (new): Elysium.KhdbMerger appends shards in prefix
order with the same rules as before (32 hex, optional ":digits"
suffix dropped, uppercase output, global sort enforced, duplicates
removed and counted) and Elysium.KhdbText.Validate checks a finished
file in one pass. Compiled with Add-Type, so it runs unchanged on
Windows PowerShell 5.1 and PowerShell 7 with no extra install.
- Merge-ShardsToFile and Validate-KHDBFile keep their signatures and
error messages; the per-line Convert-KHDBLineToHash is removed.
Merge-ShardsToFile prints progress per shard and a duplicate count.
- Update-ElysiumKhdb skips the rebuild when the manifest hash equals
the one recorded by the last successful build, nothing was downloaded
or removed, and khdb.txt exists. The khdb-build.sha256 marker is
written only after the new file is moved into place, so a failed run
cannot leave a state that skips a needed rebuild.
Measured on pwsh 7 (macOS): 5,000,000 hashes in 256 shards merged in
3.6 s and validated in 1.2 s. The Windows PowerShell 5.1 timing still
needs measuring on the VM.
Tests: 39/39 pass. New: lowercase and HASH:count normalization with
cross-shard duplicates removed and counted; malformed content named
by shard and line; bare or non-digit count suffixes rejected;
duplicate warning on validation; a 300,000-hash, 256-shard merge
checked against a sorted reference. Update: unchanged manifest keeps
khdb.txt (same write time), a missing khdb.txt is rebuilt with no
download needed.