Files
elysium/tests/FakeStorage.ps1
tomas.kracmar 06bcd22430 refactor(storage): one HTTP storage path; server-verified uploads; parallel on 5.1
Client storage access had four implementations: raw SAS HTTP and
native SigV4 for downloads, Az.Storage cmdlets and SigV4 or AWS Tools
for the hash export upload, plus separate serial and parallel download
loops. The parallel loop needed PowerShell 7 (ForEach-Object -Parallel)
and injected helper functions into its runspaces as text through
Invoke-Expression.

Now Private/Storage.ps1 has one path:
- Get-StorageConfig: provider settings, validated up front (Azure
  account/container/SAS, S3 endpoint/bucket/keys; s3Region defaults to
  us-east-1, s3ForcePathStyle to true).
- New-StorageRequest: GET/PUT request for Azure (SAS URI) or S3
  (SigV4). A PUT carries a digest the service verifies before storing:
  Content-MD5 on Azure, the signed x-amz-content-sha256 on S3.
- Invoke-StorageDownload: streams to disk; retries network errors and
  408/429/5xx (5 attempts, 2-30 s backoff), fails at once on other
  statuses (previously a 403/404 was retried for ~30 s).
- Invoke-StorageUpload: one PUT, service error body surfaced.
- Invoke-InModuleRunspace: runs a module function per parameter set in
  a runspace pool whose sessions import the module normally - works on
  Windows PowerShell 5.1 and 7, no code injection, failures collected
  and rethrown after all workers finish.

Update-ElysiumKhdb (434 -> 186 lines) fetches the manifest with
Invoke-StorageDownload and runs Receive-KhdbShard (download, size and
SHA-256 check, move into place) per shard through the pool, so
-MaxParallelTransfers now works on 5.1 too.

Send-ElysiumHashExport (230 -> 132 lines) validates storage settings
before replicating anything and uploads with Invoke-StorageUpload. The
download-back comparison, the delete-on-mismatch branch and the
Az.Storage container probe are gone: a mismatched body is rejected by
the service and never stored, and the credentials no longer need read
or list access to the upload location.

Removed: Get-FunctionDefinitionText, New-HttpClient,
Invoke-DownloadWithRetry, Invoke-S3HttpDownloadWithRetry,
Ensure-AWSS3Module, New-S3Client, Invoke-S3PutFile/GetToFile/DeleteFile,
Get-FileSha256Hex, Get-FileChecksum; the s3UseAwsTools setting. TLS
1.2 is set once at module import.

Known limit, marked in code: MD5 is unavailable on FIPS-enforced
Windows, so Azure uploads would fail there (S3 is unaffected);
Test-ElysiumPasswordQuality already refuses FIPS hosts.

Tests now run the real HttpClient/SigV4/runspace code against
tests/FakeStorage.ps1, an S3-style store on HttpListener that, like S3,
only accepts a PUT whose body matches x-amz-content-sha256. Covered:
incremental sync with 1/2/4 runspaces, a tampered shard failing the
run while keeping the previous KHDB, the publish -> update round trip
with 8 runspaces, Azure PUT headers (Content-MD5, block blob), up-front
settings validation, an S3 upload with no read-back, a rejected upload
surfacing the service error, 404 failing without retry, and a 503
retried once. 30/30 pass on pwsh 7; mutations (404 treated as
transient, worker failures dropped) each fail exactly the matching
test. Windows PowerShell 5.1 runs are still to do.
2026-11-04 13:20:31 +01:00

77 lines
3.7 KiB
PowerShell

# Test helper: a minimal S3-style object store on HttpListener, served from a background runspace so
# the real HttpClient/SigV4/runspace-pool code can run against it on any OS.
# - GET <bucket>/<key> serves $Root/<bucket>/<key>, 404 if missing.
# - PUT stores the body only if it matches x-amz-content-sha256, else 400 BadDigest (as S3 does).
# - Keys containing '/flaky/' answer 503 to their first GET; keys containing '/reject/' refuse PUTs.
# Requests are logged as "<METHOD> <bucket>/<key>"; Get-FakeStorageLog drains the log.
function Start-FakeStorage {
param([Parameter(Mandatory)][string]$Root)
$probe = [System.Net.Sockets.TcpListener]::new([System.Net.IPAddress]::Loopback, 0)
$probe.Start(); $port = $probe.LocalEndpoint.Port; $probe.Stop()
$listener = [System.Net.HttpListener]::new()
$listener.Prefixes.Add("http://localhost:$port/")
$listener.Start()
$log = [System.Collections.Concurrent.ConcurrentQueue[string]]::new()
$serve = {
param($listener, $root, $log)
$gets = @{}
while ($listener.IsListening) {
try { $context = $listener.GetContext() } catch { break }
$request = $context.Request
$response = $context.Response
$key = [uri]::UnescapeDataString($request.Url.AbsolutePath.TrimStart('/'))
$path = Join-Path $root $key
$log.Enqueue("$($request.HttpMethod) $key")
try {
if ($request.HttpMethod -eq 'GET') {
$gets[$key] = 1 + [int]$gets[$key]
if ($key -like '*/flaky/*' -and $gets[$key] -eq 1) {
$response.StatusCode = 503
} elseif (Test-Path -LiteralPath $path -PathType Leaf) {
$bytes = [System.IO.File]::ReadAllBytes($path)
$response.ContentLength64 = $bytes.Length
$response.OutputStream.Write($bytes, 0, $bytes.Length)
} else {
$response.StatusCode = 404
}
} elseif ($request.HttpMethod -eq 'PUT') {
$buffer = [System.IO.MemoryStream]::new()
$request.InputStream.CopyTo($buffer)
$body = $buffer.ToArray()
$sha = [System.Security.Cryptography.SHA256]::Create()
$actual = [BitConverter]::ToString($sha.ComputeHash($body)).Replace('-', '').ToLowerInvariant()
if ($key -like '*/reject/*' -or $request.Headers['x-amz-content-sha256'] -ne $actual) {
$response.StatusCode = 400
$message = [System.Text.Encoding]::UTF8.GetBytes('<Error><Code>BadDigest</Code></Error>')
$response.OutputStream.Write($message, 0, $message.Length)
} else {
[void][System.IO.Directory]::CreateDirectory((Split-Path -Path $path -Parent))
[System.IO.File]::WriteAllBytes($path, $body)
}
}
} finally {
$response.Close()
}
}
}
$shell = [powershell]::Create()
[void]$shell.AddScript($serve.ToString()).AddArgument($listener).AddArgument($Root).AddArgument($log)
[pscustomobject]@{ Url = "http://localhost:$port"; Listener = $listener; Shell = $shell; Handle = $shell.BeginInvoke(); Log = $log }
}
function Stop-FakeStorage {
param([Parameter(Mandatory)]$Server)
$Server.Listener.Stop()
$Server.Listener.Close()
$Server.Shell.Dispose()
}
function Get-FakeStorageLog {
param([Parameter(Mandatory)]$Server)
$entry = $null
while ($Server.Log.TryDequeue([ref]$entry)) { $entry }
}