Client storage access had four implementations: raw SAS HTTP and native SigV4 for downloads, Az.Storage cmdlets and SigV4 or AWS Tools for the hash export upload, plus separate serial and parallel download loops. The parallel loop needed PowerShell 7 (ForEach-Object -Parallel) and injected helper functions into its runspaces as text through Invoke-Expression. Now Private/Storage.ps1 has one path: - Get-StorageConfig: provider settings, validated up front (Azure account/container/SAS, S3 endpoint/bucket/keys; s3Region defaults to us-east-1, s3ForcePathStyle to true). - New-StorageRequest: GET/PUT request for Azure (SAS URI) or S3 (SigV4). A PUT carries a digest the service verifies before storing: Content-MD5 on Azure, the signed x-amz-content-sha256 on S3. - Invoke-StorageDownload: streams to disk; retries network errors and 408/429/5xx (5 attempts, 2-30 s backoff), fails at once on other statuses (previously a 403/404 was retried for ~30 s). - Invoke-StorageUpload: one PUT, service error body surfaced. - Invoke-InModuleRunspace: runs a module function per parameter set in a runspace pool whose sessions import the module normally - works on Windows PowerShell 5.1 and 7, no code injection, failures collected and rethrown after all workers finish. Update-ElysiumKhdb (434 -> 186 lines) fetches the manifest with Invoke-StorageDownload and runs Receive-KhdbShard (download, size and SHA-256 check, move into place) per shard through the pool, so -MaxParallelTransfers now works on 5.1 too. Send-ElysiumHashExport (230 -> 132 lines) validates storage settings before replicating anything and uploads with Invoke-StorageUpload. The download-back comparison, the delete-on-mismatch branch and the Az.Storage container probe are gone: a mismatched body is rejected by the service and never stored, and the credentials no longer need read or list access to the upload location. Removed: Get-FunctionDefinitionText, New-HttpClient, Invoke-DownloadWithRetry, Invoke-S3HttpDownloadWithRetry, Ensure-AWSS3Module, New-S3Client, Invoke-S3PutFile/GetToFile/DeleteFile, Get-FileSha256Hex, Get-FileChecksum; the s3UseAwsTools setting. TLS 1.2 is set once at module import. Known limit, marked in code: MD5 is unavailable on FIPS-enforced Windows, so Azure uploads would fail there (S3 is unaffected); Test-ElysiumPasswordQuality already refuses FIPS hosts. Tests now run the real HttpClient/SigV4/runspace code against tests/FakeStorage.ps1, an S3-style store on HttpListener that, like S3, only accepts a PUT whose body matches x-amz-content-sha256. Covered: incremental sync with 1/2/4 runspaces, a tampered shard failing the run while keeping the previous KHDB, the publish -> update round trip with 8 runspaces, Azure PUT headers (Content-MD5, block blob), up-front settings validation, an S3 upload with no read-back, a rejected upload surfacing the service error, 404 failing without retry, and a 503 retried once. 30/30 pass on pwsh 7; mutations (404 treated as transient, worker failures dropped) each fail exactly the matching test. Windows PowerShell 5.1 runs are still to do.
77 lines
3.7 KiB
PowerShell
77 lines
3.7 KiB
PowerShell
# Test helper: a minimal S3-style object store on HttpListener, served from a background runspace so
|
|
# the real HttpClient/SigV4/runspace-pool code can run against it on any OS.
|
|
# - GET <bucket>/<key> serves $Root/<bucket>/<key>, 404 if missing.
|
|
# - PUT stores the body only if it matches x-amz-content-sha256, else 400 BadDigest (as S3 does).
|
|
# - Keys containing '/flaky/' answer 503 to their first GET; keys containing '/reject/' refuse PUTs.
|
|
# Requests are logged as "<METHOD> <bucket>/<key>"; Get-FakeStorageLog drains the log.
|
|
|
|
function Start-FakeStorage {
|
|
param([Parameter(Mandatory)][string]$Root)
|
|
|
|
$probe = [System.Net.Sockets.TcpListener]::new([System.Net.IPAddress]::Loopback, 0)
|
|
$probe.Start(); $port = $probe.LocalEndpoint.Port; $probe.Stop()
|
|
$listener = [System.Net.HttpListener]::new()
|
|
$listener.Prefixes.Add("http://localhost:$port/")
|
|
$listener.Start()
|
|
$log = [System.Collections.Concurrent.ConcurrentQueue[string]]::new()
|
|
|
|
$serve = {
|
|
param($listener, $root, $log)
|
|
$gets = @{}
|
|
while ($listener.IsListening) {
|
|
try { $context = $listener.GetContext() } catch { break }
|
|
$request = $context.Request
|
|
$response = $context.Response
|
|
$key = [uri]::UnescapeDataString($request.Url.AbsolutePath.TrimStart('/'))
|
|
$path = Join-Path $root $key
|
|
$log.Enqueue("$($request.HttpMethod) $key")
|
|
try {
|
|
if ($request.HttpMethod -eq 'GET') {
|
|
$gets[$key] = 1 + [int]$gets[$key]
|
|
if ($key -like '*/flaky/*' -and $gets[$key] -eq 1) {
|
|
$response.StatusCode = 503
|
|
} elseif (Test-Path -LiteralPath $path -PathType Leaf) {
|
|
$bytes = [System.IO.File]::ReadAllBytes($path)
|
|
$response.ContentLength64 = $bytes.Length
|
|
$response.OutputStream.Write($bytes, 0, $bytes.Length)
|
|
} else {
|
|
$response.StatusCode = 404
|
|
}
|
|
} elseif ($request.HttpMethod -eq 'PUT') {
|
|
$buffer = [System.IO.MemoryStream]::new()
|
|
$request.InputStream.CopyTo($buffer)
|
|
$body = $buffer.ToArray()
|
|
$sha = [System.Security.Cryptography.SHA256]::Create()
|
|
$actual = [BitConverter]::ToString($sha.ComputeHash($body)).Replace('-', '').ToLowerInvariant()
|
|
if ($key -like '*/reject/*' -or $request.Headers['x-amz-content-sha256'] -ne $actual) {
|
|
$response.StatusCode = 400
|
|
$message = [System.Text.Encoding]::UTF8.GetBytes('<Error><Code>BadDigest</Code></Error>')
|
|
$response.OutputStream.Write($message, 0, $message.Length)
|
|
} else {
|
|
[void][System.IO.Directory]::CreateDirectory((Split-Path -Path $path -Parent))
|
|
[System.IO.File]::WriteAllBytes($path, $body)
|
|
}
|
|
}
|
|
} finally {
|
|
$response.Close()
|
|
}
|
|
}
|
|
}
|
|
$shell = [powershell]::Create()
|
|
[void]$shell.AddScript($serve.ToString()).AddArgument($listener).AddArgument($Root).AddArgument($log)
|
|
[pscustomobject]@{ Url = "http://localhost:$port"; Listener = $listener; Shell = $shell; Handle = $shell.BeginInvoke(); Log = $log }
|
|
}
|
|
|
|
function Stop-FakeStorage {
|
|
param([Parameter(Mandatory)]$Server)
|
|
$Server.Listener.Stop()
|
|
$Server.Listener.Close()
|
|
$Server.Shell.Dispose()
|
|
}
|
|
|
|
function Get-FakeStorageLog {
|
|
param([Parameter(Mandatory)]$Server)
|
|
$entry = $null
|
|
while ($Server.Log.TryDequeue([ref]$entry)) { $entry }
|
|
}
|