26 Commits

Author SHA1 Message Date
tomas.kracmar 27a682a968 Release v2.2.2: fix replication permission check for nested groups
Test-ReplicationPermissions now uses the tokenGroups constructed
attribute to resolve all effective SIDs in the caller's Kerberos
token, including nested group memberships. This replaces the
previous MemberOf walk which missed indirect entitlement and
could produce false-positive missing-permission errors.

All versions bumped to unified v2.2.2.
2026-06-09 11:41:14 +02:00
tomas.kracmar 255cfe0a17 chore: centralize version and add Bump-Version.ps1
- Add  to Elysium.Common.ps1 as the single
  runtime source of truth for version strings.
- Update Update-KHDB.ps1 User-Agent to reference .
- Update Test-WeakADPasswords.ps1 usage beacon payload to reference
  .
- Add Bump-Version.ps1 release helper that updates the centralized
  variable, ASCII headers across .ps1/.py files, runtime references,
  and prints a CHANGELOG stub.
2026-06-09 11:14:20 +02:00
tomas.kracmar 09c30f97e9 Release v2.2.1: DRY refactoring and housekeeping
Consolidated duplicated helpers into Elysium.Common.ps1:
- Settings parsing (Read-KeyValueSettingsFile, Read-ElysiumSettings, Get-SettingsValue)
- Azure Blob URI builder (Build-BlobUri)
- S3 SigV4 signing helpers and AWS module bootstrap
- AD credential validation and replication permission pre-check
- Parallel execution helper (Get-FunctionDefinitionText)

Test-WeakADPasswords.ps1 and Extract-NTHashes.ps1 now import
Elysium.Common.ps1 for the first time. Update-KHDB.ps1 and
Prepare-KHDBStorage.ps1 removed their local duplicates.

Deleted legacy Settings.ps1 (superseded by ElysiumSettings.txt).
Removed stray placeholder comment in Elysium.ps1.

All versions bumped to unified v2.2.1.
2026-06-09 10:52:19 +02:00
tomas.kracmar 5127c2d096 fix(Test-WeakADPasswords): surface replication permission success to console
Replace Write-Verbose with Write-Host so operators see the
permissions verification result without needing -Verbose.
2026-06-09 09:56:24 +02:00
tomas.kracmar ad1db86232 Release v2.2.0: unified versioning and replication permission pre-check
- Unified project versioning (v2.2.0) across all scripts, settings template,
  and documentation. All components now share a single version number.
- Added Test-ReplicationPermissions to Test-WeakADPasswords.ps1 to validate
  the three required AD replication extended rights before DCSync, providing
  clear fail-fast errors when permissions are missing.
- Updated CHANGELOG.md with unified versioning strategy and release history.
- Updated README.md with versioning section and improved troubleshooting docs.
2026-06-09 09:43:43 +02:00
tomas.kracmar 60a7671ceb Fix KHDB password match format handling 2026-03-16 16:38:19 +01:00
Tom Frost 787360c706 Fail fast when FIPS policy is enabled for weak password test 2026-02-17 13:28:39 +01:00
Tom Frost baaee8dc53 Handle DSInternals FIPS bootstrap WriteErrorException 2026-02-17 13:15:14 +01:00
Tom Frost b582bb24b3 Bump Test-WeakADPasswords to v1.4.1 2026-02-17 13:03:34 +01:00
Tom Frost 7f1df7b102 Improve weak password test credential diagnostics and docs 2026-02-17 12:40:04 +01:00
tomas.kracmar 7874c0e65b Clarified privileges needed 2025-12-08 16:24:26 +01:00
tomas.kracmar be96cbf9a5 Bumped versions 2025-11-07 20:56:02 +01:00
tomas.kracmar 42fee2ff84 PS fix 2025-11-07 20:48:52 +01:00
tomas.kracmar 2ff4964537 PS7 fixes 2025-11-07 20:45:53 +01:00
tomas.kracmar 6e4cc874b0 PS 7 bug fixes 2025-11-07 20:42:58 +01:00
tomas.kracmar ec27206453 PS7 fixes 2025-11-07 20:38:26 +01:00
tomas.kracmar a55ef3713f Bug fixes 2025-11-07 20:33:21 +01:00
tomas.kracmar bda19432e2 Update to prefer PS7 if available 2025-11-07 20:21:03 +01:00
tomas.kracmar 7c2bb65a86 Parallel transfers 2025-11-07 18:14:43 +01:00
tomas.kracmar 5a64558bb9 Fix PS 5.1 -Depth handling 2025-11-07 16:27:00 +01:00
tomas.kracmar 4b1b841383 KHDB rework 2025-11-07 15:58:35 +01:00
tomas.kracmar 964e91d20f Fixing handling errors in datasets 2025-10-21 15:34:44 +02:00
tomas.kracmar 353352eeb2 Improve UPN export handling 2025-10-21 14:27:16 +02:00
tomas.kracmar 05e9358357 Versions bump 2025-10-21 13:42:46 +02:00
tomas.kracmar 5799881418 Fixing table sorting and S3 upload 2025-10-21 13:35:09 +02:00
tomas.kracmar 0d9a460057 Fix S3 download 2025-10-20 19:19:14 +02:00
14 changed files with 3462 additions and 551 deletions
+4
View File
@@ -3,3 +3,7 @@ khdb.txt
khdb.txt.zip
ElysiumSettings.txt
/Reports
/khdb-shards
khdb-manifest.json
/elysium/.vscode
/.vscode
+125
View File
@@ -0,0 +1,125 @@
# Elysium — Agent Context
> This file is written for AI coding agents. The project language is English; all code comments, documentation, and settings are in English.
## Project Overview
Elysium is a Windows-focused Active Directory weak-password assessment toolkit. It is implemented entirely in PowerShell (with one Python decryption helper) and is designed to be run from a dedicated, trusted Windows host by an operator with replication-equivalent rights on the target AD domain.
The tool performs three core workflows:
1. **Update Known-Hashes Database (KHDB)** — Download an incremental, sharded hash database from remote storage (Azure Blob or S3-compatible).
2. **Test Weak AD Passwords** — Use DSInternals to replicate account data and test passwords against the local KHDB, producing timestamped text reports.
3. **Extract and Send Current Hashes** — Pull NTLM hashes (without usernames) from live AD, compress, encrypt, and upload them back to the tool provider to improve the KHDB. This step is optional.
4. **Update Lithnet Password Protection Store** — Populate a local Lithnet Password Protection store with compromised hashes, plaintext passwords, and banned words.
## Technology Stack
- **Runtime:** Windows PowerShell 5.1 or PowerShell 7 (`pwsh`).
- **Required PowerShell modules:** `DSInternals`, `ActiveDirectory` (RSAT).
- **Optional PowerShell modules:** `Az.Storage`, `AWS.Tools.S3` / `AWSPowerShell.NetCore`, `LithnetPasswordProtection`.
- **Python 3:** `decrypt.py` requires `pycryptodome` for decrypting uploaded payload files.
- **Network:** TLS 1.2+ enforced; native S3 SigV4 signing implemented in pure PowerShell/.NET (no AWS Tools required).
## Project Structure
```
Elysium.ps1 # Main menu / orchestrator script
Elysium.Common.ps1 # Shared helpers: pwsh relauncher, Windows PS fallback
ElysiumSettings.txt # Live configuration (key=value format, # comments)
ElysiumSettings.txt.sample # Documented template for operators
Update-KHDB.ps1 # KHDB downloader: manifest + shard incremental update
Prepare-KHDBStorage.ps1 # KHDB publisher: split source into shards, build manifest, upload
Test-WeakADPasswords.ps1 # AD password-quality test with DSInternals
Extract-NTHashes.ps1 # Hash extraction, compression, encryption, upload
Update-LithnetStore.ps1 # Lithnet Password Protection store importer
Uninstall.ps1 # Self-removal script
decrypt.py # Python 3 decryptor for .enc payloads
Settings.ps1 # Legacy settings file (superseded by ElysiumSettings.txt)
khdb.txt # Local merged KHDB file (generated by Update-KHDB.ps1)
Reports/ # Generated reports and transcript logs
```
## Configuration Format
`ElysiumSettings.txt` uses a simple `Key=value` format. Lines starting with `#` are comments. Values are **not** quoted by default; if quotes are used they are stripped by some parsers.
Key sections:
- `StorageProvider``Azure` or `S3`
- Azure: `storageAccountName`, `containerName`, `sasToken`
- S3: `s3EndpointUrl`, `s3Region`, `s3BucketName`, `s3AccessKeyId`, `s3SecretAccessKey`, `s3ForcePathStyle`, `s3UseAwsTools`
- KHDB layout: `KhdbManifestPath`, `KhdbShardPrefix`, `KhdbLocalShardDir`
- App: `InstallationPath`, `ReportPathBase`, `WeakPasswordsDatabase`, `CheckOnlyEnabledUsers`
- Lithnet: `LithnetStorePath`, `LithnetSyncHibp`, `LithnetHashSources`, `LithnetPlaintextSources`, `LithnetBannedWordSources`
- Telemetry (optional): `UsageBeaconUrl`, `UsageBeaconMethod`, `UsageBeaconInstanceId`, `UsageBeaconTimeoutSeconds`
- Domains: `Domain1Name`, `Domain1DC`, `Domain2Name`, `Domain2DC`, ...
## PowerShell Edition Handling
This project has nuanced PowerShell edition rules that must be preserved:
- **Scripts that benefit from parallelism** (`Update-KHDB.ps1`, `Prepare-KHDBStorage.ps1`, `Update-LithnetStore.ps1`, `Uninstall.ps1`) auto-relaunch under `pwsh` (PowerShell 7+) via `Restart-WithPwshIfAvailable` in `Elysium.Common.ps1`.
- **Scripts that require legacy modules** (`Test-WeakADPasswords.ps1`, `Extract-NTHashes.ps1`) auto-relaunch under `Windows PowerShell` (`powershell.exe`) via `Restart-WithWindowsPowerShellIfAvailable`, because `DSInternals` and `ActiveDirectory` often need the Desktop edition.
- `Elysium.ps1` (the menu) checks `$PSVersionTable.PSEdition` and invokes the appropriate child process for options 2 and 3.
When modifying any script, keep the `. $commonHelper` import and the corresponding `Restart-With*IfAvailable` call at the top.
## Code Style Guidelines
- Strict mode and error handling are mandatory at the top of every script:
```powershell
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
```
- Every operational script starts a **transcript** into `Reports/logs/` (or `%TEMP%` for uninstall) and stops it in a `finally` block.
- Functions use `Verb-Noun` naming. Private helpers are also `Verb-Noun`.
- Settings parsing is done manually via `Get-Content` and string splitting on `=`, not `ConvertFrom-StringData`.
- Paths are resolved relative to `$PSScriptRoot` whenever possible.
- Large file operations use `StreamReader` / `StreamWriter` with explicit `UTF8Encoding($false)` (no BOM) and large buffers (1 MiB preferred).
- S3-native functions (SigV4) are duplicated in scripts that need them because they run in parallel `ForEach-Object -Parallel` scopes where module imports are not preserved.
## Build / Run Commands
There is no package manager or build system. The tool is run directly:
```powershell
# Main menu
.\Elysium.ps1
# Direct invocation (respects auto-relaunch behavior)
.\Update-KHDB.ps1
.\Update-KHDB.ps1 -MaxParallelTransfers 8
.\Test-WeakADPasswords.ps1
.\Extract-NTHashes.ps1
.\Update-LithnetStore.ps1
.\Prepare-KHDBStorage.ps1 -SourcePath .\khdb.txt -OutputRoot .\publish -StorageProvider S3 -SkipUpload
.\Uninstall.ps1
```
No tests or CI pipelines exist in this repository.
## Testing & Validation
- **KHDB validation:** `Validate-KHDBFile` in `Update-KHDB.ps1` checks 32-hex format, sorting, and duplicates.
- **DSInternals compatibility:** `Resolve-DSInternalsWeakHashFile` in `Test-WeakADPasswords.ps1` normalizes legacy `HASH:count` lines into a temporary hash-only file.
- **Upload integrity:** After every upload, the script downloads the blob back and compares SHA256 checksums before deleting local artifacts.
- **Credential pre-check:** `Test-WeakADPasswords.ps1` validates credentials against the chosen DC with `Get-ADDomain` and checks the three replication extended rights before attempting DCSync.
## Security Considerations
- The operator passphrase is stored in the **user-level** environment variable `ELYSIUM_PASSPHRASE`. Do not log or serialize it.
- `Test-WeakADPasswords.ps1` refuses to run when Windows FIPS policy is enabled (`HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy\Enabled = 1`) because DSInternals operations may fail or be limited.
- The extract-and-send workflow uploads **hashes only** (no usernames). Encryption uses PBKDF2 (SHA-256, 100k iterations, random 16-byte salt) + AES-256-CBC with a 4-byte magic header `ELY1`.
- S3 requests are signed with native SigV4 via `System.Net.Http.HttpClient` so no AWS credentials file or third-party module is required. If AWS Tools are used (`s3UseAwsTools = true`), the native path is still available as a fallback.
- Reports contain sensitive AD data. They are written to `Reports/` under the script root by default.
- All network operations enforce TLS 1.2+ (`[System.Net.ServicePointManager]::SecurityProtocol`).
## Notes for Agents
- Do **not** introduce new package-manager files (e.g., `package.json`, `pyproject.toml`, `Cargo.toml`). This project is intentionally dependency-light and script-driven.
- When adding new settings, keep the `key=value` flat format and update both `ElysiumSettings.txt.sample` and any parser that reads the file.
- If you add a new operational script, include the standard banner, strict mode, transcript logging, and the `Elysium.Common.ps1` relauncher pattern.
- The `.gitignore` excludes `khdb.txt`, `ElysiumSettings.txt`, `Reports/`, `khdb-shards/`, and `khdb-manifest.json`.
+161
View File
@@ -0,0 +1,161 @@
##################################################
## ____ ___ ____ _____ _ _ _____ _____ ##
## / ___/ _ \| _ \| ____| | \ | | ____|_ _| ##
## | | | | | | |_) | _| | \| | _| | | ##
## | |__| |_| | _ <| |___ _| |\ | |___ | | ##
## \____\__\_\_| \_\_____(_)_| \_|_____| |_| ##
## Move fast and fix things. ##
##################################################
## Project: Elysium ##
## File: Bump-Version.ps1 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
#Requires -Version 5.1
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
<#
.SYNOPSIS
Bumps the unified Elysium version across all project files.
.DESCRIPTION
Updates the centralized $ElysiumVersion variable, ASCII headers in all
operational scripts, the settings template, and runtime references
(User-Agent, usage beacon, etc.). Optionally stubs a new CHANGELOG entry.
.PARAMETER NewVersion
The new version string to apply (e.g. 2.2.2).
.PARAMETER SkipChangelog
Do not print a CHANGELOG entry stub.
.EXAMPLE
.\Bump-Version.ps1 -NewVersion 2.2.2
#>
param(
[Parameter(Mandatory = $true)]
[string]$NewVersion,
[switch]$SkipChangelog
)
$scriptRoot = $PSScriptRoot
if (-not $scriptRoot) { $scriptRoot = (Get-Location).Path }
# ---------------------------------------------------------------------------
# Validate input
# ---------------------------------------------------------------------------
if ($NewVersion -notmatch '^\d+\.\d+\.\d+$') {
throw "Version must be in semantic format X.Y.Z (e.g. 2.2.2). Got: '$NewVersion'"
}
# ---------------------------------------------------------------------------
# Determine current version from Elysium.Common.ps1
# ---------------------------------------------------------------------------
$commonPath = Join-Path -Path $scriptRoot -ChildPath 'Elysium.Common.ps1'
if (-not (Test-Path -LiteralPath $commonPath)) {
throw "Elysium.Common.ps1 not found at $commonPath"
}
$commonContent = Get-Content -LiteralPath $commonPath -Raw
$currentVersionMatch = [regex]::Match($commonContent, "\`$script:ElysiumVersion\s*=\s*'([^']+)'")
if (-not $currentVersionMatch.Success) {
throw "Could not determine current version from Elysium.Common.ps1"
}
$oldVersion = $currentVersionMatch.Groups[1].Value
if ($oldVersion -eq $NewVersion) {
Write-Warning "Current version is already $NewVersion. Nothing to do."
return
}
Write-Host "Bumping Elysium from $oldVersion -> $NewVersion"
# ---------------------------------------------------------------------------
# Helper: replace in file
# ---------------------------------------------------------------------------
function Edit-FileVersion {
param(
[string]$Path,
[string]$Old,
[string]$New
)
$content = Get-Content -LiteralPath $Path -Raw
$newContent = $content.Replace($Old, $New)
if ($newContent -eq $content) {
Write-Verbose " No changes in $(Split-Path -Leaf $Path)"
} else {
Set-Content -LiteralPath $Path -Value $newContent -NoNewline -Encoding UTF8
Write-Host " Updated $(Split-Path -Leaf $Path)"
}
}
# ---------------------------------------------------------------------------
# 1. Central version variable
# ---------------------------------------------------------------------------
Write-Host "`n[1/4] Updating centralized version variable..."
Edit-FileVersion -Path $commonPath -Old "`$script:ElysiumVersion = '$oldVersion'" -New "`$script:ElysiumVersion = '$NewVersion'"
# ---------------------------------------------------------------------------
# 2. ASCII headers in scripts and templates
# ---------------------------------------------------------------------------
Write-Host "`n[2/4] Updating script headers..."
$headerTargets = Get-ChildItem -Path $scriptRoot -File | Where-Object {
$_.Extension -in @('.ps1', '.py') -or $_.Name -eq 'ElysiumSettings.txt.sample'
}
foreach ($file in $headerTargets) {
$content = Get-Content -LiteralPath $file.FullName -Raw
# The header pattern: ## Version: X.Y.Z ##
$pattern = "## Version:\s+$([regex]::Escape($oldVersion))\s+##"
$replacement = "## Version: $NewVersion ##"
$newContent = [regex]::Replace($content, $pattern, $replacement)
if ($newContent -ne $content) {
Set-Content -LiteralPath $file.FullName -Value $newContent -NoNewline -Encoding UTF8
Write-Host " Updated $($file.Name)"
}
}
# ---------------------------------------------------------------------------
# 3. Runtime references (safety net)
# ---------------------------------------------------------------------------
Write-Host "`n[3/4] Updating runtime version references..."
$runtimeTargets = Get-ChildItem -Path $scriptRoot -Filter '*.ps1' -File
foreach ($file in $runtimeTargets) {
$content = Get-Content -LiteralPath $file.FullName -Raw
$newContent = $content
# User-Agent patterns: 'Elysium/2.2.1 (+...)' or "Elysium/2.2.1 (+...)"
$newContent = [regex]::Replace($newContent,
"Elysium/$([regex]::Escape($oldVersion))",
"Elysium/$NewVersion")
# Literal string assignments: version = '2.2.1'
$newContent = [regex]::Replace($newContent,
"version\s*=\s*'$([regex]::Escape($oldVersion))'",
"version = '$NewVersion'")
if ($newContent -ne $content) {
Set-Content -LiteralPath $file.FullName -Value $newContent -NoNewline -Encoding UTF8
Write-Host " Updated runtime refs in $($file.Name)"
}
}
# ---------------------------------------------------------------------------
# 4. CHANGELOG stub
# ---------------------------------------------------------------------------
if (-not $SkipChangelog) {
Write-Host "`n[4/4] CHANGELOG entry stub (copy-paste ready):"
$today = (Get-Date).ToString('yyyy-MM-dd')
$stub = @"
---
## [$NewVersion] $today
### Changed
- (describe your change here)
"@
Write-Host $stub
Write-Host "`nAppend the above to CHANGELOG.md, then commit and tag."
}
Write-Host "`nDone. Review the changes with: git diff --stat"
+156 -13
View File
@@ -1,30 +1,173 @@
# Changelog
## 2025-10-10
All notable changes to the Elysium project are documented in this file.
### Test-WeakADPasswords.ps1 v1.3.0
Starting with **v2.2.0**, Elysium uses a **unified project version**. All scripts, settings templates, and documentation share the same version number so operators can verify consistency at a glance. Releases prior to v2.2.0 used per-script versioning; those entries are preserved below under their original dates.
---
## [2.2.2] — 2026-06-09
### Fixed
- `Test-ReplicationPermissions` (in `Elysium.Common.ps1`) now resolves the caller's **effective token SIDs** via the `tokenGroups` constructed attribute instead of walking `MemberOf` directly. This correctly accounts for nested group memberships and avoids false-positive "missing permissions" errors when the account is entitled through nested groups.
---
## [2.2.1] — 2026-06-09
### Changed
- **DRY refactoring — shared helpers consolidated into `Elysium.Common.ps1`:**
- Moved `Read-KeyValueSettingsFile`, `Read-ElysiumSettings`, and `Get-SettingsValue` from `Prepare-KHDBStorage.ps1` and `Update-KHDB.ps1` into the common helper.
- Moved `Build-BlobUri` and Azure URI helpers from `Update-KHDB.ps1` into the common helper.
- Moved `Get-FunctionDefinitionText` from all scripts that duplicated it into the common helper.
- Moved `Get-ValidatedADCredential` and `Test-ReplicationPermissions` from `Test-WeakADPasswords.ps1` into the common helper.
- Moved all native S3 SigV4 helpers (`Ensure-AWSS3Module`, `New-S3Client`, `HmacSha256`, `GetSignatureKey`, `BuildAuthHeaders`, `BuildS3Uri`, etc.) from `Extract-NTHashes.ps1` into the common helper.
- `Test-WeakADPasswords.ps1` and `Extract-NTHashes.ps1` now import `Elysium.Common.ps1` (they previously did not), reducing duplication and ensuring consistent behavior.
- `Update-KHDB.ps1` and `Prepare-KHDBStorage.ps1` removed their local copies of helpers already available in the common module.
- Removed legacy `Settings.ps1` (superseded by `ElysiumSettings.txt`).
- Minor cleanup: removed stray placeholder comment in `Elysium.ps1`.
---
## [2.2.0] — 2026-06-09
### Changed
- **Unified versioning:** All PowerShell scripts, the settings template, and documentation now share a single project version (`2.2.0`). This replaces the previous per-script versioning model.
### Test-WeakADPasswords.ps1
- Added `Test-ReplicationPermissions` helper that validates the three required AD replication extended rights (`Replicating Directory Changes`, `Replicating Directory Changes All`, `Replicating Directory Changes In Filtered Set`) against the domain object's DACL before attempting DCSync. Missing permissions now produce a clear, fail-fast error instead of an opaque `Access is denied` later in the workflow.
---
## Historical Releases (per-script versioning)
### 2026-03-16
#### Test-WeakADPasswords.ps1 v1.4.5
Fixed:
- Normalizes legacy `HASH:count` KHDB files into a temporary hash-only list before calling `DSInternals`, so dictionary matches no longer fail silently when clients have older database content.
- Warns when KHDB normalization is required instead of leaving the weak-password match section empty without explanation.
#### Update-KHDB.ps1 v2.1.1
Fixed:
- Rebuilds the merged local `khdb.txt` as a DSInternals-compatible hash-only file even when upstream shards still contain legacy `HASH:count` lines.
- Tightened KHDB merge validation so malformed shard content is surfaced during update rather than producing a silently unusable weak-password database.
#### Prepare-KHDBStorage.ps1 v1.1.1
Fixed:
- Accepts legacy `HASH:count` source input but writes deduplicated hash-only shards for downstream DSInternals consumers.
#### README.md
Changed:
- Corrected the KHDB format documentation to require one NT hash per line and documented the automatic legacy-format normalization.
### 2026-02-17
#### Test-WeakADPasswords.ps1 v1.4.4
Changed:
- Added startup FIPS policy detection (`HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy\Enabled`) with fail-fast behavior and explicit remediation steps to avoid opaque DSInternals runtime failures.
#### Test-WeakADPasswords.ps1 v1.4.3
Fixed:
- Added explicit handling for `Microsoft.PowerShell.Commands.WriteErrorException,DSInternals.Bootstrap.psm1` so known FIPS bootstrap errors are downgraded to a controlled warning when possible, with a clear fail message if DSInternals cannot load under policy.
#### Test-WeakADPasswords.ps1 v1.4.2
Fixed:
- DSInternals module import now handles the known FIPS bootstrap warning as non-fatal when the module successfully loads, preventing repeated `SecurityError` noise during startup.
#### Test-WeakADPasswords.ps1 v1.4.1
Changed:
- Added credential pre-validation against the selected domain controller before running `Get-ADReplAccount`, including retry prompts for rejected credentials.
- Improved error diagnostics to distinguish invalid credentials from missing replication permissions (`Access is denied`).
- Added optional `-Credential` parameter to `Test-WeakADPasswords` for callers that need to provide credentials non-interactively.
#### README.md
Changed:
- Updated weak-password testing documentation to reflect credential pre-check behavior and added a short troubleshooting section for common authentication/permissions failures.
### 2025-10-30
#### Update-KHDB.ps1 v2.0.0
Changed:
- Replaced single-archive workflow with manifest-driven, two-hex shard downloads that verify SHA256/size before in-place updates.
- Added incremental refresh logic, stale shard cleanup, and automatic rebuild of the merged `khdb.txt` for downstream scripts.
- Hardened validation to stream-check merged output while preserving strict TLS, retry, and transcript behaviour.
#### ElysiumSettings.txt.sample v1.3.0
Added:
- Documented `KhdbManifestPath`, `KhdbShardPrefix`, and `KhdbLocalShardDir` defaults for the shard-aware updater.
#### README.md
Changed:
- Described the manifest/shard update flow so operators understand the incremental download model and automatic cleanup.
#### Prepare-KHDBStorage.ps1 v1.0.0
Added:
- Helper script to split `khdb.txt` (or a directory/list of `.gz` HIBP slices) into two-hex shards, build the JSON manifest, and push the package to Azure Blob Storage or S3-compatible endpoints.
- Validation step that tallies and quarantines malformed hashes before sharding, writing `invalid-hashes.txt` plus a console summary so bad data never reaches storage.
- Optional `-ShowProgress` mode emitting periodic `Write-Progress` updates (interval configurable) so large ingests visibly tick forward.
- Automatic reconstruction of HIBP NTLM hashes (file-prefix + suffix) so partially stored hashes still produce full 32-hex values in the shards, plus per-prefix deduplication that keeps the highest observed count.
- `-ForcePlainText` switch to skip `.gz` expansions entirely and treat the source as pre-built hash lines (skipped entries are reported separately).
- Emits a merged `khdb-clean.txt` alongside the shards for DSInternals or offline review, including SHA256 fingerprints for both manifest and clean output.
- Automatic checkpoint/resume when `-ForcePlainText` is used (configurable via `-CheckpointPath`, disable with `-NoCheckpoint`) so large ingests can be paused and resumed without reprocessing prior shards.
### 2025-10-26
#### Test-WeakADPasswords.ps1 v1.3.3
Added:
- Opt-in usage beacon that fires a single HTTP request (GET/POST/PUT) after settings load, suitable for pre-signed S3 URLs, and only includes script name, version, and a UTC timestamp (plus optional instance ID).
- Instance identifier header/body support and configurable timeout so adopters can differentiate deployments without collecting user data.
#### ElysiumSettings.txt.sample v1.2.0
Added:
- Documented `UsageBeacon*` keys (URL, method, instance ID, timeout) so telemetry stays disabled by default but easy to enable.
#### README.md
Added:
- Usage beacon section explaining how to configure the lightweight tracking call and what metadata is transmitted.
### 2025-10-21
#### Extract-NTHashes.ps1 v1.2.1
Fixed:
- Corrected SigV4 host header formatting so non-default ports serialize without parser errors.
- Hardened hashing helpers to avoid `ComputeHash` overload ambiguity under Windows PowerShell.
- Domain selection menu now respects the configured numeric order.
#### Test-WeakADPasswords.ps1 v1.3.2
Changed:
- Switched to the sorted KHDB path when driving `Test-PasswordQuality`, eliminating full linear scans and avoiding malformed-line crashes on massive datasets.
#### Test-WeakADPasswords.ps1 v1.3.1
Fixed:
- Domain picker now renders in numeric order from settings for predictable operator workflows.
- UPN export now relies on structured weak-password results, so dictionary hit UPN lists are populated reliably.
### 2025-10-10
#### Test-WeakADPasswords.ps1 v1.3.0
Added:
- `CheckOnlyEnabledUsers` flag wired from settings to filter accounts prior to `Test-PasswordQuality`.
- Transcript logging to `Reports/logs/test-weakad-<timestamp>.log`.
### Extract-NTHashes.ps1 v1.2.0
#### Extract-NTHashes.ps1 v1.2.0
Added:
- Transcript logging to `Reports/logs/extract-hashes-<timestamp>.log`.
### Elysium.ps1 v1.1.0
#### Elysium.ps1 v1.1.0
Updated:
- Added strict error handling (`$ErrorActionPreference='Stop'`) and `Set-StrictMode`.
- Resolved script invocations via `$PSScriptRoot` to avoid CWD issues.
### Elysium.ps1 v1.2.0
#### Elysium.ps1 v1.2.0
Added:
- Transcript logging to `Reports/logs/orchestrator-<timestamp>.log` and graceful shutdown without `exit`.
### Uninstall.ps1 v1.1.0
#### Uninstall.ps1 v1.1.0
Added:
- Transcript logging to `%TEMP%/Elysium/logs/uninstall-<timestamp>.log` so logs persist after directory removal.
### Update-KHDB.ps1 v1.1.0
#### Update-KHDB.ps1 v1.1.0
Added/Updated:
- Robust settings validation and SAS token normalization.
- Safe URL construction with `UriBuilder` and custom User-Agent.
@@ -34,7 +177,7 @@ Added/Updated:
- KHDB validation: format check (32-hex), deduplication and normalization.
- Transcript logging to `Reports/logs/update-khdb-<timestamp>.log`.
### Test-WeakADPasswords.ps1 v1.2.0
#### Test-WeakADPasswords.ps1 v1.2.0
Updated:
- Enforced modules via `#Requires`; removed runtime installs.
- Added strict mode and error preference.
@@ -42,7 +185,7 @@ Updated:
- Ensured report directory creation and sane defaults (`Reports`).
- Removed stray top-level loop; UPN enrichment occurs during report generation only.
### Extract-NTHashes.ps1 v1.1.0
#### Extract-NTHashes.ps1 v1.1.0
Updated:
- Enforced modules via `#Requires`; added strict mode.
- Fixed variable ordering bug and unified filename scheme with domain prefix.
@@ -50,18 +193,18 @@ Updated:
- Normalized SAS token and verified container existence; checksum verified before cleanup; artifacts retained on failure.
- Paths resolved relative to `$PSScriptRoot`; ensured report base directory exists.
### ElysiumSettings.txt.sample v1.1.0
#### ElysiumSettings.txt.sample v1.1.0
Updated:
- `ReportPathBase` default changed to `Reports` (relative) and added guidance on required modules and replication rights.
- Added optional `CheckOnlyEnabledUsers=true` example flag.
## Extract-NTHashes.ps1
### Extract-NTHashes.ps1
### version 1.1.1
#### version 1.1.1
**Updated:**
- UPNs of the accounts with passwords found in dictionary were moved into separate report (one UPN at a line) to enable further automation.
### version 1.1.0
#### version 1.1.0
**Added:**
- UPN retrieval (this will prolong the time needed to run the script significantly)
- Better error handling
+394
View File
@@ -0,0 +1,394 @@
$script:ElysiumVersion = '2.2.2'
function Invoke-RestartWithExecutable {
param(
[string]$ExecutablePath,
[hashtable]$BoundParameters,
[object[]]$UnboundArguments
)
if (-not $ExecutablePath) { return }
if (-not $PSCommandPath) { return }
$argList = @('-NoLogo', '-NoProfile', '-File', $PSCommandPath)
if ($BoundParameters) {
foreach ($entry in $BoundParameters.GetEnumerator()) {
$key = "-$($entry.Key)"
$value = $entry.Value
if ($value -is [System.Management.Automation.SwitchParameter]) {
if ($value.IsPresent) { $argList += $key }
} else {
$argList += $key
$argList += $value
}
}
}
if ($UnboundArguments) {
$argList += $UnboundArguments
}
& $ExecutablePath @argList
exit $LASTEXITCODE
}
function Restart-WithPwshIfAvailable {
param(
[hashtable]$BoundParameters,
[object[]]$UnboundArguments
)
if ($PSVersionTable.PSVersion.Major -ge 7 -or $PSVersionTable.PSEdition -eq 'Core') { return }
$pwsh = Get-Command -Name 'pwsh' -ErrorAction SilentlyContinue
if (-not $pwsh) { return }
Write-Host ("PowerShell 7 detected at '{0}'; relaunching script under pwsh..." -f $pwsh.Path)
Invoke-RestartWithExecutable -ExecutablePath $pwsh.Path -BoundParameters $BoundParameters -UnboundArguments $UnboundArguments
}
function Restart-WithWindowsPowerShellIfAvailable {
param(
[hashtable]$BoundParameters,
[object[]]$UnboundArguments
)
if ($PSVersionTable.PSEdition -eq 'Desktop') { return }
$powershellCmd = Get-Command -Name 'powershell.exe' -ErrorAction SilentlyContinue
$powershellPath = $null
if ($powershellCmd) {
$powershellPath = $powershellCmd.Path
} else {
$defaultPath = Join-Path -Path $env:SystemRoot -ChildPath 'System32\WindowsPowerShell\v1.0\powershell.exe'
if (Test-Path -LiteralPath $defaultPath) {
$powershellPath = $defaultPath
}
}
if (-not $powershellPath) {
Write-Warning 'Windows PowerShell (powershell.exe) was not found; continuing under current host.'
return
}
Write-Host ("Windows PowerShell detected at '{0}'; relaunching script under powershell.exe..." -f $powershellPath)
Invoke-RestartWithExecutable -ExecutablePath $powershellPath -BoundParameters $BoundParameters -UnboundArguments $UnboundArguments
}
# ---------------------------------------------------------------------------
# Settings loading
# ---------------------------------------------------------------------------
function Read-KeyValueSettingsFile {
param([Parameter(Mandatory)][string]$Path)
$result = @{}
if (-not (Test-Path -LiteralPath $Path)) { return $result }
foreach ($line in (Get-Content -LiteralPath $Path)) {
if ($null -eq $line) { continue }
$trimmed = $line.Trim()
if (-not $trimmed) { continue }
if ($trimmed.StartsWith('#')) { continue }
$kv = $line -split '=', 2
if ($kv.Count -ne 2) { continue }
$key = $kv[0].Trim()
$value = $kv[1].Trim()
if (-not $key) { continue }
if ($value.StartsWith("'") -and $value.EndsWith("'") -and $value.Length -ge 2) {
$value = $value.Substring(1, $value.Length - 2)
}
$result[$key] = $value
}
return $result
}
function Read-ElysiumSettings {
param([Parameter(Mandatory)][string]$ScriptRoot)
$settingsPath = Join-Path -Path $ScriptRoot -ChildPath 'ElysiumSettings.txt'
if (-not (Test-Path -LiteralPath $settingsPath)) {
throw "Settings file not found at $settingsPath"
}
return Read-KeyValueSettingsFile -Path $settingsPath
}
function Get-SettingsValue {
param(
[hashtable]$Settings,
[string]$Key
)
if (-not $Settings) { return $null }
if ($Settings.ContainsKey($Key)) { return $Settings[$Key] }
return $null
}
# ---------------------------------------------------------------------------
# Parallel execution helpers
# ---------------------------------------------------------------------------
function Get-FunctionDefinitionText {
param([Parameter(Mandatory)][string]$Name)
$cmd = Get-Command -Name $Name -CommandType Function -ErrorAction Stop
return $cmd.ScriptBlock.Ast.Extent.Text
}
# ---------------------------------------------------------------------------
# Azure Blob Storage helpers
# ---------------------------------------------------------------------------
function Build-BlobUri {
param(
[string]$Account,
[string]$Container,
[string]$Sas,
[string]$BlobName
)
if ([string]::IsNullOrWhiteSpace($Account)) { throw 'storageAccountName is missing or empty.' }
if ([string]::IsNullOrWhiteSpace($Container)) { throw 'containerName is missing or empty.' }
if ([string]::IsNullOrWhiteSpace($Sas)) { throw 'sasToken is missing or empty.' }
if ([string]::IsNullOrWhiteSpace($BlobName)) { throw 'BlobName cannot be empty.' }
$sas = $Sas.Trim()
if (-not $sas.StartsWith('?')) { $sas = '?' + $sas }
$normalizedBlob = $BlobName.Replace('\', '/').TrimStart('/')
$builder = [System.UriBuilder]::new("https://$Account.blob.core.windows.net/$Container/$normalizedBlob")
$builder.Query = $sas.TrimStart('?')
return $builder.Uri.AbsoluteUri
}
# ---------------------------------------------------------------------------
# Storage path utilities
# ---------------------------------------------------------------------------
function Combine-StoragePath {
param(
[string]$Prefix,
[string]$Name
)
$cleanName = $Name.Replace('\', '/').TrimStart('/')
if ([string]::IsNullOrWhiteSpace($Prefix)) { return $cleanName }
$normalizedPrefix = $Prefix.Replace('\', '/').Trim('/')
if ([string]::IsNullOrEmpty($normalizedPrefix)) { return $cleanName }
return "$normalizedPrefix/$cleanName"
}
# ---------------------------------------------------------------------------
# AWS SigV4 / S3 helpers
# ---------------------------------------------------------------------------
function Get-Bytes([string]$s) { return [System.Text.Encoding]::UTF8.GetBytes($s) }
function Get-HashHex([byte[]]$bytes) {
if ($null -eq $bytes) { $bytes = [byte[]]@() }
$sha = [System.Security.Cryptography.SHA256]::Create()
try {
$ms = New-Object System.IO.MemoryStream -ArgumentList (,$bytes)
try {
$hash = $sha.ComputeHash([System.IO.Stream]$ms)
} finally { $ms.Dispose() }
return ([BitConverter]::ToString($hash)).Replace('-', '').ToLowerInvariant()
} finally { $sha.Dispose() }
}
function HmacSha256([byte[]]$key, [string]$data) {
$h = [System.Security.Cryptography.HMACSHA256]::new($key)
try {
$b = [System.Text.Encoding]::UTF8.GetBytes($data)
$ms = New-Object System.IO.MemoryStream -ArgumentList (,$b)
try {
return $h.ComputeHash([System.IO.Stream]$ms)
} finally { $ms.Dispose() }
} finally { $h.Dispose() }
}
function GetSignatureKey([string]$secret, [string]$dateStamp, [string]$regionName, [string]$serviceName) {
$kDate = HmacSha256 (Get-Bytes ('AWS4' + $secret)) $dateStamp
$kRegion = HmacSha256 $kDate $regionName
$kService = HmacSha256 $kRegion $serviceName
HmacSha256 $kService 'aws4_request'
}
function UriEncode([string]$data, [bool]$encodeSlash) {
if ($null -eq $data) { return '' }
$enc = [System.Uri]::EscapeDataString($data)
if (-not $encodeSlash) { $enc = $enc -replace '%2F', '/' }
return $enc
}
function BuildCanonicalPath([System.Uri]$uri) {
$segments = $uri.AbsolutePath.Split('/')
$encoded = @()
foreach ($s in $segments) { $encoded += (UriEncode $s $false) }
$path = ($encoded -join '/')
if (-not $path.StartsWith('/')) { $path = '/' + $path }
return $path
}
function ToHex([byte[]]$b) { ([BitConverter]::ToString($b)).Replace('-', '').ToLowerInvariant() }
function BuildAuthHeaders($method, [System.Uri]$uri, [string]$region, [string]$accessKey, [string]$secretKey, [string]$payloadHash) {
$algorithm = 'AWS4-HMAC-SHA256'
$timestamp = (Get-Date).ToUniversalTime()
$amzDate = $timestamp.ToString('yyyyMMddTHHmmssZ')
$dateStamp = $timestamp.ToString('yyyyMMdd')
$hostHeader = $uri.Host
if (-not $uri.IsDefaultPort) { $hostHeader = "${hostHeader}:$($uri.Port)" }
$canonicalUri = BuildCanonicalPath $uri
$canonicalQueryString = ''
$canonicalHeaders = "host:$hostHeader`n" + "x-amz-content-sha256:$payloadHash`n" + "x-amz-date:$amzDate`n"
$signedHeaders = 'host;x-amz-content-sha256;x-amz-date'
$canonicalRequest = "$method`n$canonicalUri`n$canonicalQueryString`n$canonicalHeaders`n$signedHeaders`n$payloadHash"
$credentialScope = "$dateStamp/$region/s3/aws4_request"
$stringToSign = "$algorithm`n$amzDate`n$credentialScope`n$((Get-HashHex (Get-Bytes $canonicalRequest)))"
$signingKey = GetSignatureKey $secretKey $dateStamp $region 's3'
$signature = ToHex (HmacSha256 $signingKey $stringToSign)
$authHeader = "$algorithm Credential=$accessKey/$credentialScope, SignedHeaders=$signedHeaders, Signature=$signature"
@{
'x-amz-date' = $amzDate
'x-amz-content-sha256' = $payloadHash
'Authorization' = $authHeader
}
}
function BuildS3Uri([string]$endpointUrl, [string]$bucket, [string]$key, [bool]$forcePathStyle) {
$base = [System.Uri]$endpointUrl
$builder = [System.UriBuilder]::new($base)
$normalizedKey = $key.Replace('\', '/').TrimStart('/')
if ($forcePathStyle) {
$path = $builder.Path.TrimEnd('/')
if ([string]::IsNullOrEmpty($path)) { $path = '/' }
$builder.Path = ($path.TrimEnd('/') + '/' + $bucket + '/' + $normalizedKey)
} else {
$builder.Host = "$bucket." + $builder.Host
$path = $builder.Path.TrimEnd('/')
if ([string]::IsNullOrEmpty($path)) { $path = '/' }
$builder.Path = ($path.TrimEnd('/') + '/' + $normalizedKey)
}
return $builder.Uri
}
function Ensure-AWSS3Module {
try { $null = [Amazon.S3.AmazonS3Client]; return } catch {}
try { Import-Module -Name AWS.Tools.S3 -ErrorAction Stop; return } catch {}
try { Import-Module -Name AWSPowerShell.NetCore -ErrorAction Stop; return } catch {}
throw "AWS Tools for PowerShell not found. Install with: Install-Module AWS.Tools.S3 -Scope CurrentUser"
}
function New-S3Client {
param(
[string]$EndpointUrl,
[string]$Region,
[string]$AccessKeyId,
[string]$SecretAccessKey,
[bool]$ForcePathStyle = $true
)
Ensure-AWSS3Module
$creds = New-Object Amazon.Runtime.BasicAWSCredentials($AccessKeyId, $SecretAccessKey)
$cfg = New-Object Amazon.S3.AmazonS3Config
if ($EndpointUrl) { $cfg.ServiceURL = $EndpointUrl }
if ($Region) { try { $cfg.RegionEndpoint = [Amazon.RegionEndpoint]::GetBySystemName($Region) } catch {} }
$cfg.ForcePathStyle = [bool]$ForcePathStyle
return (New-Object Amazon.S3.AmazonS3Client($creds, $cfg))
}
# ---------------------------------------------------------------------------
# Active Directory credential and permission helpers
# (requires the ActiveDirectory module to be loaded before calling)
# ---------------------------------------------------------------------------
function Get-ValidatedADCredential {
param (
[Parameter(Mandatory)][string]$DomainName,
[Parameter(Mandatory)][string]$Server,
[int]$MaxAttempts = 3
)
for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++) {
$credential = Get-Credential -Message "Enter AD credentials with replication rights for $DomainName (attempt $attempt/$MaxAttempts)"
if ($null -eq $credential) {
throw "Credential prompt was cancelled."
}
try {
Get-ADDomain -Server $Server -Credential $credential -ErrorAction Stop | Out-Null
Write-Verbose ("Credential pre-check succeeded for '{0}' against '{1}'." -f $credential.UserName, $Server)
return $credential
} catch {
$message = $_.Exception.Message
if ($message -match 'rejected the client credentials|unknown user name|bad password|logon failure') {
Write-Warning ("Credentials were rejected for '{0}' (attempt {1}/{2})." -f $credential.UserName, $attempt, $MaxAttempts)
if ($attempt -lt $MaxAttempts) { continue }
throw "Credentials were rejected by domain controller '$Server' after $MaxAttempts attempts."
}
throw "Credential pre-check failed against '$Server': $message"
}
}
}
function Test-ReplicationPermissions {
param(
[Parameter(Mandatory)][string]$DomainDN,
[Parameter(Mandatory)][string]$Server,
[Parameter(Mandatory)][System.Management.Automation.PSCredential]$Credential
)
$requiredRights = [ordered]@{
'Replicating Directory Changes' = [guid]'1131f6aa-9c07-11d1-f79f-00c04fc2dcd2'
'Replicating Directory Changes All' = [guid]'1131f6ab-9c07-11d1-f79f-00c04fc2dcd2'
'Replicating Directory Changes In Filtered Set' = [guid]'89e95b76-444d-4c62-991a-0facbeda640c'
}
$callerSids = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase)
try {
$samName = $Credential.UserName -replace '^.*\\', ''
$adUser = Get-ADUser -Identity $samName -Server $Server -Credential $Credential `
-Properties SID, DistinguishedName -ErrorAction Stop
[void]$callerSids.Add($adUser.SID.Value)
# tokenGroups is a constructed attribute containing all SIDs in the user's token,
# including nested group memberships — more reliable than walking MemberOf recursively
$userDe = New-Object System.DirectoryServices.DirectoryEntry(
"LDAP://$Server/$($adUser.DistinguishedName)",
$Credential.UserName,
$Credential.GetNetworkCredential().Password
)
$userDe.RefreshCache(@('tokenGroups'))
foreach ($sidBytes in $userDe.Properties['tokenGroups']) {
$sid = New-Object System.Security.Principal.SecurityIdentifier($sidBytes, 0)
[void]$callerSids.Add($sid.Value)
}
} catch {
Write-Warning ("Could not resolve account SIDs for replication permission pre-check: {0}. Skipping." -f $_.Exception.Message)
return
}
$acl = $null
try {
$de = New-Object System.DirectoryServices.DirectoryEntry(
"LDAP://$Server/$DomainDN",
$Credential.UserName,
$Credential.GetNetworkCredential().Password
)
$acl = $de.ObjectSecurity.GetAccessRules(
$true, $true, [System.Security.Principal.SecurityIdentifier])
} catch {
Write-Warning ("Could not read domain object ACL for replication permission pre-check: {0}. Skipping." -f $_.Exception.Message)
return
}
$missing = @()
foreach ($rightName in $requiredRights.Keys) {
$guid = $requiredRights[$rightName]
$granted = $false
foreach ($ace in $acl) {
if ($ace.AccessControlType -ne [System.Security.AccessControl.AccessControlType]::Allow) { continue }
if (-not ($ace.ActiveDirectoryRights -band [System.DirectoryServices.ActiveDirectoryRights]::ExtendedRight)) { continue }
if ($ace.ObjectType -ne $guid) { continue }
if ($callerSids.Contains($ace.IdentityReference.Value)) { $granted = $true; break }
}
if (-not $granted) { $missing += $rightName }
}
if ($missing.Count -gt 0) {
throw ("Account '{0}' is missing the following replication permissions on '{1}':`n - {2}`n`nGrant these extended rights on the domain object to allow DCSync-based hash retrieval." -f `
$Credential.UserName, $DomainDN, ($missing -join "`n - "))
}
Write-Host ("[+] Replication permissions verified for '{0}'." -f $Credential.UserName)
}
+39 -12
View File
@@ -7,7 +7,7 @@
##################################################
## Project: Elysium ##
## File: Elysium.ps1 ##
## Version: 1.2.0 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
@@ -21,13 +21,13 @@ Elysium.ps1 offers a menu to perform various actions:
2. Test Weak AD Passwords
3. Extract and Send Current Hashes for KHDB Update
4. Uninstall the tool
5. Exit
5. Update Lithnet Password Protection store
6. Exit
#>
# Safer defaults
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
# Define the path to the settings file
$settingsFilePath = Join-Path -Path $PSScriptRoot -ChildPath "ElysiumSettings.txt"
@@ -52,8 +52,6 @@ if ([string]::IsNullOrEmpty($passphrase)) {
Write-Host "Passphrase found in environment variables."
}
# Continue with the rest of your script...
function Start-OrchestratorTranscript {
param([string]$BasePath)
try {
@@ -69,6 +67,20 @@ function Start-OrchestratorTranscript {
function Stop-OrchestratorTranscript { try { Stop-Transcript | Out-Null } catch {} }
function Invoke-WindowsPowerShellScript {
param([string]$ScriptPath)
$powershellCmd = Get-Command -Name 'powershell.exe' -ErrorAction SilentlyContinue
if (-not $powershellCmd) {
throw "Windows PowerShell (powershell.exe) was not found. Install it or run the script from a Desktop edition session."
}
$args = @('-NoLogo', '-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', $ScriptPath)
& $powershellCmd.Path @args
$exitCode = $LASTEXITCODE
if ($exitCode -ne 0) {
throw ("Windows PowerShell script '{0}' exited with code {1}." -f $ScriptPath, $exitCode)
}
}
function Show-Menu {
param (
[string]$Title = 'Elysium Tool Main Menu'
@@ -78,8 +90,9 @@ function Show-Menu {
Write-Host "1: Download/Update Known-Hashes Database (KHDB)"
Write-Host "2: Test Weak AD Passwords"
Write-Host "3: Extract and Send Current Hashes for KHDB Update"
Write-Host "4: Uninstall"
Write-Host "5: Exit"
Write-Host "4: Update Lithnet Password Protection Store"
Write-Host "5: Uninstall"
Write-Host "6: Exit"
}
Start-OrchestratorTranscript -BasePath $PSScriptRoot
@@ -94,27 +107,41 @@ do {
}
'2' {
Write-Host "Testing Weak AD Passwords..."
& (Join-Path -Path $PSScriptRoot -ChildPath 'Test-WeakADPasswords.ps1')
$testScript = Join-Path -Path $PSScriptRoot -ChildPath 'Test-WeakADPasswords.ps1'
if ($PSVersionTable.PSEdition -eq 'Desktop') {
& $testScript
} else {
Invoke-WindowsPowerShellScript -ScriptPath $testScript
}
}
'3' {
Write-Host "Extracting and Sending Current Hashes..."
& (Join-Path -Path $PSScriptRoot -ChildPath 'Extract-NTHashes.ps1')
$extractScript = Join-Path -Path $PSScriptRoot -ChildPath 'Extract-NTHashes.ps1'
if ($PSVersionTable.PSEdition -eq 'Desktop') {
& $extractScript
} else {
Invoke-WindowsPowerShellScript -ScriptPath $extractScript
}
}
'4' {
Write-Host "Updating Lithnet Password Protection store..."
& (Join-Path -Path $PSScriptRoot -ChildPath 'Update-LithnetStore.ps1')
}
'5' {
Write-Host "Uninstalling..."
& (Join-Path -Path $PSScriptRoot -ChildPath 'Uninstall.ps1')
}
'5' {
'6' {
Write-Host "Exiting..."
# end loop; transcript will be stopped after the loop
$userSelection = '5'
$userSelection = '6'
}
default {
Write-Host "Invalid selection, please try again."
}
}
pause
} while ($userSelection -ne '5')
} while ($userSelection -ne '6')
} finally {
Stop-OrchestratorTranscript
}
+28 -1
View File
@@ -8,7 +8,7 @@
##################################################
## Project: Elysium ##
## File: ElysiumSettings.txt ##
## Version: 1.1.0 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
@@ -36,6 +36,14 @@ s3SecretAccessKey =
s3ForcePathStyle = true
s3UseAwsTools = false
# KHDB Shard Settings
#####################
# The KHDB update script downloads a manifest plus per-prefix shards (default shard size 2).
# These values control the remote object names and local storage directory.
KhdbManifestPath=khdb/manifest.json
KhdbShardPrefix=khdb/shards
KhdbLocalShardDir=khdb-shards
# Application Settings
######################
InstallationPath=
@@ -43,6 +51,25 @@ ReportPathBase=Reports
WeakPasswordsDatabase=khdb.txt
# CheckOnlyEnabledUsers=true
# Lithnet Password Protection Settings
######################################
LithnetStorePath=
LithnetSyncHibp=false
LithnetHashSources=khdb.txt
LithnetPlaintextSources=
LithnetBannedWordSources=
# Telemetry (optional)
######################
# These values are empty by default so no telemetry is sent.
# Provide a pre-signed URL (for example, an S3 PUT) to receive a single beacon
# when the weak-password test starts. Only script name, version, and timestamp
# are transmitted; you can set UsageBeaconInstanceId to differentiate deployments.
UsageBeaconUrl=
UsageBeaconMethod=GET # GET, POST, or PUT
UsageBeaconInstanceId=
UsageBeaconTimeoutSeconds=5
# Notes:
# - Required PowerShell modules: DSInternals, ActiveDirectory
# For Azure uploads: Az.Storage
+135 -225
View File
@@ -6,8 +6,8 @@
## \____\__\_\_| \_\_____(_)_| \_|_____| |_| ##
##################################################
## Project: Elysium ##
## File: Extract-NTLMHashes.ps1 ##
## Version: 1.2.0 ##
## File: Extract-NTHashes.ps1 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
@@ -22,8 +22,14 @@ This script will connect to selected domain (defined in ElysiumSettings.txt) usi
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
$scriptRoot = $PSScriptRoot
[string]$commonHelper = Join-Path -Path $PSScriptRoot -ChildPath 'Elysium.Common.ps1'
if (-not (Test-Path -LiteralPath $commonHelper)) { throw "Common helper not found at $commonHelper" }
. $commonHelper
Restart-WithWindowsPowerShellIfAvailable -BoundParameters $PSBoundParameters -UnboundArguments $MyInvocation.UnboundArguments
function Start-ExtractTranscript {
param([string]$BasePath)
try {
@@ -39,156 +45,18 @@ function Start-ExtractTranscript {
function Stop-ExtractTranscript { try { Stop-Transcript | Out-Null } catch {} }
Start-ExtractTranscript -BasePath $scriptRoot
try {
# Import settings
Write-Host "Loading settings..."
$ElysiumSettings = @{}
$settingsPath = Join-Path -Path $scriptRoot -ChildPath "ElysiumSettings.txt"
if (-not (Test-Path $settingsPath)) {
Write-Error "Settings file not found at $settingsPath"
exit
}
Get-Content $settingsPath | ForEach-Object {
if (-not [string]::IsNullOrWhiteSpace($_) -and -not $_.StartsWith("#")) {
$keyValue = $_ -split '=', 2
if ($keyValue.Count -eq 2) {
$ElysiumSettings[$keyValue[0].Trim()] = $keyValue[1].Trim()
}
}
}
function Normalize-ReportPath([string]$p) {
if ([string]::IsNullOrWhiteSpace($p)) { return (Join-Path -Path $scriptRoot -ChildPath 'Reports') }
if ([System.IO.Path]::IsPathRooted($p)) { return $p }
return (Join-Path -Path $scriptRoot -ChildPath $p)
}
# Storage provider selection (Azure by default)
$storageProvider = $ElysiumSettings['StorageProvider']
if ([string]::IsNullOrWhiteSpace($storageProvider)) { $storageProvider = 'Azure' }
# Azure settings
$storageAccountName = $ElysiumSettings['storageAccountName']
$containerName = $ElysiumSettings['containerName']
$sasToken = $ElysiumSettings['sasToken']
# S3-compatible settings
$s3EndpointUrl = $ElysiumSettings['s3EndpointUrl']
$s3Region = $ElysiumSettings['s3Region']
$s3BucketName = $ElysiumSettings['s3BucketName']
$s3AccessKeyId = $ElysiumSettings['s3AccessKeyId']
$s3SecretAccessKey = $ElysiumSettings['s3SecretAccessKey']
$s3ForcePathStyle = $ElysiumSettings['s3ForcePathStyle']
$s3UseAwsTools = $ElysiumSettings['s3UseAwsTools']
if ([string]::IsNullOrWhiteSpace($s3Region)) { $s3Region = 'us-east-1' }
try { $s3ForcePathStyle = [System.Convert]::ToBoolean($s3ForcePathStyle) } catch { $s3ForcePathStyle = $true }
try { $s3UseAwsTools = [System.Convert]::ToBoolean($s3UseAwsTools) } catch { $s3UseAwsTools = $false }
function Ensure-AWSS3Module {
# Ensure AWS SDK types are available via AWS Tools for PowerShell
try {
$null = [Amazon.S3.AmazonS3Client]
return
} catch {
try { Import-Module -Name AWS.Tools.S3 -ErrorAction Stop; return } catch {}
try { Import-Module -Name AWSPowerShell.NetCore -ErrorAction Stop; return } catch {}
throw "AWS Tools for PowerShell not found. Install with: Install-Module AWS.Tools.S3 -Scope CurrentUser"
}
}
function New-S3Client {
param(
[string]$EndpointUrl,
[string]$Region,
[string]$AccessKeyId,
[string]$SecretAccessKey,
[bool]$ForcePathStyle = $true
)
Ensure-AWSS3Module
$creds = New-Object Amazon.Runtime.BasicAWSCredentials($AccessKeyId, $SecretAccessKey)
$cfg = New-Object Amazon.S3.AmazonS3Config
if ($EndpointUrl) { $cfg.ServiceURL = $EndpointUrl }
if ($Region) {
try { $cfg.RegionEndpoint = [Amazon.RegionEndpoint]::GetBySystemName($Region) } catch {}
}
$cfg.ForcePathStyle = [bool]$ForcePathStyle
return (New-Object Amazon.S3.AmazonS3Client($creds, $cfg))
}
# Native S3 SigV4 (no AWS Tools) helpers
function Get-Bytes([string]$s) { return [System.Text.Encoding]::UTF8.GetBytes($s) }
function Get-HashHex([byte[]]$bytes) {
$sha = [System.Security.Cryptography.SHA256]::Create()
try { return ([BitConverter]::ToString($sha.ComputeHash($bytes))).Replace('-', '').ToLowerInvariant() } finally { $sha.Dispose() }
}
function Get-FileSha256Hex([string]$path) {
$sha = [System.Security.Cryptography.SHA256]::Create()
$fs = [System.IO.File]::OpenRead($path)
try { return ([BitConverter]::ToString($sha.ComputeHash($fs))).Replace('-', '').ToLowerInvariant() } finally { $fs.Close(); $sha.Dispose() }
}
function HmacSha256([byte[]]$key, [string]$data) {
$h = [System.Security.Cryptography.HMACSHA256]::new($key)
try { return $h.ComputeHash((Get-Bytes $data)) } finally { $h.Dispose() }
}
function GetSignatureKey([string]$secret, [string]$dateStamp, [string]$regionName, [string]$serviceName) {
$kDate = HmacSha256 (Get-Bytes ('AWS4' + $secret)) $dateStamp
$kRegion = HmacSha256 $kDate $regionName
$kService = HmacSha256 $kRegion $serviceName
return (HmacSha256 $kService 'aws4_request')
}
function UriEncode([string]$data, [bool]$encodeSlash) {
if ($null -eq $data) { return '' }
$enc = [System.Uri]::EscapeDataString($data)
if (-not $encodeSlash) { $enc = $enc -replace '%2F','/' }
return $enc
}
function BuildCanonicalPath([System.Uri]$uri) {
$segments = $uri.AbsolutePath.Split('/')
$encoded = @()
foreach ($seg in $segments) { $encoded += (UriEncode $seg $false) }
$path = ($encoded -join '/')
if (-not $path.StartsWith('/')) { $path = '/' + $path }
return $path
}
function ToHex([byte[]]$bytes) { return ([BitConverter]::ToString($bytes)).Replace('-', '').ToLowerInvariant() }
function BuildAuthHeaders($method, [System.Uri]$uri, [string]$region, [string]$accessKey, [string]$secretKey, [string]$payloadHash) {
$algorithm = 'AWS4-HMAC-SHA256'
$amzdate = (Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ')
$datestamp = (Get-Date).ToUniversalTime().ToString('yyyyMMdd')
$hostHeader = $uri.Host
if (-not $uri.IsDefaultPort) { $hostHeader = "$hostHeader:$($uri.Port)" }
$canonicalUri = BuildCanonicalPath $uri
$canonicalQueryString = ''
$canonicalHeaders = "host:$hostHeader`n" + "x-amz-content-sha256:$payloadHash`n" + "x-amz-date:$amzdate`n"
$signedHeaders = 'host;x-amz-content-sha256;x-amz-date'
$canonicalRequest = "$method`n$canonicalUri`n$canonicalQueryString`n$canonicalHeaders`n$signedHeaders`n$payloadHash"
$credentialScope = "$datestamp/$region/s3/aws4_request"
$stringToSign = "$algorithm`n$amzdate`n$credentialScope`n$((Get-HashHex (Get-Bytes $canonicalRequest)))"
$signingKey = GetSignatureKey $secretKey $datestamp $region 's3'
$signature = ToHex (HmacSha256 $signingKey $stringToSign)
$authHeader = "$algorithm Credential=$accessKey/$credentialScope, SignedHeaders=$signedHeaders, Signature=$signature"
return @{ 'x-amz-date' = $amzdate; 'x-amz-content-sha256' = $payloadHash; 'Authorization' = $authHeader }
}
function BuildS3Uri([string]$endpointUrl, [string]$bucket, [string]$key, [bool]$forcePathStyle) {
$base = [System.Uri]$endpointUrl
$ub = [System.UriBuilder]::new($base)
if ($forcePathStyle) {
$p = ($ub.Path.TrimEnd('/'))
if ([string]::IsNullOrEmpty($p)) { $p = '/' }
$ub.Path = ($p.TrimEnd('/') + '/' + $bucket + '/' + $key)
} else {
$ub.Host = "$bucket." + $ub.Host
$p = $ub.Path.TrimEnd('/')
if ([string]::IsNullOrEmpty($p)) { $p = '/' }
$ub.Path = ($p.TrimEnd('/') + '/' + $key)
}
return $ub.Uri
}
function Invoke-S3PutFile([string]$endpointUrl, [string]$bucket, [string]$key, [string]$filePath, [string]$region, [string]$ak, [string]$sk, [bool]$forcePathStyle) {
$uri = BuildS3Uri -endpointUrl $endpointUrl -bucket $bucket -key $key -forcePathStyle $forcePathStyle
$payloadHash = Get-FileSha256Hex -path $filePath
@@ -206,6 +74,7 @@ function Invoke-S3PutFile([string]$endpointUrl, [string]$bucket, [string]$key, [
if (-not $resp.IsSuccessStatusCode) { throw "S3 PUT failed: $([int]$resp.StatusCode) $($resp.ReasonPhrase)" }
} finally { if ($req) { $req.Dispose() }; if ($stream) { $stream.Close(); $stream.Dispose() }; $client.Dispose() }
}
function Invoke-S3GetToFile([string]$endpointUrl, [string]$bucket, [string]$key, [string]$targetPath, [string]$region, [string]$ak, [string]$sk, [bool]$forcePathStyle) {
$uri = BuildS3Uri -endpointUrl $endpointUrl -bucket $bucket -key $key -forcePathStyle $forcePathStyle
$payloadHash = (Get-HashHex (Get-Bytes ''))
@@ -224,13 +93,6 @@ function Invoke-S3GetToFile([string]$endpointUrl, [string]$bucket, [string]$key,
} finally { if ($req) { $req.Dispose() }; $client.Dispose() }
}
# Retrieve the passphrase from a user environment variable
$passphrase = [System.Environment]::GetEnvironmentVariable("ELYSIUM_PASSPHRASE", [System.EnvironmentVariableTarget]::User)
if ([string]::IsNullOrWhiteSpace($passphrase)) { Write-Error 'Passphrase not found in ELYSIUM_PASSPHRASE environment variable.'; exit }
# Timestamp
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
function Protect-FileWithAES {
param (
[Parameter(Mandatory = $true)]
@@ -243,7 +105,6 @@ function Protect-FileWithAES {
[string]$Passphrase
)
# Derive key with PBKDF2 (HMACSHA256) + random salt
$rng = [System.Security.Cryptography.RandomNumberGenerator]::Create()
$salt = New-Object byte[] 16
$rng.GetBytes($salt)
@@ -265,7 +126,6 @@ function Protect-FileWithAES {
$outFileStream = [System.IO.File]::Create($OutputFile)
try {
# File header: magic 'ELY1' (4 bytes), salt (16 bytes), IV (16 bytes)
$magic = [System.Text.Encoding]::ASCII.GetBytes('ELY1')
$outFileStream.Write($magic, 0, $magic.Length)
$outFileStream.Write($salt, 0, $salt.Length)
@@ -287,6 +147,7 @@ function Protect-FileWithAES {
Write-Host "File has been encrypted (PBKDF2+AES-256-CBC): $OutputFile"
}
function Get-FileChecksum {
param (
[string]$Path,
@@ -297,79 +158,128 @@ function Get-FileChecksum {
try {
$hashBytes = $hasher.ComputeHash($stream)
return [BitConverter]::ToString($hashBytes) -replace '-', ''
}
finally {
} finally {
$stream.Close()
$hasher.Dispose()
}
}
# Extract NTLM hashes
$reportBase = Normalize-ReportPath -p $ElysiumSettings['ReportPathBase']
if (-not (Test-Path $reportBase)) { New-Item -Path $reportBase -ItemType Directory -Force | Out-Null }
Start-ExtractTranscript -BasePath $scriptRoot
try {
Write-Host "Loading settings..."
$ElysiumSettings = Read-ElysiumSettings -ScriptRoot $scriptRoot
# Build domain details from settings
$DomainDetails = @{}
for ($i = 1; $ElysiumSettings.ContainsKey("Domain${i}Name"); $i++) {
# Storage provider selection (Azure by default)
$storageProvider = $ElysiumSettings['StorageProvider']
if ([string]::IsNullOrWhiteSpace($storageProvider)) { $storageProvider = 'Azure' }
# Azure settings
$storageAccountName = $ElysiumSettings['storageAccountName']
$containerName = $ElysiumSettings['containerName']
$sasToken = $ElysiumSettings['sasToken']
# S3-compatible settings
$s3EndpointUrl = $ElysiumSettings['s3EndpointUrl']
$s3Region = $ElysiumSettings['s3Region']
$s3BucketName = $ElysiumSettings['s3BucketName']
$s3AccessKeyId = $ElysiumSettings['s3AccessKeyId']
$s3SecretAccessKey = $ElysiumSettings['s3SecretAccessKey']
$s3ForcePathStyle = $ElysiumSettings['s3ForcePathStyle']
$s3UseAwsTools = $ElysiumSettings['s3UseAwsTools']
if ([string]::IsNullOrWhiteSpace($s3Region)) { $s3Region = 'us-east-1' }
try { $s3ForcePathStyle = [System.Convert]::ToBoolean($s3ForcePathStyle) } catch { $s3ForcePathStyle = $true }
try { $s3UseAwsTools = [System.Convert]::ToBoolean($s3UseAwsTools) } catch { $s3UseAwsTools = $false }
# Retrieve the passphrase from a user environment variable
$passphrase = [System.Environment]::GetEnvironmentVariable("ELYSIUM_PASSPHRASE", [System.EnvironmentVariableTarget]::User)
if ([string]::IsNullOrWhiteSpace($passphrase)) { throw 'Passphrase not found in ELYSIUM_PASSPHRASE environment variable.' }
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$reportBase = Normalize-ReportPath -p $ElysiumSettings['ReportPathBase']
if (-not (Test-Path $reportBase)) { New-Item -Path $reportBase -ItemType Directory -Force | Out-Null }
# Build domain details from settings (ordered to keep numeric index order)
$DomainDetails = [ordered]@{}
for ($i = 1; $ElysiumSettings.ContainsKey("Domain${i}Name"); $i++) {
$DomainDetails["$i"] = @{
Name = $ElysiumSettings["Domain${i}Name"]
DC = $ElysiumSettings["Domain${i}DC"]
DA = $ElysiumSettings["Domain${i}DA"]
}
}
}
# User selects a domain
Write-Host "Select a domain to extract NTLM hashes:"
$DomainDetails.GetEnumerator() | ForEach-Object { Write-Host "$($_.Key): $($_.Value.Name)" }
$selection = Read-Host "Enter the number of the domain"
$selectedDomain = $DomainDetails[$selection]
# User selects a domain
Write-Host "Select a domain to extract NTLM hashes:"
$DomainDetails.GetEnumerator() | Sort-Object { [int]$_.Key } | ForEach-Object { Write-Host "$($_.Key): $($_.Value.Name)" }
$selection = Read-Host "Enter the number of the domain"
$selectedDomain = $DomainDetails[$selection]
if (-not $selectedDomain) {
Write-Error "Invalid selection."
exit
}
if (-not $selectedDomain) {
throw "Invalid selection."
}
# Update script variables based on selected domain
$domainController = $selectedDomain.DC
$credential = Get-Credential -Message "Enter AD credentials with replication rights for $($selectedDomain.Name)"
$domainController = $selectedDomain.DC
$domainPrefix = ($selectedDomain.Name -replace "\W", "_")
$baseName = "${domainPrefix}_NTLM_Hashes_$timestamp"
$exportPath = Join-Path -Path $scriptRoot -ChildPath "$baseName.txt"
$compressedFilePath = Join-Path -Path $scriptRoot -ChildPath "$baseName.zip"
$encryptedFilePath = Join-Path -Path $scriptRoot -ChildPath "$baseName.enc"
$blobName = "$baseName.enc"
# Validate credentials and replication permissions before attempting DCSync
$hasADModule = $null -ne (Get-Module -Name ActiveDirectory -ErrorAction SilentlyContinue)
if (-not $hasADModule) {
try { Import-Module ActiveDirectory -ErrorAction Stop; $hasADModule = $true } catch {}
}
$ntlmHashes = Get-ADReplAccount -All -Server $domainController -Credential $credential |
Where-Object { $_.NTHash } |
ForEach-Object { [BitConverter]::ToString($_.NTHash).Replace("-", "") } |
Sort-Object -Unique
if ($hasADModule) {
$credential = Get-ValidatedADCredential -DomainName $selectedDomain.Name -Server $domainController
try {
$domainInfo = Get-ADDomain -Server $domainController -Credential $credential -ErrorAction Stop
Test-ReplicationPermissions -DomainDN $domainInfo.DistinguishedName `
-Server $domainController -Credential $credential
} catch {
throw $_.Exception.Message
}
} else {
Write-Warning "ActiveDirectory module not available; skipping credential pre-check and replication permission verification."
$credential = Get-Credential -Message "Enter AD credentials with replication rights for $($selectedDomain.Name)"
if ($null -eq $credential) { throw "Credential prompt was cancelled." }
}
$ntlmHashes | Out-File -FilePath $exportPath
Write-Host "NTLM hashes have been extracted to: $exportPath"
$domainPrefix = ($selectedDomain.Name -replace "\W", "_")
$baseName = "${domainPrefix}_NTLM_Hashes_$timestamp"
$blobName = "$baseName.enc"
# Compress extracted NTLM hashes
Compress-Archive -Path $exportPath -DestinationPath $compressedFilePath
Write-Host "File has been compressed: $compressedFilePath"
# Use a temp directory for all sensitive intermediate files so they are
# never written to the installation directory and are always cleaned up.
$tmpDir = New-Item -ItemType Directory -Path ([System.IO.Path]::Combine(
[System.IO.Path]::GetTempPath(), "elysium-extract-" + [System.Guid]::NewGuid())) -Force
$exportPath = Join-Path -Path $tmpDir.FullName -ChildPath "$baseName.txt"
$compressedFilePath = Join-Path -Path $tmpDir.FullName -ChildPath "$baseName.zip"
$encryptedFilePath = Join-Path -Path $tmpDir.FullName -ChildPath "$baseName.enc"
$tempDownloadPath = $null
# Encrypt the compressed file
Protect-FileWithAES -InputFile $compressedFilePath -OutputFile $encryptedFilePath -Passphrase $passphrase
Write-Host "File has been encrypted: $encryptedFilePath"
try {
$ntlmHashes = Get-ADReplAccount -All -Server $domainController -Credential $credential |
Where-Object { $_.NTHash } |
ForEach-Object { [BitConverter]::ToString($_.NTHash).Replace("-", "") } |
Sort-Object -Unique
# Calculate the local file checksum
$localFileChecksum = Get-FileChecksum -Path $encryptedFilePath
$ntlmHashes | Out-File -FilePath $exportPath
Write-Host "NTLM hashes have been extracted to temporary file."
if ($storageProvider -ieq 'S3') {
# S3-compatible path (e.g., IDrive e2) without requiring AWS Tools
if ([string]::IsNullOrWhiteSpace($s3BucketName)) { Write-Error 's3BucketName is missing in settings.'; exit }
if ([string]::IsNullOrWhiteSpace($s3AccessKeyId) -or [string]::IsNullOrWhiteSpace($s3SecretAccessKey)) { Write-Error 's3AccessKeyId / s3SecretAccessKey missing in settings.'; exit }
if ([string]::IsNullOrWhiteSpace($s3EndpointUrl)) { Write-Error 's3EndpointUrl is required for S3-compatible storage.'; exit }
Compress-Archive -Path $exportPath -DestinationPath $compressedFilePath
Write-Host "File has been compressed."
Protect-FileWithAES -InputFile $compressedFilePath -OutputFile $encryptedFilePath -Passphrase $passphrase
$localFileChecksum = Get-FileChecksum -Path $encryptedFilePath
if ($storageProvider -ieq 'S3') {
if ([string]::IsNullOrWhiteSpace($s3BucketName)) { throw 's3BucketName is missing in settings.' }
if ([string]::IsNullOrWhiteSpace($s3AccessKeyId) -or [string]::IsNullOrWhiteSpace($s3SecretAccessKey)) { throw 's3AccessKeyId / s3SecretAccessKey missing in settings.' }
if ([string]::IsNullOrWhiteSpace($s3EndpointUrl)) { throw 's3EndpointUrl is required for S3-compatible storage.' }
$usedAwsTools = $false
if ($s3UseAwsTools) {
try {
$s3Client = New-S3Client -EndpointUrl $s3EndpointUrl -Region $s3Region -AccessKeyId $s3AccessKeyId -SecretAccessKey $s3SecretAccessKey -ForcePathStyle:$s3ForcePathStyle
# Upload
$putReq = New-Object Amazon.S3.Model.PutObjectRequest -Property @{ BucketName = $s3BucketName; Key = $blobName; FilePath = $encryptedFilePath }
$null = $s3Client.PutObject($putReq)
Write-Host "Encrypted file uploaded to S3-compatible bucket (AWS Tools): $blobName"
@@ -378,7 +288,6 @@ if ($storageProvider -ieq 'S3') {
$getResp = $s3Client.GetObject($getReq)
$getResp.WriteResponseStreamToFile($tempDownloadPath, $true)
$getResp.Dispose()
$downloadedFileChecksum = Get-FileChecksum -Path $tempDownloadPath
$usedAwsTools = $true
} catch {
Write-Warning "AWS Tools path failed or not available. Falling back to native HTTP (SigV4). Details: $($_.Exception.Message)"
@@ -391,48 +300,49 @@ if ($storageProvider -ieq 'S3') {
Write-Host "Encrypted file uploaded to S3-compatible bucket: $blobName"
$tempDownloadPath = [System.IO.Path]::GetTempFileName()
Invoke-S3GetToFile -endpointUrl $s3EndpointUrl -bucket $s3BucketName -key $blobName -targetPath $tempDownloadPath -region $s3Region -ak $s3AccessKeyId -sk $s3SecretAccessKey -forcePathStyle:$s3ForcePathStyle
$downloadedFileChecksum = Get-FileChecksum -Path $tempDownloadPath
}
}
else {
# Azure Blob Storage path (default)
} else {
$sas = $sasToken
if ([string]::IsNullOrWhiteSpace($sas)) { Write-Error 'sasToken is missing in settings.'; exit }
if ([string]::IsNullOrWhiteSpace($sas)) { throw 'sasToken is missing in settings.' }
$sas = $sas.Trim(); if (-not $sas.StartsWith('?')) { $sas = '?' + $sas }
try { Import-Module Az.Storage -ErrorAction Stop } catch {}
$storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -SasToken $sas
# Ensure container exists
$container = Get-AzStorageContainer -Name $containerName -Context $storageContext -ErrorAction SilentlyContinue
if (-not $container) { Write-Error "Azure container '$containerName' not found or access denied."; exit }
if (-not $container) { throw "Azure container '$containerName' not found or access denied." }
# Upload the encrypted file to Azure Blob Storage
Set-AzStorageBlobContent -File $encryptedFilePath -Container $containerName -Blob $blobName -Context $storageContext | Out-Null
Write-Host "Encrypted file uploaded to Azure Blob Storage: $blobName"
# Download the blob to a temporary location to verify
$tempDownloadPath = [System.IO.Path]::GetTempFileName()
Get-AzStorageBlobContent -Blob $blobName -Container $containerName -Context $storageContext -Destination $tempDownloadPath -Force | Out-Null
# Calculate the downloaded file checksum
$downloadedFileChecksum = Get-FileChecksum -Path $tempDownloadPath
}
# Compare the checksums
if ($localFileChecksum -eq $downloadedFileChecksum) {
Write-Host "The file was correctly uploaded. Checksum verified."
# Clean up local and temporary files only on success
Remove-Item -Path $exportPath, $compressedFilePath, $encryptedFilePath, $tempDownloadPath -Force
if ($storageProvider -ieq 'S3') {
Write-Host "Local and temporary files cleaned up after uploading to S3-compatible storage."
} else {
Write-Host "Local and temporary files cleaned up after uploading to Azure Blob Storage."
}
}
else {
Write-Warning "Checksum verification failed. Keeping local artifacts for investigation: $exportPath, $compressedFilePath, $encryptedFilePath"
if (Test-Path $tempDownloadPath) { Remove-Item -Path $tempDownloadPath -Force }
}
$downloadedFileChecksum = Get-FileChecksum -Path $tempDownloadPath
if ($localFileChecksum -eq $downloadedFileChecksum) {
Write-Host "The file was correctly uploaded. Checksum verified."
Remove-Item -Path $encryptedFilePath -Force
Remove-Item -Path $tempDownloadPath -Force
if ($storageProvider -ieq 'S3') {
Write-Host "Upload to S3-compatible storage completed and verified."
} else {
Write-Host "Upload to Azure Blob Storage completed and verified."
}
} else {
Write-Warning "Checksum verification failed. Encrypted file preserved for investigation: $encryptedFilePath"
if ($tempDownloadPath -and (Test-Path $tempDownloadPath)) {
Remove-Item -Path $tempDownloadPath -Force -ErrorAction SilentlyContinue
}
}
} finally {
# Always delete plaintext hashes and compressed archive regardless of outcome.
foreach ($f in @($exportPath, $compressedFilePath)) {
if ($f -and (Test-Path $f)) {
Remove-Item -Path $f -Force -ErrorAction SilentlyContinue
}
}
}
Write-Host "Script execution completed."
} finally {
File diff suppressed because it is too large Load Diff
+61 -6
View File
@@ -12,27 +12,82 @@ Sensitive operations are confined only to the dedicated host. In the third step,
## Prerequisities
* **Windows Host:** A Windows machine with PowerShell and DSInternals suite installed.
* **Administrative Access:** Local admin privileges on the host for installation and updating.
* **Domain Credentials:** A domain user account with Domain Admin privileges for each tested AD domain. This account should be active only during testing.
* **Domain Credentials:** For weak-password testing (option 2), an account with the three replication rights (`Replicating Directory Changes`, `Replicating Directory Changes All`, `Replicating Directory Changes In Filtered Set`) on the domain naming context; Domain Admin also works but is not required. Keep this account disabled and enable only when running tests.
* **Network Requirements:** A stable connection to the domain controller in each tested AD domain and internet access (specific hostnames/IP addresses will be provided).
## Versioning and Releases
Elysium uses a **unified project version** starting with v2.2.0. Every script, the settings template, and the documentation share the same version number so you can verify consistency at a glance by checking the header of any `.ps1` file. Releases are tagged in Git (`v<major>.<minor>.<patch>`) and documented in `CHANGELOG.md`.
Prior to v2.2.0, each script carried its own version number; those historical versions are preserved in the changelog for reference.
---
## Operation
### Install and update
This tool is provided in private git repository. Installation and updating is done with cloning and pulling from this repository.
During first run, the tool will ask for passphrase that will be used to encrypt/decrypt sensitive content.
After installation, edit ElysiumSettings.txt, check all variables and add domains to test.
All scripts automatically relaunch under PowerShell 7 (`pwsh`) when it is installed so that features like parallel transfers are available; if pwsh is missing they continue under Windows PowerShell 5.1 with the legacy single-threaded behavior. The two DSInternals-driven workflows (menu options 2 and 3) load the legacy `ActiveDirectory` and `DSInternals` modules, so they automatically fall back to Windows PowerShell even if pwsh is present.
### Update Known-Hashed Database (KHDB)
Run script Elysium.ps1 as an administrator and choose option 1 (Update Known-Hashes Database).
The script downloads the database from the configured storage (Azure Blob or S3-compatible), decompresses it and updates the current database.
The updater now pulls a manifest plus individual hash shards (two-hex prefix layout) from the configured storage (Azure Blob or S3-compatible), verifies checksums, replaces only changed shards, and rebuilds `khdb.txt` for local use. Deleted shards listed in the manifest are removed automatically. When PowerShell 7 is available the downloader automatically fetches up to `-MaxParallelTransfers` shards in parallel (default `5`); on Windows PowerShell 5.1 it reverts to the original sequential behavior. Override the concurrency as needed when running the script directly (for example `.\Update-KHDB.ps1 -MaxParallelTransfers 8`).
To publish an updated shard set, run `Prepare-KHDBStorage.ps1` against your sorted `khdb.txt` (or point it at the directory/list of the Have I Been Pwned `.gz` slices). The helper reconstructs the full 32hex NTLM values (prefix + remainder), deduplicates per hash (keeping the largest count), splits by the first two hex characters, writes a manifest (`version`, `sha256`, `size`, entry counts), and can upload the resulting files directly to Azure Blob Storage (via SAS) or S3-compatible endpoints using SigV4. Invalid or malformed entries are omitted automatically, and a short report (aggregate counts + `invalid-hashes.txt`) is produced for review. Example:
```powershell
.\Prepare-KHDBStorage.ps1 -SourcePath .\khdb.txt `
-OutputRoot .\publish `
-StorageProvider S3 `
-S3EndpointUrl https://s3.example.com `
-S3BucketName private-khdb `
-S3AccessKeyId AKIA... `
-S3SecretAccessKey ... `
-ManifestRemotePath khdb/manifest.json `
-ShardRemotePrefix khdb/shards
```
Use `-SkipUpload` to stage files locally, or `-StorageProvider Azure` with `storageAccountName`/`containerName`/`sasToken` when targeting Azure Blob Storage. Add `-ShowProgress` (optionally tune `-ProgressUpdateInterval`) if you want a running `Write-Progress` indicator while the hashes are being split. Pass `-ForcePlainText` when your `khdb.txt` already contains complete hashes and you want `.gz` references treated as invalid instead of being expanded. When you only need to push an already prepared package, combine `-UploadOnly` with `-OutputRoot` pointing at the existing shard directory and choose the storage provider to perform an upload-only run. Missing storage values are pulled from `ElysiumSettings.txt` automatically (override the path with `-SettingsPath`) so you dont have to retype S3/Azure credentials for every run. On PowerShell 7, `Prepare-KHDBStorage.ps1` can push shards concurrently by setting `-MaxParallelTransfers` (default `5`); Windows PowerShell 5.1 automatically falls back to serial uploads.
Every run also emits a cleaned, DSInternals-friendly `khdb-clean.txt` beside the shards so you can inspect or distribute the merged list before publishing.
When `-ForcePlainText` is specified the script automatically keeps a checkpoint (default: `<output>/khdb.checkpoint.json`) and resumes from it on the next run so massive inputs dont restart from scratch. Use `-CheckpointPath` to relocate that file or `-NoCheckpoint` to disable the behavior entirely.
### Test Weak AD passwords
Run script Elysium.ps1 as an administrator and choose option 2 (Test Weak AD Passwords).
The script will then ask for the domain to be tested and upon choice will ask for domain administrator password. The DA username is already provided in the script for each domain.
The tool then connects to Domain Controller and tests all enabled users in the domain against KHDB. PDF report with findings is then generated.
The script lists domains in the same order as they appear in `ElysiumSettings.txt`. After you pick one, it prompts for credentials and validates them against the selected domain controller before running the password-quality test.
The tool connects to the selected Domain Controller and compares accounts against KHDB (respecting the optional `CheckOnlyEnabledUsers` flag if configured). A timestamped text report is saved under `Reports`, and accounts with dictionary hits are also exported to a dedicated UPN-only text file to support follow-up automation.
The KHDB file is consumed by DSInternals as a sorted hash list with one NT hash per line (for example `HASH`). Do not include `:count` suffixes in `khdb.txt`; the packaging and update scripts normalize legacy `HASH:count` input to the hash-only format automatically.
#### Least privileges for password-quality testing
The DSInternals cmdlets (`Get-ADReplAccount`/`Test-PasswordQuality`) pull replicated password data, which requires DCSync-style rights. The account that runs option 2 does not have to be a Domain Admin if it has these permissions on the domain naming context:
- `Replicating Directory Changes`
- `Replicating Directory Changes All`
- `Replicating Directory Changes In Filtered Set` (needed on 2008 R2+ to read password hashes)
To delegate, enable Advanced Features in ADUC, right-click the domain, choose *Delegate Control…*, pick the service account, select *Create a custom task*, apply to *This object and all descendant objects*, and tick the three replication permissions above. Keep this account disabled and only activate it for scheduled tests.
#### Common errors
- `The server has rejected the client credentials.` or `Credentials ... were rejected`:
The supplied username/password is invalid for the selected domain controller, or the session is not running in the expected domain context. Re-run and provide valid domain credentials.
- `Account '<user>' is missing the following replication permissions ...`:
Starting with v2.2.0, the script pre-validates the three required replication extended rights against the domain object ACL before attempting DCSync. If this error appears, delegate the listed rights (see *Least privileges* above) and retry.
- `Get-ADReplAccount: Access is denied`:
Credentials are valid, but the account does not have the three replication permissions listed above. This error should now be rare because the pre-check catches most permission issues early; if it still occurs, verify the account is not restricted by an additional conditional access or Group Policy setting.
- `Only FIPS certified cryptographic algorithms are enabled in .NET`:
This warning comes from DSInternals under FIPS-enforced environments. Hash-quality operations that rely on MD5 may be limited.
#### Optional usage beacon
If you want to know the script was executed without collecting telemetry, set a pre-signed URL (for example, an S3 `PUT` URL) in `UsageBeaconUrl` inside `ElysiumSettings.txt`. When present, the weak-password script issues a single request as soon as it loads the settings. Only the script name, its version, a UTC timestamp, and the optional `UsageBeaconInstanceId` value are sent, and network failures never block the run. Choose the HTTP verb via `UsageBeaconMethod` (`GET`, `POST`, or `PUT`) and adjust the timeout with `UsageBeaconTimeoutSeconds` if your storage endpoint needs more time.
### Send current hashes for update KHDB
Run script Elysium.ps1 as an administrator and choose option 3 (Extract and Send Hashes).
The tool will then ask for domain and password of domain administrator. With correct credentials, the tool will then extract current hashes (no history) of non-disabled users, compresses and encrypts them and uploads them to the configured storage (Azure Blob or S3-compatible) for pickup by the tool provider.
Domains are listed in configuration order, after which the script prompts for the replication-capable account password. With valid credentials, it extracts current NTLM hashes (no history) for active accounts, compresses the results, encrypts them with the configured passphrase, and uploads the payload to the configured storage (Azure Blob or S3-compatible). A checksum-verified round-trip download confirms the upload before local artifacts are removed.
### Update Lithnet Password Protection store
Run script Elysium.ps1 as an administrator and choose option 5 (Update Lithnet Password Protection Store).
Configure the target folder via `LithnetStorePath` in `ElysiumSettings.txt` (the location created with `Open-Store`). The script automatically imports the `khdb.txt` file unless you override/add additional NTLM hash lists in `LithnetHashSources` (comma or semicolon separated). You can also populate plaintext password lists (`LithnetPlaintextSources`) and banned-word files (`LithnetBannedWordSources`), or enable `LithnetSyncHibp=true` to seed the store directly from the Have I Been Pwned API (using `Sync-HashesFromHibp`). Behind the scenes the helper loads the `LithnetPasswordProtection` module, opens the store, runs [`Import-CompromisedPasswordHashes`](https://docs.lithnet.io/password-protection/advanced-help/powershell-reference/import-compromisedpasswordhashes)/`Import-CompromisedPasswords`/`Import-BannedWords` for each configured file, and then closes the store.
S3-compatible usage notes:
- No AWS Tools required. The scripts can sign requests using native SigV4 via .NET and HttpClient.
- No AWS Tools required. The scripts sign requests using native SigV4 via .NET and HttpClient, including non-default endpoint ports.
- To force using AWS Tools instead, set `s3UseAwsTools = true` in `ElysiumSettings.txt` and install `AWS.Tools.S3`.
### Uninstallation
Run script Elysium.ps1 as an administrator and choose option 4 (Uninstall).
+326 -48
View File
@@ -8,7 +8,7 @@
##################################################
## Project: Elysium ##
## File: Test-WeakADPasswords.ps1 ##
## Version: 1.3.0 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
@@ -21,10 +21,14 @@ Weak AD password finder component of Elysium tool.
This script will test the passwords of selected domain (defined in ElysiumSettings.txt) using DSInternals' Test-PasswordQuality cmdlet. It writes its output to a report file which is meant to be shared with the internal security team. The report now includes UPNs for each account mentioned.
#>
# Enable verbose output
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
$VerbosePreference = "Continue"
[string]$commonHelper = Join-Path -Path $PSScriptRoot -ChildPath 'Elysium.Common.ps1'
if (-not (Test-Path -LiteralPath $commonHelper)) { throw "Common helper not found at $commonHelper" }
. $commonHelper
$VerbosePreference = "SilentlyContinue"
$scriptRoot = $PSScriptRoot
@@ -52,6 +56,63 @@ function Start-TestTranscript {
function Stop-TestTranscript { try { Stop-Transcript | Out-Null } catch {} }
function Invoke-UsageBeacon {
param(
[string]$Url,
[string]$Method = 'GET',
[int]$TimeoutSeconds = 5,
[string]$InstanceId
)
if ([string]::IsNullOrWhiteSpace($Url)) { return }
$normalizedMethod = 'GET'
if (-not [string]::IsNullOrWhiteSpace($Method)) {
$normalizedMethod = $Method.ToUpperInvariant()
}
if ($normalizedMethod -notin @('GET', 'POST', 'PUT')) {
$normalizedMethod = 'GET'
}
$requestParams = @{
Uri = $Url
Method = $normalizedMethod
ErrorAction = 'Stop'
}
$invokeWebRequestCmd = $null
try { $invokeWebRequestCmd = Get-Command -Name Invoke-WebRequest -ErrorAction Stop } catch { }
if ($invokeWebRequestCmd -and $invokeWebRequestCmd.Parameters.ContainsKey('UseBasicParsing')) {
$requestParams['UseBasicParsing'] = $true
}
if ($TimeoutSeconds -gt 0 -and $invokeWebRequestCmd -and $invokeWebRequestCmd.Parameters.ContainsKey('TimeoutSec')) {
$requestParams['TimeoutSec'] = $TimeoutSeconds
}
if (-not [string]::IsNullOrWhiteSpace($InstanceId)) {
$requestParams['Headers'] = @{ 'X-Elysium-Instance' = $InstanceId }
}
if ($normalizedMethod -in @('POST', 'PUT')) {
$payload = [ordered]@{
script = 'Test-WeakADPasswords'
version = $ElysiumVersion
ranAtUtc = (Get-Date).ToUniversalTime().ToString('o')
}
if (-not [string]::IsNullOrWhiteSpace($InstanceId)) {
$payload['instanceId'] = $InstanceId
}
$requestParams['ContentType'] = 'application/json'
$requestParams['Body'] = ($payload | ConvertTo-Json -Depth 3 -Compress)
}
try {
Invoke-WebRequest @requestParams | Out-Null
Write-Verbose ("Usage beacon sent via {0}." -f $normalizedMethod)
} catch {
Write-Verbose ("Usage beacon failed: {0}" -f $_.Exception.Message)
}
}
# Current timestamp for both report generation and header
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
@@ -66,31 +127,32 @@ $footer = "`r`n==== End of Report ===="
Start-TestTranscript -BasePath $scriptRoot
try {
# Import settings
Write-Verbose "Loading settings..."
$ElysiumSettings = @{}
$settingsPath = Join-Path -Path $scriptRoot -ChildPath "ElysiumSettings.txt"
# Ensure the settings file exists
if (-not (Test-Path $settingsPath)) {
Write-Error "Settings file not found at $settingsPath"
exit
}
# Load settings from file
try {
Get-Content $settingsPath | ForEach-Object {
if (-not [string]::IsNullOrWhiteSpace($_) -and -not $_.StartsWith("#")) {
$keyValue = $_ -split '=', 2
if ($keyValue.Count -eq 2) {
$ElysiumSettings[$keyValue[0].Trim()] = $keyValue[1].Trim()
}
}
}
$ElysiumSettings = Read-ElysiumSettings -ScriptRoot $scriptRoot
Write-Verbose "Settings loaded successfully."
} catch {
Write-Error ("An error occurred while loading settings: {0}" -f $_.Exception.Message)
exit
$usageBeaconUrl = $ElysiumSettings['UsageBeaconUrl']
$usageBeaconMethod = $ElysiumSettings['UsageBeaconMethod']
$usageBeaconInstanceId = $ElysiumSettings['UsageBeaconInstanceId']
$usageBeaconTimeoutSeconds = $null
if ($ElysiumSettings.ContainsKey('UsageBeaconTimeoutSeconds')) {
$parsedTimeout = 0
if ([int]::TryParse($ElysiumSettings['UsageBeaconTimeoutSeconds'], [ref]$parsedTimeout)) {
$usageBeaconTimeoutSeconds = $parsedTimeout
}
}
if (-not [string]::IsNullOrWhiteSpace($usageBeaconUrl)) {
$beaconParams = @{ Url = $usageBeaconUrl }
if (-not [string]::IsNullOrWhiteSpace($usageBeaconMethod)) {
$beaconParams['Method'] = $usageBeaconMethod
}
if (-not [string]::IsNullOrWhiteSpace($usageBeaconInstanceId)) {
$beaconParams['InstanceId'] = $usageBeaconInstanceId
}
if ($null -ne $usageBeaconTimeoutSeconds) {
$beaconParams['TimeoutSeconds'] = $usageBeaconTimeoutSeconds
}
Invoke-UsageBeacon @beaconParams
}
# Define the function to extract domain details from settings
@@ -99,7 +161,7 @@ function Get-DomainDetailsFromSettings {
[hashtable]$Settings
)
$domainDetails = @{}
$domainDetails = [ordered]@{}
$counter = 1
while ($true) {
$nameKey = "Domain${counter}Name"
@@ -135,6 +197,28 @@ if ($runningInPSCore -and -not $onWindows) {
throw 'This script requires Windows when running under PowerShell 7 (AD/DSInternals are Windows-only).'
}
function Test-IsFipsPolicyEnabled {
if (-not $onWindows) { return $false }
try {
$fipsReg = Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy' -Name Enabled -ErrorAction Stop
return ([int]$fipsReg.Enabled -eq 1)
} catch {
return $false
}
}
if (Test-IsFipsPolicyEnabled) {
throw @"
FIPS policy is enabled on this host (HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy\Enabled = 1).
Test-WeakADPasswords uses DSInternals/AD replication operations that are not fully compatible with this policy in this environment.
Remediation:
1. Run this script from a dedicated non-FIPS workstation/jump host.
2. If approved by your security policy, temporarily disable local FIPS policy for this host, run the test, then re-enable it.
3. If FIPS must remain enforced, use an alternative fully FIPS-validated workflow/tool for weak password assessment.
"@
}
function Test-IsAdmin {
try {
$wi = [Security.Principal.WindowsIdentity]::GetCurrent()
@@ -237,6 +321,44 @@ function Import-CompatModule {
$params['UseWindowsPowerShell'] = $true
}
}
if ($Name -eq 'DSInternals') {
# DSInternals can emit a FIPS MD5 warning via Write-Error during import; treat it as non-fatal if the module loads.
$params['ErrorAction'] = 'SilentlyContinue'
$importErrors = @()
try {
Import-Module @params -ErrorVariable +importErrors
} catch {
$fqid = [string]$_.FullyQualifiedErrorId
$message = $_.Exception.Message
$isFipsBootstrapError = ($fqid -match 'DSInternals\.Bootstrap\.psm1') -and ($message -match 'Only FIPS certified cryptographic algorithms are enabled in \.NET')
if (-not $isFipsBootstrapError) { throw }
Write-Warning "DSInternals bootstrap reported FIPS restrictions. Continuing if the module is available."
}
$moduleLoaded = [bool](Get-Module -Name $Name -ErrorAction SilentlyContinue)
if (-not $moduleLoaded) {
$fipsErrorSeen = @($importErrors | Where-Object { $_.Exception.Message -match 'Only FIPS certified cryptographic algorithms are enabled in \.NET' }).Count -gt 0
if ($fipsErrorSeen) {
throw "DSInternals could not be loaded under current FIPS policy. Use a host/policy that allows required algorithms for DSInternals."
}
if ($importErrors.Count -gt 0) { throw $importErrors[0] }
throw "Failed to import module '$Name'."
}
$fipsErrors = @($importErrors | Where-Object { $_.Exception.Message -match 'Only FIPS certified cryptographic algorithms are enabled in \.NET' })
if ($fipsErrors.Count -gt 0) {
Write-Warning "DSInternals loaded under FIPS policy. MD5-dependent DSInternals checks may be limited."
}
$nonFipsErrors = @($importErrors | Where-Object { $_.Exception.Message -notmatch 'Only FIPS certified cryptographic algorithms are enabled in \.NET' })
if ($nonFipsErrors.Count -gt 0) {
Write-Warning ("DSInternals import reported non-fatal warning(s): {0}" -f $nonFipsErrors[0].Exception.Message)
}
Write-Verbose ("Imported module '{0}' (Core={1}, Windows={2})" -f $Name, $runningInPSCore, $onWindows)
return
}
Import-Module @params
Write-Verbose ("Imported module '{0}' (Core={1}, Windows={2})" -f $Name, $runningInPSCore, $onWindows)
}
@@ -320,17 +442,117 @@ function Get-UserUPN {
# (removed stray top-level loop; UPN enrichment happens during report generation below)
function Resolve-DSInternalsWeakHashFile {
param(
[Parameter(Mandatory)][string]$Path
)
if (-not (Test-Path -LiteralPath $Path)) {
throw "Weak password hashes file not found at '$Path'."
}
$compatibleRegex = '^[0-9A-F]{32}$'
$legacyRegex = '^[0-9A-Fa-f]{32}(:\d+)?$'
$lineNumber = 0
$previousHash = $null
$duplicateCount = 0
$legacyEntryCount = 0
$needsNormalization = $false
$reader = $null
try {
$reader = New-Object System.IO.StreamReader($Path, [System.Text.Encoding]::UTF8, $true)
while (($line = $reader.ReadLine()) -ne $null) {
$lineNumber++
$trimmed = $line.Trim()
if ($trimmed.Length -eq 0) { continue }
if ($trimmed -notmatch $legacyRegex) {
throw ("Weak password hashes file '{0}' contains invalid content at line {1}: '{2}'." -f $Path, $lineNumber, $trimmed)
}
if ($trimmed -notmatch $compatibleRegex) {
$needsNormalization = $true
if ($trimmed.Contains(':')) { $legacyEntryCount++ }
}
$normalizedHash = ($trimmed.Split(':', 2)[0]).ToUpperInvariant()
if ($line -cne $trimmed -or $trimmed -cne $normalizedHash) {
$needsNormalization = $true
}
if ($null -ne $previousHash) {
if ($normalizedHash -lt $previousHash) {
throw "Weak password hashes file '$Path' is not sorted alphabetically at line $lineNumber."
}
if ($normalizedHash -eq $previousHash) {
$duplicateCount++
$needsNormalization = $true
}
}
$previousHash = $normalizedHash
}
} finally {
if ($reader) { $reader.Dispose() }
}
if (-not $needsNormalization) {
return [pscustomobject]@{
Path = $Path
IsTemporary = $false
}
}
$tmpPath = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), ('elysium-khdb-' + [System.Guid]::NewGuid().ToString() + '.txt'))
$encoding = New-Object System.Text.UTF8Encoding($false)
$reader = $null
$writer = $null
$lastWrittenHash = $null
try {
$reader = New-Object System.IO.StreamReader($Path, [System.Text.Encoding]::UTF8, $true)
$writer = New-Object System.IO.StreamWriter($tmpPath, $false, $encoding)
while (($line = $reader.ReadLine()) -ne $null) {
$trimmed = $line.Trim()
if ($trimmed.Length -eq 0) { continue }
$normalizedHash = ($trimmed.Split(':', 2)[0]).ToUpperInvariant()
if ($normalizedHash -eq $lastWrittenHash) { continue }
$writer.WriteLine($normalizedHash)
$lastWrittenHash = $normalizedHash
}
} finally {
if ($reader) { $reader.Dispose() }
if ($writer) { $writer.Dispose() }
}
$normalizationReasons = @()
if ($legacyEntryCount -gt 0) { $normalizationReasons += "$legacyEntryCount legacy HASH:count entries" }
if ($duplicateCount -gt 0) { $normalizationReasons += "$duplicateCount duplicate hashes" }
if ($normalizationReasons.Count -eq 0) { $normalizationReasons += 'format normalization' }
Write-Warning ("Normalized weak password hashes file for DSInternals compatibility ({0}). Temporary file: {1}" -f ($normalizationReasons -join ', '), $tmpPath)
return [pscustomobject]@{
Path = $tmpPath
IsTemporary = $true
}
}
# Function to test for weak AD passwords
function Test-WeakADPasswords {
param (
[hashtable]$DomainDetails,
[string]$FilePath,
[bool]$CheckOnlyEnabledUsers = $false
[bool]$CheckOnlyEnabledUsers = $false,
[System.Management.Automation.PSCredential]$Credential
)
# User selects a domain
Write-Host "Select a domain to test:"
$DomainDetails.GetEnumerator() | ForEach-Object { Write-Host "$($_.Key): $($_.Value.Name)" }
$DomainDetails.GetEnumerator() | Sort-Object { [int]$_.Key } | ForEach-Object { Write-Host "$($_.Key): $($_.Value.Name)" }
$selection = Read-Host "Enter the number of the domain"
if (-not ($DomainDetails.ContainsKey($selection))) {
@@ -341,12 +563,33 @@ function Test-WeakADPasswords {
$selectedDomain = $DomainDetails[$selection]
Write-Verbose "Selected domain: $($selectedDomain.Name)"
# Prompt for DA credentials
$credential = Get-Credential -Message "Enter AD credentials with replication rights for $($selectedDomain.Name)"
if ([string]::IsNullOrWhiteSpace($selectedDomain["DC"])) {
Write-Error ("Domain '{0}' does not have a configured DC in ElysiumSettings.txt." -f $selectedDomain.Name)
return
}
if ($null -eq $Credential) {
$credential = Get-ValidatedADCredential -DomainName $selectedDomain.Name -Server $selectedDomain["DC"]
} else {
$credential = $Credential
Write-Verbose ("Using credential supplied by caller: {0}" -f $credential.UserName)
}
# Verify the account has the three replication extended rights before attempting DCSync
try {
$domainInfo = Get-ADDomain -Server $selectedDomain["DC"] -Credential $credential -ErrorAction Stop
Test-ReplicationPermissions -DomainDN $domainInfo.DistinguishedName `
-Server $selectedDomain["DC"] -Credential $credential
} catch {
Write-Error $_.Exception.Message
return
}
# Performing the test
Write-Verbose "Testing password quality for $($selectedDomain.Name)..."
$resolvedHashFile = $null
try {
$resolvedHashFile = Resolve-DSInternalsWeakHashFile -Path $FilePath
$accounts = Get-ADReplAccount -All -Server $selectedDomain["DC"] -Credential $credential
if ($CheckOnlyEnabledUsers) {
Write-Verbose "Filtering to only enabled users per settings."
@@ -355,50 +598,85 @@ function Test-WeakADPasswords {
if ($_.PSObject.Properties.Name -contains 'Enabled') { $_.Enabled } else { $true }
}
}
$testResults = $accounts | Test-PasswordQuality -WeakPasswordHashesFile $FilePath
$testResults = $accounts | Test-PasswordQuality -WeakPasswordHashesSortedFile $resolvedHashFile.Path
Write-Verbose "Password quality test completed."
} catch {
Write-Error ("An error occurred while testing passwords: {0}" -f $_.Exception.Message)
$message = $_.Exception.Message
if ($message -match 'Access is denied') {
Write-Error ("Access denied while reading replication data from '{0}' using '{1}'. Ensure this account has Replicating Directory Changes, Replicating Directory Changes All, and Replicating Directory Changes In Filtered Set on the domain." -f $selectedDomain["DC"], $credential.UserName)
return
}
if ($message -match 'rejected the client credentials|unknown user name|bad password|logon failure') {
Write-Error ("Credentials for '{0}' were rejected by '{1}'. Re-run and provide valid domain credentials." -f $credential.UserName, $selectedDomain["DC"])
return
}
Write-Error ("An error occurred while testing passwords: {0}" -f $message)
return
} finally {
if ($resolvedHashFile -and $resolvedHashFile.IsTemporary -and (Test-Path -LiteralPath $resolvedHashFile.Path)) {
try { Remove-Item -LiteralPath $resolvedHashFile.Path -Force -ErrorAction Stop } catch { }
}
}
# Report generation with dynamic content and UPNs
$reportPath = Join-Path -Path $reportPathBase -ChildPath "$($selectedDomain.Name)_WeakPasswordReport_$timestamp.txt"
$upnOnlyReportPath = Join-Path -Path $reportPathBase -ChildPath "$($selectedDomain.Name)_DictionaryPasswordUPNs_$timestamp.txt"
# Build a lookup of SAM account names to UPNs for dictionary hits by leveraging structured results
$dictionaryLogonNames = @()
foreach ($result in @($testResults)) {
if ($null -ne $result -and $null -ne $result.WeakPassword) {
$dictionaryLogonNames += $result.WeakPassword
}
}
$dictionaryLogonNames = $dictionaryLogonNames | Sort-Object -Unique
$dictionarySamToUpn = @{}
$upnReportContent = @()
foreach ($logonName in $dictionaryLogonNames) {
$samAccountName = $logonName -replace '^.*\\', ''
if (-not [string]::IsNullOrWhiteSpace($samAccountName) -and -not $dictionarySamToUpn.ContainsKey($samAccountName)) {
Write-Verbose "Looking up UPN for $samAccountName (dictionary hit)"
$upn = Get-UserUPN -SamAccountName $samAccountName -Domain $selectedDomain.DC -Credential $credential
$dictionarySamToUpn[$samAccountName] = $upn
if ($upn -ne "UPN not found") {
$upnReportContent += $upn
}
}
}
Write-Verbose "Generating report at $reportPath"
$reportContent = @($header, ($testResults | Out-String).Trim(), $footer) -join "`r`n"
$lines = $reportContent -split "`r`n"
$newReportContent = @()
$upnReportContent = @()
$collectingUPNs = $false
foreach ($line in $lines) {
$newReportContent += $line
# Start collecting UPNs after detecting the relevant section in the report
if ($line -match "Passwords of these accounts have been found in the dictionary:") {
$collectingUPNs = $true
continue
}
# Stop collecting UPNs if a new section starts or end of section is detected
if ($collectingUPNs -and $line -match "^\s*$") {
if ($collectingUPNs) {
if ($line -match '^\s*$') { continue }
if ($line -match '^\s*-{2,}') { continue }
if ($line -match '^\s*(SamAccountName|LogonName)\b') { continue }
if ($line -match '^[^\s].*:\s*$' -and $line -notmatch 'dictionary') {
$collectingUPNs = $false
continue
}
# Regex to match the SAMAccountName from the report line and collect UPNs if in the target section
if ($collectingUPNs -and $line -match "^\s*(\S+)\s*$") {
$samAccountName = $matches[1]
Write-Verbose "Looking up UPN for $samAccountName"
$upn = Get-UserUPN -SamAccountName $samAccountName -Domain $selectedDomain.DC -Credential $credential
$newReportContent += " UPN: $upn"
# Collect UPNs only for accounts found in the dictionary section
if ($upn -ne "UPN not found") {
$upnReportContent += $upn
$firstToken = ($line.Trim() -split '\s+')[0]
if (-not [string]::IsNullOrWhiteSpace($firstToken)) {
$samAccountName = $firstToken -replace '^.*\\', ''
if ($dictionarySamToUpn.ContainsKey($samAccountName)) {
$upnValue = $dictionarySamToUpn[$samAccountName]
$newReportContent += " UPN: $upnValue"
}
}
}
}
+8 -1
View File
@@ -7,7 +7,7 @@
##################################################
## Project: Elysium ##
## File: Uninstall.ps1 ##
## Version: 1.1.0 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
@@ -19,6 +19,13 @@ Uninstall script for the Elysium AD password testing tool.
This script will remove the Elysium tool and its components (scripts, configurations, and any generated data) from the system, and then delete itself.
#>
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
[string]$commonHelper = Join-Path -Path $PSScriptRoot -ChildPath 'Elysium.Common.ps1'
if (-not (Test-Path -LiteralPath $commonHelper)) { throw "Common helper not found at $commonHelper" }
. $commonHelper
Restart-WithPwshIfAvailable -BoundParameters $PSBoundParameters -UnboundArguments $MyInvocation.UnboundArguments
function Start-UninstallTranscript {
try {
$base = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), 'Elysium', 'logs')
+648 -195
View File
@@ -7,32 +7,34 @@
##################################################
## Project: Elysium ##
## File: Update-KHDB.ps1 ##
## Version: 1.1.0 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
<#
.SYNOPSIS
Known hashes database update script for the Elysium AD password testing tool.
Known-hashes database updater for the Elysium AD password testing tool.
.DESCRIPTION
This script downloads khdb.txt.zip from the designated storage (Azure Blob or S3-compatible), validates and decompresses it, and atomically updates the current version with backup and logging.
Downloads a sharded KHDB manifest, performs incremental shard updates, validates
checksums, and atomically refreshes the merged khdb.txt for downstream scripts.
Supports Azure Blob Storage (via SAS) and S3-compatible endpoints (SigV4).
#>
# safer defaults
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
# ensure TLS 1.2
[string]$commonHelper = Join-Path -Path $PSScriptRoot -ChildPath 'Elysium.Common.ps1'
if (-not (Test-Path -LiteralPath $commonHelper)) { throw "Common helper not found at $commonHelper" }
. $commonHelper
Restart-WithPwshIfAvailable -BoundParameters $PSBoundParameters -UnboundArguments $MyInvocation.UnboundArguments
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor [System.Net.SecurityProtocolType]::Tls12
# Resolve paths
$scriptRoot = $PSScriptRoot
function Start-UpdateTranscript {
param(
[string]$BasePath
)
param([string]$BasePath)
try {
$logsDir = Join-Path -Path $BasePath -ChildPath 'Reports/logs'
if (-not (Test-Path $logsDir)) { New-Item -Path $logsDir -ItemType Directory -Force | Out-Null }
@@ -48,19 +50,6 @@ function Stop-UpdateTranscript {
try { Stop-Transcript | Out-Null } catch {}
}
function Read-ElysiumSettings {
$settings = @{}
$settingsPath = Join-Path -Path $scriptRoot -ChildPath 'ElysiumSettings.txt'
if (-not (Test-Path $settingsPath)) { throw "Settings file not found at $settingsPath" }
Get-Content $settingsPath | ForEach-Object {
if ($_ -and -not $_.Trim().StartsWith('#')) {
$kv = $_ -split '=', 2
if ($kv.Count -eq 2) { $settings[$kv[0].Trim()] = $kv[1].Trim().Trim("'") }
}
}
return $settings
}
function Get-InstallationPath([hashtable]$settings) {
$p = $settings['InstallationPath']
if ([string]::IsNullOrWhiteSpace($p)) { return $scriptRoot }
@@ -69,136 +58,35 @@ function Get-InstallationPath([hashtable]$settings) {
}
function New-HttpClient {
Add-Type -AssemblyName System.Net.Http
Add-Type -AssemblyName System.Net.Http -ErrorAction SilentlyContinue
$client = [System.Net.Http.HttpClient]::new()
$client.Timeout = [TimeSpan]::FromSeconds(600)
$client.DefaultRequestHeaders.UserAgent.ParseAdd('Elysium/1.1 (+Update-KHDB)')
$client.DefaultRequestHeaders.UserAgent.ParseAdd("Elysium/$ElysiumVersion (+Update-KHDB)")
return $client
}
function Build-BlobUri([string]$account, [string]$container, [string]$sas) {
if ([string]::IsNullOrWhiteSpace($account)) { throw 'storageAccountName is missing or empty.' }
if ([string]::IsNullOrWhiteSpace($container)) { throw 'containerName is missing or empty.' }
if ([string]::IsNullOrWhiteSpace($sas)) { throw 'sasToken is missing or empty.' }
$sas = $sas.Trim()
if (-not $sas.StartsWith('?')) { $sas = '?' + $sas }
$ub = [System.UriBuilder]::new("https://$account.blob.core.windows.net/$container/khdb.txt.zip")
$ub.Query = $sas.TrimStart('?')
return $ub.Uri.AbsoluteUri
}
function Ensure-AWSS3Module {
try { $null = [Amazon.S3.AmazonS3Client]; return } catch {}
try { Import-Module -Name AWS.Tools.S3 -ErrorAction Stop; return } catch {}
try { Import-Module -Name AWSPowerShell.NetCore -ErrorAction Stop; return } catch {}
throw "AWS Tools for PowerShell not found. Install with: Install-Module AWS.Tools.S3 -Scope CurrentUser"
}
function New-S3Client {
function Invoke-DownloadWithRetry {
param(
[string]$EndpointUrl,
[string]$Region,
[string]$AccessKeyId,
[string]$SecretAccessKey,
[bool]$ForcePathStyle = $true
[System.Net.Http.HttpClient]$Client,
[string]$Uri,
[string]$TargetPath,
[string]$Activity
)
Ensure-AWSS3Module
$creds = New-Object Amazon.Runtime.BasicAWSCredentials($AccessKeyId, $SecretAccessKey)
$cfg = New-Object Amazon.S3.AmazonS3Config
if ($EndpointUrl) { $cfg.ServiceURL = $EndpointUrl }
if ($Region) { try { $cfg.RegionEndpoint = [Amazon.RegionEndpoint]::GetBySystemName($Region) } catch {} }
$cfg.ForcePathStyle = [bool]$ForcePathStyle
return (New-Object Amazon.S3.AmazonS3Client($creds, $cfg))
}
# Native S3 SigV4 (no AWS Tools) helpers
function Get-Bytes([string]$s) { return [System.Text.Encoding]::UTF8.GetBytes($s) }
function Get-HashHex([byte[]]$bytes) { $sha=[System.Security.Cryptography.SHA256]::Create(); try { ([BitConverter]::ToString($sha.ComputeHash($bytes))).Replace('-', '').ToLowerInvariant() } finally { $sha.Dispose() } }
function HmacSha256([byte[]]$key, [string]$data) { $h=[System.Security.Cryptography.HMACSHA256]::new($key); try { $h.ComputeHash((Get-Bytes $data)) } finally { $h.Dispose() } }
function GetSignatureKey([string]$secret, [string]$dateStamp, [string]$regionName, [string]$serviceName) {
$kDate = HmacSha256 (Get-Bytes ('AWS4' + $secret)) $dateStamp
$kRegion = HmacSha256 $kDate $regionName
$kService = HmacSha256 $kRegion $serviceName
HmacSha256 $kService 'aws4_request'
}
function UriEncode([string]$data, [bool]$encodeSlash) { $enc=[System.Uri]::EscapeDataString($data); if (-not $encodeSlash) { $enc = $enc -replace '%2F','/' }; $enc }
function BuildCanonicalPath([System.Uri]$uri) { $segments=$uri.AbsolutePath.Split('/'); $encoded=@(); foreach($s in $segments){ $encoded += (UriEncode $s $false) }; $p=($encoded -join '/'); if (-not $p.StartsWith('/')){ $p='/' + $p }; $p }
function ToHex([byte[]]$b) { ([BitConverter]::ToString($b)).Replace('-', '').ToLowerInvariant() }
function BuildAuthHeaders($method, [System.Uri]$uri, [string]$region, [string]$accessKey, [string]$secretKey, [string]$payloadHash) {
$algorithm = 'AWS4-HMAC-SHA256'
$amzdate = (Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ')
$datestamp = (Get-Date).ToUniversalTime().ToString('yyyyMMdd')
$hostHeader = $uri.Host; if (-not $uri.IsDefaultPort) { $hostHeader = "$hostHeader:$($uri.Port)" }
$canonicalUri = BuildCanonicalPath $uri
$canonicalQueryString = ''
$canonicalHeaders = "host:$hostHeader`n" + "x-amz-content-sha256:$payloadHash`n" + "x-amz-date:$amzdate`n"
$signedHeaders = 'host;x-amz-content-sha256;x-amz-date'
$canonicalRequest = "$method`n$canonicalUri`n$canonicalQueryString`n$canonicalHeaders`n$signedHeaders`n$payloadHash"
$credentialScope = "$datestamp/$region/s3/aws4_request"
$stringToSign = "$algorithm`n$amzdate`n$credentialScope`n$((Get-HashHex (Get-Bytes $canonicalRequest)))"
$signingKey = GetSignatureKey $secretKey $datestamp $region 's3'
$signature = ToHex (HmacSha256 $signingKey $stringToSign)
$authHeader = "$algorithm Credential=$accessKey/$credentialScope, SignedHeaders=$signedHeaders, Signature=$signature"
@{ 'x-amz-date' = $amzdate; 'x-amz-content-sha256' = $payloadHash; 'Authorization' = $authHeader }
}
function BuildS3Uri([string]$endpointUrl, [string]$bucket, [string]$key, [bool]$forcePathStyle) {
$base=[System.Uri]$endpointUrl; $ub=[System.UriBuilder]::new($base)
if ($forcePathStyle) { $p=$ub.Path.TrimEnd('/'); if ([string]::IsNullOrEmpty($p)){ $p='/' }; $ub.Path = ($p.TrimEnd('/') + '/' + $bucket + '/' + $key) }
else { $ub.Host = "$bucket." + $ub.Host; $p=$ub.Path.TrimEnd('/'); if ([string]::IsNullOrEmpty($p)){ $p='/' }; $ub.Path = ($p.TrimEnd('/') + '/' + $key) }
$ub.Uri
}
function Invoke-S3HttpDownloadWithRetry([string]$endpointUrl, [string]$bucket, [string]$key, [string]$targetPath, [string]$region, [string]$ak, [string]$sk, [bool]$forcePathStyle) {
Add-Type -AssemblyName System.Net.Http -ErrorAction SilentlyContinue
$client = [System.Net.Http.HttpClient]::new()
$retries=5; $delay=2
try {
for($i=0;$i -lt $retries;$i++){
try {
$uri = BuildS3Uri -endpointUrl $endpointUrl -bucket $bucket -key $key -forcePathStyle $forcePathStyle
$payloadHash = (Get-HashHex (Get-Bytes ''))
$req = [System.Net.Http.HttpRequestMessage]::new([System.Net.Http.HttpMethod]::Get, $uri)
$hdrs = BuildAuthHeaders -method 'GET' -uri $uri -region $region -accessKey $ak -secretKey $sk -payloadHash $payloadHash
$req.Headers.TryAddWithoutValidation('x-amz-date', $hdrs['x-amz-date']) | Out-Null
$req.Headers.TryAddWithoutValidation('Authorization', $hdrs['Authorization']) | Out-Null
$req.Headers.TryAddWithoutValidation('x-amz-content-sha256', $hdrs['x-amz-content-sha256']) | Out-Null
$resp = $client.SendAsync($req).Result
if (-not $resp.IsSuccessStatusCode) { throw "HTTP $([int]$resp.StatusCode) $($resp.ReasonPhrase)" }
$totalBytes = $resp.Content.Headers.ContentLength
$stream = $resp.Content.ReadAsStreamAsync().Result
$fs = [System.IO.File]::Create($targetPath)
try {
$buffer = New-Object byte[] 8192
$totalRead = 0
while (($read = $stream.Read($buffer, 0, $buffer.Length)) -gt 0) {
$fs.Write($buffer, 0, $read)
$totalRead += $read
if ($totalBytes) { $pct = ($totalRead * 100.0) / $totalBytes; Write-Progress -Activity "Downloading khdb.txt.zip" -Status ("{0:N2}% Complete" -f $pct) -PercentComplete $pct } else { Write-Progress -Activity "Downloading khdb.txt.zip" -Status ("Downloaded {0:N0} bytes" -f $totalRead) -PercentComplete 0 }
}
} finally { $fs.Close(); $stream.Close() }
if ($resp) { $resp.Dispose() }
return
} catch {
if ($i -lt ($retries - 1)) { Write-Warning "Download failed (attempt $($i+1)/$retries): $($_.Exception.Message). Retrying in ${delay}s..."; Start-Sleep -Seconds $delay; $delay=[Math]::Min($delay*2,30) }
else { throw }
} finally { if ($req){ $req.Dispose() } }
}
} finally { $client.Dispose() }
}
function Invoke-DownloadWithRetry([System.Net.Http.HttpClient]$client, [string]$uri, [string]$targetPath) {
$retries = 5
$delay = 2
for ($i = 0; $i -lt $retries; $i++) {
for ($attempt = 0; $attempt -lt $retries; $attempt++) {
try {
$resp = $client.GetAsync($uri, [System.Net.Http.HttpCompletionOption]::ResponseHeadersRead).Result
if (-not $resp.IsSuccessStatusCode) {
$code = [int]$resp.StatusCode
$response = $Client.GetAsync($Uri, [System.Net.Http.HttpCompletionOption]::ResponseHeadersRead).Result
if (-not $response.IsSuccessStatusCode) {
$code = [int]$response.StatusCode
if (($code -ge 500 -and $code -lt 600) -or $code -eq 429 -or $code -eq 408) { throw "Transient HTTP error $code" }
throw "HTTP error $code"
}
$totalBytes = $resp.Content.Headers.ContentLength
$stream = $resp.Content.ReadAsStreamAsync().Result
$fs = [System.IO.File]::Create($targetPath)
$totalBytes = $response.Content.Headers.ContentLength
$stream = $response.Content.ReadAsStreamAsync().Result
$fs = [System.IO.File]::Create($TargetPath)
try {
$buffer = New-Object byte[] 8192
$totalRead = 0
@@ -207,18 +95,21 @@ function Invoke-DownloadWithRetry([System.Net.Http.HttpClient]$client, [string]$
$totalRead += $read
if ($totalBytes) {
$pct = ($totalRead * 100.0) / $totalBytes
Write-Progress -Activity "Downloading khdb.txt.zip" -Status ("{0:N2}% Complete" -f $pct) -PercentComplete $pct
Write-Progress -Activity $Activity -Status ("{0:N2}% Complete" -f $pct) -PercentComplete $pct
} else {
Write-Progress -Activity "Downloading khdb.txt.zip" -Status ("Downloaded {0:N0} bytes" -f $totalRead) -PercentComplete 0
Write-Progress -Activity $Activity -Status ("Downloaded {0:N0} bytes" -f $totalRead) -PercentComplete 0
}
}
} finally {
$fs.Close(); $stream.Close()
$fs.Close()
$stream.Close()
}
Write-Progress -Activity $Activity -Completed -Status 'Completed'
return
} catch {
if ($i -lt ($retries - 1)) {
Write-Warning "Download failed (attempt $($i+1)/$retries): $($_.Exception.Message). Retrying in ${delay}s..."
if ($attempt -lt ($retries - 1)) {
Write-Warning "Download of '$Uri' failed (attempt $($attempt + 1)/$retries): $($_.Exception.Message). Retrying in ${delay}s..."
Start-Sleep -Seconds $delay
$delay = [Math]::Min($delay * 2, 30)
} else {
@@ -228,31 +119,346 @@ function Invoke-DownloadWithRetry([System.Net.Http.HttpClient]$client, [string]$
}
}
function Validate-KHDBFile([string]$path) {
if (-not (Test-Path $path)) { throw "Validation failed: $path not found" }
$lines = Get-Content -Path $path -Encoding UTF8
if (-not $lines -or $lines.Count -eq 0) { throw 'Validation failed: file is empty.' }
$regex = '^[0-9A-Fa-f]{32}$'
$invalid = $lines | Where-Object { $_ -notmatch $regex }
if ($invalid.Count -gt 0) {
throw ("Validation failed: {0} invalid lines detected." -f $invalid.Count)
function Invoke-S3HttpDownloadWithRetry {
param(
[string]$EndpointUrl,
[string]$Bucket,
[string]$Key,
[string]$TargetPath,
[string]$Region,
[string]$AccessKeyId,
[string]$SecretAccessKey,
[bool]$ForcePathStyle,
[string]$Activity
)
Add-Type -AssemblyName System.Net.Http -ErrorAction SilentlyContinue
[System.Net.Http.HttpClient]$client = [System.Net.Http.HttpClient]::new()
$retries = 5
$delay = 2
try {
for ($attempt = 0; $attempt -lt $retries; $attempt++) {
$request = $null
try {
$uri = BuildS3Uri -endpointUrl $EndpointUrl -bucket $Bucket -key $Key -forcePathStyle $ForcePathStyle
$payloadHash = (Get-HashHex (Get-Bytes ''))
$headers = BuildAuthHeaders -method 'GET' -uri $uri -region $Region -accessKey $AccessKeyId -secretKey $SecretAccessKey -payloadHash $payloadHash
$request = [System.Net.Http.HttpRequestMessage]::new([System.Net.Http.HttpMethod]::Get, $uri)
foreach ($kvp in $headers.GetEnumerator()) {
$request.Headers.TryAddWithoutValidation($kvp.Key, $kvp.Value) | Out-Null
}
$response = $client.SendAsync($request, [System.Net.Http.HttpCompletionOption]::ResponseHeadersRead).GetAwaiter().GetResult()
$null = $response.EnsureSuccessStatusCode()
$totalBytes = $response.Content.Headers.ContentLength
$stream = $response.Content.ReadAsStreamAsync().Result
$fs = [System.IO.File]::Create($TargetPath)
try {
$buffer = New-Object byte[] 8192
$totalRead = 0
while (($read = $stream.Read($buffer, 0, $buffer.Length)) -gt 0) {
$fs.Write($buffer, 0, $read)
$totalRead += $read
if ($totalBytes) {
$pct = ($totalRead * 100.0) / $totalBytes
Write-Progress -Activity $Activity -Status ("{0:N2}% Complete" -f $pct) -PercentComplete $pct
} else {
Write-Progress -Activity $Activity -Status ("Downloaded {0:N0} bytes" -f $totalRead) -PercentComplete 0
}
}
} finally {
$fs.Close()
$stream.Close()
}
if ($response) { $response.Dispose() }
Write-Progress -Activity $Activity -Completed -Status 'Completed'
return
} catch {
if ($attempt -lt ($retries - 1)) {
Write-Warning "Download of '$Key' failed (attempt $($attempt + 1)/$retries): $($_.Exception.Message). Retrying in ${delay}s..."
Start-Sleep -Seconds $delay
$delay = [Math]::Min($delay * 2, 30)
} else {
throw
}
} finally {
if ($request) { $request.Dispose() }
}
}
} finally {
$client.Dispose()
}
}
function Get-FileSha256Lower {
param([string]$Path)
if (-not (Test-Path -LiteralPath $Path)) { throw "File not found: $Path" }
return (Get-FileHash -Path $Path -Algorithm SHA256).Hash.ToLowerInvariant()
}
function Ensure-Directory {
param([string]$Path)
if ([string]::IsNullOrWhiteSpace($Path)) { return }
if (-not (Test-Path -LiteralPath $Path)) {
New-Item -Path $Path -ItemType Directory -Force | Out-Null
}
}
function Get-BooleanSetting {
param(
[string]$Value,
[bool]$Default = $false
)
if ([string]::IsNullOrWhiteSpace($Value)) { return $Default }
$parsed = $Default
if ([System.Boolean]::TryParse($Value, [ref]$parsed)) { return $parsed }
return $Default
}
function Get-RelativePath {
param(
[string]$BasePath,
[string]$FullPath
)
$base = (Resolve-Path -LiteralPath $BasePath).ProviderPath
$full = (Resolve-Path -LiteralPath $FullPath).ProviderPath
if (-not $base.EndsWith([System.IO.Path]::DirectorySeparatorChar)) {
$base = $base + [System.IO.Path]::DirectorySeparatorChar
}
$baseUri = New-Object System.Uri($base, [System.UriKind]::Absolute)
$fullUri = New-Object System.Uri($full, [System.UriKind]::Absolute)
$relativeUri = $baseUri.MakeRelativeUri($fullUri)
$relativePath = [System.Uri]::UnescapeDataString($relativeUri.ToString())
return $relativePath.Replace('/', [System.IO.Path]::DirectorySeparatorChar)
}
function Load-Manifest {
param([string]$Path)
$raw = Get-Content -LiteralPath $Path -Encoding UTF8 -Raw
return $raw | ConvertFrom-Json
}
function Validate-Manifest {
param([psobject]$Manifest)
if (-not $Manifest) { throw 'Manifest is empty or invalid JSON.' }
if (-not $Manifest.shards) { throw 'Manifest does not contain a shards collection.' }
$seen = New-Object 'System.Collections.Generic.HashSet[string]' ([System.StringComparer]::OrdinalIgnoreCase)
foreach ($entry in $Manifest.shards) {
if (-not $entry) { throw 'Manifest contains null shard entries.' }
$name = [string]$entry.name
if ([string]::IsNullOrWhiteSpace($name)) { throw 'Manifest shard entry is missing name.' }
$hash = [string]$entry.sha256
if ([string]::IsNullOrWhiteSpace($hash) -or $hash.Length -ne 64) { throw "Manifest shard '$name' is missing a valid sha256." }
$sizeValue = [string]$entry.size
$sizeParsed = 0L
if (-not [long]::TryParse($sizeValue, [ref]$sizeParsed) -or $sizeParsed -lt 0) {
throw "Manifest shard '$name' has invalid size."
}
if (-not $seen.Add($name)) { throw "Manifest contains duplicate shard name '$name'." }
}
if ($Manifest.shardSize -and [int]$Manifest.shardSize -ne 2) {
throw "Manifest shardSize $($Manifest.shardSize) is not supported. Expected shardSize 2."
}
}
function Convert-KHDBLineToHash {
param(
[Parameter(Mandatory)][string]$Line,
[string]$SourceName,
[int]$LineNumber
)
$trimmed = $Line.Trim()
if ($trimmed.Length -eq 0) { return $null }
if ($trimmed -notmatch '^[0-9A-Fa-f]{32}(:\d+)?$') {
throw ("Invalid KHDB content in '{0}' at line {1}: '{2}'." -f $SourceName, $LineNumber, $trimmed)
}
return ($trimmed.Split(':', 2)[0]).ToUpperInvariant()
}
function Merge-ShardsToFile {
param(
[psobject]$Manifest,
[string]$ShardsRoot,
[string]$TargetPath
)
$encoding = New-Object System.Text.UTF8Encoding($false)
$writer = New-Object System.IO.StreamWriter($TargetPath, $false, $encoding)
$previousHash = $null
try {
foreach ($entry in ($Manifest.shards | Sort-Object name)) {
$relative = [string]$entry.name
$shardPath = Join-Path -Path $ShardsRoot -ChildPath $relative
if (-not (Test-Path -LiteralPath $shardPath)) {
throw "Missing shard on disk: $relative"
}
$reader = New-Object System.IO.StreamReader($shardPath, [System.Text.Encoding]::UTF8, $true)
$lineNumber = 0
try {
while (($line = $reader.ReadLine()) -ne $null) {
$lineNumber++
$normalizedHash = Convert-KHDBLineToHash -Line $line -SourceName $relative -LineNumber $lineNumber
if ($null -eq $normalizedHash) { continue }
if ($previousHash -and $normalizedHash -lt $previousHash) {
throw "Shard merge would produce an unsorted KHDB file at '$relative' line $lineNumber."
}
if ($normalizedHash -eq $previousHash) { continue }
$writer.WriteLine($normalizedHash)
$previousHash = $normalizedHash
}
} finally {
$reader.Dispose()
}
}
} finally {
$writer.Dispose()
}
}
function Validate-KHDBFile {
param([string]$Path)
if (-not (Test-Path -LiteralPath $Path)) { throw "Validation failed: $Path not found." }
$regex = '^[0-9A-Fa-f]{32}$'
$lineNumber = 0
$previous = $null
$duplicates = 0
$reader = New-Object System.IO.StreamReader($Path, [System.Text.Encoding]::UTF8, $true)
try {
while (($line = $reader.ReadLine()) -ne $null) {
$lineNumber++
$trimmed = $line.Trim()
if ($trimmed.Length -eq 0) { continue }
if ($trimmed -notmatch $regex) {
throw "Validation failed: unexpected format at line $lineNumber ('$trimmed')."
}
$normalized = $trimmed.ToUpperInvariant()
if ($previous -and $normalized -lt $previous) {
Write-Warning ("Validation warning: line {0} is out of order." -f $lineNumber)
}
if ($normalized -eq $previous) { $duplicates++ }
$previous = $normalized
}
} finally {
$reader.Dispose()
}
if ($lineNumber -eq 0) { throw 'Validation failed: KHDB file is empty.' }
if ($duplicates -gt 0) {
Write-Warning ("Validation warning: detected {0} duplicate hash entries (file remains unchanged)." -f $duplicates)
}
}
function Remove-EmptyDirectories {
param([string]$Root)
if (-not (Test-Path -LiteralPath $Root)) { return }
Get-ChildItem -LiteralPath $Root -Directory -Recurse | Sort-Object FullName -Descending | ForEach-Object {
$childItems = Get-ChildItem -LiteralPath $_.FullName -Force
if (-not $childItems) {
Remove-Item -LiteralPath $_.FullName -Force
}
}
# Deduplicate and normalize line endings
$unique = $lines | ForEach-Object { $_.Trim() } | Where-Object { $_ } | Sort-Object -Unique
Set-Content -Path $path -Value $unique -Encoding ASCII
}
function Update-KHDB {
param(
[ValidateRange(1, 64)]
[int]$MaxParallelTransfers = 5
)
Start-UpdateTranscript -BasePath $scriptRoot
try {
$settings = Read-ElysiumSettings
$settings = Read-ElysiumSettings -ScriptRoot $scriptRoot
$installPath = Get-InstallationPath $settings
if (-not (Test-Path $installPath)) { New-Item -Path $installPath -ItemType Directory -Force | Out-Null }
Ensure-Directory $installPath
$psSupportsParallel = ($PSVersionTable.PSVersion.Major -ge 7)
$effectiveParallelTransfers = if ($MaxParallelTransfers -lt 1) { 1 } else { [int]$MaxParallelTransfers }
$parallelDownloadsEnabled = $psSupportsParallel -and $effectiveParallelTransfers -gt 1
if (-not $psSupportsParallel -and $effectiveParallelTransfers -gt 1) {
Write-Verbose "Parallel transfers requested but PowerShell $($PSVersionTable.PSVersion) does not support ForEach-Object -Parallel; using serial downloads."
}
$parallelAzureDownloadHelpers = $null
$parallelAzureDownloadHelperList = @()
$parallelS3DownloadHelpers = $null
$parallelS3DownloadHelperList = @()
if ($parallelDownloadsEnabled) {
$parallelAzureDownloadHelpers = @{
'Build-BlobUri' = Get-FunctionDefinitionText 'Build-BlobUri'
'Invoke-DownloadWithRetry' = Get-FunctionDefinitionText 'Invoke-DownloadWithRetry'
'New-HttpClient' = Get-FunctionDefinitionText 'New-HttpClient'
'Get-FileSha256Lower' = Get-FunctionDefinitionText 'Get-FileSha256Lower'
}
$parallelAzureDownloadHelperList = $parallelAzureDownloadHelpers.GetEnumerator() | ForEach-Object {
[pscustomobject]@{ Name = $_.Key; Definition = $_.Value }
}
$parallelS3DownloadHelpers = @{}
@(
'Get-Bytes',
'Get-HashHex',
'HmacSha256',
'ToHex',
'GetSignatureKey',
'UriEncode',
'BuildCanonicalPath',
'BuildAuthHeaders',
'BuildS3Uri',
'Invoke-S3HttpDownloadWithRetry',
'Get-FileSha256Lower'
) | ForEach-Object {
$parallelS3DownloadHelpers[$_] = Get-FunctionDefinitionText $_
}
$parallelS3DownloadHelperList = $parallelS3DownloadHelpers.GetEnumerator() | ForEach-Object {
[pscustomobject]@{ Name = $_.Key; Definition = $_.Value }
}
}
$storageProvider = $settings['StorageProvider']
if ([string]::IsNullOrWhiteSpace($storageProvider)) { $storageProvider = 'Azure' }
$client = $null
$manifestBlobPath = $settings['KhdbManifestPath']
if ([string]::IsNullOrWhiteSpace($manifestBlobPath)) { $manifestBlobPath = 'khdb/manifest.json' }
$remoteShardPrefix = $settings['KhdbShardPrefix']
if ([string]::IsNullOrWhiteSpace($remoteShardPrefix)) { $remoteShardPrefix = 'khdb/shards' }
$localShardDirName = $settings['KhdbLocalShardDir']
if ([string]::IsNullOrWhiteSpace($localShardDirName)) { $localShardDirName = 'khdb-shards' }
$localShardRoot = Join-Path -Path $installPath -ChildPath $localShardDirName
Ensure-Directory $localShardRoot
$localManifestPath = Join-Path -Path $installPath -ChildPath 'khdb-manifest.json'
$tmpDir = New-Item -ItemType Directory -Path ([System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), "elysium-khdb-" + [System.Guid]::NewGuid())) -Force
$manifestTempPath = Join-Path -Path $tmpDir.FullName -ChildPath 'manifest.json'
$downloadTempRoot = Join-Path -Path $tmpDir.FullName -ChildPath 'shards'
Ensure-Directory $downloadTempRoot
Write-Host "Fetching manifest ($manifestBlobPath) from $storageProvider storage..."
$s3Bucket = $null
$s3EndpointUrl = $null
$s3Region = $null
$s3AK = $null
$s3SK = $null
$forcePathStyle = $true
$s3UseAwsTools = $false
$storageAccountName = $null
$containerName = $null
$sasToken = $null
if ($storageProvider -ieq 'S3') {
$s3Bucket = $settings['s3BucketName']
$s3EndpointUrl = $settings['s3EndpointUrl']
$s3Region = $settings['s3Region']
@@ -260,66 +466,314 @@ function Update-KHDB {
$s3SK = $settings['s3SecretAccessKey']
$s3Force = $settings['s3ForcePathStyle']
$s3UseAwsTools = $settings['s3UseAwsTools']
try { $s3Force = [System.Convert]::ToBoolean($s3Force) } catch { $s3Force = $true }
try { $s3UseAwsTools = [System.Convert]::ToBoolean($s3UseAwsTools) } catch { $s3UseAwsTools = $false }
$tmpDir = New-Item -ItemType Directory -Path ([System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), "elysium-khdb-" + [System.Guid]::NewGuid())) -Force
$zipPath = Join-Path -Path $tmpDir.FullName -ChildPath 'khdb.txt.zip'
$extractDir = Join-Path -Path $tmpDir.FullName -ChildPath 'extract'
New-Item -ItemType Directory -Path $extractDir -Force | Out-Null
if ($storageProvider -ieq 'S3') {
if ([string]::IsNullOrWhiteSpace($s3Bucket)) { throw 's3BucketName is missing or empty.' }
if ([string]::IsNullOrWhiteSpace($s3AK) -or [string]::IsNullOrWhiteSpace($s3SK)) { throw 's3AccessKeyId / s3SecretAccessKey missing or empty.' }
if ([string]::IsNullOrWhiteSpace($s3EndpointUrl)) { throw 's3EndpointUrl is required for S3-compatible storage.' }
$forcePathStyle = Get-BooleanSetting -Value $s3Force -Default $true
try { $s3UseAwsTools = [System.Convert]::ToBoolean($s3UseAwsTools) } catch { $s3UseAwsTools = $false }
if ($parallelDownloadsEnabled -and $s3UseAwsTools) {
Write-Warning 'Parallel shard downloads require the SigV4 HTTP path; disabling AWS Tools mode for this run.'
$s3UseAwsTools = $false
}
$downloadKey = Combine-StoragePath -Prefix $null -Name $manifestBlobPath
if ($s3UseAwsTools) {
try {
$s3Client = New-S3Client -EndpointUrl $s3EndpointUrl -Region $s3Region -AccessKeyId $s3AK -SecretAccessKey $s3SK -ForcePathStyle:$s3Force
Write-Host "Downloading KHDB from S3-compatible storage (AWS Tools)..."
# Use AWS SDK stream method into file for progress parity
$req = New-Object Amazon.S3.Model.GetObjectRequest -Property @{ BucketName = $s3Bucket; Key = 'khdb.txt.zip' }
$resp = $s3Client.GetObject($req)
try { $resp.WriteResponseStreamToFile($zipPath, $true) } finally { $resp.Dispose() }
$client = New-S3Client -EndpointUrl $s3EndpointUrl -Region $s3Region -AccessKeyId $s3AK -SecretAccessKey $s3SK -ForcePathStyle:$forcePathStyle
$req = New-Object Amazon.S3.Model.GetObjectRequest -Property @{ BucketName = $s3Bucket; Key = $downloadKey }
$resp = $client.GetObject($req)
try { $resp.WriteResponseStreamToFile($manifestTempPath, $true) } finally { $resp.Dispose() }
} catch {
Write-Warning "AWS Tools path failed or not available. Falling back to native HTTP (SigV4). Details: $($_.Exception.Message)"
Write-Host "Downloading KHDB from S3-compatible storage..."
Invoke-S3HttpDownloadWithRetry -endpointUrl $s3EndpointUrl -bucket $s3Bucket -key 'khdb.txt.zip' -targetPath $zipPath -region $s3Region -ak $s3AK -sk $s3SK -forcePathStyle:$s3Force
Write-Warning "AWS Tools download failed for manifest: $($_.Exception.Message). Falling back to SigV4 HTTP."
Invoke-S3HttpDownloadWithRetry -EndpointUrl $s3EndpointUrl -Bucket $s3Bucket -Key $downloadKey -TargetPath $manifestTempPath -Region $s3Region -AccessKeyId $s3AK -SecretAccessKey $s3SK -ForcePathStyle:$forcePathStyle -Activity 'Downloading manifest'
}
} else {
Write-Host "Downloading KHDB from S3-compatible storage..."
Invoke-S3HttpDownloadWithRetry -endpointUrl $s3EndpointUrl -bucket $s3Bucket -key 'khdb.txt.zip' -targetPath $zipPath -region $s3Region -ak $s3AK -sk $s3SK -forcePathStyle:$s3Force
Invoke-S3HttpDownloadWithRetry -EndpointUrl $s3EndpointUrl -Bucket $s3Bucket -Key $downloadKey -TargetPath $manifestTempPath -Region $s3Region -AccessKeyId $s3AK -SecretAccessKey $s3SK -ForcePathStyle:$forcePathStyle -Activity 'Downloading manifest'
}
} else {
$storageAccountName = $settings['storageAccountName']
$containerName = $settings['containerName']
$sasToken = $settings['sasToken']
$uri = Build-BlobUri -account $storageAccountName -container $containerName -sas $sasToken
$client = New-HttpClient
Write-Host "Downloading KHDB from Azure Blob Storage..."
Invoke-DownloadWithRetry -client $client -uri $uri -targetPath $zipPath
try {
$uri = Build-BlobUri -Account $storageAccountName -Container $containerName -Sas $sasToken -BlobName $manifestBlobPath
Invoke-DownloadWithRetry -Client $client -Uri $uri -TargetPath $manifestTempPath -Activity 'Downloading manifest'
} finally {
if ($client) { $client.Dispose() }
}
}
Write-Host "Download completed. Extracting archive..."
Expand-Archive -Path $zipPath -DestinationPath $extractDir -Force
$manifest = Load-Manifest -Path $manifestTempPath
Validate-Manifest -Manifest $manifest
if ($manifest.shardPrefix) {
$remoteShardPrefix = [string]$manifest.shardPrefix
Write-Verbose "Using shard prefix from manifest: $remoteShardPrefix"
}
$extractedKHDB = Get-ChildItem -Path $extractDir -Recurse -Filter 'khdb.txt' | Select-Object -First 1
if (-not $extractedKHDB) { throw 'Extracted archive does not contain khdb.txt.' }
$remoteShardPrefix = $remoteShardPrefix.Replace('\', '/')
# Validate content
Validate-KHDBFile -path $extractedKHDB.FullName
Write-Host ("Manifest downloaded. Found {0} shard(s)." -f $manifest.shards.Count)
if ($manifest.version) {
Write-Host ("Remote version: {0}" -f $manifest.version)
}
# Compute target path and backup
$targetKHDB = Join-Path -Path $installPath -ChildPath 'khdb.txt'
if (Test-Path $targetKHDB) {
$localManifest = $null
if (Test-Path -LiteralPath $localManifestPath) {
try {
$localManifest = Load-Manifest -Path $localManifestPath
} catch {
Write-Warning ("Failed to parse existing manifest. Full refresh will occur: {0}" -f $_.Exception.Message)
}
}
$localManifestMap = @{}
if ($localManifest -and $localManifest.shards) {
foreach ($entry in $localManifest.shards) {
if ($entry.name) { $localManifestMap[$entry.name] = $entry }
}
}
$downloadQueue = [System.Collections.ArrayList]::new()
$remoteNameSet = New-Object 'System.Collections.Generic.HashSet[string]' ([System.StringComparer]::OrdinalIgnoreCase)
foreach ($entry in $manifest.shards) {
$name = [string]$entry.name
[void]$remoteNameSet.Add($name)
$expectedHash = ([string]$entry.sha256).ToLowerInvariant()
$expectedSize = 0L
if (-not [long]::TryParse([string]$entry.size, [ref]$expectedSize)) {
throw "Cannot parse size for shard '$name'."
}
$localPath = Join-Path -Path $localShardRoot -ChildPath $name
$needsDownload = $true
if (Test-Path -LiteralPath $localPath) {
$localInfo = Get-Item -LiteralPath $localPath
if ($localInfo.Length -eq $expectedSize) {
$localManifestEntry = $null
if ($localManifestMap.ContainsKey($name)) {
$localManifestEntry = $localManifestMap[$name]
}
if ($localManifestEntry -and ([string]$localManifestEntry.sha256).ToLowerInvariant() -eq $expectedHash) {
$needsDownload = $false
} else {
$localHash = Get-FileSha256Lower -Path $localPath
if ($localHash -eq $expectedHash) { $needsDownload = $false }
}
}
}
if ($needsDownload) {
[void]$downloadQueue.Add([pscustomobject]@{
Name = $name
Sha256 = $expectedHash
Size = $expectedSize
})
}
}
if ($downloadQueue.Count -gt 0) {
Write-Host ("{0} shard(s) require download or refresh." -f $downloadQueue.Count)
} else {
Write-Host 'All shards already up to date; verifying manifest and combined file.'
}
$storageClient = $null
$storageHttpClient = $null
$isS3 = ($storageProvider -ieq 'S3')
try {
if ($isS3) {
$s3Bucket = $settings['s3BucketName']
$s3EndpointUrl = $settings['s3EndpointUrl']
$s3Region = $settings['s3Region']
$s3AK = $settings['s3AccessKeyId']
$s3SK = $settings['s3SecretAccessKey']
$s3Force = $settings['s3ForcePathStyle']
$forcePathStyle = Get-BooleanSetting -Value $s3Force -Default $true
$useAwsTools = $settings['s3UseAwsTools']
try { $useAwsTools = [System.Convert]::ToBoolean($useAwsTools) } catch { $useAwsTools = $false }
if ($downloadQueue.Count -gt 0 -and -not $parallelDownloadsEnabled) {
if ($useAwsTools) {
$storageClient = New-S3Client -EndpointUrl $s3EndpointUrl -Region $s3Region -AccessKeyId $s3AK -SecretAccessKey $s3SK -ForcePathStyle:$forcePathStyle
}
$storageHttpClient = @{
Endpoint = $s3EndpointUrl
Bucket = $s3Bucket
Region = $s3Region
AccessKey = $s3AK
SecretKey = $s3SK
ForcePath = $forcePathStyle
}
}
} else {
if ($downloadQueue.Count -gt 0 -and -not $parallelDownloadsEnabled) {
$storageHttpClient = New-HttpClient
}
}
if ($parallelDownloadsEnabled -and $downloadQueue.Count -gt 0) {
Write-Host ("Downloading shards with up to {0} concurrent transfer(s)..." -f $effectiveParallelTransfers)
$remotePrefixForParallel = if ([string]::IsNullOrWhiteSpace($remoteShardPrefix)) { $null } else { $remoteShardPrefix.Replace('\', '/').Trim('/') }
$parallelDownloadHelpers = if ($isS3) { $parallelS3DownloadHelperList } else { $parallelAzureDownloadHelperList }
$downloadQueue.ToArray() | ForEach-Object -Parallel {
$entry = $PSItem
try {
if ($null -eq $entry) { return }
foreach ($helper in $using:parallelDownloadHelpers) {
if (-not (Get-Command $helper.Name -ErrorAction SilentlyContinue)) {
Invoke-Expression $helper.Definition
}
}
$name = [string]$entry.Name
if ([string]::IsNullOrWhiteSpace($name)) {
throw "Parallel shard entry missing name: $(ConvertTo-Json $entry -Compress)"
}
$expectedHash = ([string]$entry.Sha256).ToLowerInvariant()
$expectedSize = [long]$entry.Size
$remoteKey = $name.Replace('\', '/').TrimStart('/')
if (-not [string]::IsNullOrWhiteSpace($using:remotePrefixForParallel)) {
$remoteKey = $using:remotePrefixForParallel + '/' + $remoteKey
}
$stagingPath = Join-Path -Path $using:downloadTempRoot -ChildPath $name
$stagingParent = Split-Path -Path $stagingPath -Parent
if ($stagingParent -and -not (Test-Path -LiteralPath $stagingParent)) {
[System.IO.Directory]::CreateDirectory($stagingParent) | Out-Null
}
$activity = ("Downloading shard: {0}" -f $name)
if ($using:isS3) {
Invoke-S3HttpDownloadWithRetry -EndpointUrl $using:s3EndpointUrl -Bucket $using:s3Bucket -Key $remoteKey -TargetPath $stagingPath -Region $using:s3Region -AccessKeyId $using:s3AK -SecretAccessKey $using:s3SK -ForcePathStyle:$using:forcePathStyle -Activity $activity
} else {
$client = $null
try {
$client = New-HttpClient
$blobUri = Build-BlobUri -Account $using:storageAccountName -Container $using:containerName -Sas $using:sasToken -BlobName $remoteKey
Invoke-DownloadWithRetry -Client $client -Uri $blobUri -TargetPath $stagingPath -Activity $activity
} finally {
if ($client) { $client.Dispose() }
}
}
$downloadInfo = Get-Item -LiteralPath $stagingPath
if ($downloadInfo.Length -ne $expectedSize) {
throw "Shard '$name' size mismatch. Expected $expectedSize bytes, got $($downloadInfo.Length)."
}
$actualHash = Get-FileSha256Lower -Path $stagingPath
if ($actualHash -ne $expectedHash) {
throw "Shard '$name' checksum mismatch. Expected $expectedHash, got $actualHash."
}
$finalPath = Join-Path -Path $using:localShardRoot -ChildPath $name
$parentDir = Split-Path -Path $finalPath -Parent
if ($parentDir -and -not (Test-Path -LiteralPath $parentDir)) {
[System.IO.Directory]::CreateDirectory($parentDir) | Out-Null
}
Move-Item -LiteralPath $stagingPath -Destination $finalPath -Force
Write-Host ("Shard '{0}' updated." -f $name)
} catch {
throw ("Shard '{0}': {1}" -f $entry.name, $_.Exception.Message)
}
} -ThrottleLimit $effectiveParallelTransfers
} else {
$downloadIndex = 0
foreach ($entry in $downloadQueue.ToArray()) {
$downloadIndex++
if ($null -eq $entry) { continue }
$name = [string]$entry.Name
if ([string]::IsNullOrWhiteSpace($name)) {
throw "Shard entry missing name: $(ConvertTo-Json $entry -Compress)"
}
$expectedHash = ([string]$entry.Sha256).ToLowerInvariant()
$expectedSize = [long]$entry.Size
$activity = "Downloading shard $downloadIndex/$($downloadQueue.Count): $name"
$remoteKey = Combine-StoragePath -Prefix $remoteShardPrefix -Name $name
$stagingPath = Join-Path -Path $downloadTempRoot -ChildPath $name
Ensure-Directory (Split-Path -Path $stagingPath -Parent)
if ($isS3) {
if ($storageClient) {
try {
$request = New-Object Amazon.S3.Model.GetObjectRequest -Property @{ BucketName = $s3Bucket; Key = $remoteKey }
$response = $storageClient.GetObject($request)
try { $response.WriteResponseStreamToFile($stagingPath, $true) } finally { $response.Dispose() }
} catch {
Write-Warning "AWS Tools download failed for shard '$name': $($_.Exception.Message). Falling back to SigV4 HTTP."
Invoke-S3HttpDownloadWithRetry -EndpointUrl $storageHttpClient.Endpoint -Bucket $storageHttpClient.Bucket -Key $remoteKey -TargetPath $stagingPath -Region $storageHttpClient.Region -AccessKeyId $storageHttpClient.AccessKey -SecretAccessKey $storageHttpClient.SecretKey -ForcePathStyle:$storageHttpClient.ForcePath -Activity $activity
}
} else {
Invoke-S3HttpDownloadWithRetry -EndpointUrl $storageHttpClient.Endpoint -Bucket $storageHttpClient.Bucket -Key $remoteKey -TargetPath $stagingPath -Region $storageHttpClient.Region -AccessKeyId $storageHttpClient.AccessKey -SecretAccessKey $storageHttpClient.SecretKey -ForcePathStyle:$storageHttpClient.ForcePath -Activity $activity
}
} else {
$blobUri = Build-BlobUri -Account $storageAccountName -Container $containerName -Sas $sasToken -BlobName $remoteKey
Invoke-DownloadWithRetry -Client $storageHttpClient -Uri $blobUri -TargetPath $stagingPath -Activity $activity
}
$downloadInfo = Get-Item -LiteralPath $stagingPath
if ($downloadInfo.Length -ne $expectedSize) {
throw "Shard '$name' size mismatch. Expected $expectedSize bytes, got $($downloadInfo.Length)."
}
$actualHash = Get-FileSha256Lower -Path $stagingPath
if ($actualHash -ne $expectedHash) {
throw "Shard '$name' checksum mismatch. Expected $expectedHash, got $actualHash."
}
$finalPath = Join-Path -Path $localShardRoot -ChildPath $name
Ensure-Directory (Split-Path -Path $finalPath -Parent)
Move-Item -LiteralPath $stagingPath -Destination $finalPath -Force
Write-Host ("Shard '{0}' updated." -f $name)
}
}
} finally {
if ($storageClient) { $storageClient.Dispose() }
if ($storageHttpClient -is [System.Net.Http.HttpClient]) { $storageHttpClient.Dispose() }
}
$existingShards = @()
if (Test-Path -LiteralPath $localShardRoot) {
$existingShards = Get-ChildItem -LiteralPath $localShardRoot -File -Recurse
}
$removed = 0
foreach ($file in $existingShards) {
$relative = Get-RelativePath -BasePath $localShardRoot -FullPath $file.FullName
if (-not $remoteNameSet.Contains($relative)) {
Remove-Item -LiteralPath $file.FullName -Force
$removed++
}
}
if ($removed -gt 0) {
Write-Host ("Removed {0} stale shard(s)." -f $removed)
Remove-EmptyDirectories -Root $localShardRoot
}
Copy-Item -LiteralPath $manifestTempPath -Destination $localManifestPath -Force
Write-Host ("Manifest saved locally to {0}" -f $localManifestPath)
$khdbName = if ([string]::IsNullOrWhiteSpace($settings['WeakPasswordsDatabase'])) { 'khdb.txt' } else { $settings['WeakPasswordsDatabase'] }
$combinedTarget = Join-Path -Path $installPath -ChildPath $khdbName
$combinedTemp = Join-Path -Path $tmpDir.FullName -ChildPath 'khdb-combined.txt'
Write-Host "Rebuilding combined KHDB file..."
Merge-ShardsToFile -Manifest $manifest -ShardsRoot $localShardRoot -TargetPath $combinedTemp
Validate-KHDBFile -Path $combinedTemp
if (Test-Path -LiteralPath $combinedTarget) {
$ts = Get-Date -Format 'yyyyMMdd-HHmmss'
$backupPath = Join-Path -Path $installPath -ChildPath ("khdb.txt.bak-$ts")
Copy-Item -Path $targetKHDB -Destination $backupPath -Force
Write-Host "Existing KHDB backed up to $backupPath"
$backupPath = Join-Path -Path $installPath -ChildPath ("$khdbName.bak-$ts")
Copy-Item -LiteralPath $combinedTarget -Destination $backupPath -Force
Write-Host ("Existing KHDB backed up to {0}" -f $backupPath)
}
# Atomic-ish replace: move validated file into place
Move-Item -Path $extractedKHDB.FullName -Destination $targetKHDB -Force
Write-Host "KHDB updated at $targetKHDB"
Move-Item -LiteralPath $combinedTemp -Destination $combinedTarget -Force
Write-Host ("KHDB merged file refreshed at {0}" -f $combinedTarget)
Write-Host "KHDB update completed successfully."
} catch {
Write-Error ("KHDB update failed: {0}" -f $_.Exception.Message)
@@ -330,6 +784,5 @@ function Update-KHDB {
}
}
# Execute the update function
Update-KHDB
Write-Host "Script execution completed."
+167
View File
@@ -0,0 +1,167 @@
##################################################
## ____ ___ ____ _____ _ _ _____ _____ ##
## / ___/ _ \| _ \| ____| | \ | | ____|_ _| ##
## | | | | | | |_) | _| | \| | _| | | ##
## | |__| |_| | _ <| |___ _| |\ | |___ | | ##
## \____\__\_\_| \_\_____(_)_| \_|_____| |_| ##
##################################################
## Project: Elysium ##
## File: Update-LithnetStore.ps1 ##
## Version: 2.2.2 ##
## Support: support@cqre.net ##
##################################################
<#
.SYNOPSIS
Populates a Lithnet Password Protection store with compromised passwords and banned words.
.DESCRIPTION
Reads configuration from ElysiumSettings.txt (or a provided settings file), opens the Lithnet
Password Protection store, optionally synchronizes with Have I Been Pwned, imports local NTLM
hash lists, plaintext password lists, and banned-word files.
#>
[CmdletBinding()]
param(
[string]$SettingsPath,
[string[]]$HashFiles,
[string[]]$PlaintextFiles,
[string[]]$BannedWordFiles,
[switch]$SkipHibpSync
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
[string]$commonHelper = Join-Path -Path $PSScriptRoot -ChildPath 'Elysium.Common.ps1'
if (-not (Test-Path -LiteralPath $commonHelper)) { throw "Common helper not found at $commonHelper" }
. $commonHelper
Restart-WithPwshIfAvailable -BoundParameters $PSBoundParameters -UnboundArguments $MyInvocation.UnboundArguments
function Read-KeyValueSettings {
param([string]$Path)
$result = @{}
if (-not (Test-Path -LiteralPath $Path)) { throw "Settings file not found at $Path" }
Get-Content -LiteralPath $Path | ForEach-Object {
$line = $_
if (-not $line) { return }
$trimmed = $line.Trim()
if (-not $trimmed) { return }
if ($trimmed.StartsWith('#')) { return }
$kv = $line -split '=', 2
if ($kv.Count -ne 2) { return }
$key = $kv[0].Trim()
$value = $kv[1].Trim()
if (-not $key) { return }
if ($value.StartsWith("'") -and $value.EndsWith("'") -and $value.Length -ge 2) {
$value = $value.Substring(1, $value.Length - 2)
}
$result[$key] = $value
}
return $result
}
function Get-BooleanSetting {
param(
[string]$Value,
[bool]$Default = $false
)
if ([string]::IsNullOrWhiteSpace($Value)) { return $Default }
try { return [System.Convert]::ToBoolean($Value) } catch { return $Default }
}
function Get-ListFromSetting {
param([string]$Value)
if ([string]::IsNullOrWhiteSpace($Value)) { return @() }
return ($Value -split '[,;]' | ForEach-Object { $_.Trim() } | Where-Object { $_ })
}
function Resolve-ExistingPath {
param([string]$PathValue, [string]$Description)
if ([string]::IsNullOrWhiteSpace($PathValue)) { throw "$Description path was not provided." }
if (-not (Test-Path -LiteralPath $PathValue)) {
throw "$Description not found at '$PathValue'."
}
return (Resolve-Path -LiteralPath $PathValue).Path
}
if (-not $SettingsPath) {
$SettingsPath = Join-Path -Path $PSScriptRoot -ChildPath 'ElysiumSettings.txt'
}
$settings = Read-KeyValueSettings -Path $SettingsPath
$storePathSetting = $settings['LithnetStorePath']
$storePath = Resolve-ExistingPath -PathValue $storePathSetting -Description 'Lithnet store'
$settingsHashSources = Get-ListFromSetting -Value $settings['LithnetHashSources']
$settingsPlainSources = Get-ListFromSetting -Value $settings['LithnetPlaintextSources']
$settingsBannedSources = Get-ListFromSetting -Value $settings['LithnetBannedWordSources']
$hashSourcePaths = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase)
foreach ($path in @($HashFiles) + $settingsHashSources) {
if ([string]::IsNullOrWhiteSpace($path)) { continue }
$resolved = Resolve-ExistingPath -PathValue $path -Description 'Hash list'
[void]$hashSourcePaths.Add($resolved)
}
if ($hashSourcePaths.Count -eq 0) {
$defaultKhdb = Join-Path -Path $PSScriptRoot -ChildPath 'khdb.txt'
if (Test-Path -LiteralPath $defaultKhdb) {
[void]$hashSourcePaths.Add((Resolve-Path -LiteralPath $defaultKhdb).Path)
} else {
throw 'No hash files were provided via parameters or LithnetHashSources, and khdb.txt was not found.'
}
}
$plaintextSourcePaths = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase)
foreach ($path in @($PlaintextFiles) + $settingsPlainSources) {
if ([string]::IsNullOrWhiteSpace($path)) { continue }
$resolved = Resolve-ExistingPath -PathValue $path -Description 'Plaintext password list'
[void]$plaintextSourcePaths.Add($resolved)
}
$bannedWordSourcePaths = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase)
foreach ($path in @($BannedWordFiles) + $settingsBannedSources) {
if ([string]::IsNullOrWhiteSpace($path)) { continue }
$resolved = Resolve-ExistingPath -PathValue $path -Description 'Banned word list'
[void]$bannedWordSourcePaths.Add($resolved)
}
$syncHibp = Get-BooleanSetting -Value $settings['LithnetSyncHibp'] -Default:$false
if ($SkipHibpSync) { $syncHibp = $false }
Write-Host "Importing LithnetPasswordProtection module..."
try {
Import-Module -Name LithnetPasswordProtection -ErrorAction Stop | Out-Null
} catch {
throw "LithnetPasswordProtection module not found: $($_.Exception.Message). Install it from https://github.com/lithnet/password-protection."
}
Write-Host ("Opening Lithnet store at '{0}'..." -f $storePath)
Open-Store -Path $storePath
$storeOpened = $true
try {
if ($syncHibp) {
Write-Host 'Synchronizing compromised hashes from Have I Been Pwned (this can take a while)...'
Sync-HashesFromHibp
}
foreach ($hashFile in ($hashSourcePaths.ToArray() | Sort-Object)) {
Write-Host ("Importing NTLM hash list '{0}'..." -f $hashFile)
Import-CompromisedPasswordHashes -Filename $hashFile
}
foreach ($plainFile in ($plaintextSourcePaths.ToArray() | Sort-Object)) {
Write-Host ("Importing plaintext password list '{0}'..." -f $plainFile)
Import-CompromisedPasswords -Filename $plainFile
}
foreach ($bannedFile in ($bannedWordSourcePaths.ToArray() | Sort-Object)) {
Write-Host ("Importing banned word list '{0}'..." -f $bannedFile)
Import-BannedWords -Filename $bannedFile
}
Write-Host 'Lithnet store update completed successfully.'
} finally {
if ($storeOpened) {
try { Close-Store } catch {}
}
}