Project / Agent & Engineering
PowerShell Reliability Framework
Reliability checks for AI-generated PowerShell.
A portable Agent Skill: Probe → Classify → Rule Check → Propose(-WhatIf) → Verify, with machine-readable rules, positive/negative examples, and a scoring benchmark.
v0.1.0-alpha.4
Overview Project overview
The PowerShell Reliability Framework (PRF) is a portable Agent Skill that enforces a reliability protocol before generating, reviewing, or executing PowerShell.
Motivation Why it exists
When an LLM generates and runs PowerShell directly, common failures are environment assumptions, destructive parameters, quoting and encoding drift, and missing verification. PRF moves these checks up front instead of patching things after the script fails.
Architecture Key architecture
- The entry point is SKILL.md: Probe → Classify(R/W/D/N/C) → Draft → Rule Check → Propose(-WhatIf) → Confirm → Execute → Verify → Report.
- The knowledge base covers object pipelines, PowerShell 5.1/7.x differences, safety rules, enterprise patterns, anti-patterns, and here-string syntax.
- rules/reliability-rules.yaml provides machine-readable rules; examples/ provides paired positive/negative cases; benchmark/cases.json provides scoring cases.
- rules/routing-labels.yaml adds reviewed risk/topic labels for every rule; the L0 gate checks vocabulary, dependencies, corpus order, and the rules_blob binding.
Capabilities Capabilities
- The v0.1 repository contains 109 rules, 110 benchmark cases, 12 positive/negative example pairs, and 6 knowledge documents; 16 rules are resident.
- alpha.3 adds a reviewed routing-label sidecar for every rule, audited benchmark references (329 → 244), KS/KC/AP knowledge IDs, a nine-item rule backlog, and a Windows PowerShell 5.1 CI lane.
- alpha.4 rewrites the SEC-006/009/019/023 bad examples as non-runnable descriptions after an antivirus false positive, refreshes rules_blob, and aligns the safety notice; rules and detection semantics are unchanged.
- L0 package-integrity checking via validate.py and L1 rule/example consistency self-check via rule-selftest.py.
- Portable to hosts that support Agent Skills, with environment-probe scripts for target machines.
Limitations Known limitations
- v0.1 is a Skill package, not yet a static analyzer or sandboxed executor.
- PSScriptAnalyzer integration and custom rules are planned for v0.2; sandboxed execution for v0.3.
- The README states explicitly that no measured effectiveness is claimed before the first double-blind benchmark completes.
- Attack shapes in examples/bad are negative samples only and must not be executed.
- Still alpha: rules, labels, and benchmark references may change before 0.1.0. The L2 model A/B comparison has not run, and the Windows lane does not yet include a byte-level encoding probe.
- Older copies of the rule corpus may still be flagged by antivirus scanners; benchmark and knowledge files can still mention the same cmdlets. Replacing a copy does not guarantee a scanner will stop flagging it.
Validation Testing & validation
- alpha.4 passes L0 (routing labels, knowledge IDs, package integrity), L1 self-test (37 pattern-bearing rules, 0 FAIL), L3 parse (25/25 .ps1), PSScriptAnalyzer on pwsh 7.6.9, and actionlint locally.
- On GitHub Actions, both jobs passed for main @ 6f66946; the Windows job ran on Windows PowerShell 5.1.26100.33648. Routing recall is unchanged at 244/244 and 101/101 with the closure.
- Benchmark case counts and routing recall are repository data, not a claim of real-world effectiveness; the L2 model A/B comparison has not run.