Eigenoid phase implementation history

Federation initiative phase history plus consolidated reference docs.

What this document is

The numbered phases in Part 1 (Phase 0 through Phase 3D) were a single coordinated initiative: enable two operators on the internet -- each running their own SPIRE server -- to federate their agents so they can talk to each other over A2A securely. Phase 3D is the climax: two operators can now run eigenoid federation and pair their stacks without editing YAML by hand. Each phase was implemented by an independent agent on a dedicated branch, who left a PHASE_*_REPORT.md at repo root for the next agent in the chain. Those handoff files, the two associated audit catalogues, the manual acceptance runbook, the pre-Phase-3 repo review, and the parallel OpenClaw feature track are consolidated in Part 1. Part 2 holds the project's operational reference docs (architecture, auth, discovery, federation operator guide, quickstart, testing). For day-to-day orientation while making code changes, see AGENTS.md at repo root.

Contents

Part 1 -- Federation initiative phase history

1. Phase 0 -- Consolidation and foundations
2. Phase 1 -- Real delegation (Ed25519)
3. Phase 1.5 -- Identity binding
4. Phase 2 -- Grants and capabilities
5. Phase 2.1 -- Hardening
6. Phase 3A -- Federation schema
7. Phase 3B -- SPIRE federation activation
8. Phase 3C -- Cross-TD enforcement
9. Phase 3D -- Federation bootstrap UX
10. Audit -- SPIFFE ID usage
11. Audit -- Silent-failure paths
12. Runbook -- Phase 3c manual acceptance
13. Repo review -- gap analysis
14. OpenClaw innovations proposal (parallel track)

Part 2 -- Reference documentation

1. Architecture overview
2. Delegation security model
3. Dual-mode authentication (mTLS + JWT-SVID)
4. Discovery providers
5. SPIRE federation
6. Quick start guide
7. Enterprise governance testing guide

Part 1 -- Federation initiative phase history

A coordinated effort to enable two operators on the internet (each running their own SPIRE server) to federate their agents and call each other over A2A securely. Phase 0 lays foundations; Phase 3D ships the friend-request bootstrap UX.

Phase 0 -- Consolidation and foundations

Status: shipped

Branch	claude/eigenoid-phase-0-consolidation-Gft3g
Scope	Consolidate middleware, introduce typed values, thread identifiers through chain envelopes.
Tests	49 new tests across 7 files

Phase 0 is deliberately non-feature and lands 14 consolidation and forward-compat deliverables. It unifies SimplifiedAgent and BaseAgent middleware and helpers into a shared canon, introduces a typed SpiffeID, per-agent state directories, a KeyStore with atomic writes, an InstanceIdentity discovery primitive, and threads chain_id and version through the audit pipeline. No behavior change; all existing tests pass unchanged.

Deliverables

ID	Description
D1	Consolidate SimplifiedAgent + BaseAgent middleware/helpers into eigenoid/core/auth.py
D2	Forward-compatible DelegationHop.from_dict with metadata["_preserved"]
D3	Unify hop producer schemas + A2A metadata/header symmetry
D4	Typed SpiffeID value object in eigenoid/core/spiffe_id.py
D5	chain_id + version threaded through chain envelopes
D6	schema_version column on audit SQLite
D7	Single source of truth for trust domain via resolve_trust_domain
D8	Per-agent state directories under EIGENOID_HOME
D9	KeyStore with load-or-generate, atomic writes, 0o600 perms
D10	InstanceIdentity + /.well-known/eigenoid-instance.json
D11	Discovery consolidation -- expanded AgentEndpoint, async StaticDiscovery
D12	Multi-TD docker-compose fixture + xfail test
D13	Minimal CI workflow (GitHub Actions, pytest)
D14	Deprecate _sign_hop / _verify_hop_signature

Key files

• eigenoid/core/auth.py -- new, shared middleware (245 LOC)
• eigenoid/core/instance.py -- instance identity (147 LOC)
• eigenoid/core/keystore.py -- load-or-generate keys (148 LOC)
• eigenoid/core/spiffe_id.py -- typed value (94 LOC)
• eigenoid/delegation/chain.py -- chain_id threading, deprecation warnings
• .github/workflows/test.yml -- minimal CI

Env vars introduced

• EIGENOID_HOME -- optional override of default state-dir path

Deviations and known follow-ups

• SimplifiedAgent._AuthMiddleware kept as subclass-like copy (collision risk deferred to Phase 1)
• AgentDiscovery delegates via composition + private event loop, not replacement
• String-sliced SPIFFE IDs still exist in 15 sites -- deferred to Phase 1 for migration
• Deprecated crypto helpers still called by agent + executor -- Phase 1 to gate behind capability-token path
• DEFAULT_TRUST_DOMAIN module-level constant kept for backward compat

Phase 1 -- Real delegation (Ed25519 hops)

Status: shipped

Branch	claude/eigenoid-phase-1-delegation-Xweoo
Scope	Make the delegation chain cryptographically meaningful.
Tests	9 regression tests, 611 total pass
Latency	172 us (1-hop), 503 us (3-hop), 1.2 ms (7-hop)

Phase 1 replaces the SHA-256 placeholder with Ed25519 hop signatures over JCS canonical bytes, wires verification into the request middleware before the handler is invoked, captures Bearer id_token in approval records, and adds an opt-in chain-aware permission mode that intersects role permissions across delegation depth. The signing key persists at <EIGENOID_HOME>/agents/<agent_type>/signing.key.

Deliverables

ID	Description
D1	Per-agent Ed25519 signing key, AgentIdentity helper, signingPublicKeyPem on the agent card
D2	Replace SHA-256 with Ed25519 over JCS canonical bytes; NUL-delimited chain format
D3	Embed signer's PEM on hop as signer_public_key_pem
D4	verify_chain_dicts in middleware pre-handler; 403 + audit on failure; configurable timestamp tolerance
D5	Approval server captures Bearer id_token; _enforce_approval re-validates signature / expiry / issuer + email match
D6	compute_effective_permissions(chain, role_map) -- intersects when chain length > 1; gated by chain_aware_permissions: bool = False
D7	is_cross_domain_boundary(prev, next) helper; chain.cross_domain_hop audit event (logging only)
D8	9 regression tests

Key files

• eigenoid/core/agent_identity.py -- new, Ed25519 signing wrapper (147 LOC)
• eigenoid/core/jcs.py -- new, vendored RFC 8785 canonicalizer (137 LOC)
• eigenoid/delegation/chain.py -- +386 LOC sign/verify rewrite
• eigenoid/permissions/manager.py -- +107 LOC, compute_effective_permissions
• eigenoid/agent.py -- +273 LOC identity boot, sign on outbound, middleware verify

Env vars and flags

• EIGENOID_CHAIN_AWARE_PERMISSIONS (default false)
• Config: chain_timestamp_tolerance_seconds (default 5.0)

Deviations

• JCS vendored to keep dependency count stable
• Signing key separate from instance key (independent rotation schedules)
• _enforce_approval accepts missing approved key (wire compat during Phase 1 to Phase 2 migration)
• Hop signature covers hop identity claims, not the capability field (Phase 2 addition)
• No key rotation policy -- Phase 2 work item

Phase 1.5 -- Identity binding for signed hops

Status: shipped

Branch	claude/identity-binding-delegation-6ckhc
Scope	Prove the public key belongs to the SPIFFE ID the hop claims.
Tests	19 new tests; 28 pass with Phase 1; zero regressions
Overhead	254 us steady-state; ~6.7 ms first-contact TOFU

Phase 1 embeds the signer's public key and verifies Ed25519 against it -- which only proves whoever signed held the matching private key. It does not prove the key belongs to the SPIFFE ID the hop claims. Phase 1.5 adds that cross-reference: a persistent pubkey-pin ledger, pin-on-first-contact with agent-card SPIFFE-ID binding, signed rotation-attestation mechanism, manual-admin pin API, and revocation. All runs in middleware so forged chains never reach handlers.

Deliverables

ID	Description
D1	PubkeyPinStore -- SQLite-backed ledger; pubkey_pins + pin_rotation_history tables; full pin/get/verify/update/revoke API
D2	PubkeyResolver ABC + 4 implementations (AgentCard, Chained, ManualAdmin, Static); TTL cache
D3	IdentityBindingVerifier wired into middleware after Phase 1 signature math; 403 + audit on failure; gated by EIGENOID_IDENTITY_BINDING_ENABLED (default off)
D4	RotationAttestation signed by old key over JCS; rotation store + verifier; /.well-known/pubkey-rotation.json endpoint; eigenoid pubkeys rotate CLI
D5	POST / GET /admin/pubkey-pins + revoke; gated by EIGENOID_ADMIN_EMAILS; manual-admin resolver layered first
D6	Real-keypair forgery test -- attacker with valid Ed25519 key claiming any SPIFFE ID rejected
D7	19 regression tests
D8	examples/testing_capability/phase_1_5/ with 8 scenario modules

Key files

• eigenoid/core/pubkey_pins.py -- pin ledger + schema (416 LOC)
• eigenoid/core/pubkey_resolver.py -- ABC + 4 implementations (435 LOC)
• eigenoid/core/rotation.py -- attestations + store + verifier (513 LOC)
• eigenoid/delegation/identity_binding.py -- per-hop binding check (224 LOC)
• eigenoid/cli/main.py -- eigenoid pubkeys rotate subcommand (+122 LOC)

Env vars and flags

• EIGENOID_IDENTITY_BINDING_ENABLED (default false; opt-in while Phase 1 demos run)
• EIGENOID_PUBKEY_CACHE_TTL (default 60 s)
• EIGENOID_ADMIN_EMAILS (comma-separated allowlist)

Deviations

• Binding check gated off by default to preserve Phase 1 demo compatibility
• Peer-cert SPIFFE-ID binding uses asyncio streams (not httpx) to extract ssl_object.getpeercert()
• BFS rotation-chain walk is order-independent but naive on pathological ordering
• Admin API uses update_pin for create (admins are authoritative)

Phase 2 -- Grants and capabilities

Status: shipped

Branch	claude/eigenoid-phase-2-grants-FGRrq
Scope	"What is this caller allowed to ask for" -- grants + capabilities layer.
Tests	51 (Phase 1/1.5/2 combined); 655 in full suite
Overhead	575 us capability verify (cache hit); 208 us grant store cold hit

Phase 2 introduces Grant (persistent authorization record scoped to issuer + subject + skill), Capability (short-lived Ed25519-JWT minted off a grant), DelegationPolicy enforcement, and the _system.grant_request / _system.grant_request_status built-in skills for the approval workflow. Enforcement is gated by EIGENOID_REQUIRE_GRANT_CITATION (defaults false) so every Phase 0/1/1.5 test passes unchanged. The verifier reuses the Phase 1.5 PubkeyPinStore + PubkeyResolver -- no parallel cache or TOFU path.

Deliverables

ID	Description
D1	Grant, DelegationPolicy, IssuanceContext + constraints_hash
D2	GrantBackend ABC + SqliteGrantBackend with schema_version, indexing, full API
D3	CapabilityIssuer/Verifier ABCs; JwtCapabilityIssuer/Verifier using EdDSA; reuses Phase 1.5 infra
D4	DelegationPolicyEnforcer + match_spiffe_pattern (exact, wildcard, trust-domain)
D5	PendingApprovalStore (approvals.db); four-state lifecycle; condition variables; web UI hook
D6	GrantRequestHandler for _system.grant_request*
D7	ReceivedGrantStore (received_grants.db) indexed by (issuer, skill)
D8	DelegationHop.capability field (base64 on wire); construct_outbound_hop helper
D9	Agent.initialize() bootstrap; register_skill blocks _system.* prefix
D10	_enforce_grant_citation in executor before permissions; admin routes; approval web UI at /approvals; auto-attach + auto-request capabilities
D11	19 regression tests
D12	4 integration tests (1.5 + 2 cooperation)
D13	12 demo scenarios

Key files

• eigenoid/grants/ -- new package, 6 files, 983 LOC (models, store, request handler, received store)
• eigenoid/capabilities/ -- new package, 3 files, 610 LOC (ABCs, policy DSL, JWT impl)
• eigenoid/approval/web_ui/ -- Starlette sub-app + templates + JS (276 LOC)
• eigenoid/approval/pending_store.py -- SQLite + condition variables (401 LOC)
• eigenoid/agent.py -- +547 LOC bootstrap, enforcement, auto-request, admin routes

Env vars and flags

• EIGENOID_REQUIRE_GRANT_CITATION (default false)
• EIGENOID_AUTO_REQUEST_GRANTS (default false)
• EIGENOID_APPROVAL_MODE (default terminal; also web, both)
• EIGENOID_CAPABILITY_CACHE_TTL (default 30.0)
• EIGENOID_GRANT_POLL_TIMEOUT_SECONDS (default 60.0)
• EIGENOID_GRANT_SYNC_WAIT_SECONDS (default 0)
• EIGENOID_OIDC_ISSUER, EIGENOID_OIDC_CLIENT_ID (required for approval auth)
• EIGENOID_ADMIN_EMAILS (optional allowlist)

Deviations

• _system. prefix enforced at register_skill, not invoke time (early fail, not hot-path)
• Capabilities signed with Phase 1 hop-signing key (no separate cap-signing key; reduces operator burden)
• Capability expiry clamped to grant expiry
• Constraints hash = SHA-256 of sorted JSON (not JCS) to avoid cross-implementation coupling
• Capability on last hop only, matches Phase 1.5 hop-signing model
• Framework default false; demo YAML defaults true

Phase 2.1 -- Hardening + pre-Phase-3 cleanup

Status: shipped

Branch	claude/phase-2.1-hardening-lpzWJ
Scope	Three Phase 2 follow-ups + SPIFFE-ID + silent-failure cleanup pass.
Tests	36 new Phase 2.1 tests; 655 pre-existing pass; zero regressions

Phase 2.1 closes the browser approval flow (PKCE + state cookie + OIDC session), fixes the audit-DB default path under ~/.eigenoid/agents/<agent_type>/, resolves URL targets to real SPIFFE IDs via agent cards instead of string fabrication, adopts SpiffeID typed values at four high-impact sites, catalogs 49 SPIFFE-ID usage sites, fixes three high-priority silent-failure swallows, and establishes chain_id at the request boundary so originator audit events correlate with downstream.

Deliverables

ID	Description
D1	Browser cookie-session OIDC for /approvals/ (PKCE, state cookie, signed session cookie, redirect on 401)
D2	Audit DB default ~/.eigenoid/agents/<agent_type>/audit.db; startup writability self-check
D3	_resolve_target_spiffe_id resolves URLs via agent card or fails fast (no fabrication)
D4	SpiffeID typed-value adoption (4 high-impact sites); AUDIT_SPIFFE_ID_USAGE.md catalog
D5	Regression tests for three Phase 2 fix commits (audience-equal-issuer, audience-equal-subject, self-issued cap)
D6	Silent-failure audit + 3 high-priority swallows fixed; SILENT_FAILURE_AUDIT.md
D7	chain_id established at request boundary (inbound reuse, autonomous mint); SDK bridge forwarding
D7+	Configurable grant-request poll cadence (default backed off from 0.25 s to 1.0 s)

Key files

• eigenoid/agent.py -- +397 / -21 LOC, web UI auth routes, chain_id boundary, SPIFFE resolution
• eigenoid/approval/web_ui/ -- rewrite + new session.py (168 LOC), OIDC cookie flow
• eigenoid/core/pubkey_resolver.py -- +58 LOC, fetch_card_spiffe_id bootstrap path
• eigenoid/client.py -- +20 / -5 LOC, OIDC token cache split (ImportError vs other)
• eigenoid/a2a/context.py -- chain_id forwarding in SpiffeCallContextBuilder

Env vars and flags

• EIGENOID_OIDC_REDIRECT_URI (optional, for reverse-proxy deployments)
• EIGENOID_GRANT_POLL_INTERVAL_SECONDS (default 1.0; previously hardcoded 0.25)

Deviations

• OIDC flow built from scratch (no existing pattern to match against)
• D5 regression tests are in-process (bug location was JWT/cap path, not network)
• _resolve_target_spiffe_id falls back to legacy synthesis when no resolver configured (test backward compat, logs WARNING)
• D2 self-check uses filesystem probe instead of emit+flush (avoids cross-loop audit drop)

Phase 3A -- Federation schema plumbing (no behavioral change)

Status: shipped

Branch	(merged as 94e387f)
Scope	Add federation: schema; thread effective trust domain; no behavior change for non-federated deployments.
Tests	27 (16 parser + 4 byte-identical + 7 instance identity); 680 in full suite

Phase 3A introduces the optional federation: block in eigenoid.yaml that declares peer trust domains and bundle endpoints. The schema parses, validates, and threads an effective trust domain through StackConfig to artifact-generation call sites. Zero new SPIRE-level functionality. No federation HCL is emitted yet. The no-federation byte-identical contract is locked in: users without a federation: block see byte-identical generated artifacts to pre-3A. The https_web profile is rejected at parse time.

Deliverables

ID	Description
D1	FederationPeerConfig, FederationBundleEndpointConfig, StackFederationConfig in orchestration/config.py
D2	StackConfig.effective_trust_domain() resolving federation vs top-level vs default
D3	public_url on LLM / Database / Custom agent configs
D4	generate_compose accepts trust_domain kwarg; no HCL emitted
D5	InstanceIdentity.trust_domain (additive, default None)
D6	CLI logs info line when federation parsed but not yet activated
D7	16 unit tests (parser, validation, env-var interpolation, public_url)
D8	4 acceptance tests (no-federation byte-identical, federation parses, https_web rejected)
D9	3 instance-identity tests (trust-domain field)

Env vars introduced

• None.

Deviations

• Schema plumbing only; runtime behavior unchanged
• Legacy eigenoid/federation.py FederationConfig dataclass deleted (zero callers)

Phase 3B -- SPIRE-level federation activation

Status: shipped

Branch	(merged with 3A as a single PR)
Scope	Emit SPIRE federation HCL, map bundle endpoint port, -federatesWith on entries, ship export/import CLI.
Tests	721 passed total; 25 new artifacts + 12 new CLI tests

Phase 3B activates SPIRE-level federation on top of 3A. Generates federation { bundle_endpoint { ... } } HCL, maps port 8443 in Docker Compose, appends -federatesWith <peer-td> to every spire-server entry create invocation, threads public_url into the agent launcher, introduces container-name namespacing for same-machine two-stack deployments, and ships eigenoid federation export / import CLI for manual bundle exchange. A startup hook re-applies persisted peer bundles after volume teardown.

Three bugs surfaced during the live two-stack acceptance test and were fixed mid-run: missing echo.py fixture (0417361), compose project-name collision (b3e8f6d), and CLI hard-coded the legacy container name (ae32757). Each got a regression test.

Deliverables

• D1: Federation HCL emission in server.conf (default endpoint_spiffe_id, multiple peers)
• D2: Bundle endpoint port mapping (8443 default, custom port support, extra_hosts for Linux)
• D3: -federatesWith on every entry create (per-peer, per-entry)
• D4: public_url plumbing into agent.run(...) in the launcher
• D5: eigenoid federation export produces JSON with endpoint URL, endpoint SPIFFE ID, trust bundle PEM, TD; import updates YAML + persists bundle
• D6: Container-name namespacing (<stack-slug>-eigenoid-spire-server)
• D7: Startup hook apply_persisted_peer_bundles reapplies after docker compose down -v
• D8: StackFederationConfig.endpoint_spiffe_id + resolved_endpoint_spiffe_id()
• D9: public_bundle_endpoint_url parsing

Key files

• eigenoid/orchestration/container.py -- HCL rendering, port mapping, -federatesWith in scripts, container-name helpers, EIGENOID_PORT injection
• eigenoid/cli/federation.py -- new, 250 LOC, export/import + startup hook
• eigenoid/federation.py -- generate_federation_spire_config deleted (subsumed by HCL path)
• tests/test_federation_artifacts.py -- new, 25 tests
• tests/test_federation_cli.py -- new, 12 tests

Deviations

• Docker Compose v2 with name: field required -- older v2.2.3 rejects it; users must upgrade
• host.docker.internal works natively on macOS; Linux requires extra_hosts: host.docker.internal:host-gateway
• eigenoid status --isolated label display stale on federated stacks -- not fixed (3c/3d item)
• Agent build image shared across stacks -- second up rebuilds if source differs

Phase 3C -- Cross-trust-domain enforcement

Status: shipped

Branch	(merged, code WIP f54ffe9)
Scope	Application-layer policy on top of SPIRE federation. Inbound + outbound rejection; outbound runs before DNS/TCP.
Tests	11 new tests; 769 total (1 pre-existing deselected); 104 3a/3b regression tests pass

Three pieces: (1) federated trust domains plumbed from eigenoid.yaml into IdentityProvider._federated_trust_domains at boot; (2) cross-TD pubkey resolution routed by peer trust domain with per-TD SSL contexts, failing closed for unknown TDs; (3) inbound and outbound enforcement rejecting non-federated cross-TD traffic with 403 and a chain.cross_domain_rejected audit event. The outbound gate runs before DNS or TCP, the load-bearing security property. Legacy chain.cross_domain_hop observability event still fires.

Live two-stack-plus-gamma acceptance test passed 2026-04-28: alpha and beta federated, gamma not; alpha to gamma rejected before network, alpha to and from beta allowed.

Deliverables

• D1: EIGENOID_FEDERATION_PEERS env var auto-injected by _build_agent_service (JSON list); not emitted for single-TD stacks (3A contract preserved)
• D2: Agent.initialize calls configure_federation(provider, peers) after provider.initialize()
• D3: IdentityProvider.is_trust_domain_federated(td) predicate
• D4: _AuthMiddleware.dispatch extends cross_domain_hop branch: non-federated, 403 + chain.cross_domain_rejected
• D5: core/auth.py AuthMiddleware.dispatch mirror enforcement
• D6: Agent._enforce_outbound_cross_domain called from Agent.call/stream_call after _resolve_target_spiffe_id, before transport selection
• D7: TLSContextManager.client_ssl_context_for_peer(spiffe_id) -- same-TD uses existing context, federated uses per-TD context, unknown returns None
• D8: X509SVID.create_client_ssl_context_for_trust_domain(td) with that TD's bundle as trust root
• D9: X509SVID.trust_bundles_by_domain -- filled by fetch_x509_svid walking all bundles
• D10: AgentCardPubkeyResolver._fetch_and_verify uses client_ssl_context_for_peer, falls back to legacy for test stubs
• D11: 11 regression tests covering all three pieces + fail-closed path

Env vars and flags

• EIGENOID_FEDERATION_PEERS (auto-injected by framework; not user-configurable)

Deviations

• Fail-closed on unknown TDs (no network round-trip)
• _federated_trust_domains read in 5 places -- consolidation deferred to Phase 4
• Outbound enforcement chain_id best-effort (reads RequestContext, None outside A2A request)
• Resolver fallback to legacy client_ssl_context kept for one Phase 1.5 test stub
• Pre-existing test_audit_self_check_raises_on_unwritable_path deselected (sandbox runs as root)

Phase 3D -- Federation bootstrap UX (with cross-machine fix)

Status: shipped

Branch	(merged with cross-machine restart-robustness fix)
Scope	Friend-request bootstrap with 12-char pairing code, /admin/federations UI, per-peer policy, revocation.
Tests	19 new tests; pre-existing 113-test federation suite updated and passing

Asymmetric friend-request flow: initiator generates a 12-character pairing code, publishes its SHA-256 hash on /.well-known/eigenoid-federation.json, sends the code out-of-band; acceptor enters code + peer metadata, the framework verifies the hash at peer, applies the bundle, and POSTs a confirm callback that finalizes both sides. Pairing codes: 12 chars, ~60-bit entropy, 10-minute TTL, single-use on confirm, 5-attempt rate limit.

Two cross-machine fixes landed mid-phase: per-agent state directories persist via host bind-mount (instance.json, signing keys, pin store, audit ledger, federation records all survive eigenoid up --isolate restarts); the bundle commitment is fetched fresh on every accept (cached PEM becoming stale on CA rotation).

Deliverables

• D1: Asymmetric flow (initiator pending, acceptor active, confirm callback finalizes)
• D2: Pairing code generation (12 chars, SHA-256 hash, 10-min TTL, single-use)
• D3: /.well-known/eigenoid-federation.json public endpoint (hash list, no auth)
• D4: /admin/federations web UI for initiator add-peer and acceptor accept-incoming
• D5: /federation/confirm public endpoint for callback
• D6: Per-peer policy hook (currently all federated TDs fully trusted)
• D7: Revocation path (operator can revoke peer relationship)
• D8: Email transport (logging sender for dev; SMTP-upgradeable)
• D9: SPIFFE pinning per 3d.1 -- operator supplies trust anchor + SPIFFE ID; fetcher validates chain + SAN
• D10: Per-agent state persistence via host bind-mount + init container chown (19 tests)
• D11: Fresh bundle fetch on every accept (CA rotation gap, 7 tests)
• D12: 5 tests for asymmetric two-stack lifecycle

Env vars and flags

• EIGENOID_FEDERATION_PUBLIC_ADMIN_URL -- for cross-machine bootstrap; acceptor enters this URL on form
• EIGENOID_FEDERATION_LOCAL_TRUST_BUNDLE_PEM_PATH -- for 3d.1 SPIFFE pinning

Deviations and known follow-ups (Phase 4)

• Operator-supplied trust anchor is the only auth for the bundle endpoint at first fetch -- Phase 4: https_web profile via public CA
• Short typed pairing code is MITM-exposed -- Phase 4: fingerprint-based handshake with optional QR code
• Callback auth concentrates authority in pairing code -- Phase 4: ephemeral private-key-signed callbacks
• Confirm-side bundle failure has no automated retry -- Phase 4: admin retry endpoint
• Bundle apply + state flip not transactional -- Phase 4: write-ahead log
• Initiator observability limited between initiate and accept -- Phase 4: shorter windows + stronger monitoring

Audit -- SPIFFE ID usage (Phase 2.1 D4)

Why this audit existed. The Phase 2 URL-synthesis bug (spiffe://eigenoid.local/agent/https://librarian:9400) revealed that bare-string SPIFFE IDs were propagating with zero validation. Phase 3 federation extracts trust domains from peer-cert SPIFFE IDs at every cross-TD hop, so any unparseable ID becomes a hard error there. Phase 2.1 catalogued every CREATE / PARSE / COMPARE site so Phase 3 could land safely.

Inventory totals

Class	Sites
CREATE (synthesis from components)	18
PARSE (string-strip / structural extract)	14
COMPARE (trust domain)	5
COMPARE (full SPIFFE ID ==)	12
Total non-test production sites	49

Disposition rules

• migrated -- uses eigenoid.core.spiffe_id.SpiffeID
• kept-as-string -- full-SPIFFE-ID string equality; does not need trust-domain structure
• pending-phase-3 -- should migrate before federation lands but does not block 2.1

Concrete fixes applied in the PR

• eigenoid/agent.py:212 -- mTLS SPIFFE-ID trust-domain extraction migrated to SpiffeID.try_parse(...).trust_domain
• eigenoid/agent.py:248 -- JWT-SVID subject trust-domain extraction migrated to SpiffeID.try_parse
• eigenoid/agent.py:_dispatch_approvals -- both mTLS and JWT branches migrated
• eigenoid/core/pubkey_resolver.py -- added AgentCardPubkeyResolver.fetch_card_spiffe_id(base_url)
• eigenoid/agent.py:_resolve_target_spiffe_id -- removed URL-in-path synthesis; resolves via card or raises

Audit -- silent-failure paths (Phase 2.1 D6)

Why this audit existed. Phase 2 integration found two silent-failure bugs (auto-request flow, OIDC token cache) that masqueraded as normal operation. Phase 2.1 catalogued ~25 except clauses across security-critical surfaces to find and fix the pattern before Phase 3 added more code.

Inventory totals

Disposition	Count	Action
re-raise / log-warn	3	Applied in PR
log-then-continue (was silent)	2	Applied in PR
keep-silent (already logging)	~14	Documented, no change
keep-silent (intentional)	~6	Documented, no change

Concrete fixes applied in the PR

• Stream cancellation event enqueue (agent.py:833) -- WARNING log before re-raising asyncio.CancelledError
• OIDC token cache lookup (client.py:183) -- broad except (ImportError, Exception) split: ImportError silent, others WARNING
• Auto-request grant failures (agent.py:2929+) -- grant_request.auto_sent_failed already emitted; swallow scoped tightly
• D2 audit self-check (agent.py:_audit_store_self_check) -- originally returned on error; now raises RuntimeError with clear message

Runbook -- Phase 3c manual acceptance

Purpose. Self-contained step-by-step verification of Phase 3c (IdentityProvider integration, cross-TD pubkey resolution, cross-TD enforcement) against real SPIRE servers, real bundle exchange, real container boundaries.

Verification points

1. Branch verification (commit f54ffe9 in history) + unit test baseline (11 passed)
2. Setup: three working dirs (alpha, beta, gamma) with echo.py + eigenoid.yaml (federates_with initially empty)
3. First up: both stacks unfederated, registration succeeds with no warnings
4. Bundle export: alpha and beta export to JSON files with all required fields
5. Bundle import: alpha imports beta's bundle, beta imports alpha's; yaml updated, sidecar persisted, applied immediately
6. Second up: federated registration succeeds, persisted bundles re-applied
7. D1 verification: EIGENOID_FEDERATION_PEERS env shows peer TDs (["beta.local"] on alpha, ["alpha.local"] on beta)
8. D2 verification: SPIRE bundle list shows both local and federated bundles on each side
9. D3 outbound verification: alpha to gamma rejected before network; gamma logs show no inbound request
10. D5 inbound verification: 2-hop chain (alpha to gamma) rejected with 403 + chain.cross_domain_rejected; legacy chain.cross_domain_hop also fires

Out-of-band prerequisites

• Docker + GHCR access (pulls ghcr.io/spiffe/spire-server:1.9.6, ghcr.io/spiffe/spire-agent:1.9.6)
• macOS Docker Desktop, or Linux with host.docker.internal reachable or routable via DNS
• Eigenoid source tree; working dirs must be under the source so _find_project_root finds Dockerfile.agent
• Three YAML configs from fixtures, three echo.py scripts, enough disk for three SPIRE datastores

Repo review -- gap analysis

Date / context. Pre-Phase-3 architecture review. Establishes the foundation for all post-2.0 work (conversation memory, federation, CLI tools, instrumentation).

Architecture summary at review time

Eigenoid is a governed multi-agent AI communication framework built on SPIFFE/SPIRE. Core: Agent base class with @skill decorators, JWT-SVID auth, delegation chains with permission reduction, RBAC, approval workflows, container isolation, OIDC integration. Identity: per-agent SPIFFE ID, SPIRE-issued JWT-SVID for bearer tokens, OIDC-linked IDs for human users. Delegation: DelegationChainManager creates typed hops; PermissionReducer progressively strips dangerous permissions. Discovery: YAML / env-var lookups (not yet A2A Agent Card consumption). Skill system: JSON Schema from type hints, permission requirements, auto-registration on agent card. LLM agent: single-turn OpenAI calls with delegated skill discovery at runtime (no conversation memory). Two working end-to-end demos: metadata-skills (agent-to-agent) and approval-gate (human approval).

Gaps identified (with current status)

#	Gap	Severity	Status
1	Weak token validation -- JWT signature never verified	P0	fixed (Phase 1)
2	Dual server divergence -- agent.py and a2a/server.py unreconciled	P0	fixed (Phase 1)
3	Non-standard skill routing -- _skill in data payload	P1	fixed (Phase 1)
4	No dev mode without SPIRE	P1	partial
5	Client-side does not follow A2A Agent Card patterns	P1	partial
6	mTLS transport missing -- uses JWT-SVID over HTTP only	P1	fixed (Phase 1)
7	Dynamic peer registry missing (no consent model)	P2	open
8	Agent Registry service missing (central card store)	P2	open
9	Agent Card missing skills serialization	P1	fixed (Phase 1)
10	Custom agents invisible to LLM routing	P2	fixed
11	No observability / monitoring (OpenTelemetry, structured logs)	P2	open
12	No persistent audit store	P1	fixed (Phase 2)
13	No rate limiting or circuit breakers	P2	open
14	No secret management integration	P2	open
15	No agent versioning or rollback	P2	open
16	No SDK for other languages	P3	open
17	No federation across trust domains	P2	fixed (Phase 3A-D)
18	No testing utilities for developers	P2	partial
19	No policy-as-code engine (OPA / Cedar)	P2	open
20	No streaming / long-running task support	P2	partial
21	No retry / resilience patterns	P2	open
22	No pre-built conversational UI	P1	fixed (OpenClaw)
23	No conversation memory / session state	P0	fixed (OpenClaw)

OpenClaw innovations proposal

Date	2026-03-23; updated 2026-03-27 (Phases 1-2), 2026-03-31 (Phases 3-4)
Status	Phases 1-4 complete; Phases 5-6 draft
Builds on	docs/REPO_REVIEW.md Gap #13 (Conversation Memory)

Naming note

The "Phase 1-6" numbering in this proposal refers to the feature-integration phases of the OpenClaw arc specifically (conversation memory, governed filesystem, memory search, dynamic tools, instruction updates, CLI tool adapter). These are unrelated to the delegation-security Phase 0-3D numbering above.

Scope. Integrates OpenClaw's five patterns -- filesystem-as-memory, dynamic tool loading, self-modifying instructions, pre-compaction flush, governed memory search -- into Eigenoid on top of the existing governance primitives (SPIFFE identity, permission reduction, delegation chains, audit store).

Sub-phase	Status	Summary
1. Governed conversation memory (Gap #13)	complete	ConversationStore + filesystem backend, A2A propagation, governed visibility (permission + depth filtering), LLMAgent multi-turn, pre-compaction flush, BM25 memory search (zero deps), session lifecycle, terminal chat (chat.py with OIDC + recall/convs). 33 conversation tests; eigen_echo demo. ~2200 LOC.
2. Governed filesystem access	complete	FilesystemPermission with from_string parser, FilesystemGuard with symlink/TOCTOU protection, depth-based reduction (depth 0 full, 2 workspace, 3+ read, 5+ none), container mount enforcement, file_read/write/list skills via FileToolsMixin, LLMAgent.ask() routing. 64 tests. ~1300 LOC.
3. Memory search	complete	Pure-Python BM25 index (zero external deps), identity-scoped search (participant filtering), memory_search + memory_get skills, MemorySearchMixin. 17 tests. ~500 LOC.
4. Dynamic tools + instruction update	complete	ToolRegistry with approved-source validation, permission ceiling enforcement, DynamicToolsMixin (install/uninstall/list), InstructionUpdateMixin (update_instructions always requires approval), two source types (Python @skill modules + CLI tools via SKILL.md manifests + subprocess adapter). 26 tests. ~800 LOC.
5. CLI tool adapter + remote registries	draft	RemoteSourceLoader (git clone + pip install in container), SKILL.md manifest parser (YAML frontmatter), CLIToolAdapter (subprocess to @skill-compatible handler), container hardening recommendations (seccomp, gVisor, Kata). ~400 LOC estimated.
6. Advanced memory	draft	Permission-based content redaction on search results, optional vector search provider integration (OpenAI, Ollama, local GGUF). ~600 LOC estimated.

Part 2 -- Reference documentation

Operational and architectural reference for current shipped behavior. Originally lived under docs/ in the repo; consolidated here.

Architecture overview

Source: docs/ARCHITECTURE.md

System design

Eigenoid is the governance layer for agent-to-agent communication. It ensures every message between agents carries proof of origin, every delegation chain is tracked and scoped, and every dangerous action requires human approval -- so enterprises can deploy multi-agent systems that pass security and compliance review.

Under the hood, Eigenoid integrates SPIFFE/SPIRE with the A2A Protocol to create a verifiable, auditable communication fabric for multi-agent systems.

Core components

1. IdentityProvider

The core identity provider for SPIFFE-enabled authentication.

Responsibilities:

• Initialize SPIFFE Workload API connection
• Fetch and cache X.509 and JWT SVIDs
• Validate tokens and claims
• Support both real SPIRE and mock mode

Key methods:

• initialize() - Set up SPIFFE components
• issue_svid() - Issue JWT-SVID for an agent
• validate_token() - Validate incoming JWT-SVID
• fetch_x509_svid() - Fetch X.509-SVID for mTLS
• validate_peer_spiffe_id() - Validate a peer's SPIFFE ID against trusted domains

2. TLSContextManager

Manages SSL contexts for mTLS with automatic certificate rotation.

Responsibilities:

• Build server, client, and verify-only SSL contexts from X.509-SVIDs
• Atomically swap contexts on SPIRE certificate rotation
• Provide thread-safe access to current contexts

SSL contexts:

• server_ssl_context - Server cert + trust bundle, CERT_OPTIONAL (passed to uvicorn)
• client_ssl_context - Client cert + trust bundle, check_hostname=False (for direct/mTLS transport)
• verify_only_ssl_context - Trust bundle only, no client cert (for proxied/JWT transport)

3. PeerCertMiddleware

Raw ASGI middleware that extracts TLS peer certificates from uvicorn connections.

Responsibilities:

• Access the SSL transport via uvicorn's internal request cycle
• Extract peer certificate and inject into scope["peer_cert"]
• Run before all other middleware so the cert is available for auth

5. DelegationChainManager

Manages the creation and verification of delegation chains.

Phased implementation. This component is specified across three shipped phases -- see Delegation security model for the authoritative end-to-end reference. Phase 0 unified the hop schema with a forward-compat bucket and added chain_id + version. Phase 1 replaced the SHA-256 placeholder with real Ed25519 signatures over RFC 8785 (JCS) canonical bytes, chained to the previous hop's signature; every hop embeds its signer's PEM public key. Phase 1.5 added identity binding (PubkeyPinStore, IdentityBindingVerifier) so the embedded pubkey is cross-checked against a pin instead of being self-attested.

Responsibilities:

• Create hops with the unified Phase 0 schema (Phase 0)
• Sign outbound hops with the agent's per-agent Ed25519 signing key over JCS canonical bytes (Phase 1)
• Verify inbound chains at the middleware before set_context (Phase 1)
• Cross-check every hop's embedded pubkey against the per-SPIFFE-ID pin store, running TOFU on first contact and walking rotation attestations on mismatch (Phase 1.5)

Key methods:

• DelegationHop.from_dict() -- forward-compat parse (Phase 0)
• sign_hop_dict(hop, identity, previous_signature) -- embeds signer_public_key_pem and writes the base64 Ed25519 signature (Phase 1)
• verify_chain_dicts(chain) -- structural + cryptographic check (Phase 1)
• IdentityBindingVerifier.verify_chain_binding(chain) -- pin check per hop (Phase 1.5)

6. PermissionManager

Handles permission inheritance and reduction.

Responsibilities:

• Apply inheritance strategies
• Progressive permission reduction
• Financial limit adjustments
• Action filtering by depth

Strategies:

• INHERIT_ALL - No reduction
• PROGRESSIVE_REDUCTION - Reduce by depth
• ROLE_BASED - Based on agent role
• CUSTOM - Custom reduction function

7. ApprovalTracker

Manages human approval workflows.

Responsibilities:

• Create approval requests
• Track approval decisions
• Embed approvals in SVIDs
• Verify approvals

Policies:

• ANY - Any one approver
• ALL - All approvers required
• MAJORITY - Majority needed
• THRESHOLD - Minimum number

8. ConversationStore

Persistent conversation memory with governed visibility.

Responsibilities:

• Store messages with sender_spiffe_id, delegation_depth, and content
• Create/close conversations with participant tracking
• Return governed history filtered by delegation depth and permissions
• Support pre-compaction flush (LLM summarizes old messages)

Backends:

• Filesystem (eigenoid/conversation/filesystem.py) -- per-conversation directories with JSON files, atomic writes via fcntl.flock
• Interface allows future backends (Redis, PostgreSQL)

Governed access:

• get_governed_history() filters by participant SPIFFE ID, delegation depth, and conversation:read permission
• Root callers (depth 0) see all messages
• Deeper agents see fewer messages
• Without conversation:read, content is replaced with {"redacted": true}
• Agents always see their own messages regardless of depth

9. OIDC authentication

Client-to-agent authentication via OIDC (separate from agent-to-agent mTLS).

Components:

• eigenoid/auth/oidc.py -- Full OIDC authorization code flow with PKCE. Opens browser, starts local callback server, exchanges code for tokens, validates id_token via issuer JWKS.
• eigenoid/auth/storage.py -- Token persistence at ~/.eigenoid/auth.json. Stores id_token, refresh_token, email, expires_at.
• eigenoid/auth/resolver.py -- Identity resolution: returns verified OIDC email or falls back to self-declared identity.

Middleware integration: The _AuthMiddleware in agent.py tries two validation paths:

1. SPIRE JWT-SVID validation (agent-to-agent)
2. OIDC id_token validation via issuer JWKS (client-to-agent, configured via EIGENOID_OIDC_ISSUER and EIGENOID_OIDC_CLIENT_ID)

10. SVIDValidator

Validates SPIFFE SVIDs and delegation chains.

Responsibilities:

• Validate JWT-SVID signatures
• Check expiration
• Verify trust bundles
• Validate delegation chains

Data flow

1. Human authentication to SPIFFE identity

Human (OAuth2/SAML)
    v
Enterprise IdP
    v
SPIRE Server (creates JWT-SVID)
    v
Human initiates A2A request
    v
Orchestrator receives JWT-SVID

text

2. Agent-to-agent delegation

Parent Agent
    v
Create delegation hop  (unified Phase 0 schema, chain_id + version)
    v
Sign hop with per-agent signing.key       (Phase 1: Ed25519 over
    v                                      JCS, chained to prev sig)
Embed signer_public_key_pem               (Phase 1: per-hop pubkey)
    v
Apply permission reduction
    v
Attach chain to A2A metadata.spiffe.chain (+ X-Delegation-Chain header)
    v
Child Agent's AuthMiddleware:
  1. verify_chain_dicts()                 (Phase 1: signature math)
  2. IdentityBindingVerifier              (Phase 1.5: pin check /
     if EIGENOID_IDENTITY_BINDING_ENABLED  TOFU / rotation walk)
    v
set_context -> skill handler executes
    v
Child extends chain, signs its own hop, forwards to next target

text

See Delegation security model for the hop wire format, the chained-signature derivation, and the pin-store lifecycle.

3. Client-to-agent authentication (OIDC)

User runs: eigenoid login
    v
Browser opens OIDC provider (Google)
    v
Authorization code returned to local callback
    v
Code exchanged for tokens (id_token, refresh_token)
    v
Tokens saved to ~/.eigenoid/auth.json
    v
chat.py loads id_token, attaches Authorization: Bearer
    v
Agent middleware validates via issuer JWKS
    v
auth_context.method = "oidc", email from claims

text

4. Multi-turn conversation

User sends message
    v
Agent resolves/creates conversation_id
    v
Save user message (sender_spiffe_id, delegation_depth)
    v
Load governed history (filtered by depth + permissions)
    v
Send history + question to LLM
    v
LLM routes to delegate agent (mTLS)
    v
Format result with conversation context
    v
Save agent response
    v
Return answer + conversation_id

text

5. Human approval workflow

Agent encounters approval requirement
    v
Create ApprovalRequest
    v
Notify human approvers
    v
Record approval decisions
    v
Embed approval in JWT-SVID
    v
Agent proceeds with approved action

text

Security model

Trust establishment

1. SPIRE Server is the root of trust
2. Trust Bundle contains root CA certificates
3. X.509-SVIDs enable mTLS -- identity proved via client certificate at the TLS layer
4. JWT-SVIDs are signed by SPIRE Server -- identity proved via Bearer token at the HTTP layer
5. Hostname verification is disabled -- SPIFFE uses URI SANs (spiffe://...), not DNS SANs. Trust is established via the SPIRE trust bundle, not DNS hostname matching

Delegation chain integrity

1. Ed25519 hop signatures (Phase 1) -- every hop carries a base64 signature over previous_signature || 0x00 || JCS(hop_without_signature). The chained input binds each hop to its predecessor so tampering with hop N invalidates hop N+1. The signer's PEM public key is embedded per-hop as signer_public_key_pem.
2. Parent-child SPIFFE linkage verified (Phase 0) -- each hop's parent_spiffe_id must equal the previous hop's spiffe_id.
3. Timestamp ordering enforced with tolerance (Phase 1) -- configurable via EIGENOID_CHAIN_TIMESTAMP_TOLERANCE (default 5.0 s) to absorb inter-host clock skew without accepting outright reversals.
4. Identity binding (Phase 1.5, gated by EIGENOID_IDENTITY_BINDING_ENABLED) -- the embedded pubkey is cross-checked against a per-SPIFFE-ID pin. First contact TOFUs against the peer's mTLS-authenticated agent card (with peer-cert SPIFFE-ID verification); subsequent mismatches walk the peer's published rotation attestations; revoked pins always reject. Without Phase 1.5, an attacker with their own valid Ed25519 keypair could sign a hop claiming any SPIFFE ID and the math would pass.
5. Permission inheritance validated (Phase 1 D6, opt-in via EIGENOID_CHAIN_AWARE_PERMISSIONS) -- role permissions are intersected along the chain.

All of the above run in AuthMiddleware.dispatch before set_context; forged chains 403 before any skill handler executes. See Delegation security model for the per-branch verification flow.

Permission reduction

Depth 0 (Human): All permissions
    v
Depth 1: Remove nothing
    v
Depth 2: Remove DELETE
    v
Depth 3: Remove DELETE, APPROVE
    v
Depth 4: Remove CREATE, UPDATE, DELEGATE
    v
Depth 5: Read-only

text

Financial limits

Base: $5000
Depth 1: $2500 (50% reduction)
Depth 2: $1250
Depth 3: $625
Depth 4: $312.50
Depth 5: $156.25
Minimum: $100

text

Deployment architecture

Production setup

+-----------------+
|  Enterprise IdP |
|  (OAuth2/SAML)  |
+--------+--------+
         |
    +----v-----+
    |  SPIRE   |
    |  Server  |
    +----+-----+
         |
    +----v-----+
    |  SPIRE   |
    |  Agent   |
    +----+-----+
         |
    +----v-------------+
    |  A2A Agents      |
    |  (Orchestrator,  |
    |   Planner, etc)  |
    +------------------+

text

High availability

• Multiple SPIRE Servers with shared datastore
• SPIRE Agents on each host
• Load balancer for A2A endpoints
• PostgreSQL for SPIRE persistence

Performance considerations

SVID caching

• Cache X.509-SVIDs for 5 minutes
• Cache JWT-SVIDs for 5 minutes
• Refresh before expiration

Batch operations

• Fetch multiple SVIDs in parallel
• Batch validation operations
• Connection pooling

Monitoring

Export metrics for:

• Delegation depth distribution
• Permission check latency
• SVID validation failures
• Approval response times

Extension points

Custom permission reducers

def custom_reducer(permissions, role, depth):
    # Your custom logic
    return filtered_permissions

manager.inherit_permissions(
    parent_permissions=perms,
    child_role="agent",
    depth=2,
    custom_reducer=custom_reducer
)

python

Custom approval policies

tracker = ApprovalTracker(
    auto_approve_patterns=[
        "*.read",
        "low_risk.*"
    ]
)

python

Custom validation

class CustomValidator(SVIDValidator):
    async def validate_jwt_svid(self, token, audience):
        # Add custom validation logic
        is_valid, spiffe_id = await super().validate_jwt_svid(token, audience)

        # Additional checks
        if not self.check_custom_claim(token):
            return False, None

        return is_valid, spiffe_id

python

Audit store

The framework includes a persistent, pluggable audit store that records every authentication, authorization, and skill lifecycle event. It is integrated directly into the agent request pipeline -- no additional wiring required.

Backends:

• SQLite (default) -- single-file database per agent, queryable with standard SQL
• JSONL -- append-only line-delimited JSON, suitable for log aggregation pipelines

Event types recorded:

• auth.success / auth.rejected -- authentication outcomes
• skill.invoked / skill.completed / skill.failed -- full skill lifecycle with duration
• permission.allowed / permission.denied -- RBAC decisions
• delegation.scope_checked -- delegation chain validation

Architecture:

• Async-safe buffered writes (configurable batch size and flush interval)
• NullAuditStore when disabled (zero overhead, bool() returns False)
• SqliteBackend creates directories and schema automatically at runtime
• In container mode, each agent gets an isolated Docker volume for its audit DB

Enable with audit_enabled = True on any Agent subclass or set EIGENOID_AUDIT_ENABLED=true as an environment variable.

See examples/audit_logs/ for a working two-agent demo.

Recent additions

6. Governed conversation memory (complete) -- Fully implemented. Per-agent session state with delegation-aware visibility. Conversations persisted to filesystem (JSON in container volumes), keyed by conversation_id. History filtered by the requester's delegation depth and permissions -- agents deeper in the chain see less. Includes pre-compaction flush for LLM context windows. See examples/eigen_echo/ for the working demo and OpenClaw innovations proposal for the spec (originally Gap #13 in the repo review).
7. OIDC client authentication (complete) -- Client-to-agent auth via OIDC Bearer tokens. Agent middleware validates both SPIRE JWT-SVIDs (agent-to-agent) and OIDC id_tokens (client-to-agent). Tested end-to-end with Google OIDC.
8. Interactive chat UI (complete) -- Terminal chat (examples/eigen_echo/chat.py) with OIDC auth, /recall (governed history), /convs (list conversations), /new (fresh thread).
9. Phase 1: real delegation (complete) -- Replaces the Phase 0 SHA-256 placeholder with real Ed25519 signatures over RFC 8785 (JCS) canonical bytes. Each hop is signed by the per-agent signing key (signing.key under <EIGENOID_HOME>/agents/<name>/), chained to the previous hop's signature, and verified inside the request middleware before any skill handler runs. Approval records carry a re-validatable OIDC id_token. See Phase 1 and examples/testing_capability/phase_1/.
10. Phase 1.5: identity binding (complete) -- Closes Phase 1's open question ("does this pubkey actually belong to this SPIFFE ID?"). Adds PubkeyPinStore (TOFU pin ledger), PubkeyResolver ABC + chain (agent-card / manual admin / static), signed RotationAttestations served at /.well-known/pubkey-rotation.json, and an admin API for pre-provisioned pins. Wires into the request middleware so forged chains never reach a handler. Gated by EIGENOID_IDENTITY_BINDING_ENABLED. See Phase 1.5 and examples/testing_capability/phase_1_5/.
11. Phase 2: grants and capabilities (complete) -- Authorization layer on top of identity. Grants are long-lived, revocable records the issuer persists (grants.db); capabilities are short-lived Ed25519-JWT tokens minted off them and attached to outbound delegation hops as the capability field. Two reserved built-ins (_system.grant_request, _system.grant_request_status) drive the operator-approval bootstrap; a Starlette web UI mounts at /approvals (web-mode). The verifier reuses the Phase 1.5 PubkeyPinStore + PubkeyResolver -- no parallel pubkey cache. Gated by EIGENOID_REQUIRE_GRANT_CITATION (default off). Auto-request and stale-cap eviction make the loop self-healing. See Phase 2 and examples/testing_capability/phase_2/.

Capability model: skills vs tools vs CLI execution

Eigenoid has three distinct capability layers. Understanding the terminology prevents confusion when reading the codebase or building agents.

Layer 1: A2A skills (protocol layer)

Concept	Usage
@skill decorator	Defines a capability an agent exposes to other agents
register_skill()	Programmatically registers a capability on the agent
AgentSkill (A2A SDK)	The A2A protocol type advertised in Agent Cards
skill_id in message metadata	How callers route requests to a specific capability
_skills / _skill_metadata	Internal dicts on the Agent class

This is the protocol interface. Every callable capability on an agent is a "skill" in A2A terms -- it appears in the Agent Card at /.well-known/agent.json, and callers invoke it via metadata.skill_id in JSON-RPC messages. The @skill decorator, register_skill() method, and all related internals use "skill" terminology because they implement the A2A specification.

Layer 2: Dynamic tools (runtime extension layer)

Concept	Usage
ToolRegistry	Manages approved tool sources and enforces installation governance
InstalledToolRecord	Audit record for a dynamically installed tool
install_tool / uninstall_tool	User-facing skill IDs for tool management
load_tool_module()	Loads a Python module from an approved source
DynamicToolsMixin	Mixin that adds tool management capabilities to an agent

This is the runtime extension mechanism. Tools are Python modules that agents install at runtime from approved sources. Once installed, tools get registered as A2A skills (via register_skill()) so they're callable through the protocol. The governance layer applies: approved sources only, permission ceiling enforcement, human approval workflow, and full audit trail.

The bridge point is the register_skill() call inside DynamicToolsMixin -- that's where a tool crosses from the runtime layer into the protocol layer.

Layer 3: CLI execution (container runtime layer) -- implemented

Concept	Usage
CLIToolAdapter (eigenoid/tools/cli_adapter.py)	Wraps CLI commands as governed async skill handlers
ToolManifest (eigenoid/tools/manifest.py)	Parsed from SKILL.md YAML frontmatter -- name, subcommand, input schema, required binaries
make_cli_handler()	Generates an async handler: ath <subcommand> --flag value returns JSON stdout returns dict
_load_remote_source() (eigenoid/tools/registry.py)	pip install (PyPI) or git clone fallback returns discover SKILL.md returns build adapters

This is the container runtime layer (Phase 5). Agents run in isolated Docker containers. CLI execution gives them the ability to shell out to installed binaries -- ath weather --city London returns JSON to stdout, which the adapter parses back into a Python dict. Unlike in-process Python tools (Layer 2), CLI tools run as subprocesses with their own process isolation.

The adapter reads SKILL.md files with YAML frontmatter (supporting both flat and openclaw metadata formats), generates async handlers that call asyncio.create_subprocess_exec() with explicit env propagation, parse JSON stdout, and return the result dict. The handler is registered via register_skill() -- same bridge point as Layer 2. The Dockerfile.agent sets PYTHONUSERBASE=/app/tools-user and PIP_NO_CACHE_DIR=1 so that runtime pip install --user works for non-root UIDs (10001+), and /app/tools-user/bin is on PATH so installed binaries are immediately discoverable.

See examples/dynamic_tools_cli/ for the working end-to-end demo with both local Python tools and CLI packages from Agent-cli-ToolHub. Tested live: ath weather --city London returns real weather data inside the container.

How Layer 2 and Layer 3 differ

	Layer 2: in-process Python tools	Layer 3: CLI tool packages
Format	Single .py file with @skill-decorated async functions	A pip-installable package with CLI subcommands (ath weather, ath search)
Discovery	Scans module for @skill decorator + extracts metadata from it	skills/ath-<name>/SKILL.md files with YAML frontmatter
Invocation	Calls the Python function directly in-process	asyncio.create_subprocess_exec() returns JSON to stdout
I/O contract	kwargs in, dict out (in-process)	argparse args in, JSON stdout/stderr out with exit codes
Dependencies	Assumed none (just load the .py)	pip install the package (PyPI or from git clone)
Loading	importlib loads {path}/{tool_id}.py	_load_remote_source() handles clone/install/discover

End-to-end flow (implemented)

User says: "Install Agent-cli-ToolHub from toolhub"
    -> LLM routes to _local/install_tool(tool_id="Agent-cli-ToolHub", source="toolhub")
    -> ToolRegistry validates "toolhub" is on the approved list
    -> ApprovalServer prompts human (if require_approval: true)
    -> pip install cli-tools-hub (PyPI) -- if fails, fall back:
    -> git clone --depth=1 https://github.com/andylow92/Agent-cli-ToolHub /tmp/tools/toolhub/
    -> pip install --user --no-cache-dir /tmp/tools/toolhub/ (from clone)
    -> discover_skill_mds() finds skills/*/SKILL.md manifests
    -> skill_filter applied: only ath_weather + ath_search registered
    -> For each: parse_skill_md() -> CLIToolAdapter -> register_skill()
    -> Permission ceiling checked for each tool
    -> AuditStore.emit("tools.installed", source="toolhub", ...)
    -> Schema invalidated -> next request rebuilds LLM routing context

User says: "What's the weather in London?"
    -> LLM routes to _local/ath_weather(city="London")
    -> CLI handler runs: ath weather --city London (subprocess, env inherited)
    -> JSON stdout parsed -> result dict returned
    -> LLM formats: "London: 12 degrees C, overcast clouds, humidity 69%"

text

Implementation

File	LOC	What it does
eigenoid/tools/manifest.py	430	SKILL.md parser: YAML frontmatter, openclaw metadata format, ## Usage schema inference
eigenoid/tools/cli_adapter.py	235	make_cli_handler() + CLIToolAdapter + _build_cli_args() + binary pre-check
eigenoid/tools/registry.py (additions)	~100	_load_remote_source(): pip install (PyPI to git clone fallback), SKILL.md discovery, skill_filter
Dockerfile.agent (additions)	6	PYTHONUSERBASE, PIP_NO_CACHE_DIR, HOME, /app/tools-user writable dir
tests/test_cli_adapter.py	688	61 unit tests: manifest parsing, subprocess mocking, arg building, error handling
examples/dynamic_tools_cli/	~1000	Full example: eigenoid.yaml, chat.py, tool_agent.py, tools/, skills/, 23 integration tests

The existing governance layer (approved sources, permission ceiling, human approval, audit trail) applies unchanged -- Phase 5 widened the funnel of what can be loaded without changing how it is governed.

How the layers compose

+---------------------------------------------------------+
|                    A2A Protocol                          |
|         (Agent Cards, skill_id routing, JSON-RPC)        |
|                                                          |
|   +--------------+  +--------------+  +--------------+   |
|   |  @skill       |  | Dynamic Tool |  |  CLI Tool     |  |
|   |  (in-process) |  | (installed   |  |  (subprocess  |  |
|   |              |  |  at runtime) |  |  in container)|  |
|   +--------------+  +------+-------+  +------+-------+  |
|                            |                  |          |
|                     register_skill()    register_skill() |
|                      (bridge point)      (bridge point)  |
+---------------------------------------------------------+
                              |
                    +---------v----------+
                    |   Governance Layer   |
                    |  - Permission ceiling|
                    |  - Approval workflow |
                    |  - RBAC enforcement  |
                    |  - Audit trail       |
                    |  - Delegation depth  |
                    +---------------------+

text

All three layers converge at register_skill() -- the single point where any capability becomes an A2A-visible, governed skill. The governance layer applies uniformly regardless of how the capability was defined.

Security boundaries

Boundary	What it prevents
ToolRegistry (capability boundary)	Agent can only call registered skills. No shell access unless a skill wraps it. LLM cannot escalate permissions -- the permission ceiling ensures installed tools can't exceed the installer's own permissions.
Container (process boundary)	Namespace/cgroup isolation. Agent can't see host files, network, or processes. Each agent gets its own filesystem, UID, and network namespace.
Kernel (shared)	Docker containers share the host Linux kernel. Every syscall goes directly to the host kernel -- there is no hypervisor boundary. Isolation is purely at the namespace/cgroup level.

Threat model by layer:

• LLM tries to escalate via skill calls -- Blocked by ToolRegistry + RBAC. Low risk.
• Malicious tool module runs code at import -- Contained by Docker namespaces, but kernel-level escape is theoretically possible. Medium risk.
• Misconfigured container (privileged mode, mounted Docker socket) -- Trivial escape. High risk -- operational concern.

Hardening for untrusted tool sources (recommended for production):

1. gVisor or Kata Containers -- intercept syscalls in userspace or run each container in a lightweight VM. Adds a real kernel boundary.
2. Seccomp profiles -- restrict which syscalls the container can make. Block ptrace, mount, unshare.
3. Read-only root filesystem -- docker run --read-only with explicit writable tmpfs mounts.
4. Drop all capabilities -- --cap-drop=ALL, add back only what's needed.
5. No new privileges -- --security-opt=no-new-privileges prevents setuid escalation.

For the Eigenoid threat model -- agents calling each other with delegated SPIFFE identities -- Docker namespaces are sufficient. The ToolRegistry is the real security boundary. But for running untrusted third-party tool modules (arbitrary repos from GitHub), the container is the last line of defense and gVisor/Kata is recommended. The pip install moment is where the real risk lives -- not the skill invocation, but the code that runs at import time before governance kicks in.

Delivered so far

The delegation-security story has shipped in three phases, each with its own branch and regression suite. Per-phase deliverables, deviations, and latency tables are archived in this document; Delegation security model is the consolidated reference for current shipped behavior.

• (complete) Phase 0 -- Consolidate and prepare. Unified hop schema with forward-compat bucket, chain_id + hop version, typed SpiffeID, InstanceIdentity + Ed25519 keystore pattern, per-agent state dirs under $EIGENOID_HOME/agents/<name>/, multi-TD test fixture, schema versioning on every on-disk store.
• (complete) Phase 1 -- Real delegation. Per-agent signing.key distinct from the Phase 0 instance key, Ed25519 hop signatures over JCS canonical bytes chained to the previous hop's signature, every hop embeds signer_public_key_pem, verify_chain_dicts() in AuthMiddleware.dispatch (403 before set_context on tamper), OIDC id-token-as-signed-approval with email-match enforcement, chain-aware permission intersection (opt-in), cross-trust-domain audit logging.
• (complete) Phase 1.5 -- Identity binding. PubkeyPinStore SQLite ledger, AgentCardPubkeyResolver with peer-cert SPIFFE-ID binding check, IdentityBindingVerifier wired into the middleware after Phase 1's signature math, RotationAttestation signed by the OLD key, /.well-known/pubkey-rotation.json, admin API at /admin/pubkey-pins with revocation, eigenoid pubkeys rotate CLI.

Future enhancements

1. Grant store / capability tokens -- Phase 2. Capability-based authorization on top of the Phase 1.5 identity binding.
2. Federated trust -- Cross-domain delegation enforcement. Phase 1 D7 landed the audit hook (chain.cross_domain_hop); Phase 3 adds the policy knob.
3. Attestation -- Hardware-based identity.
4. Policy engine -- OPA / Cedar integration. Phase 4.
5. Tamper-evident audit -- Signed/chained audit entries, SIEM export.
6. Revocation -- Real-time credential revocation beyond the Phase 1.5 admin pin-revocation path.
7. Governed filesystem access -- Permission-scoped file_read/file_write skills, reduced by delegation depth. See OpenClaw innovations proposal (Proposal 1).
8. Dynamic skill installation -- Governed tool loading with approval workflow and permission ceiling. See OpenClaw innovations proposal (Proposal 2).
9. Memory search -- BM25 + identity-scoped search over conversation history. See OpenClaw innovations proposal (Proposal 4a).
10. Self-modifying instructions -- Approval-gated system prompt updates with audit trail. See OpenClaw innovations proposal (Proposal 5).
11. CLI tool adapter (complete) -- Remote tool registry support with git clone / pip install into container, SKILL.md manifest parsing (YAML frontmatter + openclaw metadata format), and subprocess-based execution via asyncio.create_subprocess_exec. Agents install CLI tool packages (e.g., Agent-cli-ToolHub) as governed skills with the same approval workflow, permission ceiling, and audit trail as in-process Python tools. PyPI install with automatic git clone fallback, skill_filter for selective registration, --no-cache-dir + PYTHONUSERBASE for non-root container UIDs. See examples/dynamic_tools_cli/ and eigenoid/tools/cli_adapter.py.

Delegation security model

Source: docs/DELEGATION_SECURITY.md

This document is the authoritative reference for Eigenoid's delegation-chain security model. It consolidates what each released phase added, how the wire format evolved, and how inbound chains are verified today. If you're looking for high-level system design see Architecture overview; if you want the dual-auth perimeter see Dual-mode authentication. This doc is about the signed-delegation story specifically.

Every section is labeled with the phase that introduced the primitive so readers can tell shipped behavior from roadmap.

Phase roadmap

Phase	Scope	Status
0 -- Consolidate and prepare	Agent consolidation, discovery consolidation, hop schema unification, forward-compat from_dict, typed SpiffeID, chain_id + hop version, InstanceIdentity with Ed25519 keypair pattern, per-agent state dirs, multi-TD test fixture, minimal CI, schema versioning	Merged (PR #1)
1 -- Real delegation	Per-agent signing keys (distinct from instance key), Ed25519 hop signatures over JCS canonical bytes, verify_chain_dicts() at middleware, OIDC id-token-as-signed-approval, chain-aware _enforce_permissions, explicit kill of the Phase-0 SHA-256 placeholder	Merged (PR #2)
1.5 -- Identity binding	PubkeyPinStore, AgentCardPubkeyResolver with peer-cert SPIFFE-ID binding check, IdentityBindingVerifier in middleware, RotationAttestation signed by OLD key, /.well-known/pubkey-rotation.json, /admin/pubkey-pins, eigenoid pubkeys rotate CLI	Merged (PR #3)

Subsequent phases (grant store, capability tokens, pending-approvals UI, cross-trust-domain enforcement) shipped under Phase 2 / 2.1 / 3A-3D and are archived in Part 1 of this document. That archive also covers the OpenClaw feature track (conversation memory, governed filesystem, dynamic tools, instruction updates), which uses its own internal "Phase 1-6" numbering -- unrelated to the delegation-security phases enumerated above.

Key material -- two keys per agent

Every running Eigenoid agent holds two independent Ed25519 keypairs, each per-agent (not per-process-instance, not per-SVID). Both live under $EIGENOID_HOME/agents/<agent_type>/ with mode 0o600, parent directory 0o700.

Key	File	Introduced	Purpose
Instance key	agent.key	Phase 0	Identifies the deployment. Pinned by peers. Published in /.well-known/eigenoid-instance.json.
Signing key	signing.key	Phase 1	Signs outbound delegation hops. Rotates independently of SPIRE X.509-SVIDs (which rotate much more often). Its public half is published on the agent card as signingPublicKeyPem.

The Phase 0 KeyStore.load_or_generate() primitive backs both files -- atomic write, fsync, rename; defensive chmod 0o600 on load. The Phase 1 AgentIdentity value object wraps a KeyStore at the signing.key path and exposes ergonomic sign(message: bytes) / verify(signature, message) methods.

Splitting instance-identity from signing-identity is load-bearing for Phase 1.5: the instance key is long-lived and pinned by peers via the instance document, while the signing key rotates on operator cadence and its rotations are bridged via signed attestations (below).

Wire format

Phase 0 -- the unified hop

Phase 0 unified five dict-literal producers into a single canonical shape with a forward-compat bucket and a version field:

{
    "hop_index":        int,            # position in the chain, 0-indexed
    "spiffe_id":        str,            # this hop's SPIFFE ID
    "parent_spiffe_id": str | None,     # previous hop's spiffe_id (None on root)
    "timestamp":        iso8601_str,    # UTC
    "action":           str,            # skill / operation being delegated
    "chain_id":         uuid_str,       # same on every hop of a given chain
    "version":          2,              # hop schema version (bumped when fields become first-class)
    # ... optional: permissions, restrictions, approval_info, metadata
}

python

DelegationHop.from_dict() stashes any unknown top-level keys into metadata["_preserved"] and re-emits them on to_dict() -- older readers never silently drop fields added by newer writers. This is the foundation Phase 1 and Phase 1.5 both layered additive fields onto.

Phase 1 -- signed hop

Phase 1 added two fields, produced by the issuer and checked by the verifier:

{
    # ... Phase 0 fields ...
    "signer_public_key_pem": str,       # PEM SubjectPublicKeyInfo of the hop signer
    "signature":             base64str, # Ed25519 signature, 64 raw bytes
}

python

The signature covers:

previous_signature_base64 || 0x00 || JCS(hop_without_signature)

text

• JCS (RFC 8785) -- lexicographic object-key ordering, canonical number rendering, deterministic string escaping. Implemented in eigenoid/core/jcs.py.
• Chained input -- each hop's signature input includes the previous hop's signature, so tampering with hop N invalidates hop N+1. previous_signature is the empty string for the root hop.
• NUL delimiter -- 0x00 is unambiguous because base64 never contains NUL bytes. An attacker cannot extend the previous signature to collide with the canonical payload.

Phase 1.5 -- adds nothing to the hop

Phase 1.5 introduces no new hop fields. It layers identity binding on top of the Phase 1 hop: every signer_public_key_pem is cross-checked against a per-SPIFFE-ID pin that the verifier maintains separately. The hop wire format is unchanged -- this is a receiver-side tightening.

Separately, Phase 1.5 introduces a new document, RotationAttestation, served at /.well-known/pubkey-rotation.json. It is not a hop -- it's signed evidence published by an agent to justify a signing-key rotation.

Verification flow (Phase 1 + 1.5 combined)

For every inbound request, AuthMiddleware.dispatch runs the following before any skill handler is invoked:

+---------------------------------------------+
| 1. Extract delegation_chain from request    |   Phase 0
|    (A2A metadata first, then                |
|    X-Delegation-Chain header)               |
+----------------+----------------------------+
                 v
+---------------------------------------------+
| 2. verify_chain_dicts():                    |   Phase 1
|    - hop_index monotonic                    |
|    - parent_spiffe_id linkage               |
|    - timestamp ordering (with tolerance)    |
|    - Ed25519 signature math                 |
|                                             |
|    Fail -> 403 + chain.verify_failed audit  |
+----------------+----------------------------+
                 v
+---------------------------------------------+
| 3. is_cross_domain_boundary() logging       |   Phase 1 (D7)
|    (audit-only, not enforcement)            |
+----------------+----------------------------+
                 v
+---------------------------------------------+
| 4. IdentityBindingVerifier (if enabled)     |   Phase 1.5
|    For every hop:                           |
|      verify_pin(spiffe_id, embedded_pem)    |
|      match     -> accept                    |
|      no_pin    -> TOFU via agent card       |
|      mismatch  -> walk rotation chain       |
|      revoked   -> always reject             |
|                                             |
|    Fail -> 403 + chain.identity_binding_    |
|             failed audit                    |
+----------------+----------------------------+
                 v
          continue to handler

text

Phase 1.5 is gated behind EIGENOID_IDENTITY_BINDING_ENABLED=true, default false -- matching the Phase 1 precedent set by EIGENOID_CHAIN_AWARE_PERMISSIONS. Operators flip it on once every peer on the network has discovery wired for every other peer (needed for TOFU to work).

What Phase 1.5 closes

Phase 1's own report called out the residual gap:

The embedded signer_public_key_pem is self-attested. An attacker who generates their own real Ed25519 keypair, signs a hop properly with it, embeds their own pubkey, and claims to be any SPIFFE ID will pass verification today. The math verifies. The identity binding does not exist.

Phase 1.5 closes it by pinning the pubkey observed for each SPIFFE ID on first contact (TOFU) and rejecting any later presentation that disagrees with the pin, unless a signed rotation attestation authorizes the change.

The pin lifecycle

 first hop from SPIFFE ID X      -->  resolver fetches X's agent card
                                      over mTLS; checks peer-cert SPIFFE
                                      ID equals X; extracts
                                      signingPublicKeyPem;
                                      compares to embedded pubkey;
                                      match -> PIN

 subsequent hop from X, same pem -->  pin MATCH, accept (cache-hit path,
                                      no resolver fetch)

 subsequent hop from X, diff pem -->  pin MISMATCH. Walk X's published
                                      rotation attestations:
                                      - each link signed by the OLD key
                                        listed on that link
                                      - verifier BFS old_pem -> new_pem
                                      - contiguous chain found -> UPDATE
                                        pin (source=rotation_attestation)
                                      - no chain -> 403 as possible forgery

 admin POST /admin/pubkey-pins/X/revoke
                                 --> all future hops from X rejected
                                      regardless of pubkey

text

Backing store: SQLite at $EIGENOID_HOME/agents/<name>/pubkey_pins.db, two tables (pubkey_pins, pin_rotation_history), both carrying a schema_version column from day one.

Rotation attestations

A RotationAttestation is a self-verifying document:

{
    "attestation_id": uuid_str,
    "spiffe_id":      str,
    "old_pubkey_pem": pem_str,
    "new_pubkey_pem": pem_str,
    "rotated_at":     iso8601_str,
    "attestation_signature": base64str,   # Ed25519 signature by the OLD private key
}

python

The signature covers JCS canonical bytes of the first five fields, so a verifier holding only the old pubkey can validate the attestation without any out-of-band data. Each agent persists its own history at $EIGENOID_HOME/agents/<name>/rotation_attestations.db and serves the recent N (default 64) at /.well-known/pubkey-rotation.json.

The eigenoid pubkeys rotate --agent <name> CLI atomically:

1. Loads the current signing identity (AgentIdentity).
2. Generates a new Ed25519 keypair.
3. Builds the attestation and signs it with the old private key.
4. Persists the attestation.
5. Atomically replaces signing.key on disk (temp file + fsync + rename, 0o600).

Gotcha: the running agent process loaded its AgentIdentity at startup. A docker restart (or equivalent) is needed for the agent card to serve the new pubkey. This is operational-by-design -- key rotation is a deliberate maintenance event.

The manual-admin pin path

For enterprise deployments that prefer to pre-provision pubkey pins out-of-band instead of trusting TOFU, the admin API writes pins directly:

POST /admin/pubkey-pins              body: {spiffe_id, pubkey_pem, source: "manual_admin"}
GET  /admin/pubkey-pins              list all pins (optionally filtered)
POST /admin/pubkey-pins/<sid>/revoke body: {reason}

text

Gated by EIGENOID_ADMIN_EMAILS (comma-separated OIDC email allowlist). A ManualAdminPubkeyResolver sits in front of AgentCardPubkeyResolver inside the default ChainedPubkeyResolver, so admin-provisioned pubkeys preempt TOFU.

Identity binding in the middleware

File: eigenoid/delegation/identity_binding.py.

class IdentityBindingVerifier:
    async def verify_chain_binding(self, chain: list[dict]) -> tuple[bool, str | None]:
        for i, hop in enumerate(chain):
            spiffe_id = hop.get("spiffe_id")
            presented = hop.get("signer_public_key_pem")
            if not spiffe_id or not presented:
                continue                           # nothing to bind for unsigned stubs
            result = await self._pin_store.verify_pin(spiffe_id, presented)
            if result.status == "match":           continue
            if result.status == "revoked":         raise ...
            if result.status == "no_pin":          await self._tofu(...)
            if result.status == "mismatch":        await self._handle_mismatch(...)
        return True, None

python

Wire-up point: _AuthMiddleware.dispatch calls verify_chain_binding() after verify_chain_dicts() and before setting request.state.auth_context. Rejections return 403 and emit chain.identity_binding_failed audit events.

Approval as signed evidence (Phase 1 D5)

Phase 1's approval path re-uses the OIDC id_token as cryptographic evidence of who approved what:

1. The approval server's Bearer handler captures the incoming Authorization: Bearer <id_token> and persists both the token and its issuer into the approval_info block on the hop.
2. _enforce_approval downstream re-validates the token's signature, expiry, and issuer against the agent's OIDC config, then asserts token.email == approval_info.approved_by.
3. An email mismatch raises PermissionError("email mismatch") before the skill runs.

The cryptographic check sits on top of the wire-compat check -- older records without approved: True are still honored, but newer records go through the signature + email validation path.

Chain-aware permissions (Phase 1 D6)

PermissionManager.compute_effective_permissions(chain, role_map) intersects role permissions along a multi-hop chain when EIGENOID_CHAIN_AWARE_PERMISSIONS=true. Default false to preserve Phase 0 / single-hop behavior. Single-hop chains take the existing fast path in either mode.

Cross-domain audit (Phase 1 D7)

is_cross_domain_boundary(prev_hop, next_hop) returns True whenever two adjacent hops belong to different trust domains. The middleware emits a chain.cross_domain_hop audit event per crossing. This is logging only -- the policy knob lives in Phase 3. Operators can inspect the audit log today to see where cross-domain delegation is actually happening before deciding allow/deny policy.

Operator surfaces summary

Surface	Phase	Auth	Purpose
GET /.well-known/agent-card.json	0 (card) + 1 (signingPublicKeyPem field)	none	Peer discovers this agent's signing pubkey
GET /.well-known/eigenoid-instance.json	0	none	Peer learns instance UUID + instance pubkey
GET /.well-known/pubkey-rotation.json	1.5	none	Peer walks this agent's rotation history
GET /admin/pubkey-pins	1.5	OIDC admin	List pins held by this agent
POST /admin/pubkey-pins	1.5	OIDC admin	Pre-provision a pin (source=manual_admin)
POST /admin/pubkey-pins/<sid>/revoke	1.5	OIDC admin	Hard-reject all future hops from <sid>
eigenoid pubkeys rotate --agent <name>	1.5	local filesystem access to $EIGENOID_HOME	Rotate this agent's signing.key, emit signed attestation

Feature flags

Flag	Default	Effect when true
EIGENOID_CHAIN_TIMESTAMP_TOLERANCE	5.0 (seconds)	Phase 1 -- permits adjacent hops' timestamps to regress by up to N seconds (absorbs inter-host clock skew)
EIGENOID_CHAIN_AWARE_PERMISSIONS	false	Phase 1 -- multi-hop chains get their role permissions intersected along the chain
EIGENOID_IDENTITY_BINDING_ENABLED	false	Phase 1.5 -- inbound hops' pubkeys are cross-checked against the pin store, 403 on failure
EIGENOID_ADMIN_EMAILS	(empty)	Phase 1.5 -- comma-separated OIDC email allowlist for the /admin/* routes
EIGENOID_PUBKEY_CACHE_TTL	60 (seconds)	Phase 1.5 -- in-process cache TTL on agent-card resolver lookups

Demos

Path	What it proves	Runtime
examples/testing_capability/phase_0/run_demo.py	Phase 0 primitives (keystore, typed hops, instance identity, from_dict forward-compat)	less than 1 s
examples/testing_capability/phase_1/run_demo.py	Phase 1 in-process: real signatures, middleware 403 on tamper, timestamp tolerance	less than 1 s
examples/testing_capability/phase_1/run_scenarios.py	Phase 1 D5 (OIDC approval) + D6 (chain-aware permissions)	less than 1 s
examples/testing_capability/phase_1/ + eigenoid up --isolate	Phase 1 over real mTLS + SPIRE in Docker	~30 s
examples/testing_capability/phase_1_5/run_demo.py	All 8 Phase 1.5 contracts via the real primitives	less than 1 s
Same phase_1/ Docker stack with EIGENOID_IDENTITY_BINDING_ENABLED=true	Phase 1.5 over real mTLS + SPIRE	~1 min

The phase_1_5 README has a full operator playbook for the Docker path, including the reverse-discovery env vars TOFU needs and a one-shot Python forgery script that signs with an attacker-owned keypair so Phase 1 math passes (the contrast that defines the Phase 1.5 contract).

Regression coverage

Test file	Phase	Count
tests/test_keystore.py	0	keystore load/generate, chmod fixup
tests/test_paths.py	0	per-agent state dirs
tests/test_instance_identity.py	0	instance doc serialize/deserialize
tests/test_spiffe_id.py	0	typed SpiffeID
tests/test_chain_id_threading.py	0	chain_id propagation through A2A metadata
tests/test_phase1_delegation.py	1	9 tests -- signatures, middleware 403, timestamp tol, OIDC approval, chain-aware perms
tests/test_phase1_5_identity_binding.py	1.5	19 tests -- pin lifecycle, TOFU, rotation walk, admin pins, revocation, DNS-hijack, cache TTL, middleware integration

Total: 28 regression tests pass across Phase 1 + Phase 1.5 end to end; Phase 0 surfaces have never regressed. Run:

python -m pytest tests/test_phase1_delegation.py \
                 tests/test_phase1_5_identity_binding.py -v

bash

Further reading

• Architecture overview -- overall system design, component responsibilities
• Dual-mode authentication -- mTLS vs JWT-SVID vs OIDC, middleware evaluation order
• Discovery providers -- transport hint selection, agent endpoint resolution
• SPIRE federation -- cross-trust-domain SPIRE federation
• Enterprise governance testing guide -- manual verification procedures for operators

Discovery providers

Source: docs/DISCOVERY.md

Discovery is a separate concern from authentication. Eigenoid provides a pluggable discovery interface that resolves agent identifiers to routable endpoints with transport hints.

Transport hints

Each resolved endpoint includes a transport hint that drives auth mechanism selection:

• "direct" -- use mTLS (X.509-SVID client certificate + trust bundle, no Bearer header)
• "proxied" -- use JWT-SVID Bearer token (trust bundle only for TLS verification, no client cert)

These map to two distinct SSL contexts: client_ssl_context (mTLS with client cert) and verify_only_ssl_context (trust bundle only). See Dual-mode authentication for details.

URL scheme note: When EIGENOID_MTLS_ENABLED=true, all agents serve HTTPS. Inter-agent URLs must use https://. The container orchestrator auto-upgrades http://eigenoid-agent-* URLs to https:// when mTLS is enabled.

Built-in providers

StaticDiscovery

Reads agent endpoints from config files, environment variables, or programmatic registration.

from eigenoid import StaticDiscovery

discovery = StaticDiscovery(trust_domain="eigenoid.local")

# Programmatic registration
discovery.register("booking-agent", "http://localhost:8001", transport="direct")
discovery.register("external-agent", "https://gateway.example.com/agents/ext", transport="proxied")

# Resolve
endpoint = await discovery.resolve("booking-agent")
print(endpoint.url)        # "http://localhost:8001"
print(endpoint.transport)  # "direct"
print(endpoint.spiffe_id)  # "spiffe://eigenoid.local/agent/booking-agent"

python

Environment variables:

EIGENOID_AGENT_BOOKING_AGENT=http://localhost:8001
EIGENOID_AGENT_BOOKING_AGENT_TRANSPORT=direct

bash

YAML config (eigenoid.yaml):

agents:
  booking-agent:
    url: http://localhost:8001
    transport: direct
    spiffe_id: spiffe://eigenoid.local/agent/booking-agent
  external-agent:
    url: https://gateway.example.com/agents/ext
    transport: proxied

yaml

AgentCardDiscovery

Fetches /.well-known/agent-card.json from known base URLs (A2A-native discovery).

from eigenoid import AgentCardDiscovery

discovery = AgentCardDiscovery(
    base_urls=[
        "http://localhost:8001",
        "http://localhost:8002",
        "http://localhost:8003",
    ],
)

endpoint = await discovery.resolve("booking-agent")
print(endpoint.agent_card)  # Full A2A agent card

python

ChainedDiscovery

Tries multiple providers in order. First result wins.

from eigenoid import ChainedDiscovery, StaticDiscovery, AgentCardDiscovery

discovery = ChainedDiscovery([
    StaticDiscovery(),                  # Check config first
    AgentCardDiscovery(base_urls=[...]), # Then try agent cards
])

python

Custom providers

Implement DiscoveryProvider to add custom discovery:

from eigenoid import DiscoveryProvider, AgentEndpoint

class ConsulDiscovery(DiscoveryProvider):
    """Discover agents via Consul service registry."""

    async def resolve(self, agent_id: str) -> AgentEndpoint | None:
        service = await consul.health.service(agent_id)
        if service:
            return AgentEndpoint(
                url=f"https://{service.address}:{service.port}",
                transport="direct",
                spiffe_id=f"spiffe://my-domain/agent/{agent_id}",
            )
        return None

    async def list_agents(self) -> list[AgentEndpoint]:
        services = await consul.catalog.services()
        return [
            AgentEndpoint(url=s.url, transport="direct")
            for s in services
        ]

python

AgentEndpoint fields

Field	Type	Description
url	str	Routable endpoint URL
transport	str	"direct" or "proxied"
agent_card	dict | None	A2A agent card if available
spiffe_id	str | None	Agent's SPIFFE ID
metadata	dict	Provider-specific metadata

Dual-mode authentication (mTLS + JWT-SVID)

Source: docs/DUAL_AUTH.md

Eigenoid supports two authentication mechanisms simultaneously. The caller selects which to use based on its transport path to the target agent.

When to use each mechanism

Scenario	Mechanism	Why
Container isolation (eigenoid up --isolate)	mTLS (X.509-SVID)	Each container gets an X.509-SVID from the shared SPIRE agent socket. Agents serve HTTPS with mutual TLS by default.
Direct agent-to-agent (same network, mTLS enabled)	mTLS (X.509-SVID)	Strongest auth. Mutual TLS validates both parties at the transport layer. Peer certificates must contain a spiffe:// SAN URI.
Through L7 proxy, API gateway, or load balancer	JWT-SVID Bearer	Proxies terminate TLS, so client certs don't survive the hop. JWT travels in the HTTP header.
Cross-network (federated trust domains)	Either	After SPIRE federation, both mechanisms work across trust domains.

Architecture

+-----------------------------------------------------+
|                  Transport Layer                     |
|                                                     |
|  Direct connectivity --> mTLS (X.509-SVID)         |
|  Through gateway/proxy --> JWT-SVID Bearer token    |
|                                                     |
+-----------------------------------------------------+
|                 Identity Layer                       |
|                                                     |
|  SPIRE Workload API provides both:                  |
|    - X.509-SVID (cert + key + trust bundle)        |
|    - JWT-SVID (signed token + trust bundle)        |
|                                                     |
+-----------------------------------------------------+
|                Application Layer                     |
|                                                     |
|  auth_context populated from whichever mechanism    |
|  authenticated the caller. Same shape either way.   |
|                                                     |
|  Delegation chains use JWT-SVIDs in A2A metadata    |
|  regardless of transport auth mechanism.            |
+-----------------------------------------------------+

text

auth_context shape

Regardless of which mechanism authenticated the caller, downstream code receives the same auth_context structure:

auth_context = {
    "authenticated": True,           # bool: was caller authenticated?
    "method": "mtls",               # "mtls" | "jwt-svid" | None
    "spiffe_id": "spiffe://eigenoid.local/agent/caller",
    "trust_domain": "eigenoid.local",
    "claims": {                      # JWT claims if jwt-svid, cert metadata if mtls
        "sub": "spiffe://eigenoid.local/agent/caller",
        # ...
    },
    "delegation_chain": [],          # List of delegation hops
    "delegation_depth": 0,           # Number of hops
}

python

When neither mechanism authenticates (and require_auth=False):

auth_context = {
    "authenticated": False,
    "method": None,
    "spiffe_id": None,
    "trust_domain": None,
    "claims": {},
    "delegation_chain": [],
    "delegation_depth": 0,
}

python

Middleware evaluation order

1. Check mTLS: Was a client certificate presented and validated at the TLS layer?
   • Yes: extract SPIFFE ID from X.509 SAN, set method="mtls"
   • No: continue to step 2
2. Check JWT: Is there a valid Authorization: Bearer header?
   • Yes: validate JWT-SVID against trust bundle, set method="jwt-svid"
   • Invalid JWT is always rejected (401), even when require_auth=False
   • No Bearer header: continue to step 3
3. Neither mechanism authenticated:
   • require_auth=True: reject with 401
   • require_auth=False: set authenticated=False, continue
4. Verify delegation chain (Phase 1): run verify_chain_dicts() on the inbound chain -- hop_index monotonicity, parent-spiffe-id linkage, timestamp ordering with tolerance (EIGENOID_CHAIN_TIMESTAMP_TOLERANCE, default 5.0 s), and Ed25519 signature math for every hop. Any failure returns 403 with chain.verify_failed audit event, before set_context.
5. Cross-domain audit (Phase 1 D7): emit a chain.cross_domain_hop event for every adjacent-hop trust-domain change. Logging only, no enforcement.
6. Identity-binding check (Phase 1.5, gated by EIGENOID_IDENTITY_BINDING_ENABLED=true): for every hop, verify the embedded signer_public_key_pem against the PubkeyPinStore. Possible outcomes per hop:
   • match -- continue
   • no_pin -- TOFU via AgentCardPubkeyResolver (peer-cert SPIFFE-ID binding check on the mTLS fetch), pin on success, 403 on mismatch
   • mismatch -- walk the peer's /.well-known/pubkey-rotation.json attestations with RotationVerifier, update pin on success, 403 on failure
   • revoked -- always 403

Rejections emit chain.identity_binding_failed.

Important: If a Bearer token is present, it MUST validate. Steps 4-6 run regardless of which auth mechanism passed -- a legitimate mTLS or JWT peer cannot smuggle a forged delegation chain. The only path to unauthenticated access is presenting neither a client cert nor a Bearer token, with require_auth=False.

See Delegation security model for the wire format of the hops that steps 4-6 inspect and the pin-store lifecycle they enforce.

Accessing auth info in skill handlers

from eigenoid import Agent, skill, context

class MyAgent(Agent):
    name = "my-agent"

    @skill("sensitive_action")
    async def sensitive_action(self, data: dict):
        # Check authentication method
        if context.auth_method == "mtls":
            print("Caller authenticated via mTLS")
        elif context.auth_method == "jwt-svid":
            print("Caller authenticated via JWT-SVID")

        # Check trust domain (useful for federation)
        if context.trust_domain != "eigenoid.local":
            print(f"Cross-domain call from {context.trust_domain}")

        return {"processed_by": context.caller_spiffe_id}

python

Outbound call auth selection

When one agent calls another, the transport mode determines which SSL context and auth header to use:

# Direct call (mTLS) -- client cert + trust bundle, no Bearer header
result = await self.call(
    agent="https://booking-agent:8001",
    skill="book",
    transport="direct",
    **kwargs,
)

# Through gateway (JWT-SVID) -- trust bundle only, no client cert, Bearer header
result = await self.call(
    agent="https://gateway.example.com/agents/booking",
    skill="book",
    transport="proxied",
    **kwargs,
)

# Auto-detect (default)
result = await self.call(agent="https://booking-agent:8001", skill="book", **kwargs)

python

The transport flag selects between two distinct SSL contexts:

Transport	SSL context	Auth header
direct	client_ssl_context -- trust bundle + X.509-SVID client cert (mTLS)	None
proxied	verify_only_ssl_context -- trust bundle only, no client cert	Authorization: Bearer <jwt-svid>

This separation is critical: the proxied context must NOT present a client certificate, because in production an L7 proxy terminates TLS and the client cert never reaches the agent. If the client always sent a cert, the JWT path would be untestable.

Auto-detection heuristics:

• Hostnames starting with gateway., proxy., api., lb., ingress. -- proxied
• All other hostnames -- direct

Delegation chains

Delegation chains are carried in A2A message metadata (not HTTP headers) so they survive L7 proxies that strip custom headers. Phase 0 unified the hop schema (spiffe_id / parent_spiffe_id, chain_id, version); Phase 1 added signer_public_key_pem + signature:

{
  "params": {
    "message": {
      "metadata": {
        "spiffe": {
          "version": "1.0",
          "chain": [
            {
              "hop_index": 0,
              "spiffe_id": "spiffe://eigenoid.local/agent/a",
              "parent_spiffe_id": null,
              "timestamp": "2026-04-22T22:53:06.827533+00:00",
              "action": "ask",
              "chain_id": "81dce0a6-ee05-4cb1-925a-3cb69f4db104",
              "version": 2,
              "signer_public_key_pem": "-----BEGIN PUBLIC KEY-----\nMCow...\n-----END PUBLIC KEY-----\n",
              "signature": "JVXRXpU32rA..."
            },
            {
              "hop_index": 1,
              "spiffe_id": "spiffe://eigenoid.local/agent/b",
              "parent_spiffe_id": "spiffe://eigenoid.local/agent/a",
              "timestamp": "2026-04-22T22:53:07.015102+00:00",
              "action": "lookup_amenity",
              "chain_id": "81dce0a6-ee05-4cb1-925a-3cb69f4db104",
              "version": 2,
              "signer_public_key_pem": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n",
              "signature": "..."
            }
          ],
          "currentDepth": 2,
          "maxDepth": 5
        }
      }
    }
  }
}

json

For backward compatibility, X-Delegation-Chain and X-Delegation-Depth HTTP headers are also sent but A2A metadata takes priority.

Container isolation with SPIRE

When running eigenoid up --isolate, Eigenoid starts a self-contained SPIRE server and agent inside Docker alongside the agent containers. All containers share the same SPIRE agent socket and trust domain. Authentication uses mTLS with X.509-SVIDs by default, with JWT-SVID as a fallback for proxied transport.

How it works:

1. A Docker volume (spire-sockets) shares the SPIRE agent socket across all containers
2. Each agent container runs as a unique UID (starting at 10001, incrementing per agent)
3. SPIRE uses the unix workload attestor to match container UIDs to registered SPIFFE IDs
4. Agents serve HTTPS using X.509-SVIDs from SPIRE. Client certificates prove identity at the transport layer (mTLS)
5. For proxied transport (through gateways/load balancers), agents fall back to JWT-SVID Bearer tokens
6. --pid host is required on all containers for the unix attestor to work
7. Peer certificates must contain a spiffe:// SAN URI -- certificates without one are rejected

How eigenoid up --isolate sets it up

The generated Docker Compose configuration includes:

volumes:
  spire-sockets:    # Shared SPIRE agent socket
  spire-data:       # SPIRE server data
  spire-agent-data: # SPIRE agent state

networks:
  eigenoid-net:
    driver: bridge

services:
  eigenoid-spire-server:
    image: ghcr.io/spiffe/spire-server:1.9.6
    volumes:
      - spire-data:/opt/spire/data/server
      - ./spire/server.conf:/opt/spire/conf/server/server.conf:ro

  eigenoid-spire-agent:
    image: ghcr.io/spiffe/spire-agent:1.9.6
    pid: host    # Required for unix workload attestor
    volumes:
      - spire-sockets:/opt/spire/sockets
      - spire-agent-data:/opt/spire/data/agent

  eigenoid-agent-my-agent:
    image: eigenoid-agent:local
    user: "10001:10001"    # Unique UID for SPIRE attestation
    pid: host
    volumes:
      - spire-sockets:/opt/spire/sockets:ro
    environment:
      SPIFFE_ENDPOINT_SOCKET: unix:///opt/spire/sockets/agent.sock
    networks:
      - eigenoid-net

yaml

SPIRE workload registration

Eigenoid automatically registers each agent with the container SPIRE server using unix UID selectors:

# Automatically run by eigenoid up --isolate
spire-server entry create \
  -socketPath /opt/spire/data/server/api.sock \
  -spiffeID spiffe://eigenoid.local/agent/my-agent \
  -parentID spiffe://eigenoid.local/spire/agent/join_token/<uuid> \
  -selector unix:uid:10001

bash

Running a demo client against container agents

To test from outside the agent containers, run a client container on the same Docker network. See examples/metadata_skills/README.md for the full walkthrough.

mTLS transport details

Agents communicate over mTLS (mutual TLS) using X.509-SVIDs from SPIRE. Both the server and client present certificates, proving identity at the transport layer. No JWT token travels over the wire for direct agent-to-agent calls.

TLSContextManager

TLSContextManager builds and manages three SSL contexts from each X.509-SVID:

Context	Purpose	Contents
server_ssl_context	Serve HTTPS (passed to uvicorn)	Server cert + key + trust bundle, CERT_OPTIONAL
client_ssl_context	Direct (mTLS) outbound calls	Client cert + key + trust bundle, check_hostname=False
verify_only_ssl_context	Proxied (JWT) outbound calls	Trust bundle only -- no client cert

All contexts are built in-memory from SPIRE's X.509-SVID (cert chain + private key + trust bundle). Certificate rotation is automatic -- a background watcher rebuilds all three contexts when SPIRE rotates certificates (default: every hour).

Why check_hostname=False

SPIFFE X.509-SVIDs use URI SANs (spiffe://eigenoid.local/agent/echo-agent), not DNS SANs. Python's default SSL hostname verification checks the hostname against DNS SANs, which fails. Trust is established via the SPIRE trust bundle and the SPIFFE ID in the certificate's URI SAN, not via DNS hostname matching.

Server-side

ssl.CERT_OPTIONAL allows both mTLS and JWT-only callers. The auth middleware checks for a client cert first, then falls back to JWT validation. Peer certificates are extracted via PeerCertMiddleware, which accesses the SSL transport through uvicorn's internal request cycle object.

Client-side

• Direct: httpx.AsyncClient(verify=client_ssl_context) presents the X.509-SVID as a client certificate and validates the server's certificate against the SPIRE trust bundle.
• Proxied: httpx.AsyncClient(verify=verify_only_ssl_context) validates the server's certificate but does NOT present a client cert. Authentication is via Authorization: Bearer <jwt-svid>.

Container URL scheme

When EIGENOID_MTLS_ENABLED=true, all agents serve HTTPS. Inter-agent URLs must use https://. The container orchestrator (container.py) auto-upgrades http://eigenoid-agent-* URLs to https:// when mTLS is enabled, preventing misconfiguration.

Testing both transport modes

# mTLS mode (default) -- auth_method: "mtls"
docker run --rm -it \
  --network eigenoid-generated_eigenoid-net \
  --pid host \
  --user 10003:10003 \
  -v eigenoid-generated_spire-sockets:/opt/spire/sockets:ro \
  -e SPIFFE_ENDPOINT_SOCKET=unix:///opt/spire/sockets/agent.sock \
  -e ECHO_AGENT_URL=https://eigenoid-agent-echo-agent:9100 \
  -e CALLER_AGENT_URL=https://eigenoid-agent-caller-agent:9101 \
  eigenoid-agent:local \
  python /app/eigenoid-src/examples/metadata_skills/demo.py all

# JWT-SVID mode -- auth_method: "jwt-svid"
docker run --rm -it \
  --network eigenoid-generated_eigenoid-net \
  --pid host \
  --user 10003:10003 \
  -v eigenoid-generated_spire-sockets:/opt/spire/sockets:ro \
  -e SPIFFE_ENDPOINT_SOCKET=unix:///opt/spire/sockets/agent.sock \
  -e ECHO_AGENT_URL=https://eigenoid-agent-echo-agent:9100 \
  -e CALLER_AGENT_URL=https://eigenoid-agent-caller-agent:9101 \
  eigenoid-agent:local \
  python /app/eigenoid-src/examples/metadata_skills/demo.py --transport=proxied all

bash

SPIRE federation

Source: docs/FEDERATION.md

When agents in different SPIFFE trust domains need to talk, SPIRE federation lets each side's SPIRE server learn the other's CA roots so mTLS and JWT-SVID validation succeed across the trust-domain boundary. Eigenoid layers application-level enforcement on top so an agent only accepts cross-domain traffic from explicitly-federated peers.

This doc is the operator entry point. The original step-by-step manual acceptance runbook is preserved in Runbook -- Phase 3c manual acceptance.

How it works

Trust Domain A (alpha.local)         Trust Domain B (beta.local)
+------------------+                 +------------------+
|  SPIRE Server A  | <-- bundle ---> |  SPIRE Server B  |
|  federates_with: |     exchange    |  federates_with: |
|   beta.local     |  (SPIRE layer)  |   alpha.local    |
+--------+---------+                 +--------+---------+
         |                                    |
+--------+---------+                 +--------+---------+
|  Eigenoid Agent  |-- mTLS + audit -|  Eigenoid Agent  |
|  IdentityProvider|   + 3c policy   |  IdentityProvider|
|  ._federated_    |     gate        |  ._federated_    |
|   trust_domains  |                 |   trust_domains  |
|   = {beta.local} |                 |   = {alpha.local}|
+------------------+                 +------------------+

text

Two layers of trust:

1. SPIRE layer (Phase 3a + 3b) -- cross-loaded CA bundles let mTLS handshakes and JWT-SVID validation succeed for peers in federated domains. Without this, the TLS layer rejects the peer cert before the application sees it.
2. Application layer (Phase 3c) -- the agent's IdentityProvider._federated_trust_domains set is the source of truth for "is this peer trust domain federated?". The middleware and outbound call paths use that set to gate cross-TD chain hops and outbound calls. A peer in a federated TD is admitted; a peer in any other TD is rejected with HTTP 403 and an audited chain.cross_domain_rejected event.

The 3c policy is binary: federated or not. Per-peer scope reduction and skill allowlists are deferred to Phase 3d.

Setup

1. Configure eigenoid.yaml on each side

Add a federation: block. Start with federates_with: [] -- bundle exchange will populate it later.

# alpha-stack/eigenoid.yaml
version: "1"
name: alpha-stack

federation:
  trust_domain: "alpha.local"
  bundle_endpoint:
    address: "0.0.0.0"
    port: 8443
  public_bundle_endpoint_url: "https://host.docker.internal:8443"
  federates_with: []

agents:
  echo:
    type: custom
    port: 8001
    path: ./echo.py

yaml

The mirror config on the other side has trust_domain: beta.local and bundle endpoint port 8444. See tests/fixtures/federation/two_stack_alpha.yaml and two_stack_beta.yaml for working examples.

2. Bring both stacks up unfederated

( cd alpha-stack && eigenoid up --isolate )
( cd beta-stack  && eigenoid up --isolate )

bash

eigenoid up --isolate runs docker compose up -d --build (already detached), generates SPIRE server config with the federation { ... } HCL block from your yaml, exposes the bundle endpoint port, and registers workload entries.

3. Exchange bundles via the CLI

( cd alpha-stack && eigenoid federation export --output /tmp/alpha-bundle.json )
( cd beta-stack  && eigenoid federation export --output /tmp/beta-bundle.json  )

( cd alpha-stack && eigenoid federation import /tmp/beta-bundle.json )
( cd beta-stack  && eigenoid federation import /tmp/alpha-bundle.json )

bash

federation export reads the local SPIRE server's CA bundle and emits a JSON envelope with trust_domain, bundle_endpoint_url, bundle_endpoint_spiffe_id, and trust_bundle_pem.

federation import does three things atomically:

• Rewrites eigenoid.yaml to add the peer to federates_with.
• Persists the bundle under .eigenoid-generated/federation-bundles/<peer-td>.pem so it survives the next eigenoid up's down -v volume teardown.
• If the SPIRE server is running, immediately calls spire-server bundle set to load the bundle.

4. Recycle so federation HCL takes effect

( cd alpha-stack && eigenoid up --isolate )
( cd beta-stack  && eigenoid up --isolate )

bash

eigenoid up always begins with docker compose down -v (wipes SPIRE's volume). On the way back up, the startup hook apply_persisted_peer_bundles re-applies each sidecar .pem via spire-server bundle set before workload registration runs, so entry create -federatesWith spiffe://<peer-td> succeeds. SPIRE agents on each side then receive an X.509-SVID whose trust bundle includes the peer's CA roots.

5. Verify

docker exec alpha-stack-eigenoid-spire-server \
    /opt/spire/bin/spire-server bundle list \
    -socketPath /opt/spire/data/server/api.sock

bash

bundle list enumerates federated bundles. Seeing beta.local listed on the alpha side proves cross-loading worked.

docker exec alpha-stack-eigenoid-agent-echo printenv EIGENOID_FEDERATION_PEERS

bash

["beta.local"] confirms the runtime IdentityProvider integration (Phase 3c piece 1) -- the env var is plumbed by _build_agent_service and consumed in Agent.initialize to call configure_federation.

What changes for agents after federation

Before Phase 3c, federation was purely a SPIRE-layer concern: agents neither needed nor could express any policy about which TDs they trusted at the application layer. Phase 3c added enforcement.

With federation configured:

• mTLS / JWT-SVID validation succeeds for peers from federated TDs (SPIRE layer, unchanged from 3a/3b).
• IdentityProvider._federated_trust_domains is populated at startup from EIGENOID_FEDERATION_PEERS (env var emitted by the compose generator from federation.federates_with). The new is_trust_domain_federated() predicate is the canonical check.
• AgentCardPubkeyResolver routes cross-TD lookups through a per-TD SSL context (X509SVID.create_client_ssl_context_for_trust_domain) pinned to that TD's bundle. Same-TD lookups use the existing client_ssl_context. Lookups for unknown TDs fail closed in code with no network round-trip.
• The inbound auth middleware (eigenoid/agent.py:_AuthMiddleware, eigenoid/core/auth.py:AuthMiddleware) gates every cross-TD hop in a delegation chain against is_trust_domain_federated. Hops into a non-federated TD return HTTP 403 and emit chain.cross_domain_rejected. Hops into a federated TD continue to emit the legacy chain.cross_domain_hop audit event and pass through to the handler.
• The outbound call path (Agent._enforce_outbound_cross_domain, called from Agent.call and Agent.stream_call) refuses to start a request whose target is in a non-federated TD. The check runs before DNS / TCP / TLS, so a non-federated target never sees the attempt -- that's the load-bearing security property of Phase 3c.

Without a federation: block in eigenoid.yaml:

Behavior is byte-identical to pre-3c single-TD operation. No env var is emitted, no cross-TD path is exercised, no policy is enforced. This is the 3a byte-identical contract; it's locked in by tests/test_isolate_byte_identical.py and tests/test_phase_3c_cross_td_enforcement.py::test_piece1_build_agent_service_no_env_for_single_td.

Programmatic access

For embedding outside the standard eigenoid up flow:

from eigenoid.federation import configure_federation

configure_federation(
    agent.identity_provider,
    federated_domains=["partner.com"],
)

python

configure_federation is what Agent.initialize calls when EIGENOID_FEDERATION_PEERS is set. The lower-level agent.identity_provider.add_federated_trust_domain("partner.com") is also available.

Audit events

Event	Emitted by	When	Fields
chain.cross_domain_hop	inbound middleware	every cross-TD hop in the chain (logging only -- pre-3c behavior preserved)	caller, from_spiffe_id, to_spiffe_id, hop_index
chain.cross_domain_rejected	inbound middleware (3c)	the destination TD of a cross-TD hop is not in _federated_trust_domains	chain_id, hop_index, from_trust_domain, to_trust_domain, reason="not_federated"
chain.cross_domain_rejected	outbound Agent._enforce_outbound_cross_domain (3c)	outbound target is in a non-federated TD	the same plus skill, target_spiffe_id

The legacy chain.cross_domain_hop event is preserved exactly so existing observability dashboards keep working. The new chain.cross_domain_rejected event is purely additive and is the canonical signal of a federation policy denial.

Security notes

• Federation is bidirectional at the SPIRE layer (each side must list the other in federates_with) and at the application layer (each side's IdentityProvider._federated_trust_domains must contain the peer). The compose generator and bundle-import CLI keep these in lock-step; manual editing of eigenoid.yaml followed by eigenoid up is the supported way to add or remove peers.
• Each domain retains its own CA -- federation cross-loads CA certificates, not key material.
• Removing a peer from federation.federates_with and recycling takes effect on the next up. In-flight chains already past the middleware are not actively cancelled; revocation of in-flight delegations is Phase 3d.
• Per-peer scope reduction (e.g. "trust beta.local only for read_* skills"), skill allowlists, and pairing flows are explicitly not in 3c -- they are deferred to 3d. The 3c policy is binary: federated or not.

What 3c does NOT do (deferred to 3d)

Phase 3d (friend-request bootstrap, /admin/federations UI, per-peer policy, revocation) ships these. Summary of 3c's exclusions:

• No per-peer policy or scope reduction.
• No /admin/federations HTTP routes / pairing UI.
• No _system.federate reserved skill.
• No federation.cross_domain_verified or federation.revoked_peer audit events.
• No automated bundle exchange / pairing codes.
• No revocation of already-mid-flight chains.
• No hot reload of federation config -- adding a peer requires eigenoid down && eigenoid up --isolate (or just eigenoid up again, which does the down internally).

See also

• Part 1 of this document -- archived per-phase reports: schema plumbing (3A), SPIRE-level activation (3B), application-layer enforcement (3C), friend-request bootstrap UX (3D); includes the original Phase 3c manual acceptance runbook.
• tests/test_phase_3c_cross_td_enforcement.py -- 11 in-process regression tests for the three pieces.

Quick start guide

Source: docs/QUICKSTART.md

This guide will help you get started with Eigenoid -- governed communication for multi-agent AI. Eigenoid traces, scopes, and gates every agent-to-agent interaction so your multi-agent system can move from staging to production.

Prerequisites

• Python 3.9+
• Docker (for eigenoid up --isolate -- the recommended path)
• Linux or macOS

Installation

pip install -e .

bash

Quickest path: container isolation (recommended)

The fastest way to see Eigenoid in action with full SPIRE authentication is eigenoid up --isolate. This starts a self-contained SPIRE server, SPIRE agent, and your agents inside Docker -- no host SPIRE installation needed.

cd examples/metadata_skills
eigenoid up --isolate

bash

Then follow the step-by-step instructions in examples/metadata_skills/README.md to run the demo client inside a container on the same Docker network. This exercises:

• Metadata-based skill routing (metadata.skill_id)
• mTLS authentication via X.509-SVIDs (default transport)
• JWT-SVID authentication via Bearer tokens (--transport=proxied)
• Agent-to-agent delegation chains with mTLS
• Full auth context propagation (auth_method, caller_spiffe_id, trust_domain)

Alternative: manual SPIRE setup (advanced)

If you need to run SPIRE on the host (e.g., for production or custom configurations):

1. Install SPIRE

cd eigenoid
sudo bash scripts/install_spire.sh

bash

2. Configure SPIRE

sudo cp config/spire/server.conf /opt/spire/conf/server/
sudo cp config/spire/agent.conf /opt/spire/conf/agent/

bash

3. Start SPIRE services

sudo systemctl start spire-server
sudo systemctl start spire-agent

bash

4. Register agents

bash scripts/register_agents.sh

bash

Note: When running agents as host processes (without --isolate), the agents and any client must all connect to the same host SPIRE agent socket. If they connect to different SPIRE instances (or none at all), authentication will fail silently -- responses will show auth_method: None. Use eigenoid up --isolate to avoid this entirely.

Running examples

Metadata skills demo (recommended first example)

The examples/metadata_skills/ directory is the most complete example. It includes two agents, a demo client, and 13 unit tests:

# Unit tests (no SPIRE needed)
cd examples/metadata_skills
python test_routing.py

# Full demo with SPIRE auth (requires eigenoid up --isolate)
# See examples/metadata_skills/README.md for container setup steps

bash

Delegation security demos (Phase 0 / 1 / 1.5)

The examples/testing_capability/ directory has one runnable demo per delegation-security phase. Each runs in under a second, needs no Docker, and exercises the real primitives.

# Phase 0 -- forward-compat hop schema + keystore + instance identity
python examples/testing_capability/phase_0/run_demo.py

# Phase 1 -- real Ed25519 hop signatures, middleware 403 on tamper
python examples/testing_capability/phase_1/run_demo.py

# Phase 1.5 -- identity binding, TOFU, rotation attestations, the
# real-keypair forgery test (definition of done for Phase 1.5)
python examples/testing_capability/phase_1_5/run_demo.py

bash

The full operator playbook -- including the Docker stack with mTLS, the eigenoid pubkeys rotate CLI, and a one-shot attack script -- lives in examples/testing_capability/phase_1_5/README.md. See Delegation security model for the consolidated security reference.

Basic delegation

python examples/basic_delegation.py

bash

Demonstrates human-to-agent delegation, JWT-SVID creation, and delegation chain tracking.

Permission reduction

python examples/multi_level_delegation.py

bash

Shows how permissions are progressively reduced at each delegation level.

Approval workflow

python examples/approval_flow.py

bash

Demonstrates human approval for high-value transactions.

Depth limits

python examples/depth_limit.py

bash

Shows enforcement of maximum delegation depth.

Testing

Run the test suite:

pytest

bash

With coverage:

pytest --cov=eigenoid --cov-report=html

bash

Phase progression

The examples/testing_capability/ directory has end-to-end demos for each cumulative security layer. Run them in order to see how Eigenoid incrementally tightens guarantees:

Phase	What it adds	Demo
0	Forward-compat foundations (chain envelope, instance identity)	examples/testing_capability/phase_0/
1	Real Ed25519 hop signatures over JCS bytes	examples/testing_capability/phase_1/
1.5	Pubkey pin store + signed rotation; binds SPIFFE ID to pubkey	examples/testing_capability/phase_1_5/
2	Grants + capabilities (authorization layer)	examples/testing_capability/phase_2/

Per-phase deliverables and design notes are archived in Part 1 of this document.

Next steps

• Read the Architecture overview
• See Part 1 of this document for the per-phase archive (Phase 2 operator guide is under Phase 2)
• Review Security Considerations
• Explore Configuration Options
• Check out the API Reference

Troubleshooting

SPIRE agent connection issues

# Check agent status
sudo systemctl status spire-agent

# Check socket
ls -la /opt/spire/run/agent/api.sock

# View logs
sudo journalctl -u spire-agent -f

bash

Permission errors

Ensure the spire user has proper permissions:

sudo chown -R spire:spire /opt/spire
sudo chmod 755 /opt/spire/run/agent

bash

Support

• GitHub Issues: Report bugs
• Documentation: Full docs

Enterprise governance testing guide

Source: docs/TESTING_GUIDE.md

Manual testing guide for verifying Eigenoid governance in live Docker containers.

Prerequisites

# Start the SPIRE infrastructure and build containers
eigenoid up --isolate

bash

1. Verify SPIFFE IDs and UIDs per container

Each agent runs as a unique UID with a corresponding SPIFFE ID.

# Check UID inside each container
docker exec eigenoid-agent-gateway-agent id
# Expected: uid=10001(eigenoid) gid=10001(eigenoid)

docker exec eigenoid-agent-processor-agent id
# Expected: uid=10002(eigenoid) gid=10002(eigenoid)

docker exec eigenoid-agent-archiver-agent id
# Expected: uid=10003(eigenoid) gid=10003(eigenoid)

# Verify SPIFFE IDs via SPIRE
docker exec eigenoid-spire-server spire-server entry show -socketPath /opt/spire/data/server/api.sock
# Expected output includes:
#   Entry ID: ...
#   SPIFFE ID: spiffe://eigenoid.local/agent/gateway-agent
#   Selector: unix:uid:10001
#
#   Entry ID: ...
#   SPIFFE ID: spiffe://eigenoid.local/agent/processor-agent
#   Selector: unix:uid:10002
#
#   Entry ID: ...
#   SPIFFE ID: spiffe://eigenoid.local/agent/archiver-agent
#   Selector: unix:uid:10003

bash

Pass: Each agent has a unique UID, and SPIRE entries bind each UID to the correct SPIFFE ID. Fail: Duplicate UIDs or missing SPIRE entries.

2. Query audit logs

Audit events are stored in SQLite inside each container.

# List all audit events for an agent
docker exec eigenoid-agent-worker-agent \
  sqlite3 /app/data/audit.db \
  "SELECT timestamp, event_type, json_extract(data, '$.caller'), json_extract(data, '$.skill') FROM events ORDER BY timestamp DESC LIMIT 20;"

# Expected output:
# 2024-01-15T10:30:00.123Z|permission.allowed|spiffe://eigenoid.local/agent/manager-agent|status_check
# 2024-01-15T10:30:01.456Z|delegation.scope_checked|spiffe://eigenoid.local/agent/manager-agent|deploy_production
# 2024-01-15T10:30:02.789Z|permission.denied|spiffe://eigenoid.local/agent/rogue-agent|deploy_production

# Count events by type
docker exec eigenoid-agent-worker-agent \
  sqlite3 /app/data/audit.db \
  "SELECT event_type, COUNT(*) FROM events GROUP BY event_type;"

# Expected:
# auth.success|5
# permission.allowed|3
# permission.denied|1
# delegation.scope_checked|4

bash

Pass: Events include permission.allowed, permission.denied, delegation.scope_checked, and auth.success. Fail: No events, missing event types, or timestamps out of order.

3. Verify mTLS enforcement

Agents with require_auth: true (default) reject unauthenticated requests.

# Try to call an agent from the host without a client certificate
curl -k https://localhost:9501/ \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":{"messageId":"test","role":"user","parts":[{"kind":"data","data":{"service":"api"},"metadata":{"skill_id":"status_check"}}]}}}'

# Expected: HTTP 401
# {"error": "Authentication required. Provide client certificate (mTLS) or Authorization: Bearer token (JWT-SVID)."}

# Verify from inside the Docker network (agent-to-agent with JWT-SVID works)
docker exec eigenoid-agent-manager-agent \
  curl -s http://eigenoid-agent-worker-agent:9501/.well-known/agent-card.json

# Expected: HTTP 200 (agent card is a public endpoint)

bash

Pass: Host curl without cert returns 401; agent-to-agent works. Fail: Unauthenticated request succeeds.

4. Inspect inbound allowlists

Each agent gets an EIGENOID_INBOUND_SKILL_ALLOWLIST environment variable.

# Check allowlist for DataStore agent
docker exec eigenoid-agent-datastore-agent \
  printenv EIGENOID_INBOUND_SKILL_ALLOWLIST

# Expected (JSON):
# {"spiffe://eigenoid.local/agent/analyst-agent": ["fetch", "store"]}

# Check allowlist for Archiver agent
docker exec eigenoid-agent-archiver-agent \
  printenv EIGENOID_INBOUND_SKILL_ALLOWLIST

# Expected:
# {"spiffe://eigenoid.local/agent/processor-agent": ["archive", "retrieve"]}

# Coordinator/Gateway should have empty allowlist (nobody delegates to them)
docker exec eigenoid-agent-coordinator-agent \
  printenv EIGENOID_INBOUND_SKILL_ALLOWLIST

# Expected: {} (empty JSON object)

bash

Pass: Each agent's allowlist matches the delegation graph in eigenoid.yaml. Fail: Missing allowlist, wrong skills, or wrong SPIFFE IDs.

5. Verify filesystem isolation

Each agent has its own Docker volume. Files are not shared.

# Write a file inside Gateway's workspace
docker exec eigenoid-agent-gateway-agent \
  sh -c 'echo "gateway secret" > /app/workspace/test.txt'

# Verify Gateway can read it
docker exec eigenoid-agent-gateway-agent \
  cat /app/workspace/test.txt
# Expected: gateway secret

# Verify Archiver CANNOT see Gateway's file (separate volume)
docker exec eigenoid-agent-archiver-agent \
  cat /app/workspace/test.txt
# Expected: File not found (or different content from Archiver's own workspace)

# Verify Processor is read-only
docker exec eigenoid-agent-processor-agent \
  sh -c 'echo "test" > /app/workspace/forbidden.txt'
# Expected: Read-only file system error

# Verify Processor CAN read
docker exec eigenoid-agent-processor-agent \
  ls /app/
# Expected: Lists files from Processor's read-only mount

# Check host-side volume paths are separate
ls -la data/
# Expected:
# data/gateway-agent/app/workspace/
# data/processor-agent/
# data/archiver-agent/app/workspace/

bash

Pass: Each agent only sees its own files; Processor cannot write. Fail: Files visible across agents or Processor can write.

6. Verify SPIRE registration entries

# List all SPIRE entries
docker exec eigenoid-spire-server \
  spire-server entry show -socketPath /opt/spire/data/server/api.sock

# For each entry, verify:
# 1. SPIFFE ID matches: spiffe://eigenoid.local/agent/<agent-name>
# 2. Selector matches: unix:uid:<expected-uid>
# 3. Parent ID is the SPIRE agent's SPIFFE ID

# Check the registration script that was generated
cat .eigenoid-generated/register.sh

# Expected content includes:
# spire-server entry create \
#   -socketPath "$SERVER_SOCK" \
#   -spiffeID spiffe://eigenoid.local/agent/gateway-agent \
#   -parentID "$AGENT_SPIFFE_ID" \
#   -selector unix:uid:10001 || true

bash

Pass: Each agent has a SPIRE entry with correct SPIFFE ID and UID selector. Fail: Missing entries or mismatched UIDs.

7. End-to-end delegation chain test

Test a full delegation chain with live agents.

# Call Gateway's ingest skill
curl -k https://localhost:9600/ \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <valid-jwt-svid>" \
  -d '{
    "jsonrpc": "2.0",
    "id": "e2e-test",
    "method": "message/send",
    "params": {
      "message": {
        "messageId": "test-1",
        "role": "user",
        "parts": [{
          "kind": "data",
          "data": {"data": "test payload"},
          "metadata": {"skill_id": "ingest"}
        }]
      }
    }
  }'

# Expected: 200 OK with result showing status: "ingested"
# Check audit logs on each agent to verify the chain was propagated

bash

Pass: Request succeeds, delegation chain visible in audit logs. Fail: 401/403 errors or missing chain entries.

Phase 1 -- inspecting signed hops in the audit log

Under Phase 1 every hop carries signer_public_key_pem + signature (64-byte Ed25519). You can confirm the chain is really signed by reading the middleware's audit events and, on tamper, the chain.verify_failed event:

docker exec eigenoid-agent-<name> sqlite3 \
  /app/workdir/.eigenoid/audit.db \
  "SELECT event_type, substr(data, 1, 120) FROM audit_events
   WHERE event_type LIKE 'chain.%'
   ORDER BY id DESC LIMIT 10;"

bash

Expect rows for chain.cross_domain_hop (Phase 1 D7) whenever a trust-domain crossing happens, and chain.verify_failed whenever a forged chain is rejected pre-handler.

Phase 1.5 -- inspecting pubkey pins and rotation attestations

When EIGENOID_IDENTITY_BINDING_ENABLED=true is set on the agent, the middleware also writes to pubkey_pins.db and surfaces pin-related audit events:

# List the pins this agent holds
docker exec eigenoid-agent-<name> sqlite3 \
  /app/workdir/.eigenoid/agents/<name>/pubkey_pins.db \
  "SELECT spiffe_id, source, substr(pubkey_pem, 28, 24)
   FROM pubkey_pins;"

# Inspect pin-related audit events (TOFU, rotation accepted, binding
# failed)
docker exec eigenoid-agent-<name> sqlite3 \
  /app/workdir/.eigenoid/audit.db \
  "SELECT event_type, substr(data, 1, 120) FROM audit_events
   WHERE event_type LIKE 'pubkey.%' OR event_type LIKE 'chain.identity%'
   ORDER BY id DESC LIMIT 10;"

# Published rotation attestations (self-verifying -- signed by the OLD key)
curl -sk https://localhost:<port>/.well-known/pubkey-rotation.json \
  | python -m json.tool

bash

See Delegation security model for the pin lifecycle and examples/testing_capability/phase_1_5/README.md for the full Docker walkthrough including a real-keypair forgery script.

Running automated tests (no Docker)

All governance tests can run without Docker, SPIRE, or API keys:

# Run all three example test suites
python -m pytest examples/governed_delegation/ examples/approval_gate/ examples/secure_perimeter/ -v

# Run individual suites
python -m pytest examples/governed_delegation/test_governed_delegation.py -v  # 17 tests
python -m pytest examples/approval_gate/test_approval_gate.py -v              # 18 tests
python -m pytest examples/secure_perimeter/test_secure_perimeter.py -v        # 20 tests

# Run existing framework tests
python -m pytest tests/ -v

bash

---

This document consolidates the project's phase-handoff reports, audit catalogues, runbook, repo review, OpenClaw proposal, and the seven operational reference docs that previously lived under docs/. All source markdown files were removed after consolidation; this is the archived record.

For day-to-day orientation while making code changes, see AGENTS.md at repo root.