Private Blockchain Developer

# Senior Hyperledger Besu / Blockchain Security Engineer — Chrysalis Production Build by Monday ## URGENT — HARD DEADLINE: MONDAY, SEPTEMBER 7, 2026 We are looking for a senior hands-on **Hyperledger Besu / EVM / distributed-systems security engineer** to productionize **Chrysalis**, our blockchain-backed evidence, provenance, attestation, and verifiable transaction-record infrastructure for AI agents and autonomous systems. This is an immediate-start engineering sprint. **You must be able to start immediately and work through the weekend as necessary to reach the Monday production gate.** Chrysalis already exists as part of our broader AI security architecture. This is **not a greenfield blockchain experiment**, token project, cryptocurrency product, or DeFi application. Chrysalis uses **Hyperledger Besu as a permissioned blockchain/evidence layer** to provide durable, cryptographically verifiable records of agent actions, authorization, policy decisions, tool execution, MCP activity, and security events. We need an engineer who can work directly in the existing codebase and Besu environment, identify production gaps, complete the implementation, harden the network, validate consensus and evidence integrity, and deliver a working production deployment by Monday. # What Chrysalis Does AI agents increasingly: * Invoke tools * Call MCP servers * Access enterprise systems * Exercise delegated authority * Execute workflows * Make decisions * Act on behalf of humans and organizations Traditional logs alone are insufficient to establish an independently verifiable history of these actions. Chrysalis creates a verifiable evidence chain around them. It should allow an authorized party to establish: * Who initiated an action * Which identity was involved * Which agent acted * Under whose authority * Which authorization was issued * Which policy was evaluated * What constraints applied * What MCP server or tool was called * What execution occurred * What result was returned * When it occurred * Whether the historical evidence has subsequently been changed The blockchain is therefore not the application itself. **Hyperledger Besu provides the trust and integrity foundation for Chrysalis evidence.** # Technology Direction The production architecture is based on: * **Hyperledger Besu** * Permissioned EVM network * Enterprise/private blockchain architecture * Cryptographically signed evidence * Transaction receipts * Smart contracts where appropriate * Durable off-chain evidence where appropriate * On-chain hashes/commitments * Merkle structures where appropriate * Enterprise identity and authorization * Production Kubernetes infrastructure The engineer should be able to make sensible decisions regarding which evidence belongs directly on-chain and which should remain off-chain with cryptographic commitments anchored to Besu. We do **not** want large confidential payloads placed unnecessarily on-chain. # Monday Production Objective By Monday, we need a working production-capable Chrysalis path demonstrating: **Agent action → identity and authorization context → Chrysalis evidence creation → evidence hash / commitment → Hyperledger Besu transaction → confirmed ledger record → transaction/evidence receipt → evidence retrieval → independent verification.** The production gate must use the actual Besu network. Mocked blockchain calls do not satisfy the requirement. # Hyperledger Besu Requirements The selected engineer must review and productionize the existing Besu implementation. Responsibilities may include: * Besu node deployment * Genesis configuration * Network configuration * Permissioning * Validator configuration * Consensus configuration * Peer discovery * Bootnodes * RPC configuration * TLS * Node keys * Account management * Smart-contract deployment * Contract upgrades where applicable * Transaction submission * Transaction monitoring * Receipt generation * Block confirmation * Failure handling * Network recovery * Node replacement * Chain persistence * Monitoring * Backup/recovery * Kubernetes deployment # Consensus and Validator Layer The engineer must review the current consensus implementation and ensure it is appropriate for the Chrysalis production architecture. This includes validation of: * Validator membership * Validator keys * Quorum assumptions * Block production * Consensus failure behavior * Validator loss * Network partition behavior * Node resynchronization * Validator addition/removal * Genesis and chain configuration * Production key custody If the existing Chrysalis architecture uses **QBFT**, the engineer must be comfortable operating and troubleshooting Besu QBFT in production. The engineer should not arbitrarily change the consensus model without reviewing the existing architecture and requirements. # Evidence Architecture Chrysalis should maintain structured evidence associated with actions such as: * Human authorization * Agent authorization * Delegated authority * Policy evaluation * MCP invocation * Tool execution * API execution * Resource access * Security decision * Execution result * Denied action * Failed action Evidence records should support appropriate fields including: * Evidence ID * Event ID * Timestamp * Tenant * Principal * Principal type * Agent identity * Session/execution ID * Delegating principal * Action * Resource * MCP server * Tool * Authorization reference * Approval reference * Policy * Policy version * Constraints * Input digest * Output/result digest * Correlation ID * Trace ID * Blockchain network * Chain ID * Smart-contract address * Transaction hash * Block number * Block hash * Evidence hash * Signature * Key ID * Schema version * Execution status The engineer should reconcile this with the existing Chrysalis schema rather than unnecessarily redesigning the platform. # On-Chain vs. Off-Chain Evidence The engineer must design or validate a production-safe pattern for evidence storage. We expect a model such as: ### Off-chain Potentially sensitive or high-volume evidence can remain in controlled durable storage. Examples: * Full request payloads * Full response payloads * Sensitive business data * Large execution artifacts * Detailed traces ### On-chain Besu can preserve the cryptographic proof necessary to verify the evidence. Examples: * Evidence hash * Merkle root * Evidence ID * Timestamp/reference * Authorization reference * Principal/agent reference where appropriate * Event type * Schema/version * Signature reference * Evidence commitment * Previous commitment where required The final design must preserve integrity without unnecessarily exposing confidential information on the ledger. # Smart Contracts The engineer should review and complete the Chrysalis contract layer. Potential contract responsibilities may include: * Evidence anchoring * Evidence registration * Receipt references * Evidence-root commitments * Identity references * Authorization references * Evidence versioning * Verification metadata * Revocation/status events where applicable Contracts must be: * Version controlled * Tested * Deterministic * Gas-aware * Upgrade strategy documented * Access controlled * Protected from unauthorized evidence submission * Designed for a permissioned network This is not a public-token contract project. # Receipt Generation Every relevant evidence submission should produce a machine-verifiable Chrysalis receipt. A receipt should be capable of referencing: * Chrysalis evidence ID * Evidence digest * Transaction hash * Block number * Block hash * Chain ID * Contract address * Event/log index where applicable * Signer * Signing key ID * Timestamp * Schema/version * Verification information The receipt should allow another authorized system to independently determine whether the associated evidence is represented on the Chrysalis ledger. # Verification Service Chrysalis needs a verification path. Given an evidence record or receipt, the verifier should be able to establish: 1. Is the receipt structurally valid? 2. Is the digital signature valid? 3. Does the referenced Besu transaction exist? 4. Is it on the expected chain? 5. Was it committed to a valid block? 6. Does the expected smart-contract event exist? 7. Does the stored evidence digest match the on-chain commitment? 8. Has the off-chain evidence been altered? 9. Is the evidence associated with the expected identity/authorization context? 10. Is the schema/version understood? Verification failures must fail explicitly. # Witness / Secure / Sentry Integration Chrysalis forms an evidence layer for our broader security architecture. It must support evidence originating from systems including: * **Witness** — authorization and approval * **Secure** — security/posture controls * **Sentry** — MCP security/gateway activity Chrysalis does not need to rebuild these systems. Instead, it needs stable typed interfaces for ingesting their security evidence. Examples include: **Witness decision → Chrysalis evidence object → evidence commitment → Besu transaction → Chrysalis receipt** and: **MCP request through Sentry → policy/security decision → tool invocation → execution result → Chrysalis evidence → Besu transaction → verifiable receipt** # Authorization Evidence A major requirement is proving not merely that an AI agent performed an action, but that it possessed the appropriate authority. Where available, Chrysalis should capture or reference: * Principal * Agent * Delegating principal * Authorization ID * Approval * Scope * Resource * Constraints * Policy * Policy version * Expiration * Decision * Decision timestamp * Execution timestamp * Result A missing approval or authorization record must never silently become a valid authorization record. # Key Management Production signing and validator keys must be handled securely. The engineer should review: * Besu node keys * Validator keys * Application signing keys * Transaction signing * Smart-contract administrative keys * Key rotation * Backup/recovery * Compromise handling * Separation of duties Keys must not be hardcoded into: * Source code * Docker images * Helm charts * Kubernetes manifests * CI/CD configuration * Git repositories The architecture should be compatible with hardware-backed or enterprise key management. Experience with **HSMs, PKCS#11, Vault, cloud KMS, or equivalent enterprise key management** is highly valuable. # Permissioning Chrysalis is a controlled enterprise blockchain. The engineer must implement or validate: * Node permissioning * Account permissioning where required * RPC access restrictions * Validator controls * Administrative access * Network-level access * Service authentication * Tenant authorization * Contract authorization We do not want an unrestricted public Ethereum-style RPC endpoint exposed to the Internet. # Failure Testing The Monday implementation must include actual failure testing. Examples: * Stop one Besu node. * Restart a Besu node. * Lose a validator. * Recover a validator. * Restart Chrysalis. * Submit duplicate evidence. * Submit malformed evidence. * Submit unauthorized evidence. * Submit evidence during partial node failure. * Verify a valid transaction. * Verify a nonexistent transaction. * Modify off-chain evidence and demonstrate hash mismatch. * Attempt cross-tenant retrieval. * Redeploy the application while preserving blockchain state. # Observability We must be able to operate the Besu network. Required visibility should include, where appropriate: * Node health * Peer count * Validator status * Current block * Block production * Block interval * Transaction pool * Transactions submitted * Transactions confirmed * Failed transactions * RPC latency * Consensus problems * Peer connectivity * Evidence submissions * Evidence confirmation latency * Verification requests * Verification failures * Smart-contract errors * Storage errors * API errors Prometheus/Grafana and OpenTelemetry experience is valuable. # Day 1 — Architecture and Besu Audit Immediately: * Build and run Chrysalis. * Review the existing Besu network. * Review genesis configuration. * Review consensus. * Review validators. * Review permissioning. * Review smart contracts. * Review evidence schema. * Review transaction submission. * Review receipts. * Review verification. * Review storage. * Review keys. * Identify mocked/in-memory/test-only components. * Identify production blockers. Then begin fixing them immediately. We expect working code on Day 1. # Day 2 — Besu and Evidence Core Complete or harden: * Besu deployment * Validators * Consensus * Permissioning * Contract deployment * Evidence schema * Evidence hashing * On-chain commitment * Transaction submission * Transaction confirmation * Receipt generation * Off-chain persistence * Verification # Day 3 — Security, Integration and Failure Testing Complete: * Witness/Secure/Sentry evidence interface * Authentication * Authorization * Tenant isolation * Key-management integration * Duplicate/replay protection * Tamper testing * Node-failure testing * Validator-failure testing * Transaction-failure testing * Evidence-verification tests * Monitoring * Metrics * Logging * Deployment automation # Monday — Production Gate Before acceptance, demonstrate: 1. Production Chrysalis service running. 2. Production Besu network running. 3. Validators healthy. 4. Consensus operating. 5. Permissioning enabled. 6. Chrysalis evidence submitted. 7. Evidence anchored to Besu. 8. Transaction confirmed. 9. Blockchain receipt generated. 10. Evidence retrieved. 11. Evidence independently verified. 12. Off-chain evidence modified. 13. Modification detected. 14. Invalid receipt rejected. 15. Unauthorized evidence submission rejected. 16. Besu node restarted successfully. 17. Validator failure scenario tested. 18. Chrysalis restarted without losing evidence. 19. Redeployment does not destroy ledger state. 20. Monitoring operational. # Monday Definition of Done The production gate is: **Agent/security event → Chrysalis → evidence digest → authenticated Besu transaction → consensus → committed block → verifiable blockchain receipt → evidence retrieval → cryptographic verification.** And separately: **Stored evidence altered → digest no longer matches on-chain commitment → verification fails.** A design document, architecture diagram, smart-contract prototype, local single-node Besu demo, or PowerPoint does not satisfy this requirement. We need working production software. # Required Experience Strong hands-on experience with several of: * Hyperledger Besu * Ethereum / EVM * Solidity * Permissioned blockchain * QBFT / BFT consensus * Distributed systems * Blockchain node operation * Smart contracts * Blockchain RPC * Transaction lifecycle * Cryptography * PKI * Digital signatures * Merkle trees * Security engineering * Go / Java / Rust / TypeScript * Kubernetes * Docker * PostgreSQL * Object storage * CI/CD * Prometheus/Grafana * Enterprise key management # Strongly Preferred Particularly valuable: * Production Hyperledger Besu deployments * Besu QBFT * Multi-validator networks * Besu Kubernetes deployments * HSM integration * PKCS#11 * Enterprise blockchain * Permissioning * Smart-contract security * Zero Trust * SPIFFE/SPIRE * AI agent infrastructure * Model Context Protocol * High-throughput event systems * Evidence/provenance platforms # We Are NOT Looking For Do not apply if your blockchain experience is primarily: * Cryptocurrency trading * NFTs * Meme coins * Token launches * DeFi frontend development * Wallet UI development * Public-chain tutorials * Smart-contract boot camps * Blockchain architecture presentations without operating production networks We need someone who has actually built and operated distributed blockchain infrastructure. # Application Questions ## 1. Monday Commitment Can you start immediately and work through Monday, September 7, 2026? State the exact number of hours you can personally commit before the production gate. ## 2. Hyperledger Besu Describe the largest Hyperledger Besu network you have personally deployed. Include: * Number of nodes * Number of validators * Consensus mechanism * Deployment environment * Your personal responsibilities ## 3. QBFT What production experience do you have with QBFT or another Byzantine fault-tolerant consensus protocol? ## 4. Besu Failure Scenario A four-validator Besu network stops finalizing blocks. Walk us through exactly how you diagnose it. ## 5. Evidence Architecture An AI agent performs an action under delegated human authorization. Describe what you would: * Store off-chain * Hash * Put on-chain * Sign * Return in the Chrysalis receipt ## 6. Smart Contracts Describe a production Solidity/EVM contract system you personally implemented or audited. ## 7. Tampering Someone modifies the off-chain evidence six months after the event. How does Chrysalis prove that the evidence no longer matches the original transaction? ## 8. Key Security How would you protect: * Validator keys * Transaction signing keys * Contract administrative keys? ## 9. Besu on Kubernetes Describe how you would deploy a persistent multi-validator Besu network on Kubernetes without risking chain loss during routine deployment. ## 10. Start Time What exact time can you begin? # IMPORTANT APPLICATION INSTRUCTION Begin your proposal with: **CHRYSALIS BESU MONDAY** Then write: **Available engineering hours before Monday: XX** Applications without these two lines will not be prioritized. # Engagement This is initially a high-intensity sprint culminating in the Monday production gate. Strong engineers may continue into the subsequent Chrysalis work, including: * HSM-backed validator/signing infrastructure * Advanced key custody * Multi-region Besu * Additional validators * Disaster recovery * Evidence replication * High-throughput evidence batching * Merkle-root anchoring * Advanced receipt verification * AI agent provenance * MCP provenance * Policy evidence * Compliance evidence * External verification * Security analytics * Large-scale production hardening The immediate requirement is straightforward: **By Monday, Chrysalis must be operating on Hyperledger Besu and producing durable, verifiable blockchain-backed evidence of AI-agent and security actions.**