Databricks/GCP Quant Engineer Build and Hand Over End-to-End Market & Signal Spider Rock trading

Senior Databricks/GCP Quant Engineer — Build and Hand Over a Fully Operational End-to-End SpiderRock WebSocket Market Data & Signal System from Scratch Contract Structure This engagement has two required stages: 1. Initial end-to-end build, deployment, validation, documentation, and handover 2. Ongoing weekly maintenance, monitoring, quantitative strategy development, and system improvement The engagement does not end when the initial build is deployed. The selected engineer must be available and willing to continue under an ongoing weekly contract after the system has been proven operational in production. Project Overview I am seeking a senior quantitative systems engineer with proven experience in: * Databricks or Google Cloud * Persistent real-time WebSocket market-data ingestion * Algorithmic trading systems * Options and equity market data * Streaming data engineering * Quantitative strategy development * Production monitoring and operational support The engineer will build a completely new SpiderRock market-data, research, backtesting, and real-time signal-generation system from zero. This is a greenfield project. The selected engineer should not assume that any previous code, notebooks, cloud configuration, schemas, tables, or infrastructure will be reused. The final system must be: * Fully operational from end to end * Production-ready * Secure * Automated * Governed * Documented * Monitorable * Recoverable * Fully owned and controlled by me * Suitable for continuous development of additional quantitative strategies My preferred environment is Databricks. A Google Cloud-native design may also be proposed, but only when the applicant can clearly demonstrate that it will provide equal or better results than Databricks in: * Security * Reliability * Real-time performance * Data governance * Scalability * Monitoring * Operational simplicity * Cost control * Long-term maintainability I do not want a solution that requires me to manually operate VMs, SSH sessions, terminals, background scripts, Kubernetes clusters, or multiple disconnected cloud services. Mandatory SpiderRock Documentation Review Applicants must review the official SpiderRock MLink documentation before submitting a proposal. Do not submit a generic proposal based only on general WebSocket knowledge. Your proposal must demonstrate that you understand the SpiderRock connection and subscription lifecycle, including: * Authentication * Administrative messages * Subscription requests * Subscription acknowledgements * Bootstrap or cache checkpoints * Live updates * Heartbeats * Connection failures * Reauthentication * Reconnection * Resubscription * Message schemas * Message primary keys * Current-state messages versus event messages Begin your proposal with: MLINK DOCUMENTATION REVIEWED Immediately after that statement, provide a concise explanation of how you would manage the SpiderRock authentication, subscription, reconnect, and resubscription lifecycle. Proposals that do not demonstrate prior review of the SpiderRock documentation will not be considered. SpiderRock Scope The system must be designed to support and govern approximately 154 entitled SpiderRock message families. These families may include data relating to: * Equity quotes * Equity prints * Exchange imbalances * Equity market summaries * Option NBBO * Option prints * Option open interest * Option volume * Volatility * Greeks * Volatility surfaces * Skew * Risk factors * Historical volatility * Reference data * Definitions * Futures * Market marks * Related institutional analytics The system must not consist of 154 manually duplicated collectors or pipelines. The architecture must use: * A governed schema registry * A configuration-driven subscription manifest * Reusable ingestion components * Metadata-driven routing * Version-controlled configurations * Standardized validation and normalization rules SpiderRock experience is a strong asset. Applicants without direct SpiderRock experience may be considered only if they can prove comparable production experience with institutional equity, options, futures, exchange, or high-throughput market-data feeds. Required Final Outcome The selected engineer must design, build, deploy, validate, document, and hand over a complete operational workflow: SpiderRock MLink WebSocket ↓ Authentication and session management ↓ Governed subscription manifest ↓ Raw vendor-message preservation ↓ Schema registry and validation ↓ Normalized governed data ↓ Real-time current-state engine ↓ Cross-family quantitative features ↓ Strategy and signal engine ↓ Historical backtesting and forward validation ↓ Live scanner and operational dashboard ↓ Alerts, logs, monitoring, and recovery The system must operate automatically without requiring someone to manually open a notebook, terminal, VM, or Python process every day. Defined Project Workflow Phase 1 — Requirements and Technical Design The engineer will work directly with me to define: * Trading objectives * Initial strategies * Required message families * Required symbols or market scope * Signal time horizons * Signal ranking requirements * Liquidity rules * Risk filters * Data-retention requirements * Dashboard requirements * Alert requirements * Expected operating hours * Expected support requirements Deliverables: * Final architecture diagram * Technology selection * Databricks versus GCP justification * Security design * Environment design * Data-flow diagram * Subscription design * Storage design * Strategy-development workflow * Milestone implementation plan * Acceptance criteria for each milestone No production credentials should be required during the initial design stage. Phase 2 — New Secure Environment Create a new environment under accounts owned and controlled by me. The environment must include: * Development environment * Testing environment * Production environment * Separate identities and permissions * Least-privilege access * Secure secrets management * Git-based source control * Automated deployment * Centralized logging * Centralized monitoring * Cost monitoring * Backup and recovery controls No production dependency may remain under the contractor’s personal cloud account, GitHub account, Databricks account, or other external account. Phase 3 — SpiderRock Connectivity Proof Build a limited production-quality SpiderRock MLink connection using a controlled subset of message families and symbols. The proof must demonstrate: * Successful secure authentication * Successful WebSocket connection * Successful administrative logon confirmation * Successful subscription requests * Successful subscription acknowledgements * Bootstrap or cache checkpoint handling * Receipt of actual SpiderRock vendor records * Separation of control messages and data messages * Raw-message preservation * Vendor timestamp preservation * Receipt timestamping * Automatic reconnect * Automatic reauthentication * Automatic resubscription * Stable operation * Measured message rate * Measured processing latency * Measured memory and compute usage A disconnected session must be deliberately tested. The system must prove that it reconnects and restores subscriptions without manual intervention. Phase 4 — Governed Schema Registry Build a governed schema registry for the entitled SpiderRock message families. The registry must maintain, where available: * Message-family name * Entitlement status * Schema version * Schema hash * Field names * Field descriptions * Data types * Primary-key structure * Event-time fields * Vendor timestamp fields * Current-state versus event classification * Supported subscription filters * Requested views * Update-latency settings * Destination tables or streams * Data-quality rules * Strategy dependencies * Schema-change history * Active or inactive status The engineer must define how schema additions and changes are detected, reviewed, approved, deployed, and documented. Phase 5 — Governed Subscription Manifest Create a configuration-driven subscription manifest controlling: * Enabled message families * Symbols or primary keys * Requested fields * Subscription filters * Connection groups * Subscription batches * Update latency * Expected freshness * Expected message rate * Data destination * Retention requirements * Priority * Strategy dependencies * Environment-specific settings Subscription changes should normally be made through governed configuration rather than by rewriting the WebSocket engine. Phase 6 — Production WebSocket Engine Build a production-grade SpiderRock WebSocket service capable of: * Persistent outbound WebSocket operation * Secure authentication * Subscription management * Subscription batching * Dynamic configuration * Administrative-message processing * Acknowledgement processing * Cache-checkpoint processing * Heartbeat monitoring * Stale-connection detection * Automatic reconnect * Automatic reauthentication * Automatic resubscription * Retry and backoff controls * Backpressure handling * Message-rate monitoring * Processing-latency monitoring * Gap detection where technically supported * Clean startup and shutdown * Automatic restart * Controlled deployment * Complete timestamped logging The WebSocket service must not rely on an engineer manually opening a terminal or notebook. The applicant must explain where the persistent WebSocket process should run and why. Phase 7 — Governed Data Architecture Create clearly separated data layers. Raw Layer The raw layer must: * Preserve the original vendor message exactly as received * Preserve control messages separately from data records * Record receipt timestamps * Record connection and session information * Record message-family information * Preserve malformed or rejected records * Prevent silent modification of source data Normalized Layer The normalized layer must: * Apply governed schemas * Apply correct data types * Standardize timestamps * Validate required fields * Deduplicate where appropriate * Track schema versions * Separate state records from event records * Record exclusions and rejection reasons Current-State Layer The current-state layer must: * Maintain the latest valid record using the correct SpiderRock primary key * Prevent older records from replacing newer records * Track freshness * Identify stale state * Maintain synchronized equity, option, volatility, surface, imbalance, print, and market-regime state Feature Layer The feature layer must: * Calculate quantitative variables * Combine related message families * Maintain event-time alignment * Track feature freshness * Record strategy dependencies * Record calculation versions * Prevent stale or future information from contaminating calculations Signal Layer The signal layer must preserve: * Strategy name * Strategy version * Symbol * Direction * Score * Confidence * Signal tier * Input features * Source-family timestamps * Entry conditions * Invalidation conditions * Targets * Time horizon * Expiry * Rejection reasons * Complete lineage back to source messages Every signal must be reproducible from its stored source data and strategy version. Phase 8 — Quantitative Strategy Framework Build a reusable framework for developing, testing, comparing, and operating quantitative trading strategies. The framework must support: * Cross-family feature engineering * Stateful real-time calculations * Event-time joins * ASOF or nearest-prior joins * Freshness controls * Stale-data rejection * Liquidity filters * Spread filters * Volume filters * Open-interest filters * Direction classification * Signal scoring * Signal ranking * Confidence scoring * Entry conditions * Invalidation conditions * Target calculations * Signal expiry * Reason codes * Strategy versioning * Strategy comparison * Full signal explainability The system must not generate unexplained black-box signals. Each signal must clearly show why it fired, which data families contributed, which filters passed, which filters failed, and whether all required inputs were fresh. Phase 9 — Backtesting and Forward Validation The system must support: * Historical feature reconstruction * Historical signal reconstruction * Backtesting * Walk-forward validation * Out-of-sample testing * Live shadow-mode testing * Strategy-version comparison * Transaction-cost assumptions * Spread assumptions * Slippage assumptions * Signal-decay measurement * Market-regime analysis * False-positive analysis * Failure-reason analysis * Prevention of look-ahead bias * Prevention of data leakage No strategy should move directly from an idea into production without documented testing and forward validation. Phase 10 — Dashboards and Alerts Create an operational dashboard showing: * WebSocket connection status * Authentication status * Subscription status * Active message families * Last message received by family * Message rate by family * Processing latency * Storage latency * Reconnect count * Error count * Rejected records * Stale families * Infrastructure utilization * Current cost where available Create a trading-signal dashboard showing: * Ranked signals * Symbol * Direction * Strategy * Strategy version * Score * Confidence * Signal tier * Entry conditions * Invalidation conditions * Targets * Time horizon * Data freshness * Contributing message families * Rejection reasons * Current market regime Alerts must be created for: * Connection loss * Authentication failure * Subscription failure * Repeated reconnects * Stale data * Processing lag * Storage failure * Signal-engine failure * Unexpected message-rate changes * Infrastructure or cost thresholds Phase 11 — Production Deployment The engineer must deploy the complete system into production and prove: * Automatic startup * Automatic recovery * Stable ingestion * Stable normalization * Stable feature calculation * Stable signal generation * Stable storage * Stable dashboards * Stable alerts * Accurate monitoring * Complete logging * Correct access controls * Reproducible deployment Production deployment is not considered complete simply because the code runs once. Phase 12 — Seven-Day Production Verification After production deployment, the selected engineer must remain responsible for the system during a minimum seven-consecutive-day verification period. For each day, the engineer must provide a timestamped production verification report containing: * WebSocket uptime * Authentication status * Subscription status * Connection interruptions * Reconnect results * Resubscription results * Message counts by active family * Last-message timestamps * Processing latency * Storage latency * Rejected records * Excluded records * Stale-data incidents * Errors * Restarts * Raw-to-normalized reconciliation * Feature-engine status * Signal-engine status * Signals generated * Signal input freshness * Infrastructure utilization * Daily cost where available * Problems identified * Corrections completed * Remaining risks * Exact next technical action Daily evidence must include: * Written production status report * Timestamped log evidence * Dashboard evidence * Short screen recording showing the system operating Final acceptance requires seven consecutive days of documented production operation without an unresolved critical failure. A critical failure must be corrected and successfully revalidated. Phase 13 — Complete Handover The engineer must provide: * Full source code * Git repositories * Infrastructure definitions * Environment configurations * Schema registry * Subscription manifest * Data dictionary * Architecture diagrams * Deployment instructions * Recovery instructions * Monitoring instructions * Strategy-development guide * Backtesting guide * Operational runbook * Incident-response runbook * Access-control documentation * Credential-rotation procedure * Cost-control guide * Disaster-recovery procedure * Recorded handover sessions The handover must prove that the system is fully owned and controllable by me. Phase 14 — Ongoing Weekly Maintenance and Strategy Development The selected engineer must be willing to remain engaged after production acceptance under an ongoing weekly contract. The continuing scope will include: * Production monitoring * Incident response * WebSocket maintenance * Schema changes * Subscription changes * Performance optimization * Cost optimization * Data-quality improvements * New message-family integration * New feature development * Quantitative strategy discussions * Strategy implementation * Backtesting * Walk-forward testing * Live forward validation * Signal-quality analysis * Dashboard improvements * Alert improvements * Documentation updates * Security updates * Capacity planning This is intended to become a long-term working relationship with continuous strategy development and production improvement. Applicants seeking only a short one-time coding task should not apply. Mandatory Candidate Experience The selected engineer must have proven professional experience in several of the following: * Algorithmic trading systems * Quantitative trading research * Equity or options market data * Institutional market-data feeds * Persistent WebSocket clients * High-throughput real-time ingestion * Databricks * Delta Lake * Unity Catalog * Databricks Jobs * Databricks streaming workloads * Stateful stream processing * Event-time processing * Watermarks * Schema registries * Data contracts * Metadata-driven pipelines * Backpressure * Checkpointing * Replay * Deduplication * Idempotency * Python asynchronous programming * Google Cloud managed services * Google Cloud Storage * IAM and secrets management * Infrastructure as code * GitHub-based development * Production observability * Options market microstructure * NBBO * Trades and prints * Open interest * Implied volatility * Greeks * Skew * Volatility surfaces * Exchange imbalances * Backtesting * Walk-forward validation * Prevention of look-ahead bias and data leakage General Python experience alone is not sufficient. TradingView, Pine Script, MetaTrader, and generic crypto-bot experience alone are not sufficient. Databricks or GCP Evidence Required Applicants must provide proven examples of similar systems built using Databricks or Google Cloud. For each example, state: * Your exact role * The system’s purpose * Data source * Asset class * Streaming technology * Approximate message rate or data volume * Cloud architecture * Storage architecture * Monitoring approach * Production duration * Your ongoing support responsibility An anonymized architecture diagram, code sample, dashboard, schema registry, data-flow design, or client reference is preferred. Do not claim experience that cannot be verified during the interview process. Security, Confidentiality, and Ownership Before receiving access, the selected engineer must sign appropriate: * Confidentiality agreement * Intellectual-property assignment * Data-use restrictions * Non-disclosure terms * Access and credential-handling terms Requirements include: * All accounts must be owned by me * All code must be stored in repositories owned by me * All infrastructure must be under accounts owned by me * All generated intellectual property must belong to me * No API key may be embedded in source code * Credentials must be stored in an approved secret-management system * Contractor access must use a separate least-privilege identity * No unauthorized subcontractors * No proprietary data may be copied to personal devices * No code, data, schema, or strategy logic may be reused for another client * No production dependency may remain under the contractor’s account * Access must be removable without disrupting production * All contractor access must be removed or reduced at handover as agreed Broader production access will be granted in controlled stages after technical and security requirements are satisfied. Application Instructions Begin your proposal with: MLINK DOCUMENTATION REVIEWED Then answer every question below directly. 1. Summarize the SpiderRock MLink authentication, subscription, acknowledgement, checkpoint, reconnect, and resubscription lifecycle. 2. Describe two production systems where you consumed a persistent real-time market-data WebSocket or comparable streaming feed. 3. For each system, provide: * Vendor or feed type * Asset class * Approximate message rate * Cloud environment * Your exact responsibility * Production duration 4. Have you worked directly with SpiderRock MLink? 5. When you have not worked with SpiderRock, identify the closest institutional market-data protocol you have implemented. 6. How would you support approximately 154 SpiderRock message families without manually building 154 separate collectors? 7. How would you create and govern: * The schema registry * The subscription manifest * Schema-version history * Message-family dependencies * Strategy dependencies 8. Explain how you distinguish: * Control messages from data messages * Bootstrap or cache records from live updates * Current-state records from event records * Vendor timestamps from receipt, processing, and storage timestamps 9. Where should the persistent SpiderRock WebSocket process run, and why? 10. Compare: * Databricks continuous processing * A managed external WebSocket service * A Google Cloud-native managed service 11. Explain how you would implement: * Reconnection * Reauthentication * Resubscription * Backpressure * Checkpointing * Deduplication * Replay * Gap detection * Automatic restart 12. How would you align equity, option, volatility, surface, print, and imbalance records without introducing look-ahead bias? 13. How would you preserve complete lineage from a generated signal back to the original SpiderRock messages? 14. Describe your algorithmic trading and quantitative strategy-development experience. 15. Provide an example of a strategy framework you built that included backtesting and live forward validation. 16. Explain why you recommend Databricks or Google Cloud for this project. 17. Provide an anonymized example of a similar: * Streaming architecture * Schema registry * Subscription manifest * Data model * Monitoring dashboard * Signal-lineage design 18. Are you willing to sign confidentiality, intellectual-property, and data-use agreements before receiving access? 19. Are you available for the seven-day post-deployment production-verification period? 20. Are you willing to continue under an ongoing weekly maintenance and strategy-development contract after production acceptance? 21. State your weekly availability after the initial build. 22. Explain what evidence you will provide to prove that the completed system is operating correctly from end to end. Generic proposals will not be considered. Applications focused primarily on TradingView, Pine Script, MetaTrader, generic crypto bots, or unverified AI trading claims will not be considered.