Looking for Senior Software Engineer (Python, TypeScript, or Go) – AI Code Evaluation

Job Title: Senior Software Engineer (Python, TypeScript, or Go) – AI Code Evaluation (US & Canada) Location: Remote Overview: We are looking for Senior Software Engineers with 5+ years of hands-on experience building production systems in Python, TypeScript, or Go (Golang) to do AI code evaluation work for our hiring partner. In this role, you will work on evaluating, benchmarking, and improving the software engineering capabilities of frontier Large Language Models (LLMs) and autonomous coding agents. Rather than simple snippet labeling, this work involves repository-level code review, debugging multi-step agent execution traces, building reproducible test harnesses, and authoring rigorous evaluation rubrics grounded in real-world production engineering standards. This is a remote role for about 20 hours a week, starting with a 3-month contract. Candidates must be based in eligible states/provinces in the United States or Canada, authorized to work without sponsorship, and willing to complete standard identity verification and a short technical interview and coding step before contracting directly with our partner. Job Responsibilities: - Perform deep technical reviews and audits of AI-generated code, pull requests, and multi-file repository patches across Python, TypeScript, or Go. - Analyze and debug multi-step AI agent traces (tool calls, terminal executions, file edits, and reasoning steps) to pinpoint root causes of failures, hallucinations, or regressions. - Design realistic, production-grade software engineering benchmark tasks (bug fixes, feature implementations, system refactoring, and API integrations) derived from real-world architectures. - Author clear, deterministic evaluation rubrics and grading criteria covering functional correctness, algorithmic efficiency, idiomatic code style, type safety, and security best practices. - Build and maintain automated verification test suites and execution harnesses using pytest (Python), Jest/Vitest (TypeScript), or the Go testing package to programmatically validate model outputs. - Write clean, well-architected reference solutions ("golden patches") and edge-case unit/integration tests that handle boundary conditions, concurrency issues, and error handling. - Provide structured, actionable technical feedback on where and why model-generated solutions fail production standards. Requirements: - 5+ years of professional software engineering experience designing, building, and maintaining production systems. - Deep production mastery in at least one of the following core ecosystems: 1) Python: Backend services and APIs (FastAPI, Django, or Flask), async programming, data modeling (Pydantic, SQLAlchemy), static typing (mypy), and automated testing with pytest. 2) TypeScript: Full-stack or backend production systems (Node.js, React, Next.js, or NestJS), strict TypeScript type systems, schema validation (Zod), asynchronous state handling, and testing with Jest, Vitest, or Playwright. 3) Go (Golang): High-concurrency backend services, REST/gRPC APIs, idiomatic error handling, goroutines/channels, CLI/platform tooling, and table-driven unit/integration testing with go test. - Strong proficiency with Git workflows, reading large unfamiliar codebases quickly, and debugging complex stack traces across multi-service environments. - Solid understanding of software architecture, RESTful/GraphQL API design, relational and NoSQL databases, and secure coding practices. - Location Requirement: Must physically reside in the United States or Canada. Due to independent contractor compliance rules, candidates based in California, New Jersey, Massachusetts, Connecticut, or Vermont are not eligible. - Work Authorization: Must be legally authorized to work in the United States or Canada without current or future visa sponsorship (US Citizen, Green Card, Canadian Citizen, or Canadian PR). - Availability & Contracting: Available for about 20 hours a week on a 3-month contract to start, willing to complete standard identity verification, a short interview, and a coding step, and willing to be contracted directly by our hiring partner. - Excellent written and verbal English communication skills to articulate complex technical critiques clearly and concisely. Nice to Have: - Hands-on experience with AI code evaluation, RLHF (Reinforcement Learning from Human Feedback), SWE-bench style repository benchmarks, or building RL execution environments for coding agents. - Experience with Docker, containerized sandbox environments, and CI/CD pipelines (GitHub Actions, GitLab CI) for reproducible test execution. - Track record as a technical lead, staff engineer, or primary code reviewer enforcing engineering standards in production teams. - Active open-source contributions or experience working with polyglot microservice architectures combining Python, TypeScript, and Go.