About the Project

We are building high-quality coding evaluation tasks based on a conversational AI SDK. We are looking for senior software engineers to validate these tasks for clarity, correctness, test quality, and overall readiness.

Role Overview

As a Senior Full-Stack Developer, you will review coding task bundles. The role goes beyond checking whether tests pass; you will assess whether each task is clear, fair, technically sound, and suitable for evaluating real software engineering ability.

This role is best suited for someone comfortable reading unfamiliar codebases, running local and Docker-based test harnesses, reviewing pull-request-style changes, and documenting precise technical feedback.

Key Responsibilities

• Review coding evaluation tasks for clarity, technical correctness, and overall quality.
• Assess whether task instructions, expected behavior, and tests are aligned.
• Run validation workflows using local and Docker-based environments.
• Identify issues with unclear requirements, weak tests, flaky behavior, or incomplete task setup.
• Review code changes and test results with a senior engineering lens.
• Document findings clearly and recommend whether tasks should be accepted, revised, or rejected.

Required Qualifications

• 5+ years of professional software engineering experience in backend or full-stack development.

• Strong Java, react, php, react/js and python development experience, including package structure, imports, object-oriented design, exceptions, typing, and testable code.

• Hands-on experience with pytest or similar test frameworks, including fixtures, mocks, regression tests, and failure analysis.

• Strong Git and command-line skills, including checking out commits, applying patches, reviewing diffs, and inspecting commit history.

• Experience reviewing production code for correctness, maintainability, edge cases, backward compatibility, and regression risk.

• Ability to debug local or containerized test environments and distinguish real code failures from setup or dependency issues.

• Strong written communication skills and attention to detail.

Preferred Qualifications

• Experience with SDKs, developer tools, frameworks, or API design.

• Experience with Docker, CI workflows, reproducible environments, or benchmark evaluation harnesses.

• Prior experience with code review, technical interviewing, annotation workflows, or AI coding-agent evaluation.

• Experience mentoring engineers or reviewing complex pull requests in a senior capacity.

• 5+ yrs of software development experience in various programming languages.
• 3+ yrs of experience if you have open source contributions.

What Success Looks Like

A successful validator consistently identifies tasks that are clear, fair, deterministic, and safe to release. They catch issues such as unclear prompts, missing skeleton stubs, brittle tests, weak verifiers, flaky harnesses, privacy problems, or leaked golden solutions, and provide practical feedback that helps improve the task before release.

Show more
Show less