AI Evaluation Rubric Auditor
Status: Open
- Pay
- $60–$85/hr
- Location
- Remote — Canada, United States, ARG, BOL, Brazil, CHL, COL, ECU, GUY, PRY, PER, URY, and United Kingdom
- Employment type
- Contractor
- Hours
- 15 hrs/week
- Posted
- 2026-11-16
Status: Open
Not the right fit? These open roles are close matches. Browse all AI gigs →
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. 1\. Overview We are hiring an experienced counsel-level lawyer to work directly with a leading AI lab's research and program management teams, improving how frontier AI models reason about real legal work. This listing is for practitioners at counsel or senior associate level, with roughly 8 to 15 years of post-qualification practice. If you are earlier in your career, or if you are at partner or General Counsel level, please apply to the corresponding Legal Domain Expert listing instead — we run separate listings by seniority so the rate matches the experience. Your legal expertise is the substance of this role. You will review the quality of legal knowledge work tasks, write the instruction specs and golden solutions that define what "correct" looks like, and build the benchmarks that show whether the model is genuinely improving. We are looking for a practising specialist rather than a generalist. This is a full-time W-2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI lab as part of their extended workforce. You will be provisioned with client-issued accounts and equipment, and will work inside the client's own tools alongside their research teams. Location: This is a hybrid role based in the Bay Area, California. You must live in the Bay Area and work on-site with the client's team multiple days each week, when required. This is not a remote role. If you do not currently live in the Bay Area, you must be willing to relocate there at your own cost before the engagement starts — relocation assistance is not provided. 2\. Key Responsibilities Data QA and reviews: Vet the quality of legal knowledge work tasks and model outputs — spotting missing behaviors, thin reasoning, and answers that read well but would not survive professional scrutiny. Instruction specs and golden datasets: Write high-quality instruction specs, produce golden solutions to legal problems, and define new legal tasks that reflect how the work is actually done in practice. Benchmarks and domain depth: Design challenging legal tasks and evaluation sets, and help build legal-specific skills and tools together with the research team. Calibration: Work with client researchers and specialists in adjacent fields to keep standards consistent, translating tacit legal judgment into explicit, teachable criteria. 3\. Core Qualifications Education: Juris Doctor (JD) from an accredited law school; a highly ranked school is strongly preferred. Experience: 8 to 15 years of substantive post-qualification legal practice at a reputable institution — an established law firm, a corporate legal department, a regulatory body, or a court. Internships and clerkships alone do not count. Seniority: Currently at or has reached counsel, senior associate, senior in-house counsel, or Assistant General Counsel level, with real ownership of matters. Domain depth: Genuine specialization in at least one substantive practice area, for example corporate and transactional, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labor, or tax. Licensure: Admission to at least one U.S. state bar, in good standing. AI fluency: Hands-on working use of large language models in your professional work, and the judgment to tell a well-reasoned answer from a plausible-sounding wrong one. Availability: Able to commit reliably to 40 hours per week for an initial engagement of 6 months. Location: Living in the Bay Area, California, and able to work on-site with the client's team multiple days each week, when required. Candidates not currently based in the Bay Area must be willing to relocate there at their own cost; relocation assistance is not provided. Excellent written communication, and the ability to give precise, well-structured written feedback. About Cincinnatus LLC Cincinnatus LLC is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for contingent and contract-based opportunities. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve part-time or full-time commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus LLC. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Status: Open
$85–$120/hr
Verified Nov 19, 2026
Role Overview Mercor is hiring experienced FP&A professionals to build and review evaluation tasks for AI agents. Each task is set in a realistic simulated company, with the budgets, forecasts, models, emails and business apps a finance team really uses. You make sure each task reads like a real request, the reference answer holds up against the source data, and the grading rubric checks what a senior FP&A lead would check. Key Responsibilities Refine AI-drafted task prompts so they read like a real request from a CFO or business partner, without giving the answer away. Check reference models and outputs against the world's files and app records, and correct what is wrong. Write and tighten grading rubrics: pass/fail criteria with exact values, plus the judgment calls a generalist would miss. Read how leading AI models attempt each task and confirm they fail for fair, defensible reasons. Work across annual budgeting and long-range planning, rolling, revenue, cash-flow and headcount forecasts, variance analysis and KPI reporting, board and investor reporting, three-statement and scenario modeling, and business partnering on pricing and cost decisions. Ideal Qualifications 3+ years of hands-on FP&A experience (Analyst, Senior Analyst, Manager or above). Time in the field matters more than title. Bachelor's degree in finance, accounting, economics or a related field. Ownership of budgeting or forecasting cycles, strong driver-based and scenario modeling, and fluency in a planning tool such as Anaplan, Adaptive or Planful. CFA, MBA or CPA is a plus, as is prior experience on a Mercor RL project. Details Pay: $100/hour. You start with a short onboarding session and a paid skills assessment. Once you pass, you can claim your first task right away. Strong performers can move into reviewer and world lead roles.
Status: Open
$100/hr
Verified Nov 15, 2026
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. 1\. Overview We are hiring a senior accounting and audit domain expert to work directly with a leading AI lab's research and program management teams, improving how frontier AI models reason about real accounting and audit work. Your accounting expertise is the substance of this role. You will review the quality of accounting knowledge work tasks, write the instruction specs and golden solutions that define what "correct" looks like, and build the benchmarks that show whether the model is genuinely improving. We are looking for a practicing specialist rather than a generalist. This is a full-time W-2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI lab as part of their extended workforce. You will be provisioned with client-issued accounts and equipment, and will work inside the client's own tools alongside their research teams. Location: This is a hybrid role based in the Bay Area, California. You must live in the Bay Area and work on-site with the client's team multiple days each week, when required. This is not a remote role. If you do not currently live in the Bay Area, you must be willing to relocate there at your own cost before the engagement starts. Relocation assistance is not provided. 2\. Key Responsibilities Data QA and reviews: Vet the quality of accounting and audit knowledge work tasks and model outputs, spotting missing behaviors, thin reasoning, misapplied standards, unsupported accounting treatments, and answers that read well but would not survive review by a reviewing partner. Instruction specs and golden datasets: Write high-quality instruction specs, produce golden solutions to accounting and audit problems, and define new tasks that reflect how the work is actually done in practice. Benchmarks and domain depth: Design challenging accounting and audit tasks and evaluation sets, and help build domain-specific skills and tools together with the research team. Calibration: Work with client researchers and specialists in adjacent fields to keep standards consistent, translating tacit accounting judgment into explicit, teachable criteria. 3\. Core Qualifications Credentials: An active professional credential, for example CPA, CA, CIA, CFE, or CMA, alongside a bachelor's or master's degree in accounting or a closely related field. Experience: 4+ years of substantive accounting or audit experience at a recognized institution, for example a public accounting firm, a corporate controllership or finance function, an internal audit department, or a regulator. Bookkeeping-only roles do not count. Domain depth: Genuine specialization in at least one area, for example external audit and assurance, technical accounting and financial reporting under US GAAP or IFRS, internal audit and SOX controls, tax, forensic accounting and fraud examination, or revenue recognition. Seniority: Clear progression to a senior level, for example Audit Manager, Senior Manager, Accounting Manager, Controller, Director of Accounting, or Partner, with real ownership of engagements or reporting cycles. AI fluency: Hands-on working use of large language models in your professional work, and the judgment to tell a well-reasoned answer from a plausible-sounding wrong one. Availability: Able to commit reliably to 40 hours per week for an initial engagement of 6 months. Location: Living in the Bay Area, California, and able to work on-site with the client's team multiple days each week, when required. Candidates not currently based in the Bay Area must be willing to relocate there at their own cost. Relocation assistance is not provided. Excellent written communication, and the ability to give precise, well-structured written feedback. About Cincinnatus LLC Cincinnatus LLC is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for contingent and contract-based opportunities. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve part-time or full-time commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus LLC. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Status: Open
$60–$100/hr
Verified Nov 19, 2026