AI Benchmark Engineer | Native Language Specialist - Chinese Mandarin - Remote

Multilingual coding benchmarks

Design, build, and validate Terminal-Bench tasks that test language models on software challenges in multilingual environments. This remote freelance project examines prompt language effects, processing of content outside English, and difficult locale or encoding cases in terminal workflows.

Tasks should reveal whether a model can work directly with the target language. Datasets and files must remain in that language so that the benchmark tests multilingual handling instead of relying on English translations.

Engineering deliverables

• Build realistic task environments using datasets and files in your native language.
• Develop prompts and tasks that identify where a coding agent fails when operating in that language.
• Support reference implementations that provide robust solutions for the benchmark tasks.
• Write reliable, deterministic verifier scripts; use rubric based judging only when necessary.
• Inspect execution logs and calibrate task difficulty from easy to very hard using standard Terminal-Bench run configurations.
• Run comparisons across different model tiers, including Haiku, Sonnet, and Opus.
• Participate in creation, human review, calibration review, and audit, alongside automated model checks.

The quality process checks fairness, grammar, and benchmark integrity. Human review has four stated layers: task creation, human review, calibration review, and audit, with automated language model checks alongside them.

Execution logs are used to understand model behavior and adjust the difficulty rating. Reference implementations establish working solutions, while deterministic verifier scripts provide a reliable check of task results. The work therefore requires both realistic multilingual software environments and careful verification of the outcomes.

Required background

The experience requirement is 5+ years in software engineering. Candidates should have a record at leading technology organizations and/or a degree from a highly regarded engineering university. Native or near-native fluency, nuanced command of grammar and register, and strong English are required.

Strong Python, shell scripting, and data processing skills are essential. You should be experienced with terminal and command line development and familiar with coding agents.

Multilingual technical depth

• Encoding and decoding robustness, including Unicode normalization.
• Locale conventions such as collation, casing, and non-Gregorian dates.
• Text input and output, toolchain interoperability, and safe string handling.
• Where relevant, bidirectional or right to left text, font fallback, and typography or rendering in user interfaces and artifacts.

Working arrangement

This is independent contractor work with a flexible schedule. Project demand determines the work available, so hours are not guaranteed and the engagement is intended to provide supplemental income. It is not presented as a dependable replacement for a full time job.

Contractors choose when to work, but accepted assignments carry commitments to quality and timely delivery. Most tasks require availability of at least 2 hours per day or 10 hours per week, so you should be able to sustain the required commitment once you accept work.

The 1099 arrangement does not include employee benefits such as health insurance, paid leave, or retirement contributions. Contractors handle their own tax obligations. The source recommends discussing those obligations with a tax professional before starting.

Projects can provide experience across different industries and domains, with opportunities to expand a portfolio and develop relevant skills. Rates are described as competitive, with prompt payment; no specific rate is stated.

Selection process

Prepare an updated CV in English. Selection includes a generative AI assessment, followed by onboarding and profile setup for eligible applied AI projects.