- Paya Lebar Road Geylang Southeast (Singapore) Singapore
Full Time Large Language Model (LLM) Evaluation Engineer Jobs, Salary up to SGD 12,000 in UMELIFE (SINGAPORE) PTE. LTD. Southeast (Singapore) - Maukerja
Large Language Model (LLM) Evaluation Engineer
UMELIFE (SINGAPORE) PTE. LTD.
Share
Save
Working Location
Job Description
Responsibilities
Job Responsibilities
Build and maintain an automated LLM evaluation pipeline covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing. The pipeline should support one-click evaluation, historical result comparison, and regression testing.
Conduct general capability evaluations using benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis.
Conduct Agent capability evaluations, including setting up evaluation environments and tracking metrics for benchmarks such as BFCL, τ-bench, and GAIA.
Design and execute persona/role-playing evaluation frameworks, covering metrics such as identity recognition, role compatibility, multi-turn stability, and style consistency.
Record and analyze evaluation results from training runs, conduct comparative analysis and anomaly detection, and produce checkpoint evaluation reports.
Conduct regular intermediate evaluations during the pre-training stage to track the evolution and improvement of model capabilities.
Job Requirements
Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related field.
Familiarity with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
Strong proficiency in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.
Familiarity with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation.
Experience in evaluation data analysis and visualization.
Detail-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results.
Preferred Qualifications / Nice to Have
Experience with Agent evaluation, particularly BFCL, τ-bench, GAIA, or SWE-bench.
Experience with persona or role-playing evaluation, such as CharacterBench or RMTBench.
Experience with evaluation automation and CI/CD integration.
Understanding of model training workflows, with the ability to understand the relationship between training checkpoints and evaluation results.
UMELIFE is a groundbreaking intelligent social lifestyle platform. It use intelligent socialization, emotional companionship as the starting point, redefining the concept of close friends, and ushering in a whole new lifestyle in the era of intelligence. Utilizing AI technology, we offer users five functional modules including emotional socialization, entertainment games, news updates, dynamic posting, and intelligent price comparison, all closely related to daily life, providing people of all ages and genders with a fun-filled intelligent living experience. Our platform is backed by investment from Brightoil Group, a renowned entity specializing in energy and upstream oil and gas exploration.
Build and maintain an automated LLM evaluation pipeline covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing. The pipeline should support one-click evaluation, historical result comparison, and regression testing.
Conduct general capability evaluations using benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis.
Conduct Agent capability evaluations, including setting up evaluation environments and tracking metrics for benchmarks such as BFCL, τ-bench, and GAIA.
Design and execute persona/role-playing evaluation frameworks, covering metrics such as identity recognition, role compatibility, multi-turn stability, and style consistency.
Record and analyze evaluation results from training runs, conduct comparative analysis and anomaly detection, and produce checkpoint evaluation reports.
Conduct regular intermediate evaluations during the pre-training stage to track the evolution and improvement of model capabilities.
Job Requirements
Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related field.
Familiarity with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
Strong proficiency in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.
Familiarity with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation.
Experience in evaluation data analysis and visualization.
Detail-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results.
Preferred Qualifications / Nice to Have
Experience with Agent evaluation, particularly BFCL, τ-bench, GAIA, or SWE-bench.
Experience with persona or role-playing evaluation, such as CharacterBench or RMTBench.
Experience with evaluation automation and CI/CD integration.
Understanding of model training workflows, with the ability to understand the relationship between training checkpoints and evaluation results.
UMELIFE is a groundbreaking intelligent social lifestyle platform. It use intelligent socialization, emotional companionship as the starting point, redefining the concept of close friends, and ushering in a whole new lifestyle in the era of intelligence. Utilizing AI technology, we offer users five functional modules including emotional socialization, entertainment games, news updates, dynamic posting, and intelligent price comparison, all closely related to daily life, providing people of all ages and genders with a fun-filled intelligent living experience. Our platform is backed by investment from Brightoil Group, a renowned entity specializing in energy and upstream oil and gas exploration.
Important Information
Never provide your bank or credit card details when applying for jobs. Do not transfer any money or complete unrelated online surveys. If you see something suspicious, Report this Job ad.
Learn More