jobs in Shopee

Kerja Sepenuh Masa, Agent Reliability - Evaluation Algorithm Engineer di Shopee - Maukerja

Agent Reliability - Evaluation Algorithm Engineer

Shopee

Sepenuh Masa

Singapore

Kongsi
Simpan

Lokasi Kerja

  • Singapore

Penerangan Kerja

Tanggungjawab

Job Description

  • Design and own the evaluation system for LLM Agents, including benchmark construction, LLM-as-Judge with human annotation, and regression/comparison pipelines. Continuously monitor product metrics to guide Agent and model optimization.
  • Build Agent reliability analysis, including error attribution and uncertainty estimation across multi-step trajectories, and turn failure cases into targeted improvements.
  • Drive evaluation-guided Agent optimization, including Harness design (planning, verification, self-refine), tool calling, context engineering, and memory.
  • Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL), including data pipelines and reward model design.
  • Build and optimize RAG/retrieval systems and the Agent tool layer (search, code interpreters, sandboxes, MCP) and evaluate their impact on end-to-end Agent quality.
  • Explore frontier research in Agent reliability and evaluation and drive a full closed loop from research to deployment.

Requirements

  • Master’s degree or above in Artificial Intelligence, Computer Science, Mathematics, or a related field.
  • At least 1 year of full-time industry experience building and deploying production LLM Agent systems (task planning, orchestration, tool calling), including designing evaluation sets and metrics (LLM-as-Judge, human annotation, error attribution) to drive iteration.
  • Hands-on experience fine-tuning LLMs via SFT and DPO (GRPO/RL a plus), and building and optimizing RAG/retrieval systems (recall, ranking, embedding fine-tuning).
  • Good programming skills; proficient in Python (SQL and PyTorch are a plus). Familiar with training and inference frameworks such as verl, vLLM, FSDP, or LLaMA-Factory.
  • Strong analytical skills, with a solid grasp of statistics and experiment design. Sustained curiosity about frontier AI, strong self-drive, and the ability to collaborate closely with teams to drive a full closed loop from research to deployment.

Peringatan Penting

Jangan pernah kongsikan maklumat bank atau kad kredit anda semasa memohon pekerjaan. Elakkan membuat sebarang pembayaran atau mengisi survey yang tidak berkaitan. Jika ada yang mencurigakan, sila laporkan iklan pekerjaan ini segera.

Lebih Lanjut