- Singapore
Full Time Agent Reliability - Evaluation Algorithm Engineer Jobs, in Shopee - Maukerja
Working Location
Job Description
Responsibilities
Department Engineering and Technology
LevelExperienced (Individual Contributor)
LocationSingapore
The Engineering and Technology team is at the core of the Shopee platform development. The team is made up of a group of passionate engineers from all over the world, striving to build the best systems with the most suitable technologies. Our engineers do not merely solve problems at hand; We build foundations for a long-lasting future. We don't limit ourselves on what we can or can't do; we take matters into our own hands even if it means drilling down to the bottom layer of the computing platform. Shopee's hyper-growing business scale has transformed most "innocent" problems into huge technical challenges, and there is no better place to experience it first-hand if you love technologies as much as we do.
Job Description:
- Design and own the evaluation system for LLM Agents, including benchmark construction, LLM-as-Judge with human annotation, and regression/comparison pipelines. Continuously monitor product metrics to guide Agent and model optimization.
- Build Agent reliability analysis, including error attribution and uncertainty estimation across multi-step trajectories, and turn failure cases into targeted improvements.
- Drive evaluation-guided Agent optimization, including Harness design (planning, verification, self-refine), tool calling, context engineering, and memory.
- Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL), including data pipelines and reward model design.
- Build and optimize RAG/retrieval systems and the Agent tool layer (search, code interpreters, sandboxes, MCP) and evaluate their impact on end-to-end Agent quality.
- Explore frontier research in Agent reliability and evaluation and drive a full closed loop from research to deployment.
Requirements:
- Master’s degree or above in Artificial Intelligence, Computer Science, Mathematics, or a related field.
- At least 1 year of full-time industry experience building and deploying production LLM Agent systems (task planning, orchestration, tool calling), including designing evaluation sets and metrics (LLM-as-Judge, human annotation, error attribution) to drive iteration.
- Hands-on experience fine-tuning LLMs via SFT and DPO (GRPO/RL a plus), and building and optimizing RAG/retrieval systems (recall, ranking, embedding fine-tuning).
- Good programming skills; proficient in Python (SQL and PyTorch are a plus). Familiar with training and inference frameworks such as verl, vLLM, FSDP, or LLaMA-Factory.
- Strong analytical skills, with a solid grasp of statistics and experiment design. Sustained curiosity about frontier AI, strong self-drive, and the ability to collaborate closely with teams to drive a full closed loop from research to deployment.
Important Information
Never provide your bank or credit card details when applying for jobs. Do not transfer any money or complete unrelated online surveys. If you see something suspicious, Report this Job ad.
Learn More