AI Engineer (Evaluation)

AI Singapore (AISG) is a national initiative driven by the Ministry of Digital Development and Information (MDDI), the Infocomm Media Development Authority (IMDA), and the National Research Foundation, Singapore (NRF).

AISG brings together Singapore-based research institutions and the broader ecosystem of AI start-ups and companies to support impactful research, develop talent and power Singapore's AI efforts.

For more information on AI Singapore, please visit https://www.aisingapore.org .

This position will be hosted at the Nanyang Technological University (NTU) under VP (Artificial Intelligence & Digital Economy)’s office and we welcome you to join our community.

Duties and Responsibilities:

- Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).

- Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.

- Work with partners to collect, translate and verify evaluation datasets.

- Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.

- Collaborate with cross-functional teams within AI Products to design and resolve issues.

- Maintain code repository and documentation standards.

- Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.

Requirements:

- Graduate from a degree program in computer science, AI, data science or related fields (or equivalent practical experience).

- Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.

- Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.

- Writing production level code in Python and using version control systems such as Git.

- Strong written and verbal communication skills.

- Independent learner that is capable of reading and understanding research papers.

- Fluent in English and one other Southeast Asian language for the purposes of understanding how to build quality evaluations from a multilingual and multicultural perspective.

We regret that only shortlisted candidates will be notified.

Hiring Institution: NTU