Generative Ai Evaluation Engineer

Design generative AI evaluation systems for EA-SPORTS FC at Electronic Arts, combining LLM-as-judge patterns, metrics, and automated testing. Work hybrid in Burnaby, BC.This Software Engineer III role sits within FC Quality Verification Engineering. You will create evaluation frameworks and developer tools that identify quality issues earlier. Your work spans dataset curation, model evaluation, validation criteria, error analysis, and deterministic checks.Key Responsibilities:
Develop evaluation frameworks for generative AI systemsImplement LLM-as-judge patterns and rubric-based scoringPerform automated calibration alongside deterministic validation checksDefine methods for exploring complex software behaviourProduce reproducible diagnostics and support delivered softwareRequirements:
Experience with generative system evaluation frameworksKnowledge of LLM architectures, concepts, and failure modesExperience curating datasets and defining validation criteriaExperience with quantitative and qualitative evaluation metricsMaster of Science or equivalent engineering experienceBring generative AI evaluation, LLM analysis, and software testing expertise to Electronic Arts’ EA-SPORTS FC engineering work.
#J-18808-Ljbffr