Director of AI Quality Job Description We are seeking a Director of AI Quality to join our team, though remote work in the US with travel is available. This role involves building and leading the AI Quality Engineering organization, focusing on functional testing, scenario testing, automation, and regression across various platforms, including Windows, Android, and cloud environments. Responsibilities + Build and lead the AI Quality Engineering organization, covering functional testing, scenario testing, automation, and regression. + Define and execute a comprehensive AI testing strategy across Windows, Android, cloud, and cross-device experiences. + Architect the AI automation framework for UI, system interactions, agent flows, and multimodal input/output validation. + Own model and agent evaluation pipelines, including accuracy, latency, grounding, hallucination detection, safety checks, and guardrail tests. + Build tooling for validating local versus cloud inference paths, RAG reliability, perception models, and agent orchestration flows. + Partner with SRE & Delivery to define quality gates, release criteria, and gating policies for the release train. + Create a unified scenario library across features like Learning Zone, Notion integration, cross-device continuity, Local Agent, and more. + Develop a risk-based testing strategy to support parallel prototyping while maintaining high confidence in integration and release. + Work closely with Platform, Core Intelligence, Experience, and Cloud teams to identify quality gaps early and enforce quality standards. + Lead the cultural shift toward 'testability by design' and raise the quality bar across all engineering teams. Essential Skills + 12+ years of experience in quality engineering, test automation, or AI/ML validation, with at least 5 years leading teams. + Strong background in building automation frameworks for complex distributed or cross-device products. + Hands-on experience testing AI models, RAG/LLM systems, perception models, or agentic behaviors. + Deep experience with UI automation (Windows, Android), API testing, performance validation, and scenario-based testing. + Experience working with telemetry, logs, and debugging across large-scale systems. Additional Skills & Qualifications + Ability to work across geographies and manage cross-functional alignment with platform, intelligence, cloud, and experience teams. + Proven success driving quality culture in fast-paced engineering organizations. + 25% global travel capability. Work Environment This role is part of a technical leadership team responsible for test strategy, automation, scenario validation, and evaluation frameworks for AI models and agentic behaviors. You will be at the forefront of architecting test systems and methodologies that ensure end-to-end quality across the entire Quantum stack. The work environment supports a hybrid AI platform that spans Windows, Android, and Cloud, providing an innovative and dynamic space to work. Job Type & Location This is a Permanent position based out of Chicago, IL. Pay and Benefits The pay range for this position is $300000.00 - $475000.00/yr. www.lenovobenefits.com can be accessed for more information Workplace Type This is a hybrid position in Chicago,IL. Application Deadline This position is anticipated to close on Feb 16, 2026. About Actalent Actalent is a global leader in engineering and sciences services and talent solutions. We help visionary companies advance their engineering and science initiatives through access to specialized experts who drive scale, innovation and speed to market. With a network of almost 30,000 consultants and more than 4,500 clients across the U.S., Canada, Asia and Europe, Actalent serves many of the Fortune 500. The company is an equal opportunity employer and will consider all applications without regard to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law. If you would like to request a reasonable accommodation, such as the modification or adjustment of the job application process or interviewing due to a disability, please email actalentaccommodation@actalentservices.com (%20actalentaccommodation@actalentservices.com) for other accommodation options.