AI CoE Shared Service QA and Support -Vice President

Key Responsibilities

• Lead the end-to-end QA strategy, framework, and execution for enterprise AI services including Prompt Hub, RAG-as-a-Service, Fine-tuning Studio, and Agents Marketplace.
• Design and maintain test automation suites and validation pipelines for GenAI-based microservices, APIs, and data pipelines.
• Establish procedures for model output validation, performance benchmarking, and guardrail enforcement (e.g., hallucination detection, grounding checks, bias tests).
• Own the L2/L3 support processes for GenAI services; manage ticketing queues, triage incidents, and drive root-cause analysis with Engineering.
• Coordinate UAT cycles, release readiness, and production rollouts with product managers, platform teams, and DevOps.
• Develop and maintain monitoring dashboards, observability metrics, and alerting mechanisms to ensure production stability and SLA adherence.
• Act as the primary interface for audit, compliance, and InfoSec queries related to QA artifacts and operational support.
• Maintain comprehensive documentation and knowledge base for support workflows, known issues, and remediation steps.

Qualification & Requirements

• Hands-on experience with QA practices and tools across software and ML/AI pipelines (e.g., pytest, unittest, Postman, JMeter, LangSmith, MLflow).
• Strong working knowledge of GenAI lifecycle components – prompt testing, model evaluation, RAG, embedding validation, etc.
• Familiarity with RESTful APIs, container orchestration (Docker/Kubernetes), and cloud environments (Azure, AWS).
• Experience setting up automated CI/CD pipelines for test execution, using tools such as GitLab CI, Jenkins, or Azure DevOps.
• Understanding of governance frameworks for AI, including Responsible AI principles and audit requirements.
• Strong analytical and communication skills; ability to work cross-functionally with engineers, product owners, and risk managers.
• Proven leadership in handling production incidents, managing on-call schedules, and delivering continuous improvements.

Education & Preferred Qualifications

• B.Tech/M.Tech/MCA or equivalent in Computer Science, Engineering, or related fields.
• 17 + years of experience in QA and/or production support, preferably in data/AI/ML platforms.
• Experience with ITIL or equivalent service management framework is a plus.
• GenAI/ML certification or experience working in a Center of Excellence setup preferred

Benefits

State Street offers a comprehensive benefits package to its employees, including health insurance, retirement plans, and time off, with options for both full-time and part-time employees. Key benefits include health insurance with options through Cigna, a 401k plan with company matching, and vacation time, including four weeks after one year of service and a birthday off.