Job Summary

Responsible for leading the transition of our Generative AI initiatives from experimental proofs-of-concept to resilient, enterprise-scale production systems. This role is pivotal in establishing the architectural standards and operational rigor required to maintain high-performing, secure, and cost-effective LLM applications within our cloud and on-premise environment.

How will you contribute

1. LLM Experimentation & Evaluation Frameworks
- Architect and maintain systematic environments for prompt versioning, hyperparameter tuning, and model comparison.
- Establish automated evaluation pipelines utilizing metrics such as faithfulness, relevancy, and toxicity to benchmark model iterations.
- Implement Human-in-the-loop (HITL) workflows to validate automated metrics against domain-specific banking requirements.
2. Architectural Oversight & Integration
- Define and govern the end-to-end architecture for LLM-based systems, specifically focusing on Retrieval-Augmented Generation (RAG) and Agentic workflows.
- Ensure seamless synchronization between data ingestion layers, vector databases, andinference endpoints.
3. Production Deployment
- Design and implement CI/CD/CT (Continuous Testing) pipelines tailored for the non-deterministic nature of Generative AI.
- Manage model serving infrastructure, focusing on optimizing throughput, minimizinglatency, and implementing robust auto-scaling strategies.
- Ensure proper audit-ready documentation of Gen AI / LLM assets and testing standards.
4. Adversarial Red Teaming & Security Governance
- Lead proactive red teaming exercises to identify and mitigate risks including prompt injection, jailbreaking, and inadvertent PII exposure.
- Deploy real-time guardrails . PII masking/redaction layers and moderation layers to ensure alignment with corporate compliance and safety standards.
- Conduct due-diligence on 3rd party model, tool, infrastructure and Gen AI solution providers.
5. GenAI Lifecycle & Cost Management:
- Monitor and optimize the Total Cost of Ownership (TCO) by analyzing token consumption, cache hit ratios, and model performance-to-cost efficiency.
- Oversee version control and lifecycle management for foundational models, fine-tuned adapters, and embedding models.

What will make you successful

Degree in Mathematics, Statistics, Computer Science, Engineering, Physics or related field.
5+ years of total experience in Data Science, Software Engineering, or DevOps.
Actual experience in deploying scalable, secure LLM applications in a production-grade cloud and hybrid environment
Experience in utilizing MLOps, DevOps frameworks and tools.
Core Engineering: Expert proficiency in Python and SQL. Competency in API development (e.g. FastAPI/Flask). Orchestration & Workflow Management: Mastery of LangChain, LlamaIndex, etc for managing complex application logic.
Vector Database Operations: Advanced administration of vector database operations, including index optimization and metadata filtering strategies.
LLM Observability: Proficiency in specialized monitoring platforms for trace analysis and drift detection.

Senior Machine Learning Engineer - LLMOps

Machine Learning Engineer