Technical Lead (ETL)

Duties and Responsibilities

• Skilled Offshore ETL/Data Engineer Contractor to support enterprise data integration, transformation, and modernization initiatives.

• Will build scalable data pipelines and contribute to next-gen data solutions incorporating AI/ML capabilities along with legacy ETL modernization.

• Design, develop, and maintain ETL pipelines using Informatica, Azure Data Factory (ADF), and Databricks

• Perform migration of legacy ETL workflows (Informatica) to Databricks using Python/PySpark

• Analyze existing ETL workflows and re-engineer into optimized Spark-based transformations

• Develop data processing and transformation solutions using Python and PySpark

• Apply AI/ML techniques for data enrichment, anomaly detection, and predictive insights (where applicable)

• Build and optimize SQL queries, data models, and transformations

• Schedule and monitor jobs using AutoSys • Integrate data from multiple sources:

  o Relational databases (SQL Server, DB2) o Files (CSV, XML, JSON)

  o Mainframe systems

  o Streaming platforms like Kafka

• Perform data validation, reconciliation, and ensure data quality

• Troubleshoot ETL/pipeline failures and optimize performance

• Collaborate closely with onshore teams for development and production support

Non-negotiable/Required

• Strong experience in Informatica ETL development

• Proven experience in Informatica to Databricks migration

• Strong programming skills in Python and PySpark

• Hands-on experience with Databricks and Azure Data Factory (ADF)

• Proficient in SQL (complex query development and optimization)

• Experience with AutoSys job scheduling

• Experience integrating data from:

  o DB / DB2 / Mainframe systems

  o Files and streaming platforms (Kafka)

• Solid understanding of ETL re-engineering, transformation logic conversion, and Spark optimization

Nice-to-have/Advantage

AI / ML Skills (Preferred)

• Basic to intermediate understanding of AI/ML concepts and data pipelines for ML workloads

• Experience using Databricks ML / MLflow / notebooks for model tracking and experimentation

• Ability to support data preparation for machine learning models (feature engineering, dataset curation)

• Exposure to Python ML libraries (e.g., Pandas, Scikit-learn)

• Experience or familiarity with:

  o Anomaly detection / predictive analytics use cases

  o Data pipelines supporting AI workflows

• Understanding of LLM / GenAI integration with data platforms (preferred but optional) Preferred Qualifications

• Experience with Azure cloud data ecosystem

• Exposure to Data Lake / Lakehouse architecture

• Familiarity with CI/CD and DevOps practices

• Experience in batch + streaming data processing patterns

Soft Skills

• Strong communication and coordination with offshore/onshore teams

• Ability to work independently in a fast-paced environment

• Proactive in identifying modernization and AI-driven opportunities