Job SummaryWe are looking for a highly skilled Data Engineer (Senior Analyst) with strong experience in ETL processes, data quality, and production support. The ideal candidate will have hands-on expertise in Azure-based data platforms, Databricks, and modern data engineering tools, along with a solid background in troubleshooting, data validation, and analytics support.
Key Responsibilities- Monitor and manage ETL pipelines and workflows (e.g., Airflow DAGs) to ensure smooth and timely data processing
- Perform data analysis, validation, and cleansing to maintain high data quality and integrity
- Provide L2/L3 production support, including incident management, root cause analysis, and issue resolution
- Develop, optimize, and maintain data pipelines using Azure Databricks, PySpark, SparkSQL, and Scala
- Design and implement data ingestion and transformation processes using Azure Data Factory and related tools
- Collaborate with cross-functional teams, including Data Scientists and BI Developers, to deliver reliable datasets
- Validate and reconcile backend data with reporting tools such as Power BI
- Create reusable notebooks, scripts, and templates for scalable data processing
- Develop and maintain technical documentation, including data workflows, test cases, and operational procedures
- Support deployment activities and ensure operational readiness of data solutions
- Communicate effectively with stakeholders to provide updates, insights, and resolution plans
Required Skills & Qualifications- Bachelor's degree in Information Technology, Computer Science, or a related field
- 5+ years of experience in data engineering, ETL development, or application support
- Strong experience with: (1) Azure Databricks; (2) Azure Data Factory; (3) SQL and data warehousing concepts; and (4) PySpark, SparkSQL, and/or Scala
- Experience with workflow orchestration tools such as Airflow
- Familiarity with cloud storage solutions (e.g., Azure Blob Storage, AWS S3)
- Experience in Power BI data validation and reporting support
- Strong analytical and problem-solving skills with a focus on data quality
- Experience in production support and incident management tools (e.g., ServiceNow)
- Ability to create clear technical documentation and reusable assets
Preferred Qualifications- Experience with Microsoft Fabric or modern data platform ecosystems
- Knowledge of Snowflake or other cloud data warehouses
- Certifications in Azure or Databricks are a plus
- Experience in Agile or Scrum environments
Key Competencies- Data Quality & Governance
- Production Support & Troubleshooting
- ETL Development & Optimization
- Stakeholder Communication
- Continuous Improvement & Documentation
