Role Summary
We are looking for a strong Senior Data Engineer to design, build, and optimize scalable data pipelines supporting analytics, reporting, and operational workloads within a modern cloud data platform environment.
This role requires hands-on expertise in Azure Databricks, strong engineering discipline, and the ability to translate complex insurance business requirements into production-ready technical solutions
Main Responsibilities
- Design, develop, and maintain scalable data pipelines on Azure Databricks.
- Build robust ingestion frameworks using batch, streaming, and CDC (Change Data Capture) patterns.
- Convert complex insurance business requirements into clear, actionable technical implementations.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Implement incremental processing strategies and data lifecycle management.
- Design and maintain medallion architecture (Bronze/Silver/Gold layers) where applicable.
- Ensure high standards of data quality, reliability, and observability.
- Implement DevOps best practices, including CI/CD pipelines for data workflows.
- Collaborate closely with data modelers, analysts, and platform teams to deliver production-ready datasets.
- Ensure adherence to security, governance, and compliance standards across all data flows.
- Document architecture, pipelines, and technical decisions clearly.
Qualifications & Experience
- Bachelor's Degree in Information Technology or relevant fields.
- More than 5 years of experience in Data Engineering.
- Strong hands-on experience with Azure Databricks and Spark / PySpark.
- Experience with batch processing, streaming (e.g., structured streaming), and CDC frameworks.
- Strong SQL skills and understanding of data warehousing concepts.
- Experience designing scalable data lake / lakehouse architectures.
- Experience with CI/CD and DevOps practices for data platforms.
- Solid understanding of data quality, governance, and security principles.
- Experience working in insurance or financial services is highly preferred.
- Ability to work independently and deliver production-grade solutions.
- Experience in multi-market or regional data platforms.
- Knowledge of performance tuning and cost optimization in Databricks.
- Exposure to data cataloging or governance tools.
- Experience with cloud-native monitoring and observability tools.
