Key Responsibilities:
- Create proof-of-concepts and implement best-fit data/platform solutions aligned to the business/product owner's initiatives
- Promote and define enterprise-grade data engineering standards, including coding practices, testing frameworks, version control, and CI/CD processes
- Design, build, support complete data lifecycle from ingestion, transformation, storage, access ensuring data integrity, security, and quality
- Collaborate closely with data scientists, data analysts, data product owners, third-party teams, and business stakeholders to translate needs into data-driven/technical requirements and solutions
- Ensure alignment with regulatory requirements, data governance frameworks, and internal data policies, fostering trust and accountability in data assets
- Keep updated of emerging data technologies and tools, evaluating and integrating them where they can deliver strategic advantage
- Identify opportunities for innovation in data infrastructure and automation
- Implement observability practices to monitor data platforms performance and reliability, practicing continuous improvement where necessary
- Coach and mentor junior to mid-level data engineers, creating high-performing engineering culture
Qualifications:
- 2+ years of experience in data engineering
- Proven and validated experience in designing and building large-scale data platforms or pipelines (batch, real-time) with varying security/governance compliance requirements from different business entities/data sources
- Understanding of data architectures (data lake, data warehouse, lakehouse, delta architecture)
- Hands-on experience with various Cloud platform services to support data engineering and data lifecycle management
- Proficiency in data manipulation and pipeline development with scripting background
- Working knowledge of data cataloging, lineage tracking and access controls with familiarity on regulatory requirements and data privacy/security best practices
- Comfortable in designing, implementing, and managing CI/CD pipelines (for data pipelines), version control, Infrastructure as Code with testing and monitoring frameworks
- Experience implementing data quality, observability, and pipeline monitoring
- Prior experience in a lead and architect capacity, including technical leadership and decision making
Skills and Competencies:
Essential
- Design, implementation, maintenance, and support of organization-wide data solutions and end-to-end data pipelines
- Expertise in data modeling, normalization, and schema design
- Enablement and operationalization of cloud components in relation to the data solutions
- Ingestion and integration of structured, semi-structured, and unstructured data from diverse data sources
- Design and implementation of CI/CD solutions for data pipelines
- Understanding of data access controls, lineage, and privacy regulations
- Establish, enable, and implement governance, compliance requirements, security of data and data pipelines via the Data Platform
- Implement pipeline health checks, monitoring, and alerting
- Cost and resource utilization optimization of pipelines, data solutions
- Ability to align data/data platform initiatives with data productization and long-term organization goals
- Strong written and verbal communication to business proponents around design and implementation of data solutions and end-to-end data pipelines
- Encourage innovation and continuous improvement in data platform/data delivery
- Proficiency on various Data Engineering Tools:
- Platform Databricks
- Programming - Python, SQL, Shell Scripting
- Orchestration Apache AirFlow
- Processing Apache Spark, Kafka
- Storage S3, NoSQL DB
- Cloud AWS
- Integration AWS Integration Services, APIs, CDCs
- CI/CD GitHub, GitHub Actions, Databricks Asset Bundles, IaC (Terraform)
Desirable
- Knowledge on Azure and GCP data engineering-related services
- Skilled in managing multiple SIs around data delivery
- Has other relevant Data, DevOps, and Cloud certifications
- Data Engineering Tools:
- Platform - Microsoft Fabric
- Cloud - GCP, Azure
- Integration - AWS Integration Services, various iPaaS
Education / Certifications:
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field
- Databricks Fundamentals Certification
- AWS Cloud Practitioner Certification
- Other related certifications are a plus
