Data Engineer

Key Duties and Responsibilities

Data product engineering

• Build and maintain Bronze to Silver to Gold transformations and automated data-cleaning pipelines.
• Create aggregated tables that shorten the path from raw data to dashboards and Genie insights.
• Design reusable customer, product, location, asset, material, transaction and process models.

Quality and correctness

• Implement business rules, historical and slowly changing logic, reconciliations and data-quality controls.
• Build data-quality monitoring and publish quality metrics for certified datasets.
• Investigate and resolve data defects raised by consumers.

Performance and operations

• Optimize storage, partitioning, compute, orchestration and cost.
• Implement testing, version control and CI/CD for pipelines.
• Support production pipelines, including on-call or scheduled support arrangements.

Enablement

• Support feature pipelines for machine learning, forecast inputs, semantic layers, Genie Spaces and AI applications.
• Document lineage, definitions and refresh behaviour for every certified dataset.

Key Deliverables

• Certified datasets and reusable domain models.
• Tested, version-controlled and documented pipelines.
• Data-quality dashboards and control reports.
• Lineage and technical documentation.
• Performance and cost-optimization improvements.

Accountability and Success Measures

• Correctness of data against source and business rules.
• Freshness and reliability of certified data products.
• Performance, cost efficiency and maintainability of pipelines.
• Completeness of lineage and documentation.
• Reproducibility — a result produced today can be reproduced tomorrow.

Working Relationships

• Internal: Systems Integration Engineers; BI & AI Context Engineers; Data Scientists; AI / LLM Engineers; Data Governance Specialist; Data & AI Translators; Data Engineering Capability Head.

External: platform vendor support.

QUALIFICATIONS

• Bachelor's degree in Computer Science, Information Technology, Engineering, Statistics or a related field.
• Three or more years building production data pipelines. Demonstrated delivery of dimensional or domain models serving analytics and machine learning workloads. Databricks or equivalent Lakehouse experience strongly preferred.
• Preferred: Databricks Certified Data Engineer Associate or Professional. Optional: cloud platform data engineering certification.

Technical Skills

• Advanced SQL and Python; PySpark.
• Databricks, Delta Lake, Unity Catalog and medallion architecture.
• Dimensional and domain data modelling.
• Orchestration and workflow scheduling.
• Data-quality frameworks and testing.
• Git, CI/CD and code review practice.
• Performance tuning and cost management.

Behavioural Competencies

• Rigor — will not ship a number that has not been reconciled.
• Systems thinking about downstream consumers.
• Ownership of production outcomes, not just code delivery.
• Constructive code review and knowledge sharing.
• Continuous improvement and automation instinct.