Data Engineer

Role responsibilities

● Design, develop and maintain data pipelines that extract, load and transform data from various sources into the data warehouse

● Design and implement data models that align with business requirements and support efficient querying and analysis

● Ensure the team's orchestration pipelines are reliable, scalable and efficient whilst considering factors including data volume, velocity and variety

● Work across data engineering teams, vendors and internal stakeholders to contribute to the continuous improvement of data products, processes and systems.

● Follow data engineering best practices including coding, DevOps, documentation and metadata standardizations

● Experience with the management and nature of audience tracking and subscribers datasets

Employee responsibilities

● Take ownership and accountability for assigned work, including development and end-to-end testing

● Ability to conduct comprehensive feasibility assessments of architectural and data modelling options, systematically evaluating the trade-offs, advantages, and limitations of each proposed solution

● Ensure high standard and ongoing quality of Engineering output whilst meeting project timelines

● Document existing and future solution designs to enhance the team's collective knowledge

Qualifications, Experience & Skills

Essential qualifications, experience & skills

● Fluent in Python programming language

● Proficient in SQL

● Understanding of OLTP concepts including normalised schema design

● Understanding of OLAP concepts and data model design

● Experience leading the designing and building data pipelines, ETL processes, and managing data infrastructure

● Experience leading workflows automation, managing task dependencies and transformations using Apache Airflow and DBT

● Experience in real time streaming pipelines like Kafka/pubsub and Pyspark/dataflows

Problem-Solving & Leadership:

● A strong analytical mindset, a passion for resolving complex technical challenges, and the ability to find creative solutions to issues

● Excellent communication and interpersonal skills, with the ability to lead technical discussions, build consensus, and work effectively with cross-functional teams

● Deep understanding of cloud migration strategies, best practices, and potential pitfalls

Desirable qualifications, experience & skills

● A bachelor's or master's degree in computer science, software engineering, data engineering, data science, information technology or a related field

● Experience with Google Cloud Platform, AWS or Azure including services for data storage, processing, and analytics

● Experience with cloud deployment and test automation using a CI/CD solution such as Cloud Build or Concourse

● Familiarity with consumer behaviour data (Google Analytics, Adobe Analytics or Snowplow Analytics) and/or subscribers data

● Proficient usage of AI to gain working efficiency to focus on higher impact architectural tasks