E
Senior Data Engineer
E
Senior Data Engineer
Eclaro5-7 Years
- Posted 8 hours ago
- Be among the first 10 applicants
Job Description
Role: Senior Data Engineer
Skill
- Fluent in Python programming language
- Proficient in SQL
- Understanding of OLTP concepts including normalised schema design
- Understanding of OLAP concepts and data model design
- Experience leading the designing and building data pipelines, ETL processes, and managing data infrastructure
- Experience leading workflows automation, managing task dependencies and transformations using Apache Airflow and DBT
- Experience in real time streaming pipelines like Kafka/pubsub and, Pyspark/dataflows
- A strong analytical mindset, a passion for resolving complex technical challenges, and the ability to find creative solutions to issues
- Excellent communication and interpersonal skills, with the ability to lead technical discussions, build consensus, and work effectively with cross-functional teams
- Deep understanding of cloud migration strategies, best practices, and potential pitfalls
- Experience with Google Cloud Platform, AWS or Azure including services for data storage, processing, and analytics
- Experience with cloud deployment and test automation using a CI/CD solution such as Cloud Build or Concourse
- Familiarity with consumer behaviour data (Google Analytics, Adobe Analytics or Snowplow Analytics) and/or subscribers data
- Proficient usage of AI to gain working efficiency to focus on higher impact architectural tasks
Experience and Role Responsibilities
- 5 + Senior Data Engineer
- Design, develop and maintain data pipelines that extract, load and transform data from various sources into the data warehouse
- Design and implement data models that align with business requirements and support efficient querying and analysis
- Ensure the team's orchestration pipelines are reliable, scalable and efficient whilst considering factors including data volume, velocity and variety
- Work across the data engineering teams, vendors and internal stakeholders to contribute to the continuous improvement of data products, processes and systems.
- Follow data engineering best practices including coding, DevOps, documentation and metadata standardisations
- Experience with the management and nature of audience tracking and subscribers datasets
- Take ownership and accountability for assigned work, including development and end-to-end testing
- Ability to conduct comprehensive feasibility assessments of architectural and data modelling options, systematically evaluating the trade-offs, advantages, and limitations of each proposed solution
- Ensure high standard and ongoing quality of Engineering output whilst meeting project timelines
- Document existing and future solution designs to enhance the team's collective knowledge
More Info
Key Skills
Cloud migration strategies
OLTP concepts including normalised schema design
Pyspark dataflows
Snowplow Analytics
ETL processes
OLAP concepts and data model design
Real time streaming pipelines
About Company
Skills:
Data Lineage, Metadata Management, Pyspark, Incident Management, Sql, Data Quality, Data Governance, Python, AWS, ETL operations, Data pipeline scheduling, Control-M, Monitoring
Skills:
Data Engineer, Python, Sql, Pandas, BigQuery, Etl, Docker, Polar
Skills:
data sharing , snowflake , Java, Data Modeling, Automated Testing, Sql, Performance Tuning, Git, Python, AWS, Data Pipeline Integration, Generative AI, DevOps practices, Platform Governance, Cost Optimization, AI-assisted engineering tools, Access Control, Amazon Bedrock
Skills:
Databricks, Power Bi, Sql, Python, Data Governance, Data Visualization, Data Modeling, Data Ingestion, R, AI Integration
Skills:
Gcp, Apache Spark, Databricks, Tableau, Python, Sql, AWS, Airflow, Bitbucket Pipelines, dbt