Data Engineer
- Posted a month ago
- Be among the first 10 applicants
Job Description
Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across various systems. You will collaborate with team members to enhance data workflows and contribute to the overall efficiency of data management practices. Roles & Responsibilities: - Optimize queries for speed and efficiency, design and implement snowflake schema and create fact dimension tables - Design, develop, and maintain scalable data pipelines using Apache Spark within Databricks. Build batch and streaming data workflows. - Implement ETL/ELT processes to transform raw data into curated datasets. - Implement data access controls and permissions using Unity Catalog. - Required active participation/contribution in team discussions. - Collaborate with cross-functional teams to gather requirements and translate them into technical specifications. - Monitor and optimize data pipelines for performance and reliability. - Document data processes and workflows to ensure clarity and knowledge sharing within the team. - Stay updated with industry trends and best practices in data engineering to continuously improve processes. Professional & Technical Skills: - Required Skill: Expert proficiency in Databricks Unified Data Analytics Platform. - Strong knowledge of data pipeline architecture and implementation. - Experience with data integration and transformation techniques. - Familiarity with cloud-based data platforms and distributed computing. - Ability to troubleshoot and resolve data-related issues efficiently. - Proficient in optimizing data workflows for scalability and performance. Additional Information: - The candidate should have minimum 2 years of experience in Databricks Unified Data Analytics Platform. - This position is based at our Manila office. Minimum 2 year(s) of experience is required
Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across various systems. You will collaborate with team members to enhance data workflows and contribute to the overall efficiency of data management practices. Roles & Responsibilities: - Optimize queries for speed and efficiency, design and implement snowflake schema and create fact dimension tables - Design, develop, and maintain scalable data pipelines using Apache Spark within Databricks. Build batch and streaming data workflows. - Implement ETL/ELT processes to transform raw data into curated datasets. - Implement data access controls and permissions using Unity Catalog. - Required active participation/contribution in team discussions. - Collaborate with cross-functional teams to gather requirements and translate them into technical specifications. - Monitor and optimize data pipelines for performance and reliability. - Document data processes and workflows to ensure clarity and knowledge sharing within the team. - Stay updated with industry trends and best practices in data engineering to continuously improve processes. Professional & Technical Skills: - Required Skill: Expert proficiency in Databricks Unified Data Analytics Platform. - Strong knowledge of data pipeline architecture and implementation. - Experience with data integration and transformation techniques. - Familiarity with cloud-based data platforms and distributed computing. - Ability to troubleshoot and resolve data-related issues efficiently. - Proficient in optimizing data workflows for scalability and performance. Additional Information: - The candidate should have minimum 2 years of experience in Databricks Unified Data Analytics Platform. - This position is based at our Manila office. Minimum 2 year(s) of experience is required
More Info
Key Skills
cloud-based data platforms
Databricks Unified Data Analytics Platform
Unity Catalog
data integration and transformation techniques
data access controls
data pipeline architecture
About Company
Skills:
data engineering , snowflake , Python development, Microsoft Azure, SQL Server, SQL expertise, ETL/ELT data pipelines
Early Applicant
Skills:
snowflake , Data Lineage, S3, Data Modeling, Automated Testing, Version Control, Sql, ELT, Data Quality, Lambda, Etl, AWS, Data Pipeline Performance Optimization, AI Agent Development, Cloud-based Data Platforms, Cost Management, Operational Automation, Code Reviews, CI CD, AI-assisted Coding Tools, Software Engineering Principles, Glue, Observability
Skills:
data engineering , snowflake , Spark, Python, Sql, Etl, ELT, Generative AI solution development, AI data architecture
Skills:
Databricks Unified Data Analytics Platform, data pipeline development and management, cloud-based data storage and processing solutions, data integration and transformation techniques, data governance and quality assurance practices
Skills:
snowflake , AWS, Databricks, Sql, RDBMS, Hadoop, Pyspark, Power Bi, Python, Gcp, Git, Scala, Databricks Unity Catalog, Microsoft Fabric, Looker, Microsoft Synapse, Microsoft Azure Data Factory