Lead Data Engineer
- Posted a month ago
- Be among the first 10 applicants
Job Description
Position: Lead Data Engineer
Location: Bengaluru
Experience: 10+ years
Job Summary:
We are seeking an experienced Senior Data Engineer with a strong background in PySpark, Spark, and Big Data technologies. The ideal candidate will have over five years of hands-on experience designing, building, and optimizing large-scale data processing systems and pipelines.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes using PySpark and Spark.
- Perform data wrangling, transformation, and integration across diverse data sources.
- Optimize the performance and reliability of data processing workflows.
- Implement data quality frameworks to ensure data accuracy and integrity.
- Develop and maintain data architectures and data models to support business needs.
- Collaborate with cross-functional teams including data analysts, software engineers, and business stakeholders to understand data requirements.
- Stay updated with the latest advancements in Big Data tools, frameworks, and best practices.
- Mentor and guide junior data engineers in technical and professional growth.
Required Skills and Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- 5+ years of experience in data engineering roles.
- Expertise in PySpark and Spark for large-scale data processing.
- Strong understanding of Big Data ecosystems and technologies (e.g., Hadoop, Hive).
- Hands-on experience with ETL development, data modeling, and data warehousing.
- Proficiency in programming languages such as Python, Scala, or Java.
- Experience with cloud platforms like AWS, Azure, or Google Cloud.
- Strong problem-solving skills and a detail-oriented mindset.
- Excellent communication and collaboration skills.
Preferred Qualifications:
- Experience with containerization and orchestration tools like Docker and Kubernetes.
- Knowledge of SQL and relational database management systems.
- Familiarity with data orchestration tools like Airflow, Prefect, or similar.
More Info
Key Skills
About Company
Skills:
Github, SAP, Data Manipulation, Data Modelling, AWS Glue, Pytest, Redshift, ELT, Apache Airflow, Azure Data Factory, Kinesis, Sonarqube, Microsoft Azure, Databricks, Data Integration, AWS, Azure DevOps, CI CD, GitHub Actions, Stream Analytics, Synapse Analytics, Azure SQL DB
Skills:
Data Modeling, Redshift, Sql, ELT, Jenkins, Python, AWS, Etl, Airflow, SageMaker, GitHub Actions, dbt, Dagster, CI/CD
Skills:
S3, Cloudformation, Redshift, Sql, Spark, Python, AWS, AWS CDK, Lake Formation, Infrastructure as code, Glue, CI/CD, Athena
Skills:
Testing Frameworks, Python, Data Manipulation, Version Control, Performance Tuning, Workspace Management, Sql, Git, Spark, Databricks, Advanced Apache Spark Programming, Lakehouse Architecture, Role-Based Access Control, Databricks Notebooks, Delta Live Tables, DB Asset Bundles, modular design, CTEs, Job Orchestration, Cluster Configuration, Unity Catalog, CI CD pipelines, Delta Lake, Advanced SQL window functions, MLflow Integration, error handling
Skills:
Apache Spark, Sql, ELT, Apache Airflow, Apache Kafka, Databricks, Python, Etl, Apache Iceberg, cdc, Delta UniForm, BigLake, Medallion Architecture, Unity Catalog, dbt, Delta Lake, GCP BigQuery, Lakehouse architecture