Lead Data Engineer

The Lead Data Engineer is responsible for designing, developing, and optimizing data infrastructure to ensure reliable pipelines, scalable architecture, and high-quality data for analytics and business needs. The role includes leading a team of data engineers, collaborating with cross-functional stakeholders, and implementing best practices in data transformation, validation, governance, and cloud-based solutions.

Duties And Responsibilities

Lead and mentor a team of data engineers, fostering collaboration, innovation, and continuous improvement.

Define and enforce data engineering standards, coding practices, and architectural guidelines.

Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable solutions.

Design and maintain high-performance ETL/ELT pipelines and workflows.

Build and optimize data lake, data warehouse, and streaming data solutions.

Ensure systems support structured, semi-structured, and unstructured data sources.

Develop data transformation and preparation processes to deliver clean, reliable, analytics-ready datasets.

Implement validation, anomaly detection, and automated quality checks to ensure accuracy and consistency.

Define and execute data remediation strategies to resolve issues with minimal business disruption.

Implement and manage cloud-based data platforms (AWS, Azure, or GCP).

Apply modern frameworks (e.g., Spark, Kafka, Airflow, DBT) for efficient data processing.

Oversee data modeling, schema design, and database optimization (SQL/NoSQL).

Ensure accuracy, consistency, and availability of data across platforms.

Implement governance, lineage, validation frameworks, and security best practices.

Monitor, troubleshoot, and optimize pipelines to meet SLAs and performance benchmarks.

Stay current with emerging data engineering and analytics technologies.

Recommend improvements to systems, tools, and processes to enhance scalability, reliability, and quality.

Other job-related activities that may be assigned from time to time.

Job Specifications

Education

• At least graduate with a Bachelor's or Master's Degree in IT, Computer Science, Engineering, or any related course.
  
  

Related Work Experience -

7+ years of experience in data engineering, with at least 2-3 years in a leadership or senior technical role.

Proven expertise in building and managing enterprise-level data pipelines and architectures.

Strong experience with one or more cloud platforms (AWS, Azure, GCP).

Knowledge

• Knowledgeable in the following:
  
  

Proficiency in SQL and at least one programming language (Python, Java, Scala).

Hands-on experience with big data tools (Spark, Hadoop, Kafka).

Strong knowledge of data transformation, preparation, validation, and remediation techniques.

Familiarity with data warehousing solutions (Snowflake, Redshift, BigQuery, Synapse).

Experience with workflow orchestration (Airflow, Prefect, Luigi).

Solid understanding of DevOps and CI/CD for data pipelines.

Skills

Excellent communication and interpersonal skills to collaborate effectively with developers, analysts, and business stakeholders.

Strong ability to document data requirements and translate them into technical solutions.

Proven leadership and mentoring capabilities to guide and develop team members.

Ability to work seamlessly across technical and non-technical teams.

Strong problem-solving mindset with focus on detail, data quality, and scalability.

Ability to thrive in a fast-paced environment and meet tight SLAs.