Cloud Data Engineer

Core Expertise:

●       Experience: At least 5 years in a data engineering role working on validation of data integration, processing and transformation pipelines and cloud infrastructure.

●       Strong Python skills for test frameworks, API integration, and data validation, emphasizing code quality and TDD.

●       Demonstrated proficiency with SQL for deep-dive data reconciliation, data profiling, and backend verification

●       Combines cloud data engineering with QA practices (planning, automation, and reporting) to validate data pipeline integrity, scalability, and performance for production readiness.

●       Practical knowledge of big data storage formats (Apache Parquet, ORC), including schema validation, file size optimization testing, and identifying data corruption pitfalls.

●       Demonstrated client-facing experience, multi-cultural team experience, technical leadership, team leadership.

●       Data Modeling Verification: Experience reviewing and verifying various data models for structural integrity and performance:

o   Relational modeling

o   Dimensional modeling (Nice to have)

o   NoSQL modeling

●       Working knowledge of IaC on AWS (CloudFormation or CDK)

●       Working knowledge of AWS Services:

o   Required: Glue, IAM, Lambda, DynamoDb, Step Functions, S3, CloudFormation or CDK

o   Nice-to-have: Athena, Kinesis, MSK, MWAA, SQS

●       Test Lifecycle Management: Skilled in end-to-end data test planning, scenario design, automated script writing, execution, and quality reporting.

●       Pipeline Orchestration Testing: Experience validating the orchestration and failure boundaries of data flows and pipelines.