Data Engineer

Key Responsibilities

Integration & Workflow Engineering

• Analyse and re-platform existing workflow-based integrations from one platform to another while maintaining functional parity.
• Implement orchestration logic including triggers, conditional routing, retries, exception handling, and state management.
• Design and implement automation patterns to handle platform limitations (e.g., looping, batching, pagination, throttling).
• Ensure workflows are idempotent, fault tolerant.
• Support environment-based deployments (DEV / UAT / PROD) with configuration-driven designs.

Data Engineering & ETL

• Design and build ETL pipelines for ingesting flat files (e.g., CSV) into relational databases.
• Handle schema validation, basic schema evolution, data quality checks, and error reconciliation.
• Optimize data ingestion for performance, scalability, and reliability.
• Collaborate with application teams to understand upstream and downstream data dependencies.

Cloud & Big Data Platform Contributions

• Build and maintain data pipelines on cloud-based big data platforms using distributed processing frameworks.
• Contribute to Lakehouse-style data storage that supports both batch and streaming data.
• Work with modern table formats that support incremental processing, versioning, and historical queries.
• Support use cases such as append-heavy datasets, high-write event data, and analytical queries.

Operations, Quality & Observability

• Implement logging, monitoring, and alerting for workflows and data pipelines.
• Support operational readiness including runbooks, deployment procedures, and rollback strategies.
• Participate in root-cause analysis and continuous improvement of data pipelines.
• Ensure adherence to data governance, security, and compliance standards.

Required Skills & Experience

Technical Skills

• Strong experience with workflow orchestration / integration platforms.
• Solid understanding of ETL concepts and hands-on experience with file-based ingestion.
• Proficient in SQL and working knowledge of relational databases (e.g., SQL Server or equivalent).
• Experience with distributed data processing frameworks (e.g., Spark).
• Familiarity with streaming and batch data processing concepts.
• Practical experience with cloud platforms and managed data services.
• Understanding of CI/CD principles for data and integration workflows.

Programming & Automation

• Experience with scripting or programming languages commonly used in data engineering (e.g., Python).
• Ability to build reusable utilities for batching, retries, pagination, and error handling.
• Experience with REST APIs and structured data formats (JSON, CSV).

Desired (Good to Have)

• Exposure to modern Lakehouse table formats supporting incremental processing and time travel.
• Experience with stream processing engines (e.g., Flink or Spark Streaming).
• Familiarity with query engines used for analytical access.
• Experience working in regulated environments (banking, financial services, or manufacturing).
• Knowledge of data observability and quality frameworks.

Experience Level

• Typically, 3-5 years of relevant experience in data engineering, integration engineering, or similar roles.
• Proven ability to work independently on moderately complex problems while collaborating within a larger team.

Soft Skills & Competencies

• Strong analytical and problem-solving skills.
• Ability to understand existing systems and translate requirements into working solutions.
• Clear communication with technical and non-technical stakeholders.
• Detail-oriented with a focus on reliability and maintainability.
• Comfortable working across multiple teams and applications.