Senior Data Engineer

Data Operations Senior Engineer

Job Description

The Data Operations Senior Engineer is responsible for operating, monitoring, and continuously improving production data workloads supporting SLHK. The role focuses on DataOps, ETL operations, cloud data platforms, automation, and production support, ensuring data pipelines meet operational, quality, and SLA requirements.

Key Qualifications

• At least 8+ years of overall IT/data engineering experience.
• 3–6 years of hands-on experience in DataOps, ETL, data engineering, or production data operations.
• Strong experience with AWS cloud services and cloud-based data environments.
• Extensive hands-on experience with SQL for data analysis, troubleshooting, validation, and reconciliation.
• Strong programming experience in Python for automation, data processing, and operational tooling.
• Hands-on experience with Apache Spark / PySpark for large-scale data processing.
• Experience with enterprise scheduling and orchestration tools such as Control-M, AutoSys, Tidal, Stonebranch, Redwood, ActiveBatch, or Airflow.
• Experience in production support, incident management, change/release execution, and SLA monitoring.
• Exposure to Data Quality, Data Lineage, Metadata Management, and Data Governance processes is a plus.

Key Responsibilities

Scheduling & Production Monitoring

• Own and manage daily/weekly production schedules using enterprise scheduling tools such as Control-M/Airflow.
• Maintain calendars, job dependencies, alerts, and operational health checks.
• Monitor AWS-based data workloads and ensure timely completion of critical data pipelines.

Data Operations & ETL Support

• Operate and troubleshoot production ETL/data pipelines across cloud and enterprise environments.
• Use SQL, Python, and Spark/PySpark to investigate data issues, perform validations, and support data processing.
• Identify opportunities to improve pipeline reliability, performance, and operational efficiency.

Incident & Problem Management

• Perform first-line triage and troubleshooting of production data incidents.
• Analyze job failures, data discrepancies, performance issues, and SLA breaches.
• Document RCA, corrective actions, and preventive measures while maintaining runbooks and knowledge bases.

Change & Release Execution

• Execute production deployments, configuration changes, and data pipeline updates following established change controls.
• Prepare implementation steps, validation procedures, rollback plans, and complete deployment evidence.

Data Quality Operations

• Execute data quality checks, reconciliations, and validation activities.
• Investigate data discrepancies using SQL, Python, and available AWS/data platform tools.
• Coordinate remediation activities with Data Governance, Data Quality, and Data Engineering teams.

Automation & Continuous Improvement

• Develop Python-based automation to reduce manual operational tasks and improve monitoring.
• Optimize SQL queries and Spark/PySpark workloads where applicable.
• Identify opportunities to automate scheduling, health checks, reporting, reconciliation, and operational processes.

Service Reporting

• Monitor and report job success rates, processing latency, failures, and SLA compliance.
• Automate operational reporting and dashboards to reduce manual effort.
• Provide accurate operational metrics and evidence for service reviews and governance activities.