Data Engineer

JOB SUMMARY

The Data Engineer is responsible for designing, developing, and maintaining data processing systems that ensure efficient data collection, transformation, and integration. The role focuses on designing scalable architectures for batch and real-time processing, applying transformation and modeling techniques to ensure data quality, consistency, and availability for analytics and decision support.

PRINCIPAL DUTIES

1. Build and maintain pipelines to ingest high-volume, high-velocity data from shopfloor equipment, testers, PLCs, IoT gateways, and external systems using APIs, message queues, and industrial protocols.

2. Develop scalable ETL/ELT workflows and real-time streaming pipelines for sensor, production, and transactional data; ensure pipelines support both batch analytics and low-latency operational insights.

3. Design relational, dimensional, and time-series data models optimized for manufacturing analytics, BI dashboards, and AI/ML workloads in cloud and on-premise environments.

4. Write and optimize SQL queries, stored procedures, and Python-based scripts to process, validate, and automate data transformations; leverage orchestration frameworks for workflow scheduling.

5. Apply data governance frameworks (lineage, metadata, access control) and implement validation, reconciliation, and anomaly detection to ensure compliance and trusted analytics.

6. Monitor pipeline performance, automate failure recovery, and fine-tune query execution and resource usage for reliable, production-grade pipelines.

7. Enable AI/ML workflows by preparing structured, high-quality training data, managing feature pipelines, and supporting agent-based automation.

8. Work with BI analysts, data scientists, and integration engineers to align pipelines with enterprise reporting, predictive models, and other initiatives.

9. Produce technical documentation, data dictionaries, and lineage diagrams; ensure reproducibility of data workflows through version control and CI/CD practices.

10. Evaluate emerging data technologies to enhance scalability, performance, and automation.

11. Perform other related tasks as needed to support enterprise data initiatives and business objectives.

JOB SPECIFICATIONS

1. Formal Education:

- Graduate of Computer Science, Information Technology, Data Science, Engineering, or other related fields.

2. Experience:

- 2-3 years of experience in data engineering, data integration, or related fields.

3. Training/Skills:

- Proficient in data pipeline development, ETL processes, and data integration.

- Experienced in database design and warehousing and optimization.

- Familiar with big data frameworks for large-scale processing.

- Hands-on experience with cloud platforms for data storage and analytics.

- Skilled in workflow orchestration for managing pipelines and machine learning workflows.

- Knowledgeable in data security, privacy, and governance best practices.

- Strong skills in documentation, stakeholder engagement, and presentations.