The Senior Data Engineer will play a critical role in designing, building, and maintaining scalable data pipelines and data warehouse solutions to support business intelligence and analytics initiatives. This position ensures data integrity, accessibility, and performance across multiple platforms, enabling data-driven decision-making throughout the organization.
Responsibilities
- Design, develop, and maintain scalable data pipelines for efficient data ingestion and transformation.
- Implement and optimize data warehouse architecture, ensuring high performance and reliability.
- Utilize dbt for data transformation and modeling to support analytics workflows.
- Manage ETL/ELT processes, including incremental loads and Change Data Capture (CDC).
- Integrate data from APIs, databases, and various file formats (JSON, XML, Parquet).
- Ensure data quality, lineage, and impact analysis for all data processes.
- Collaborate with cross-functional teams to define data requirements and deliver actionable insights.
- Optimize database performance through indexing, query tuning, and advanced SQL techniques.
- Implement CI/CD practices for data pipelines and maintain version control using Git.
- Work with cloud platforms (GCP) and Snowflake to design and manage data warehousing solutions.
- Leverage GCP services such as BigQuery, Cloud Storage, Cloud Functions, Cloud Composer, Dataflow, and Pub/Sub for data processing and orchestration.
Qualifications:
- 37 years of professional experience as a Data Analyst or Analytics Engineer.
- Proven experience in building and maintaining scalable data pipelines.
- Hands-on experience with data warehouse architecture and implementation.
- Advanced SQL skills, including complex query writing, optimization, CTEs, window functions, and performance tuning.
- Proficiency in PostgreSQL (including postgres_fdw), MySQL, and SQL Server.
- Strong Python skills (Pandas, NumPy, SQLAlchemy), Bash/shell scripting, REST API integration, and Git version control.
- Expertise in ETL/ELT design and implementation, handling structured and semi-structured data, and CDC processes.
- Deep understanding of Snowflake architecture (streams, tasks, time travel, optimization).
- Familiarity with dimensional modeling and star schema design.
- Hands-on experience with GCP services: BigQuery, Cloud Storage, Cloud Functions, Cloud Composer, Dataflow, and Pub/Sub.
