Senior Data Engineer
- Posted 19 days ago
- Be among the first 20 applicants
Job Description
Responsibilities
- Design, develop, test, deploy, and maintain robust and scalable ELT data pipelines using dbt (data build tool) for data transformation within Snowflake.
- Orchestrate and schedule complex data workflows using Apache Airflow, ensuring timely and reliable data delivery.
- Develop connectors and scripts (primarily in Python) to extract data from various source systems (APIs, databases, files, streaming platforms) and load it into Snowflake.
- Implement data ingestion strategies (batch and streaming) using Snowflake's capabilities (e.g., Snowpipe, external stages).
- Optimize Snowflake warehouse usage, query performance, and overall data platform efficiency.
- Manage and monitor Snowflake resources, ensuring cost-effectiveness and scalability.
- Implement and enforce data governance, security (e.g., RBAC, data masking), and privacy best practices within Snowflake.
- Assist in schema design, table optimization (clustering, partitioning), and data loading strategies.
- Solve key business problems through using an appropriate mix of strategic thinking and computational methods.
- Develop and uphold best practices with respect to change management, documentation and data protocols.
- Bachelor/Master degree in Analytics, Data Science, Mathematics, Computer Science, Information Systems, Computer Engineering, or related technical field.
- Demonstrated mastery of complex SQL queries, analytical functions, stored procedures, and performance tuning.
- 5+ years of hands-on experience with SQL or any Data warehouse/Data Lake, including data loading, transformations, performance optimization, and security features.
- Proven experience in building and managing complex data transformation pipelines using dbt, including Jinja templating, macros, tests, and documentation.
- Solid experience in designing, developing, and deploying production-grade data pipelines using Apache Airflow (DAGs, Operators, Sensors, XComs).
- Strong Python scripting skills for data manipulation, API integrations, and Airflow DAG development.
- Analytical and independent problem solver. Meticulous with high attention to detail.
- Strong communicator with ability to switch hats between data/technical speak and business/layperson speak.
- Solid understanding of data warehousing concepts, dimensional modeling (star/snowflake schemas), and data lake architectures.
- Deep understanding of Extract, Load, Transform (ELT) or ETL principles and best practices.
- Familiarity with data quality frameworks, data lineage, and data governance principles.
- Experience working in a digital banking or financial services environment is highly advantageous, with an understanding of financial data concepts and regulatory requirements.
More Info
Key Skills
data quality frameworks
data ingestion
About Company
Skills:
data segmentation , Data structures, Apache Spark, Data Modeling, Data mining, Sql, Metadata Management, Data Transformation, Databricks, DataFlow, Python, dependency management, Database design and development, MLflow, Prefect, ETL processes, Experiment tracking, workload management, A/B testing platforms, Feature stores
Skills:
Apache Airflow, Data Analytics, Hadoop, Apache Spark, Kafka, Data Visualization, Python, Sql, Etl, ELT, dashboard development
Skills:
Pyspark, Sql, Azure Synapse, Azure Data Factory, Python, Parquet, OneLake, dimensional data modelling, Medallion Architecture, Delta formats, Lakehouse design, Microsoft Fabric
Skills:
snowflake , BigQuery, T-sql, Power Bi, Ms Sql Server, PostgreSQL, PowerShell, Avro, Sap Ecc6, SSIS, Ssas, Databricks, Excel, Apache Arrows, Apache Iceberg, Parquet, DuckLake, DuckDB, Synapse, Apache Hudi, ADLS2
Skills:
Big Data Hadoop, Apache Airflow, Kubernetes, MinIO