Data Engineer
- Posted a month ago
- Over 50 applicants have applied
Job Description
Roles and Responsibilities
- Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Databricks.
- Develop complex ETL processes using Python scripts and SQL queries to process large datasets stored in Azure Data Lake Storage.
- Collaborate with cross-functional teams to gather requirements for new data warehousing solutions and implement scalable architectures on Azure Data Warehouse.
- Troubleshoot issues related to ADF pipeline failures, Azure Synapse connections, and SQL queries.
More Info
Job Type:
Industry:
Function:
Employment Type:
About Company
Skills:
Hive, Hadoop, Gcp, Pyspark, Spark, Sql
Skills:
S3, Github, Pyspark, Lambda, Ec2, Kinesis, Docker, Shell scripting, Python, AWS, Scala, Redshift, Sql, Git, Kubernetes, GenAI, Airflow, AI coding assistants, LLM tools, RAG pipelines, LangChain, prompt engineering, dbt, Glue, LlamaIndex
Skills:
Github, Pyspark, Pandas, Numpy, Gcp, Docker, Databricks, Azure, Python, AWS, scikit-learn, Great Expectations, Kedro
Skills:
Power Bi, SQL Server, Apache Spark, Tableau, Data Governance, Data Modeling, Data Warehousing, Sql, MySQL, Data Integration, Oracle, Azure, Python, AWS, Data lakes, Big data platforms
Skills:
Spark SQL, Distributed Computing, Pyspark, Data Warehousing, Apache Spark, Databricks, Sql, ELT, Etl, Delta Lake, Lakehouse architecture