SUMMARY
The Senior Data Engineer is responsible for designing, building, and maintaining robust data pipelines using Python, SQL Server (T-SQL, Stored Procedures), and integrating various APIs within a modern Azure infrastructure. The role supports critical data operations in regulated environments.
JOB RESPONSIBILITIES
- Design and implement scalable ETL pipelines with Python and SQL Server for data ingestion, transformation, and integration from diverse sources, including RESTful APIs.
- Develop and optimize T-SQL stored procedures to fulfill business logic and reporting needs.
- Architect secure, efficient data workflows leveraging Azure Data Factory, Azure SQL, and Azure Functions.
- Ensure data quality, lineage, and compliance with regulations such as HIPAA and SOC2.
- Collaborate with software engineering teams to align data structures and APIs with application requirements, including those based on .NET and C++ environments.
- Work cross-functionally with data analysts, security engineers, and DevOps to support comprehensive data needs.
- Monitor and troubleshoot pipeline failures and data discrepancies.
- Lead data governance initiatives, best practices, and performance tuning efforts.
- Mentor junior engineers and participate in peer code reviews.
QUALIFICATIONS
- 5+ years of experience in Data Engineering or a similar role.
- Strong proficiency in SQL Server, including stored procedures and performance tuning.
- Advanced skills in Python for ETL development and automation.
- Hands-on experience with Azure data services such as Data Factory, Blob Storage, and Azure SQL.
- Experience integrating data from RESTful APIs and third-party sources.
- Knowledge of data privacy and compliance standards, especially HIPAA and SOC2, relevant to financial and healthcare sectors.
- Strong problem-solving abilities and capacity to work independently in a fast-paced, regulated environment.
- Excellent written and verbal communication skills.
- Experience with additional Azure services like Azure Synapse, Azure DevOps, and Azure Functions.
- Understanding of data modeling and warehousing concepts.
- Exposure to cybersecurity data, SIEM tools, or SOC operations is advantageous.
- Background in MSP/MSSP environments or consulting.
- Familiarity with Power BI or other data visualization tools.
- Knowledge of .NET Framework and C#-based APIs, particularly in data consumption contexts.
JOB REQUIREMENTS
- Should be willing to accept a long-term work-from-home arrangement.
- Should be amenable to a permanent night shift schedule.
