We are seeking a highly skilled and analytical Senior Data Engineer to join our growing Data & Analytics team. In this role, you will be instrumental in designing, building, and optimizing our next-generation data platform. You will lead the convergence of our established Databricks environments with our emerging Microsoft Fabric adoption strategy.

The ideal candidate is someone who feels equally at home writing highly optimized PySpark/SQL code in Databricks notebooks as they are architecting end-to-end Lakehouse solutions within MS Fabric utilizing OneLake, Data Factory, and Synapse Data Engineering.

Key Responsibilities

1. Data Pipeline Engineering & Architecture

• Design & Build: Develop scalable, robust batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL across both Databricks and Microsoft Fabric environments.
• Lakehouse Implementation: Implement and maintain the Medallion Architecture (Bronze, Silver, Gold layers) ensuring ACID compliance and schema enforcement using Delta Lake.
• Data Integration: Ingest and unify structured, semi-structured, and unstructured data from diverse enterprise sources (e.g., ERPs, CRMs, APIs, Webhooks).

2. Platform Optimization & Governance

• Performance Tuning: Optimize Databricks cluster utilization and Spark jobs (partitioning, caching, memory management).
• Capacity Management: Monitor and manage Fabric capacity utilization, workspace sizing, and Compute Unit (CU) consumption to ensure cost-effective operations.
• Data Governance: Enforce data security, lineage, and compliance frameworks utilizing Databricks Unity Catalog, Microsoft Purview, and Fabric item-level/row-level security.

3. Analytics & Semantic Modeling

• Semantic Layer: Design and optimize Fabric-native semantic models and Power BI Direct Lake configurations to ensure sub-second reporting performance.
• Collaboration: Partner closely with Data Scientists, BI Developers, and business stakeholders to turn raw data into production-ready data products.

4. DataOps & Mentorship

• CI/CD: Automate deployment pipelines for data workloads using Azure DevOps or GitHub Actions across Dev, Test, and Prod environments for both Databricks and Fabric workspaces.
• Data Quality: Implement automated data quality frameworks to ensure data reliability and proactive monitoring.

Leadership: Provide technical mentorship, code reviews, and architectural guidance to junior and mid-level data engineers

Required Technical Qualifications

• Experience: 5+ years of dedicated data engineering experience in an enterprise environment.
• Databricks Expertise: Deep, hands-on experience with Azure Databricks, Delta Live Tables, Unity Catalog, Autoloader, and workflow orchestration.
• Microsoft Fabric Expertise: Proven hands-on exposure to the MS Fabric ecosystem, including OneLake, Fabric Lakehouses/Warehouses, Data Factory, and Dataflow Gen2.
• Programming Languages: Expert-level proficiency in Python (PySpark) and SQL. Familiarity with Scala or Java is a major plus.
• Cloud Infrastructure: Strong background in Azure cloud resources (ADLS Gen2, Azure Key Vault, Microsoft Entra ID).
• Data Modeling: Strong understanding of dimensional modeling, Star Schemas, and Data Vault techniques.

Preferred Qualifications & Certifications

• Experience implementing Generative AI / Agentic AI data ingestion and vector database integration.

Certifications:

• Databricks Certified Data Engineer Professional
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)

Show more
Show less