Job Summary

We are seeking a highly skilled Senior Data Engineer to join the APEX engineering team and contribute to the design, development, and evolution of our next-generation data platform.

The successful candidate will play a key role in building scalable and reliable data pipelines on Databricks, leveraging PySpark and Python to process large-scale datasets and support critical business processes. The role requires close collaboration with Backend Engineering teams to integrate data services with a Java-based ecosystem through APIs and microservices.

Data Platform Development:

• Design, develop, and maintain enterprise-grade data pipelines using PySpark on different systems as Databricks.
• Build efficient ETL/ELT processes for large-scale data ingestion, transformation, and delivery.
• Design and optimize Delta Lake data models and storage structures.
• Implement data quality controls, reconciliation mechanisms, and monitoring solutions.
• Analyze and optimize Spark workloads for performance and cost efficiency.

Backend Integration:

• Develop integrations between Databricks workloads and Java/Spring Boot applications.
• Design and consume REST APIs for data exchange between platform components.
• Support event-driven and service-oriented architectures.
• Collaborate with backend teams to ensure end-to-end data flow reliability and scalability.

Software Engineering Excellence:

• Develop reusable Python frameworks and libraries to standardize data processing.
• Apply software engineering best practices including clean code, testing, code reviews, and documentation.
• Contribute to CI/CD implementation and deployment automation.
• Participate in architecture reviews and technical design discussions.
• Strong Adoption of AI tools, AI agents and Agentic AI to bring more efficiency to deliver value. Usage of AI within databricks. Proactively propose news usage / AI tools on our technical environment being open minded. 

Operational Excellence:

• Ensure reliability, observability, and supportability of production pipelines.
• Investigate and resolve performance, stability, and data consistency issues.
• Participate in production support and root cause analysis activities.
• Implement monitoring and alerting mechanisms across the data platform.

Leadership & Collaboration:

• Collaborate with architects, product owners, business analysts, and development teams.
• Contribute to technical roadmaps and platform modernization initiatives.

Technical Skills:

• 7+ years of experience in Data Engineering or Software Engineering.
• Strong expertise in: 
  • Databricks
  • Apache Spark / PySpark
  • Python
  • SQL
• Proven experience designing and implementing enterprise data pipelines.
• Strong understanding of: 
  • Delta Lake
  • Data Modeling
  • Data Quality
  • Metadata Management
  • Data Governance
• Experience integrating data platforms with Java/Spring Boot applications.
• Solid knowledge of REST APIs and Microservices architectures.
• Experience with Git and CI/CD practices.

Soft Skills:

• Strong analytical and problem-solving abilities.
• Excellent communication and stakeholder management skills.
• Ability to lead technical discussions and influence architecture decisions.
• Self-driven, proactive, and results-oriented mindset.
• Strong collaboration skills within multicultural and distributed teams.

Technical Environment:

• Data Platform: Databricks, Dataworks
• Languages: Python, PySpark, SQL, Java
• Backend: Spring Boot, REST APIs
• Cloud: Microsoft Azure, Alibaba
• Storage:  Azure storage, Parquet
• DevOps: Azure DevOps, Git
• Monitoring: Grafana, ELK