Full-Time Senior Data Engineer - Databricks Optimization Specialist

We are a dynamic team managing the central cloud infrastructure, including the Databricks platform. Our mission is to empower teams, ensuring high-performance, cost-effective solutions optimizing workloads and improving efficiency.

We seek a Senior Data Engineer focused on Databricks optimization to help users fine-tune workloads and contribute to cloud infrastructure enhancements.

As a Senior Data Engineer – Databricks Optimization Specialist, you'll optimize Databricks workloads, ensuring high performance while minimizing costs and resource usage.

You'll work closely with users to enhance Spark query performance, fine-tune resource allocation, and implement best practices in Databricks.

Key Responsibilities:

• Optimize Databricks workloads for efficiency and cost reduction.
• Analyze and fine-tune Spark queries for optimal performance.
• Provide best practices for Databricks resource allocation and cluster management.
• Troubleshoot performance bottlenecks.
• Implement cost-saving measures and automate workload management.
• Develop and document performance optimization strategies.
• Stay updated with Databricks and Spark advancements.
• Support cloud infrastructure projects and advise on Databricks-Azure integrations.
• Assist in cloud infrastructure implementation and support various use cases.

Education: Degree in Computer Science, Data Engineering, or a related field (or equivalent experience).

Experience: Proven experience as a Data Engineer with a strong focus on Databricks optimization. Hands-on expertise in PySpark and Spark performance tuning. Experience in query performance tuning and workload optimization. Knowledge of cost-saving techniques within Databricks.

Know-how: Deep understanding of distributed computing and big data processing. Expertise in Databricks resource allocation, cluster management, and automation. Familiarity with Azure cloud services.

Languages: Fluency in English (written and spoken) is required.

Working Style and Methods: Strong problem-solving skills and ability to troubleshoot performance bottlenecks. Ability to work independently while collaborating with teams. Proactive in staying updated with Databricks and Spark advancements.

Personality: High level of initiative, ownership, and autonomy. Excellent communication and collaboration skills. Passion for efficiency, optimization, and continuous improvement.

What is the last date for applying to the job?

Which countries are accepted for this remote job?