Full-Time Azure Databricks Engineer

We are seeking a skilled Azure Databricks Engineer to design, develop, and optimize large-scale data processing systems using Azure Databricks. The ideal candidate will have expertise in Apache Spark, data engineering pipelines, and Azure cloud technologies. You will collaborate with cross-functional teams to build robust, secure, and scalable data solutions.

Typical Duties and Responsibilities:

• Design and implement data ingestion and transformation pipelines using Azure Databricks and other Azure data services.
• Develop ETL/ELT processes for structured, semi-structured, and unstructured data.
• Optimize and tune Apache Spark jobs for performance and cost efficiency.
• Build and manage scalable data lakehouse solutions using Delta Lake and Azure Data Lake Storage.
• Integrate Databricks with Azure Synapse Analytics, Data Factory, and other Azure resources.
• Implement security best practices: role-based access control, encryption, and data masking.
• Collaborate with data scientists and analysts to operationalize machine learning models using MLflow.
• Automate workflows with Databricks Jobs and CI/CD pipelines.
• Monitor and troubleshoot performance issues in Databricks clusters and Spark applications.

Requirements:

• 2+ years of experience with Azure Databricks and Apache Spark for large-scale data processing.
• Strong programming skills in Python, Scala, and SQL.
• Expertise in Azure Data Lake, Blob Storage, Azure Synapse, and Azure Data Factory.
• Hands-on experience with CDC tools (e.g., Debezium, SQL Server CDC).
• Expertise in configuring and managing CDC pipelines in Azure.
• Experience with Delta Lake architecture.
• Knowledge of Spark job performance tuning.
• Strong understanding of data security and governance.
• Experience with CI/CD for Databricks and Infrastructure as Code (e.g., Terraform, ARM templates).

Education and Certifications (preferred):

• Bachelor's degree in IT or related field.
• Databricks Certified Associate Developer for Apache Spark or related certification.
• Microsoft Azure Data Engineer Associate Certification.

Additional Notes:

• Legally authorized to work in the US without sponsorship.

What is the last date for applying to the job?

Which countries are accepted for this remote job?