Full-Time Azure Databricks Engineer
Resultant is hiring a remote Full-Time Azure Databricks Engineer. The career level for this job opening is Experienced and is accepting Indianapolis, IN based applicants remotely. Read complete job description before applying.
Resultant
Job Title
Posted
Career Level
Career Level
Locations Accepted
Share
Job Details
We are seeking a skilled Azure Databricks Engineer to design, develop, and optimize large-scale data processing systems using Azure Databricks. The ideal candidate will have expertise in Apache Spark, data engineering pipelines, and Azure cloud technologies. You will collaborate with cross-functional teams to build robust, secure, and scalable data solutions.
Typical Duties and Responsibilities:
- Design and implement data ingestion and transformation pipelines using Azure Databricks and other Azure data services.
- Develop ETL/ELT processes for structured, semi-structured, and unstructured data.
- Optimize and tune Apache Spark jobs for performance and cost efficiency.
- Build and manage scalable data lakehouse solutions using Delta Lake and Azure Data Lake Storage.
- Integrate Databricks with Azure Synapse Analytics, Data Factory, and other Azure resources.
- Implement security best practices: role-based access control, encryption, and data masking.
- Collaborate with data scientists and analysts to operationalize machine learning models using MLflow.
- Automate workflows with Databricks Jobs and CI/CD pipelines.
- Monitor and troubleshoot performance issues in Databricks clusters and Spark applications.
Requirements:
- 2+ years of experience with Azure Databricks and Apache Spark for large-scale data processing.
- Strong programming skills in Python, Scala, and SQL.
- Expertise in Azure Data Lake, Blob Storage, Azure Synapse, and Azure Data Factory.
- Hands-on experience with CDC tools (e.g., Debezium, SQL Server CDC).
- Expertise in configuring and managing CDC pipelines in Azure.
- Experience with Delta Lake architecture.
- Knowledge of Spark job performance tuning.
- Strong understanding of data security and governance.
- Experience with CI/CD for Databricks and Infrastructure as Code (e.g., Terraform, ARM templates).
Education and Certifications (preferred):
- Bachelor's degree in IT or related field.
- Databricks Certified Associate Developer for Apache Spark or related certification.
- Microsoft Azure Data Engineer Associate Certification.
Additional Notes:
- Legally authorized to work in the US without sponsorship.