Contractor Software Development Advisor

Key Responsibilities:
Work in an agile environment with a development team to:

• Design, develop, and maintain scalable data pipelines using software development patterns.
• Implement data processing solutions using Python and PySpark on a cloud-native Lakehouse data platform.
• Write efficient SQL queries to extract, transform, and load data.
• Collaborate with product management and analysts to understand data requirements and deliver solutions.
• Optimize and troubleshoot data pipelines for performance and reliability.
• Ensure data quality and integrity through comprehensive testing and validation processes.
• Follow DevOps principles and use CI/CD to deploy and operate data pipelines.

Required Skills:

• Proficiency in Python and PySpark.
• Strong experience with SQL and database management.
• Knowledge of software development patterns and best practices in data engineering.
• Experience with ETL/ELT processes and data pipeline orchestration.
• Proficiency developing using version control, automated testing, and deployments using git-based tools like GitHub and GitHub Actions.

Preferred Skills:

• Understanding of testing methodologies for data pipelines, including unit testing, integration testing, and end-to-end testing.
• Knowledge of data governance and data security best practices.
• Familiarity with data warehousing concepts and tools.
• Experience with cloud platforms (e.g., Azure, AWS, Google Cloud Platform) with Azure preferred.
• Knowledge of big data technologies (e.g., Microsoft Fabric, Azure Synapse, Lakehouse, Databricks).
• Familiarity with advanced data orchestration tooling and development frameworks like dbt or Airflow.
• Experience working in a healthcare related industry.

Qualifications:

• Intense intellectual curiosity and an ability to view old problems with a fresh perspective.
• Bachelor s degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in data engineering or a related role.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork abilities.

ESSENTIAL JOB FUNCTIONS

• Specific vision abilities, including close vision, operate computer screen for extending periods of time
• May be required to sit or stand for extended periods of time
• Ability to read, write and speak the English language fluently