Full-Time Senior Data Engineer (AI and ML frameworks)

Data Standardization and Transformation: Convert diverse data structures from various EHR systems into a unified format based on FHIR standards. Map and normalize incoming data to the FHIR data model, ensuring consistency and completeness.

Kafka Integration: Consume and process events from the Kafka stream produced by the Data Writer Module. Deserialize and validate incoming data to ensure adherence to required standards.

Data Segmentation: Separate data streams for warehousing and AI model training, applying specific preprocessing steps for each purpose. Prepare and validate data for storage and machine learning model training.

Error Handling and Logging: Implement robust error handling mechanisms to track and resolve data mapping issues. Maintain detailed logs for auditing and troubleshooting purposes.

Data Ingestion and Processing: Use LLMs to extract structured data from EHRs, research articles, and clinical notes. Ensure semantic consistency and interoperability during data ingestion.

Knowledge Graph Construction: Integrate extracted data into a knowledge graph, representing entities and relationships for semantic data integration. Implement contextual understanding and querying of complex relationships within the knowledge graph (KG).

Advanced Predictive Modeling: Leverage KGs and LLMs to enhance data interoperability and predictive analytics. Develop frameworks for contextualized insights and personalized medicine recommendations.

Feedback Loop: Continuously update the knowledge graph with new data using LLMs, ensuring up-to-date and relevant insights.

Work Closely with Cross-Functional Teams: Collaborate with data scientists, AI specialists, and software engineers to design and implement data processing solutions. Communicate effectively with stakeholders to align on goals and deliverables.

Contribute to Engineering Culture: Foster a culture of innovation, collaboration, and continuous improvement within the engineering team.

Technical Skills:

1. Deep understanding of patterns and software development practices for event-driven architectures
2. Hands-on experience with stateful stream data processing solutions (Kafka or similar streaming platforms)
3. Strong knowledge of data serialization/deserialization using various data formats (at minimum JSON and Avro), and integration with schema registries
4. Proven Python software development expertise, with experience in data processing and integration
5. Practical experience building end-to-end solutions with Apache Flink or a similar platform
6. Experience with containerization and orchestration using Kubernetes (K8s) and Helm, especially on Google Kubernetes Engine (GKE)
7. Familiarity with Google Cloud Platform (GCP) or a similar cloud platform
8. Hands-on experience implementing data quality solutions for schema-on-read or schema-less data
9. Hands-on experience integrating with Apache Kafka, particularly the Confluent Platform
10. Familiarity with AI and ML frameworks
11. Proficiency in SQL and experience with both relational and NoSQL databases
12. Experience with graph databases like Neo4j or RDF-based systems
13. Experience in the healthcare domain and familiarity with healthcare standards such as FHIR and HL7 for data interoperability

WOULD BE A PLUS: Experience with web data scraping.

Personal Profiles:

• Strong problem-solving skills, with the ability to design innovative solutions for complex data integration and processing challenges
• Excellent communication skills, with the ability to articulate complex technical concepts and work effectively with various stakeholders
• Commitment to improving healthcare through data-driven solutions and technology
• Stay abreast of the latest technologies and industry trends while continually improving your skills and knowledge
• Ability to work in a collaborative environment, valuing diverse perspectives and contributing to a positive team culture

What is the last date for applying to the job?

Which countries are accepted for this remote job?