Contractor LLM Data Engineer

We are looking for a Data Engineer with expertise in data pipelines, workflow orchestration, and data integration.

Responsibilities will include:

• Monitoring and maintaining data pipelines, troubleshooting new errors, and addressing format drift
• Extracting and enriching additional data elements from diverse sources
• Reprocessing and validating large datasets in batch workflows
• Designing and integrating new data sources into existing pipelines
• Aligning and integrating extracted data with the core application data model to ensure consistency and usability
• Participating in code reviews
• Contributing to project success by keeping a close eye on team velocity, project scope, budget, and timeline
• Negotiating with clients to align project scope with budget and timeline, if needed

Must Have Competencies:

• Strong proficiency in Python
• Experience with data/workflow orchestration tools (e.g., Prefect, Dagster, Airflow)
• A thorough understanding ETL & data transformation for the ingestion of industry standard LLMs (OpenAI, Claude, etc)
• Familiarity with Large Language Models (LLMs)
• Skilled in interfacing with APIs (OpenAI, Google Gemini/Vertex, etc.) using wrapper libraries such as Instructor, LiteLLM, etc.
• Practical experience in prompt engineering
• Ability to work with structured outputs and potentially tool calling
• 5+ years of general experience in backend (Ruby on Rails, Elixir Phoenix, Python Django, or Node Express) and/or native app development (React Native, Flutter, Android, AOSP, Kotlin/Java).

Nice to Have Competencies:

• Experience with Google Cloud Platform (GCP), particularly Cloud Run and Cloud Tasks
• Knowledge of search technologies, including embeddings and vector databases for semantic search, as well as keyword-based search (BM25)
• Familiarity with PySpark for batch data processing
• Experience working with LLMs, Vector Databases, and other generalist AI-enabled application patterns
• Client-facing experience
• Product management experience
• Engineering management experience

What is the last date for applying to the job?

Which countries are accepted for this remote job?