Full-Time Senior Data Engineer

Job Description

A Senior Data Engineer - Datalake is a problem solver who analyzes models, designs, creates, modifies, and supports a set of complex systems, processes, or operations that enable optimal business capabilities.

KEY DUTIES/RESPONSIBILITIES

• Creates Application and System Design Documents and Develops Applications / Reports / Systems / Enterprise Solutions
• Estimates component/application system level and Enterprise solution work efforts
• Creates RFI/RFP requests / responses for vendor product evaluations
• Designs, Develops, and Implements Complex Business Rules
• Responsible for fulfilling end-user requests. Provides on call support as required. Periodically provides guidance to and assists in training less experienced individuals. Routinely operates under deadlines and may be subject to extreme workloads
• Delivers personal tasks on time and leads the delivery of tasks for natural or cross-functional workgroup
• Participates on initiatives with deliverables and meets quality standards on time
• Leads cross-functional initiatives with deliverables and meets quality standards on time

Qualifications

EDUCATION

• Bachelor’s degree or equivalent work experience

EXPERIENCE

• 5+ years of experience in Data Engineering or ETL Development role
• Strong experience with PySpark, Python for building solid data pipelines
• Strong experience with Iceberg, Hive, S3, Trino
• Hands-on experience with Hadoop ecosystems, relational databases and SQL queries
• Hands-on experience with Apache Ranger, Rancher/Kubernetes preferred
• Hands-on experience with Talend, Red Point or other ETL technologies an advantage
• Experience with Agile Software Development methodologies
• Experience with GitLab, CI/CD process and ServiceNow etc..

KNOWLEDGE/SKILLS/ABILITIES

• Solid programming skills in object-oriented/functional scripting languages like Python, PySpark for building data pipelines with experience in testing, logging to ensure quality of code and data observability (Required)
• Experience in distributed systems and parallel data processing using big data tools such as Spark, PySpark, Hadoop, Kafka, Hive. (Required)
• Proficiency in querying with relational databases (Required)
• Strong knowledge of Linux/Unix-based computer systems (Required)
• Strong experience with Iceberg, Hive, S3, Trino (Required)
• Experience in building Data Processing pipelines using ETL tools like Talend, SSIS etc.. (Required)
• Hands-on experience with Apache Ranger, Rancher/Kubernetes
• Understanding of Machine Learning models and algorithms interfacing with Data Science team
• Proficiency in data visualization tools to showcase insights using Tableau, matplotlib etc..
• Nice to have AWS cloud experiences in Redshift, Lambda, Sage Maker, Glue etc..
• Experience with building Rest API
• Excellent data analytical, conceptual, and problem-solving skills
• Excellent communication skills to promote cross-team collaboration