Full-Time Data Engineer

Job Overview

The Data Engineer will be responsible for designing and building modern data platforms to support data-driven decision making. The Data Engineer will execute technical implementation of data engineering and visualization projects.

The Data Engineer will help build a data and analytics consulting practice by taking part in recruiting efforts, creating technical collateral, and staying on top of technology trends with ongoing training and certifications. The entire consulting team will be responsible for building long-term strategic relationships with clients and participating in all aspects of project delivery

Key Responsibilities

• Design, implement, and develop data pipelines to collect and process large amounts of data from various sources.
• Implement data storage solutions that are scalable, secure, and efficient, such as data warehouses and databases.
• Develop and implement data validation and testing processes to ensure that data is processed accurately and efficiently.
• Automate data collection, processing, and reporting processes to minimize manual work and improve efficiency.
• Create high quality documents to capture problem statements, requirements, solutions and designs.
• Contribute to the development of reusable, repeatable collateral for use across the practice.
• Implement data quality checks and validation processes to ensure the accuracy, consistency, and security of data.
• Monitor data pipelines and processes, troubleshoot issues, and implement solutions to prevent recurrence
• Maintain comprehensive documentation of data architectures, processes, and workflows to facilitate knowledge sharing and collaboration.

Key Requirements

Must Have

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 3-5 years of experience in data engineering
• Excellent problem-solving, organization, debugging, and analytical skills.
• Ability to work independently and in a team environment.
• Excellent communication skills for effectively expressing ideas to team members and clients.
• Understanding of relational database concepts and SQL.
• Hands on experience in one RDBMS (MSSQL/MySQL/PostgreSQL)
• Knowledge of cloud computing platforms, such as Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure
• Hands on Experience in Data processing frameworks such as Apache Spark/Beam/Flink
• Experience in building data pipelines using orchestration tools or equivalent cloud services like Apache Airflow or NiFi, as well as ETL tools such as Azure Data Factory, AWS Glue, Matillion, or similar platforms.
• Hands on Experience with any one of data warehousing solutions, such as Snowflake/ Google BigQuery, or Databricks
• Debug and optimize existing data infrastructure and processes as needed.
• Experience with at least one programming language, such as Python/ Java

Nice To Have

• Experience building large-scale, high throughput, 24x7 data systems.
• Any Data engineering certifications on any of the clouds
• Experience with Visualization tools, such as Power BI, Looker, Tableau, and QuickSight.
• Experience in designing, implementing, and managing event-driven data pipelines using Kafka or similar cloud-native streaming services like AWS Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub.
• Exposure to machine learning algorithms, AI, and/or LLM, with implementation in practice.
• Experience with legacy data systems (e.g. Hadoop, Informatica).
• Experience in Migrating Data Engineering workloads from on-prem to cloud or between clouds
• Experience in GCP Data services is preferred