Back to all jobs

Data Engineer

Mid Posted about 8 hours ago Himalayas
Engineer

AI summary: Designs and maintains large-scale real-time data pipelines using Kafka, Kafka Streams, and distributed SQL engines while collaborating with stakeholders on data infrastructure improvements.

Description

• Working with stakeholders to grow the amount and kinds of customer-defined targetings we can calculate in real-time • Develop new features for our in-house streaming key/value DBs • Improve resource utilization across all stages of our data pipelines, from validation and ingestion to storage to dynamic queries • Collaborate with developers and stakeholders to design services in order to meet product and business requirements Requirements• Strong experience designing, building, implementing, and maintaining large real-time data pipelines with Kafka and Kafka Streams • Significant expertise with data consistency, throughput, and event-driven system concepts • Fluent in Python and/or Java • Familiar with designing and deploying data jobs • Experience with managing and monitoring service infrastructure and deployments • Worked with Kafka, Kafka Streams, Trino (or any distributed SQL engine), Airflow, Kubernetes, Docker, Helm, Jenkins, Prometheus, Grafana, Elasticsearch • Nice to have: Basic knowledge of AWS services such as S3 and RDS, and Terraform Benefits• A chance to be a part of a casual but professional environment where you will have a safe place to try, fail and lear • Coaching from our tech leads to advance your soft and technical skills and set your own development path • Defined and organized the onboarding process for both, the company and the project • Competitive compensation depending on experience and skills • Private pension and medical insurance for you and your family.