Data Engineer - Data Platform Team

Description We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications. The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies , with hands-on expertise in Apache Spark and Python Scala . You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions. Key Responsibilities - Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets. - Build and optimize distributed data applications using Apache Spark and Python Scala . - Develop reliable, high-performance data pipelines for batch and streaming workloads . - Design and manage data workflows using Apache Airflow . - Build and operate data workloads on AWS , with strong usage of Amazon S3 for large-scale data storage. - Work with large datasets to ensure data quality, consistency, reliability, and performance . - Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions. - Optimize data workflows for scalability, reliability, performance, and cost efficiency . - Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance. Requirements Candidates who demonstrate: - 5+ years of experience in Data Engineering, Big Data Engineering, or a similar role. - Strong hands-on experience with Apache Spark and Scala. - Experience designing, building, and maintaining large-scale ETL pipelines. - Strong hands-on experience with AWS, particularly Amazon S3. - Hands-on experience with Apache Airflow for workflow orchestration and scheduling. - Strong SQL skills and a solid understanding of distributed data processing concepts. - Experience working with batch and/or streaming data pipelines. - Excellent debugging, problem-solving, and performance optimization skills. - Strong communication and collaboration skills. Good to Have - Experience with Databricks and the broader Databricks data platform. - Familiarity with streaming technologies such as Apache Kafka. - Experience working on large-scale data platforms handling high-volume data workloads. - Exposure to additional AWS data services and cloud-native data architectures. Skills:- Apache Kafka, databricks and ETL