Snr Data Engineer

JD – Snr Software Engineer (ETL) Experience & Expectations : - Leverage extensive experience (4 to 8 years overall ETL experience, to assist in solution design and delivery along with build of new ETLs). - We are seeking an experienced ETL Developer with strong expertise in Big Data (Spark, Cloudera). - Experience orchestrating workflows using AWS Step Functions (state machines) for reliable and scalable data pipelines. - Ability to implement end-to-end serverless data architectures integrating Glue, Lambda, S3, and Redshift Core Responsibilities : - Build and maintain high volume ETL/ELT pipelines across Hadoop (HDFS, Hive, Spark, Kafka) and AWS (Glue, EMR, Lambda, Step Functions, Redshift). - Develop distributed data processing solutions using PySpark, Spark SQL, and scalable cloud serverless patterns. - Implement reusable data ingestion frameworks for batch, ability to design & implement Orchestration process and Leverage AI - Optimize data workflows using partitioning, bucketing, compression, file formats (Parquet/ORC). - Understanding hybrid data lake architectures using S3 + HDFS, ensuring data governance and best practices are adheres - Experience to deliver complex projects in an Agile environment - Assist in Design and build the robust, scalable and secure software solutions across the having no/least adoption - Define clear technical specifications and make architecture decisions that align with business goals and long-term scalability. - Implement best practices (including secure code guidelines) through the implementation of unit tests, automation, leverage and code reviews. Drive continuous improvement in code quality and maintainability. - Troubleshooting issues and proactively solving problems as they arise, ensuring the smooth operation of full stack applications - Ability to understand the data flow diagram, data modelling and Lineages - Job orchestration using Airflow, Control M, Step Functions, or event-driven triggers. - Ensure data is protected and compliant with regulatory standards. - Work closely with business stakeholders to enable high quality datasets. - Work on best practice adoption and provide guidance to peers/juniors in team. - Ability to respond on incidents, and troubleshooting Spark performance issues, job failures, and cluster bottlenecks. - Collaborate closely with team members, QA and cross product teams to streamline release processes. - Collaborate with business stakeholders to gather, analyse, and translate data into technical solutions Technical Skills : - Strong experience with the AWS data stack (S3, Glue, EMR, Lambda, Kinesis, Redshift, Step Functions etc.,). - Strong hands-on expertise in Scala, PySpark , Spark optimization techniques, HiveQL, and distributed computing. - Good understanding of Hadoop ecosystem (HDFS, Hive, Spark, YARN, Kafka). - Good work experience in SQL in hive and impala - Proficiency in at least one scripting/programming language: Python, Shell scripting . - Strong experience with CI/CD , GitHub, Git commands. - Expertise in ETL and Data Warehousing and cloud concepts. - Good understanding of data modelling (star/snowflake), partitioning strategies, and schema evolution. - Expertise in data profiling and decision making. - Able to understand, design and create data flow diagrams. - Able to understand the architecture and design end-to-end data flow. - Hands-on experience with Airflow, or Control‑M , or other orchestrators. - To monitor and support BAU and year end activities, if needed. - Exposure to security and compliance aspects in Cloud. - Familiarity with serverless patterns and containerization (Docker, ECS/EKS). Other Requirements - Strong logical and analytical, problem-solving, and communication skills. - Communicate effectively and concisely with multiple stakeholders and coordinate and collaborate with cross functional teams. - AWS certifications (Data Engineer, or Developer) are a plus. Detail-Oriented and proactive in problem-solving and issue resolution We offer you a competitive total rewards package, continuing education & training, and tremendous potential with a growing worldwide organization. DISCLAIMER: Nothing in this job description restricts management's right to assign or reassign duties and responsibilities of this job to other entities; including but not limited to subsidiaries, partners, or purchasers of Alight business units. .