Custom Software Engineer - Pyspark

As a Custom Software Engineer, a typical day involves designing and developing tailored software solutions that enhance various system components and applications. The role requires working within dynamic project environments, applying modern development frameworks, and following agile methodologies to ensure the delivery of scalable and efficient software. Collaboration with cross-functional teams is a key aspect, enabling the continuous improvement and adaptation of solutions to meet evolving business requirements and technical challenges. Roles & Responsibilities: - Expected to be an SME, collaborate and manage the team to perform.- Responsible for team decisions.- Engage with multiple teams and contribute on key decisions.- Provide solutions to problems for their immediate team and across multiple teams.- Lead the coordination of project activities to ensure timely delivery and quality standards.- Mentor junior team members to support their professional growth and skill development.- Facilitate effective communication between stakeholders to align technical solutions with business goals. Professional & Technical Skills: - Must To Have Skills: Proficiency in PySpark.- Strong experience in developing scalable data processing pipelines using distributed computing frameworks.- Ability to optimize and troubleshoot complex data workflows for performance and reliability.- Familiarity with integrating PySpark applications within cloud-based environments and data platforms.- Experience in writing clean, maintainable code following best practices and coding standards.- Competence in collaborating with cross-functional teams to design and implement end-to-end software solutions.- Extensive experience in design, build and deployment of Python-based applications with Experience in various AWS services such as EMR, API Gateway, RDS instance, and Lambda.- Hands-on experience in relational databases preferably Oracle and PostgreSQL and writing complex SQL queries.- Understand complex data sets and ETL processes, and how they can be optimized using Spark.- Collaborate with other teams to integrate Spark jobs into the overall data pipeline.- Knowledge of Java and Snowflake would be an added advantage.- Ability to monitor and tune data loads and queries. Additional Information: - The candidate should have minimum 5+ years of overall IT experience, which includes building & deployment of Bigdata applications using PySpark and AWS cloud.- This position is based at our Bengaluru office.- A 15 years full time education is required.