Staff Data Engineer

Job Description Position Overview We seek a results-oriented Data Engineer with a minimum of 6+ years of experience in data pipeline development within cloud environments. The successful candidate shall be responsible for designing, constructing, and optimizing Azure-based data ingestion and transformation pipelines using PySpark and Spark SQL. This role requires collaboration with cross-functional teams to deliver high-quality, reliable, and scalable data solutions. Duties and Responsibilities - Design, develop, and maintain high-performance ETL/ELT pipelines using PySpark and Spark SQL using cloud-native components in Databricks - Build and orchestrate data workflows in AZURE . - Implement hybrid data integration between on-premise databases and Azure Databricks using tools such as ADF, HVR/Fivetran, and secure network configurations. - Enhance/optimize Spark jobs for performance, scalability, and cost efficiency. - Implement and enforce best practices for data quality, governance, and documentation. - Collaborate with data analysts, data scientists, and business users to define and refine data requirements. - Support CI/CD processes and automation tools and version control systems like Git. - Perform root cause analysis, troubleshoot issues, and ensure the reliability of data pipelines.