Azure Data Engineer

Job Summary We are seeking a skilled Azure Data Engineer with hands-on experience in Azure Data Services, Azure Databricks, Python, PySpark, and SQL. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions to support business intelligence, analytics, and reporting requirements. Key Responsibilities - Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and PySpark. - Build and optimize data processing workflows using Python and SQL. - Develop and manage data ingestion pipelines from multiple structured and unstructured data sources. - Work with Azure Data Factory (ADF) to orchestrate and schedule data pipelines. - Implement data transformation and cleansing logic using PySpark. - Optimize SQL queries and Spark jobs for performance and scalability. - Collaborate with data architects, analysts, and business stakeholders to understand data requirements. - Ensure data quality, integrity, and governance across the data platform. - Monitor, troubleshoot, and resolve production data pipeline issues. - Follow coding standards, version control, and CI/CD best practices. Required Skills - Strong experience with Microsoft Azure cloud services. - Hands-on experience with Azure Databricks. - Strong programming skills in Python. - Expertise in PySpark for large-scale data processing. - Strong SQL skills, including query optimization and performance tuning. - Experience with Azure Data Factory (ADF). - Knowledge of Delta Lake, Spark SQL, and Databricks notebooks. - Experience with Git or Azure DevOps for source code management. - Understanding of data warehousing concepts and ETL/ELT processes. Preferred Skills - Experience with Azure Synapse Analytics. - Knowledge of Delta Live Tables (DLT). - Experience with Azure Data Lake Storage (ADLS Gen2). - Familiarity with Unity Catalog and data governance. - Exposure to CI/CD pipelines and infrastructure-as-code. - Experience working in Agile/Scrum environments. Skills:- Azure Data Engineer, Azure Databricks, Python, PySpark and SQL