Data Engineer

Job Description: - Develop and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Python. - Build data ingestion frameworks for structured and unstructured data sources. - Design data transformation processes using Medallion Architecture (Bronze, Silver, Gold). - Develop Delta Lake solutions with focus on data quality, reliability, and governance. - Develop and maintain Databricks Notebooks, Jobs, and Workflows. - Optimize Spark jobs and Databricks workloads for performance and cost efficiency. - Orchestrate data workflows using Azure Data Factory (ADF) and Databricks Workflows. - Implement data validation, reconciliation, monitoring, and exception handling. - Support CI/CD pipelines using Git and Azure DevOps. - Participate in solution design, code reviews, and production support. - Collaborate with business analysts, data scientists, and application teams to deliver analytical datasets.