Data Engineer

Description: Mandatory skills | - Corporate Banking Knowledge. - Strong hands-on experience in developing, optimizing, and managing ETL pipelines, particularly for large-scale, real-time data. - Advanced proficiency in SQL. - Expertise in Python programming languages. - In-depth experience with Azure services (e.g., Azure Data Factory, Azure Databricks) - Strong experience with Cloudera/Hadoop ecosystem, including HDFS, Hive, Spark, - Knowledge of data governance, data quality, and data security best practices. - Any Data engineer Professional Certification. - Strong analytical and problem-solving skills | Desired Skills | - Real-time Data Processing Basic experience with machine learning models and tools such as Azure ML | Roles & Responsibilities | - Data Pipeline Management Design and optimize ETL pipelines to integrate corporate banking data into the data lake (Cloudera/Hadoop, Azure + Databricks) - Data Integration & Transformation Integrate data from multiple sources, transforming it for analytics using Apache Spark and Databricks. - Automation & Optimization Automate data processes and optimize pipeline performance. - Data Quality & Validation Implement data quality checks to ensure accurate and reliable data. - Collaboration Work with business teams and IT to design and document data solutions. | Overall years of experience | 10+ years of experience in data engineering, with a focus on large-scale data pipelines. |