Sr. Software Engineer

JOB DESCRIPTION Senior Data Engineer Data & Analytics Engineering | About the Role We are looking for a Senior Data Engineer to join our growing Data & Analytics team. In this role, you will design, build, and maintain scalable data pipelines and platforms that power critical business decisions. You will be a key contributor in driving our cloud-first data strategy, leveraging Azure Databricks as a core technology and Azure Data Factory (ADF) as Good to have feature. You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment. Key Responsibilities Data Pipeline & Architecture - Design, develop, and maintain robust ETL/ELT pipelines using Azure Databricks and Azure Data Factory (ADF). - Architect and implement scalable data lakehouse solutions on Azure using Delta Lake. - Build and optimize data workflows across batch and streaming workloads (Spark Structured Streaming, Event Hubs). - Define and enforce data modeling best practices (star schema, data vault, medallion architecture). Databricks Engineering - Develop and optimize Spark-based data transformations using PySpark and Spark SQL in Databricks. - Manage Databricks clusters, jobs, and workspace configurations for performance and cost efficiency. - Implement Delta Live Tables (DLT) pipelines for declarative, auto-scaling data transformations. - Leverage Unity Catalog for data governance, lineage tracking, and access control. - Utilize Databricks Asset Bundles (DABs) and CI/CD practices for deployment automation. Azure Data Factory (ADF) – (Good To Have Feature) - Build, schedule, and monitor complex ADF pipelines with parameterized templates. - Integrate ADF with Azure Key Vault, Linked Services, and Integration Runtimes (SHIR/Azure IR). - Implement incremental load patterns, watermarking, and change data capture (CDC) strategies. - Troubleshoot pipeline failures and optimize ADF pipeline performance and cost. Data Quality & Governance - Implement data quality frameworks and validation checks across pipelines. - Enforce data cataloging, lineage, and metadata management practices. - Collaborate with data governance teams to ensure compliance with data policies and regulations (GDPR, HIPAA). Collaboration & Leadership - Mentor junior data engineers and conduct code reviews. - Work closely with data scientists and ML engineers to productionize machine learning models. - Partner with DevOps/Cloud teams on infrastructure-as-code (Terraform/Bicep) for data platform provisioning. - Document architecture decisions, pipeline designs, and operational runbooks. Required Qualifications - 7+ years of hands-on experience with Azure Databricks (Spark SQL, Delta Lake, PySpark). - Strong proficiency in SQL preferred and Python. - Deep understanding of distributed computing principles and the Apache Spark ecosystem. - Experience with Azure data services: ADLS Gen2, Azure Synapse, Azure SQL, Event Hubs / Kafka. - Solid understanding of data warehousing concepts and dimensional modeling. - Experience with version control (Git) and CI/CD tools (Azure DevOps, GitHub Actions). - Familiarity with infrastructure-as-code tools (Terraform or ARM/Bicep). Preferred Qualifications - Databricks Certified Data Engineer Associate or Professional certification. - Microsoft Certified: Azure Data Engineer Associate (DP-203). - Experience with Delta Live Tables (DLT) and Databricks Workflows. - Familiarity with Power BI or other BI/reporting tools. - Experience with Scala or Java for Spark development. Technology Stack Core Platforms | Azure Databricks (Preferred), Azure Data Factory (ADF) (Good to have) | Data Storage | Azure Data Lake Storage Gen2, Delta Lake, Azure SQL, Cosmos DB | Languages | Python (PySpark), SQL | Orchestration | ADF Triggers, Databricks Workflows, Apache Airflow | Monitoring | Azure Monitor, Log Analytics, Databricks Cluster Policies | Governance | Unity Catalog, Azure Purview, Azure Key Vault – (Good to have) | BI / Reporting | Power BI, Tableau | Key Competencies - Strong analytical and problem-solving skills with attention to detail. - Excellent communication skills — ability to translate complex technical concepts to non-technical stakeholders. - Self-motivated and able to manage competing priorities in an agile environment. - Collaborative team player with a growth mindset and eagerness to mentor others. - Proactive in identifying performance bottlenecks and proposing architectural improvements. *** Concord is an execution partner helping organizations drive digital transformation, modernization, and scalable technology solutions. We deliver results that solve real business challenges. We operate globally and are growing fast, shaping the future of technology. Join a team trusted by top companies to drive strategic growth and operational excellence! Powered by JazzHR