Data Engineer (Spark/Scala)

We are looking for an experienced Data Engineer (Spark/Scala) with strong hands-on expertise in Apache Spark, Databricks, Scala, PySpark, Python, and SQL . The role involves designing and developing large-scale data pipelines across on-premises and cloud environments , working with multiple file systems and data formats, modernizing legacy workflows, and supporting hybrid data architectures. Key Responsibilities - Design, develop, and maintain scalable data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark . - Build and support complex on-premises data workflows and hybrid on-prem-to-cloud data integration solutions. - Integrate data across HDFS, NAS, on-prem file shares, Amazon S3 , and other storage platforms. - Work with multiple data formats including JSON, Parquet, CSV, Avro, Fixed-Length, and Excel . - Develop optimized SQL queries for data extraction, transformation, and loading. - Connect to multiple relational and non-relational databases and implement performance-efficient data extraction strategies. - Develop and maintain workflow orchestration using Apache Airflow or similar scheduling tools . - Write clean, production-grade Python code for data processing, automation, and engineering utilities. - Develop unit, integration, and data-quality tests for data pipelines. - Troubleshoot pipeline failures, performance bottlenecks, data quality issues, and complex multi-system integration problems. - Support migration and modernization of legacy on-premises data processes to hybrid/cloud environments. - Collaborate with Data Scientists, Analysts, Application Engineers, and other stakeholders. - Create technical documentation covering pipelines, data flows, architecture, and data lineage. - Support cloud integration initiatives, particularly across Azure environments. - Leverage coding assistants and AI agents to improve development productivity and automate engineering tasks. Required Skills - Strong hands-on experience with Apache Spark and Databricks . - Strong experience with Scala/Spark Scala and PySpark . - Strong Python programming skills. - Strong SQL , including complex joins, query optimization, and performance tuning. - Hands-on experience with Amazon S3 . - Experience working with HDFS, NAS, on-prem file systems , and cloud storage. - Strong experience handling JSON, Parquet, CSV, Avro, Fixed-Length, and Excel data formats. - Experience extracting data efficiently from multiple databases. - Strong understanding of complex on-premises data workflows and multi-system integrations. - Experience building hybrid on-prem/cloud data pipelines . - Strong troubleshooting and production support skills. Secondary Skills - Azure cloud services. - Apache Airflow or similar workflow orchestration tools. - Automated unit and integration testing. - Data quality validation and monitoring. - Data lineage and technical documentation. Good to Have - Working knowledge of Java . - Experience with Prefect . - Familiarity with React for internal tools or dashboards. - Experience using AI coding assistants and AI agents . - PBM / Pharmacy Benefit Management / Healthcare domain experience. Preferred Candidate Profile Candidates with strong experience in Scala + Spark + Databricks + PySpark , combined with on-premises data engineering and hybrid cloud integration , will be preferred. Key Skills: Apache Spark, Scala, Spark Scala, Databricks, PySpark, Python, SQL, Amazon S3, HDFS, Airflow, Azure, On-Premise Data Engineering, ETL, Data Pipelines.