Senior Engineer - IT

Chennai Full time R000109479 **Job Description:** **Key Responsibilities** + Design, develop, and maintain scalable data pipelines using **Databricks and PySpark** + Build robust **ETL/ELT workflows** for batch and near real-time data processing + Write optimized and efficient **SQL queries** for data transformation, validation, and analysis + Work with large-scale distributed datasets using **Spark-based processing frameworks** + Perform data ingestion from multiple sources (APIs, files, databases) into data lake/lakehouse architecture + Ensure **data quality, integrity, and consistency** through validations and monitoring + Optimize pipeline performance and cost (partitioning, caching, query tuning, etc.) + Collaborate with cross-functional teams (Data Ops, Business, Analytics) to deliver data solutions + Troubleshoot pipeline failures and implement proactive monitoring and alerting + Maintain proper **documentation** for data models, transformations, and workflows **Required Skills & Experience** + Strong hands-on experience in: + **Databricks (Lakehouse platform, notebooks, jobs, clusters)** + **PySpark / Apache Spark** + **Advanced SQL (joins, window functions, performance tuning)** + Solid understanding of **ETL concepts, data warehousing, and data modeling** + Experience building **scalable data pipelines** for structured and semi-structured data + Familiarity with **data lake / delta lake architecture** + Good understanding of **performance optimization techniques** in Spark and SQL + Experience with **version control (Git)** and deployment practices + Strong problem-solving and analytical skills **Good to Have** + Experience with **cloud platforms (AWS preferably)** + Exposure to **CI/CD pipelines (Jenkins)** + Knowledge of **Delta Lake, Unity Catalog, or similar governance tools** + Understanding of **data quality frameworks / monitoring tools** + Experience working in **Agile / Scrum teams** **Location:** This position can be based in any of the following locations: Chennai **Current Guardian Colleagues: Please apply through the internal Jobs Hub in Workday** Every day, Guardian helps our 29 million customers realize their dreams through a range of insurance and financial products and services. Our Purpose, to inspire well-being, guides our dedication to the colleagues, consumers, and communities we serve. We know that people count, and we go above and beyond to prepare them for the life they want to live, focusing on their overall well-being - mind, body, and wallet. As one of the largest mutual insurance companies, we put our customers first. Behind every bright future is a GuardianTM. Learn more about Guardian at guardianlife.com . Visa Sponsorship: Guardian Life is not currently or in the foreseeable future sponsoring employment-based visas (e.g., such as an H-1B). In order to be a successful applicant, you must be legally authorized to work in the United States, without the need for employer sponsorship/support now or at any time in the future.