Data Engineer

Role & Responsibilities: - Design, develop, and maintain ELT/ETL pipelines on GCP using Dataflow/Beam, Dataproc/Spark, and Airflow/Composer. - Model and optimize datasets in BigQuery using partitioning, clustering, materialized views, and UDFs. - Build streaming and near-real-time data ingestion using Pub/Sub, Dataflow, and CDC where applicable. - Implement data quality checks, validation frameworks, and SLAs, monitoring pipelines via Cloud Monitoring/Logging. - Optimize performance and cost across GCS, Dataproc autoscaling, and BigQuery slot usage. - Contribute to coding, CI/CD standards, observability, documentation, and perform code reviews. - Partner with Analytics, BI, and ML teams to productize datasets and ensure strong data contracts. - Support production operations, including on-call rotations for critical pipelines. Preferred Candidate Profile: - Hands-on expertise in GCP data stack: BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow). - Strong experience with Spark (PySpark or Scala) for batch processing. - Solid Airflow DAG design knowledge (idempotent tasks, backfills, retries, SLAs). - Advanced SQL and data modeling (star/snowflake schemas, slowly changing dimensions, partition strategies). - Proficiency in Python (preferred) or Scala/Java. - Experience with Git and CI/CD tools (Cloud Build, GitHub Actions, GitLab CI). - Familiarity with GCP security and governance (IAM, service accounts, secrets management, VPC-SC). - Strong debugging, problem-solving, and communication skills. Good-to-Have: - Snowflake (migration, performance tuning, tasks/streams). - Terraform for infrastructure as code on GCP. - Kafka or other streaming tools; Cloud Run/Functions for glue services. - BI exposure (Looker, Tableau, Power BI). - GCP Professional Data Engineer certification. Skills: Spark, Gcp, Sql, Java, BigQuery, Python Experience: 4.00-5.00 Years