Job Description

Senior Data Platform Engineer

As a Senior Data Platform Engineer, you will drive the architectural execution of the Data platform. You will take ownership of the system's scalability and consistency, ensuring the Operational Data Store functions as a reliable, highly available operational source for all global Master Data. You will act as a technical leader, mentoring the engineering team and collaborating with cross-functional integration teams to ensure seamless data flow.

Expanded Responsibilities

• Platform Architecture & Management: Architect and maintain mission-critical data lake and warehouse environments on Google Cloud Platform (GCP), utilizing services such as BigQuery, Pub/Sub, and Cloud Storage.
• Infrastructure & Deployment: Design, deploy, and manage robust, scalable infrastructure using Terraform to ensure environmental consistency across all deployment stages.
• Data Pipeline Engineering: Engineer high-performance, distributed data pipelines for real-time and batch streaming using technologies like Dataflow and Kafka Lead workflow orchestration by designing, deploying, and monitoring complex Directed Acyclic Graphs (DAGs) in Apache Airflow (Cloud Composer).
• Software Development: Develop custom backend tooling, internal APIs, and microservices in Golang and Python to enhance platform capabilities.
• Data Governance & Modelling: Manage the end-to-end data warehouse lifecycle, employing dbt (Data Build Tool) for clean, documented, and tested data models.
• DevOps & Reliability: Drive DevOps and Site Reliability Engineering (SRE) practices, owning CI/CD pipelines and utilizing Docker for seamless, highly reliable deployments. Optimize and architect diverse database engines (SQL and NoSQL), focusing on performance tuning, schema design, and security.
• Database Management: Architect and optimize various database engines (SQL and NoSQL), focusing on performance tuning, schema design, and security.

Technical Profile Additions

• Programming - Advanced proficiency in Golang (for systems) and Python (for data/automation).
• Infrastructure as Code (IaC): Terraform
• CI/CD & Version Control: Jenkins, Gitlab
• Programming Languages: Python (with expertise in FastAPI and Prefect; well-versed in functional programming and a strong grasp of object-oriented programming)
• GCP Expertise:
  • Cloud Run (Services, Jobs, Functions)
  • Databases: BigQuery, Cloud SQL for PostgreSQL & , Cloud Spanner
  • Cloud Storage
  • Managed Airflow (Cloud Composer)
  • Pub/Sub, Eventarc
  • IAMs: Ability to construct necessary permissions following the principle of least privilege; capability to analyze permission requests for potential security consequences.
  • Artifact Registry
  • GCP Monitoring and Alerting
  • Secret Manager
  • Vertex AI and Workbench Instances: Setup and permission management.
  • GCP Networking
• Containerization: Docker
• Architecture: Advanced skills in architecting data pipelines on Cloud (GCP preferred)
• Data Architecture Theory: Advanced understanding of Data Mesh Architecture and its comparison to the centralized approach.

INTERMEDIATE LEVEL IS ACCEPTABLE:

• Transformations: dbt
• Frontend Technologies: React.js

NICE TO HAVE:

• Secrets Management: Akeyless
• Project Management & Documentation: Jira, Confluence, ServiceNow
• Data Ingestion & Reverse ETL: Fivetran (including Fivetran Activations)

Educational Background & Qualifications

Required Education:

• Master’s or Bachelor’s Degree in Computer Science, Data Engineering, Software Engineering, or a related technical field (e.g., Electromechanical Engineering with a focus on Automation)
• Specialized Coursework: Strong academic foundation in Distributed Systems, Database Management Systems (DBMS), Algorithm Design, and Real-time Computing.

Preferred Certifications:

• Google Cloud Professional Data Engineer or Google Cloud Professional Cloud Architect
• APICS or similar certifications in Logistics/Supply Chain (a significant plus for understanding the "why" behind the warehouse data).