Job Description
Job Summary
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Key Responsibilities
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Skill Requirements
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Other Requirements
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).