Job Description

Your Role:

• You'll own the whole lifecycle of data pipelines running in 15+ regions, building them from start to finish.
• You'll write big, distributed processing jobs to handle massive datasets.
• You'll build and maintain pipelines that feed our product's access layer—think threat intelligence, assets, and detections.
• Quality is huge: you’ll build frameworks to check data health, run conditional checks, and keep everything observable
• You’ll implement distributed tracing across all your pipelines.

Team & Culture:

• We believe in owning your work end-to-end without annoying bottlenecks.
• We're AI-native, using custom plugins and protocols to make development smoother.
• We take engineering rigor seriously (think dead-code detection, constant security scans, and always-on production monitoring).
• Yes, we have on-call, but we’ve built tools to automate the pain out of it.

What You'll Need:

• 4+ years building production data pipelines at scale
• Solid Python 3.10+ chops (Pydantic, async, decorators).
• Apache Airflow - DAG authoring, MWAA or self-hosted production experience
• PySpark - job authoring, optimization, EMR or Databricks
• Snowflake expertise (Snowpark, schema design).
• AWS knowledge (S3, SQS/SNS, EMR).
• Data quality tooling: Great Expectations, Monte Carlo, or equivalent
• CI/CD with artifact packaging and staged deployments
• Strong testing discipline: pytest, moto, integration tests against real or mocked cloud services