Job Description

Responsibilities

Team Introduction

The Data Infrastructure SRE team is responsible for the reliability, scalability, and efficiency of the compute platform that powers our products and data workloads. We operate large-scale Kubernetes environments and AI infrastructure across heterogeneous CPU and GPU resources, including cluster control planes, worker nodes, workload scheduling, container runtimes, and resource management systems. Our focus is to keep shared compute services available, predictable, and cost-efficient while enabling product and platform teams to run critical workloads safely at scale.

This role includes participation in a rotational on-call schedule to provide continuous coverage for critical compute infrastructure. You will respond to, troubleshoot, and resolve production incidents involving Kubernetes clusters, CPU and GPU nodes, containers, scheduling, and resource availability. Our team works across multiple time zones and follows rigorous change management and post-incident review practices to maintain platform stability.

As a Site Reliability Engineer focused on compute and AI Infra, you will operate and improve large-scale Kubernetes platforms, manage cluster and node lifecycle changes, diagnose workload and scheduling failures, and automate repetitive tasks. You will work with senior engineers and partner teams to improve the reliability, performance, and efficiency of the compute platform.

Responsibilities

• Kubernetes and AI Infra reliability: Own the reliability of large-scale Kubernetes and GPU clusters. Lead incident response, root-cause analysis, and corrective actions across control planes, nodes, container runtimes, scheduling, and host-level dependencies.
• Cluster and node lifecycle management: Operate CPU and GPU clusters through provisioning, upgrades, scaling, configuration changes, repair, replacement, and decommissioning. Build safe rollout, rollback, and validation mechanisms for cluster-level changes.
• Heterogeneous resource management and scheduling: Improve placement, utilization, isolation, and availability for online, batch, data, and AI/ML workloads across CPU and GPU resources. Optimize quotas, priorities, preemption, affinity, autoscaling, and accelerator-aware scheduling.
• Capacity, performance, and cost: Own compute capacity planning, resource forecasting, and performance tuning across clusters and failure domains. Balance workload demand, reliability, utilization, and infrastructure cost.
• SLOs, observability, and operational readiness: Define SLOs and error budgets for compute and AI Infra services. Improve monitoring, alerting, runbooks, production-readiness reviews, and failure recovery for clusters and workloads.
• Automation and technical leadership: Build maintainable tooling and automated remediation to reduce operational toil and improve deployment safety. Partner with workload owners and infrastructure teams on platform standards, design reviews, and operational practices.

Qualifications

Minimum Qualifications

• Bachelor’s degree in Computer Science, a related technical field, or equivalent practical experience.
• 3+ years of experience in Site Reliability Engineering, Production Engineering, Systems Engineering, or a similar infrastructure role.
• Strong hands-on experience operating Kubernetes in production, including troubleshooting clusters, nodes, pods, controllers, and workload scheduling.
• Experience managing compute resources across CPU and GPU nodes, including capacity, allocation, isolation, utilization, and failure recovery.
• Strong knowledge of Linux systems, container runtimes, cgroups, namespaces, networking fundamentals, and host-level performance troubleshooting.
• Proficiency in Go, Python, Bash, or another programming language used for automation and infrastructure tooling.

Preferred Qualifications

• Experience operating large, multi-cluster or multi-region Kubernetes fleets, including GPU clusters and AI Infrastructure at scale.
• Deep knowledge of Kubernetes architecture and internals, including the API server, etcd, scheduler, controller manager, kubelet, admission control, and custom controllers or operators.
• Experience with heterogeneous resource scheduling and management, including CPU/GPU requests and limits, quotas, priorities, preemption, affinity, taints and tolerations, autoscaling, bin packing, workload isolation, and accelerator-aware placement.
• Experience supporting distributed AI/ML training or inference workloads and diagnosing GPU availability, utilization, topology, device-plugin, or scheduling issues.
• Experience with container runtimes, infrastructure-as-code, configuration management, and Kubernetes deployment tooling such as Terraform, Ansible, Helm, or GitOps workflows.
• Experience designing AI Infra reliability mechanisms, including SLOs, capacity models, failure-domain planning, observability, automated remediation, and workload recovery.
• A systematic approach to complex troubleshooting, strong written and verbal communication, and clear ownership during high-impact incidents.

Job Information

About Us

Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Lemon8, CapCut and Pico as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.​

Why Join ByteDance

Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.​

As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.​

Diversity & Inclusion​

ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.​