Job Description

We are looking for the best

At 42dot, our AI Data Pipeline Engineer architect and scale global data pipelines that ingest and process data from worldwide sources. You will design and operate high-throughput systems to reliably deliver petabyte-scale data to our large-scale GPU infrastructure, powering mission-critical AI workloads.

Responsibilities

• Design and build high-performance, scalable data pipelines to support diverse AI and Machine Learning initiatives across the organization.

• Architect and implement multi-region data infrastructure to ensure global data availability and seamless synchronization.

• Develop flexible pipeline architectures that allow for complex branching and logic isolation to support multiple concurrent AI projects.

• Optimize large-scale data processing workloads using Databricks and Spark to maximize throughput and minimize processing costs.

• Maintain and evolve the containerized data environment on Kubernetes, ensuring robust and reliable execution of data workloads.

• Collaborate with AI researchers and platform teams to streamline the flow of high-quality data into training and evaluation pipelines.

Qualifications

• Extensive professional experience in building and operating production-grade data pipelines for massive-scale AI/ML datasets.

• Strong proficiency in distributed processing frameworks, particularly Apache Spark and the Databricks ecosystem.

• Deep hands-on experience with workflow orchestration tools like Apache Airflow for managing complex dependency graphs.

• Solid understanding of Kubernetes and containerization for deploying and scaling data processing components.

• Proficiency in distributed messaging systems such as Apache Kafka for high-throughput data ingestion and event-driven architectures.

• Expert-level programming skills in Python for system-level optimizations.

• Strong knowledge of cloud-native services and best practices for building secure and scalable data infrastructure.

• Logical approach to problem-solving with the persistence to identify and resolve root causes in complex, large-scale systems.

• Strong communication skills to effectively collaborate with cross-functional teams and external partners.

Preferred Qualifications

• Experience in architecting global, multi-region data pipelines and solving challenges related to cross-border data transfer and latency.

• Practical experience or a strong interest in implementing distributed computing frameworks like Ray for AI workloads.

• Experience in building real-time or near-real-time pipelines using Spark Streaming or Flink.

• Familiarity with Infrastructure as Code (IaC) tools such as Terraform to manage complex data environments.

• Understanding of the end-to-end ML lifecycle (MLOps) and how data infrastructure supports model experimentation and deployment.

Interview Process

• 서류ì „í˜• - 온라인 코딩테스트 - í™”ìƒë©´ì ‘ (1시간 내외) - ëŒ€ë©´ë©´ì ‘ (3시간 내외) - 최종합격

• ì „í˜•ì ˆì°¨ëŠ” 직무별로 다르게 ìš´ì˜ë ìˆ˜ 있으며, ì¼ì • 및 상황에 따라 ë³€ë™ë ìˆ˜ 있습니다.

• ì „í˜•ì¼ì • 및 결과는 지원서에 ë“±ë¡í•˜ì‹ ì´ë©”ì¼ë¡œ 개별 안내드립니다.

Additional Information

• ì´ë ¥ì„œ ì œì¶œ 시 주민등록번호, 가족관계, 혼인 여부, 연봉, 사진, ì‹ ì²´ì¡°ê±´, ì¶œì‹ ì§€ì—­ 등 ì±„ìš©ì ˆì°¨ë²•ìƒ 요구 금지된 ì •ë³´ëŠ” ì œì™¸ 부탁드립니다.

• ëª¨ë“ ì œì¶œ 파일은 30MB 이하의 PDF 양식으로 업로드를 부탁드립니다. (ì´ë ¥ì„œ 업로드 중 ë¬¸ì œê°€ 발생한다면 ì§€ì›í•˜ì‹œê³ ìž 하는 포지션의 URLê³¼ 함께 ì´ë ¥ì„œë¥¼ recruit@42dot.ai으로 ì „ì†¡ 부탁드립니다.)

• 인터뷰 프로세스 종료 후 지원자의 동의하에 평판조회가 ì§„í–‰ë ìˆ˜ 있습니다.

• 국가보훈대상자 및 취업보호 대상자는 ê´€ê³„ë²•ë ¹ì— 따라 우대합니다.

• ìž¥ì• ì¸ ê³ ìš© 촉진 및 직업재활법에 따라 ìž¥ì• ì¸ 등록증 소지자를 우대합니다.

• 42dot은 의뢰하지 않은 서치펌의 ì´ë ¥ì„œë¥¼ 받지 않으며, 요청하지 않은 ì´ë ¥ì„œì— 대해 수수료를 지불하지 않습니다.

※ 지원 ì „ 아래 내용을 ê¼­ 확인해 주세요.

• 42dot이 일하는 방식, 42dot Way 보러가기 →

• 42dot만의 업무몰입 프로그램, Employee Engagement Program 보러가기 →