Job Description

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Spark QA Engineer in India.

This role is focused on ensuring the reliability, scalability, and performance of large-scale data processing systems built on Apache Spark. You will work on validating complex Spark-based ETL pipelines, SQL workloads, and distributed applications that power critical data platforms. The position blends functional testing, automation, and performance engineering in highly distributed environments. You will collaborate closely with engineering teams to identify bottlenecks, optimize workloads, and ensure system stability across cloud and on-prem Spark clusters. The role involves hands-on work with performance benchmarking and deep analysis of distributed computing behavior. It is a highly technical position where your testing expertise directly influences the quality and efficiency of enterprise-scale data systems.

Accountabilities:

• Perform functional, integration, and automated testing of Apache Spark jobs, Spark SQL queries, and end-to-end ETL data pipelines across distributed systems.
• Execute performance, scalability, and benchmarking tests to evaluate Spark workloads under varying data volumes and cluster configurations.
• Set up, configure, and validate Spark environments across platforms such as YARN, Kubernetes, Databricks, EMR, Dataproc, Mesos, and standalone clusters.
• Identify performance bottlenecks, troubleshoot distributed system issues, and work closely with engineering teams to drive optimizations.
• Develop and maintain QA strategies, automation frameworks, and CI/CD-integrated testing processes for Spark-based applications.
• Lead QA initiatives, provide technical guidance, and mentor team members to improve testing quality and engineering practices.

Requirements:

• 5+ years of QA or software testing experience with strong hands-on expertise in Apache Spark and distributed data systems.
• Strong understanding of Spark architecture, optimization techniques, and distributed computing principles.
• Experience in functional, integration, and performance testing of large-scale data processing systems.
• Proficiency in Python or Java for test automation and framework development.
• Experience working with Kubernetes and cloud-based Spark environments (e.g., EMR, Dataproc, Databricks).
• Familiarity with CI/CD pipelines and automation frameworks for testing and deployment validation.
• Strong analytical and debugging skills with the ability to diagnose complex system-level issues.

Benefits:

• Competitive compensation aligned with experience and expertise
• Remote-first work environment with flexibility and autonomy
• Opportunity to work on large-scale distributed data systems and cutting-edge Spark-based architectures
• Exposure to cloud-native technologies and modern data engineering stacks
• Learning and upskilling opportunities in big data, performance engineering, and automation
• Collaborative engineering culture with mentorship and technical growth opportunities
• Health and wellness benefits as per company policy
• Participation in impactful projects driving enterprise-scale data processing

How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best! Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1