Experience Range: With at least 5-6 years of experience in data engineering, including substantial hands-on expertise in Databricks and PySpark
Key Responsibilities:
1. Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
2. Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
3. Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
4. Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
5. Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
6. Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
7. Monitor and tune performance of data processing jobs for optimal throughput and reliability
8. Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage
Required Skills:
1. Advanced proficiency in Databricks platform
2. Expertise in PySpark for distributed data processing and transformation
3. Strong SQL skills for analytics and data manipulation
4. Experience with data warehousing architectures
5. Hands-on experience designing and maintaining stored procedures
6. Proficiency in Apache Spark for large-scale data processing
Preferred Skills:
1. Experience with Azure Databricks or AWS Glue
2. Familiarity with CI/CD pipelines for Databricks workflows
3. Knowledge of Databricks performance optimization techniques
4. Exposure to data governance and security best practices in cloud environments
5. Experience with real-time data streaming frameworks such as Apache Kafka
Desired Qualifications:
1. Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
2. Certification in Databricks or Apache Spark
3. Certification in SQL or data warehousing technologies
Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption. Founded in 2014 as a digitally native full-service digital transformation services and consulting firm, we apply our expertise in customer experience transformation, data analytics, Artificial Intelligence (AI), platform and product engineering, cloud infrastructure, and security to help clients quickly innovate for growth, create digital products, build service platforms, and drive smarter, data-driven performance.
Headquartered in Dallas, Texas, we are powered by a diverse global team of world-class professionals across the U.S., the UK, Romania, Canada, Mexico and India, and are certified a Great Place to Work®. We help clients harness the transformative potential of the four superpowers of technology: cloud computing, Internet of Things (IoT), AI, and mobility. We bring deep expertise across the full spectrum of digital capabilities:
• Accelerating customer experience transformation to drive growth, customer advocacy, and superior customer experience
• Powering intelligent enterprises by harnessing the potential of data, analytics, and AI
• Crafting products of relevance with a product mindset and high-performance engineering
• Enabling enterprise agility with resilient cloud infrastructure and security
To learn more, please visit https://www.brillio.com/ and follow us here or @brillioglobal.
