Job Description

About the Role We are looking for a hands-on Data Engineer (7+years) to build and optimize scalable data pipelines and analytical datasets on the Databricks platform. You will work closely with Analytics/BI, Product, and Business teams to enable data-driven decision-making. Retail / eCommerce domain exposure is a strong plus, along with the ability to translate business needs into reliable and performant data solutions.

Key Responsibilities:

• Design, develop, and maintain robust ETL/ELT pipelines using Databricks (Spark) and Python (PySpark). • Develop and optimize complex transformations using SQL (joins, window functions, CTEs, query tuning). • Build curated datasets and data models to support reporting, dashboards, and advanced analytics use cases. • Implement pipeline reliability best practices: data quality checks, monitoring, alerting, and reconciliation. • Optimize Databricks workloads for performance and cost (cluster sizing, partitioning strategies, caching, file formats). • Work with structured and semi-structured data (JSON, CSV, Parquet/Delta) and handle schema evolution. • Collaborate with stakeholders to understand business KPIs and deliver data solutions aligned to retail/eCommerce metrics (sales, orders, returns, inventory, customer cohorts). • Follow engineering best practices for version control (Git), documentation, reusable code patterns, and testing. • Good to have: Support or migrate Alteryx workflows into Python/Databricks pipelines.

Must-Have Skills & Qualifications :

• 7-13 years of experience in Data Engineering / Data Warehousing / Big Data. • Strong hands-on experience with Databricks (Jobs/Workflows, notebooks, cluster concepts, Spark tuning fundamentals). • Strong programming skills in Python (PySpark preferred). • Excellent SQL skills, including performance tuning and writing complex analytical queries. • Experience building scalable pipelines and working with large datasets in distributed environments. • Strong understanding of data engineering concepts: ETL/ELT, orchestration, data validation, and observability. • Familiarity with modern data storage formats and practices (Delta/Parquet, partitioning, incremental loads).

Good-to-Have Skills:

• Retail / eCommerce domain knowledge (customer behavior, funnel metrics, pricing/promotions, inventory, catalog, order lifecycle). • Alteryx (workflow development, optimization, scheduling, or migration to Databricks). • Experience with Lakehouse patterns and Delta Lake features (e.g., MERGE, OPTIMIZE, Z ORDER). • Experience with orchestration tools (e.g., Airflow, ADF, Databricks Workflows). • Cloud experience: AWS / Azure / GCP (S3/ADLS/GCS, IAM basics, security controls). • CI/CD exposure for data pipelines, code reviews, and automated deployments. Preferred Traits • Strong problem-solving skills and a mindset for root-cause analysis. • Ownership and accountability for production-grade pipelines. • Ability to communicate with both technical and non-technical stakeholders. • Comfort working in fast-paced environments with evolving requirements.