Job Description

We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem.

Responsibilities

Design and maintain batch and real-time data pipelines.
Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services.
Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting.
Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR.
Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift.
Design analytical data models, including Star and Snowflake schemas.
Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik.
Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation.
Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control.
Monitor production data pipelines and proactively address operational and performance issues.

Mandatory Requirements

Proven experience as a Senior Data Engineer or in a similar role.
Advanced proficiency in Python and SQL.
Hands-on experience with:
- Apache Spark / PySpark
- Apache Kafka
- Apache Airflow
Strong AWS experience, particularly with:
- Amazon S3
- Amazon Redshift
- AWS EMR
Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling.
Strong understanding of batch and streaming architectures.
Experience with performance optimization, Git, testing practices, and production support.

Technical Stack

Python
SQL
Apache Spark / PySpark
Apache Kafka
Apache Airflow
AWS EMR
Amazon S3
Amazon Redshift
Git
Qlik

About Be | Shaping the Future

🇮🇹

Be | Shaping the Future, parte del gruppo Engineering, è tra i principali player in ambito Management Consulting e ICT Solutions. La Società fornisce servizi di Business Advisory, Information Technology e Digital Engagement.

Grazie alla combinazione di competenze specialistiche, tecnologie proprietarie avanzate e un patrimonio di esperienza consolidata, il Gruppo supporta primarie istituzioni finanziarie, assicurative e industriali nelle più importanti sfide della trasformazione digitale.

Con oltre 4.000 risorse e sedi in Italia, Regno Unito, Germania, Austria, Svizzera, Lussemburgo, Spagna, Romania, Repubblica Ceca, Polonia, Albania e Ucraina.

🇬🇧

Be | Shaping the Future, part of the Engineering Group, is one of the leading players in Management Consulting and ICT Solutions. The Company provides Business Advisory, Information Technology and Digital Engagement services.

Through a combination of specialized expertise, advanced proprietary technologies and a wealth of proven experience, the Group supports leading financial, insurance and industrial institutions in the most important challenges of digital transformation.

With over 4,000 employees and offices in Italy, the UK, Germany, Austria, Switzerland, Luxembourg, Spain, Romania, the Czech Republic, Poland, Albania, and Ukraine.

Industry

Consulting & Advisory

Company Size

1,001-5,000 employees

Headquarters

Milano, IT

Year Founded

2008

Website

be-tse.it

Social Media