Job Description

We are seeking an experienced Data Engineer/Architect with strong expertise in Cloudera NiFi, data pipeline optimization, and ETL/ELT processes. This role involves assessing, enhancing, and supporting scalable data integration solutions.

Key Responsibilities

NiFi Platform Assessment & Optimization

• Evaluate existing Cloudera NiFi flows for performance, scalability, and maintainability

• Identify improvements to simplify architecture and reduce operational overhead

• Recommend best practices, reusable patterns, and standards

Data Integration & Pipeline Support

• Develop and enhance ETL/ELT pipelines using NiFi

• Integrate data from databases, APIs, filesystems, streaming platforms, and cloud storage

• Troubleshoot pipeline failures, latency, and performance issues

• Implement monitoring, error handling, and recovery mechanisms

Python & Custom Development

• Use Python for data transformation, automation, and scripting

• Build integrations between NiFi and external systems

Governance, Security & Operations

• Ensure secure data movement with authentication, authorization, and encryption

• Support deployment, versioning, and operational processes

• Align platform changes with compliance and operational standards

Advisory & Collaboration

• Act as a technical advisor to data and platform teams

• Document findings, recommendations, and improvements

• Provide knowledge transfer and long-term support guidance

Requirements

Required Skills

• Hands-on experience with Cloudera NiFi in production

• Strong understanding of ETL/ELT and data pipeline architecture

• Proficiency in Python for scripting and automation

• Experience with Linux and distributed systems

• Ability to analyze and optimize existing data systems

Preferred Skills

• Experience with Cloudera ecosystem (Kafka, HDFS, Hive, Iceberg)

• Familiarity with CI/CD for data pipelines

• Experience in monitoring and performance tuning

• Exposure to cloud or hybrid data environments

Engagement Expectations

• Hands-on, deliverables-focused role

• Ability to work independently and collaborate with teams

• Strong communication and documentation skills