Description:

• 
  Designing,
  developing, and optimizing Azure Data Factory and Databricks pipelines
  from Oracle databases to Lakehouses
• 
  Designing
  and optimizing data models using Medallion Architecture principles
• 
  Designing,
  developing and optimizing data connections from Databricks Medallion
  Architecture to on-premises data sources for downstream consumers.
• 
  Designing
  and optimizing relational data models
• 
  Translating
  Informatica ETL to Azure Data Factory and Databricks ELT

Knowledge Transfer:

• 
  Transfer
  From DataOps/Cloud Data Engineer to Designated CSC Resource

When Knowledge Will Be Transferred:

• 
  Knowledge
  transfer must be completed one week prior to the end of the project or one
  week prior to the consultant leaving the ministry.

What Knowledge Will Be Transferred:

• 
  All
  deliverables, including design/supporting/release/training documents must
  be checked into designated version control repositories (for example, git
  repositories or SharePoint). All final documents and working drafts
  related to project requirements or solution design must be stored on
  designated project repositories (for example, SharePoint site, HPQC, TFS,
  etc.)
• 
  Project
  manager and designated ministry staff must be regularly informed in
  writing (by email) of where documentation has been stored and must be
  provided a minimum of one walk-through of all documentation as part of the
  final knowledge transfer activities.

• 
  Knowledge
  will be transferred through 1 on 1 sessions, emails, document updates and
  document review with the team.

Requirements

Experience and Skill Set Requirements:

Must Haves:

• 
  Experience
  developing ETL processes with Informatica, Azure Data Factory and
  Databricks (Python, Lakeflow, and SQL Optimization)
• 
  Experience
  with solution development with Databricks in Medallion Architecture.
• 
  Experience
  with Cloud data engineer must be familiar and experienced with different
  programming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS.
  Microsoft Fabric) and be able to integrate with many different platforms
  to create data pipelines, jobs, automate tasks, and write scripts.
• 
  Experience
  with Data pipeline and workflow development, orchestration, deployment,
  and automation and specializing in programming and pipelines to create and
  managing the dataflow, Delta Lake indexing, parallelism and movement of
  data.

Skill Set Requirements:

Technical Experience:

• 
  Experience
  developing ETL processes with Informatica, Azure Data Factory and
  Databricks (Python, Lakeflow, and SQL Optimization)
• 
  Experience
  with solution development with Databricks in Medallion Architecture.
• 
  Experience
  with Cloud data platforms, data management and data exchange tools and
  technologies
• 
  Experience
  with Commercial and opensource data and database development and
  management and specializing in data storage setting (Delta Lake) and
  managing cloud Data As a Service (DaaS). application Database as a Service
  (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms
  (both in the cloud and on-premise).
• 
  Experience
  with Data pipeline and workflow development, orchestration, deployment,
  and automation and specializing in programming and pipelines to create and
  managing the dataflow, Delta Lake indexing, parallelism and movement of
  data.
• 
  Experience
  with Cloud data engineer must be familiar and experienced with different
  programming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS.
  Microsoft Fabric) and be able to integrate with many different platforms
  to create data pipelines, jobs, automate tasks, and write scripts.
• 
  Experience
  in Continuous Integration/Continuous Development/Deployment (CI/CD) and
  Data provisioning Automation
• 
  Experience
  with digital product, data analysis, data exchange, data integration APIs,
  data provisioning, and data security
• 
  Extensive
  experience in designing/developing and implementing data conversation and
  migration of VLD (Very large Data) of Online analytical processing (OLAP)
  and online transaction processing (OLTP) environments to Cloud
  Software-as-a-service (SaaS), Platform-as-a-service (PaaS) and
  Infrastructure-as-a-service (IaaS) environments.
• 
  Experience
  in design, development and implementation of fact/dimension model, data
  mapping, data warehouse, data lakehouse for enterprise.
• 
  Demonstrated
  experience developing star schema multi-dimensional models and documenting
  detailed design models.
• 
  Experience
  managing Cloud Data services for project delivery, including storage,
  repositories, Data Lakehouse, key vault, virtual machine, Azure Storage
  Account, SIHR, parquet files etc.
• 
  Experience
  with structured, semi-structured, unstructured data collection, ingestion,
  provisioning and exchange technological development of enterprise data
  warehouse and data lake and data lakehouse solutions and operational
  support.
• 
  Experience
  with DataOPS performance monitoring and tuning
• 
  Implementing
  and enforcing data quality checks and principles, including data
  validation, profiling, cleansing, and monitoring across pipelines
• 
  Applying
  data governance practices and information architecture standards,
  including development of conceptual, logical, and physical data models
• 
  Integrating
  and managing Microsoft Entra ID for secure authentication, authorization,
  and role-based access control across data platforms
• 
  Implementing
  data anonymization and masking techniques to protect sensitive and
  regulated data in compliance with privacy requirements
• 
  Developing
  and maintaining data lineage reports to provide end-to-end visibility and
  traceability of data movement and transformations
• 
  Databricks
  and/or Microsoft Fabric certifications are considered an asset

Core Skills:

• 
  Excellent
  analytical, problem-solving and decision-making skills; verbal and
  written communication skills; presentation skills; interpersonal and
  negotiation skills
• 
  Demonstrates
  development and solution design experience with technology, data,
  databases, statistics, applications, and networking techniques, tools and
  practices that enables the candidate to lead the design and development of
  business intelligence projects.
• 
  Experience
  in DataOPS best practices, and Agile project development and deployment
• 
  Demonstrates
  experience in working with and analyzing large and diverse data sets, data
  visualization, KPI development and data profiling to identify
  relationships and themes.
• 
  Demonstrates
  experience manipulating and analyzing complex, high-volume data from
  structured and unstructured sources.
• 
  Demonstrates
  experience creating detailed documentation for the tools that are
  developed, such as business requirements, system requirements and
  technical requirements.
• 
  Demonstrates
  experience with advanced data visualization tools and techniques,
  designing high quality interfaces to present information in a meaningful
  way.
• 
  Troubleshooting
  of production issues and creating fixes for any unplanned deployments.
• 
  Defect
  investigation, resolution and assignment to team members using
  defect/issue.
• 
  Demonstrated
  experience in coordinating and planning for the modernization of large,
  complex multi-platform, multitier information technology systems.
• 
  Experience
  with Azure DevOps.
• 
  Experience
  with AODA/WCAG compliance.
• 
  A
  team player with a track record for meeting deadlines

Project Experience and Techniques:

• 
  Demonstrates
  development and design experience with SDLC processes, Agile and Waterfall
  methodologies.
• 
  Demonstrates
  experience conducting system testing and unit testing, and conducting SIT,
  SAT and UAT Testing.
• 
  Demonstrates
  experience communicating regarding day-to-day tasks and issue tracking,
  reporting and facilitating resolution of issues and risks for all project
  activities.
• 
  Demonstrates
  experience gathering and consolidating input into changes required to
  business and/or system requirements, change requests, project artefacts,
  as well as leading requirement gathering sessions.

General Skills:

• 
  Demonstrates
  experience problem solving and decision making.
• 
  Demonstrates
  experience communicating clearly in both written and verbal formats.
• 
  Demonstrates
  experience working with and leading multiple teams representing various
  areas in delivering the project.
• 
  Demonstrates
  ability to work collaboratively, with the ability to coordinate multiple
  projects with competing priorities and a track record for meeting strict
  deadlines.
• 
  Demonstrates
  experience working collaboratively with other team members and/or groups
  and leading a team to ensure optimal solution integration.
• 
  Demonstrates
  experience leading a team and with providing technical advice and
  guidance.
• 
  Demonstrates
  experience working with both the business users and IT development teams
  to ensure business requirements are properly reflected in the system
  design and technical specifications.

Organization Experience:

• 
  Previous
  public sector experience in an organization of equivalent size.