Job Description

Key Responsibilities

• 
  Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization.
• 
  Perform data extraction, cleaning, transformation, and data flow management. Web scraping may also be a part of the work scope in data extraction.
• 
  Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• 
  Integrate and collate data silos in a manner that is both scalable and compliant.
• 
  Collaborate with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers, and Data Analysts to build scalable, data-driven products.
• 
  Be responsible for developing backend APIs and working on databases to support applications.
• 
  Work in an Agile environment that practices Continuous Integration and Continuous Delivery.
• 
  Work closely with fellow developers through pair programming and code review processes.

Experience and Skills Needed

• 
  Proficient in general data cleaning and transformation (e.g., SQL, pandas, R, etc.) to ensure data accuracy and consistency.
• 
  Proficient in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Service (DMS), Python, AWS Lambda, ECS Container Tasks, EventBridge, AWS Glue, Spring).
• 
  Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostGIS, MySQL, SQLite, VoltDB, Cassandra, etc.).
• 
  Experience in cloud technologies such as GCP, GCC (i.e., AWS, Azure, Google Cloud).
• 
  Experience and passion for data engineering in a big data environment using cloud platforms such as GCP, GCC (i.e., AWS, Azure, Google Cloud).
• 
  Experience with building production-grade data pipelines and ETL/ELT data integration.
• 
  Knowledge of system design, data structures, and algorithms.
• 
  Familiar with data modelling, data access, and data storage infrastructure such as Data Marts, Data Lakes, Data Virtualization, and Data Warehouses for efficient storage and retrieval.
• 
  Familiar with REST APIs and web requests/protocols in general.
• 
  Familiar with big data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ).
• 
  Familiar with W3C Document Object Model and customised web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js, etc.).
• 
  Familiar with data governance policies, access control, and security best practices.
• 
  Comfortable with at least one scripting language (e.g., SQL, Python).
• 
  Comfortable working in both Windows and Linux development environments.
• 
  Interest in being the bridge between engineering and analytics.

Bonus Experience (Added Advantage)

• 
  Experience building data engineering pipelines that require integration with search indexes.
• 
  Experience with Airflow and RDBMS integration and implementation (e.g., MySQL).
• 
  Experience with either Snowflake, Databricks, or an equivalent provider