Data Product Developer

Job Responsibility:

Data Product Developer

THE OPPORTUNITY

Total Systems Solutions Consulting Inc. is seeking a Data Product Developer to work for our public sector client at The Ministry of Finance in the Province of British Columbia. This is a remote full-time role with availability Monday to Friday from 8:00am to 5:00pm in the Pacific Time Zone.

Consultant candidates are expected to start ASAP and the contract length is 2 years and may be required to attend in-person meetings in Victoria, BC.

Project Context

The Ministry develops data products for use by Ministry staff in the form of datasets, reports and analytics. Currently, this is accomplished with a variety of manual and semi-automated processes using a wide variety of technologies.

In 2022, the Ministry began the development of an ‘Finance Data Strategy' to understand its current state, leverage learnings from other jurisdictions to set a direction, and articulate its vision for effectively managing data moving forward (the “Strategy”). The Strategy included recommendations and a high-level, multi-year, strategic reference roadmap. In 2023, the Ministry received endorsement of a four year, detailed implementation plan for the Strategy.

The Strategy and the implementation plan are part of a comprehensive four-year initiative to modernize the Ministry information technology, which started in 2022/2023. Several projects detailed in the implementation plan have been started, and the Ministry is preparing to undertake another phase of the implementation plan by beginning component development detailed in the implementation plan in 2024- 2025.

Specifically, this phase will consist of several overlapping components each with their own phases and expected deliverables, each either building upon an existing Ministry asset or creating a new ministry asset to fulfil on the Ministry's overall data strategy. Each component is envisaged and structured as a business-driven design process, followed by an iterative and end-to-end development process that delivers value in the form of a Ministry asset early in the project lifespan and then iteratively throughout the life of the component.

The components fall into two main groups. The first group of components is the development of the Ministry data operations platform. The data operations platform is a collection of components forming a set of services that will be available to Ministry staff to enable them to create high quality data products primarily for internal use, in the form of datasets, reports and analytics in a structured, maintainable and supportable way. The second group of projects is to populate the data operations platform with data by creating data pipelines and, in some cases, by manually gathering metadata from unstructured sources, to assist the Ministry data teams to make their data products more discoverable, accessible and usable.

Scope of Project

Anticipated Components

Certain anticipated components are as follows:

Finance Data Store Service Ministry data analysts and data scientists require access to a variety of database management technologies suited to their specific and evolving business and analysis needs. The Finance Data Store, currently in active development, is an effort to provide a variety of highly available and supported database systems that provide data for analysis used by ministry staff performing data analysis tasks and creating data products.

Asset: Finance Data Store Objective: Establish Finance Data Store as a production ready service

Finance Data Catalogue Metadata Collection The Finance Data Catalogue is a single place within the Ministry of Finance where staff can locate and discover data products, reports and analytics for use in analysis and informed decision making. The catalogue will go into production in May 2024. As a new repository of metadata, it will be populated with metadata describing the data assets of the Ministry. This Finance Data Catalogue Metadata Collection project goal is to collect metadata from a variety of sources including the source databases and supporting documents and placing that metadata into the metadata catalogue for use by ministry staff.

Asset: Finance Data Catalogue

Objective: Populate Finance Data Catalogue with metadata

Access Management System The Access Management System provides a web-based interface to a roles-based access control administrative database that will leverage IDIR as well as provide finer grained access control mechanisms to allow the ministry to manage access to data resources in an efficient and agile manner.

Asset: Access Management System

Objective: Establish Finance Data Access Management System as a production ready service

DataOps Platform A collection of tools and processes to establish, refine and continuously improve data products in a reliable and automated way making it more usable and valuable at scale and at increasing velocity. The tools include data collection, orchestration, pipelines, code management, data quality monitoring, and testing frameworks and tools.

Asset: Finance DataOps Platform Objective: Establish DataOps Platform as a production ready service

Component: Landing Zones A secure and managed service for external data providers for sending data to the Ministry of Finance using a flexible set of data transfer protocols. The service will enable ministry staff to easily create, monitor and grant access to secure landing zones as needed for receiving data from external partners.

Asset: Landing Zones Service

Objective: Establish Landing Zone Service as a production ready service

Research Library Develop a process and toolset for supplying analytical sandboxes to data scientists for trying out new ideas in a safe environment without having to make significant changes to their working environment and without affecting other systems and without risking of exposure of sensitive information. Consisting of tools including scripts and templates for establishing development environments for a variety of project contexts along with how-to documentation, the sandboxes will provide an easy way for new data scientist staff members to quickly become productive.

Asset: Analytical Sandbox Toolset

Objective: Establish tools and documentation for data scientists

Enterprise Data Analysis Capabilities Develop and establish tools and an approach to Power BI data access and data product publishing that can be utilized by Power BI users across MFIN providing consistent and convenient access to source data and guidance for users wanting to share or publish their resulting data products. Asset: Enterprise Data Analysis Capabilities

Objective: Establish Power BI Service structures and guidelines for accessing Finance Data Store data and making Power BI related data products discoverable in the Finance Data Catalogue

Intra-Ministry Data Management Capabilities Provide a platform for the development of ad-hoc no code low code solutions using Power Apps to be used by qualified MFIN staff to develop small applications on a selfserve basis in a supported and secure environment. Asset: Intra-Ministry Data Management Capability

Objective: Establish structures, underlying infrastructure, processes and guidelines for the development of Power Apps solutions.

Data Register Capability The use of Data Registers for data from authoritative sources can ensure data consistency, integrity, and accuracy for important MFIN data assets. This approach requires robust data governance policies and supporting technology to manage access, integrity and security effectively. This component provides additional resources to assist MFIN with the capacity to explore business value opportunities, develop a conceptual design and prototype to demonstrate how this can be done.

Asset: MFIN Data Registry Design

Objective: Develop a registry design and prototype

Consultant Qualifications

The Platform Developer is a software developer who specializes in developing and configuring individual system components, so they work together as part of a larger platform. They take direction and guidance from the Platform Architect in supporting the development and maintenance of systems.

Following are the desired qualificaitons:

3+ years in software development, integration and/or solution architecture.

• Post-secondary education in Computer Science or related field or equivalent combination of education and experience.

The Data Product Developer is a software developer who specializes in creating new valuable data products or improve existing ones by developing data pipelines to transform, combine and publish data to create data products for use by data scientists and other staff members They take direction and guidance from the Data Product Manager in supporting the development and maintenance of data pipelines and data products.

They have expert hands on knowledge of software development, data pipeline development, data quality assurance, testing, debugging and operations and can troubleshoot problems as they arise.

They are expected to have hands on experience and be able to provide skills and expertise in the areas of:

• Linux and Windows operating systems
• Python programming
• Apache Airflow or similar technology
• data modelling
• software development
• CI/CD They are expected to be familiar with the technologies or equivalent technologies as described in the scope section of this RFP as well as demonstrated proficiency and experience in the following areas:
• developing supporting configuration scripts in Python, Bash and Windows PowerShell
• Microsoft Power BI
• working with business areas and Data Product Managers to gather and analyse requirements
• understanding of data management issues including privacy and security
• designing and running data tests to ensure quality and accuracy expectations are met and to detect problems
• deploying and configuring monitoring systems to ensure the ongoing performance requirements are met
• documenting and maintaining appropriate system documentation

The Data Product Developer must be able to communicate effectively, to a broad range of stakeholders, including executive and senior management using oral and written communication skills

Job Types: Full-time, Fixed term contract

Pay: Up to $80.00 per hour

Flexible Language Requirement:

• French not required

Schedule:

• Day shift
• Monday to Friday

Application question(s):

• Describe where you have worked as the Developer on a large migration project where Power BI and OpenShift were involved?

Education:

• Bachelor's Degree (preferred)

Experience:

• Power BI: 3 years (required)
• OpenShift and related tools such as Helm and Docker: 3 years (required)
• Python, Bash and Windows PowerShell: 3 years (required)
• Software Development, integration and/or solution architect: 3 years (required)

Work Location: In person