Flows

What are flows?

Flows are batch jobs that move and transform data. Collectively, they constitute the Data pipeline part of the Monitorfish architecture.

Each batch job is written as a Prefect flow which typically extracts data (from external sources and / or from tables in the Monitorfish database), processes the data and loads it back into a table of the Monitorfish database.

Flows are composed of tasks typically written as python pure functions. The UI (restricted access) enables administrators to view each flow as a diagram of its constituent tasks, to monitor their execution, see the logs and debug in case any flow run fails…

Overview of flows in Monitorfish

Flow that imports administrative areas :

Schematic of administrative areas data flow

Flows that import repositories (species, gears, infractions…) :

Schematic of repositories flows

Flows that import and perform a check-up on regulations data :

Schematic of regulation flows

Flows that perform computations that are internal to Monitorfish (alerts, fishing detection…):

Schematic of internal computations flows

Flows that contribute to maintaining an up-to-date last_positions table :

Schematic of last positions flows

List of flows