============ Contributing ============ This guide explains how to contribute to the project. .. contents:: :local: :depth: 1 ---- Joining the project team ======================== Communication ------------- Communication within the team is done on : * a Mattermost workspace * daily standup meetings The first thing to do when joining the project is to `get in touch `_ so we help you join these spaces. Project management ------------------ We use : * a `board of the tasks `_ (issues) * work on issues is always done on feature branches which are merged into ``master`` after review ---- Front end and back end development ================================== Install ------- To install front end dependencies, execute: .. code-block:: shell make install-front Run --- To run the frontend for development purposes, open a terminal and execute: .. code-block:: shell make run-front To run the backend for development purpose (with hot-reload), open another terminal and execute: .. code-block:: shell make run-back * During the first run, dependencies will be downloaded * You'll need to install `psql` to interact with the Postgres database The users for login are: - User with non super-user access: - username: "user" - password: "fish" - User with super-user access: - username: "superuser" - password: "fish" Then, insert the GIS layers to the postgres database by executing (make sure you have `psql` installed): .. code-block:: shell ./infra/remote/postgis_insert_layers.sh Export the required environment variables: .. code-block:: shell export DB_HOST=db export DB_NAME=monitorfishdb export DB_SCHEMA=public export DB_USER=postgres export DB_PASSWORD=postgres Finally add to Geoserver the layers by executing (make sure to remove your proxy if you have one configured with `unset HTTP_PROXY` and `unset http_proxy`): .. code-block:: shell ./infra/init/geoserver_init_layers.sh Test ---- To run all tests and checks clean architecture principles are respected, execute: .. code-block:: shell make test To manually add a VMS position (in NAF format) for the vessel `CABO ARTA`: .. code-block:: shell curl --data '//SR//AD/FRA//FR/NLD//RD/20210431//NA/CABO ARTA//RT/2133//FS/NLD//RC/PCVC//XR/FG78//IR/XXX2545115//DA/20210431//TI/2130//LT/55.099//LG/3.869//SP/0//CO/173//TM/POS//ER//' -X POST http://localhost:8880/api/v1/positions -H "Content-Type:application/text" Lint ---- To lint backend code, execute from the ``backend`` folder: .. code-block:: shell ./mvnw antrun:run@ktlint-format Documentation ------------- - API documentation can be found at http://localhost:8880/swagger-ui.html - Frontend documentation can be generated with .. code-block:: shell jsdoc frontend/src/domain/ frontend/src/types/ frontend/src/api/ -r -P frontend/ -d docs Problems & solutions -------------------- Windows & Docker """""""""""""""" * *During the build, there is an error while seting up docker which is used for the test database* * **Explanation** : Testing uses docker. To be able to connect to docker, Intellij requires that TLS is disabled in docker. * **Solution** : Open docker Configuration, General and click on "Expose Daemon on xxxx Witout TLS" * *Running the application with IntelliJ does not work. Spring complains about a BuildProperties beans that is missing* * **Explanation**: The buildProperties bean is constructed based on the META-INF/build-info.properties file. This file is generated by Maven, but not by IntelliJ * **Solution**: Tell IntelliJ to delegate the build to maven. To do this you must: 1. go to Build,Execution,Deployment> Build Tools > Maven > Runner 2. click the checkbox : "Delegate IDE Build/run actions to maven" ---- Data pipeline development and data science ========================================== Data processing and ETL operations are done in python using `prefect `_ and `pandas `_. Data and data sources --------------------- Data is not open and can only be accessed from inside the RIE network. To work with data, you will need to either access the private development server or work with data extracts. .. _data-install: Install ------- * The project runs on **python 3**. To manage python versions, we suggest you use `pyenv `_. See the official documentation for installation instructions. * After installing python 3 (the exact version currently used can be found in the .python-version file), install development dependencies : * If you do not have ``poetry``, install it to manage python dependencies: .. code-block:: shell curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3 - * Within the `pipeline` folder, install python dependencies : .. code-block:: shell poetry install * To install pre-commit hooks (isort, black and flake8 are configured in the pyproject.toml), run .. code-block:: shell poetry run pre-commit install Test ---- To run tests on the data pipeline, run .. code-block:: shell make test-pipeline ---- Documentation ============= Documentation is written as ``.rst`` source files, from which the documentation (this website) is built using `sphinx `__ and hosted on ReadTheDocs. .. _writing-documentation: Writing documentation --------------------- * To update the documentation, simply edit the ``.rst`` source file in ``pipeline/docs/source``. * To build the documentation locally and see the result of your updates : * you need to install the python development dependencies as described in :ref:`data-install` if you have not yet done so * build the html documentation with : .. code-block:: shell make build-docs-locally This builds the documentation in ``pipeline/docs/build/html``. You can view the built documentation in english and in french by opening ``en/index.html`` and ``fr/index.html`` in a web browser. Translations ------------ The documentation is translated with the `recommended process of ReadTheDocs `_ using `Transifex `_. Setting up Transifex """""""""""""""""""" In order to translate documentation using Transifex, you must set it up : * you need to have ``transifex-client`` installed, which is part of the :ref:`python development dependencies `. If you have not yet done so, install development dependencies * ask us for an invitation to join our organization on Transifex * in your Transifex account, create an API token in ``User Setttings`` > ``API token`` * in ``pipeline/docs``, run (with your API token instead of ``$TOKEN``) : .. code-block:: shell tx init --token $TOKEN --no-interactive Pushing documentation to Transifex for translation """""""""""""""""""""""""""""""""""""""""""""""""" After updating the ``.rst`` sources in english, if you wish to translate it : * push the documentation to Transifex with : .. code-block:: shell make push-docs-to-transifex This does several things : * cut the ``.rst`` sources into strings of uninterrupted text * create a mapping of these strings that Transifex uses to keep translation synchronized * push everything to Transifex * translate the corresponding strings in Transifex Pulling the translated documentation from Transifex """"""""""""""""""""""""""""""""""""""""""""""""""" When you are done translating in Transifex, you can pull the translated material back from Transifex with : .. code-block:: shell make pull-translated-docs-from-transifex Translated material comes in the form of ``.po`` and ``.mo`` files in the ``locale`` directory, from which sphinx will look for translations during the build. You can then :ref:`build the documentation locally ` to check the result. Updating the documentation online --------------------------------- The documentation is built by ReadTheDocs and hosted on ReadTheDocs. One the ``.rst`` source files and the translations are updated and pulled from Transifex, simply pushed the changes to ``master`` (better, create a branch and PR) and ReadTheDocs will build and update the documentation online automatically.