pipeline.src.utils
==================

.. py:module:: pipeline.src.utils


Functions
---------

.. autoapisummary::

   pipeline.src.utils.get_table
   pipeline.src.utils.delete
   pipeline.src.utils.delete_rows
   pipeline.src.utils.psql_insert_copy
   pipeline.src.utils.move
   pipeline.src.utils.remove_file


Module Contents
---------------

.. py:function:: get_table(table_name: str, schema: str, conn: sqlalchemy.engine.Connectable, logger: logging.Logger) -> sqlalchemy.Table

   Performs reflection to get a sqlalchemy Table object with metadata reflecting
   the table found in the databse. Returns resulting Table object.

   If the table is not found in the database, raises an error.


.. py:function:: delete(tables: List[sqlalchemy.Table], connection: sqlalchemy.engine.base.Connection, logger: logging.Logger, truncate: bool = False)

   Delete tables.
   Useful to wipe tables before re-inserting fresh data in ETL jobs.


.. py:function:: delete_rows(table: sqlalchemy.Table, id_column: str, ids_to_delete: Sequence, connection: sqlalchemy.engine.base.Connection, logger: logging.Logger)

   Deletes all rows of a table whose id is in ``ids_to_delete``.

   :param table: table to remove rows from
   :type table: sqlalchemy.Table
   :param id_column: name of the column in the table that contains ids to delete
   :type id_column: str
   :param ids: list-like sequence of ids to look for in the table and delete
   :type ids: Sequence
   :param connection: database connection
   :type connection: sqlalchemy.engine.base.Connection
   :param logger: logger
   :type logger: logging.Logger


.. py:function:: psql_insert_copy(table, conn, keys, data_iter)

   Execute SQL statement inserting data

   :param table:
   :type table: pandas.io.sql.SQLTable
   :param conn:
   :type conn: sqlalchemy.engine.Engine or sqlalchemy.engine.Connection
   :param keys: Column names
   :type keys: list of str
   :param data_iter:
   :type data_iter: Iterable that iterates the values to be inserted


.. py:function:: move(src_fp: pathlib.Path, dest_dirpath: pathlib.Path, if_exists: str = 'raise') -> None

   Moves a file to another directory. If the destination directory
   does not exist, it is created, as well as all intermediate directories.


.. py:function:: remove_file(fp: Union[str, pathlib.Path], ignore_errors: bool = True)