Skip to content

ETL

ETL Script for Generating and Loading Data into a Database.

This script generates data for employees, customers, products, orders, and sales, saves the data to CSV files, and loads the CSV data into a database.

Modules:

Name Description
- etl.Database.models

Database models for the project.
- etl.Database.database

Database engine and base class.
- etl.Database.data_generator

Functions to generate data for various entities.
- pandas

For data manipulation and storage in CSV.
- loguru

For logging.
- random

For random number generation.
- glob, os

For file path and system operations.

load_csv_to_table(table_name, csv_path)

Load data from a CSV file into a database table.

Parameters:

  • table_name (str): The name of the database table.
  • csv_path (str): The path to the CSV file containing data.

Returns: - None

Source code in myapp/etl/etl.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def load_csv_to_table(table_name: str, csv_path: str) -> None:
    """
    Load data from a CSV file into a database table.

    **Parameters:**

    - `table_name (str):` The name of the database table.
    - `csv_path (str):` The path to the CSV file containing data.

    **Returns:**
        - `None`
    """
    df = pd.read_csv(csv_path)
    df.to_sql(table_name, con=engine, if_exists="append", index=False)
    logger.info(f"Loading data into table: {table_name}")