Data Warehouse Project

Welcome to the Data Warehouse repository! 🚀
This project demonstrates a comprehensive data warehousing solution. Designed as a portfolio project, it highlights industry best practices in data engineering.

---

🏗️ Data Architecture

The data architecture for this project follows Medallion Architecture Bronze, Silver, and Gold layers:

Data Architecture

1. Bronze Layer: Stores raw data as-is from the source systems. Data is ingested from CSV Files into SQL Server Database.
2. Silver Layer: This layer includes data cleansing, standardization, and normalization processes to prepare data for analysis.
3. Gold Layer: Houses business-ready data modeled into a star schema required for reporting and analytics.

---

📖 Project Overview

This project involves:

1. Data Architecture: Designing a Modern Data Warehouse Using Medallion Architecture Bronze, Silver, and Gold layers.
2. ETL Pipelines: Extracting, transforming, and loading data from source systems into the warehouse.
3. Data Modeling: Developing fact and dimension tables optimized for analytical queries.

---

🚀 Project Requirements

Building the Data Warehouse (Data Engineering)

Objective

Develop a modern data warehouse using SQL Server to consolidate sales data, for downstream reporting and informed decision-making.

Specifications

• Data Sources: Import data from two source systems (ERP and CRM) provided as CSV files.
• Data Quality: Cleanse and resolve data quality issues prior to analysis.
• Integration: Combine both sources into a single, user-friendly data model designed for analytical queries.
• Scope: Focus on the latest dataset only; historization of data is not required.
• Documentation: Provide clear documentation of the data model to support both business stakeholders and analytics teams.

---

📂 Repository Structure

data-warehouse-project/
│
├── datasets/                           # Raw datasets used for the project (ERP and CRM data)
│
├── design/                                 # Project documentation and architecture details
│   ├── Data_flow_diagram.drawio            # data flow diagram
│   ├── Integration_model.drawio            # Catalog of datasets, including field descriptions and metadata
│   ├── Project_high_level_design.drawio    # Macro view
│   ├── star_data_model.drawio              # star schema for gold layer fact and dimenton representation
│
├── scripts/                                # SQL scripts for ETL and transformations
│   ├── bronze/                             # Scripts for extracting and loading raw data
│   ├── silver/                             # Scripts for cleaning and transforming data
│   ├── gold/                               # Scripts for creating analytical models
│
├── data_architecture.png                   # Macro View
├── gold_data_catalog.md                    # Final gold layer views ready for insights and project downstream flow.
├── README.md                               # Project overview and instructions
└── LICENSE                                 # License information for the repository

---

Project releated snips

Bronze Creating Tables	Bronze Loading Data	Bronze Stored Procedure	Bronze Layer Questionnaire	SL Understanding data	SL stored procedure 1	SL stored procedure 2	Gold Views for downstream

🛡️ License

This project is licensed under the MIT License. You are free to use, modify, and share this project with proper attribution.