AI-Powered Network Security System

Real-Time ETL Pipelines, Automated MLOps Retraining, and Blockchain-Secured Logs

📋 Project Overview

This project implements an integrated AI-driven, automated, and blockchain-secured network security system with real-time threat detection and immutable audit trails. It addresses the challenges of modern cybersecurity by combining machine learning, blockchain technology, and automated MLOps pipelines.

🎯 Key Features

• Automated ETL Pipeline: Extract, transform, and load network security data with blockchain logging
• AI-Powered Threat Detection: Deep learning models (MLP-GRU) for anomaly detection
• Blockchain-Secured Audit System: Tamper-proof logs for forensic analysis
• MLOps Automation: Continuous model retraining and deployment
• Real-Time Monitoring Dashboard: Streamlit-based interface for live threat analytics

🏗️ Project Structure

5th-Semester-Project-/
├── src/
│   ├── etl/                    # ETL Pipeline Module
│   ├── models/                 # AI/ML Models
│   ├── blockchain/             # Blockchain Integration
│   ├── mlops/                  # MLOps Pipeline
│   ├── api/                    # FastAPI Backend
│   └── dashboard/              # Streamlit Dashboard
├── data/
│   ├── raw/                    # Raw security datasets
│   ├── processed/              # Processed data
│   └── models/                 # Trained models
├── config/                     # Configuration files
├── tests/                      # Unit and integration tests
├── docs/                       # Documentation
├── scripts/                    # Utility scripts
├── notebooks/                  # Jupyter notebooks for exploration
└── deployment/                 # Docker & Kubernetes configs

🛠️ Technology Stack

• ETL & Processing: Python, Pandas, Apache Airflow, Apache Spark
• Data Streaming: Apache Kafka
• Machine Learning: TensorFlow, PyTorch, Scikit-learn
• MLOps: MLflow, DagsHub
• Blockchain: Web3.py, Smart Contracts, IPFS
• Databases: MongoDB, PostgreSQL
• Cloud: AWS (S3, EC2)
• Deployment: Docker, Kubernetes
• Frontend: Streamlit, FastAPI
• AI Integration: Google Gemini API

📊 Datasets Used

1. CIC-IDS2017: Intrusion detection dataset
2. UNSW-NB15: Network intrusion dataset
3. Phishing Websites Dataset
4. Real-time logs: Firewall logs, network packets, system events

🚀 Getting Started

Prerequisites

• Python 3.10+
• macOS (or Linux/Windows with WSL)
• Git
• Sufficient disk space (at least 10GB)

Installation

1. Clone the repository:

cd /Users/omsuneri/5th-Semester-Project-

2. Create and activate virtual environment:

python3 -m venv venv
source venv/bin/activate  # On macOS/Linux

3. Install dependencies:

pip install --upgrade pip
pip install -r requirements.txt

4. Set up environment variables:

cp config/.env.example config/.env
# Edit config/.env with your API keys and configurations

5. Initialize the database:

python scripts/init_database.py

6. Download datasets:

python scripts/download_datasets.py

🎮 Running the Project

1. Start the ETL Pipeline

python src/etl/pipeline.py

2. Train the AI Models

python src/models/train_model.py

3. Start the Blockchain Node (Development)

python src/blockchain/blockchain_node.py

4. Launch the Streamlit Dashboard

streamlit run src/dashboard/app.py

5. Start the FastAPI Backend

uvicorn src.api.main:app --reload --host 0.0.0.0 --port 8000

🔧 Configuration

Edit config/config.yaml to customize:

• Database connections
• API endpoints
• Model parameters
• Blockchain settings
• Alert thresholds

Add your API keys to config/.env:

GEMINI_API_KEY=your_gemini_api_key_here
MONGODB_URI=your_mongodb_connection_string
POSTGRES_URI=your_postgres_connection_string
AWS_ACCESS_KEY_ID=your_aws_access_key
AWS_SECRET_ACCESS_KEY=your_aws_secret_key

📈 Monitoring & Alerts

Access the dashboard at: http://localhost:8501

Features:

• Real-time threat detection
• Network traffic analysis
• Model performance metrics
• Blockchain audit logs
• Alert management

🧪 Testing

Run all tests:

pytest tests/ -v --cov=src

Run specific test modules:

pytest tests/test_etl.py -v
pytest tests/test_models.py -v
pytest tests/test_blockchain.py -v

📚 Documentation

Detailed documentation is available in the docs/ folder:

• Architecture Overview
• ETL Pipeline Guide
• Model Training Guide
• Blockchain Integration
• API Documentation
• Deployment Guide

🤝 Contributing

This is an academic project developed by:

• Om Santosh Suneri (UE238066)
• Shubham Choubey (UE238101)
• Sourav Biswas (UE238103)
• Tanuj Ramchandani (UE238108)
• Sehwag Meena (UE238095)
• Yatin Kumar (UE238112)

📄 License

This project is developed as part of B.E.(IT) coursework at the University Institute of Engineering & Technology, Panjab University, Chandigarh.

🙏 Acknowledgments

• Dr. Amandeep Verma Ma'am
• Amrit Sandhu Ma'am
• Department of Information Technology, UIET, Panjab University

📞 Support

For issues or questions, please create an issue in the repository or contact the project team.

---

Project Type: 5th Semester Project (B.E. Information Technology)
Institution: UIET, Panjab University, Chandigarh
Academic Year: 2023-2027