Task Offloading for the Metaverse with Deep Reinforcement Learning

M.Sc. thesis (completed, September 2025): Design a Task Offloading Policy for Computational Tasks in Metaverse Mohammad Javad Amin · Department of Electrical Engineering, Sharif University of Technology Supervisors: Prof. Babak Hossein Khalaj, Prof. Farid Ashtiani

Overview

Metaverse applications produce heavy, delay-sensitive computation on devices with little compute and limited battery, such as XR headsets and phones. This thesis designs an offloading policy for an end-device / edge / cloud system. At every time slot it decides where each task runs (locally, on its edge server, or in the cloud) so that it:

• maximizes the share of tasks completed before their deadlines (soft deadline gives the best QoE, hard deadline gives acceptable QoE), and
• minimizes energy consumption on the end devices.

There is no labeled dataset that covers these dynamic, non-stationary scenarios, so the policy is learned with deep reinforcement learning in a custom simulator. The thesis uses value-based agents with separate online and target networks: DQN and Dueling DQN (called Double DQN and Double Dueling DQN in the thesis). It compares two architectures:

Architecture	Folder	Agent(s)	Trade-off
Single agent (cloud)	single_agent/	One central agent decides for all devices.	Global view, but a huge joint action space (3^15 ≈ 14.3 M actions with 15 devices) and a 94-dimensional observation. Training is slow and every decision crosses the network to the cloud.
Multi-agent (edge)	multi_agent/	One independent agent per edge server, each deciding for its own devices.	Smaller observation and action spaces, faster convergence, and decisions close to the devices. This scales to more users.

System model

• Topology: 1 cloud server (128 GHz), 3 edge servers (24 GHz each), and N end devices (4 GHz each) split evenly across the edge servers.
• Network: 10 ms time slots; end device → edge 100 Mb/s, edge → cloud 1 Gb/s, 5 ms propagation delay.
• Tasks: each task has a size Z, a compute density C (cycles/bit), a deadline D and a priority P.
  • Foreground tasks (P = 2): interactive and light, short deadlines, high QoE impact; generated with probability 0.55 per device per slot.
  • Background tasks (P = 1): environment and asset updates, heavier, longer deadlines; generated with probability 0.30 per device per slot.
  • Soft and hard deadlines: completion before the soft deadline gives the best QoE; completion before the hard deadline (1.4 × soft) is still acceptable.
• Observation: per-task features (Z, C, D, P), queue status of the device, edge and cloud, and the device's energy use.
• Action: for each device with a new task, run it locally, on the edge, or in the cloud.
• Baselines: local only, edge only, cloud only, edge + cloud, and random.

Training: Adam, learning rate 5 × 10⁻⁵, batch size 64, γ = 0.9, replay buffer of 10⁴ transitions, 1000 episodes of 100 steps, target network updated every 100 episodes, MSE loss, ε-greedy exploration decaying exponentially from 1.0 to 0.01.

Repository structure

.
├── single_agent/            # centralized (cloud) agent
│   ├── env.py               # SingleAgentTaskOffloadingEnv: devices, edge, cloud, queues, energy
│   ├── Models.py            # DQN and Dueling DQN networks, replay buffer
│   ├── Utility.py           # Trainer (online/target networks, replay buffer) and AlgorithmTester (baselines, metrics, plots)
│   ├── master_thesis.ipynb  # training and evaluation
│   ├── saved_models/        # trained weights and observation normalizers
│   └── result/              # evaluation outputs (JSON)
├── multi_agent/             # one agent per edge server
│   ├── env.py               # MultiAgentTaskOffloadingEnv
│   ├── Models.py
│   ├── Utility.py
│   ├── master_thesis.ipynb
│   ├── saved_models/        # per-edge-agent weights for N = 3, 5, 10, 15, 20 devices per edge
│   ├── result/              # rewards, losses and CDFs per method and N
│   └── summary_N_*.csv      # per-priority summaries
├── docs/
│   ├── report.md            # English summary report
│   ├── thesis-fa.pdf        # full thesis (Persian, with English abstract)
│   └── defense-slides.pptx  # defense presentation (English)
└── requirements.txt

Running

Locally:

pip install -r requirements.txt
cd multi_agent            # or single_agent
jupyter lab master_thesis.ipynb

On Google Colab, the first cells clone the repository and change into the matching folder. Models, results and summaries are read from and written to the folder of the code that runs, so both versions work from any working directory.

Thesis

📄 English summary report: problem, system model, optimization problem, MDP, learning method, decision architectures and experimental setup.

The full thesis, including the literature review, the formal system model, the delay and energy equations, and all experiments, is in docs/thesis-fa.pdf (Persian, with an English abstract). The defense slides in English are in docs/defense-slides.pptx.

Author

Mohammad Javad Amin · GitHub · LinkedIn