With the ever increasing mobile traffic in wireless networks, radio frequency spectrum is becoming limited and overcrowded. To address the radio frequency spectrum scarcity problem, researchers proposed advanced radio technology-Cognitive Radio to make use of the uncommonly used and under-utilized licensed bands to improve overall spectrum efficiency. Mobile service providers also deploy small base stations on the streets, into shopping center and users' households in order to improve spectrum efficiency per area. In this thesis, we study cooperation schemes in cognitive radio networks as well as heterogeneous networks to reuse the existing radio frequency spectrum intelligently and improve network throughput and spectrum efficiency, reduce network power consumption and provide network failure protection capability.

In the first work of the thesis, we study a multicast routing problem in Cognitive Ratio Networks (CRNs). In this work, all Secondary Users (SUs) are assumed not self interested and they are willing to provide relay service for source SUs. We propose a new network modeling method, where we model CRNs using a Multi-rate Multilayer Hyper-Graph (MMHG). Given a multicast session of the MMHG, our goal is to find the multicast routing trees that minimize the worst case end-to-end delay, maximize the multicast rate and minimize the number of transmission links used in the multicast tree. We apply two metaheuristic algorithms (Multi-Objective Ant Colony System optimization algorithm (MOACS) and Archived Multi-Objective Simulated Annealing Optimization Algorithm (AMOSA)) in solving the problem. We also study the scheduling problem of multicast routing trees obtained from the MMHG model.

In the second work of the thesis, we study the cell outage compensation function of the self-healing mechanism using network cooperation scheme. In a heterogeneous network environment with densely deployed Femto Base Stations (FBSs), we propose a network cooperation scheme for FBSs using Coordinated Multi-Point (CoMP) transmission and reception with joint processing technique. Different clustering methods are studied to improve the performance of the network cooperation scheme.

In the final work of the thesis, we study the user cooperative multi-path routing solution for wireless Users Equipment (UEs)' streaming application using auction theory. We assume that UEs use multi-path transport layer service, and establish two paths for streaming events, one path goes through its cellular link, another path is established using a Wi-Fi connection with a neighbor UE. We study user coordinated multi-path routing solution with two different energy cost functions (LCF and EAC) and design user cooperative real-time optimization and failure protection operations for the streaming application. To stimulate UEs to participate into the user cooperation operation, we design a credit system enabled with auction mechanism.

Simulation results in this thesis show that optimal cooperation operations among network devices to reuse the existing spectrum wisely are able to improve network performance considerably. Our proposed network modeling approach in CRN helps reduce the complicated multicast routing problem to a simple graph problem, and the proposed algorithms can find most of the optimal multicast routing trees in a short amount of time. In the second and third works, our proposed network cooperation and user cooperation approaches are shown to provide better UEs' throughput compared to non-cooperation schemes. The network cooperation approach using CoMP provides failure compensation capability by preventing the system sum rate loss from having the same speed of radio resource loss, and this is done without using additional radio resources and will not have a significant adverse effect on the performance of other UEs. The user cooperation approach shows great advantage in improving service rate, improving streaming event success rate and reducing energy consumption compared to non-cooperation solution.