Wireless relay networks are promising technologies for high data rate transmissions in next generation networks. However, wireless resources, such as energy and spectrum, are limited. Wireless relay networks alone are not enough to provide high data rate transmissions. Therefore, advanced resource allocation algorithms that can fully exploit resources in wireless relay networks are crucial for network performance improvement. This dissertation has investigated how to efficiently and fairly allocate the limited resources to users in three typical wireless relay networks: orthogonal frequency division multiple access (OFDMA) relay networks, multiuser multirelay networks, and relay-based cognitive radio networks (CRNs). The main objective of this dissertation is to design new resource allocation algorithms for multiple dimensions, power, bandwidth, and relay selection, to balance the optimal resource allocation and the computational complexity while guaranteeing the users’ quality of service, achieving fairness among users, and satisfying the constraints from related techniques. Towards that end, mathematical optimization, especially convex optimization and mixed integer nonlinear programming (MINLP), is employed to address the resource allocation issues. First, the problem of subcarrier and transmission power allocation in OFDMA relay networks that seeks to maximize the sum rate and maintain proportional rate fairness among users will be addressed. Because the formulated problem is a MINLP problem with an extremely high computational complexity, a low-complexity suboptimal algorithm, which is a two-step separated subcarrier and power allocation algorithm, is proposed. Next, the problem of fair resource allocation based on the Nash bargaining solution over wireless multiuser multirelay networks is investigated. The problem is formulated as an optimization problem. And it is proven to be a convex optimization problem. When the problem is feasible, a distributed optimal algorithm, including relay selection, relay power allocation, and rate adaptation, is proposed. When the problem is infeasible, admission control is necessary. The joint admission control and fair resource allocation problem is formulated as a two-stage optimization problem. Since this problem is combinatorially difficult, it is transformed into an equivalent one-stage optimization problem. To reduce the computational complexity further, a suboptimal algorithm is then developed. Finally, a study on the problem of subcarrier pairing and power allocation in relaybased CRNs where a primary user and a secondary user (SU) coexist is conducted. The problem is formulated as a MINLP problem with the objective of maximizing the sum rate of the SU. Using the dual decomposition method, a jointly optimal subcarrier pairing and power allocation algorithm is proposed. To reduce the computational complexity further, two suboptimal algorithms are exploited as well. Simulations have been conducted to verify the performance of the proposed algorithms.