Abstract
Cooperative relaying refers to a technique that allows the source to transmit its messages to the destination via the relaying of multiple cooperative partners and, in this way, exploits the spatial diversity gains inherent in multiuser wireless systems. In this work, we examine cooperative networks with both cooperative and malicious relays and determine the optimal behavior for both kinds of relays using a game-theoretic approach. To study these issues, we formulate the problems into zero-sum games for both decode-and-forward (DF) and amplify-and-forward (AF) systems. Then, we determine the optimal relay strategies by identifying the Nash equilibrium of these problems under individual power constraints. In the DF case, we show that, with Rayleigh fading, the optimal strategy for malicious relays is to transmit independent Gaussian noise using full power at each relay and the optimal strategy for cooperative relays is to independently re-encode the source’s message into Gaussian signals and forward them to the destination. Inter-cooperation among relays is not necessary. In the AF case, we prove, for the case with only one cooperative relay, that malicious relays do not gain by overhearing the source’s message and, thus, should transmit Gaussian noise in both phases to corrupt the reception at both the cooperative relay and the destination. The optimal strategy for the cooperative relay in this case is to amplify and forward the received signal with full power at each relay. In this case, we also show that Gaussian signaling at the source is optimal. The results are verified through numerical simulations.