Abstract
The trellis-coded generalized spatial modulation (TCGSM) system with multiple code rates is investigated so as to provide configurability and to enhance efficiency in different fading channels. Various high-radix stages are then adopted to support the optimal sequence detection for different configurations. A double K-best technique is then proposed to strike a good balance between performance and complexity of Viterbi-sphere decoding. From the analysis and simulation results, 29%-69% complexity can be saved with about 1dB performance degradation for the code rate of 4/5 in the spatial domain with QPSK and 16-QAM constellations. The first K-best selection for generating branch metrics from sphere decoding can reduce the detection complexity of the constellation while the second K- best selection can be considered if a high-radix trellis is used when the number of transmit antennas is large.