Abstract
We analyze stability of the TM polarized optical solitons and nonlinear guided waves localized at a metal- dielectric interface. We demonstrate, both analytically and numerically, that the spatial solitons can experience vectorial transverse modulational instability that leads to the generation of arrays of two-dimensional TM polarized self-trapped localized beams. In a sharp contrast, we reveal that the transverse instability is completely eliminated for nonlinear surface plasmons. © 2009 Optical Society of America.