To better allow accurate identification of non-smooth signals in two-way relay networks. In this paper, an algorithm based on the two-way relay network system model, the RLS algorithm, is proposed. The algorithm introduces the recursive least squares principle algorithm to the basic cost function of nonsmooth signal identification, optimizes the signal identification model by exponential weighting, and uses the time-varying variance matrix and the optimal identification data separation matrix to make real-time estimates of the new cost function for obtaining nonsmooth signals. And a new index, the PI index, is introduced to define whether the non-stationary signal is stable and reliable. Through two sets of comparison tests, we can learn that the RLS algorithm based on the two-way relay network system is better than other algorithms in the identification of non-stationary signals. It shows that the RLS algorithm can better identify and manage the information flow in a two-way relay network, make the non-stationary signal more stable, and further improve the convergence and convergence speed of the non-stationary signal to provide a better information interaction experience for both sides of electronic communication.