The monastery has been destroyed and rebuilt many times. In 1641 the troops of anti-Ming rebel Li Zicheng sacked the monastery due to the monks' support of the Ming and the possible threat they posed to the rebels. This effectively destroyed the temple's fighting force.

Perhaps the best-known story of the Temple's destruction is that it was destroyed by the Qing government for supposed anti-Qing activities. Variously said to have taken place in 1647 under the Shunzhi Emperor, in 1674 under the Kangxi Emperor, or in 1732 under the Yongzheng Emperor, this destruction is also supposed to have helped spread Shaolin martial arts through China by means of the five fugitive monks Ng Mui, Jee Shin Shim Shee, Fung Doe Duk, Miu Hin and Bak Mei. Some accounts claim that a supposed southern Shaolin Temple was destroyed instead of, or in addition to, the temple in Henan: Ju Ke, in the Qing bai lei chao (1917), locates this temple in Fujian Province. This account states that Ming loyalists infiltrated the Southern Temple to disseminate anti-Qing ideology and that the Qing Emperor himself infiltrated the Southern Temple to learn Shaolin Kung Fu. Tibetan Lamas were said to have aided Yongzheng Emperor's army in razing the Temple with a deadly flying weapon known as "Huit Tik Tze" or a Flying guillotine. These stories commonly appear in martial arts history, fiction, and cinema.

While these latter accounts are common among martial artists, and often serve as origin stories for various martial arts styles, their accuracy is questionable. The accounts are known through often inconsistent 19th-century secret society histories and popular literature, and also appear to draw on both Fujianese folklore and popular narratives such as the Water Margin. Modern scholarly attention to the tales is mainly concerned with their role as folklore, or as clues to the history of secret societies or possible southern Shaolin temples.