Prime Minister Junichiro Koizumi should refrain from making his contentious visits to the war-related Yasukuni Shrine to avoid further damaging ties with Japan's Asian neighbors, five former prime ministers intimated Wednesday.

"We cannot overlook the rapid deterioration in Japan's relations with China and South Korea," House of Representatives Speaker Yohei Kono said after inviting Kiichi Miyazawa, Tomiichi Murayama, Ryutaro Hashimoto, Yoshiro Mori and Toshiki Kaifu to his official residence.

"It cannot be denied that one of the reasons behind (the worsening ties) is (Koizumi's) visits to Yasukuni," said Kono, who is known as a leading dove within the ruling Liberal Democratic Party.