This is the entrance to the Sentō Gosho (仙洞御所), a palace built in 1630 for the Emperor Go Mizuno-o (後水尾天皇) who was forced into retirement by the Shogun, Tokugawa Hidetada (徳川 秀忠).
It all began when the Emperor married the Shogun's daughter. The imperial couple had a son, now the Shogun's grandson, and the Shogun wished to place him on the Chrysanthemum Throne. He therefore "encouraged" Emperor Go Mizuno-o to retire. To sweeten the deal, the Shogun offered a handsome amount of money and built this palace. Unfortunately for everyone concerned, the new emperor soon died, as did Go Mizuno-o's second son. The result was a rarity in Japanese imperial history: the reign of a woman, the Empress Meishō (明正天皇).
After abdicating, the retired emperor lived in this palace for another fifty years, and he became a connoisseur of many things including women (he eventually fathered over thirty children), flower arranging and especially gardens. The palace buildings did not do as well: they burned down on six different occasions and in 1854 after the last major fire the buildings were not put up again. The palace is now open by appointment to the public (you can find out how to apply here) and is still famous for the spectacular gardens, laid out in the 1630s under the retired emperor's supervision, as well as an exquisite tea house.