Avodah Zarah 48 - Houses of Idol Worship
If one's house adjoined a building of idol worship and their common wall collapsed, it is forbidden to rebuild it. What should he do? He withdraws four steps into his property and builds the wall there, but to prevent the idol from benefit, he uses his area as a toilet or plants it with thorn-bushes.
There are three categories of idolatrous houses. A house that was built for use as an idol is prohibited. If a house was plastered and decorated for idol worship - one removes the additions, and the house is permitted. If an idol was brought into a house and then removed - the house is permitted.
Similar laws apply to stones and trees that were worshipped. However, a tree under which there is an idol is a subject of disagreement between Rabbi Shimon and the Sages. The Sages forbid it, but Rabbi Shimon allows it if the worship can be attributed to the idol underneath. It once happened in Sidon that Rabbi Shimon told to his followers to examine the heap under a worshiped tree. They found an image there, and he permitted the tree.
Art: T. Caffi - The Ruined Temple of Saturn in The Roman Forum