Shevuot (Oaths) 10 - For What Each Specific Goat Atones
According to Rabbi Yehudah, both the goats brought on Festivals and the goats brought on the New Moon atone for such violations of ritual purity laws of which one never becomes aware.
Rabbi Shimon disagrees about the goats of New Moon and says that these atone specifically for cases when one, while being pure, inadvertently eats sacrifices that are impure.
Rabbi Meir says that the atonement of all he-goats is the same: they atone for inadvertent purity violations of the Temple and of its offerings.
Rabbi Shimon ben Yehuda has a different version of what Rabbi Shimon said: that the goats atone for progressively wider range of infringements, in order from New Moon to Festivals to Yom Kippur. Said the Sages to him, "If they are so similar, can a goat designated for the New Moon be brought on Yom Kippur ?" He said, "Yes. Even though they are not the same, they have that in common that they deal with purity of the Temple and its offerings."
Paul Gauguin - Landscape with Two Goats