Dr. `Abdullah Al-Faqih, supervisor of Fatwa Centre at www.islamweb.net, states: The place of the Sacred House was known to Prophet Adam (peace and blessings be upon him). It was said that it was he who placed the foundations of the Sacred House and then Prophet Ibrahim and his son Prophet Isma`il (peace and blessings be upon both of them) raised these foundations as it is mentioned in the Qur’an.
As for all prophets having made Hajj to the Sacred House, this was reported in a number of hadiths. For example, it was reported on the authority of Anas ibn Malik (may Allah be pleased with him) that Prophet Muhammad (peace and blessings be upon him) said: “The place of the Sacred House at the time of Prophet Adam (peace and blessings be upon him) was protruded about a span of the hand or a little more (from the earth). The angels would make Hajj to it before Adam (peace and blessings be upon him). Then Prophet Adam (peace and blessings be upon him) made Hajj to it, (after which) the angels met him and asked him, ‘Where have you come from?’ He said, ‘I visited the Sacred House.’ They said, ‘The angels made Hajj to it before you’” (Reported by Al-Bayhaqi in his book Shu`ab al-Iman).
Prophet Nuh (peace and blessings be upon him) also made Hajj to the Sacred House. It was said in a hadith reported on the authority of `Urwa ibn Az-Zubair (may Allah be pleased with him and his father):“All Prophets except Hud and Salih (peace and blessings be upon all of them) made Hajj to the Sacred House. Prophet Nuh visited it; then when the flood took place and the earth drowned, the Sacred House, which was then a red height, perished under the flood. Then Almighty Allah sent Prophet Hud (peace and blessings be upon him) to call his people to the right path. Prophet Hud was so busy undertaking his mission that he did not manage to make Hajj to the Sacred House before he died. When Prophet Abraham was shown the place of the Sacred House, he made Hajj to it, and so did all the prophets who were sent after him” (Reported by Al-Bayhaqi in his book As-Sunnan).