Scholars and historians say that the Ka`bah has been reconstructed between 5 to 12 times. According to some of them, the very first construction of the Ka`bah was done by Prophet Adam (peace and blessings be upon him). After this, ProphetIbrahim and Isma`il (peace and blessings be upon them) rebuilt the Ka’bah.
Regarding this issue, Sheikh Ahmad Kutty, an Islamic scholar said:
We have no clear evidence in the sources about this, although there are some reports that indicate that the Ka`bah was built by Prophet Adam (peace and blessings be upon him) and it had been a site of pilgrimage from ancient times; as such, it had been visited by various prophets. These reports, however, do not pass the rigorous standards of scientific verification established by the scholars of Hadith.
and his son (peace and blessings be upon them) only rebuilt the , and that Prophet (peace and blessings be upon him) was guided to the site. And the reports indicate that it had been in ruins and buried by a heap of sand. We cannot go any farther than this to elaborate the details for lack of clear evidences in the sources.
It is better to leave the issue of building the Ka`bah at this point and stop speculating on such matters, leaving them to the knowledge of Allah.