Dr. Monzer Kahf, a prominent economist and counselor states: It is ok to use this money for Hajj. The least that can be said in this regard is that this is your principal and even in regards to giving loans on interest God said what means that you should take your principal back as taking it back means that no injustice is done by you or against you. This is your principal that I assume you accumulated from your savings and other Halal sources, you can then use it any way you like.”
Allah Almighty knows best.