The Prophet’s order to pull up trousers or clothes above the ankles was in the early days of Islam. The order was against the baggy clothes of the early Arabs who used to drag filth when they walk or even attract attention either while walking or while praying. It is worth mentioning here that some Arabs used to wear long clothes to show arrogance. If a person avoids arrogance and he is keen that his trousers do not drag filth, then there is nothing wrong in wearing them.
That is why men should keep their trousers a little bit above the ankles, and it may be lengthened till the ankles, but below the ankles is forbidden if it implies showing-off; if not, lengthening in this case is just makruh (blameworthy), especially if it drags on filth.

The above clarification applies to both when one is praying, initiating prayers or out of prayer.