There are clear injunctions from the Prophet (peace and blessings be upon him) forbidding men to let their garments below the ankles. The prohibition is clear if this is done to show off or as an act of haughtiness. If not, jurists differed whether this forbiddance denotes that this is still haram (punishable prohibition) or that it is just makruh (disliked or undesirable).
There are also prophetic injunctions banning the Muslim in Prayer to roll up or collect his garment to prevent it from touching the ground while prostrating. The majority of jurists hold that this is makruh to do so in Prayer.
So, generally one has to keep his trousers above his ankles to avoid falling in the prohibited area. If it happens that he joins the Prayer with his garment below his ankles, then he may roll it up inwardly or outwardly committing the lesser of the two
prohibitions.