The prostration of forgetfulness in prayer is prescribed for three reasons: doing something extra in prayer, omitting something, and being uncertain. It should be performed before the salam if you have omitted one of the obligatory parts of the prayer or if you are not sure how many rak`ahs you have done, and neither of the two choices seems more likely to be the case. And it should be done after the salam if you have added something extra to the prayer or you are not certain but one of the two choices seems more likely to be the case.
elaborating more on this, the late Sheikh Ibn `Uthaymeen, a prominent Saudi Muslim scholar (may Allah shower mercy on his soul), said: “The prostration of forgetfulness in prayer is generally prescribed for three reasons:
1.Doing something extra: for example, doing an extra bow (ruku`), prostration (sujud), standing or sitting.
2. Omitting something: such as omitting an essential part of the prayer, or omitting one of the obligatory parts of the prayer.
3. Being uncertain: such as when a person is not sure how many rak`ahs he has prayed, whether it was three or four, for example.
In the case of doing something extra, if a person adds something to his prayer – bowing, prostrating, standing or sitting – deliberately, then his prayer becomes invalid, because when he added it, that means that he did not do the prayer in the manner ordained by Allah and His Messenger (peace and blessings be upon him). The Prophet (peace and blessings be upon him) said: “Whoever does an action that is not part of this matter of ours (i.e., Islam), will have it rejected.” (Muslim)
But if he did that extra thing by mistake, then his prayer is not invalidated, but he should do the prostration of forgetfulness after saying the salam. The evidence for that is the hadeeth narrated by Abu Hurairah (may Allah be pleased with him) who said that when the Prophet (peace and blessings be upon him) said the salam after two rak`ahs in one of the two afternoon prayers, either Zuhr or `Asr, and they told him about that, he (peace and blessings be upon him) did the rest of the prayer, then he said the salam, then he prostrated twice (the prostration of forgetfulness) after saying the salam. (Al-Bukhari and Muslim)
And Ibn Mas`oud (may Allah be pleased with him) narrated that the Prophet (peace and blessings be upon him) led them in praying Zuhr and he prayed five rak`ahs. When he had finished it was said to him: “Has something been added to the prayer?” He said: “Why are you asking that?” They said: “You prayed five (rak`ahs).” So he turned towards the qiblah and prostrated twice. (Al-Bukhari and Muslim)
In the case of omitting something, if a person omits one of the essential parts of the prayer, one of the following two scenarios must apply: either he remembers it before he reaches the same point in the following rak`ah, so he has to go back and do that essential part of the prayer and whatever came after it, or he does not remember it until he reaches the same point in the following rak`ah, in which case the following rak`ah takes the place of the one in which he omitted that essential part, and he should make up the invalid rak`ah.
In either of these two cases, he should do the prostration of forgetfulness after saying the salam.
For example: a man stood up after doing the first prostration of the first rak`ah and did not sit or do the second prostration. When he started to recite Qur’an he remembers that he had not done the second prostration or sat between the two prostrations. In that case he should go back and sit as between the two prostrations, then prostrate, then stand up and do whatever is left of his prayer, and do the prostration of forgetfulness after saying the salam.
An example of one who did not remember until after he reached the same point in the following rak`ah is a man who stood up following the first prostration in the first rak`ah and did not do the second prostration or sit between the two prostrations, but he did not remember that until he sat between the two prostrations in the second rak`ah. In this case he should make the second rak`ah the first rak`ah, and add another rak`ah to his prayer, then say the salam then do the prostration of forgetfulness.
Omitting an obligatory part of the prayer – such as if he omits an obligatory part and moves on to the next part of the prayer. For example, he forgets to say SubhanRabbiyyaAl-‘A`la (Glory be to my Lord Most High) and he does not remember until he has got up from prostrating. In this case he has omitted an obligatory part of the prayer by mistake, so he should carry on with his prayer then do the prostration of forgetfulness before saying the salam, because when the Prophet (peace and blessings of be upon him) omitted the first tashahhud he carried on with his prayer and did not go back and repeat anything, then he did the prostration of forgetfulness before saying the salam.
In the case of being uncertain, such as if a person is not sure if he has done something extra or omitted something, e.g., he is not sure whether he has prayed three rak`ahs or four, one of the following two scenarios must apply:
Either he thinks that one or the other is more likely to be case, whether it is doing more or omitting something, in which case he should proceed on the basis of what he thinks is more likely to be the case, complete his prayer accordingly, then do the prostration of forgetfulness after saying the salam; or neither appears more likely to be the case, so he should proceed on the basis of what is certain, namely the lesser amount, and complete his prayer accordingly, then do the prostration of forgetfulness before saying the salam.
For example: a man prays Zuhr then he is not sure whether he is praying the third or fourth rak`ah, but he thinks it more likely that it is the third. So he should pray another rak`ah, then say the salam, then do the prostration of forgetfulness.
An example of when neither seems to him more likely to be the case is when a man is praying Zuhr and is not sure whether he is in the third rak`ah or the fourth, and neither seems to him more likely to be the case. In that case he should proceed on the basis of what is certain, which is the lesser amount. So he should regard it as the third rak`ah, then do another rak`ah and do the prostration of forgetfulness before saying the salam.
Thus, it becomes clear that this should be done before the salam if you have omitted one of the obligatory parts of the prayer or if you are not sure how many rak`ahs you have done, and neither of the two choices seems more likely to be the case. And it should be done after the salam if you have added something extra to the prayer or you are not certain but one of the two choices seems more likely to be the case.”