It is unanimously agreed upon among scholars that Salah is obligatory upon Muslims and that Muslim parents are to order their children when they are seven years old to perform it, but if the children have reached ten years old without observing Salah, their parents are to beat them for that. Beating in that case is to be with the hand, not a harmful instrument. There is a Hadith to that effect: “Order your children to perform Salah when they are seven years old, and beat them (if they have neglected it) when reaching ten years old. And separate between your male and female children in bed.”
The one who denies that Salah is obligatory is regarded as an unbeliever, for the Salah is one of the pillars of Islam, and the evidence for this is clear cut. There is a scholarly consensus upon that.
As for the one who holds that the Salah is obligatory, but he abandons it out of lack of concern (as it is the case of many Muslim people), there is a scholarly difference as to judging him.
According to the majority of the righteous Muslim predecessors and the majority of the righteous successors, including Imams Malik and Ash-Shafi`i, he who does so is not to be regarded as an unbeliever; rather, he is to be regarded as a sinner.
Another group of the righteous predecessors believed that the one who abandons Salah while still believing it to be obligatory is to be considered an unbeliever. Such an opinion was attributed to `Ali ibn Abi Talib (may Allah honor him). Such point of view was also reported to have been said by Imam Ahmad ibn Hanbal. (Yet there was another opinion in that regard also reported to have been said by Imam Ahmad.) Some of the followers of Imam Ash-Shafi`i held that opinion, too.
Each of the above opinions cited evidence to support its point of view. Some of these pieces of evidence are quotations from the Qur’an and some are Hadiths of Prophet Muhammad (peace and blessings be upon him).
It is noteworthy that a Muslim attitude toward his brother is to advise him and help him obey the obligations of Allah. It is not for ordinary people to declare others as kafirs.