In his book “Fiqh-us-Sunnah”, the late Sheikh Sayyed Sabiq (may Allah be pleased with him) states: “There is no dispute among the scholars that a congregation is a condition for the validity of Salatul Jumu`ah (Friday Prayer). This is based on the Hadith of Tariq Ibn Shihab who reported that the Prophet (peace and blessings be upon him) said: “Al-Jumu`ah is an obligation (Wajib) upon every Muslim in the community.”
However, the scholars differ on how many people are required for Salatul Jumu`ah. There are about fifteen opinions on this issue and Ibn Hajar mentions them in “Fath Al-Bari“. The view believed to be the most correct is that Salatul Jumu`ah is valid if there are two or more people present. This is based on the Hadith in which the Prophet (peace and blessings be upon him) is reported to have said: “Two or more constitute a congregation.”
Imam Ash-Shawkani states: “The other Prayers are considered to be made in congregation if there are two people present. The same applies to Salatul Jumu`ah, unless there is a reason for it to be different. There is no evidence to show that [for the purpose of the congregation] its number should be larger than that for the other Prayers.”
`Abdul Haqq says: “There is no confirmed Hadith on the number of people needed for Salatul Jumu`ah.”
Likewise, As-Sayuti holds: “There is no confirmed Hadith that states a particular number (for Salatul Jumu`ah).” This is also the opinion held by At-Tabari, Dawud, An-Nakha`i, and Ibn Hazm.“
Based on the above-mentioned Fatwa, we can conclude that it is valid for to hold Salatul Jumu`ah; with two or more persons, and it is not a condition to reach the number you mentioned in the question.