The people follow the Quraish and the Caliphate belongs to the Quraish
وَحَدَّثَنَا أَحْمَدُ بْنُ عَبْدِ اللَّهِ بْنِ يُونُسَ، حَدَّثَنَا عَاصِمُ بْنُ مُحَمَّدِ بْنِ زَيْدٍ، عَنْ أَبِيهِ، قَالَ قَالَ عَبْدُ اللَّهِ قَالَ رَسُولُ اللَّهِ صلى الله عليه وسلم " لاَ يَزَالُ هَذَا الأَمْرُ فِي قُرَيْشٍ مَا بَقِيَ مِنَ النَّاسِ اثْنَانِ " .
It has been narrated on the authority of 'Abdullah that the Messenger of Allah (ﷺ) said: The Caliphate will remain among the Quraish even if only two persons are left (on the earth),