You are forced to sit there until the whole list is downloaded and refreshed, only then you can change to Favorites.
This is extremely annoying for someone who only wants use the Favorites.
It almost makes the Favorites button pointless for pub players; naturally someone who only plays pubs would add all pubs of one type to favorites, e.g. all "EU - Objective"s. If he is forced to download the whole list first anyways, he can just sort the general list by name and gets the same result, including an unnecessary waiting time either way.
On another note, high level players will probably want to play on “Min level 10” servers, but are auto-matched into the unleveled servers, too. So if you have all “min level 10” servers in the favorites to avoid playing with low levels, you run into the same annoyance.