First, some questions:
How long is ‘really long’ ? 10 seconds ? 1 minute ?
What kind of connection are you on ? (DSL, cable, dail up, carrier pigeon) ?
Do you lag in game after you connect ? If you haven’t already, turn on the lagometer with cg_lagometer 1. This thread: http://bani.anime.net/banimod/forums/viewtopic.php?t=3550 explains what it shows.
The most likely cause is some network problem between you and the server. Assuming it happens on all servers, then it is near your end. This could be caused by some problem with the actual hardware, or by congestion, or by viruses/trojans/P2P software on your computer.
If you have checked for viruses and trojans, and made sure that all P2P software is off, and you still have lag, then you need to troubleshoot your connection. You can use tracert and ping to do this. Just ask if you need help with that.