A gap in the green lagometer graph means that during that interval, your client was not processing the updates from the server. This is usually caused by another program or driver taking all of your CPU’s resources during that time. I’ve also seen it happen when overclocking my videocard too much; the card would basically “hang” for short periods, causing the disruption. It is not something that’s caused by the connection to the server (or you would have seen a red spike).
Try closing all programs running in the background, checking your overclocking (if applicable), entering \pb_system 1 when not connected to the server (if not done already), using different/newer drivers, stuff like that.
HTH.