It does matters but not that much. Your mining rig(s) uses internet just to sync your progress with the pool and the blockchain. It happens frequently but if you have a small setup with few GPUs then it barely uses any internet.
Also, keep in mind that it’s not the speed that matters, it’s connection quality. Fluctuating internet speeds and connection can result in less profits as your rig won’t be able to send your mined shares to the pool or Blockchain itself in case of Solo mining on time.
Even 2G internet should work without any hiccups on a 4 GPU mining rig. Depends on connection quality.
Poor internet connection often results in invalid shares so that should also be noted.