There are many factors involved with buffering, but for now I will try to explain about speed in very simple terms.
Your upload speed is just as important if not more so than your download speed, streaming is a two way conversation between your device and the source. The device sends out a request for data, the source will then send that data in packets. Once received your device will then ask for the next batch and the cycle continues.
Your upload speed is usually half if not much less of what your download speed is. When your device makes a request it is uploading, in the meantime your device is using the data that it has received then waits for the next batch of packets (so it buffers or studders).
Another thing to take into consideration: Any connection can only be as fast as it's slowest component. If you have 100mbps+ download speed, but the source (server) only has 10mbps, that's as fast as it's going to get. This goes for any device whether it's your phone, computer, or #smartyourtv.