Kb64

Emulab FAQ: Troubleshooting: I wrote a small TCP application to test the bandwidth of a link/LAN, but I do not observe the bandwidth that I asked for.

Short answer: TCP needs large send and receive socket buffers in order for its throughput to approach the capacity of long fat networks (LFN) i.e. link/LANs with a large bandwidth-delay product (BDP). Use UDP instead, if your intention is just to test the bandwidth. If you need to tune the throughput of your TCP application, refer to Enabling High Performance Data Transfers.

Long Answer: In order to observe the bandwidth that you specify, it is necessary to keep the data pipe between the sender and the receiver full. For a reliable window based protocol such as TCP, the window size represents the number of unacknowledged bytes. TCP needs to keep the unacknowledged bytes around until the acks for them are received. These bytes are retained in socket buffers. On a link/LAN with bandwidth B and round-trip-time (RTT) D, the sender TCP needs to be able to transmit B times D bytes before expecting any acknowledgement, if the data pipe has to be kept full. The effective sending window is dependent on receiver advertised window besides other things. It is necessary to have sender and receiver socket buffers at least as high as BxD. If you only care about optimal end-to-end TCP throughput, then the socket buffers need to be BxD where B is the bandwidth of the bottleneck link and D is the end-to-end RTT. Refer to the following TCP mini-tutorial or a basic TCP/IP book for the gory details.