For Submission to the "Internet2 LSR" for the IPv4 Multiple-Stream Class 
Land Speed Record: 6.25Gbps from Los Angeles to Geneva, Switzerland


On February 22, 2004 we achieved average 6.25Gbit/s rate with multiple TCP Reno streams for more than 10 minutes 
from Los Angeles to Geneva using Microsoft Windows for 64 bit with Jumbo frames (9000bytes)
 
Network Setup:



Results:

9 TCP streams average throughput: 6.25Gbit/s between Los Angeles and Geneva for 10 minutes.

Monitoring of the  10GE port at Los Angeles taken from the Abilene weathermap:

 

Here is the PDF file of Traffic Statistics.

Sender Station Command Line: ntttcps -m 9,0,192.91.239.7 -a 6 -l 4300k -sb 256M -n 13000
Receiver Station Command Line: ntttcpr -m 9,1,192.91.239.7 -a 16 -l 4300k -rb 1024M -n 100000

Command Line Syntax Details:	
-m option specifies a mapping of thread information, including specifying the receiver system's IP address.
 In our test we started 9 session threads, bound to processor 0 for sender and 1 for receiver, and connecting 
 to host 192.91.239.7.
-a is specified to indicate asynchronous I/Os to the sender (6) or receiver process (16). 
-l is the length of data buffer used for the test.  TCP Segmentation Offload becomes more obvious when a larger data size is used.
-n is the number of buffers to transfer.  It changes the time it takes to complete a NTttcp test run.
-sb, -rb are the send and receive buffer sizes.  They modify the socket options SO_SNDBUF and SO_RCVBUF for the sender and receiver processes.


Sender Log:

Thread Realtime(s) Throughput(KB/s) Throughput(Mbit/s)
====== =========== ================ ==================

     0     638.700        87602.102            700.817
     1     638.730        89493.749            715.950
     2     638.720        87833.747            702.670 
     3     638.710        88896.783            711.174
     4     638.690        77220.942            617.768 
     5     638.720        88888.497            711.108
     6     638.740        87741.381            701.931 
     7     638.690        89623.448            716.988
     8     638.740        83308.814            666.471 
 
Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ======================== 
   498587.545600     638.740          26734.212                 6244.876

Total Buffers Throughput(Buffers/s) Pkts(sent/intr) Intr(count/s) Cycles/Byte 
============= ===================== =============== ============= ===========
   113237.302               177.282               1     111681.74         1.3 
 
Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ========== 
    18649794          9197520                52            1      17.03 

Length of the path:
- Los Angeles - Sunnyvale (Abilene): 489 km
- Sunnyvale - Kansas City (Abilene): 2403 km
- Kansas City - Indianapolis (Abilene) 727 km
- Indianapolis - Chicago (Abilene): 263 km
- Chicago - Geneva (LHCnet/Datatag): 7067 km
Total: 10949 km

Average speed: 6.25 Gbps

Record submitted (IPV4 Multiple-Stream class):  68,431,250,000,000,000 meters-bits/sec 

End System Configuration and Settings:
 - OS: Windows Server 2003 64-Bit Edition.
 - Two 4U Intel Quad Itanium2 SR870BN4 Servers were setup at Los Angeles and Geneva in distance of 10949 km.      
 - Intel server chipset: Intel® E8870 chipset.  
 - A s2io 10GE NIC was used at sender station and an Intel 10GE LR NIC at receiver station.
 - Performance was tested with PCI clock frequencies 133-MHz in 64-bit mode.     
 - Appliction is NTttcp test tool (part of Windows 2000 DDK), with the Jumbo Frames (9000).          

NTttcp tool is a ttcp variant optimized for Microsoft Windows.  NTttcp can be found on the latest 
Windows 2000 Driver Development Kit (DDK) CD, available from Microsoft.  It was also distributed as part 
of the Developer Conference that took place in Nov 2003 and will be part of the tools suite distributed 
at WinHEC this May 2004.  It helps measure network driver performance and throughput on different network 
topologies and hardware setups.  It also provides the customer with a multi-threaded, asynchronous 
performance benchmark for measuring achievable data transfer rate on an existing network setup.  

The binaries used in the test are:
ntttcpr.exe , which receives TCP/IP data, and 
ntttcps.exe , which sends TCP/IP data.  

NTttcp tcpdump file:
Microsoft Network Monitor format  (10,611 packets,  ~65MB)
libcap (tcpdump, Ethereal) format ((10,611 packets, ~65MB)
First 100 packets in libcap tcpdump format (647KB)
First 100 packets tcpdump in ascii text (2.6MB)


End Systems Registry Configuration:    
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters    
Tcp123Opts=1 
TcpWindowSize=256000000

NIC Driver Settings: 
Maximum Frame Size = 9000
Receive Descriptors = 16384
Transmit Descriptors = 16384
Offload TCP Segmentation = On

Traceroute:
c:\ntttcp>tracert 192.91.239.7

Tracing route to IIS_IA64 [192.91.239.7] over a maximum of 30 hops:

   1 <1 ms <1 ms <1 ms lax-hpr--caltech.cenic.net   [137.164.30.225]
   2 <1 ms <1 ms <1 ms abilene-LA--hpr-lax-gsr1-10ge.cenic.net   [137.16.25.3]
   3 7 ms 7 ms 13 ms snvang-losang.abilene.ucaid.edu   [198.32.8.95]
   4 32 ms 32 ms 33 ms dnvrng-snvang.abilene.ucaid.edu   [198.32.8.2]
   5 43 ms 43 ms 49 ms kscyng-dnvrng.abilene.ucaid.edu   [198.32.8.14]
   6 61 ms 56 ms 57 ms iplsng-kscyng.abilene.ucaid.edu   [198.32.8.80]
   7 56 ms 56 ms 56 ms chinng-iplsng.abilene.ucaid.edu   [198.32.8.76]
   8 56 ms 56 ms 56 ms r04chi-v-187.caltech.datatag.org   [192.91.236.19]
   9 172 ms 172 ms 172 ms IIS_IA64 [192.91.239.7]

Trace complete.

End-to-End Pings:
c:\ntttcp>ping 192.91.239.7 -n 20

Pinging 192.91.239.7 with 32 bytes of data:

Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120
Reply from 192.91.239.7: bytes=32 time=172ms TTL=120

Ping statistics for 192.91.239.7:
   Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
   Approximate round trip times in milli-seconds:
   Minimum = 172ms, Maximum = 172ms, Average = 172ms

Major Sponsors: