For Submission to the "Internet2 LSR" for the 
IPv4 Multiple-Stream Class Land Speed Record: 7.09Gbps 
from Geneva, Switzerland to Pasadena, USA


On May 6, 2004 during WinHEC 2004 conference we achieved average 7.09Gbit/s rate with multiple TCP Reno streams 
and transferred 860GB of data in ~16minutes (970s) from Geneva to Pasadena using Microsoft Windows for 64 bit 
with Jumbo frames (9000bytes)
 
Intercontinental 10GE Network Setup:



Results:
8 TCP streams average throughput with NTttcp: 7.09Gbit/s between Geneva and Pasadena for 970s.

Monitoring of the 10GE port at Los Angeles taken from the Abilene weathermap:

 

Here is the saved PDF file of Traffic Statistics.

Length of the path:

(Distance measured with virtual GPS)
Geneva - Chicago (LHCnet/Datatag): 7067 km
Chicago - Indianapolis (Abilene): 263 km
Indianapolis - Kansas City (Abilene) 727 km
Kansas City - Sunnyvale (Abilene): 2403 km
Sunnyvale - Pasadena (Abilene): 499 km

Total: 10959 km (~6810miles)
Average speed: 7.09 Gbps

Previous IPv4 Multi-Stream record:   68,431,250,000,000,000 meters-bits/sec
New IPv4 Multi-Stream record submitted:  77,699,310,000,000,000 meters-bits/sec  (an increase of ~14%)

NTttcp test tool command line and results:

Sender Station : ntttcps -m 8,1,137.164.30.245 -a 6 -l 5376000 -sb 268800000 -n 20000
Receiver Station: ntttcpr -m 8,1,137.164.30.245 -a 16 -l 5376000 -rb 268800000 -t 1000 -n 20000

Thread Realtime(s) Throughput(KB/s) Throughput(Mbit/s)
====== =========== ================ ==================
   0     970.625     110746.292       885.970
   1     970.656     110764.910       886.119
   2     970.625     110768.447       886.148
   3     970.640     110766.735       886.134
   4     970.547     110782.888       886.263
   5     970.640     110766.735       886.134
   6     970.640     110766.735       886.134
   7     970.656     110764.910       886.119
 

Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ========================

 860100.864000 	970.656 	26873.064                7089.021
 

Total Buffers Throughput(Buffers/s) Pkts(sent/intr) Intr(count/s) Cycles/Byte
============= ===================== =============== ============= ===========

 159993.512         164.830                  1          38880.72      1.3
 

Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ==========

 32006059 	27898468               185              0        13.01

Complete NTttcp send and receiver logs can be viewed from here.

End Host Configuration and Settings:
 - Operating systems:
    • Windows Server 2003 64-Bit Extended Systems Edition were used in the AMD64 sender machine 
       and IA64 receiver machine.
    • Win2K3 IA64 is a released product and can be downloaded from MSDN subscription.  
    • Win2K3 AMD64 currently is in final beta and can be downloaded from MSDN subscription.   
       Free download of Win2K3 AMD64 (time limited and will expire in 360 days) software is also available at: 
       http://www.microsoft.com/windowsserver2003/64bit/extended/trial/default.mspx, upon agreeing to comply 
       with the Microsoft's EULA for Windows Server 2003 for 64-Bit Extended Systems (Beta)
    • To comply with the rules of the Internet2 Land Speed Record Microsoft has provided a written permission 
       to lift "RESTRICTIONS ON USE" stated in above Microsoft's EULA. It allows testing of the beta software till 
       the product ships (which should be within a year, probably much sooner).
 - Sender station at CERN:  Newisys 4300 Quad AMD Opteron Enterprise Server with AMD-8131 chipsets. 
 - Receiver station at Caltech:  Intel Quad Itanium2 SR870BN4 Server with Intel E8870 chipset.
 - S2io 10GE NIC was used at both sender and receiver stations.
 - Network performance was obtained using PCI-X clock frequencies 133-MHz in 64-bit mode.     
 - Application is NTttcp test tool (part of Windows 2000 DDK), with the Jumbo Frames (9000).          
   • NTttcp tool is a ttcp variant optimized for Microsoft Windows OS.  NTttcp can be found on the latest 
     Windows 2000 Driver Development Kit (DDK) CD, available from Microsoft.  It was also distributed as part 
     of the Developer Conference that took place in Nov 2003 and at May 2004 WinHEC.  NTttcp can help measure 
     network driver performance and throughput on different network topologies and hardware setups.  It also 
     provides the customer with a multi-threaded, asynchronous performance benchmark for measuring 
     achievable data transfer rate on an existing network setup.  
   • The NTttcp binaries used in the test are:
      ntttcpr.exe.ia64 for IA64, which receives TCP/IP data, and 
      ntttcps.exe.amd64 for AMD64 , which sends TCP/IP data.  

NTttcp packet capture files from NetMon in MS Network Monitor format:
From Itanium2 receiver machine  (4,196 packets, ~26MB)
From Opteron sender machine (12,691 packets, ~81MB)

First 50 packets in libpcap (tcpdump, Ethereal, etc) format extracted from .cap files:
ia64-receive.tcpdump  (50 packets, ~275KB)
amd64-send.tcpdump  (50 packets, ~319KB)

Or in ascii text format:
ia64-receive.log  (50 packets, ~1.3MB)
amd64-send.log  (50 packets, ~1.5MB)

End Systems Registry Configuration:    
Complete Sender NIC Registry Settings
Complete Receiver NIC Registry Settings

S2io NIC Driver Version and Settings: 
Version: 1.7.0.16
Ethernet payload size = 9000
Checksum offload = On
Offload TCP Segmentation = On 

Traceroute from sender station:

c:\ntttcp>tracert 137.164.30.24

Tracing route to mach20-caltech.cenic.net [137.164.30.245] over a maximum of 30 hops:

 1 232 ms 116 ms 116 ms r04chi-v-570.caltech.datatag.org [192.91.239.56]
 2 116 ms 116 ms 116 ms abilene-tge.cern.ch [192.91.236.197]
 3 126 ms 125 ms 120 ms iplsng-chinng.abilene.ucaid.edu [198.32.8.77]
 4 129 ms 129 ms 129 ms kscyng-iplsng.abilene.ucaid.edu [198.32.8.81]
 5 140 ms 140 ms 140 ms dnvrng-kscyng.abilene.ucaid.edu [198.32.8.13]
 6 164 ms 164 ms 164 ms snvang-dnvrng.abilene.ucaid.edu [198.32.8.1]
 7 172 ms 187 ms 180 ms losang-snvang.abilene.ucaid.edu [198.32.8.94]
 8 172 ms 172 ms 172 ms hpr-lax-gsr1--abilene-LA-10ge.cenic.net [137.164.25.2]
 9 173 ms 172 ms 172 ms mach20-caltech.cenic.net [137.164.30.245]

Trace complete.

End-to-End Pings:

c:\ntttcp>ping 137.164.30.245 -n 10

Pinging 137.164.30.245 with 32 bytes of data:

Reply from 137.164.30.245: bytes=32 time=173ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120
Reply from 137.164.30.245: bytes=32 time=172ms TTL=120

Ping statistics for 137.164.30.245:
Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 172ms, Maximum = 173ms, Average = 172ms

Major Sponsors: