On June 25, 2004 we once again completed another Internet2 land speed record with multiple TCP Reno streams and transferred
859GB of data in ~17minutes (1,037s) from Geneva to Pasadena on a distance of 15,766 Km using Microsoft Windows for 64 bit 
with jumbo frame packets (9000bytes).
 
Intercontinental 10GE Network Setup:

Results:
8 TCP streams average throughput with NTttcp: 
Transferred 859052.821760MBytes of data in 1037s,  at rate of 6.63Gbit/s between Geneva and Pasadena.

Monitoring of the 10GE port at Los Angeles taken from the Abilene weathermap:

Here is the real-time Traffic Statistics in downloadable PDF file format.

Abilene backbone load during the transfer:

Length of the path:

(Distance measured with virtual GPS)

Average rate: 6.63 Gbps

Previous IPv4 Multi-Stream record:   77,699,310,000,000,000 meters-bits/sec 
New IPv4 Multi-Stream record submitted:  104,528,580,000,000,000 meters-bits/sec  (an increase of ~35%)

NTttcp test tool command line and results:

Sender Station : ntttcps -m 8,1,137.164.30.238 -a 8 -l 5376000 -n 20000
Receiver Station: ntttcpr -m 8,1,137.164.30.238 -a 16 -l 5376000 -rb 372520960 -t 1000 -n 20000

Thread Realtime(s) Throughput(KB/s) Throughput(Mbit/s)
====== =========== ================ ==================
   0    1037.090        103524.377            828.195
   1    1037.070        103599.008            828.792
   2    1037.020        103495.076            827.961
   3    1037.010        103646.475            829.172
   4    1037.200        103513.398            828.107
   5    1037.180        103489.667            827.917
   6    1037.180        103525.760            828.206
   7    1037.130        103520.341            828.163

Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ========================

 859052.821760    1037.200         8956.852          6626.513
 

Total Buffers Throughput(Buffers/s) Pkts(sent/intr) Intr(count/s) Cycles/Byte
============= ===================== =============== ============= ===========

 159807.922          154.076               4           19150.83      1.9

Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ==========

 12456414         95910126                11            0      26.38

Complete NTttcp send and receiver logs can be viewed from here.

End Host Configuration and Settings:
 - Operating systems:
    • Windows Server 2003 64-Bit Extended Systems Edition were used in the AMD64 sender machine 
       and IA64 receiver machine.
    • Win2K3 IA64 is a released product and available for download at no cost to members of MSDN at 
       the principal download site.  
    • Win2K3 AMD64 currently is in final beta and available for download at no cost to members of MSDN.   
       Free download of Win2K3 AMD64 (time limited and will expire in 360 days) software is also available at: 
       http://www.microsoft.com/windowsserver2003/64bit/extended/trial/default.mspx, upon agreeing to comply 
       with the Microsoft's EULA for Windows Server 2003 for 64-Bit Extended Systems (Beta)
    • To comply with the rules of the Internet2 Land Speed Record Microsoft has provided a written permission 
       to lift "RESTRICTIONS ON USE" stated in above Microsoft's EULA. It allows testing of the beta software till 
       the product ships (which should be within a year, probably much sooner).

- S2io 10GE NICs were used at both sender and receiver stations.
• Driver Version: 1.8.6.0
• Ethernet payload size = 9000
• Checksum offload = On
• Offload TCP Segmentation = On 
With Large Send Offload (LSO) enabled on the S2io NIC sender CPU utilization was reduced by 10~15%.

- Application/Tool : NTttcp version 2.2.   
• RTT was 264ms and a large TCP window 370MB was used in the NTttcp command lines.
• IP protocol : IPv4
• Transport protocol : TCP Reno

The NTttcp binaries used in the test are:
ntttcpr.exe.ia64 for IA64, which receives TCP/IP data, and 
ntttcps.exe.amd64 for AMD64 , which sends TCP/IP data.  

First 40 captured packets generated from NTttcp via NetMon in libpcap (tcpdump, Ethereal, etc) format:
Newisys Opteron sender (40 packets, ~476KB)
Itanium2 receiver  (40 packets, ~310KB)

Or in ascii text format:
Newisys Opteron sender  (40 packets, ~2,238KB)
Itanium2 receiver (40 packets, ~1,480KB)

End-to-End Pings with large packets and Traceroute results from both sender/receiver stations:

Geneva CERN Opteron Sender: 
===========================
c:\ntttcp>ping -n 10 -l 8972 -f 137.164.30.238

Pinging 137.164.30.238 with 8972 bytes of data:

Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116
Reply from 137.164.30.238: bytes=8972 time=264ms TTL=116

Ping statistics for 137.164.30.238:
Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 264ms, Maximum = 264ms, Average = 264ms c:\ntttcp>

c:\ntttcp>tracert 137.164.30.238

Tracing route to mach13-caltech.cenic.net [137.164.30.238]
over a maximum of 30 hops:

1 125 ms 125 ms 125 ms chi-vlan595-143.cern.ch [192.91.238.143]
2 125 ms 125 ms 131 ms abilene-tge.cern.ch [192.91.236.197]
3 146 ms 146 ms 146 ms nycmng-chinng.abilene.ucaid.edu [198.32.8.83]
4 150 ms 150 ms 150 ms washng-nycmng.abilene.ucaid.edu [198.32.8.85]
5 153 ms 153 ms 153 ms atlang-washng.abilene.ucaid.edu [198.32.8.65]
6 170 ms 191 ms 170 ms hstnng-atlang.abilene.ucaid.edu [198.32.8.33]
7 170 ms 170 ms 170 ms kscyng-hstnng.abilene.ucaid.edu [198.32.8.93]
8 180 ms 219 ms 180 ms dnvrng-kscyng.abilene.ucaid.edu [198.32.8.13]
9 206 ms 206 ms 206 ms sttlng-dnvrng.abilene.ucaid.edu [198.32.8.49]
10 214 ms 214 ms 214 ms snvang-sttlng.abilene.ucaid.edu [198.32.8.10]
11 263 ms 263 ms 263 ms losang-snvang.abilene.ucaid.edu [198.32.8.94]
12 263 ms 263 ms 263 ms hpr-lax-gsr1--abilene-LA-10ge.cenic.net [137.164.25.2]
13 263 ms 263 ms 263 ms mach13-caltech.cenic.net [137.164.30.238]

Trace complete.

Pasadena Caltech Itanium2 Receiver:
===================================
c:\ntttcp>ping -n 10 -l 8972 -f 192.91.238.132 (255.255.255.192)

Pinging 192.91.238.132 with 8972 bytes of data:

Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116
Reply from 192.91.238.132: bytes=8972 time=264ms TTL=116

Ping statistics for 192.91.238.132:
Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 264ms, Maximum = 264ms, Average = 264ms

c:\ntttcp>tracert 192.91.238.132

Tracing route to CERN-NEWISYS [192.91.238.132]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms lax-hpr--caltech.cenic.net [137.164.30.225]
2 <1 ms <1 ms <1 ms abilene-LA--hpr-lax-gsr1-10ge.cenic.net [137.164.25.3]
3 16 ms 19 ms 11 ms snvang-losang.abilene.ucaid.edu [198.32.8.95]
4 27 ms 27 ms 27 ms sttlng-snvang.abilene.ucaid.edu [198.32.8.9]
5 50 ms 44 ms 44 ms dnvrng-sttlng.abilene.ucaid.edu [198.32.8.50]
6 54 ms 54 ms 54 ms kscyng-dnvrng.abilene.ucaid.edu [198.32.8.14]
7 61 ms 55 ms 55 ms hstnng-kscyng.abilene.ucaid.edu [198.32.8.92]
8 75 ms 75 ms 75 ms atlang-hstnng.abilene.ucaid.edu [198.32.8.34]
9 101 ms 90 ms 90 ms washng-atlang.abilene.ucaid.edu [198.32.8.66]
10 141 ms 149 ms 139 ms nycmng-washng.abilene.ucaid.edu [198.32.8.84]
11 138 ms 149 ms 138 ms chinng-nycmng.abilene.ucaid.edu [198.32.8.82]
12 138 ms 138 ms 138 ms r04chi-v-187.caltech.datatag.org [192.91.236.198]
13 263 ms 263 ms 263 ms CERN-NEWISYS [192.91.238.132]

Trace complete.

Major Sponsors: