The team has won the SC05 bandwidth challenge (BWC) with an official bandwidth usage of 131 Gbps (Gigabits per second). This number is an average measured by the jury over a period of 15 minutes on 17 of the 22 10 Gbps waves, (FNAL/SLAC waves) being used by the team entry. The team is a collaboration of institutes including Caltech, University of Michigan, SLAC and FNAL, CERN, and Manchester. The bandwidth challenge does not only involves networks, but servers on the receiving and sending side that are connected via the wide area network. In the Caltech booth at SC05 4 server racks where placed especially for this purpose.
This year's entry involved real-world applications where real physics data was transferred based on the ROOT file format (ROOT files) which is a format frequently used by physicists. As such the bandwidth result, and lessons learned from it, will have some lasting benefits for transfer, and management of large amounts of (scientific) data.Several different protocols where used for transferring data, including: bbcp, xrootd (authored by Andy Hanushevsky from SLAC), and gridftp. Part of the data was transferred between remote SRM dcache deployments, and ones deployed at the show floor using gridftp. The extraordinary achieved bandwidth usage was made possible in part through the use of the FAST TCP protocol which was utilized by some of the above transfer protocols and is developed by Caltech professor Steven Low and his Netlab team.
The Bandwidth Challenge is an interesting benchmark of what is possible with high performance networking. It is also important for the LHC experiments (ATLAS, CMS, ALICE, LHCb) which will generate Petabytes to Exabytes of data per year, that will need to be analyzed by physicists around the world in multiple geographically dispersed computing centers (Tier-1's, Tier2's, Tier'3). In the near future most of ATLAS and CMS Tier-2's and even some Tier-3's will have 10 Gigabit connections and will want to be able to utilize them effectively. Activities like calibration and alignment of detectors for these experiments will rely upon being able to quickly move large amounts of data from CERN (the place where the LHC resides and Tier 0) to the sites responsible for that data's reduction. Part of how these huge data transfers take place is depicted in the LHC data hierarchy scheme, which will be augmented with many transfers between Tier-2's. The Bandwidth Challenge demonstrates what is possible with current networks when a focused effort is undertaken and will prepare us for enormous amounts of data that will generate increasingly more network traffic . The results of this challenge is part of the larger picture for LHC physics. We need to continue to make progress, especially "end-to-end". Efforts like this are just a step on the way to providing a robust high performance infrastructure for LHC science and other global data intensive science collaborations. More information on the context of this bandwidth challenge entry and networks in general can be found in the presentation Global Lambdas and Grids for Particle Physics by Professor Harvey Newman from Caltech and head of the team or the official press release.
The pictures on this page show measurements of individual and aggregate waves as measured by MonALISA. Within 3 hours an aggregate of 142.8 TB (Terabyte) was transferred (see figure), with sustained transfer rates ranging from 90 Gbps to 150 Gbps and a measured peak of 151 Gbps. During the whole day (24 hours) on which the bandwidth challenge took place approximately 475 TB where transferred. This number (475 TB) is lower than the team was capable of as they did not always have exclusive access to waves, outside the bandwidth challenge time slot. If you multiply the 3 hours where 142.8 TB was transferred, times 8 (to represent a whole day) you get approximately 1.1 PB (Petabyte). Transferring this amount of data in 24 hours, is equivalent to a transfer rate of 3.8 (DVD) movies per second, assuming an average size of 3.5 GB per movie.
(picture generated using MonALISA).
3 hour snapshot of total bandwidth usage, with an average throughput of more than 100 Gbps.The time on the horizontal axis in the picture is GMT (Greenwich Mean Time), which is Seattle time+8 hours. The official BWC measurement took place between 8 and 9 PM PST (equals 4 and 5 AM GMT)
 |
 |
 |
||
| Total bandwidth utilization in 2 hours surrounding the BWC. | Total size of data transferred in 2 hours surrounding the BWC (95.37TB). | Total size of data transferred in 24 hours on the day of the BWC (475 TB) | ||
 |
 |
 |
||
| Total bandwidth utilization in
3 hours surrounding the BWC, split by wave. (in and out going) |
Total size of data transferred in 2 hours surrounding the BWC, split by wave. | Total bandwidth utilization in 8 hours surrounding the BWC (230 TB transferred) | ||
|
(pictures generated using MonALISA). |
2 hour measurements of several individual waves.
In summary:
The team would like to thank all the companies, institutes and organizations who contributed to this success.
We are especially grateful to our many network partners: SCInet, LHCNet, Starlight, NLR, Internet2s Abilene and HOPI, ESnet, UltraScience Net, MiLR, FLR, CENIC, Pacific Wave, UKLight, TeraGrid, Gloriad, AMPATH, RNP, ANSP, CANARIE and JGN2;
our partner projects: US CMS, US ATLAS, D0, CDF, BaBar, US LHCNet, UltraLight, LambdaStation, Terapaths, PPDG, GriPhyN/iVDGL, LHCNet, StorCloud, SLAC IEPM, ICFA/SCIC and Open Science Grid,
our supporting agencies: DOE and NSF,
for the generosity of our vendor supporters, especially Cisco Systems, Neterion, HP, IBM, and many others, who have made this possible,
MiLR for the work that was done to loan and connect two fibers from NLR to Starlight
And the Hudson Bay Fan Company...