A paper from Microsoft on data center traffic patterns.

It measures traffic from socket level. From the data, the following are found:

  • In general, data center traffic is more regular than ISP networks, due to tighter coupling between different components
  • Because data center applications prefer placing jobs with high traffic exchange close to each other for higher bandwidth, there is a work-seeks-bandwidth pattern, i.e. high intra-rack or intra-server traffic
  • There are frequent one-to-all or all-to-one traffic, correspond to the map and reduce operation. This is a scatter-gather pattern.
  • The traffic between servers are heavy-tailed. Servers within the same rack are more likely to exchange more traffic
  • Median number of correspondance for a server is: 2 other intra-rack server, and 4 servers outside the rack
  • If define congestion as 70% of link speed, among the 150 inter-switch links, 86% of links observe congestion lasting more than 10 sec in a day and 15% links observe congestion lasting more than 100 sec in a day. Most congestion is short-lived
  • If a job sends traffic over a congested link, they are 110% more likely to fail to read the input
  • Reduce jobs are responsible for a fair amount of traffic. Extract jobs in Dryad as well, which converts raw data into input stream
  • Defining a flow as five-tuple flow, most flows are short-lived (80% flows are less than 10s), and very few are long running (less than 0.1% last longer than 200s)
  • Traffic changes quickly
  • Several times a day, all used links run close to capacity
  • TCP incast is not observed. Number of contending flows are small as well. Reasons could be, (1) the number of open connections are limited, (2) local communications such as intra-server or intra-rack are preferred.
  • A potential problem: Due to small BDP, window of TCP are small. A few packet drop will easily make TCP timeout.

Bibliographic data

@inproceedings{
   title = "The Nature of Datacenter Traffic: Measurements & Analysis",
   author = "Srikanth Kandula and Sudipta Sengupta, Albert Greenberg, Parveen Patel, Ronnie Chaiken",
   booktitle = "Proc. IMC'09",
   month = "November 4-6",
   year = "2009",
   address = "Chicago, Illinois, USA",
}