Problem Statement for High Performance Wide Area Networks
ZTE Corporation
- BDP:
- Bandwidth Delay Product
- DC:
- Data Center
- DCI:
- Data Centers Interconnection
- HPC:
- High Performance Computing
- WAN:
- Wide Area Networks
- PFC:
- Priority Flow Control
- ECN:
- Explicit Congestion Notification
- ECMP:
- Equal-Cost Multipath
- RTT:
- Round-Trip Time
- TCP:
- Transfer Control Protocol
- RDMA:
- Remote Direct Memory Access
- QUIC:
- Quick UDP Internet Connections
- FCT:
- Flow Completion Time
Technical Goals for HP-WANs
The services need to be provided in HP-WANs mainly focus on massive
data with timely transmission while multiple services may co-exist over
long-distance WANs as described below.
- Massive data transmission, high-volume data with high-speed transfer,
e.g. the data speed of a flow could be at 2Gbps~1Tbps.
- Requested completion time, the data transmission should be completed
within a requested completion time, e.g. the completion time could be
minutes~milliseconds.
- Scheduled transmission, traffic patterns could be scheduled by the
sender, e.g. data volume, start time, finish time, service type.
- Long-distance transmission over non-dedicated WANs, with multiple hops
and domains, long RTT latency, routing changes, network congestion,
packet loss, and link quality fluctuations, e.g. the distance between
two sites or DCs could be more than 100km or 1000km.
- Multiple services are co-existed with concurrent flows.
It is required to achieve high-speed data transmission within
a completion time. Moreover, it is also crucial to maximize bandwidth
utilization while ensuring fairness among multiple services. This
document outlines the technical goals for HP-WANs as described below.
- Completion time: achieve the target job completion time within seconds
to minutes, while meeting FCT requirements for all incoming traffic flows.
- High throughput: ensuring the high-speed data transmission within
a requested completion time for a flow, which could be impacted by
the bandwidth, convergence speed, start time and RTT.
- Efficient use of capacity: efficiently using available network
capacity with fairness to maximize data transfer rates and minimize
the completion time for multiple flows.
- Efficient transmission of concurrent multi-flows: ensuring fair sharing of
link resources among multiple concurrent flows, avoiding slow-flow tailing and
FCT jitter caused by competition of multi-flows.
Problem Statement
The specific requirements of HP-WANs may encompass a wide range of
aspects. These include transport-related technologies such as proxy,
flow control, QoS negotiation, congestion control, admission control
and traffic scheduling. Additionally, they also involve routing-related
technologies like traffic engineering, resource scheduling, and load
balancing.
Existing network technologies face numerous challenges and fall short
of meeting performance requirements. This document highlights the key
issues associated with HP-WANs in the following sub-sections.
Poor Convergence Speed
The traditional congestion control mechanisms perform blind transmission
by controlling the size of the congestion window with rate adjusting by
detection of overloaded links. WAN is a black box to provide unpredictable
behaviors for high-speed transmission due to the issues such as multiple
hops and domains, long Round-Trip Time (RTT), routing changes, network
congestion, packet loss, and link quality fluctuations. The BDP (Bandwidth
Delay Product) which represents the maximum amount of data that can be in
transit on the network at any given time is variable over WANs, so the
inflight data is difficult to predict for host-based congestion control
algorithms. It will lead to the poor convergence speed that the host always
takes significantly long time to identify the optimal sending rate comparing
to the requested completion time.
For example, it will use the slow start and blind detection with
unawareness of network capability leading to long convergence time
such as Cubic (e.g.over 50s), BBR (e.g.over 30s) and BBRv2 (e.g.30~50s).
BBR divides the entire process into four stages, Startup, Drain,
ProbeBW and ProbeRTT. The probe cycle of ProbeRTT state is long,
e.g. 10s. The convergence time will be multiple probe cycle which
will impact the completion time at seconds level. There is a significant
transmission capacity gaps between the appropriate sending rate and the
available network capacity. The transport protocols should signal and
collaborate with the network to negotiate the rate for the host to send
traffic.
Unscheduled Traffic
The host sending large unscheduled traffic without collaboration will
lead to the instantaneous congestion in WANs. For multiple high-speed
flows, the random arrival and departure of cross-traffic without scheduling
creates significant fluctuations for available capacity in WANs. The network
infrastructure may struggle to handle high-volume data transfers efficiently
if applications do not proactively schedule the traffic. Without awareness
of the traffic patterns, the network risks unscheduled resource allocation,
leading to low bottleneck bandwidth utilization, reduced overall throughput,
and uncontrolled completion time.
For example, for HPC applications, a large amount of data will be transmitted,
e.g. the data volumes of a single flow may be from 10G to 1TB, the host sends
the unscheduled large traffic leading to the instantaneous congestion, packet
loss, and queuing delay within network devices in WANs, resulting in low throughput.
Considering the multiple services with various types of flows, the optimal
bandwidth and transmission time may be different and the traffic is random to
join and leave without to be scheduled to multiple paths and fine-grained
network resources, which can not achieve the timely transmission. The resource
of WANs should be scheduled at the elements along the path to provide predictable
capability for high-speed transmission.
Long Feedback Loop
The congestion algorithms are implemented by controlling the size
of the congestion window and adjusting the sending rates upon the network
status feedback. It will delay the network feedback due to the long-distance
transmission delays and large RTT, resulting in the inability to adjust
the transmission rate in a timely manner. It will be challenging for congestion
control over WANs for controlling the total amount of data entering the
network to maintain the traffic at an acceptable level, leading to RTT
fluctuation due to long queues and large buffer at network devices
with high-speed transmission upon the long network state feedback loop.
Especially when multiple flows targeting an aggregating node, the maximum
value is exceeding devices buffer capacity.
For example, the loss-based congestion control algorithms, such as
Reno and CUBIC, depends on the congestion notification with packet loss.
Explicit Congestion Notification (ECN) can be used to achieve an
end-to-end congestion notification based on IP and transport layers.
When a congestion occurred, the network may signal congestion
by ECN markings or by dropping packets, and the receiver passes this
information back to the sender in transport-layer acknowledgements,
notifying the source to adjust the transmission rate. It will use the
slow start, requiring large buffer which is impacted by multiple hops
and long RTT latency over WANs.
And the congestion-based congestion control algorithms such as BBR,
depends on the measurement of congestion, it actively measures
bottleneck bandwidth (BtlBw) and round-trip propagation time (RTprop)
based on the model to calculate the BDP and then to adjust the
transmission rate to maximize throughput and minimize latency. But
BBR relies on real-time measurement of the parameters, and will
optimize the buffer overflow, but it is not significant under large
RTT, e.g. retransmission will increase when the buffer size is
less than two BDPs, thereby affecting the control precision of BBR
in long-distance networks.
Multi-flow Concurrent Transmission
An AI/HPC job may be decomposed into multiple tasks for parallel transmissions over a network.
The insufficient transmission throughput and blind competition among multiple flows will lead to
slow flow tailing and FCT transmission jitter.
For a single flow, traditional congestion control mechanisms implemented on hosts lack rate controls,
resulting in unbounded rate adjustments and the transmission rate exhibits a sawtooth-like
fluctuation. When this flow is transmitted concurrently with other flows, it causes
competing for bottleneck bandwidth, resulting in tail latency that drags down overall task throughput.
This will trigger queuing delays and congestion packet loss, creating slow flows and making the
completion time of a single flow uncontrollable.
For multiple flows within a job, the passive competition for bandwidth resources often leads
to a cyclical pattern of "peak overflows" (causing queuing delays) and "valley underflows"
(causing waiting delays), resulting in significant jitter and deviation in FCTs of multiple flows.
The FCT jitter significantly undermines job completion reliability and performance in
concurrent network environments.
Security Considerations
This document covers several of representative applications and
network scenarios that are expected to make use of HP-WAN
technologies. Each of the potential use cases does not raise
any security concerns or issues, but may have security
considerations from both the use-specific perspective and
the technology-specific perspective.
IANA Considerations
This document makes no requests for IANA action.
Acknowledgements
The authors would like to acknowledge Bin Tan, Guangping Huang, Yao Liu and
Zheng Zhang for their thorough review and very helpful comments.
References
Normative References