Flex Ethernet Technology and Application in 5G Mobile Transport Network

Research Institute of Telecommunications Transmission(RITT),China Academy of Information Communications Technology(CAICT),MIIT,Beijing 100191,China.

Abstract:With the advent of 5G era,the rise of cloud services,virtual reality/virtual reality(AR/VR),vehicle networking and other technologies has put forward new requirements for the bandwidth and delay of the bearer network.Traditional Ethernet technology cannot meet the new requirements very well.Flex Ethernet (FlexE) technology has emerged as the times require.This paper introduces the background,standardization process,functional principle,application mode and technical advantages of FlexE technology,and finally analyses its application prospects and shortcomings in 5G mobile transport network.

Keywords:Flex Ethernet (FlexE);5G;mobile transport network;network slicing

I.INTRODUCTION

The concept of Ethernet was proposed by Xerox in 1937.The original rate of Ethernet was only 2.94 Mbps.In 1982,DIX published the specification of Ethernet Version 2 (EV2) at a rate of 10 Mbps.With the development and innovation of technology,the transmission medium of Ethernet has evolved from coaxial cable to optical fiber,and the transmission rate has also evolved from 10/100/1000Mbps to 100/200/400 Gbps [1–5].Ethernet has been regarded as a low-cost scalable data networking solution in LAN environment.Until 2000,Metro Ethernet Forum (MEF) put forward the concept of telecommunication-level Ethernet,pointing out that telecommunication-level Ethernet has the characteristics of standardized services,scalability,reliability,quality of service assurance,telecommunication-level management.Telecommunication-level Ethernet also maintains the advantages of low cost,high efficiency and statistical reuse of Ethernet technology.Operators can use it to achieve multi-service access and end-toend service deployment.Figure1 shows development and application of Ethernet.With the advent of 5G era,the rise of cloud services,AR/VR,vehicle networking and other technologies has put forward new requirements for bandwidth,delay and other performance[6–11].Flex Ethernet (FlexE) emerged as the times require.

Figure1.Development and application of Ethernet.

II.FLEXE STANDARDIZATION PROCESS AND PRINCIPLE

FlexE technology is proposed by Optical Internetworking Forum (OIF).Once put forward,it has attracted wide attention.In addition to OIF,standards organizations such as ITU-T,Internet Engineering Task Force(IETF),Broad Band Forum(BBF)and China Communications Standards Association(CCSA) have also started FlexE-related standardization work.

Figure2.Structure of IEEE 802.3 and OIF FlexE.

2.1 Optical Internetworking Forum(OIF)

In 2015,OIF launched FlexE standard research and released FlexE IA version 1.0 in 2016[12].The standard defines the 100GE PHY FlexE function and is the industry’s first standard on FlexE.In this standard,the decoupling of MAC layer and PHY layer is realized by adding FlexE Shim layer on the basis of Ethernet layered structure defined in IEEE 802.3,and then the flexible rate adaptation function is realized,as shown in Figure2.

FlexE Client,FlexE Group and FlexE Shim are defined in this standard.FlexE Client corresponds to user interfaces at various rates of the network.FlexE Group consists of a series of Ethernet PHY.Each PHY is divided into 66B slots of 5Gbps rate.Hard pipeline isolation similar to TDM system can be achieved between slots.FlexE Client can be configured flexibly according to bandwidth requirements.The FlexE Client MAC rates supported by FlexE Groups are 10Gbps,40Gbps,and m × 25 Gbps.The FlexE Client MAC rates supported by FlexE Groups may support all,or only a subset of these FlexE Client rates,e.g.,m ×25 Gbps.The FlexE Client transmits data streams to FlexE Shim layer by 64B/66B encoding.FlexE’s flexible scheduling capability is mainly implemented in FlexE Shim layer.Shim layer realizes the rate adaptation between Client layer and PHY by inserting and deleting idle blocks.Then 66B blocks are distributed to different slots in Group.The allocation principle is controlled by the calendar.FlexE frame overhead/-multiframe overhead is inserted into the stream to realize the slot mapping.66B blocks are processed by Shim layer,including scrambling code,multi-channel allocation and so on,and then enter into FEC (optional) or PMA layer.The subsequent processing is the same as that defined in IEEE 802.3.In June 2018,OIF released FlexE IA version 2.0[13],which added support for 200GE/400GE PHY,maintained the frame overhead/multiframe overhead format compatible with FlexE IA version 1.0,as well as the insertion/deletion mechanism of idle code blocks for 100/200/400GE PHY rate adaptation,and increased support for time synchronization of IEEE 1588V2.The original management channel is extended to support transmission of synchronization information,Precision Time Protocol (PTP) or Synchronization Status Message (SSM).With the rise of 5G services,10GE access ring cannot meet the needs of 5G mobile backhaul network.Considering the bandwidth capacity,cost and reliability,50GE gray-light interface(10-40km)has become one of the mainstream choices in the construction of the next generation bearer network access ring.Therefore,at the fourth quarter meeting of the OIF in 2018,FlexE 2.1 Project was adopted to increase FlexE’s support for 50GBASE-R PHY.In September 2018,OIF released FlexE Neighbor Discovery IA version 1.0 for interaction capability and status information between local devices and remote devices,including Connectivity Discovery,Connectivity Verification,Remote Capability Discovery,Subgroup Integrity Verification and Deskew Capability Discovery.By extending the Link Layer Discovery Protocol (LLDP) defined in IEEE 802.1ab and interacting with the segment management channel in frame overhead,it is easy for the local control plane or SDN controller to obtain the capability information of remote devices,which is significant especially in the distributed controller scenario.

2.2 ITU Telecommunication Standardization Sector(ITU-T)

ITU-T has initiated the revision of G.709,G.798 and other standards to define the mapping mode of FlexE signals into Optical Transport Network (OTN).According to the definition in OIF FlexE IA,FlexE has three delivery modes on the transport network:Unaware,Aware and Termination.G.709 defines mapping modes for different schemas.In particular,for aware mode,100G instances in FlexE group (or subgroup) need to be compressed,filled and interleaved.After removing unconfigured and unavailable slots,Bitsynchronous Generic Mapping Procedure(BGMP)is used to map FlexE signal into OPUflex cell.In June 2018,ITU-T SG15 approved G.8023 which specifies both the functional components and the methodology that should be used in order to specify the Ethernet physical layer and Flex Ethernet interfaces.At the ITU-T SG15 plenary session held in October 2018,the project proposal put forward by China Mobile,China Academy of communications,Huawei,ZTE and Fenghuo was approved.SG15 officially launched the standardization research of g.mtn (interfaces for a metro transport).MTN includes two layers of network,namely MTN section layer and MTN path layer.MTN section layer reuses the flex frame structure and implementation logic of OIF to the greatest extent,so as to share the existing Ethernet and Flex technology and industrial chain as much as possible,especially from the perspective of chip implementation,reusing the flex frame structure and implementation logic to the greatest extent can simplify the chip design.MTN path layer takes client as basic unit,and uses S-XC(sequence cross connect)technology to form end-to-end hard channel between PE nodes and realize low delay forwarding[14].MTN path also supports MTN OAM and network protection functions.

2.3 Internet Engineering Task Force(IETF)

IETF is currently committed to the standardization of FlexE control plane.The requirements,framework,architecture and YANG model of FlexE technology are studied.The YANG data models of FlexE group and FlexE client are defined.Based on the YANG model application program,FlexE group and client can be configured and modified on FlexE-supporting equipments.Alerts and notifications from FlexE layer can be received.To combine FlexE technology with mature IP/MPLS technology,routing and signaling framework extension for FlexE links (FlexE group(PHY) and FlexE client (MAC)) is studied.The signaling protocols that need to be extended include Resource Reservation Protocol (RSVP),Link Management Protocol (LMP) and Path Computation Element Communication Protocol (PCEP).For example,RSVP protocol need to be extended to support resource reservation and establishment of FlexE group and client when a Label Switching Path (LSP) need to be established on FlexE-supporting devices.In order to establish FlexE group and client correctly,routing protocols(such as ISIS,OSPF,BGP-LS)also need to be extended to support FlexE capability claim and other functions.

2.4 Broad Band Forum(BBF)

Routing and Transport Work Area of BBF approved a new project in 2017,which defines solution architecture and equipment requirements for the FlexE technology.This innovative technology combines the ubiquity of Ethernet with the guaranteed consistency of optical network.The architecture will address the use of this technology in conventional IP/Multiprotocol Label Switching (MPLS) networks to realize 5G issues such as network slicing and service orchestration.

2.5 China Communications Standards Association(CCSA)

CCSA launched a research project FlexE and FlexO Technology Research in 2016.The application scenarios and principles of FlexE technology,the application of FlexE technology in transport network,the interconnection and interoperability between devices using FlexE technology and transmission equipment,the modification of IEEE Ethernet PCS layer by FlexE technology,and the location and role of FlexE overhead are discussed.The standard of technical requirements for Flexible Ethernet(FlexE)interface and link is formulated.

III.APPLICATION MODE AND CHARACTERISTICS OF FLEXE TECHNOLOGY

3.1 Application Modes of FlexE Technology

Figure3.Three application modes of FlexE.

According to the definition in OIF FlexE IA,FlexE technology can provide three application modes:bonding,channelization and sub-rate,as shown in Figure3.

Bonding:By bonding multiple PHYs,higher MAC rate can be achieved.If N PHYs of 50GE/100GE/200GE/400GE are bonded together,the MAC rate of N×50G/100G/200G/400G can be supported.Channelization:One or more PHYs are divided into multiple low-rate channels through FlexE Shim layer.Multiple low-rate services can share one or more PHYs’ bandwidth.Hard pipeline isolation similar to TDM services can be achieved between divided sub-channels.Such as carrying 10G,20G,45G and 25G four-way MAC data streams on a 100G PHY,or multiplexing 65G,25G and 10G MAC data streams on three 50G PHYs.Sub-rate:A single low-rate MAC data stream shares one or more paths of PHY,and achieves deceleration through a specially defined error control block.For example,only 10G MAC data stream is hosted on a 100G PHY.FlexE technology can decouple the MAC layer from the PHY layer and realize the flexible adaptation between the MAC layer rate and the PHY layer rate.High-speed MAC signals can be adapted to low-rate PHYs by bonding.Lowrate MAC can also be adapted to a high-speed PHY by sub-rate,and flexible mapping between multiple MACs and multiple PHYs can be realized by channelization.Hard isolation of pipelines can also be achieved between different MACs.

3.2 Advantages of FlexE Technology

FlexE technology has obvious advantages over traditional port bonding technology (such as LAG,MLG,etc.) and hard pipeline technology (such as SDH).Stricter bandwidth guarantee than LAG:Link Aggre-

Figure4.Architectural positioning of Link Aggregation sublayer.

gation(LAG)technology is based on the definition of IEEE 802.1AX[15].It can aggregate multiple physical ports to form a logical port to realize the load sharing of traffic at each member port according to the user-configured load sharing strategy.The architecture model of LAG is shown in Figure4.Link aggregation layer is inserted between LLC layer and MAC layer.Load sharing among member ports is usually based on the characteristic fields of Ethernet packet(such as MAC address,IP address,etc.).Therefore,HASH algorithm directly affects the effect of load sharing and bandwidth utilization.In a real network,due to the unbalanced load,it is difficult to achieve a uniform distribution of traffic on each member port,and the bandwidth utilization rate is also difficult to achieve the actual bandwidth of multiple ports.FlexE Shim layer is designed between RS layer and PCS layer.Traffic is multiplexed and de-multiplexed based on 66B blocks.It can transfer different traffic strictly on the allocated slots,and also can realize line-speed forwarding.The bandwidth utilization is very high.

More flexible rate adaptation than MLG:Multilink Gearbox (MLG) technology is also defined by OIF and released in 2015.MLG technology defines three MLG configurations:A 4×25G lane configuration is comprised of 20 MLG lanes,where two groups of eight MLG lanes can be configured to carry either four 10GBASE-R signals or a single 40GBASE-R signal,while the remaining 4 MLG lanes can carry two 10GBASE-R signals.An 8×25G lane configuration is comprised of 40 MLG lanes,where each group of eight MLG lanes can carry either four 10GBASE-R signals or a single 40GBASE-R signal.A 2×20G/1×40G lane configuration is comprised of 4 MLG lanes (similar to 40GBASE-R) to carry up to four 10GBASE-R signals.This function is similar to the channelization function of FlexE.FlexE technology is more flexible than MLG technology.MLG technology supports the bonging of one or two 100GBASE-R physical interfaces.Client rates must also be standard 10GBASE-R and 40GBASER signals.FlexE technology can theoretically bond 254 100G instances (100G instances can be either 100GBASE-R ports or 100G lanes in 200GBASE-R or 400GBASE-R).Client rate can be 10Gbps,40Gbps,and m × 25 Gbps.Higher bandwidth utilization than SDH:Synchronous Digital Hierarchy(SDH)transmits multi-rate signals through virtual containers,and realizes hard isolation between pipelines.However,due to the complex overhead system in SDH technology,bandwidth utilization is relatively low.For example,the STM-1 interface (155 Mbit/s) can carry 63 channels of E1 signal(2048 kbit/s),and the bandwidth utilization is about 0.83.FlexE IA also defines overhead frames for correct alignment,multiplexing and demultiplexing of channels.FlexE overhead uses 66B blocks for transmission,and overhead block should be inserted after every 1023×20 66B information blocks.Therefore,FlexE overhead occupies very little bandwidth and high rate utilization of effective signal bandwidth can be achieved.

IV.APPLICATION OF FLEXE TECHNOLOGY IN 5G MOBILE TRANSPORT NETWORK

Figure5.Application of FlexE in 5G mobile transport network.

The fifth generation mobile communication (5G)technology is dedicated to deal with the explosive growth of mobile data traffic,massive device connections,emerging new services and application scenarios in the future.This also puts forward new requirements for the bearer network.ITU-R defines three typical scenarios of 5G:Enhanced Mobile Broadband (eMBB),massive Machine Type of Communication (mMTC) and Ultra Reliable and Low Latency Communication(uRLLC).Among them,eMBB service mainly aims at the broadband applications such as Augmented Reality(AR)/Virtual Reality(VR)and Ultra-high definition video,requiring the bearer network to provide ultra-large bandwidth capability.mMTC service mainly aims at the application scenario of the Internet of Things(IoT),in which massive data acquisition and sensor devices need to access the network.It also requires the cooperation between base stations,which requires the bearer network to have high-precision synchronization,low-cost access,etc.URLLC services are mainly used in time-sensitive application scenarios such as Internet of Vehicles and Industrial Internet,which requires bearer network to have ultra-low delay forwarding and hard isolation capabilities.FlexE technology can meet some of the requirements very well by providing low-cost and ultralarge bandwidth logical ports,hard isolation and time synchronization function,as shown in Figure5.

Ultra-large bandwidth capacity requires the bearer network to provide high-speed physical ports,i.e.high-speed optical modules.However,the actual deployment of the network is often limited by the transmission distance and cost of high-speed optical modules.Taking 400G requirement as an example,the 400G Ethernet interface defined in IEEE 802.3 adopts PAM4 modulation format with 25G baud rate,and the transmission distance is 10km.The maximum transmission distance of mobile backhaul bearer network can reach 40-80km.It’s obvious that 400G PAM4 optical modules cannot meet the requirements.The transmission distance of more than 40 km can be achieved by 400G coherent light scheme,but the cost is too high.By using FlexE’s bonding function,four 100GBASE-R ports can be bonded into a 400G FlexE group to achieve ultra-large bandwidth transmission capacity at low cost.Hard isolation capacity requires that the bearing network can provide rigid pipelines similar to TDM.Traditional Ethernet interfaces provide a soft isolation based on statistical multiplexing.Peaks and valleys of multi-user traffic can be filled with each other to achieve higher bandwidth utilization,but it cannot achieve strict user isolation and bandwidth guarantee.When the network is congested,the high priority traffic can be guaranteed by the QoS configuration,but the forwarding delay cannot be controlled.FlexE technology can achieve hard bandwidth isolation between multiple users through slot allocation.In 5G backhaul bearer network,FlexE technology can slice physical links into a series of virtual links.Those virtual links can provide strictly hard isolation for services requiring exclusive network bandwidth resources,delay and reliability,such as Internet of Vehicles,Industrial Internet,and high-quality government-enterprise dedicated lines.As shown in Figure6,in order to verify the hard isolation characteristics of FlexE technology,we did the following experiments.Figure6 (a) shows the Flexe networking topology with 50Gbps Flexe link on the network side.We set up two clients and allocate bandwidth of 10Gbps and 40 Gbps respectively.The user side uses 10GE and 50GE ports respectively to configure the local switching service.Client 1 (10Gbps) corresponding service transmission traffic bandwidth is constant 10Gbps.Client 2 (40Gbps) corresponding service transmission traffic bandwidth gradually increased from 10Gbps to 50Gbps.Figure6 (b) shows the traffic reception of the two services.Whether the network is congested or not,service 1 can guarantee 10Gbps channel bandwidth.When the traffic of service 2 is greater than 40Gbps,client2 can only guarantee the channel bandwidth of 40Gbps,and other messages are discarded.Figure6 (c) compares the transmission delay of service 1 before and after network congestion.No matter whether service 2 is congested or not,the transmission experiment of service 1 is basically unchanged.We also replace the network side with 50GE Ethernet link,and use QoS Technology to ensure the bandwidth.Repeat the above experiments,as shown in Figure6 (d) - (f).It can be seen that although QoS technology can guarantee the bandwidth,when the network is congested,the transmission delay of the non congested services will increase obviously.This experiment proves that FlexE technology can provide TDM like rigid pipes for future networks.This bandwidth guarantee and delay guarantee technology is very important in many URLLC scenarios(such as 5G+smart grid,5G+industrial Internet,5G+smart healthcare,etc.).

It can be seen that FlexE technology can improve bandwidth capacity,reduce cost,guarantee delay and provide network hard slice in 5G mobile transport network.In OIF FlexE IA2.0,the transmission of synchronization information for IEEE 1588V2 is realized by extending management channels,which further meets the time synchronization requirements in bearer networks.

However,as a link technology,rather than a network technology,FlexE technology needs to be further expanded and perfected if it is to be fully applied in the bearer network.This paper makes a further analysis of the shortcomings of FlexE technology application in bearer network.

Figure6.Results of FlexE hard isolation.

Port Fault Handling:OIF FlexE IA2.0 specifies that when a port in a FlexE group fails,all clients’traffic on the group cannot be forwarded normally.Even if the client’s slot is independent of the fault port,the traffic cannot be forwarded normally.This obviously does not satisfy the requirement of robustness of bearer network.Starting from the reliability of the bearer network,when a physical port fails,the unrelated clients’traffic should be forwarded normally through fault isolation;the clients related to the fault port should generate a corresponding alarm to trig the switch of the protection mechanism of the client layer,such as Ethernet liner protection switching specified in ITU-T G.8031,or linear protection switching for MPLS transport profile specified in ITU-T G.8131,etc..When the fault is eliminated,the physical port is rejoined to the FlexE group.Because the whole group needs to do alignment operation,it may cause a small interruption to the clients which is independent of the fault port.If it is guaranteed within 50ms,it will not affect its application in the bearer network.

Online parameter modification:At present,FlexE clients need to be configured on FlexE group,and the services are related to FlexE clients.Due to the increase of traffic in the existing network,the reserved bandwidth may not meet the needs of customers.At this time,it is necessary to extend the equipment and modify the services.In traditional Ethernet network,bandwidth parameters can be directly modified if there is still bandwidth margin in the service bearing port;if the bandwidth of the port is saturated,it is necessary to consider to migrate the service to other port with higher rate.Similarly,in the bearer network based on FlexE technology,when the service bearer port still has idle slots,it can modify the clendar of FlexE clients online to expand the bandwidth.This operation is regulated by OIF FlexE IA.In the case of insufficient port bandwidth,more physical ports can be bonded into FlexE group to achieve bandwidth expansion.At present,OIF FlexE IA has not made clear specifications for such operations.In principle,we need to delete all the servives and FlexE clients on the FlexE group,re-establish a new FlexE group,and reconfigure the FlexE clients and services.This operation will inevitably cause the interruption of the services carried on the group,which cannot guarantee the reliability of the bearer network.To meet this requirement,FlexE technology needs to be further improved and extended to support online modification of FlexE group configuration.

Smaller time slot bandwidth:The time slot bandwidth of FlexE technology is still large,which cannot meet the application requirements of some 5G+fields.For example,distributed energy regulation information in 5G + smart grid requires millions of 2Mbps massive connections.Application scenarios of remote surgery and remote consultation in 5G + smart medical also require 10 200Mbps pipe bandwidth.The current hard pipe provided by FlexE technology cannot meet this demand.At present,the CMCC has launched a small particle solution based on the MTN interface to improve this defect.

In order to make FlexE technology better play its advantages and overcome its disadvantages in 5G mobile transport network,China Mobile Communications Corporation puts forward the concept of Slicing Packet Network (SPN),and set up a new project of G.mtn in ITU-T SG15 Q11 in October 2018.SPN insert a path layer upon FlexE shim layer.SPN defines a new network technology that combines FlexE and transport technology to face more challenges of 5G.Enhancing FlexE to an end-to-end network technology,SPN can achieve ultra-low latency forwarding by introducing 66B cross-over technology,which better adapts to uRLLC services.In addition,SPN extends the functions of OAM and Automatic Protection Switching (APS) in the path layer,further improves the management ability and enhances the robustness of the network.

V.CONSLUSION

FlexE technology achieves decoupling between MAC layer and PHY layer through lightweight enhancement technology on the basis of IEEE 802.3 Ethernet,and supports flexible adaptation between MAC layer rate and PHY layer rate.Functions of port bonding,channelization and sub-rate can meet the needs of 5G mobile transport network,such as large bandwidth,hard pipe slicing,flexible speed and time synchronization.With the further maturity and perfection of FlexE technology,it will be greatly developed and widely used in the fields of 5G bearer,data center interconnection and so on.