Experimental Demonstration of Distributed Multi-tenant Cloud/Fog and Heterogeneous SDN/NFV Orchestration for 5G Services Ricard Vilalta


Experimental Demonstration of Distributed
Multi-tenant Cloud/Fog and Heterogeneous
SDN/NFV Orchestration for 5G Services
Ricard Vilalta, Arturo Mayoral, Ramon Casellas, Ricardo Mart ´
nez, Raul Mu ˜
noz
Communication Networks Division
Centre Tecnol
ogic de Telecomunicacions de Catalunya, CTTC
Av. Carl Friedrich Gauss 7, 08860 Castelldefels, Spain Email: [email protected]
Abstract —It is expected that in 5G networks billions of smart
devices will generate huge aggregated volumes of data that will
be processed in distributed cloud/fog infrastructure. To this end,
it is required an integrated management of the network and the
cloud resources forming a converged end-to-end system. Software
Dened Networking (SDN) and Network Function Virtualization
(NFV) architectures are the key enablers to integrate both
network and cloud resources, enabling cross-optimization in both
sides.
This paper presents the experimental activities related to
5G services using the ADRENALINE testbed located at CTTC
premises in Castelldefels (Barcelona, Spain). SDN orchestration
is presented as a feasible and scalable solution for providing
end-to-end connectivity between heterogeneous networks and
cloud/edge computing. Moreover, we present the demonstration of
an SDN/NFV orchestrator to dynamically create virtual backhaul
tenants over a multi-layer (packet/optical) aggregation network
and deploy virtual network functions to better adapt the capacity
increase of Mobile Network Operators.
I. IN T RO D U C T I O N
The fth generation of mobile networks technology (5G)
is not only focused on the evolution of the radio technologies,
but with the design of a new End-to-End (E2E) converged
network and cloud infrastructure.
This converged infrastructure, illustrated in Fig. 1, is com-
posed of: E2E heterogeneous network segments covering radio
and xed access, metro aggregation, and core transport involv-
ing heterogeneous wireless and optical technologies; massive
distributed cloud computing and storage infrastructure; and
large amounts of heterogeneous smart devices and terminals
for traditional mobile broadband services (e.g., smartphones,
tablets, etc.) and IoT services (e.g. sensors, actuators, robots,
cars, drones, etc.).
From the network perspective, the 5G architecture needs to
provide high exibility, low-latency, and high-capacity in order
to support the forecasted 1000x growth in mobile data trafc
with sub-millisecond latency 1. On the control/management
side, E2E connectivity services need to be provisioned between
distributed cloud infrastructures and end users. These require-
ments can only be met by efciently integrating heterogeneous
access (RAN, xed access, satellite, Wi-Fi, personal area net-
works), optical/wireless crosshaul (fronthaul/backhaul), metro aggregation packet networks and high-capacity optical core
transport networks.
For this integration, SDN Orchestration is proposed to
coordinate, in a hierarchical, logically centralized manner,
the heterogeneous control plane technologies of the different
network segments, which may remain separated as independent
administrative domains. Moreover, the current SDN controllers
northbound interface (NBI) is highly heterogeneous and tech-
nology and vendor dependent. The STRAUSS project has
dened the rst Transport API named Control Orchestration
Protocol (COP), that abstracts the particular control plane tech-
nology of a given transport domain. COP provides a research-
oriented multi-layer approach using YANG/RESTconf 2.
At the cloud level, the demand of massive computing and
storage will dramatically be increased by new 5G services,
which will require processing and storage capabilities (e.g.,
Big Data). In addition, the impending growth of Network
Function Virtualization (NFV) 3 and Mobile Edge Comput-
ing (MEC) 4 also require cloud services for the deployment
of software functions (e.g., mobile Evolved Packet core (EPC),
local cache, rewalls). Originally, cloud services have been
implemented in core data centers (DCs) for high-computational
or long-term processing. However, the cloud is being spread to
the edge of the network (e.g., in edge DCs located in the metro
network, or even in network nodes or mobile base stations with
cloud capabilities) in order to reduce the latency of services
for the end user. This concept is referred to as fog computing
6. Therefore, 5G networks need a global orchestration for
the distributed cloud/fog implementation and the management
of heterogeneous networks.
The ongoing efforts carried out by CTTC towards the
aforementioned integration architecture are condensed in the
presented 5G SDN/NFV experimental platform for testing
advanced end-to-end IoT and mobile services 5. In this paper,
we present in detail three different use cases available for real-
life demonstration in our platform:
End-to-End SDN Orchestration of IoT Services Using
an SDN/NFV-enabled Fog Node
Hierarchical SDN Orchestration of Wireless and Op-
tical Networks
Integrated SDN/NFV Orchestration for the Dynamic
Deployment of Mobile Virtual Backhaul Networks

Fig. 1: ADRENALINE Testbed for 5G services
This paper is organized as follows, section 2 introduces the
implementation details of the CTTC 5G experimental tested;
section 3 includes the detailed description of the three 5G use
cases available for real-life demonstration in our testbed; and
nally section 4 summarizes the conclusions and future work.
II. EX P E R I M E N TA L S E T U P D E S C R I P T I O N
The cloud computing platform and transport network of
the ADRENALINE Testbed (Fig. 1) is composed by the
Cloud/Fog infrastructure, the intra-DC networks, the multi-
domain heterogeneous Wireless/Optical networks and the con-
trol/management planes, all physically installed at CTTC
premises in Castelldefels (Barcelona, Spain). The presented
demonstrations exploit the ADRENALINE testbed system
features, as well as they also integrate EXTREME testbed
(SDN-enabled wireless domain) and IoTworld testbed (based
on wireless sensor networks).
For the cloud/fog computing platform, we have deployed
OpenStack Liberty into Commercial Off The Shelf (COTS)
servers. Two availability zones have been dened in order to
emulate distributed DC locations, which are interconnected as
depicted in Fig. 1. The Fog/Edge node has been implemented using an Intel Next Unit of Computing (NUC) on top of which
is deployed an OpenStack compute node instance running in
a third availability zone.
For the intra-data center network, OpenFlow switches have
been employed. All inter-DC trafc is aggregated to the core
through the access/metro segments which are composed of
OpenFlow 1.4 switches deployed on COTS hardware with
several 1G NICs implemented by xDPD software switch. Each
Metro/Core border node includes a 10 Gb/s XFP tunable
transponder interface. Both the intra-data center networks, and
the access/metro segments are controlled with OpenDayLight
(ODL) SDN Controller, Hydrogen Service Provider release
instances using OpenFlow.
The inter-data center interconnection traverse across the
core GMPLS-controlled optical network segment. This is com-
posed of an all-optical WSON with 2 ROADMs and 2 OXCs
providing re-congurable (in space and in frequency) end-to-
end lightpaths, deploying a total of 610 km of G.652 and
G.655 optical ber, with six DWDM wavelengths per optical
link. The optical SDN controller is responsible for the inter-
data center network connectivity and it has been implemented
following the Active Stateful Path Computation Element (AS-
PCE) architecture.Optical SDN Controller
E2E SDN
Orchestrator (pABNO)
LTE/WiFi/mmWav e
EXTREM E Te stbe d ADRENALINE Te stbe d
M e tro M PLS
DWDM Cor e
SDN
Controller
UE
SDN
ControllerllllerererererererererererererererererererererererererererererererCoCoCoCoCoCoCoCoCoCoCoCoCoCo
IoTworld
IoT CO2 WSN IoT Heat
WSN
IoT GW 1
IoT GW 2
Core DC
Edge
SDN CtlEdge
Fog CtlCore DC
SDN CtlCore DC
Cloud Ctl
Cloud/Fog
Orchestrator
SDN Transport
Orchestration
(cABNO)
Wireless SDN
ControllerDN
Network
Hypervisor
SDN IT and Network Orchestrator (SINO)
NFV Orchestrator
VNF Manager #NvEPC Manager
Acce s s /Metro MPLS
MME
S-PGW
HSS
vEPC

We Will Write a Custom Essay Specifically
For You For Only $13.90/page!


order now

Fig. 2: 5G Cloud/Fog and SDN/NFV orchestrator
Fig. 3: a) Message exchange for hierarchical SDN orches-
tration; b) Network Topology abstraction at the E2E SDN
orchestrator level
Fig. 2 shows the user interface for the overall 5G Cloud/Fog
and SDN/NFV orchestrator software stack. It is composed of
the NFV orchestrator, several VNF managers, an SDN IT and
Network Orchestrator (SINO), a network hypervisor and an hi-
erarchical SDN orchestrator. The 5G Cloud/Fog and SDN/NFV
orchestrator software stack has been developed entirely in
Python by CTTC and it is able to provide multi-tenancy
over SDN/NFV-enabled multi-vendor multi-technology net-
work and cloud/fog computing resources.
III. 5G SDN/NFV E X P E R I M E N TA L P L AT F O R M
D E M O N S T R AT I O N S
Following subsections include the architectural and func-
tional description of the 5G SDN/NFV experimental platform
through the three different use cases presented in this paper.
A. Hierarchical SDN orchestration of Wireless and Optical
Networks
Hierarchical SDN Orchestration has been proposed as
a feasible solution to handle the heterogeneity of different
network domains, technologies, and vendors. It focuses on
network control and abstraction of several control domains,
whilst using standard protocols and modules. The need of
hierarchical SDN orchestration has been previously justied
in 7 with two basic purposes: the ability to scale and the
increase of security.
Fig. 4: E2E conectivity provisioning workow between IoT
gateway and virtual machine running at Edge node
In this use case, the hierarchical SDN Orchestration is
applied for the integration of wireless and optical transport
networks 8, to provide E2E connectivity between the User
Equipment (UE) and a cloud service deployed in the Core DC
location. In the wireless segment (Fig. 1), implemented over
the EXTREME Testbed, an SDN controller is in charge of the
programming of the wireless network (access and backhaul).
This SDN controller tackles the specics of the wireless
medium, implementing the proper extensions to control wire-
less devices. In the optical segment, implemented over the
ADRENALINE Testbed, we consider an SDN-enabled MPLS-
TP aggregation network, while the control of the core network
relays on an AS-PCE over a GMPLS distributed control plane.
A parent E2E SDN Orchestrator, based on the IETF
ABNO architecture in 9 (pABNO), is responsible of the
E2E provisioning across the different network segments. The
pABNO orchestrates several network segments: an SDN-
enabled wireless segment and the MPLS/Metro and Core
network segments orchestrated by a child ABNO (cABNO).
The cABNO is responsible for abstracting the multi-domain
transport segments, and it offers a simplied view to the
pABNO, thus improving scalability and security.
Fig. 3.a shows the orchestration workow. It can be ob-
served that an E2E connection is requested (POST Call) to
the pABNO. The pABNO computes the involved network con-
trollers (Wireless SDN/cABNO) and requests the underlying
connection to them. We can observe how the workow follows
inside a cABNO, which is responsible for another level of
hierarchy of the SDN orchestration process.
B. End-to-End SDN Orchestration of IoT Services Using an
SDN/NFV-enabled Fog Node
SDN is a key technology to address all the technical net-
working challenges posed by the IoT. SDN aims to overcome
the limitations of traditional IP networks, which are complex
and hard to manage in terms of network conguration and
reconguration due to faults and changes. An SDN controller
can be viewed as a network operating system which interacts
with the data plane and the network applications by means of
Application Programmable Interfaces (APIs). In this regard,
also the different needs in networking resources such as
bandwidth and delay can be managed more easily thanks to the
software programmability approach facilitated by SDN in the
network control. Another important benet of SDN is that it
paves the way for the integration of smart objects with fog and
cloud computing. More specically, thanks to the exibility ClientOptical SDN Controller
Call ACK
POST Call
cABNOpABNO
POST Call
Establish Flows
PCInitiate {src,dst}
PCRpt{LspId}
Flows ACK
Call ACK…
Establish Flows
Establish Flows
Flows ACK
Flows ACK
SDN CTL 1 W ireless
SDN CTLSDN CTL 2AP1
… FLOW_M OD Request data processing
Request data
processing
SINOSDN
Orche stratorEdge
SDN CTL
SDSDSDSDSDSDSDSDSDSDSDSDSDSDSDSD
Request VM
VM Ack
POST Call
Call ACK
Data
Create Flows Flows ACK
Request VM Fog VM
VM ACK Cloud/Fog
Orche strator
IoT GW
SDN/NFV
Edge Node
Cloud VM infoCloud VM info
1
2
3
4

Fig. 5: Setup delay. (left) Edge node, (right) Core DC
provided by SDN, the data ows of information between IoT
nodes and fog or cloud computing can be easily managed. This
enables collaborative analytics between geo-distributed smart
things.
Integrating IoT and SDN can also increase the efciency
of the network by responding to changes or events detected
by the IoT which might imply network reconguration. For
example, SDN can be used in IoT applications where the data
are transmitted from the sensors periodically in specic time
frames to schedule the requested bandwidth on the transmis-
sion paths only during the active duty cycles. Such dynamic
reconguration of the forwarding devices is only possible
via centralized applications which orchestrate IoT collected
information and network resources information jointly.
We have deployed an SDN/NFV-enabled Edge Node in
ADRENALINE Testbed for integrating wired IoT gateways
from the IoTWorld Testbed by means of E2E SDN Orchestra-
tion of integrated Cloud/Fog and network resources 6. E2E
SDN orchestration provides network connectivity between IoT
gateways and deployed virtual machines (VMs) which might
be allocated in the proposed edge node or in a DC located in
the core network.
The SDN IT and Network Orchestrator (SINO) is re-
sponsible for handling Virtual Machine (VM) and network
connectivity requests, which are processed through the Cloud
and SDN orchestrators. The orchestration process consists of
two different steps: the VM creation and network connectivity
provisioning (see Fig. 4). The SINO requests the creation
of virtual instances (VMs) to the Cloud Orchestrator, which,
is responsible for the creation of the instances. It is also
responsible to attach the VMs to the virtual switch inside the
host node (at the edge node or in a core DC). When the VMs
creation is nished, the Cloud Orchestrator replies the VMs
networking details to the integrated Cloud/Fog and network
orchestrator (MAC address, IP address and physical computing
node location). The SDN orchestrator is the responsible to
provision E2E network services. The SDN orchestrator will
provide the E2E connectivity between the requested IoT gate-
way and the deployed VM. Finally, data from IoT gateway
will ow to the processing resources located in the proposed
SDN/NFV-enabled edge node.
E2E connectivity setup delay has been measured 100 times
between IoT GW and edge node (Fig. 5.a) or core DC (Fig.
5.b). The histograms and CDFs are showed. In average the
setup delay between the IoT GW and the edge node is 456
ms, while towards the core DC is 4070 ms, due to the fact
that a bidirectional optical lightpath needs to be dynamically
established.
Fig. 6: Multi-layer aggregation network connecting RANs and
DCs and abstracted view of the backhaul networks per MNO
(left).
C. Integrated SDN/NFV Orchestration for the Dynamic De-
ployment of Mobile Virtual Backhaul Networks
In order to cope with the augmenting data trafc, MNOs
expect that virtualization of network functions (NFV) and
infrastructure (SDN) are appealing to obtain a more scalable,
cost-efcient and exible MNO deployment, in particular,
in the backhaul infrastructure. We assume that a number of
MNOs owning their radio area network (RAN) are connected
to a common physical multi-layer (packet and optical) ag-
gregation infrastructure. This shared physical infrastructure is
partitioned to compose individual virtual backhaul tenants on
top of it. Furthermore, the MNO Evolved Packet Core (EPC)
functions are as well virtualized into the cloud connected to
the aggregation network. This model enables MNOs to exibly
adjust their virtual backhaul and EPC necessities to the actual
trafc loads.
This demonstration use case experimentally assess the
dynamic computation and automatic deployment of a MNO
virtual backhaul along with a virtual EPC (vEPC) 10. The
5G Cloud/Fog and SDN/NFV orchestrator coordinates the
virtualization of heterogeneous transport technologies within
the aggregation segment as well as compute cloud resources
at the DCs.
Fig. 6 shows the physical multi-layer aggregation network
to connect MNO’s RAN and DC domains wherein virtual SDN
controller (vSDN) and vEPC are instantiated. The aggregation
network leverages the statistical multiplexing provided by
MPLS packet switching and the huge transport capacity of
optical switching applying multi-layer grooming techniques.
An MNO creating/increasing its backhaul capacity is built
upon the aggregation network as interconnected virtual packet
domains. The MNO SDN controller’s vision is an abstraction
of a set of connected packet domains (via an optical connec-
tion) providing the connectivity between the RAN and vEPC.
Each abstracted packet domain is represented by a virtual
packet node whose interfaces are mapped to the physical
incoming/outgoing links of a packet ow.
The network topology and packet resource status are used VEPC1
VM N
OVS

OVSvSDN1OFS
OFS
OFS
vEPC2
VM N
OVS

OVS
vSDN2
Aggre ga tion MPLS
pa cke t doma in
EnB VMNO2
EnB VMNO1Optica l doma in
MPLS Pa cke t Core
Doma inDC infra structure
Physical Topology
OFS
A
B C
D E
F G
MMES-PGWHS
S
S1-MMEovs
Abstracted view of
vMNO1vEPC 1
S1-U
vSDN1
OFP OFP
OFP
Abstracted MPLS node
A
C EG
H
MMES-PGWHS
S
S1-MMEovs
Abstracted view of
vMNO2vEPC2
S1-U
vSDN2
OFP OFP
OFP
Abstracted MPLS node
B C FH
Abstracted
MPLS node
Abstracted MPLS node

Fig. 7: Workow for provisioning MNO virtual backhaul
network and VNFs
to dynamically set up packet MPLS tunnels for backhauling
upcoming mobile LTE signaling and data bearers (i.e., S1-
MME and S1-U interfaces) between the RAN and vEPC 12.
The vSDN controller for the virtual backhaul is provided as a
VNF in the DC. Last but not least, the connectivity within the
DC network is virtualized connecting the core packet domain
and the deployed cloud VNFs.
Fig. 7 shows the workow between the involved functional
blocks of the SDN/NFV orchestrator to manage the creation
of an SDN-controlled virtual backhaul and the corresponding
vEPC.
Step 1 allows the NFV orchestrator to request the pro-
visioning of the vSDN controller (for the virtual backhaul)
and the vEPC. This is handled by the corresponding VNF
managers sending requests to the Compute controller of VMs
with the respective implementation (image) of the VNFs
(vSDN and vEPC). Next, in step 2, the creation of the MNO
virtual backhaul is conducted allowing the connectivity of the
created vSDN controller to congure such an infrastructure.
To do that, the MNH receives the request and computes the
domain sequence within the aggregation network in order to
connect at the packet level the MNO RAN and the vEPC.
This requires that at rst the traversed packet domains are
interconnected via an optical connection which is triggered
by the SDN orchestrator. When the optical connection is set
up (by the network hypervisor 11) at the packet level all
the domains are interconnected. For those packet domains
the SDN Orchestrator subsequently requests the packet ow
provisioning specifying ingress/egress links of those domains
to derive the abstracted (virtual) packet node forming the
targeted virtual backhaul.
Finally, a L2 ow in the DC infrastructure (e.g., Ethernet)
is created to connect the virtual (MPLS) node with the vEPC.
Once the virtual backhaul connectivity is ready, this is notied
to the NFV orchestrator, and at that time, the vSDN has a view
of the virtual packet backhaul used to transport LTE bearers
between the RAN and the vEPC. IV. C
O N C L U S I O N
Conducting real-life demonstrations of an end-to-end 5G
scenario including both IoT and mobile broadband services,
requiring the integration of heterogeneous wireless access and
optical transport networks, distributed cloud computing, and
wireless sensor and actuators networks is a very challenging
task.
CTTC has been working on the development of the rst-
known end-to-end 5G platform capable of reproducing such
an ambitious scenario. This paper has described the exist-
ing demonstrations, supported functionalities, use cases, and
preliminary results among the different experimental facilities
available at CTTC.
Further research will consist on introducing service func-
tion chainning in the ADRENALINE testbed, as well as opti-
mal resource allocation algorithms which based on constraints
(e.g., latency) decide the optimal allocation of virtualized
network functions.
AC K N OW L E D G M E N T
This work was partially funded by EU FP7 STRAUSS
(FP7-ICT-2013-EU-Japan 608528), EU FP7 COMBO
(317762), and Spanish MINECO project DESTELLO
(TEC2015-69256-R).
RE F E R E N C E S
1 5GPPP white paper, the 5G Infrastructure Public Private Partnership: the next generation of communication networks and services, March2015,
2 A. Mayoral, et al., “First experimental demonstration of distributed cloud and heterogeneous network orchestration with a common Trans-port API for E2E service provisioning and recovery with QoS”, in Proc.
OFC 2016 , Anaheim (CA), USA.
3 Network function virtualization (nfv): Architectural framework, ETSI GS NFV 002 v.1.1.1,, 2013.
4 Mobile-Edge Computing – Introductory Technical White Paper, ETSI MEC ISG, September 2014.
5 R. Mu ˜
noz, et al., The CTTC 5G end-to-end experimental platform
integrating IoT, SDN, and distributed cloud , in Proceedings of Wireless
World Research Forum Meeting 35 (WWRF), 14-16 October 2015,
Copenhagen (Denmark).
6 Ricard Vilalta, et al., “End-to-End SDN Orchestration of IoT Services Using an SDN/NFV-enabled Edge Node”, in Proc. OFC 2016, Anaheim(CA), USA.
7 R. Vilalta, et al., Hierarchical SDN Orchestration for Multi-technology Multi-domain Networks with Hierarchical ABNO, ECOC 2015.
8 R. Vilalta, et al., “Hierarchical SDN Orchestration of Wireless and Optical Networks with E2E Provisioning and Recovery for Future 5GNetworks”, in Proc. OFC 2016, Anaheim (CA), USA.
9 D. King, and A. Farrel, “A PCE-based Architecture for Application- based Network Operations”, IETF RFC 7491, 2015.
10 R. Martinez, et al., “Integrated SDN/NFV Orchestration for the Dy- namic Deployment of Mobile Virtual Backhaul Networks over a Multi-
layer (Packet/Optical) Aggregation Infrastructure”, in Proc. OFC 2016,
Anaheim (CA), USA.
11 R. Vilalta, et al., “Multi-Tenant Transport Networks with SDN/NFV”, IEEE/OSA Journal of Lightwave Technology, Vol. 34, No. 8, 2016.
12 R. Martinez, et. al., “Experimental Validation of a SDN Orchestrator for the Automatic Provisioning of Fixed and Mobile Services”, in Proc
of ECOC 2015.NFV
orch.vSDN
mnger.vEPC
mnger.SINOCloud CtlNetwork
HypervisorSDN
Pkt D1SDN
Opt. D2SDN
Pkt D3SDN
DC D4
VM req w/ SDN ctrler image
VM rep w/ IP & MAC address
VM req w/ EPC image
VM rep w/ addressing of EPC elements (MME, SGW/PGW, et c.
Creation of the VMs for the
vSDN ctrlr and vEPC1
Req for MNO virtual backhaul w/ SDN ctrlr IP address
Req for pkt flow on D1 b/w in/out ports
Rep pkt flow
Req for opt. tunnel on D2 between in/out ports
Req for creating an opt. connection
Rep created opt. connection
Rep for opt. tunnel on D2
Req for pkt flow on D3 b/w in/out ports
Rep pkt flow
Req for pkt flow towards the vEPC
Rep pkt flow
Creation of MNO virtual
backhaul
2
All the requests are duplicated to allow
bidirectional connectivity
Rep MNO virtual backhaul
Req for e2e connectivity D1, D3 and D4 w/ in/out ports
Rep for e2e pkt connectivity SDN
Orchestrator

x

Hi!
I'm Darin!

Would you like to get a custom essay? How about receiving a customized one?

Check it out