Virtualization

Intro

Virtualization is the art of subdividing resources provided by modern computers in order to achieve maximum performance, isolated execution, maximum utilization, resource managed hardware sharing and so on. Virtualization is not a novel concept. Mainframes designed in the late '70s incorporated the idea of isolating hardware design and hardware-agnostic software execution. Virtualization exists in all aspects of modern high performance computing. We try to experiment on Virtualized I/O, especially in Network Device Virtualization.

Storage and Network I/O

We believe that modern High Performance Interconnection Networks provide abstractions that can be exploited in Virtual Machine execution environments but lack support in sharing architectures. Previous work has shown that integrating the semantics of Virtualization in specialized software that runs on Network Processors can isolate and finally minimize the overhead on the VM Hypervisor concerning access to the device by Guest VMs. Direct I/O has been proposed as the solution to the CPU overhead imposed by guest VM transparent services that can lead to low throughput for high bandwidth links. However, minimizing CPU overhead comes at the cost of giving away the benefits of the device driver model. Integrating protocol offload support (present in most modern NICs) in virtual network device drivers can lead to performance improvement. Bypassing the Hypervisor in data movement can also minimize the overhead imposed by heavy I/O but at the cost of safety and memory protection.

We envision a Virtualization-enabled High performance Network Interface that can achieve line-rate throughput and optimized sharing of Network I/O in Virtual Machines by utilizing commodity hardware and innovative resource-sharing virtualization architectures.

High-performance I/O in Virtualized Environments

Device access in Virtualization environments is often realized by specific layers within the hypervisor which allow VMs to interface with the hardware. A common practice for such an interface is a split driver model. These specific layers host a backend driver while guest VM kernels host a frontend driver exposing a generic device API to guest kernels or user-space. The backend exports a communication mechanism to the frontend along with interrupt routing, page flipping and shared memory techniques.

On the other hand, intelligent interconnects provide NICs that offload protocol processing and achieve fast message exchange, suitable for HPC applications. These NICs feature specific hardware such as DMA engines, volatile memory, I/O or network processors and an interface to the physical medium. To avoid the overhead associated with user-space -- kernel-space communication, HPC interconnects often utilize a user-level networking approach. Thus, the NIC can export a virtual instance of a network interface directly to an application. Our work is focused on integrating these semantics into the VMM split driver model.

To evaluate our framework we experiment on optimizing the data exchange path between an application running in a Xen VM and a 10Gbps interface. To provide intelligence to the network controller we choose to apply our approach on a Myrinet Myri-10G NIC and a custom 10GbE interface which consists of an I/O processor, a number of DMA engines and a commodity 10GbE NIC.

The split driver model poses difficulties for user-level direct NIC access in VMs. To enable VMM-bypass techniques, we need to let VMs have direct access to certain NIC resources. The building block of our framework is the backend which allows the frontend to communicate with the NIC's core driver. The frontend driver communicates with the backend via an event channel mechanism. Contrary to Xen's netfront / netback architecture, our framework utilizes the backend in conjunction with the NIC's core driver to grant pages to the VM user space and install mappings that can simulate the normal case while the netfront driver uses these channels as a data path (to send or receive packets).

MyriXen (ongoing)

Data access in HPC infrastructures is realized via user-level networking and OS-bypass techniques through which nodes can communicate with high bandwidth and low-latency. Virtualizing physical components requires hardware-aided software hypervisors to control I/O device access. As a result, line-rate bandwidth or lower latency message exchange over 10GbE interconnects hosted in Cloud Computing infrastructures can only be achieved by alleviating software overheads imposed by the Virtualization abstraction layers, namely the VMM and the driver domains which hold direct access to I/O devices. We have designed MyriXen, a framework in which Virtual Machines efficiently share network I/O devices bypassing overheads imposed by the VMM or the driver domains. MyriXen permits VMs to optimally exchange messages with the network via a high performance NIC, leaving security and isolation issues to the Virtualization layers. Smart Myri-10G NICs provide hardware abstractions that facilitate the integration of the MX semantics in the Xen split driver model. With MyriXen, multiple VMs exchange messages using the MX message passing protocol over Myri-10G interfaces as if the NIC was assigned solely to them. We believe that MyriXen can integrate message passing based application in clusters of VMs provided by Cloud Computing infrastructures with near-native performance.

Summary

Our framework allows VMs to share an HPC NIC efficiently and exchange messages with the network. It is a thin split driver layer running on top of the NIC's core driver and consists of the backend driver in the driver domain and the frontend drivers in the VMs. Our current agenda consists of evaluating our prototype in order to estimate our framework's efficiency. In the future, we plan to experiment on fine-tuning the NIC's intelligence and propose a high-performance interconnection architecture for Virtualized environments based on commodity components.

See the Virtualization Section in our Bibliography Portal for selected publications concerning Virtualization techniques.

Publications

• A. Nanos, G. Goumas and N. Koziris, “Exploring I/O Virtualization Data paths for MPI Applications in a Cluster of VMs: A Networking Perspective”, Proceedings of the 5th Workshop on Virtualization in High-Performance Cloud Computing, held in conjunction with Euro-par 2010, Ischia - Naples, Italy, 30 Aug. - 3 Sep., 2010 [ pdf ] [ slides ]
• A. Nanos and N. Koziris, “MyriXen: Message Passing in Xen Virtual Machines over Myrinet and Ethernet”, Proceedings of the 4th Workshop on Virtualization in High-Performance Cloud computing, held in conjunction with Euro-par 2009, Delft, The Netherlands, 24-28 August, 2009 [ pdf ] [ slides ]