VOFS (DFS): Federative, disconnected, peer-to-peer filesystem for Grid4All

Telex

Apart from DFS we have implemented a fully functional and efficient generic multilog toolkit for general evaluation of the multilog design.

Creating a VO-aware filesystem with DFS

In Grid4All, DFS is used to create a VO-aware filesystem for VO members. It's basic function is to create shared workspaces from existing VO member files and storage, or other resources contributed to the system. The scenario looks like this:

• Peer-to-peer, web-like navigation.

• Notifications for file modifications and publish-subscribe messaging
• Integration with the OS. DFS can be mounted in Linux and MacOS.

• Integration with the OS. DFS can be mounted in Linux and MacOS.

Read an article on DFS architecture

<-- PERCENT INCLUDE{"CSLab.DFSArch"}% -->

• A. Chazapis, G. Tsoukalas, G. Verigakis, K. Kourtis, A. Sotiropoulos and N. Koziris, "Global-scale peer-to-peer file services with DFS," in Proceedings of the 8th IEEE/ACM International Conference on Grid Computing (Grid 2007), Austin, TX, USA, September 2007 [pdf]

DFS: Federative, disconnected, peer-to-peer filesystem for Grid4All

Overview

Grid4All

Our team is involved in Grid4All data storage, researching and developing DFS, a peer-to-peer distributed filesystem with a modern and democratic character, that is most suitable for the project. Currently, it is in an early prototype status.

The most prominent features of DFS are:

• Peer-to-peer, web-like navigation. Other's computers are accessible just by typing their address in the path.
• Federation of files. You can create workspaces and populate them by linking in files anywhere on the internet.
• Federation of storage. Peers can contribute storage to each other, creating storage pools. Peers create files using this distributed pooled storage.
• Disconnected operation. Files are locally cached and remain available when disconnected from the network. Synchronisation is done automatically, if conflicts do not occur.
• Notifications for file modifications and publish-subscribe messaging
• Integration with the OS. DFS can be mounted in Linux and MacOS.

Publications

• A. Chazapis, G. Tsoukalas, G. Verigakis, K. Kourtis, A. Sotiropoulos and N. Koziris, "Global-scale peer-to-peer file services with DFS," in Proceedings of the 8th IEEE/ACM International Conference on Grid Computing (Grid 2007), Austin, TX, USA, September 2007 [pdf]

Internet and its services are central to European life - at work and at home, indoors and outdoors. The next logical step is to provide the possibility to plug-and-upgrade personal computing capacity and to secure collaborative edutainment space on the fly through managed sharing of IT resources over the Internet. Grid4All will enable Grid services to evolve from high performance computing niche markets to a multipurpose service for anybody.

Grid4All - by leveraging the large base of broadband users - is aiming to advance the pervasiveness of Grid computing across society.

Official Project Homepage http://grid4all.elibel.tm.fr/

About Grid4All

Description of the project

Grid4All embraces the vision of a "democratic" Grid as a ubiquitous utility whereby domestic users, small organizations and enterprises may draw on resources on the Internet without having to individually invest and manage computing and IT resources. This project funded by the 6° Framework Programme of the European Commission involves institutional and industrial partners. The global budget is slightly superior to 4 800 000 euros.

Grid4All will achieve the following objectives for the Grid community:

• Alleviate administration and management of large scale distributed IT infrastructure — by pioneering the application of component based management architectures to self-organizing peer-to-peer overlay services;
• Provide self-management capabilities — improve scalability, resilience to failures and volatility thus paving the way to mature solutions enabling deployment of Grids on the wide Internet;
• Widen the scope of Grid technologies by enabling on-demand creation and maintenance of dynamically evolving scalable virtual organisations even short lived.
• Apply advanced application frameworks for collaborative data sharing applications executing in dynamic environments;
• Capitalize on Grids as revenue generating sources to implement utility models of computing but using resources on the Internet.

Grid4All will help to bring global computing to the broader society beyond that of academia and large enterprises by providing an opportunity to small organisations and individuals to reap the cost benefit of resource sharing without however the burdens of management, security, and administration.

The consortium will demonstrate this by applying Grid4All in two different application domains: collaborative tools for e-learning targeting schools and digital content processing applications targeting residential users.

WP1

WP1: Overlay Infrastructure and Programming Models

The objective of WP1 is to provide middleware and services provided by the other work packages with the basic infrastructure, models and tools to realize the Grid4All vision. In other words this WP provides the architectural model, framework and core services of the platform. Based on peer-to-peer overlay network technology and component model, WP1 provides framework (tools, APIs, and programming models) for the construction of self-managing peer-to-peer applications. It also provides communication, routing, messaging and lookup functionalities and infrastructure support for the construction of reliable and self-managing services and applications, including that of creation and management of Grid4All virtual organisations (mapped as an overlay service).

Furthermore WP1 will provide the models and tools to allow system and application developers to easily use the loosest and weakest synchronization and consistency models consisted with system and application goals. In particular a framework to understanding and instrument suitable replication protocols based on the Action-Constraint Framework (ACF) will be developed.

Partners involved: INRIA, KTH, SICS, FT, ANTARES

WP2

WP2: Virtual Organization and Resource Management

Grid4All proposes a framework based approach for the creation and management of virtual organisations. Within Grid4All virtual organisations (VO) are run time entities federating resources, services and members. VOs encapsulate all management functions such as resource allocation, membership management, performance management, monitoring, communication and security, simplyfing the construction of dynamic and distributed virtual communities.

The objectives of this WP are to provide an autonomic management framework for dynamic virtual organizations spanning multiple management domains and to develop supporting tools for the construction of Grid4All self-management domains.

WP2 will provide middleware and VO management services. It will use the framework and core services provided by WP1 to build higher level Grid management functions as overlay services: Creation, deletion and management of dynamic virtual organisations. Control of resources, services and users forming the virtual organisation. Service/resource discovery based on semantic descriptions of Grid4All entities

Market based resource management for provisioning of resources and services to virtual organisations. An implementation of a scheduling service – deployed within a virtual organisation

Partners involved: FT, INRIA, KTH, SICS, ICCS, UPRC, UPC, ANTARES

WP3

WP3: Data Storage

The Grid4All usage scenarios, centred on collaboration and utility computing for communities, small enterprises and individuals call for simple, secure and resilient access to shared data. Distributed access to shared data may occur on a large scale. Applications such as multimedia, collaboration tools and e-learning systems have specific and non-traditional storage needs. Furthermore, the storage mechanisms need to be adapted to the VO concept and to run above the large-scale, dynamic, heterogeneous environment of Grid4All.

Within the Grid4All structure, WP3 provides data services constructed using self-managing software framework provided by WP1. Work performed within WP3 plan to implement a distributed file service for a large scale heterogeneous Grid and to export a storage infrastructure built upon secure site-wide storage pools. It is also planned to federate views exported from different organisations into a unified file system for a Virtual Organisation across management domains. Finally a semantic storage architecture will be explored for a collaborative work and at large scale.

Partners involved: INRIA, KTH, SICS, ICCS, UPRC, FT, ANTARES

WP4

WP4: Applications

WP4 provides both the applications and directs the use of scenarios. This WP provides not only the implementation of proof-of-concept applications, but also the framework and API permitting the development of new applications.

This WP will have to determine scenarios and derive requirements for infrastructure and applications specific for the Grid4All intended uses. Partners involved will have to understand the complexity imposed by this environment on applications and to define algorithms to overcome situations and limitations. WP4 also plans to define and develop an API and selective transparency mechanisms to facilitate application development. The last action within this WP will be to produce application level traces and benchmarks to evaluate and validate the Grid4All infrastructure.

Partners involved: UPRC, UPC, ANTARES, ICCS, INRIA, FT

WP5

WP5: Integration and Evaluation

WP5 is in charge of:

• Integrating the different middleware building blocks developed within the work packages WP1-WP4 in an iterative fashion in order to build the Grid4All middleware infrastructure. This includes the production of guidelines, packaging and configuration tools.
• Defining an evaluation plan for the platform that identifies relevant performance metrics derived from the requirement analysis phase performed by WP1-WP4. This plan will also detail how the Grid4All system will be hosted on the test-bed(s) selected for the evaluation.
• Deploying the Grid4All middleware infrastructure along with the applications developed in WP4 on the chosen test-beds.
• Evaluating the Grid4All platform on the basis of evaluation plan and providing feedback to technical WP (WP1-WP4) in order to improve the platform both from a functional and performance point of view

Partners involved: FT, INRIA, KTH, SICS, ICCS, UPRC, UPC, ANTARES

WP6

WP6: Dissemination and Exploitation

This WP sets the basis for the execution of a good dissemination campaign of the project and for the exploitation of the obtained results. A preliminary awareness about the project will be created and a set of dissemination actions will be developed. These actions will be developed by the whole consortium in order to drive the project to a Europe-wide dissemination. An Exploitation Plan will be created defining the actions that will be jointly taken by the consortium partners following the project to further the exploitation of the results. Communication and public relations addressing the different category of target audience (end users, service providers, VARs, application providers) will be a part of this WP. This also includes identification and participation in public events where Grid4All can be demonstrated.

Actions developed within this WP are: Dissemination and technology transfer plan definition, establishment of dissemination standard, quality assurance procedure, integrated project web site and e-newsletter, organisation of events and relation with other media, Grid4All contacts database, establishment of external collaborations and communication and public relations.

Partners involved: FT, INRIA, KTH, SICS, ICCS, UPRC, UPC, ANTARES

WP7

WP7: Project Management

This work package implements the management of the project in a way that technical objectives of the project are met with respect to the available budget, and the financial reporting to the European Commission. This includes control over the consistency of work between partners and WP. Internal assessment of project progress is organized; as well as preparatory actions for external evaluation are managed. Moreover, dissemination activities as implemented in WP will directly triggered from WP7.

Management and assessment of the project will be supported by the virtual project management infrastructure, which will be also provided by WP7.

Partners involved: FT, INRIA

-- ArisSotiropoulos - 04 Mar 2008