python peerPlatform.py -c MediaZ.ini

GridTorrent Installation

GridTorrent is written in Java and doesn't need any installation. Just download the code from here and unzip it to a directory

GridTorrent Operation Modes

GridTorrent can run as :

a. a standalone torrent application to upload/download a file b. a server attached in a terminal which can accept remote requests for download/upload files c. a daemon (linux service) that is detached from any terminals and has the same functionality as the above server d. a shell that provides functionality of remote administrating GridTorrent servers/daemons

GridTorrent standalone application

To run the GridTorrent standalone application execute gtorrent.sh in Linux or gtorrent.exe in Windows. With the -h option a help message is provided.

[azisi@gridvm5 GrediaGridTorrent]$ gtorrent.sh -h
usage: java -jar GridTorrent.jar [ -f filename | -i id ] -r rls
 -b,--blocksize <arg>        The block size that will be used when
                             transmitting a file
 -e,--failurerate <arg>      The failure rate that will be used when
                             sending a file
 -f,--filename <arg>         The filename to seed
 -h                          Print help for this application
 -i,--id <arg>               The id/hash of the file to download
 -o,--propertiesfile <arg>   The GridTorrent.properties file
 -p,--piecesize <arg>        The piece size that will be used when
                             uploading a file
 -r,--rls <arg>              The rls endpoint (DRLS/
 -t,--seedtime <arg>         The seconds to seed after downloading the
                             file
 -u,--uploadservers <arg>    A comma separated list of servers to use for
                             uploading the file <host:port,host:port>

For easier visual integration the command line arguments can be included in a file and then the file can be given as a unique command line argument. For example we make the following sample with gtorrent file extension:

[azisi@gridvm5 GrediaGridTorrent]$ cat sample.gtorrent 
-r 
https://gridvm6.cslab.ece.ntua.gr:8443
-f 
dummyfile
-u 
gridvm1.cslab.ece.ntua.gr:48620,gridvm2.cslab.ece.ntua.gr:48620

If the user has selected the gtorrent.sh/gtorrent.exe to open the .gtorrent by default, then by double-clicking the .gtorrent file, GridTorrent will find the file named dummyfile in the data/ directory, connect to the DRLS to publish it and send remote download command to gridvm1/gridvm2 GridTorrent servers to start downloading the file.

GridTorrent server

The GridTorrent server can be invoked by executing gtserver.sh. We can interact with the server only with remote commands that can be issued by the GridTorrent shell or the GridTorrent standalone application. According to the etc/log4j.properties diagnostic messages will be print in the terminal or the log file.

GridTorrent daemon

The GridTorrent daemon can be invoked by executing gtdaemon.sh start and can be stopped with gtdaemon.sh stop

We can interact with the daemon only with remote commands that can be issued by the GridTorrent shell or the GridTorrent standalone application. According to the etc/log4j.properties diagnostic messages will be print in the log file.

GridTorrent shell

Executing gtshell.sh the GridTorrent shell will be initiated and we will be able to connect to remote servers. The help command shows the implemented remote commands

[azisi@gridvm5 GrediaGridTorrent]$ gtshell.sh 
Java GridTorrent Shell starting
# help
GridTorrent commands
start name rlsip:rlsport ip:port 
         Starts a new download on server listening on ip:port
                         if the LFN=name exists in the data directory then GT starts to share the file
                         else GT search in the RLS for the UID=name and tries to downloaded
stop uid ip:port   
         Stops an existing download on server listening on ip:port
delete lfn ip:port 
         Stop and deleting local files for existing download on server listening on ip:port
list ip:port       
         Listing existing downloads UIDs/LFNs on server listening on ip:port
stats uid ip:port  
         Show download statistics for the specific UID on server listening on ip:port
shutdown ip:port   
         Shutting down server listening on ip:port
sleep sec          
         The shell sleeps for sec seconds
exit               
         Exiting/Quiting GridTorrentClient
quit               
         Exiting/Quiting GridTorrentClient

GridTorrent and Globus Security Infrastructure (GSI)

GridTorrent uses GSI to secure the data and control channel when communicating with other peers. These features can be enabled/disabled using the suitable options in etc/GridTorrent.properties By default all security checks are disabled.

The available security checks are:

• authentication: Both client and server check the remote peer's certificate in order to authenticate. The certificate must be signed by a mutually trusted CA. (the CA files - cert/policy - must stored in /etc/grid-security/certificates/ or $HOME/.globus/certificates/)
• verification: Authentication is done as described above and also all messages have a cryptographic checksum in order to prevent data modification.
• encryption: Authentication is done as described above and also all messages are transmitted in encrypted form.

When GridTorrent starts, it expects to find a valid Globus proxy credential in the default location. This proxy credential is the output of a grid-proxy-init command or the fivo-authn login operation. If the proxy credential is not found or is not valid, GridTorrent tries to find the default Globus certificates usually situated in $HOME/.globus/user{cert,key}.pem In windows $HOME usually points to C:\Documents and Settings\username

• BankingZ.ini: INI file for the banking scenario

• MediaZ.ini: INI file for the media scenario

• keywords.ini: INI file for searches of keywords

GridTorrent

GridTorrent constitutes the data transfer layer of the Gredia middleware. Based on peer-to-peer techniques, it allows clients to download files from multiple sources while uploading them to other users at the same time, rather than obtaining them from a central server. At the same time, it takes advantage of the striped version of GridFTP protocol to directly communicate with GridFTP servers. Moreover, instead of the centralized RLS Grid component, GridTorrent relies on a Kademlia-based overlay for efficient and scalable Replica Location.

SFC module and Kademlia overlay

This server software implements the component responsible for constructing a Kademlia-based overlay supporting the storage and querying of metadata files and can be downloaded from here. The indexing is based on Space Filling Curves (Z-curve or Hilbert curve). The initialization of the nodes that accommodate the enabled operations in the Metadata overlay is accomplished by the execution of peerPlatform.py. The description of the nodes and the indexing method is included in a INI file, which is given as parameter to the specific executable.

Client setup

The command line client software can be downloaded from here and the source code from here . The client software enables the user to perform rich queries and to store metadata descriptions on the RDLS. The client can be executed both from windows/linux machines with the only prerequisite is a java installation. After the zip is extracted, the directory config contains the file metadataclient.properties. The propery servername is the dns name of the server executing RDLS. The property proxyfile is the file name of a valid proxy certificate (that must reside in this directory). The executables have a .sh extension for linux and a .bat extension for windows. We now describe the client executables and their usage.

queryMedia : this executable is used to query the media overlay. It takes the following parameters: author date_created date_modified category type status privacy_status keywords(separated by " ").  When searching for a range, the attribute value must be replaced by from_value..to_value (with ".." between them). For the special ALL value, just give "..".

queryBanking : The same for the banking overlay. the parameters are the following: customer_name vat_number.

storeMedia: This executable is used to upload a metadata description to the overlay. the parameters are : path_to_local_filename  author date_created date_modified category type status privacy_status keywords(separated by " ")

storeBanking: Upload metadata description to the banking overlay. The parameters are: path_to_local_filename customer_name vat_number privacy_status.

RDLS

RDLS (Rich Data Location Services) is the layer that exposes the middleware's functionality through standard Web Services functionality. In the following, we give installation instructions of how to setup a gredia middleware server and how to install the client software and access the functionality of the installed services.

Server setup

RDLS is based on the web services layer of Globus Toolkit v.4. Globus can be installed as a whole, or only the web services layer. To install the whole toolkit, download the file  Globus Toolkit v.4.01 and follow the installation instructions from here. If you wish to install only the web services layer, download it from here and follow the installation instructions from here. When the installation is completed, install the gredia middleware services package  (download it from the bottom of the page) using the following command:

globus-deploy-gar gredia_MetaData.gar

-- IoannisKonstantinou - 08 Jun 2009