GrediaMiddlewareInstallationInstructions < CSLab

CSLab Web>GrediaMiddlewareInstallationInstructions (2009-06-19, AthanassiaAssiki) (raw view)

In this page we present installation instructions for the gredia middleware platform. Specifically, we present instructions for the following components: RDLS, SFC, !Gridtorrent.

---+ RDLS         

RDLS (Rich Data Location Services) is the layer that exposes the middleware's functionality through standard Web Services functionality. In the following, we give installation instructions of how to setup a gredia middleware server and how to install the client software and access the functionality of the installed services. 
---++ Server setup       

RDLS is based on the web services layer of Globus Toolkit v.4. Globus can be installed as a whole, or only the web services layer. To install the whole toolkit, download the file  [[http://www-unix.globus.org/toolkit/survey/index.php?download=gt4.0.1-all-source-installer.tar.gz][Globus Toolkit v.4.01]] and follow the installation instructions from [[http://www.globus.org/toolkit/docs/4.0/admin/docbook/index.html][here]]. If you wish to install only the web services layer, download it from [[http://www-unix.globus.org/toolkit/survey/index.php?download=ws-core-4.0.1-src.tar.gz][here]] and follow the installation instructions from [[http://www.globus.org/toolkit/docs/4.0/admin/docbook/ch09.html][here]]. When the installation is completed,[[%ATTACHURL%/gredia_MetaData.gar][download]]  the gredia middleware services package  and install it using the following command:

globus-deploy-gar gredia_MetaData.gar

The source code can be found  [[%ATTACHURL%/gredia_metadata_services_source_code.zip][here]]

---++ Client setup   

The command line client software can be downloaded from [[http://www.cslab.ntua.gr/~ikons/gredia_metadata_client.zip][here]] and the source code from  [[%ATTACHURL%/gredia_metadata_client_source_code.zip][here]] . The client software enables the user to perform rich queries and to store metadata descriptions on the RDLS. The client can be executed both from windows/linux machines with the only prerequisite is a java installation. After the zip is extracted, the directory config contains the file metadataclient.properties. The propery servername is the dns name of the server executing RDLS. The property proxyfile is the file name of a valid proxy certificate (that must reside in this directory). The executables have a .sh extension for linux and a .bat extension for windows. We now describe the client executables and their usage.

*queryMedia* : this executable is used to query the media overlay. It takes the following parameters: author date_created date_modified category type status privacy_status keywords(separated by " ").  When searching for a range, the attribute value must be replaced by
from_value..to_value (with ".." between them). For the special ALL
value, just give "..".

*queryBanking* : The same for the banking overlay. the parameters are the following: customer_name vat_number.

*storeMedia*: This executable is used to upload a metadata description to the overlay. the parameters are : path_to_local_filename  author date_created date_modified category type status privacy_status keywords(separated by " ")

*storeBanking*: Upload metadata description to the banking overlay. The parameters are: path_to_local_filename customer_name vat_number privacy_status.

---++ SFC module and Kademlia overlay

The Python implementation of this server software responsible for constructing a Kademlia-based overlay supporting the storage and querying of metadata files and can be downloaded from [[%ATTACHURL%/SFC_p2pOverlay.tar.bz2][here]]. Just unzip it to a directory.  Python2.5.x and the package [[%ATTACHURL%/pysqlite.org][pysqlite-2.5.x]] are required. The indexing is based on Space Filling Curves (Z-curve or Hilbert curve). The initialization of the nodes that accommodate the enabled operations in the Metadata overlay is accomplished by the execution of peerPlatform.py. The description of the nodes and the indexing method is included in a INI file, which is given as parameter to the specific executable (-c option). Examples of INI files for the media scenario ([[%ATTACHURL%/MediaZ.ini][here]]), the banking scenario ([[%ATTACHURL%/BankingZ.ini][here]]) and keyword searches ([[%ATTACHURL%/keywords.ini][here]]) have been attached. For the media overlay for instance, type: 

<verbatim>
python peerPlatform.py -c MediaZ.ini
</verbatim>


---+ !GridTorrent

!GridTorrent constitutes the data transfer layer of the Gredia middleware. Based on peer-to-peer techniques, it allows clients to download files from multiple sources while uploading them to other users at the same time, rather than obtaining them from a central server. At the same time, it takes advantage of the striped version of !GridFTP protocol to directly communicate with !GridFTP servers. Moreover, instead of the centralized RLS Grid component, !GridTorrent relies on a  Kademlia-based overlay for efficient and scalable Replica Location.

---++ !GridTorrent Installation

!GridTorrent is written in Java and doesn't need any installation. Just download the code from  [[http://www.cslab.ece.ntua.gr/~azisi/GridTorrent-1.3.tar.gz][here]] and unzip it to a directory

---+++ !GridTorrent Operation Modes

!GridTorrent can run as :

  a. a standalone torrent application to upload/download a file 
  b. a server attached in a terminal which can accept remote requests for download/upload files
  c. a daemon (linux service) that is detached from any terminals and has the same functionality as the above server
  d. a shell that provides functionality of remote administrating !GridTorrent servers/daemons

---++++ !GridTorrent standalone application

 To run the !GridTorrent standalone application execute gtorrent.sh in Linux or gtorrent.exe in Windows.
 With the -h option a help message is provided.

<verbatim>
[azisi@gridvm5 GrediaGridTorrent]$ gtorrent.sh -h
usage: java -jar GridTorrent.jar [ -f filename | -i id ] -r rls
 -b,--blocksize <arg>        The block size that will be used when
                             transmitting a file
 -e,--failurerate <arg>      The failure rate that will be used when
                             sending a file
 -f,--filename <arg>         The filename to seed
 -h                          Print help for this application
 -i,--id <arg>               The id/hash of the file to download
 -o,--propertiesfile <arg>   The GridTorrent.properties file
 -p,--piecesize <arg>        The piece size that will be used when
                             uploading a file
 -r,--rls <arg>              The rls endpoint (DRLS/
 -t,--seedtime <arg>         The seconds to seed after downloading the
                             file
 -u,--uploadservers <arg>    A comma separated list of servers to use for
                             uploading the file <host:port,host:port>
</verbatim>

 For easier visual integration the command line arguments can be included in a file and then the file can be given as a unique command line argument. For example we make the following sample with gtorrent file extension:

<verbatim>
[azisi@gridvm5 GrediaGridTorrent]$ cat sample.gtorrent 
-r 
https://gridvm6.cslab.ece.ntua.gr:8443
-f 
dummyfile
-u 
gridvm1.cslab.ece.ntua.gr:48620,gridvm2.cslab.ece.ntua.gr:48620
</verbatim>

If the user has selected the gtorrent.sh/gtorrent.exe to open the .gtorrent by default, then by double-clicking the .gtorrent file, !GridTorrent will find the file named dummyfile in the data/ directory, connect to the DRLS to publish it and send remote download command to gridvm1/gridvm2 !GridTorrent servers to start downloading the file.

---++++ !GridTorrent server

The !GridTorrent server can be invoked by executing gtserver.sh. 
We can interact with the server only with remote commands that can be issued by the !GridTorrent shell or the !GridTorrent standalone application. According to the etc/log4j.properties diagnostic messages will be print in the terminal or the log file.

---++++ !GridTorrent daemon 

The !GridTorrent daemon can be invoked by executing gtdaemon.sh start and can be stopped with gtdaemon.sh stop 

We can interact with the daemon only with remote commands that can be issued by the !GridTorrent shell or the !GridTorrent standalone application. According to the etc/log4j.properties diagnostic messages will be print in the log file.

---++++ !GridTorrent shell 

Executing gtshell.sh the !GridTorrent shell will be initiated and we will be able to connect to remote servers. The help command shows the implemented remote commands

<verbatim>
[azisi@gridvm5 GrediaGridTorrent]$ gtshell.sh 
Java GridTorrent Shell starting
# help
GridTorrent commands
start name rlsip:rlsport ip:port 
         Starts a new download on server listening on ip:port
                         if the LFN=name exists in the data directory then GT starts to share the file
                         else GT search in the RLS for the UID=name and tries to downloaded
stop uid ip:port   
         Stops an existing download on server listening on ip:port
delete lfn ip:port 
         Stop and deleting local files for existing download on server listening on ip:port
list ip:port       
         Listing existing downloads UIDs/LFNs on server listening on ip:port
stats uid ip:port  
         Show download statistics for the specific UID on server listening on ip:port
shutdown ip:port   
         Shutting down server listening on ip:port
sleep sec          
         The shell sleeps for sec seconds
exit               
         Exiting/Quiting GridTorrentClient
quit               
         Exiting/Quiting GridTorrentClient
</verbatim>

---++++ !GridTorrent and Globus Security Infrastructure (GSI) 

!GridTorrent uses GSI to secure the data and control channel when communicating with other peers.
These features can be enabled/disabled using the suitable options in etc/GridTorrent.properties
By default all security checks are disabled.

The available security checks are:
   * authentication: Both client and server check the remote peer's certificate in order to authenticate. The certificate must be signed by a mutually trusted CA. (the CA files - cert/policy - must stored in /etc/grid-security/certificates/ or $HOME/.globus/certificates/)
   * verification: Authentication is done as described above and also all messages have a cryptographic checksum in order to prevent data modification.
   * encryption: Authentication is done as described above and also all messages are transmitted in encrypted form.

When !GridTorrent starts, it expects to find a valid Globus proxy credential in the default location. This proxy credential is the output of a grid-proxy-init command or the fivo-authn login operation. If the proxy credential is not found or is not valid, !GridTorrent tries to find the default Globus certificates usually situated in $HOME/.globus/user{cert,key}.pem In windows $HOME usually points to C:\Documents and Settings\username

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
ini	BankingZ.ini	r1	manage	8.0 K	2009-06-18 - 12:49	AthanassiaAssiki	INI file for the banking scenario
gz	GridTorrent.tar.gz	r1	manage	120.2 K	2009-06-18 - 12:51	KaterinaDoka	GridTorrent source code
ini	MediaZ.ini	r1	manage	8.0 K	2009-06-18 - 12:49	AthanassiaAssiki	INI file for the media scenario
bz2	SFC_p2pOverlay.tar.bz2	r1	manage	462.5 K	2009-06-18 - 11:29	AthanassiaAssiki	Source code of SFC module and Kademlia P2P overlay
gar	gredia_MetaData.gar	r1	manage	48.2 K	2009-06-08 - 17:14	IoannisKonstantinou	Gredia middleware services
zip	gredia_metadata_client_source_code.zip	r1	manage	94.7 K	2009-06-08 - 20:35	IoannisKonstantinou	Source code of metadata client
zip	gredia_metadata_services_source_code.zip	r1	manage	1184.8 K	2009-06-08 - 20:35	IoannisKonstantinou	Source code of metadata services
ini	keywords.ini	r1	manage	10.9 K	2009-06-18 - 12:50	AthanassiaAssiki	INI file for searches of keywords