Go to the U of M home page
School of Physics & Astronomy
School of Physics and Astronomy Wiki

User Tools

Trace:
computing:department:data:moving:globus

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
computing:department:data:moving:globus [2015/12/04 19:31] – [First, create a Globus account] allancomputing:department:data:moving:globus [2022/02/03 14:55] (current) – [Globus - high speed data transfer] cse-sull0153
Line 3: Line 3:
 Globus is a software tool to transfer files across the web in a reliable, high-performance and secure way. It provides fault-tolerant, fire-and-forget data transfer using simple web (or command line) interfaces. It is appropriate for transferring very large files and datasets. Globus is a software tool to transfer files across the web in a reliable, high-performance and secure way. It provides fault-tolerant, fire-and-forget data transfer using simple web (or command line) interfaces. It is appropriate for transferring very large files and datasets.
  
-We have a Globus Connect endpoint (currently) named **umnphys#data**.+We have a Globus Connect endpoint (currently) named **umn.edu CSE POSIX home directories and data** which lets you access Physics data storage.
  
 ===== How to use Globus ===== ===== How to use Globus =====
Line 12: Line 12:
  
   * Point your browser at http://www.globus.org and click Sign Up.   * Point your browser at http://www.globus.org and click Sign Up.
-  * On the Create an Account page, fill in the information (your name, email address, username, password, etc.) and read the terms, then click Register.+  * On the Create an Account page, fill in the information (your name, @umn.edu email address, username, password, etc.) and read the terms, then click Register.
   * You will receive an email with a link which you need to follow to confirm your new Globus account.   * You will receive an email with a link which you need to follow to confirm your new Globus account.
  
Line 18: Line 18:
 ==== Transferring files between Physics and other endpoints ==== ==== Transferring files between Physics and other endpoints ====
  
- +    * Use your web browser to visit https://www.globus.org/ 
-<note> +    * Press the Login button 
-Before you can connect to our endpoint with Globus, we first need to record your "certificate subject" in our database to link it to your Physics account. You can obtain this information from the **CILogon web site** - CILogon is the service which globus uses to connect with the university authentication infrastructure. +    * Choose "University of Minnesota" for your existing organizational login, and click the "Log on" button, which takes you to the standard UMN login page. 
- +    * Click on "Endpoints Shared With You" and search for CSE. 
-    * Use your web browser to visit https://cilogon.org +    * Select "umn.edu CSE POSIX home directories and data" 
-    * Choose "University of Minnesota" as your Identity Provider, and click the "Log on" button, which takes you to the standard UMN login page. +    * Click on Open File Manager 
-    * The page will then display information, including the certificate subject, which looks something like: ''/DC=org/DC=cilogon/C=US/O=University of Minnesota/CN=Your Name A12345'' +    * By default it will select your CSE Linux home directory, though this shouldn't used for large bulk transfers!
-    * Copy this into the "Edit Physics directory information" page at [[https://www.physics.umn.edu/resources/myphys/|MyPhys]], in the "Globus CILogon certificate" field. **Make sure you copy the entire subject line, starting from "/DC=org...".** Our system should then update our endpoint account mapping within about 30 minutes. +
- +
-Note that this particular process is specific to our system in Physics; other places may handle it differently. For the MSI endpoint, you should send your details to ''help@msi.umn.edu'' +
-</note> +
- +
-To transfer files between the Physics cluster and other endpoints such as MSI. +
- +
-  * Point your browser at http://www.globus.org, click Sign In, and click Transfer Files. A web page with two Endpoint fields will display. +
-  * In one Endpoint field, pull down the expand menu and select our endpoint, **umnphys#data**. +
-    * //Remember you have to register your certificate (as described above) before you can connect to the Physics endpoint.// +
-    * You can also type into the field to filter the endpoints +
-    * On selecting the endpoint, you may get redirected to the standard University of Minnesota web login (unless you already have a session active). Log in, after which you are returned to the Globus file transfer page. +
-    * By default it will select your physics home directory, though this shouldn't used for large bulk transfers!+
     * You then need to update the "Path" field to point at the desired physics data directory (for example, ''/data/gammaray'') - the path ''/data'' itself is not a writable location.     * You then need to update the "Path" field to point at the desired physics data directory (for example, ''/data/gammaray'') - the path ''/data'' itself is not a writable location.
     * The contents of that directory will then display     * The contents of that directory will then display
-  * In the other Endpoint field, pull down the expand menu and select the appropriate site - for example, msihpc#panfs to connect to the main MSI storage.+    * In the other Endpoint field, pull down the expand menu and select the appropriate site - for example, msihpc#panfs to connect to the main MSI storage.
  
 To transfer files between endpoints, select a file or directory from each list, then click one of the large arrow buttons to tell Globus the desired direction of the transfer. To transfer files between endpoints, select a file or directory from each list, then click one of the large arrow buttons to tell Globus the desired direction of the transfer.
Line 51: Line 38:
  
 After that, the procedure is similar to that described above. If you are running Globus Connect, the first name in the list of available endpoints is your local computer, though you can select any site where you have an account. After that, the procedure is similar to that described above. If you are running Globus Connect, the first name in the list of available endpoints is your local computer, though you can select any site where you have an account.
 +
 +===== Additional notes =====
 +
 +The transfer speed will depend greatly on the nature of the data set. For example, the number of parallel streams initiated is dependent on the size of the files being transferred.  2 streams for files less than 50MB, 4 streams for files between 50Mb and 250MB and 8 streams for files >250MB.
 +
 +To use UDT instead of TCP as the tranfer protocol, put 'useudtplease' in the 'Label this Transfer' box (this may be useful for very distant transfers with high latency (BDP) on the network path).
 +
computing/department/data/moving/globus.1449279095.txt.gz · Last modified: 2015/12/04 19:31 by allan

Page Tools