A2CPS Data Transfer and Management Guide

 

Outline

Data Transfer and Management Guide

Transferring A2CPS project data from your local machine to TACC’s Corral system can be accomplished using two methods: command line tools (scpsftprsync) and graphical tools (Cyberduck).

 Data Transfer Methods

Usage Mode Transfer Method
Command Line Tool scp
Command Line Tool sftp
Command Line Tool rsync
Graphical Tool Cyberduck

 

 What is Corral?

 

Corral is one of TACC’s systems dedicated to storage and data management. Corral can be used for storing protected data on a secure location of Corral (i.e. Corral Secure). As an A2CPS project user, you have access to Corral’s secure server “secure.corral.tacc.utexas.edu”, as well as the secure Corral folder /corral-secure/projects/A2CPS. Corral Secure’s system is to be used exclusively for transferring and accessing A2CPS project data.

 

Prerequisites for A2CPS Project User

There are two requirements needed prior to transferring data:

  • A TACC Account
  • Multi-Factor Authentication (MFA) pairing with the TACC Token app

As an A2CPS project user, you should already have access to a TACC account through the TACC Portal. If you are unsure of your TACC account username, please consult the “TACC Account Request Confirmation” email you have received from TACC Accounting Staff, or contact acantu@tacc.utexas.edu.

 

TACC now requires MFA pairing. To set up MFA, you can follow this tutorial.

 

 Using Command Line Tools to Transfer and Organize Data

A common method of transferring files between TACC resources and/or your local machine is through the command line.

 

 scp, sftp, & rsync

These three command line tools are secure and can be used to accomplish data transfer. You can run these commands directly from the terminal if your local system runs Linux or macOS.

 

Note: It is possible to use these command line tools if your local machine runs Windows, but you will need to use a ssh client (ex. PuTTY). However, for the scope of this project and ease in data transfer, it is recommended that Windows users follow our Transfer Data with Cyberduck guide below.

 

 

For users that are new to the command line, using either scp or sftp to transfer data is advised.

 

 Prerequisites for Data Transfer with Command Line Tools

Before we begin, you will need to know:

  • the path to your data file(s) on your local system
  • the path to your transfer directory on Corral

 Determining the Path to Your Data File(s) on Your Local System

In order to transfer your project data, you will need to know where your file(s) is located on your local system. To do so, navigate through the file system on your local computer.

 

For example, you can do so on Mac by going to Finder. Common data file locations such as Desktop and Documents should be accessible through the Favorites pane on the left-hand side of the window. You can also do Finder > Go to find other common locations on your computer.

 

Once you have navigated to where your file(s) is located, take note of its path by right-clicking on the file and selecting Get Info.

 

 Figure 1. Use Get Info to determine “Where” the path of your data file(s) is
 

For example, a file located in an a2cps-data folder in Documents would have the following path:

 

/Users/myname/Documents/a2cps-data/my_file.txt

 

 Determining the Path to Your Transfer Directory on Corral

A transfer directory on Corral has already been set up for you and is available for data transfer. The transfer directory path is individualized to your research center/institution.

 

For example, for a user that is affiliated with the institution University of Texas at Austin, the path to their transfer directory on Corral would be:

 

/corral-secure/projects/A2CPS/submissions/utaustin/

 

Note: The transfer directory name will always be one word and in lowercase.

 

If you are unsure of your transfer directory path, please consult the email you have received from acantu@tacc.utexas.edu, or contact acantu@tacc.utexas.edu directly.

 

 Transfer Data with scp

scp copies files between hosts on a network. To transfer a file (ex. my_file.txt) to Corral’s secure system via scp, open a terminal on your local computer and navigate to the path where your data file is located.

 

localhost$ cd Documents/a2cps-data/

 

 

Assuming your TACC username is jdoe and you are affiliated with UT Austin, a scp transfer that pushes my_file.txt from your local computer to Corral’s secure system would look like this:

 

localhost$ scp ./my_file.txt jdoe@secure.corral.tacc.utexas.edu:/corral-secure/projects/A2CPS/submissions/utaustin/

 

Note: This command will copy your data file directly to your individualized transfer directory on Corral’s system.

 

If you have not done so already, enter this command in your terminal, replacing the file name, TACC username, and your individualized transfer directory name appropriately.

 

After entering the command, you will be prompted to login to Corral’s secure system by entering the password associated with your TACC account as well as the token value generated from your TACC token app.

 

The data transfer was successful if your terminal output looks like this:

 

my_file.txt     100% ##  #.# KB/s  ##:##

 

 

If you wish to learn more about scp, you can do so here.

 

Transfer Data with sftp

sftp is a file transfer program that allows you to interactively navigate between your local file system and Corral’s remote secure system. To transfer a file (ex. my_file.txt) to Corral’s secure system via sftp, open a terminal on your local computer and navigate to the path where your data file is located. 

 

localhost$ cd Documents/a2cps-data/

 

 

Assuming your TACC username is jdoe and you are affiliated with UT Austin, a sftp transfer that pushes my_file.txt from your local computer to Corral’s secure system would look like this:

 

localhost$ sftp jdoe@secure.corral.tacc.utexas.edu:/corral-secure/projects/A2CPS/submissions/utaustin

Password:

TACC Token Code:

Connected to secure.corral.tacc.utexas.edu.

Changing to: /corral-secure/projects/A2CPS/submissions/utaustin

sftp>


 

If you have not done so already, enter this command in your terminal, replacing the TACC username and your individualized transfer directory name appropriately.

At this point, you have logged in to Corral’s secure system and been redirected to your transfer directory. To confirm your location on Corral, enter the following command:

 

sftp> pwd
Remote working directory: /corral-secure/projects/A2CPS/submissions/utaustin

 

 

To list the files currently in your transfer directory:

 

ftp> ls

utaustin_dir.txt

 

 

To list the files currently in your local directory (ex. /Users/myname/Documents/a2cps-data/):

 

sftp> lls

my_file.txt

 

Note: The leading l in the lls command denotes that you are listing the contents of your local working directory.

 

To transfer my_file.txt from your local computer to /corral-secure/projects/A2CPS/submissions/utaustin:

 

sftp> put my_file.txt

Uploading my_file.txt to /corral-secure/projects/A2CPS/submissions/utaustin/my_file.txt

my_file.txt     100% ##  #.#KB/s   ##:##

 

To check if my_file.txt is in the utaustin subfolder:

 

sftp> ls

my_file.txt

utaustin_dir.txt

 

To exit out of sftp on the terminal:

 

ftp> bye

localhost1$

 

If you wish to learn more about sftp, you can do so here.

 

 Transfer Data with rsync 

rsync is a file copying tool that can reduce the amount of data transferred by sending only the differences between the source files on your local system and the existing files in your transfer directory. To transfer a file (ex. my_file.txt) to Corral’s secure system via rsync, open a terminal on your local computer and navigate to the path where your data file is located.  

 

localhost$ cd Documents/a2cps-data/

 

Assuming your TACC username is jdoe and you are affiliated with UT Austin, a rsync transfer that pushes my_file.txt from your local computer to Corral’s secure system would look like this:

 

localhost$ rsync ./my_file.txt jdoe@secure.corral.tacc.utexas.edu:/corral-secure/projects/A2CPS/submissions/utaustin/

 

If you have not done so already, enter this command in your terminal, replacing the TACC username and your individualized transfer directory name appropriately.

 

If the command returns 0 in your terminal, the data transfer was successful.

 

If you wish to learn more about rsync and how to synchronize your file transfer, you can do so here or follow the “Transferring files using scp and rsync” section of our Stampede2 guide.

 

 Transfer Data with Cyberduck

Cyberduck is a free graphical tool for data transfer and is an alternative to using the command line. With a drag-and-drop interface, it is easy to transfer a file from your local system to Corral’s secure system. You can use Cyberduck for Windows or macOS.

 

For Windows

Download and install Cyberduck for Windows on your local machine.

Once installed, click “Open Connection” in the top left corner of your Cyberduck window.

 Figure 2. Windows Cyberduck and “Open Connection” setup screen

To setup a connection, type in the server name secure.corral.tacc.utexas.edu. Add your TACC username and password in the spaces provided. If the "More Options" area is not shown, click the small triangle button to expand the window; this will allow you to enter the path to your transfer directory (ex. /corral-secure/projects/A2CPS/submissions/tacc) so that when Cyberduck opens the connection you will immediately be in your individualized transfer directory on Corral. Click the "Connect" button to open your connection.

Consult Figure 3. below to ensure the information you have provided is correct. If you have not done so already, replace the “Path” with the path to your individualized transfer directory.

 

 Figure 3. Windows “Open Connection” setup screen

 

Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value.

 

Once connected, you can navigate through your remote file hierarchy using familiar graphical navigation techniques. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to Corral.

  For Mac

Download and install Cyberduck for macOS on your local machine.

Once installed, go to Bookmark > New Bookmark to setup a connection.

 

Note: You cannot select “Open Connection” in the top left corner of your Cyberduck window as macOS’ setup screen is missing the “More Options” button.

 

To setup a connection using “New Bookmark", type in the server name secure.corral.tacc.utexas.edu. Add your TACC username and password in the spaces provided. If the "More Options" area is not shown, click the small triangle or button to expand the window; this will allow you to enter the path to your transfer directory (ex. /corral-secure/projects/A2CPS/submissions/tacc) so that when Cyberduck opens the connection you will immediately be in your individualized transfer directory on Corral. As you fill out the information, Cyberduck will be creating the bookmark for you. Exit out of the setup screen and click on your newly created bookmark to launch the connection.

 

 

 Figure 4. macOS “New Bookmark” setup screen

Consult Figure 4. above to ensure the information you have provided is correct. If you have not done so already, replace the “Path” with the path to your individualized transfer directory.

 

Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value.

 

Once connected, you can navigate through your remote file hierarchy using familiar graphical navigation techniques. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to Corral.

 References