Great Lakes Description

Great Lakes is the shared, Linux-based high-performance computing (HPC) cluster available to researchers at the University of Michigan. Great Lakes consists of approximately 13,000 cores – including compute nodes composed of multiple CPU cores, GPUs, large-memory nodes, and support for simulation, modeling, machine learning, data science, genomics, and more.

Ross is renting some computer cores on university Great Lakes cluster. Contact us to get access to the Great Lakes cores.

Data Storage

Every Great Lakes account holder has 80GB of default storage available in his/her home directory. For additional storage, we are renting university Turbo storage that is mounted on Great Lakes. The location is /nfs/turbo/ross-research/ . Users may create their personal folders named as their uniqname at this location. It is requested that the researchers remove their project data from Turbo storage once the project is completed.

Contact us to get the permission to use the Turbo storage. Great Lakes access alone will not guarantee access to the Turbo storage.

Software Available

There is a large number of software packages available on Great Lakes. Some of them are:

  • Matlab
  • R
  • SAS
  • Stata
  • Python(Anaconda)

You may search the software that you want by keywork or by specific name. We will talk about this in deatil in the Load Modules below.

Connecting to Great Lakes

Set Up the Account

  1. First, you need to fill out the account request form from ARC-TS.
  2. Once you have received an email from ARC-TS confirming approval of your account, please contact us so that your name may be added to the allocation we are renting.

Set Up Duo

  1. Duo app is available on both iOS and Android platform. You may use either smartphone or tablet.
    Open the App Store or the Google Play Store, search “Duo Mobile” and install it.
  2. terminal
  3. Get your Duo enrolled by following the instructions from Information and Technology Services (ITS).

Set Up VPN

  1. Great Lakes login is restricted to campus IP addresses. If you are trying to log in from off campus, or using an unauthenticated wireless network such as MGuest, Install UMVPN software on your computer (if you haven't already installed it).

Install PuTTY

  1. You will first need to install the latest version of PuTTY available at https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
  2. Under “Package Files”, install 32-bit or 64-bit version of PuTTY depending on your computer system.
  3. To install 64-bit version, click on it and press “Run” as shown below.
  4. search terminal
  5. It will open the PuTTY setup window as below, press Next.
  6. search terminal
  7. Press 'Next' to install to the default folder.
  8. search terminal
  9. Press 'Install'
  10. search terminal
  11. You may be asked to allow the PuTTY application to make changes to your device, which is generally fine, so press 'Yes'.
  12. search terminal
  13. The PuTTY installation is now completed. Press 'Finish' to exit the Setup. You may unclick the 'View README file'.
  14. search terminal

Log in

  1. Open the recently added PuTTY app in your machine, and enter greatlakes.arc-ts.umich.edu in the 'Host Name' field, and make sure that the connection type is SSH.
  2. search terminal
  3. Under Connection / SSH / X11 select "Enable X11 forwarding".
  4. terminal
  5. Under Session, You may want to save the session with the specifications identified in above two steps. You may save the session by giving it a name e.g. 'My Great Lakes Cluster', and then click 'Save'. It will show up in the saved sessions list after saved.
  6. terminal
  7. After pressing 'Open' in the configuration window above, you will get a 'PuTTy Security Alert' asking if you trust the host. You may safely press 'Yes' to proceed.
  8. terminal
  9. Enter your uniqname in the new window pops up, and press 'Enter'.
  10. terminal
  11. Enter your University of Michigan password and hit 'Enter'. Please note that you won’t be able to see the password as you type, so don’t worry about this and continue typing the password. Press enter when you are done entering the password. It will ask you to verify the login through Duo, Phone call or SMS.
  12. terminal
  13. After the account is verified, you will be successfully logged into the Great Lakes.
  14. terminal

Tips

Here are a few helpful hints on where to set some theme settings in PuTTY:

  • Under Window / Appearance
    • Cursor Appearance
  • Under Window / Selection
    • Action of mouse buttons
  • Under Window / Colors
    • You will see a list of possible colors to change (such as foreground, background, etc)
    • For each color you want to change: Highlight the color type, select Modify, click on the desired color from the color palette, and click OK.

Note: to make these options permanent you must then re-save the "My Great Lakes cluster" session you created in 'Step k' above.

Transfer Files

You may use either Cyberduck (an SFTP client) or command line to transfer files between local machine and remote Great Lakes server.

Cyberduck

  1. Download Cyberduck.
  2. Visit https://cyberduck.io/ and click on the first box i.e. 'Download Cyberduck for Windows' ( it will work for both Windows 7/10), then press 'Run'.

    terminal
  3. It will ask you if you want this app to make changes to your device. Press 'Yes', and you will see the following Cyberduck Setup screen.
  4. Press Install to install the Cyberduck. Once installation successfully completed, close the window.

    terminal
  5. Now open newly installed Cyberduck Desktop app. Press 'Open Connection'.
  6. terminal
  7. In the new connection window, choose SFTP from dropdown menu. Put server address as
    greatlakes-xfer.arc-ts.umich.edu, and enter your uniqname and password, and press 'Connect' as shown below:
  8. terminal
  9. You are now successfully connected to the Great Lakes server. You will see the list of yours files/folders sitting on the server.
  10. You may now simply drag or drop the files between your local computer and server. Alternatively, you may right click on the files/folders on the server to download them on your computer.

    terminal

Command Line

You can also use command line to transfer your files to Great Lakes server.

  1. Open PuTTY.
  2. Use cd to change to the directory where the files are lying. For example, I have changed the directory to Desktop below.

    terminal
  3. Type the following on the command line
    scp localfile uniquename@greatlakes-xfer.arc-ts.umich.edu:remotefile
  4. *Replace the localfile and remotefile with the name of the file that you want to transfer and the name you want them to has on your Great Lakes cluster. Also, replace the uniquename with your Great Lakes user name.

    Or if you want to transfer the entire directory, you can type the following on the command line:
    scp -r localdir uniquename@greatlakes-xfer.arc-ts.umich.edu:remotedir

    For example, I transfered the directory named 'first_job' to my Great Lakes cluster.

    terminal

    If you are transfering the file for the first time, you may encounter with the question about ECDSA key fingerprint. You can just type in 'yes' and continue the connecting.

    terminal
  5. Finally, enter the password and the file will be successfully transfered.
  6. terminal

Load Modules

Everytime before submitting your job, you will need to load the required software module(s).

Module Name

Below are the Great Lakes names of commonly used software modules.

Software Module Name
Matlab matlab
R R/3.6.1
SAS sas
Stata stata-mp / stata-se
Python (Anaconda) python-anaconda3/2019.03

Besides, there are also two ways for you to search the name of the module:

  • Use module av module_name to see if a particular module is available on the Great Lakes.
  • Use module spider search_keyword to see all possible modules that contains that "search_keyword".

Tips: After you get the list of the module, you may use the up/down key to go through it and the press "q" to quit the view mode.

    terminal

Loading and Unloading Modules

After selecting the modules that you want, you may load or unload the modules by using the following commands.

  • Use module load module_name to load the module.
  • Use module unload module_name to unload the module.
    *replace the module_name with the name of the module

For example, we have loaded the matlab and sas modules below.

    terminal

Check the Module(s) Loaded

After loading the required module(s), you may use module list to see all the loaded module(s).

Here are the two modules we have loaded before.

    terminal

Saving Default Module List

If you are using certain modules very frequently, it will be convenient for you to set up a default module list. The modules in the list will be automatically uploaded when you log into your Great Lakes account. Here are the steps to save the default module list.

The steps:

  1. Log out and re-log in your Great Lakes account to ensure there is no module has been loaded.
  2. Use module purge to clear all the modules.
  3. Use module load module_name load the module(s) that you want to put on the default list.
  4. Use module save default to save the default list.
  5. terminal

Installing Custom Packages

R

If you need to use some custom packages in your R job, there are two things that need to be done.

  1. Create a folder in your Great Lakes home directory to store the custom packages.
  2. You may use the command mkdir -p ~/Rlibrary/3.4 to create a folder in your Great Lakes home directory.

  3. Create a file named .Renviron in your Great Lakes home directory by using touch ~/.Renviron.
    Use open ~/.Renviron to open the file.
    Finally, add the following text in the file, and save/close the file.
    R_LIBS_USER=~/Rlibrary/3.4

Python Virtual Environment

Most of the time, the Anaconda may provides the library you may need. If you want to use a library that is not included in Anaconda, you need to create a virtual environment to install that library in your Great Lakes.

  1. First, go to your Great Lakes home directory and load the Python module, like we load the python-3.5.2 for example.
  2. terminal
  3. Create the virtual environment by typing in the command line virtualenv --system-site-packages ~/your_environment_name
  4. terminal
  5. Use source ~/your_environment_name/bin/activate to activate the environment.
  6. terminal
  7. Then use pip install to install a library that you want to use, and it may be used in any job that you runs on Great Lakes.
  8. terminal

Batch Script

Batch script contains the codes that communicate with Great Lakes. The sbatch command is used to submit a batch script to Slurm. Using this script, you may ask for computing resources and supply information about the allocation to charge. Without this script you cannot schedule/execute your job on the Great Lakes. Here is how a sample Batch script looks like.

    terminal

The meaning of each option are shown as below:

    terminal

Below is a Batch script template.




Important Things for Batch Script:

  • Using Stata or SAS
  • If you are using some softwares that need license, like Stata and SAS. You need to add one line to the batch script, #BATCH --licenses=application@slurmdb:N. The application should be replaced by the name of the software, and the N should be replaced by the number of licenses that you need.

    Take the license for Stata for example, the line you need add is:

    #BATCH --licenses=stata@slurmdb:1

    Put it in the submission options part, before executing your program.


  • Access Internet via Great Lakes
  • If your job needs to access Internet via Great Lakes, you need to add some lines below to the batch script.

    export http_proxy="http://proxy.arc-ts.umich.edu:3128/"
    export https_proxy="http://proxy.arc-ts.umich.edu:3128/"
    export ftp_proxy="http://proxy.arc-ts.umich.edu:3128/"
    export no_proxy="localhost,127.0.0.1,.localdomain,.umich.edu"
    export HTTP_PROXY="${http_proxy}"
    export HTTPS_PROXY="${https_proxy}"
    export FTP_PROXY="${ftp_proxy}"
    export NO_PROXY="${no_proxy}"


    Put it in the submission options part, before executing your program.

Submitting Jobs

Files on Great Lakes

Make sure all of the files that you need are on the Great Lakes, this may includes your script, data, and Batch script file.

Submitting Jobs

Use sbatch filename.sh to submit your job, and get a job number. In the example, the job number is 85262.

    terminal

Check Job Status and Delete Job

Use squeue -u uniquename to check status of your jobs. Replace the uniquename with your Great Lakes user name.

Use scontrol show job job_number to check the status of a specific job.

Use scancel job_number to delete a job.

Interactive Jobs

Besides using Batch script, you may also submit jobs in an interactive way by using srun. You can see more details at their main page.

Great Lakes OnDemand

Great Lakes OnDemand allows researchers to access Great Lakes resources interactively using a web browser.

Log in to Great Lakes OnDemand

  1. Before logging-in to Great Lakes OnDemand, please make sure that you have Great Lakes account and Duo. If you are out of university network, you will need to install VPN. Just follow the instructions available at Connecting to Great Lakes to set up the account, Duo, and VPN.
  2. Then, go to https://greatlakes.arc-ts.umich.edu or click the button below.
  3. Enter your uniqname and password, and hit Login.
  4. terminal
  5. Choose the Duo verification and approve login on your mobile phone.
  6. terminal
  7. Then, you will get to the dashboard of the Great Lakes OnDemand.
  8. terminal

Transfer Files

  1. Click Files button and select Home Directory.
  2. terminal
  3. Then, you will get to the File Explorer page that shows all your files on the Great Lakes server. You can manage your files by clicking upload, download, create new directory options.
  4. terminal

Submit a job

  1. Click Jobs button and select Job Composer.
  2. terminal
  3. Then, you will get to the dashboard of the Job Composer. On this page, you can create Batch script, submit job, stop job, and delete job.
  4. terminal
  5. If you want to create a new job, click New Job and select From Default Template.
  6. terminal
  7. Then, you will see a default Batch script that is created automatically.
  8. terminal
  9. Edit this default Batch script by clicking Edit Files
  10. terminal
  11. Make the .sh file being selected. Then, click Edit button.
  12. terminal
  13. Copy and paste the Batch script below into the file, after the job headers. Also, make sure to ask for the right amount of resources in each configuration.
  14. #SBATCH --job-name=example_job
    #SBATCH --mail-type=BEGIN,END
    #SBATCH --nodes=1
    #SBATCH --ntasks-per-node=1
    #SBATCH --mem-per-cpu=1000m
    #SBATCH --time=1:00:00
    #SBATCH --account=singlaro
    #SBATCH --partition=standard
    terminal
  15. Then, click Save button.
  16. terminal
  17. Finally, go back to the job composer page, and click Submit button.
  18. terminal
  19. The job will start running.
  20. terminal
  21. After completed, you can click Edit Files button to check the output file.
  22. terminal terminal

Launch interactive desktop with MATLAB

  1. Click Interactive Apps and select MATLAB.
  2. terminal
  3. Before launching the interactive desktop with MATLAB, you will need to fill the following form. Put in singlaro in the accout line, put in the memory you need in the Memory per core, and fill in the number of hours that you need. Then, click "Launch" button.
  4. terminal
  5. You will see your job has been created and queued. You may need to wait for a few seconds before it shows running status.
  6. terminal
  7. After the header part turns into "Running" status, you can click the "Launch noVNC in New Tab" button at the bottom.
  8. terminal
  9. Then, you will see the interactive desktop with MATLAB started.
  10. terminal

Launch Jupyter Notebook Server

  1. Click Interactive Apps and select Jupyter Notebook.
  2. terminal
  3. Before launching the interactive desktop with Jupyter Notebook, you will need to fill the following form. Put in singlaro in the accout line, put in the memory you need in the Memory per core, and fill in the number of hours that you need. Then, click "Launch" button.
  4. terminal
  5. Then, you will see your job has been created and queued. You may need to wait for a few seconds before it shows running status.
  6. terminal
  7. After the header part turns into "Running" status, you can click the "Connect to Jupyter" button at the bottom.
  8. terminal
  9. Then, you will see the Jupyter Notebook started.
  10. terminal

Launch Greatlakes remote desktop

  1. Click Interactive Apps and select Greatlakes Remote Desktop.
  2. terminal
  3. Before launching the remote desktop, you will need to fill the following form. Put in singlaro in the accout line, put in the memory you need in the Memory per core, and fill in the number of hours that you need. Then, click "Launch" button.
  4. terminal
  5. Then, you will see your job has been created and queued. You may need to wait for a few seconds.
  6. terminal
  7. After the header part turns into "Running" status, you can click the "Launch noVNC in New Tab" button at the bottom.
  8. terminal
  9. Then, you will see the remote desktop started.
  10. terminal

Display Servers

Displaying Linux-based application graphics on your Windows PC requires an "X-Windows Display Server" installed on your machine. We typically use the Xming shareware for this purpose.

  1. Go to http://sourceforge.net/projects/xming/ , and click Download
  2. terminal
  3. At the very top you will see a box in which is displayed "Your Xming X Server for Windows download will start X seconds". When "X" counts down to zero, a tape will pop-up at the bottom asking to run the executable. Select 'Run'.
  4. terminal
  5. Next, press 'Yes' to the subsequent window asking the app to make changes to your device.
  6. terminal
  7. You will be welcomed to the Xming Setup Wizard. Press 'Next'
  8. terminal
  9. Continue pressing 'Next' (and accepting defaults) on the subsequent windows until you see the final windows as below. Please note that you may select the option to create a desktop Xming icon, enroute.
  10. Select Finish to complete the Xming setup (you may or may not want to launch Xming automatically at this time).
  11. terminal
  12. You may get a Windows Security Alert asking permission for Xming to communicate on your private and public networks. Press 'Allow access' to finish the setup.
  13. terminal