Great Lakes Description

Great Lakes is the shared, Linux-based high-performance computing (HPC) cluster available to researchers at the University of Michigan. Great Lakes consists of approximately 13,000 cores – including compute nodes composed of multiple CPU cores, GPUs, large-memory nodes, and support for simulation, modeling, machine learning, data science, genomics, and more.

Get a new Great Lakes Cluster login

  • Fill out the following form to get a new Great Lakes Cluster login.
  • Being able to login does not mean that one can use the computing resources on great lakes. In order to submit computing jobs, one will need an allocation such as UMRCP.

    UMRCP Allocation

    All faculty members, and a majority of the PhD students at Ross are eligible to receive access to UMRCP (U of M Research Computing Package). UMRCP is a pretty generous package, which provides a researcher access to:

    • 80,000 compute hours
    • 10 terabytes Turbo storage
    • 100 terabytes Archive storage

    Researchers may sign up for the UMRCP by visiting the following:

    Data Storage

    Every Great Lakes account holder has 80GB of default storage available in his/her home directory. For additional storage, they have access to 10 terabytes of turbo storage (through UMRCP allocation) at location /nfs/turbo/bus-<your uniqname>/

    Software Available

    There is a large number of software packages available on Great Lakes. Some of them are:

    • Matlab
    • R
    • SAS
    • Stata
    • Python(Anaconda)

    You may search the software that you want by keywork or by specific name. We will talk about this in deatil in the Load Modules below.

    Connecting to Great Lakes

    Set Up Duo

    1. Duo app is available on both iOS and Android platform. You may use either smartphone or tablet.
      Open the App Store or the Google Play Store, search “Duo Mobile” and install it.
    2. terminal
    3. Get your Duo enrolled by following the instructions from Information and Technology Services (ITS).

    Set Up VPN

    1. Great Lakes login is restricted to campus IP addresses. If you are trying to log in from off campus, or using an unauthenticated wireless network such as MGuest, Install UMVPN software on your computer (if you haven't already installed it).

    Install PuTTY

    1. You will first need to install the latest version of PuTTY available at https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
    2. Under “Package Files”, install 32-bit or 64-bit version of PuTTY depending on your computer system.
    3. To install 64-bit version, click on it and press “Run” as shown below.
    4. search terminal
    5. It will open the PuTTY setup window as below, press Next.
    6. search terminal
    7. Press 'Next' to install to the default folder.
    8. search terminal
    9. Press 'Install'
    10. search terminal
    11. You may be asked to allow the PuTTY application to make changes to your device, which is generally fine, so press 'Yes'.
    12. search terminal
    13. The PuTTY installation is now completed. Press 'Finish' to exit the Setup. You may unclick the 'View README file'.
    14. search terminal

    Log in

    1. Open the recently added PuTTY app in your machine, and enter greatlakes.arc-ts.umich.edu in the 'Host Name' field, and make sure that the connection type is SSH.
    2. search terminal
    3. Under Connection / SSH / X11 select "Enable X11 forwarding".
    4. terminal
    5. Under Session, You may want to save the session with the specifications identified in above two steps. You may save the session by giving it a name e.g. 'My Great Lakes Cluster', and then click 'Save'. It will show up in the saved sessions list after saved.
    6. terminal
    7. After pressing 'Open' in the configuration window above, you will get a 'PuTTy Security Alert' asking if you trust the host. You may safely press 'Yes' to proceed.
    8. terminal
    9. Enter your uniqname in the new window pops up, and press 'Enter'.
    10. terminal
    11. Enter your University of Michigan password and hit 'Enter'. Please note that you won’t be able to see the password as you type, so don’t worry about this and continue typing the password. Press enter when you are done entering the password. It will ask you to verify the login through Duo, Phone call or SMS.
    12. terminal
    13. After the account is verified, you will be successfully logged into the Great Lakes.
    14. terminal

    Tips

    Here are a few helpful hints on where to set some theme settings in PuTTY:

    • Under Window / Appearance
      • Cursor Appearance
    • Under Window / Selection
      • Action of mouse buttons
    • Under Window / Colors
      • You will see a list of possible colors to change (such as foreground, background, etc)
      • For each color you want to change: Highlight the color type, select Modify, click on the desired color from the color palette, and click OK.

    Note: to make these options permanent you must then re-save the "My Great Lakes cluster" session you created in 'Step k' above.

    Transfer Files

    You may use either Cyberduck (an SFTP client) or command line to transfer files between local machine and remote Great Lakes server.

    Cyberduck

    1. Download Cyberduck.
    2. Visit https://cyberduck.io/ and click on the first box i.e. 'Download Cyberduck for Windows' ( it will work for both Windows 7/10), then press 'Run'.

      terminal
    3. It will ask you if you want this app to make changes to your device. Press 'Yes', and you will see the following Cyberduck Setup screen.
    4. Press Install to install the Cyberduck. Once installation successfully completed, close the window.

      terminal
    5. Now open newly installed Cyberduck Desktop app. Press 'Open Connection'.
    6. terminal
    7. In the new connection window, choose SFTP from dropdown menu. Put server address as
      greatlakes-xfer.arc-ts.umich.edu, and enter your uniqname and password, and press 'Connect' as shown below:
    8. terminal
    9. You are now successfully connected to the Great Lakes server. You will see the list of yours files/folders sitting on the server.
    10. You may now simply drag or drop the files between your local computer and server. Alternatively, you may right click on the files/folders on the server to download them on your computer.

      terminal

    Command Line

    You can also use command line to transfer your files to Great Lakes server.

    1. Open PuTTY.
    2. Use cd to change to the directory where the files are lying. For example, I have changed the directory to Desktop below.

      terminal
    3. Type the following on the command line
      scp localfile uniquename@greatlakes-xfer.arc-ts.umich.edu:remotefile
    4. *Replace the localfile and remotefile with the name of the file that you want to transfer and the name you want them to has on your Great Lakes cluster. Also, replace the uniquename with your Great Lakes user name.

      Or if you want to transfer the entire directory, you can type the following on the command line:
      scp -r localdir uniquename@greatlakes-xfer.arc-ts.umich.edu:remotedir

      For example, I transfered the directory named 'first_job' to my Great Lakes cluster.

      terminal

      If you are transfering the file for the first time, you may encounter with the question about ECDSA key fingerprint. You can just type in 'yes' and continue the connecting.

      terminal
    5. Finally, enter the password and the file will be successfully transfered.
    6. terminal

    Load Modules

    Everytime before submitting your job, you will need to load the required software module(s).

    Module Name

    Below are the Great Lakes names of commonly used software modules.

    Software Module Name
    Matlab matlab
    R R/3.6.1
    SAS sas
    Stata stata-mp / stata-se
    Python (Anaconda) python-anaconda3/2019.03

    Besides, there are also two ways for you to search the name of the module:

    • Use module av module_name to see if a particular module is available on the Great Lakes.
    • Use module spider search_keyword to see all possible modules that contains that "search_keyword".

    Tips: After you get the list of the module, you may use the up/down key to go through it and the press "q" to quit the view mode.

      terminal

    Loading and Unloading Modules

    After selecting the modules that you want, you may load or unload the modules by using the following commands.

    • Use module load module_name to load the module.
    • Use module unload module_name to unload the module.
      *replace the module_name with the name of the module

    For example, we have loaded the matlab and sas modules below.

      terminal

    Check the Module(s) Loaded

    After loading the required module(s), you may use module list to see all the loaded module(s).

    Here are the two modules we have loaded before.

      terminal

    Saving Default Module List

    If you are using certain modules very frequently, it will be convenient for you to set up a default module list. The modules in the list will be automatically uploaded when you log into your Great Lakes account. Here are the steps to save the default module list.

    The steps:

    1. Log out and re-log in your Great Lakes account to ensure there is no module has been loaded.
    2. Use module purge to clear all the modules.
    3. Use module load module_name load the module(s) that you want to put on the default list.
    4. Use module save default to save the default list.
    5. terminal

    Batch Script

    Batch script contains the codes that communicate with Great Lakes. The sbatch command is used to submit a batch script to Slurm. Using this script, you may ask for computing resources and supply information about the allocation to charge. Without this script you cannot schedule/execute your job on the Great Lakes. Here is how a sample Batch script looks like.

      terminal

    The meaning of each option are shown as below:

      terminal

    Below is a Batch script template.




    Important things to remember when writing a Batch script:

    • Using Stata or SAS
    • If you are using some softwares that need license, like Stata and SAS. You need to add one line to the batch script, #BATCH --licenses=application@slurmdb:N. The application should be replaced by the name of the software, and the N should be replaced by the number of licenses that you need.

      Take the license for Stata for example, the line you need add is:

      #BATCH --licenses=stata@slurmdb:1

      Put it in the submission options part, before executing your program.


    • Access Internet via Great Lakes
    • If your job needs to access Internet via Great Lakes, you need to add some lines below to the batch script.

      export http_proxy="http://proxy.arc-ts.umich.edu:3128/"
      export https_proxy="http://proxy.arc-ts.umich.edu:3128/"
      export ftp_proxy="http://proxy.arc-ts.umich.edu:3128/"
      export no_proxy="localhost,127.0.0.1,.localdomain,.umich.edu"
      export HTTP_PROXY="${http_proxy}"
      export HTTPS_PROXY="${https_proxy}"
      export FTP_PROXY="${ftp_proxy}"
      export NO_PROXY="${no_proxy}"


      Put it in the submission options part, before executing your program.

    For detailed information on writing and submitting batch scripts refer to ARC Great Lakes User Guide.

    Submitting Jobs

    Files on Great Lakes

    Make sure all of the files that you need are present on the Great Lakes, preferably in the same folder, this may include your code, data, and Batch script file.

    Submitting Jobs

    Use sbatch filename.sh to submit your job, and get a job number. In the example, the job number is 85262.

      terminal

    Check Job Status and Delete Job

    Use squeue -u uniquename to check status of your jobs. Replace the uniquename with your Great Lakes user name.

    Use scontrol show job job_number to check the status of a specific job.

    Use scancel job_number to delete a job.

    Interactive Jobs

    Besides using Batch script, you may also submit jobs in an interactive way by using srun. You can see more details at their main page.

    Great Lakes Open OnDemand

    Great Lakes OnDemand allows researchers to access Great Lakes resources interactively using a web browser.

    Log in to Great Lakes OnDemand

    1. Before logging-in to Great Lakes OnDemand, please make sure that you have a Great Lakes login account and Duo. If you are out of the university network, you will need to install VPN as well.
    2. Then, go to https://greatlakes.arc-ts.umich.edu or click the button below.
    3. Enter your uniqname and password, and hit Login.
    4. terminal
    5. Choose the Duo verification and approve login on your mobile phone.
    6. terminal
    7. Then, you will get to the dashboard of the Great Lakes OnDemand.
    8. terminal

    Transfer Files

    1. Click Files button and select Home Directory.
    2. terminal
    3. Then, you will get to the File Explorer page that shows all your files on the Great Lakes server. You can manage your files by clicking upload, download, create new directory options.
    4. terminal

    Submit a job

    1. Click Jobs button and select Job Composer.
    2. terminal
    3. Then, you will get to the dashboard of the Job Composer. On this page, you can create Batch script, submit job, stop job, and delete job.
    4. terminal
    5. If you want to create a new job, click New Job and select From Default Template.
    6. terminal
    7. Then, you will see a default Batch script that is created automatically.
    8. terminal
    9. Edit this default Batch script by clicking Edit Files
    10. terminal
    11. Make the .sh file being selected. Then, click Edit button.
    12. terminal
    13. Copy and paste the Batch script below into the file, after the job headers. Also, make sure to ask for the right amount of resources in each configuration.
    14. #SBATCH --job-name=example_job
      #SBATCH --mail-type=BEGIN,END
      #SBATCH --nodes=1
      #SBATCH --ntasks-per-node=1
      #SBATCH --mem-per-cpu=1000m
      #SBATCH --time=1:00:00
      #SBATCH --account=singlaro
      #SBATCH --partition=standard
      terminal
    15. Then, click Save button.
    16. terminal
    17. Finally, go back to the job composer page, and click Submit button.
    18. terminal
    19. The job will start running.
    20. terminal
    21. After completed, you can click Edit Files button to check the output file.
    22. terminal terminal

    Launch interactive desktop with MATLAB

    1. Click Interactive Apps and select MATLAB.
    2. terminal
    3. Before launching the interactive desktop with MATLAB, you will need to fill the following form. Put in singlaro in the accout line, put in the memory you need in the Memory per core, and fill in the number of hours that you need. Then, click "Launch" button.
    4. terminal
    5. You will see your job has been created and queued. You may need to wait for a few seconds before it shows running status.
    6. terminal
    7. After the header part turns into "Running" status, you can click the "Launch noVNC in New Tab" button at the bottom.
    8. terminal
    9. Then, you will see the interactive desktop with MATLAB started.
    10. terminal

    Launch Jupyter Notebook Server

    1. Click Interactive Apps and select Jupyter Notebook.
    2. terminal
    3. Before launching the interactive desktop with Jupyter Notebook, you will need to fill the following form. Put in singlaro in the accout line, put in the memory you need in the Memory per core, and fill in the number of hours that you need. Then, click "Launch" button.
    4. terminal
    5. Then, you will see your job has been created and queued. You may need to wait for a few seconds before it shows running status.
    6. terminal
    7. After the header part turns into "Running" status, you can click the "Connect to Jupyter" button at the bottom.
    8. terminal
    9. Then, you will see the Jupyter Notebook started.
    10. terminal

    For more information about Great Lakes Open OnDemand go to ARC OOD webpage.