Scientific Computing Without the Command LIne: Enabling HPC Code to Run Anywhere through a Web Interface with the Agave API

Find this page at http://krt3.lsu.edu/training/agave/tutorial-2-20-19/AgaveWebAPI.html

What is a Science Gateway?

Normally "Science Gateway" means a web interface used to run a complex scientific applicaiton on a high performance computer.

Normally, it means a web interface used to run a complex scientific application on a high performance computer.

More formally:

A Science Gateway is a community-developed set of tools, applications, and data that are integrated via a portal or a suite of applications, usually in a graphical user interface, that is further customized to meet the needs of a specific community. Gateways enable entire communities of users associated with a common discipline to use national resources through a common interface that is configured for optimal use. Researchers can focus on their scientific goals and less on assembling the cyberinfrastructure they require. Gateways can also foster collaborations and the exchange of ideas among researchers. (quoted from https://www.xsede.org/gateways-overview)

What is Agave

The Agave Platform (http://agaveapi.co) is an open source, science-as-a-service API platform for powering your digital lab. Agave allows you to bring together your public, private and shared high performance computing (HPC), high throughput computing (HTC), cloud and Big Data resources under a single friendly REST API

How Does Agave Work

Agave works by using a REST api and use JSON to format data.

What do the words JSON and REST mean?

  1. What is JSON? JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language (quoted from http://www.json.org/ ). Essentially, this is just a standard way to format data. You don't need to learn it, but you can.
  2. 2 What is REST? Representational state transfer (REST) or RESTful Web services are one way of providing interoperability between computer systems on the Internet. REST-compliant Web services allow requesting systems to access and manipulate textual representations of Web resources using a uniform and predefined set of stateless operations. (quoted from https://en.wikipedia.org/wiki/Representational_state_transfer )

What Capabilities Does Agave Provide

  • Run code
  • Manage data
  • Collaborate meangingfully
  • Integrate anywhere
  • Provenance

Getting Started

To get started go to the Agave web page and create an account:

  • Go to https://togo.agaveplatform.org
  • Select the Agave tenant if it isn't already selected and then Click on the "Create An Account" link in the gray bar under the "Login" button. Agave Login SCreen
  • Fill in the information on the form that comes up and click on the "Sign up" button. Agave Create Account Screen
  • Follow the instructions that come up from Agave.
  • Go your email, the one you used to sign up with, and activate your Agave account. If you don't activate the Agave account you can't participate in the tutorial because you can't login to the Agave platform.
If you don't see a menu bar on the left hand side of your Agave page widen your browser window until you do.
  • This is what you see if everything goes correctly.
    Login Success image
  • Create a Storage System

    Before we can put the other files where we need where we need them to be we must setup our our storages system on the Agave Togo web interface. On the menu on the left you will see the "Systems" link. Click on this link to go to the Systems page which looks like this: Pick Systems from the menu In the image below notice the three ovals on the top right. Click on the leftmost oval that has "Form" in it. We won't distract ourselves with the code until we learn what we are doing. Storage System On the drop down menu under the line "Select System Type" chose "Storage", this tells Agave which type of system we are defining. The storage system is where your data and output are stored.

    Click the "Next" box to go to the second page in defining the storages system. The next few pages define the details of the machine we will be using for storage. In the image below are the correct input for the input boxes. These boxes give the storage system (our shelob.hpc.lsu.edu machine) details.

    • The ID box gives the system being described an Agave unique name.
    • The Name box is a short descriptive name.
    • The Status should be "UP" as you can't use a machine that isn't up and running.
    • The Description box is a longer descriptive name.
    • The site is the information we use to define LSU systems without being system centric.
    Next part of Storage System form
    Now click the "Next" box to bring up the next part of the form.

    Fill in the information from the image below. The form is slightly longer so we will need to use two images for all of the information.

    • Protocol: Choose SFTP
    • Host: shelob.hpc.lsu.edu (just like we use to SSH in
    • Auth Server Port: 22, where we go to logon
    • Root Directory: the machines root directory
    • Home Directory: your home directory (on hpctrain20 it would be /home/hpctrain20)
    Second part of above page.
    The following image gives you the rest of the information you need for this part of the form. Do NOT hit the "Next" box until you fill it all in.
    • Proxy Tunnel: NO
    • Type: gives you choices of how you authenticate. Since we aren't going to create public/private keys for training accounts we will use the "password" choice.
    • This is your LSU HPC machine login name and password (hpctrain## if you are using a training account)
    • When you put your cursor over the Next button you will see you can not click it. This means you are done and need to submit the form.
    Second part of above page. The next screen that comes up will indicate success or failure. Hopefully you will see this: Second part of above page.
    This indicates you successfully completed the form. Now let's test and see if we entered everything correctly. On this form click on the "Browse Systems" button. If you see this you did it correctly:
    Second part of above page.

    Create the Execution System

    Now we are ready to create the form for the executable system. From the "File Browser" menu click on the "Systems" link in the left menu. Then click on the "New Systsem" link on the Systems page.
    Systems Page.

    This definition is a long one. You can fill it in using the web page listed below if you get too far behind.
    http://krt3.lsu.edu/training/agave/tutorial-2-20-19/AgaveWebAPI.html
    Just like in the first page of the build Storage you choose the "Form" link if you see code. Then you use the drop down menu to chose "Execution". First Execution System page. Click on the "Next" box. The information on this page is broken into seperate images. The first image below asks for the following:

    • ID: Agave unique execution machine name.
    • Name: short descriptive name.
    • Status: remember if a machine isn't "UP" you can't use it.
    • Description: more descriptive explanation
    • Site: "hpc.lsu.edu (or "loni.org" for LONI machines)
    • Execution Type: the type machine you are using (Condor, CLI, HPC)
    • Scheduler: The scheduler on your system. This on is "Custom_Torque"
    Second Execution System page.

    The following image is part of the same page for our execution system in Agave.

    • The first line repeats the schedule to give us continuity.
    • Max system jobs: you need to login to your machine and use your queuing commands to find this info. Fifty (50) for this machine.
    • Scratch Directory: we usually use /work/username for tutorials.
    • Work Directory: we usually use /work/username for tutorials.
    • Environment: leave blank
    • Startup Script: leave blank
    • Queues: this block describes the queue we will be using.
      • Name: Queue name on machine in use
      • Maximum Queues: your machine's maxnumber of jobs that can run in the queue at the same time.

    The image below is where we fill in the rest of the Queue information:
    • MaximumJobs: repeated from the last image for continutity
    • Maximum Nodes: is where you define the max nodes available on the system. This is found through your machine documentation.
    • Maximum Memory per Node: is where you define the max memory available on the nodes in the system. This is found through your machine documentation.
    • Maximum Processors per node: (should be 16 for Shelob) Also found in you user documentation.
    • Maximum Requested time: The max run time for any job. I used 1:00 because of htis job.
    • Custom Directives: Allows us to use needed PBS directives as we would in a PBS script written on the machine. See the lines below.

    NOTE: The special directives section below, which are cut off in the image are: #PBS -A hpc_hpcadmin3\n #PBS -l cput=${AGAVE_JOB_MAX_RUNTIME}\n #PBS -l walltime=${AGAVE_JOB_MAX_RUNTIME}\n #PBS -q ${AGAVE_JOB_BATCH_QUEUE}\n #PBS -l nodes=${AGAVE_JOB_NODE_COUNT}:ppn=16
    Click on the next link to go to the next part of the form. Queue info part 1

    In the next image we see the third fram in the Wizard form that defines the actual login information. In these examples we are using the "Password" authentication method. In real life you should use the private/public keys. You can create them directly on your machine.

      Login: Describe your connectivity for your Execution machine.
      • Host: the machine you are logging into for your execution: "shelob.hpc.lsu.edu" for this tutorial.
      • System Auth Server Port: 22 for our HPC machines
      • Protocol: The protocol we use to access our execution and storage machines.
      • proxyTunnel: are you using one to access your machines? If sso, chose yes.
      Login Authentication: Your login method and credentials.
      • Type: Password is the one we are choosing for this tutorial. I recommend one of the others in "real life".
      • Username: Your user name on the machine you are running jobs on. For this tutorial if you are using an HPC training account it will be that.
      • Password: The password we gave.
      Storage: Describe your storage machine.
      • Protocol: The file transfer protocol. For this tutorial it is SFTP because that is one of the secure protocols our HPC machines accepts.
      • Host: The machine you are logging into. "shelob.hpc.lsu.edu".
      • System Auth Server Port: "22" for our machines.
      • Root Directory: "/" the root directory on the machine.
      • Home Diretory: "/home/username" the login name you use on your machine.
      • Proxy Tunnel: "NO", yes if you are using one.
      Storage Authentication:
      • Type: Password
      • Username: your login name
      • Password: your password
    • Click submit to submit the form.

    In the image below you see the window that should come up if your submit was successful. Before we do anything else we need to test the execution system to see if we can reach it. To do this click on the "Browse Files" button. If we were successful you will see the files on your execution system.

    This image is what a file browser looks like. On the hpctrn## accoutns you won't see this many files.

    Create the Application

    This is a series of Agave screenshots you will use to create your application.

    First we name and describe our app.

    • Name: the name and the version should be globally unique together as well as unique from any other application you create.. The convention I learned when I started is that it helps to have your Agave username in it.
    • Version: As you improve versions you can use the same name with an increasing version number. Just like all software.
    • Label: a label for use in other forms.
    • Short description: short description of what the app does.
    • Long description: A longer more descriptive version of the "Short Description".
    • Tags: continuted on the next screenshot.
    The second part of this form gives us "tags" that describe the app. These come with app.

    The second page of the application creation form.

    • Deployment path: the path on your storage system where you want the output files to be stored.
    • Deployment system: This is a pull down menu with a choice from all of your storage systems.
    • Wrapper script: This is a needed script and is stored in the deployemnt (agave-deployment) directory. It is already on you storage/deployment machine.
    • Test script: a small file that is used to test the application.

    The third page of the application creation form describes the execution system.

    • Execution Type: We are using "HPC".
    • Execution System: This is the execution system we've created. It will be your only choice.
    • Default queue: We've described our queue in the execution system so we leave this blank.
    • Default node count: The number of nodes to use when running this job.
    • Default memory: The node has 64GB so we ask for this.

    • We are describing the parameter file (variable).
    • Parameter ID: a unique identifier.
    • Label: describes the parameter.
    • Prepend the argument? False unless it needs to be put into the submit script before the parameter.

    Part two of the parameter file.

    • Part two of the Parameter section of the form.
    • Min Cardinality: Minimum number of instances needed.
    • Max Cardinality: Maximum number of instances per job.
    • Parameter type: what type of parameter we expected: string, number, boolean, etc.
    • Default value: the name of the parameter file

    The ouput file description.

    Second half of the form page.

    The application was submitted to Agave successfully.

    We will choose the "Browse Files" button

    Run the app.

    Setup Shelob Account

    To use Agave to submit jobs you need to create files for the machine you will use to execute (execution machine)your job, the one you use to store (storage machine) your data on and the file for the application you are going to use. You need one file for each even if the same machine is used for both options. We have created templates for you to use. More about these as we setup our HPC account.

    All training accounts begin with hpctrn and end with two numbers starting at "01" and ending with "20" inclusive. I am using "hpctrn20".

    Use the HPC training account or your HPC account to login to Shelob.

    • Remember to open your SSh client
    • ssh username@shelob.hpc.lsu.edu
    • Logon ids and passwords on next tab - remind me to show them
    • To list your files use:
      ls -l
    • Drawgau app