Introduction to the IBM P5 Environment

Table of Contents

• Objectives
• What is the IBM Power5 (P5) System?
• Accessing the P5s
• File Systems
• Login
• Shell
• Environment Variables
• Customize Environment
• Compilers
• Libraries
• LInk Loader
• Parallel Execution

Objectives

1. To introduce you to the IBM AIX operating system environment.
2. Give you basic information that will allow you to set your environment on the LONI Power5 systems.

Return to top of page

What is the IBM Power5 (P5) System?

• 14 nodes with 8-way IBM Power5s
• 851.2 GF theoretical peak each; 4.256 TF total
• https://www.loni.org/systems/
• mailto: sys-help@loni.org

Cluster Name	Location	LONI	Hostname	Status
Bluedawg	Louisiana Tech	LONI	bluedawg.loni.org	open for LONI use
Ducky	Tulane University	LONI	bluedawg.loni.org	open for LONI use
LaCumba	Southern University	LONI	lacumba.loni.org	open for LONI use
Neptune	University of New Orleans	LONI	neptune.loni.org	open for LONI use
Zeke	University of Louisiana at Lafayette	LONI	zeke.loni.org	open for LONI use

Return to top of page

Accessing the IBM Power5 Systems

• telnet
• ftp
• rlogin
• rcp
• rsh

• Accessing the LONI Power5 Systems

  All LONI machines except Queen Bee are accessible only through Internet 2. To log into any of the LONI machines except Queen Bee you should first login to either Queen Bee or some machine on your campus. You will not be able to reach the Power5s from your home or commercial internet sites.

  If you are working on a Macintosh or Linux system there will be a built in utility called "Terminal" or "Console" that you will use to access the IBM Power5s.

  If you are working on a Windows system you will need to install an ssh client or something like Cygwin Most people I've asked recommend the application "Putty".

  Once you get your ssh client installed then you would use the following command to login to the system you want to access:

  username@machine.address

  On the machine you are using open up the terminal program and type the following using your user name instead of mine.

  Since my user name on all LONI machines is "ktraxler" I would type:

  ssh ktraxler@bluedawg.loni.org

  

  In the image above you see the machine prompt for the machine I was on followed by the "$" sign. Then the command, "ssh" which invokes the "Secure Shell" utility. The "ssh" is followed by my username, the "at" sign and then the machine address of "bluedawg.loni.org".

  The next three lines come up the very first time I access a new machine on my current machine or anytime the RSA key has been changed on the machine I am trying to connect to. At this point you need to type the word "yes" to continue with the logon.

  yes

  You will see the following:

  

  If you typed everything correctly you will see the following on your screen:

  

  Return to top of page

  
  

  ---

  
  

  System Environment

  File Systems

  For the file systems below there are two very important things to remember:
  • These file systems are NOT for long term storage so move the files as soon as you are done.
  • There is no backup at all.
  • LONI
  • Your home directory has a quota of 500MB
  • The "/work/default" directory, which is 280GB, has a quota of 20GB
  • The "/scratch/local" file system, local to each node has 140 GB
  • For large storage "/work/nsf102" is available

  Login

  • On all LONI systems your default shell is the "bash" shell.
  • On the LONI systems you change the shell and the password at http://allocations.loni.org.

  Return to top of page

  
  

  ---

  
  

  Shell

  • The shell interprets the input command by users to the underlying OS. The oS executes the command and returns any results back to the user via the shell.
  • You will use the command line to communicate with the system.
  • On the terminal session you will see the prompt, usually followed by a '$'.
  • Common shells are: bash, ksh, tcsh, csh
  • A set of shell commands and constructs can be saved into a text file and be run as a program - shell scripts.
  • Shells can track global variables that are referred to as their environment.

  Environment Variables

  • The shell's environment is used to store useful system information.
  • This information is stored as "environment variables".
  • The environment variables have names and store a value which affects the way a program and the environment operates.
  • Some of these variables are set when you first login, other variables can be customized by the user using a specific set of files contained in their HOME directory "/home/username".
  • Some variables can be customized by the user using a specific set of files contained in your home directory
  • You can view the environment variables to see how they are set:

  • echo $PATH

  • echo $USER

  • echo $CLASSPATH

  • echo $JAVA_HOME

  • You can use these environment variables to change how your environment "feels".
    

    echo $PATH

    export PATH=$PATH:/newdir1/newdir2

  

  Return to top of page

  Customize the Environment

  The system default environment is set using a file:
  • /etc/profile for Bourne shell
  • /etc/csh.cshrc for the C shell

  These files automatically set up the default path and user environment variables. A user can customize his/her environment in this shell resource file. To see the listing of all environment files in your account:

  ls -al

  

  To see the contents of your ".profile" file with environment setting:

  cat .profile

  

  You should not change the contents of either the .profile or the .bashrc unless you know what you are doing. Start off using softenv until you have learned what the contents mean.

  Return to top of page

  Softenv

  Softenv is a tool used to manage the environment amid the variety of software tools and libraries offered on an HPC environment. We use Softenv to setup environments for packages & compilers.

  Here is what your /home/username/.soft file looks like. First cat the file:

  cat .soft

  Your file is empty until you use SoftEnv to setup your environment for your hpc experience.

  Softenv commands:

  resoft

  command	description
  softenv	lists all packages available on the system
  soft add	adds a package to the .soft file; keyword is replaced by the string that represents a package
  soft delete	deletes a package from the .soft file.
  resets account to your initial environment

  Return to top of page

  Compilers

  Serial code
  Example code	Type	Compiler	Suffix
  xlc_r prog.c	C	Visual Age Comp.	.c
  xlC_r prog.C	C++	Visual Age Comp.	.C
  xlf_r prog.f	F77	Xlf	.f
  xlf90_r prog.f	F90	Xlf90	.f, .F
  xlf95_r prog.F	F95	Xlf95	.f, >F
  MPI Code
  Example code	Type	Compiler	Suffix
  mpcc_r prog.c	C	Visual Age Comp.	.c
  mpCC_r prog.c	C++	Visual Age Comp.	.C
  mpxlf_r prog.f	F77	Xlf	.f
  mpxlf90_r prog.f	F90	Xlf90	.f, .F
  mpxlf95_r prog.f	F95	Xlf95	.f, .F

  Return to top of page

  Libraries

  • ESSL (Engineering & Scientific Subroutine Library):
  • BLAS, Linear Algebra Solvers, FFTs, etc.
  • Sorting, Searching, Quadratures, Interpolation
  • Link with "-lessl" or "-lessl_r"
  • PESSL (Parallel ESSL):
  • Parallel BLAS, Subset of level 2 & 3.
  • Subset of ScaLAPACK (dense, banded, sparse).
  • Eigen System Solver and Singular Value Analysis. FFTs (2D and 3D).
  • Link with "-lpessl" or "-lpessl_r"
  • MASS (Mathematical Acceleration SubSystem):
  • consists of tuned intrinsic mathematical functions such as sin(), cos(), log() and exp().
  • supports C, C++ and Fortran.
  • has both scalar and vector versions.
  • is thread-safe.
  • usually provides better performance over the original intrinsic functions, but at the expense of reduced precision (1 to 2 bits at most).
  • Small accuracy loss, Faster, Vector version requires "vector expression"
  • Link with "-lmass" or "-lmassvp4"

Return to top of page

Link Loader Options

• Loadmap
• Specified as "-bloadmap:[no]file_name" and saves a log of linker action and messages in file_name.
• Listing is useful for debugging purposes.
• Debugging Mode:
• Specified as "-g", generates symbol and line number information
• Profiling:
• Specified as "-p" or "-pg"; generates monitoring information for runtime profiles
• List
• Specified as "-qlist" the output goes to the .list file.

Return to top of page

Parallel Job Execution

• The job run can be made either interactively or through batch queues using poe.
• Options: command line arguments or environment variables such as MPI parameters, adapter specification, number of nodes, etc.
• Execution: poe mp-compiled_executable options or mp-compiled_executable options
• Interactive execution example:
• mpxlf_r -o hello hello.f
• poe ./hello -tasks_per_node 4 -rmpool 1 -nodes 1
• "mpxlf_r" compiles with MPI the source code for Fortran 77 "hello.f" and creates an executable called "hello". The second line runs the executable through the parallel operation environment for interactive execution. It requires one node with four processors
• POE involves LoadLeveler (batch resource management and scheduling software).
• Remember that interactive runs are ONLY for debugging.

Return to top of page

LoadLeveler Batch Facility

• llsubmit which is used to submit jobs.
• llq which is used to monitor the queue and query the job status.
• llcancel which is used to kill running jobs or remove idle jobs from the queue.
• llstatus which is used to query machine status in the LoadLeveler cluster.

Return to top of page

LoadLeveler Job Queue Descriptions

Return to top of page

Submitting through llsubmit

wget http://krt3.lsu.edu/ktraxler/training/p5-env/LONI_env_lab.tar.gz

gunzip LONI_env_lab.tar.gz

tar -xvf LONI_env_lab.tar.gz

Compile C code

mpcc_r -o hello hello.c

Using a text editor on the AIX system, vi, nano, pico, etc. open a file called "hello.cmd": Now type in the following commands:

#! /bin/ksh	Tells the system to use the korn shell
job_type = parallel	Tells the system to run a parallel job
account_no = loni_training_0	tells which account to affect the allocation
environment = COPY ALL	Copies all environment settings
executable = /bin/poe	look in /bin for the poe executable
argument = !/LONI_env_lab/mpic	tells system where to look for and put files
input = /dev/null	tells system where to look for input (in this case there is none
output = ~/LONI_env_lab/mpic/hello.out	output files are found in this directory
error = ~/LONI_env_lab/mpic/hello.err	put the error file here
initialdir = ~LONI_env_lab/mpic	beginning directory
notify_user = ktraxler@lsu.edu	where to send email; these machines don't receive email
class = workq	use the queue "workq"
notification = error	notify me upon an error
node = 1	number of nodes I need
total_tasks = 8	number of processors to use is 8. This number can't be more than the number of processors in the node.
requirements = (ARCH == "Power5")	Where there is more than one architecture type use the Power5 architecture.
network.MPI = sn_single, not shared, US
queue	Now we put it into the queue

To submit the job you will:

llsubmit hello.cmd

llq -u username

Return to top of page

Moving files from site to site

tar -cvf myfiles.tar file1 dir1 file2 dir3

Compress files using gzip:

gzip filename

Transfer files:

Using either scp or sftp.

`

Return to top of page

Documentation

LONI:
https://www.loni.org/systems
https://docs.loni.org/wiki/Main_Page

User Support:

LONI: sys-help@loni.org or otrs@loni.org

Return to top of page

---

The statements and opinions included in these pages are those of Kathryn R. Traxler. Any statements and opinions included in these pages are not those of Louisiana State University or the LSU Board of Supervisors.
© 1998-2008

Mail to: Kathryn R. Traxler