More Linux

Viewing Files

• cat - shows you the contents of an entire file (you will see only the last screen full of lines)
  • the command is cat filename

  • cat test.txt

• more - shows you the contents of an entire file one screen at a time
  • the command is more filename

  • more test.txt

  • to go to the next screen hit the space bar
  • to see only the next line hit the Return key
  • to quit more hit the "q" key
• head - shows you the first 10 lines of a file
  • the command is head filename

  • head text.txt

  • to see fewer or more lines us the -n option

    • head -3 test.txt

  • tail - shows you the last 10 lines of a file

    • tail test.txt

    • tail -3 test.txt

    • shows the last 3 lines of file test.txt
  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

• Wild Cards to Use with Commands

  Wild cards can be used with certain commands to create patterns for filenames. The wildcards allow you to create many types of filenames. Some commands that wildcards can be used with are:
  • ls
  • cp
  • mv
  • rm
  
  The wildcards are:
  • ? - allows the character in this location to be replaced with any other character in the ASCII character set.

  • ls t?st.txt

  • lists all files that start with "t" and end with "st.txt" and have the second letter any character.
  • so test.txt, tist.txt, t?st.txt, t9st.txt, tmst.txt would all be shown.

  • ls ?

  • lists all files that have names with only one character
  • so a, b, 3, _, z, 9 would all be shown
  • * - replaces any number of characters.

  • ls p*

  • lists every file that starts with "p" in the directory

  • ls t*t

  • lists every file that starts and ends with "t" no matter how many other letters or characters are in it
  • [] = replaces any single character inside the brackets

  • ls t[aeo]st.txt

  • lists every file that starts with a "t", has either an "a", "e", or "o" as the second character and ends with "st.txt"
  • you can also specify a range of characters in the brackets, i.e.:
    • [a-z] any file that has any of the lower case characters from a to z inclusive
    • [!-+] any file that has any of the special characters: "!", "@", "#", "$", "%", "^", "&", "*", "(", ")", "_", "+"

  • ls t[a-z]st.txt

  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

• Learning Vi

  Vi is a text editor. It comes on every UNIX like system in the base install. There are many other text editors and different people believe different ones are the best. I am going through vi here because you can always find it on a UNIX like OS unless the system administrator has removed it (Not likely).

  Vi stands for visual editor.
  

  • To create a file, choose a name that doesn't exist in the directory
  • vi newfile This creates the new file and puts you in command mode, one of vi's two modes.
  
  Vi has two modes:
  • command mode - allows you to use the normal keys to give movement and text editing commands
    • move up or down one character, one line or one page
    • substitute one pattern for another
    • ...
  • text mode - allows you to use the normal keys to type the characters needed for your file.

  Command mode:
  This is the mode that vi comes up in. If the file doesn't exist then you will see a screen with tilde's "~". If the file does exist you will see the first page of text in the file. Lets create a new file - use your name followed by .txt. So my file would be named "kathy.txt".
  

  vi kathy.txt

  Vi basic movement commands. You must be in command mode to move around to get into command mode press the escape key . Once you are in command mode you will use the h, j, k, and l keys to move around.

  • H - moves you to the first character of the "highest" line on the screen
  • M - moves you to the first character at the "middle" of the screen
  • L - moves you to the first character at the "bottom" of the screen
  • h - moves you to the left on the current line
  • j - moves you down to the next line in the file
  • k - moves you up to the next line in the file
  • l - moves you to the right on the current line
  • $ - moves you to the end of the current line
  • ^ - moves you to the beginning of the current line
  • Control key plus "f" moves you one page toward the beginning of the file
  • Control key plus "b" move you one page toward the end of the file
  • type the colon key, ':', followed by a line number to go to that line

  Because this is a new file there is nowhere inside of the file to go. So the cursor can't move around. We have to go to text mode and create text so that we have something to move around in.
  Use the following commands to add text to a file:
  
  Vi Text mode:
  Vi basic insert text commands:

  • i - insert new text to the left of the character the cursor is on
  • I - insert new text to the left of the first non-blank character on the line
  • a - insert new text to the right of the character the cursor is on
  • A - insert new text to the right of the last character on the line
  • o - inserts a new line below the line the cursor is on
  • O - inserts a new line above the line the cursor is on

  Now that we know the basic commands lets finish adding text to this file:

  • Copy the text off the web page
    hit the 'i' key for insert mode
    Paste the text into the file
    hit the escape key Type :wq

  • ":wq" - the ":" tells vi that a command is coming up next
    the "w" tells vi to write the file to disk (save the file)
    the "q" tells vi to quit
    If you don't type the "q" vi saves the file and lets you continue

  • Now that you have text to move around in practice using the movement commands

  • Go into the file, and insert a word on the first line after the third word

  • Save the file

  • Get into text mode and go to the third line. Insert a new line after the third line and type "All good boys do fine."

  • Get into command mode, to to the 8th character on the first line and replace it with a "+". To replace it put the cursor on the 8th character, type a lower case "r" then type the plus sign.

  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  Getting Help from the System

  The basic way to get help on the system:
  • Man pages are the way that you check out the syntax of a command in UNIX like operating systems. There are on-line manuals which give you information on the commands: syntax, options, etc. To access these pages for a command you would type "man command-name". Man pages are presented using the "more" option for file viewing.

  • Type:
    man cp

  • This gives the information on the cp (copy) command.
  • whatis command-name
  • The command, whatis, gives a one-line description of the command but doesn't include the options.

  • Type:
    whatis cp

  • apropos keyword
  • Gives the command with this keyword in their description in a "cat" file viewing style.

  • Typed:
    apropos copy

  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  File System Access/Security

  When you execute the command "ls -l" (list the contents of the directory in long form) you will see something that looks like this:
  [ktraxler@is linux_II]$ ls -l
  total 80
  -rwxr-xr-x 1 ktraxler internalusers 48928 Jan 28 2005 linux.html
  -rwxr-xr-x 1 ktraxler internalusers 15650 Aug 2 10:58 linuxII.html
  -rwxr-xr-x 1 ktraxler internalusers 15795 Aug 2 09:54 linuxII.html.bak
  
  The group of characters to the left are the access rights or permissions of each file, directory or link. The first character tells whether this is a file, directory or link. The values would be:
  • '-' means a file
  • 'd' means a directory
  • 'l' means a symbolic link (like an alias or shortcut)
  The remaining symbols have context in groups of three giving the permissions of access for different groups. These permissions allow the reading, writing and execution (or pass through) for a person or group.
  • The first group of three are the permissions the owner of the file (account) has.
  • The middle three are the permissions for the group that file belongs to.
  • The last three give the permissions for anyone not the owner, or in the group (everyone else on the machine.
  
  In each group of three you will have the following permissions:
  • The first character in the group gives the read permission and can be either a 'r' or a '-'. This indicates whether or not you have permission to read the file or access the directory.
  • The second character in the group gives the write permission and can be either 'w' or '-'. This indicates whether or not you have permission to write to the file or add or remove files from the directory.
  • The third character gives the execution permission, if it is appropriate, and will be either 'x' or '-'. If this is an executable (from a program or application) you can actually run it, if it is a directory you can pass through to access the contents.
  To read a file you must have read access on the file itself and you must have executable access to the directory it is in (so you can get into it) and to any directories that are ancestors of this directory (come before).
  

  Examples of permissions:

  -rwx------	a file that only the owner can access. The owner can read, write or execute the file.
  -rwxr--r--	a file that everyone can read using vi, more, cat, etc. But only the owner can change the contents or delete it.
  drwxr-x---	a file that the owner and the owner's group can access. The owner can read, write or execute but the group can only read and execute.

  To see this on your account type:

  ls -l

  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  Changing File Access

  You have to be very careful with this type of action, if you change permissions on the wrong files you can change the operation of your computer!Only the owner of a file or directory can use the command to change the file access. The command to change this is "chmod", changing mode. The syntax of the command is: chmod ugoa+[-]rwx file-name. The plus adds a permission for an either the owner, group, or other. The minus removes that permission.

  ls -l

  
  Now, look at the permissions on the file your-name.txt. Then type:
  

  chmod a-w your-name.txt; ls -l

  
  (The semi-colon allows you to type multiple commands on one line. The commands are then executed in the order typed.)
  
  You have just changed the access on file your-name.txt so that the owner, the group and others may write to that file, if they have permission to get into the directory that contains it.

  Now, lets change the permissions so that the group has no access and other has all access modes. First type:

  ls -l

  
  Then type:

  chmod g-rwx kathy.txt; ls -l

  
  This changes the permissions for the file, "your-name.txt" so that the group has no permissions. Now you need to change the permissions so that all other users has every access permission:

  chmod o+rwx kathy.txt
  ls -l

  .
  There is another way to set permissions and that is by adding the numeric values assigned to the permissions for each group. I am told this is the way a REAL UNIX/Linux user changes permissions! The permissions have the values of:
  • Read == 4
  • Write == 2
  • Execute == 1
  So, for example the file permissions used in the examples above become:

  -rwx------	700
  -rwxr--r--	744
  drwxr-x---	750

  So to change a file or directory's permissions so that it looked like the first one:
  

  chmod 700 filename
  
  To change a file so ther permission looks like the second one:
  chmod 744 filename
  
  Your home directy is the place on the Unix/Linux server that you log into and your information is stored. This is represented by a tilde, "~". You can run the chmod command on your home directoy. The safest setting is: chmod 700 ~.

  ange a file so ther permission looks like the second one:
  

  chmod 744 filename
  
  Your home directy is the place on the Unix/Linux server that you log into and your information is stored. This is represented by a tilde, "~". You can run the chmod command on your home directoy. The safest setting is: chmod 700 ~.

  However, if you do this you can't serve any web pages that you have created because the web server doesn't have access to move through the directories to get to "public_html". The recommanded setting for your home directory, from many tutorials and textbooks, is: chmod 711 ~. This will allow the user full access and all others can pass through the home directory.
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  Grep

  The grep command is useful for finding strings of characters in text files. It is short for "Get Regular Expression and Print". To find all users using the bash shell on this machine:

  grep bash$ /etc/passwd

  The "$" sign is used to find lines ending with the characters "bash". You have to know the format of the file you want but with the /etc/passwd file the line ends with the shell.

  ps uax | grep $USER

  This command says:
  "ps" gives information about a selection of the active processes. By default it selects all process with the same effective user ID as the current user. Adding the options "uax" is a BSD syntax for viewing every process on the system with the resources you are using. Then you pipe it through grep "| grep" with for strings that match the regular expression defined by $USER (me in this case).

  M

  find . -name "*.xml"

  Now try it so we can see some specific subset of files:
  

  find . -name "*.xml" | grep $USER

  Grep Tutorials:

  • Grep Tutorial from Rutgers
  • Drew's Grep Tutorial
  • Regular-Experssions.info Grep Tutorial
  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  Awk

  Awk is another text processing tool that will make your hunt for data a lot easier. It is actually a programming language designed for text manipulation but is widely used as an on the spot fool for administration.
  Take the example on process from grep:

  ps uax | grep $USER

  now try:

  ps uax | awk '/ktraxler/'

  This is just another way to write this command and get the same information.

  ps aux | awk '/ktraxler/' {print $1, $2, $4}'

  The above command prints the first, second and fourth column of the table created by "ps aux".

  awk can do math also:

  ps uax | awk '/ktraxler/ {x += $4} END {print "total memory: " x } '

  This command gives the total memory for all of the processes for user "ktraxler".
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  The *nix Shell

  The shell is your workspace in the terminal. It is a command language interpreter that executes commands read from the standard input device or from a file.

  Shell Name	Developed by	Where	Remark
  BSH (Bourne-Again SHell)	Brian Fox and Chet Ramey	Free Software Foundation	Most common shell in Linux
  CSH (C SHell)	Bill Joy	University of California (For BSD)	The C shell'ssyntax and usage are very similar to the C programming language.
  KSH (Korn SHell)	David Korn	AT & T Bell Labs
  TCHS	See the man page. Type $ Man tcsh	--	TCSH is an enhanced but completely compatible version of the Berkeley UNIX C shell (CSH). The 't' in tcsh comes from the T in TENEX, an operating system which inspired Ken Greer, the author of tcsh, with its command-completion feature.

  To find available shells in your system type:
  cat /etc/shells

  Cat types out the contents of a file as you saw above. The file "shells" in the "/etc" directory holds all available shells in the system.

  To find your current shell type:
  echo $SHELL

  The command "echo" echos or types what follows it. The "$" sign says echo the contents of the following system variable. The "SHELL" is a system variable set up in your init file or by using the "export" command.

  To find out where your shell and other basic information is stored:

  To get back to your home directory type:
  cd
  Now, to list all files including the hidden ones type:
  ls -al

  class=action> Here is what it will look like on "Tezpur":
  
  
  

  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  Other Useful Commands

  Some of the other commands that you might find useful are:

  quota	shows you your quotas on the machine in column format with headers
  usage:	quota -v
  dr	reports the space left on the file system
  usage:	dr
  du	outputs the number of kilobyes used by each subdirectory, use in home directory
  usage:	du
  gzip	Compresses a file
  usage:	gzip filename
  gunzip	Uncompress a file
  usage:	gunzip filename
  file	classifies the named files according to the type of data they contain
  usage:	file *
  date	gives you the system date
  usage:	date

  
  

  ---

  
  

  Return to top of page

  
  

  ---

  
  

  Environment Variables

  What are they?

  The shell's environment is that umbrella of persistent knowledge under which the shell operates. This information is stored as "environmental variables", which are global variables that may be used at any point by the shell -- either internally or when explicitly required by the user. Other variables can be customized by the user using a specific set of files contained in their HOME directory. (Thanks to Brett Estrade for this paragraph.)

  Why should I know about them

Being able to know which environment variables affect which aspect of your environment means that you are able to customize your environment. In many shell scripts and using many applications or programming languages you will need to know the values of these variables.

Your global environment consists of environment varibles such as:

• PATH the variable that allows us to locate a command. Which gives all of the directories that the system should search through when looking for a specific command. To see what is in your path:

  echo $PATH

• PWD which holds the path of the working directory:

  echo $PWD

• HOME which holds the user's home directory:

  echo $HOME

• USER holds the user name of the current account:

  echo $USER

See the values

• To see the overall value of YOUR environment:

  env | more

• Another way to print the contents of your environment:

  printenv GLOBUS_PATH

Change the values

Here is a place for a warning: Do NOT change environment variables unless you understand completely what you are doing.

Suppose I setup my machine and didn't install JAVA because I'm a C and Fortran person. Now I have installed JAVA and need to make sure I can get to it.

Now, before we start changing things we have to know which shell "family" we are in. There are two major families of shell:

• Bourne Shell:
• sh
• bash
• ksh
• These are good as a login shell and as a basis for shell scripting
• C-Shell:
• csh
• tchs
• Good as a login shell. Avoid using it for shell scripting.

To create or modify a global environment you must use either "export" in the bash shells or "setenv" in the csh environment.

Bourne shell: To add a new place to look in your PATH variable:

export PATH=/admin/bin:PATH

C-shell:

setenv PATH "/admin/bin:PATH

The difference here is only the syntax of the command due to the differing shells.


---




Return to top of page




---


When you login the system default environment is set using system default files. For the Bourne shell family this would be: /etc/profile . For the C-shell family it is the /etc/csh.cshrc . These files automatically set up the default path and vital user system variables.

Bash users may create the following files in their home directory:

• ~/.bash_profile # 1. run first after /etc/profile if exists**
• ~/.bashrc # 2. run after ~/.bash_profile if exists
• ~/.profile # 3. run after ~/.bashrc if exists
• bash users may also create a file that is executed when one logs out:

  ~/.bash_logout

Remember:
~/.bash_profile must not produce any standard out, since it will break tools such as rsync.


---




Return to top of page




---




Beginning Shell Scripting

You use some text editor to write your shell scripts. After writing your scripts you n eed to change the permissions so the execute permission is on for the user:

chmod u+x myscript

. You will then need to execute your script: ./myscript

> The "hello world" program in bash shell script:

vi kathy1
#
#
#This is my first shell script!
#
clear
echo "Hello World - this is my first shell script!"
#


Now, execute the script

./kathy1

This script doesn't execute because we didn't change the permissions. So change those.< br> Now execute the script as above.

Lets create another script:

vi ginfo
#
#
# Script to print user information who currently login , current date & time
#
clear
echo "Hello $USER"
echo "Today is \c ";date
echo "Number of user login : \c" ; who | wc -l
echo "Calendar"
cal
exit 0

Now, set the permissions as before, then execute the script just like before.

Now, set the permissions as before, then execute the script just like before.
For those of us that don't use *nix languages all the time ideas for scripts are few an d far betweens but system administrators write them on a daily basis. They also use th e commands from the command line daily.

A good starting guide is: Vivek G. Gite's Shell Scripting Tutorial


---




Return to top of page




---




Shell Scripting Round Two

So we are going to look at two examples of shell scripting. The examples are trivial but they come from situations you will probably run into as you use the systems more.

Remember that shell scripting is simply solving a problem with a series of Linux/UNIX system commands.

Exercise 1

(1) Find out how many files contain a certain string in a directory and its subdirectories.

Practical application: I once worked on a package which would automatically create bunch of makefiles all having bad linking options.

To solve this one we need to:

• Find all the files that have a particular string
• Save only the file name of the files that have that string
• make sure that we get rid of duplicates
• count lines

Grep

To do the first we need to find a way to search through all files' content for a particular string. I mentioned in Introduction to Linux that grep was a utility that did this. The grep command stands for "Get Regular Experssion and Print". To find all users using the bash shell on this machine go to the prompt and type:

grep bash$ /etc/passwd



The "$" sign is used to find lines ending with the characters "bash". You have to know the format of the file you want but with the /etc/passwd file the line ends with the shell.

This command says: "ps" gives information about a selection of the active processes. By default it selects all process with the same effective user ID as the current user. Adding the options "uax" is a BSD syntax for viewing every process on the system with the resources you are using. Then you pipe it through grep "| grep" with for strings that match the regular expression defined by $USER (me in this case).

ps uax | grep $USER



Awk

Awk Awk is another text processing tool that will make your hunt for data a lot easier. It is actually a programming language designed for text manipulation but is widely used as an on the spot fool for administration. Take the example on process from grep:

ps uax | grep $USER

now try:

ps uax | awk '/ktraxler/'



Notice that awk prints out fields with the ':' as the separator. So if we know what a command outputs, and we want only a few certain fields from that output we can start with field #1 and go over.

This is just another way to write this command and get the same information.

ps aux | awk '/ktraxler/ {print $1, $2, $4}'



Now that we can see what awk and grep can do lets go back to solving our problem. The steps were:

• Find all the files that have a particular string
• Save only the file name of the files that have that string
• make sure that we get rid of duplicates
• count lines
To get rid of the duplicates we must sort uniquely. The "sort" command is used to do this with the "-u" switch.

To count the lines giving us the number of unique files with that string in it we do "wc -l".

The script will look like this: #open a new file for the shell script vi exer1.bash
#use grep to find every file in the /etc/re0.d directory that has the string "bash" in it: grep -i bash /etc/rc0.d/* | awk -F ':' '{print $1}' | sort -u | wc -l #the '|' bar means pipe the output of the last command into the next one as input #the grep command above pipes its output (a series of lines with the path) into awk #the awk command tells it ot use the ":" that the grep uses to separate the fields to tell what field we are at # #awk prints the first field and pipes it into the sort command #the "sort -u" command sorts the output uniquely and removes all duplicates #the "wc -l" counts the lines and tells us how many files there are in that directory and its sub-directories# The image of using this series as an single command instead of a file is:

vi exer1.bash
grep -i bash /etc/rc0.d/* | awk -F ':' '{print $1}' | sort -u | wc -l




---




Return to top of page




---




---

The statements and opinions included in these pages are those of Kathryn R. Traxler only. Any statements and opinions included in these pages are not those of Louisiana State University or the LSU Board of Supervisors.
© 1998-2006

Mail to: Kathryn R. Traxler

This page last updated November 8, 2006 CST