- Basic Operations
1.1. File Operations
1.2. Text Operations
1.3. Directory Operations
1.4. SSH, System Info & Network Operations
1.5. Process Monitoring Operations - Basic Shell Programming
2.1. Variables
2.2. Array
2.3. String Substitution
2.4. Other String Tricks
2.5. Functions
2.6. Conditionals
2.7. Loops
2.8. Regex
2.9. Pipes - Tricks
- Debugging
- Multi-threading
Displays all environment variables. If you want to get details of a specific variable, use echo $VARIABLE_NAME
.
export
Example:
$ export
AWS_HOME=/Users/adnanadnan/.aws
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LESS=-R
$ echo $AWS_HOME
/Users/adnanadnan/.aws
whatis shows description for user commands, system calls, library functions, and others in manual pages
whatis something
Example:
$ whatis bash
bash (1) - GNU Bourne-Again SHell
whereis searches for executables, source files, and manual pages using a database built by system automatically.
whereis name
Example:
$ whereis php
/usr/bin/php
which searches for executables in the directories specified by the environment variable PATH. This command will print the full path of the executable(s).
which program_name
Example:
$ which php
/c/xampp/php/php
Clears content on window.
cat | chmod | chown | cp | diff | file | find | gunzip | gzcat | gzip | head |
less | lpq | lpr | lprm | ls | more | mv | rm | tail | touch |
It can be used for the following purposes under UNIX or Linux.
- Display text files on screen
- Copy text files
- Combine text files
- Create new text files
cat filename
cat file1 file2
cat file1 file2 > newcombinedfile
cat < file1 > file2 #copy file1 to file2
The chmod command stands for "change mode" and allows you to change the read, write, and execute permissions on your files and folders. For more information on this command check this link.
chmod -options filename
The chown command stands for "change owner", and allows you to change the owner of a given file or folder, which can be a user and a group. Basic usage is simple forward first comes the user (owner), and then the group, delimited by a colon.
chown -options user:group filename
Copies a file from one location to other.
cp filename1 filename2
Where filename1
is the source path to the file and filename2
is the destination path to the file.
Compares files, and lists their differences.
diff filename1 filename2
Determine file type.
file filename
Example:
$ file index.html
index.html: HTML document, ASCII text
Find files in directory
find directory options pattern
Example:
$ find . -name README.md
$ find /home/user1 -name '*.png'
Un-compresses files compressed by gzip.
gunzip filename
Lets you look at gzipped file without actually having to gunzip it.
gzcat filename
Compresses files.
gzip filename
Outputs the first 10 lines of file
head filename
Shows the contents of a file or a command output, one page at a time. It is similar to more, but has more advanced features and allows you to navigate both forward and backward through the file.
less filename
Check out the printer queue.
lpq
Example:
$ lpq
Rank Owner Job File(s) Total Size
active adnanad 59 demo 399360 bytes
1st adnanad 60 (stdin) 0 bytes
Print the file.
lpr filename
Remove something from the printer queue.
lprm jobnumber
Lists your files. ls
has many options: -l
lists files in 'long format', which contains the exact size of the file, who owns the file, who has the right to look at it, and when it was last modified. -a
lists all files, including hidden files. For more information on this command check this link.
ls option
Example:
$ ls -la rwxr-xr-x 33 adnan staff 1122 Mar 27 18:44 . drwxrwxrwx 60 adnan staff 2040 Mar 21 15:06 .. -rw-r--r--@ 1 adnan staff 14340 Mar 23 15:05 .DS_Store -rw-r--r-- 1 adnan staff 157 Mar 25 18:08 .bumpversion.cfg -rw-r--r-- 1 adnan staff 6515 Mar 25 18:08 .config.ini -rw-r--r-- 1 adnan staff 5805 Mar 27 18:44 .config.override.ini drwxr-xr-x 17 adnan staff 578 Mar 27 23:36 .git -rwxr-xr-x 1 adnan staff 2702 Mar 25 18:08 .gitignore
Shows the first part of a file (move with space and type q to quit).
more filename
Moves a file from one location to other.
mv filename1 filename2
Where filename1
is the source path to the file and filename2
is the destination path to the file.
Also it can be used for rename a file.
mv old_name new_name
Removes a file. Using this command on a directory gives you an error.
rm: directory: is a directory
To remove a directory you have to pass -r
which will remove the content of the directory recursively. Optionally you can use -f
flag to force the deletion i.e. without any confirmations etc.
rm filename
Outputs the last 10 lines of file. Use -f
to output appended data as the file grows.
tail filename
Updates access and modification time stamps of your file. If it doesn't exists, it'll be created.
touch filename
Example:
$ touch trick.md
awk | cut | echo | egrep | fgrep | fmt | grep | nl | sed | sort |
tr | uniq | wc |
awk is the most useful command for handling text files. It operates on an entire file line by line. By default it uses whitespace to separate the fields. The most common syntax for awk command is
awk '/search_pattern/ { action_to_take_if_pattern_matches; }' file_to_parse
Lets take following file /etc/passwd
. Here's the sample data that this file contains:
root:x:0:0:root:/root:/usr/bin/zsh
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
So now lets get only username from this file. Where -F
specifies that on which base we are going to separate the fields. In our case it's :
. { print $1 }
means print out the first matching field.
awk -F':' '{ print $1 }' /etc/passwd
After running the above command you will get following output.
root
daemon
bin
sys
sync
For more detail on how to use awk
, check following link.
Remove sections from each line of files
example.txt
red riding hood went to the park to play
show me columns 2 , 7 , and 9 with a space as a separator
cut -d " " -f2,7,9 example.txt
riding park play
Display a line of text
display "Hello World"
echo Hello World
Hello World
display "Hello World" with newlines between words
echo -ne "Hello\nWorld\n"
Hello
World
Print lines matching a pattern - Extended Expression (alias for: 'grep -E')
example.txt
Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.
display lines that have either "Lorem" or "dolor" in them.
egrep '(Lorem|dolor)' example.txt
or
grep -E '(Lorem|dolor)' example.txt
Lorem ipsum
dolor sit amet,
et dolore magna
duo dolores et ea
sanctus est Lorem
ipsum dolor sit
Print lines matching a pattern - FIXED pattern matching (alias for: 'grep -F')
example.txt
Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
foo (Lorem|dolor)
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.
Find the exact string '(Lorem|dolor)' in example.txt
fgrep '(Lorem|dolor)' example.txt
or
grep -F '(Lorem|dolor)' example.txt
foo (Lorem|dolor)
Simple optimal text formatter
example: example.txt (1 line)
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
output the lines of example.txt to 20 character width
cat example.txt | fmt -w 20
Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.
Looks for text inside files. You can use grep to search for lines of text that match one or many regular expressions, and outputs only the matching lines.
grep pattern filename
Example:
$ grep admin /etc/passwd
_kadmin_admin:*:218:-2:Kerberos Admin Service:/var/empty:/usr/bin/false
_kadmin_changepw:*:219:-2:Kerberos Change Password Service:/var/empty:/usr/bin/false
_krb_kadmin:*:231:-2:Open Directory Kerberos Admin Service:/var/empty:/usr/bin/false
You can also force grep to ignore word case by using -i
option. -r
can be used to search all files under the specified directory, for example:
$ grep -r admin /etc/
And -w
to search for words only. For more detail on grep
, check following link.
Number lines of files
example.txt
Lorem ipsum
dolor sit amet,
consetetur
sadipscing elitr,
sed diam nonumy
eirmod tempor
invidunt ut labore
et dolore magna
aliquyam erat, sed
diam voluptua. At
vero eos et
accusam et justo
duo dolores et ea
rebum. Stet clita
kasd gubergren,
no sea takimata
sanctus est Lorem
ipsum dolor sit
amet.
show example.txt with line numbers
nl -s". " example.txt
1. Lorem ipsum
2. dolor sit amet,
3. consetetur
4. sadipscing elitr,
5. sed diam nonumy
6. eirmod tempor
7. invidunt ut labore
8. et dolore magna
9. aliquyam erat, sed
10. diam voluptua. At
11. vero eos et
12. accusam et justo
13. duo dolores et ea
14. rebum. Stet clita
15. kasd gubergren,
16. no sea takimata
17. sanctus est Lorem
18. ipsum dolor sit
19. amet.
Stream editor for filtering and transforming text
example.txt
Hello This is a Test 1 2 3 4
replace all spaces with hyphens
sed 's/ /-/g' example.txt
Hello-This-is-a-Test-1-2-3-4
replace all digits with "d"
sed 's/[0-9]/d/g' example.txt
Hello This is a Test d d d d
Sort lines of text files
example.txt
f
b
c
g
a
e
d
sort example.txt
sort example.txt
a
b
c
d
e
f
g
randomize a sorted example.txt
sort example.txt | sort -R
b
f
a
c
d
g
e
Translate or delete characters
example.txt
Hello World Foo Bar Baz!
take all lower case letters and make them upper case
cat example.txt | tr 'a-z' 'A-Z'
HELLO WORLD FOO BAR BAZ!
take all spaces and make them into newlines
cat example.txt | tr ' ' '\n'
Hello
World
Foo
Bar
Baz!
Report or omit repeated lines
example.txt
a
a
b
a
b
c
d
c
show only unique lines of example.txt (first you need to sort it, otherwise it won't see the overlap)
sort example.txt | uniq
a
b
c
d
show the unique items for each line, and tell me how many instances it found
sort example.txt | uniq -c
3 a
2 b
2 c
1 d
Tells you how many lines, words and characters there are in a file.
wc filename
Example:
$ wc demo.txt
7459 15915 398400 demo.txt
Where 7459
is lines, 15915
is words and 398400
is characters.
cd | mkdir | pwd |
Moves you from one directory to other. Running this
$ cd
moves you to home directory. This command accepts an optional dirname
, which moves you to that directory.
cd dirname
Switch to the previous working directory
cd -
Makes a new directory.
mkdir dirname
You can use this to create multiple directories at once within your current directory.
mkdir 1stDirectory 2ndDirectory 3rdDirectory
You can also use this to create parent directories at the same time with the -p (or --parents) flag. For instance, if you wanted a directory named 'project1' in another subdirectory at '/samples/bash/projects/', you could run:
mkdir -p /samples/bash/projects/project1
mkdir --parents /samples/bash/projects/project1
Both commands above will do the same thing. If any of these directories did no already exist, they would be created as well.
Tells you which directory you currently are in.
pwd
bg | cal | date | df | dig | du | fg | finger | jobs | last |
man | passwd | ping | ps | quota | scp | ssh | top | uname | uptime |
w | wget | whoami | whois | sync | curl |
Lists stopped or background jobs; resume a stopped job in the background.
Shows the month's calendar.
Shows the current date and time.
Shows disk usage.
Gets DNS information for domain.
dig domain
Shows the disk usage of files or directories. For more information on this command check this link
du [option] [filename|directory]
Options:
-h
(human readable) Displays output it in kilobytes (K), megabytes (M) and gigabytes (G).-s
(supress or summarize) Outputs total disk space of a directory and supresses reports for subdirectories.
Example:
du -sh pictures
1.4M pictures
Brings the most recent job in the foreground.
Displays information about user.
finger username
Lists the jobs running in the background, giving the job number.
Lists your last logins of specified user.
last yourUsername
Shows the manual for specified command.
man command
Allows the current logged user to change their password.
Pings host and outputs results.
ping host
Lists your processes.
ps -u yourusername
Use the flags ef. e for every process and f for full listing.
ps -ef
Shows what your disk quota is.
quota -v
Transfer files between a local host and a remote host or between two remote hosts.
copy from local host to remote host
scp source_file user@host:directory/target_file
copy from remote host to local host
scp user@host:directory/source_file target_file
scp -r user@host:directory/source_folder target_folder
This command also accepts an option -P
that can be used to connect to specific port.
scp -P port user@host:directory/source_file target_file
ssh (SSH client) is a program for logging into and executing commands on a remote machine.
ssh user@host
This command also accepts an option -p
that can be used to connect to specific port.
ssh -p port user@host
Displays your currently active processes.
Shows kernel information.
uname -a
Shows current uptime.
Displays who is online.
Downloads file.
wget file
Return current logged in username.
Gets whois information for domain.
whois domain
Does the same job as scp
command, but transfers only changed files. Useful when transferring the same folder to/from server multiple times.
rsync source_folder user@host:target_folder
rsync user@host:target_folder target_folder
Curl is a command-line tool for requesting or sending data using URL syntax. Usefull on systems where you only have terminal available for making various requests.
curl url
Use -X
or --request
to specify which method you would like invoke (GET, POST, DELETE, ...).
Use -d <data>
or --data <data>
to POST data on given URL.
kill | killall | & | nohup |
Kills (ends) the processes with the ID you gave.
kill PID
Kill all processes with the name.
killall processname
The &
symbol instructs the command to run as a background process in a subshell.
command &
nohup stands for "No Hang Up". This allows to run command/process or shell script that can continue running in the background after you log out from a shell.
nohup command
Combine it with &
to create background processes
nohup command &
The first line that you will write in bash script files is called shebang
. This line in any script determines the script's ability to be executed like a standalone executable without typing sh, bash, python, php etc beforehand in the terminal.
#!/usr/bin/env bash
Creating variables in bash is similar to other languages. There are no data types. A variable in bash can contain a number, a character, a string of characters, etc. You have no need to declare a variable, just assigning a value to its reference will create it.
Example:
str="hello world"
The above line creates a variable str
and assigns "hello world" to it. The value of variable is retrieved by putting the $
in the beginning of variable name.
Example:
echo $str # hello world
Like other languages bash has also arrays. An array is a variable containing multiple values. There's no maximum limit on the size of array. Arrays in bash are zero based. The first element is indexed with element 0. There are several ways for creating arrays in bash which are given below.
Examples:
array[0]=val
array[1]=val
array[2]=val
array=([2]=val [0]=val [1]=val)
array=(val val val)
To display a value at specific index use following syntax:
${array[i]} # where i is the index
If no index is supplied, array element 0 is assumed. To find out how many values there are in the array use the following syntax:
${#array[@]}
Bash has also support for the ternary conditions. Check some examples below.
${varname:-word} # if varname exists and isn't null, return its value; otherwise return word
${varname:=word} # if varname exists and isn't null, return its value; otherwise set it word and then return its value
${varname: word} # if varname exists and isn't null, return word; otherwise return null
${varname:offset:length} # performs substring expansion. It returns the substring of $varname starting at offset and up to length characters
Check some of the syntax on how to manipulate strings
${variable#pattern} # if the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest
${variable##pattern} # if the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest
${variable%pattern} # if the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest
${variable%%pattern} # if the pattern matches the end of the variable's value, delete the longest part that matches and return the rest
${variable/pattern/string} # the longest match to pattern in variable is replaced by string. Only the first match is replaced
${variable//pattern/string} # the longest match to pattern in variable is replaced by string. All matches are replaced
${#varname} # returns the length of the value of the variable as a character string
Bash has multiple shorthand tricks for doing various things to strings.
${variable,,} #this converts every letter in the variable to lowercase
${variable^^} #this converts every letter in the variable to uppercase
${variable:2:8} #this returns a substring of a string, starting at the character at the 2 index(strings start at index 0, so this is the 3rd character),
#the substring will be 8 characters long, so this would return a string made of the 3rd to the 11th characters.
Here are some handy pattern matching tricks
if [[ "$variable" == *subString* ]] #this returns true if the provided substring is in the variable
if [[ "$variable" != *subString* ]] #this returns true if the provided substring is not in the variable
if [[ "$variable" == subString* ]] #this returns true if the variable starts with the given subString
if [[ "$variable" == *subString ]] #this returns true if the variable ends with the given subString
The above can be shortened using a case statement and the IN keyword
case "$var" in
begin*)
#variable begins with "begin"
;;
*subString*)
#subString is in variable
;;
*otherSubString*)
#otherSubString is in variable
;;
esac
As in almost any programming language, you can use functions to group pieces of code in a more logical way or practice the divine art of recursion. Declaring a function is just a matter of writing function my_func { my_code }. Calling a function is just like calling another program, you just write its name.
function name() {
shell commands
}
Example:
#!/bin/bash
function hello {
echo world!
}
hello
function say {
echo $1
}
say "hello world!"
When you run the above example the hello
function will output "world!". The above two functions hello
and say
are identical. The main difference is function say
. This function, prints the first argument it receives. Arguments, within functions, are treated in the same manner as arguments given to the script.
The conditional statement in bash is similar to other programming languages. Conditions have many form like the most basic form is if
expression then
statement where statement is only executed if expression is true.
if [ expression ]; then
will execute only if expression is true
else
will execute if expression is false
fi
Sometime if conditions becoming confusing so you can write the same condition using the case statements
.
case expression in
pattern1 )
statements ;;
pattern2 )
statements ;;
...
esac
Expression Examples:
statement1 && statement2 # both statements are true
statement1 || statement2 # at least one of the statements is true
str1=str2 # str1 matches str2
str1!=str2 # str1 does not match str2
str1<str2 # str1 is less than str2
str1>str2 # str1 is greater than str2
-n str1 # str1 is not null (has length greater than 0)
-z str1 # str1 is null (has length 0)
-a file # file exists
-d file # file exists and is a directory
-e file # file exists; same -a
-f file # file exists and is a regular file (i.e., not a directory or other special type of file)
-r file # you have read permission
-s file # file exists and is not empty
-w file # you have write permission
-x file # you have execute permission on file, or directory search permission if it is a directory
-N file # file was modified since it was last read
-O file # you own file
-G file # file's group ID matches yours (or one of yours, if you are in multiple groups)
file1 -nt file2 # file1 is newer than file2
file1 -ot file2 # file1 is older than file2
-lt # less than
-le # less than or equal
-eq # equal
-ge # greater than or equal
-gt # greater than
-ne # not equal
There are three types of loops in bash. for
, while
and until
.
Different for
Syntax:
for name [in list]
do
statements that can use $name
done
for (( initialisation ; ending condition ; update ))
do
statements...
done
while
Syntax:
while condition; do
statements
done
until
Syntax:
until condition; do
statements
done
They are a powerful tool for manipulating and searching text. Here are some examples of regular expressions that use each metacharacter
:
`.`(dot) | `*`(asterisk) | ` `(plus) | `?`(question mark) | `|`(pipe) | `[]`(character class) | `[^]`(negated character class) | `()`(grouping) | `{}`(quantifiers) | `\`(escape) |
Matches any single character except newline.
grep h.t file.txt
Output:
hat
hot
hit
Matches zero or more occurrences of the preceding character or group.
grep ab*c file.txt
Output:
ac
abc
abbc
abbbc
Matches one or more occurrences of the preceding character or group.
grep ab c file.txt
Output:
abc
abbc
abbbc
abbbbc
Matches zero or one occurrence of the preceding character or group.
grep ab?c file.txt
Output:
ac
abc
Matches either the pattern to the left or the pattern to the right.
egrep "cat|dog" file.txt
Output:
cat
dog
Matches any character inside the brackets.
[aeiou] will match any vowel
[a-z] will match any lowercase letter
Matches any character not inside the brackets.
[^aeiou] will match any consonant
[^a-z] will match any non-lowercase letter
Groups multiple tokens together and creates a capture group.
egrep "(ab) " file.txt
Output:
ab
abab
ababab
Matches a specific number of occurrences of the preceding character or group.
egrep "a{3}" file.txt
Output:
aaa
aaaa
aaaaa
Escapes the next character to match it literally.
egrep "a\ " file.txt
Output:
a
=======
Multiple commands can be linked together with a pipe, |
. A |
will send the standard-output from command A to the standard-input of command B.
Pipes can also be constructed with the |&
symbols. This will send the standard-output and standard-error from command A to the standard-input of command B.
Run nano ~/.bash_profile
and add the following line:
alias dockerlogin='ssh [email protected] -p2222' # add your alias in .bash_profile
Run nano ~/.bashrc
and add the following line:
export hotellogs="/workspace/hotel-api/storage/logs"
Now you can use the saved path:
source ~/.bashrc
cd $hotellogs
This goes back to the days before you could rely on keyboards to have an "up" arrow key, but can still be useful. To run the last command in your history
!!
A common error is to forget to use sudo
to prefix a command requiring privileged execution. Instead of typing the whole command again, you can:
sudo !!
This would change a mkdir somedir
into sudo mkdir somedir
.
Make your bash scripts more robust by reliably performing cleanup.
function finish {
# your cleanup here. e.g. kill any forked processes
jobs -p | xargs kill
}
trap finish EXIT
When you do export FOO = BAR
, your variable is only exported in this current shell and all its children, to persist in the future you can simply append in your ~/.bash_profile
file the command to export your variable
echo export FOO=BAR >> ~/.bash_profile
You can easily access your scripts by creating a bin folder in your home with mkdir ~/bin
, now all the scripts you put in this folder you can access in any directory.
If you can not access, try append the code below in your ~/.bash_profile
file and after do source ~/.bash_profile
.
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
You can easily debug the bash script by passing different options to bash
command. For example -n
will not run commands and check for syntax errors only. -v
echo commands before running them. -x
echo commands after command-line processing.
bash -n scriptname
bash -v scriptname
bash -x scriptname
You can easily multi-threading your jobs using &
. All those jobs will then run in the background simultaneously and you can see the processes below are running using jobs
.
sleep 15 & sleep 5 &
The optional wait
command will then wait for all the jobs to finish.
sleep 10 & sleep 5 &
wait