Normally by itself the terminal app displays bash in the tab title and its not very useful if you have a large number of tabs open and want to know e.g., what is the pwd in each of the tabs. add this line in ~/.bash_profile so that the tab title displays the pwd (source: http://superuser.com/a/560393):
export PROMPT_COMMAND='echo -ne "\033]0;$(pwd)\007"'
once you modify your bash_profile, to apply the settings run . ~/.bash_profile from the cmd prompt.
to count lines in a file
wc -l log.txt
Count number of columns in a csv file
head -n 1 file.csv | sed 's/[^,]//g' | wc -c
find . -type f -exec cat {} + | wc -l
this also worked:
find src -name "*.ts" | xargs wc -l
sed and awk
to extract second column of a file, try:
awk 'BEGIN { FS = "," } ;{print $2}' log.txt
FS defines the field delimiter. summing all numbers in the third column of a text file: awk '{ x += $3 } END { print x }' file.txt
Say you have a file like this where the fields are delimited by | character:
00k2T000015uSR3QAM|0063400001Aw8epAAB|true|""|-18.00|4717430071|false|12.0|""|holy
00k2T000015v5mjQAA|0063400001BXKLvAAP|true|""|172.50|4717882609|false|12.0|""|holy
00k34000010oIxbAAE|006340000191XVqAAM|true|""|4000.00|4715645568|false|""|""|logy
00k34000013JUYXAA4|0063400001APiUOAA1|true|""|0.00|4716969707|false|12.0|""|hnol
00k2T00001Ga160QAB|0062T00001MD7dQQAT|true|""|4467.82|""|false|0.0|""|hnol
On line number 5 the 8th column is supposed to be a number but has "" in it. We want to fix that. So the task is to compare the 8th column of every line with "" and if there is a match replace it with 0. Here is how you can do it with awk:
$ awk 'BEGIN{OFS=FS="|"}$8=="\"\""{$8=0}{print}' sample.csv
sort
to find minimum number using sorting:
awk '{print $2}' log.txt | sort -n
to sort the lines by filesize:
sort -k2 -n log.txt > sorted.txt
-n tells to interpret key as number
-k2 tells to use the second field as sort key
Search for files recursively in a directory
find . -name myfile.txt
find only *.php files:
Grep
Find a string searching only the php files:
grep somestring $(find . -name "*.php")
Recursively find the number of files in a directory
Get hostname
$ hostname
Siddharths-MacBook-Pro-1674.local
https://github.com/jlevy/the-art-of-command-line
Know about cut, paste, and join to manipulate text files. Many people use cut but forget about join.
Know about wc to count newlines (-l), characters (-m), words (-w) and bytes (-c)
To replace all occurrences of a string in place, in one or more files:
perl -pi.bak -e 's/old-string/new-string/g' my-files-*.txt
To rename many files at once according to a pattern, use rename. For complex renames, repren may help.
Check what OS you’re on with uname or uname -a (general Unix/kernel info) or lsb_release -a (Linux distro info).
uname -a
Darwin Siddharths-MacBook-Pro-1674.local 14.5.0 Darwin Kernel Version 14.5.0: Tue Sep 1 21:23:09 PDT 2015; root:xnu-2782.50.1~1/RELEASE_X86_64 x86_64
Know sort’s options. For numbers, use -n, or -h for handling human-readable numbers (e.g. from du -h). Know how keys work (-t and -k). In particular, watch out that you need to write -k1,1 to sort by only the first field; -k1 means sort according to the whole line. Stable sort (sort -s) can be useful. For example, to sort first by field 2, then secondarily by field 1, you can use sort -k1,1 | sort -s -k2,2.
to total file sizes in a directory
WITSC02X6385JGH:manuscript sjain68$ du -ach chap*.adoc
116K chap01.adoc
76K chap02.adoc
40K chap03.adoc
56K chap04.adoc
84K chap05.adoc
72K chap06.adoc
56K chap07.adoc
36K chap08.adoc
24K chap09.adoc
100K chap10.adoc
44K chap11.adoc
76K chap12.adoc
60K chap13.adoc
8.0K chap14.adoc
848K total
find command
find command is used to find all files matching a pattern in a directory. E.g., list all files ending in .sql:
It searches the directory recursively by default. Above command will not list the sql files in a folder that contains spaces. To fix that use following form:
To zip files on Mac. link
BASH_SOURCE
This gives a relative path. To get absolute path use:
BASE_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
An array variable whose members are the source filenames where the corresponding shell function names in the FUNCNAME array variable are defined. The shell function ${FUNCNAME[$i]} is defined in the file ${BASH_SOURCE[$i]} and called from ${BASH_SOURCE[$i+1]}
Restart Shell
Find your IP address
3 ways of doing this. simplest way on Mac is to open Network and read it from there. Another way is to run following command (works only on Mac):
Here en0 is the identifier of the network interface (WiFi vs Ethernet for example). To my knowledge above two ways will return the local IP address. If you want to get your router’s IP address (i.e., your public IP address), run:
Split large file into chunks
will split lineorder.tbl into chunks named lineorder_00.csv, lineorder_01.csv and so on. Each chunk will have 20000000 lines in it.
Get hex dump of a file
Echo commands as they are run
An incredibly useful function is this:
echo_command() { printf '%s\n' "${*}"; "${@}"; }
What it does is echo the command and also run it. We can echo commands using set -x as well. But above comes in handy when you have a large script and don’t want to echo each and every command, only a select few.
# List of logs and who should be notified of issues
logPaths=("api.log" "auth.log" "jenkins.log" "data.log")
logEmails=("jay@email" "emma@email" "jon@email" "sophia@email")
# Look for signs of trouble in each log
for i in ${!logPaths[@]};
do
log=${logPaths[$i]}
stakeholder=${logEmails[$i]}
numErrors=$( tail -n 100 "$log" | grep "ERROR" | wc -l )
# Warn stakeholders if recently saw > 5 errors
if [[ "$numErrors" -gt 5 ]];
then
emailRecipient="$stakeholder"
emailSubject="WARNING: ${log} showing unusual levels of errors"
emailBody="${numErrors} errors found in log ${log}"
echo "$emailBody" | mailx -s "$emailSubject" "$emailRecipient"
fi
done
List all packages installed on Linux
below we list all packages and filter by postgres
$ dpkg -l | grep postgres
Get directory Size
Get size of folder and its sub-folders
du -h -d 1 /path/to/folder | sort -h
The sort -h will sort by size in ascending order
Get free space on disk
Get command used to launch a process
How to move everything under a directory to a subdirectory?
cd /path/to/your/dir
mkdir -p sub
find . -mindepth 1 -maxdepth 1 ! -name . ! -name sub -print0 | xargs -0 mv -t sub
Check if process exists
ps -p <PID> -o pid=,stat=,etime=,cmd=
- If it prints a row, the PID exists.
stat helps interpret state (R running, S sleeping, etc).
How to diff two folders?
diff -rq --brief dir1/ dir2/
> System Preferences, click Security & Privacy, then click General. If the lock at the bottom left is locked 
, click it to 
😀
(How so ever bad or good you may be at any language)
siddharth's space 