Bash and its builtin tools are great, and usually sufficient for solving a host of problems, or automating processes. However, there are tasks which require more power, where bash and its builtin tools can't easily resolve.
For example, parsing CSV files becomes very complex if using only bash. There are many formats to CSV and attempting to parse even one becomes chasing a rabit trail.
I used to use AWK, but recently learned about a particular way for embedding Python within Bash (see Why Python?). Python3 is usually included in most distributions, and comes with batteries included (standard libraries for accomplishing just about anything). It's syntax is more readable and less arcane than AWK, and is more widely taught and learned.
The example below illustrates embeddeding Python code within Bash in such a way that is pipeable to other Bash builtins/GNU utilities. Bash variables can be used within Python and opens the door to numerous possibilities.
Of course, this isn't just useful for Python, but other interpreted languages, etc. as well.
#!/bin/bash
CSV_FILE="$1"
$QUERY = "$2"
function parse_csv {
python3 - << EOF
from csv import reader
with open("$CSV_FILE", newline='') as csvfile:
r = reader(csvfile, delimiter=',', quotechar='"')
for row in r:
print('|'.join(row))
EOF
}
parse_csv | grep -i "$QUERY"
This script parses a CSV file by using Python and the csv standard module. It accomplishes this by wrapping the Python code within a Bash function. This allows you to pipe the output of the function to other Bash programs/utilities.
Passing Arguments to Python Subscript within Bash
#!/bin/bash
parse_csv(){
python3 - "$1" <<EOF
from sys import argv
from csv import reader
file = argv[1]
with open(file,newline='') as f:
r = reader(f,delimiter=',',quotechar='"')
for row in r:
print(','.join(row))
EOF
}
parse_csv ~/Downloads/output.csv | column -s'|' -t