As a full-stack developer and Linux scripter, arrays play a vital role in enabling complex logic and data structures. Mastering bash arrays unlocks new possibilities for building robust automation.
This all-encompassing guide aims to make you an arrays expert by walking through every facet of working with arrays in bash.
What Are Arrays?
An array is a special variable that allows storing multiple elements under a single name with easy access and manipulation. The values can be numbers, strings, or even other variable names that evaluate to values.
Instead of creating distinct variables:
month1="January"
month2="February"
month3="March"We can store the months in an array:
months=("January" "February" "March")The array elements can be accessed by their index number starting at 0. So months[0] evaluates to "January", months[1] evaluates to "February" etc.
Arrays form the underlying data structure for more complex storage like stacks and queues in programming. They enable storing sequential collections of data for easy sorting, ordering and access.
Key Benefits of Using Arrays
Compared to individual variables, arrays offer important advantages:
Concise Storage: Store thousands of elements under a single variable name
Ordering: Elements indexed in specific order for sorting needs
Flexibility: Add, remove elements on the fly as needed
Iteration: Loop over elements without manual print statements
Abstraction: Hide complexity of data by encapsulating in array
These factors enable array usage for algorithm implementation, automated data pipelines, cleaner code and more.
Bash itself leverages arrays internally for storing positional parameters passed to scripts $@ as well as for filenames * glob expansion.
Array Types in Bash
The Bash scripting language supports two main types of arrays:
Indexed Arrays
These conventional arrays use integer indexes starting from 0. They can store any bash data type like numbers and strings.
Creation:
arr[0]=5
arr[1]=helloAssociative Arrays
These arrays use custom string indexes instead of integers. Elements are stored and accessed using key-value pairs.
Creation:
declare -A colors
colors["red"]="#FF0000"
colors["blue"]="#0000FF" We will look at the specific syntax for indexed and associative arrays more in-depth later on.
Common Array Uses
Understanding how arrays are used in practice helps cement the conceptual foundation. Some typical use cases include:
Configuration Data
Store program configuration settings in arrays for easy access:
locations=("New York" "Los Angeles" "Chicago") Command Line Arguments
Pass array contents as arguments to commands:
convert_images ${png_files[@]}Temp Data
Use arrays as temporary data stores for transformation pipeline:
files=($(ls)) # store list of files
sort_files ${files[@]} # sort files
save_files ${files[@]} # save sorted filesCollections
Model real-world collections and data sets using arrays:
users=("john" "jane" "bob") These examples demonstrate the flexibility of arrays for scripting tasks.
Declaration of Arrays
Unique array assignment syntax allows creating arrays to hold values:
1. Compound Array Assignment
Enclosing values in () automatically assigns them to indexes starting from 0:
fruits=("Apple" "Banana" "Orange")We can verify array creation:
echo ${fruits[0]} # Apple
echo ${fruits[1]} # Banana2. Manual Indexed Assignment
Elements can be appended to specific array indexes manually:
fruits[0]="Apple"
fruits[1]="Banana"
fruits[2]="Orange"3. Read from File
Bash supports populating array directly from a file using IO redirection:
fruits=( $(cat fruits.txt) ) This reads the file content into an array where each line becomes an element.
4. String Split
Splitting a string on a delimiter into an array is possible as well:
vals="Apple,Banana,Orange"
fruits=(${vals//,/ })The // substitution operator splits vals on comma into the array.
5. Array Function
Bash even provides an array function to create arrays from the output of other commands:
fruits=( $(array fruits Apple Banana Orange) )These examples give a taste of the flexible syntax offered for array declaration.
Accessing Array Elements
Accessing populated array elements makes array utilities usable in scripts.
Indexed Access
Referring to indexed array elements directly prints the stored values:
fruits=("Apple" "Banana" "Orange")
echo ${fruits[0]} # Apple
echo ${fruits[1]} # BananaAttempting to access an undefined index auto-expands the array with null values:
echo ${fruits[3]} # nullNegative Index
Bash allows reverse array access using negative indexes starting from -1:
echo ${fruits[-1]} # Orange
echo ${fruits[-2]} # Banana This simplifies referencing the last elements.
Slicing Expansion
Part of an array can be extracted using slicing expansion syntax:
echo ${fruits[@]:0:2} # Apple Banana This prints the first 2 elements between indices 0 and 2.
For Loop
A common way to access each element is looping using bash for:
for fruit in "${fruits[@]}"; do
echo $fruit
doneThis loops through automatically printing each element.
Together these show some ways of directly accessing array contents.
Changing Array Elements
A key array feature is dynamically modifying stored values.
Assigning Indexes
Storing data to a particular index appends/overwrites value:
fruits[3]="Grape"
echo ${fruits[@]}
# Apple Banana Orange GrapeHere Grape got appended by directly addressing index 3.
Appending Elements
Adding elements is also possible using the += operator:
fruits+=("Pineapple")
echo ${fruits[@]}
# Apple Banana Orange Grape PineappleInserting Elements
Use slicing expansion to insert elements at specific positions:
fruits=("${fruits[@]:0:2}" "Watermelon" "${fruits[@]:2}")This inserts "Watermelon" at index 2 by extracting 0-2 and then merging the parts. Elements at 2 get pushed forward.
Sorting Elements
Sort elements using sort:
sorted=($(echo ${fruits[@]} | tr " " "\n" | sort))
echo ${sorted[@]}
# Apple Banana Grape Orange Pineapple WatermelonHere tr helped multi-line sort keeping elements intact.
These show some useful techniques for modifying populated arrays.
Removing Array Elements
Removing stored elements is equally important in data tasks.
Unset Elements
Use bash unset to delete array elements by index:
unset fruits[0]
echo ${fruits[@]}
# Banana Orange Grape Pineapple Watermelon For associative arrays, use the key:
unset colors["green"]Restore Deleted Elements
Restoring deleted elements to their original indexes is possible too:
fruits=("${fruits[@]}" "Apple")
echo ${fruits[@]}
# Apple Banana Orange Grape Pineapple WatermelonHere Apple got merged into its original index.
Trim Excess Elements
Removing excess elements from arrays leaving no gaps simplifies them:
trimmed=(${fruits[@]})
echo ${trimmed[@]}
# Apple Banana Orange Grape Pineapple Together these present some ways to delete elements or restore arrays.
Iterating Arrays with Loops
One of the prime reasons for using arrays is repeated access through loops for iteration.
For-in Loop
The basic for-inloop allows addressing elements directly:
for fruit in "${fruits[@]}"; do
echo $fruit
done C-style For Loop
Bash also offers C-style for loop syntax:
for((i=0; i<${#fruits[@]}; i++)); do
echo ${fruits[i]}
doneHere ${#fruits[@]} tracks array size even if changed.
While Loop
A while loop variant is possible too:
i=0
while [[ $i -lt ${#fruits[@]} ]]; do
echo ${fruits[$i]}
((i++))
doneThese give flexibility in programatically accessing arrays.
Associative Array Specifics
While indexed arrays use integer indexes, associative arrays use strings instead.
Creation
Here is an associative array storing hex color codes:
declare -A colors
colors["red"]="#FF0000"
colors["green"]="#00FF00"Access Values
Access elements by custom string indexes:
echo ${colors["red"]} # FF0000Bracket notation lookups the key.
List Keys
To output all keys:
echo ${!colors[@]}
# red greenIteration
We can iterate the associative array:
for color in "${!colors[@]}"
do
echo $color = ${colors[$color]}
done This shows the unique syntax for associative array handling.
Built-In Array Functions
Beyond language syntax, Bash offers functions to assist array manipulation:
Checking If Index Exists
Test if an array contains a specified element index:
colors=("red")
if [[ ${colors[0]+abc} ]]; then
echo "Index 0 exists"
else
echo "Index 0 does NOT exist"
fiThis checks array variable expansion to confirm index presence.
Expanding Into Arguments
Expand array elements into params for other commands with ${array[@]}:
files=(1.txt 2.txt 3.txt)
wc ${files[@]}
# counts lines, words, bytes for each filePassing the array expands it into arguments.
Getting Array Length
Find total elements stored using:
echo ${#colors[@]}This prefixes # to special length parameter.
Together these give a taste of useful array functions.
Multidimensional Arrays In Bash
Bash does not directly support multidimensional arrays. However, we can emulate 2D or 3D arrays using strings:
arr[0,0]=1
arr[0,1]=2
arr[1,0]=3
arr[1,1]=4
echo ${arr[1,0]} # 3Here arr works like a matrix using , to separate row/column indexes.
We need custom functions, but can model higher dimension data.
Arrays vs Other Languages
Developers coming from other languages may notice differences:
No Fixed Size
Bash arrays auto-grow and have no pre-set size limits for stored elements. Append freely.
No Type Safety
Arrays allow storing values of any data type. Mix strings, integers etc as needed.
No Native Methods
Methods like push, pop, shift etc need external functions unlike languages like JavaScript.
Limited Multidimension
Multi-dimension arrays require workarounds unlike Python/C#/Java native 2D/3D arrays.
Pass-By-Value
Bash passes arrays to functions by value unlike some languages pass by reference. Changes in functions do not affect caller array.
So adept bash array usage requires knowing these nuances.
Common Array Pitfalls
Like any programming concept, arrays come with some best practices:
Unset Elements Break Indexes
Deleting elements causes reindexing issues when later inserting expecting original order.
Pass By Value For Functions
Array changes inside functions do not apply to caller scope due to pass-by-value. Return arrays and assign back instead.
Iterate Defensively
Check array size before iterating in case elements got deleted or filter criteria impacts contents.
Declare Upfront
Declare arrays at start with declare -a rather than sprinkling throughout code. Eases debugging by explicitness.
These examples serve to highlight array specific gotchas to guard against.
Unlocking Array Programming Potential
Bash arrays open up possibilities previously not feasible without significant effort:
Data Pipeline Automation
Ingest, process, transform and store data automatically using arrays as temporary buffers.
Report Generation
Extract elements into desired formats like CSV for plugging into reporting engines.
Input Parsing
Read stdin or files into array, iterate values performing validations, transformations and output.
Dynamic Configuration
Change system configuration by loading files into arrays and applying changes needed.
Algorithm Implementation
Craft custom data structures like stacks plus functions to deliver advanced programming techniques.
In essence, combining array data structures with Bash‘s innate automation strengths enables tackling more complex scripting challenges.
Key Takeaways
This comprehensive tutorial aimed to illustrate arrays extensively through key takeways like:
- Arrays enable storing collections of data under a single variable name
- Indexed and associative arrays provide integer vs string based element access
- Array slice syntax offers insertion, removal and ordering of elements
- Built-in functions help check existence, get size and expand elements
- Loops allow iterating arrays printing values or passing to commands
- Multidimensional data can be emulated using string indexes
- Arrays compared to other languages have pros but also pitfalls
- Practical automation uses benefit from array data pipelines
With this deep grounding in array concepts, Bash scripting to handle data sets and collections can reach new potentials.