Shell func to group-n-count last identical lines

less than 1 minute read

This will be short. I was doing a lot of SQL imports last days and was tired of having not a helpful console output from the psql run. Something like this for all the 10 min:

...
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
...

So this small shell function that pipe intput lines while aggregating them and not displaying the repeated lines but only the counter appeared: https://gist.github.com/aleksandr-vin/a58ca4885d14831e75b434d29a4b61b3.

# Pipe intput lines while aggregating them and not displaying the repeated lines but only the counter
aggr() {
    awk '
{
    # If current line is the same as previous line
    if ($0 == prev) {
        count++;
        # Move cursor up and overwrite previous line
        printf "\033[1A[%d] %s\n", count, $0;
    } else {
        count = 1;
        printf "[%d] %s\n", count, $0;
    }
    prev = $0;
}
'
}

And now running

psql ... -f some.sql | aggr

Give you a 10 min running counter with one line (from 1 to 1585165):

[1585165] INSERT 0 1
[1]  setval
[1] ---------
[1]  1594866
[1] (1 row)
...

Nice!

Updated: