[Forensics] How to explode Watcher node extract file into separate files by record type

In actual fact the following is probably pretty much useless. You can bypass this step by hitting the main file with suitable query scripts. As I have done repeatedly over the last few months on this forum.

Might be useful for a beginner however?

Also an oppourtunity for me to practise Xenforo forum markup blocks.

Onwards:

KAG_extract=../dat/kag-stellar-export-using-Operations-TAB-2023-06-23-T02:02:41-UTC.csv

linux> echo $KAG_extract 
../dat/kag-stellar-export-using-Operations-TAB-2023-06-23-T02:02:41-UTC.csv

linux> sed '1d' $KAG_extract | sed 's/\"//g' | awk -F, '($4=="payment") {} END {print NR}'
32460

Drag out the list of Stellar record types from previous work:

rm -f tmp
cat >> tmp << EOF                                   
account_merge 8
create_account 0
inflation 9
payment 1
set_options 5
EOF

linux> cut -d " " -f1 tmp | paste -s -d " " 
account_merge create_account inflation payment set_options

copy/ paste that last line manually into here:

for i in account_merge create_account inflation payment set_options;
do
   echo "$i";
   sed '1d' $KAG_extract | sed 's/\"//g' | awk -F, -v what="$i" 'BEGIN {count=0} ($4==what) {count++;} END {print count}'; 
done

Ew. xenforo doesn't know about code=awk. Outrageous.

Run the above:

linux> for i in account_merge create_account inflation payment set_options ; do   echo "$i";   sed '1d' $KAG_extract | sed 's/\"//g' | awk -F, -v what="$i" 'BEGIN {count=0} ($4==what) {count++;} END {print count}'; done
account_merge
13804
create_account
14820
inflation
32
payment
3681
set_options
123

Looks good
Write them out to individual files

by making 1 small adaptation.
Surprisingly elegant. The flexibility of awk and bash, I mean.

for i in account_merge create_account inflation payment set_options 
do
  sed '1d' $KAG_extract | sed 's/\"//g' | awk -F, -v what="$i" '($4==what) {print}' > "$i".dat
done

print

displays all the fields in the record
same as

print $0

or even blank squiggly (not tested)

{}

Check the numbers still fit.

linux> wc -l *.dat
   13804 account_merge.dat
   14820 create_account.dat
      32 inflation.dat
    3681 payment.dat
     123 set_options.dat
   32460 total

linux> zip -j All5ZippedUp *.dat
  adding: account_merge.dat (deflated 72%)
  adding: create_account.dat (deflated 72%)
  adding: inflation.dat (deflated 67%)
  adding: payment.dat (deflated 74%)
  adding: set_options.dat (deflated 80%)

linux> zip -sf All5ZippedUp
Archive contains:
  account_merge.dat
  create_account.dat
  inflation.dat
  payment.dat
  set_options.dat
Total 5 entries (10199559 bytes)

Post
zip file is O(3M) much smaller than the original $KAG_extract file
This technique is accessible by anyone with a few hours linux hands-on ability.

Forum security does not allow 3M upload. Understnadable. No big deal.


It is out of date by a few days. The point is is to practise logic and syntax.
Another addition for The Toolbox.

Go fetch an up-to-date version of the above and play?
An excellent exercise for the interested viewer to learn the basics of data munging.