Analysing diff on hundreds of big CSV files

Sometimes we have to resolve very weird issue. I’m working with a very old, unmaintained piece of software that generates huge CSV files, containing and sorted, among others, by timestamp date.

So i tried to make this software a bit more maintanable, but that could introduce breaking changes. So here’s what i wanted to do:

• Generate the CSV with the old version
• Generate the CSV with the new, version
• Compare each pair of CSV.

What’s the best tool for that ?

Here’s the tricky part. In theory, this is simple. i had all the old version file in a old directory, and others in a new. So something like

for i in ls $(old); do
    diff "old/$i" "new/$i"
done

would work, and it does. But it’s also in this particular case completely unreadable and useless : i really need context before the different lines to debug correctly.

So i tried to find the best tool, either terminal or GUI. And there’s a lot. Most of them either failed by crashing on large CSV, or by being very unpleasant to use.

And in the end… I had the perfect tool, that i use everyday : VS code. So here’s my small script :

for i in (ls old)
      echo $i; if ! diff old/$i new/$i
          code -w --diff old/$i new/$i
      end
  end

Note: Sorry, yes, i use fish shell :)

This runs vscode, waiting for the window to be closed before opening a new file, and running the diff mode. VScode is really efficient at that, and pleasant to use. So i just hope that, if you need to have a good GUI diff one day, this post would help you !

Use VScode !!