Creating a huge test set with real data

Today we had an issue when a specific document arrived in our platform, it crashed one of our service. Thanks to our poor package-lock.json versioning policies, it was due to an unplaned upgrade of one of our library. Anyway, it should have worked in theory. We had tests, etc… even so this particular document that crashed everything ( thanks node, thanks assert in that library ). Well, that was not fun.

We didn’t identified the particular document, but the root cause was found pretty quickly. So here’s one thing i tried to avoid this in the future :

1. Capture production trafic, ( of course, anonymised :) ), from kubernetes, thanks to ksniff extension. Check how to install it : ksniff

2. Run ksniff and output to a file, it’s easier to debug afterwards : kubectl sniff my-service-6888d5885-228sz -n default -o output

3. Open output file with Wireshark, and filter the traffic. In my case, http && ( ip.dst == 10.10.10.1 ) && ( http.request.uri.path == "/v1/my-endpoint" ) ( Note : adapt the ip.dst with the ip of the my-service pod )

4. Go to File > Export packet dissections > As JSON... and save your file somewhere. In our example it will be test.json

5. Run the following command : cat test.json | jq -r '.[]._source.layers.http["http.file_data"]' | jq -c > test-set.ljson

And here we have tons of real world example. My next step will probably be to use Cypress to send these documents, check expected result in the UI, and make this an integration test.