How to filter invalid jsonlines using jq

When using jsonlines data with jq, invalid lines can cause parsing errors, e.g.

jq: parse error: Expected separator between values at line 61755, column 1954

To filter out invalid lines, you can use the following command:

jq -R 'fromjson?'

This will passthrough valid JSON lines and return null for invalid ones. You can then filter out the null values using select. The subsequent command needs to handle null values correctly, but it won’t see any invalid lines.

Example usage

cat data.json | jq -R 'fromjson?' | jq -c '{name: .node?.name?}'