Use batched NSRL insertion

[BolidCyber]

With the current NSRL insertion implementation of the update server, the server needs to load the entire NSRL database in memory. This can make the service crash if not enough RAM is available.

Using pandas, it is possible to load n lines of a CSV file using a generator, which will save a lot of memory.

[BolidCyber]

Pipeline seems to be broken:

• CybercentreCanada/assemblyline-pipeline-templates could not be found
• CybercentreCanada/assemblyline-unittest-samples could not be found

There is no public access to those repos.

[cccs-rs]

Pipeline seems to be broken:

• CybercentreCanada/assemblyline-pipeline-templates could not be found
• CybercentreCanada/assemblyline-unittest-samples could not be found

There is no public access to those repos.

Correct, although if the PR is ready for review, I think I should be able to trigger the pipeline manually for testing
Based on last commit: Pipelines

[cccs-rs]

An alternative to using pandas to lazy load CSV data: https://pypi.org/project/lazycsv?

Just in the interest of minimizing our footprint if we can 😅

[BolidCyber]

An alternative to using pandas to lazy load CSV data: https://pypi.org/project/lazycsv?

Just in the interest of minimizing our footprint if we can 😅

I do agree with that stance, pandas might not be the best library for this work. I'll try and make a port.

[BolidCyber]

An alternative to using pandas to lazy load CSV data: https://pypi.org/project/lazycsv?

Just in the interest of minimizing our footprint if we can 😅

Could we respecify the ticket ?
Actually, lazy loading is already implemented through the CSV reader. There's no need of a new library (nor pandas). I think that when we analyzed the bug (several months from now), we overthought it.

Reverting commit da56c7f should suffice as the issue is in the indefinitely growing hash_list variable, not the CSV read.

[cccs-rs]

Patch should be included in the latest release! 🚀