Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import railjson command fails with big infra #2328

Closed
flomonster opened this issue Nov 10, 2022 · 0 comments · Fixed by #2334
Closed

Import railjson command fails with big infra #2328

flomonster opened this issue Nov 10, 2022 · 0 comments · Fixed by #2334
Assignees
@Khoyo
Labels
area:editoast Work on Editoast Service difficulty:good first issue Good for newcomers kind:bug Something isn't working

Comments

@flomonster
Copy link
Contributor

Describe the bug
When trying to import an infra with more than $2^{16} - 1$ objects (of the same type), the command fails.

To Reproduce
Steps to reproduce the behavior:

  1. In editoast run this command: editoast import-railjson my_infra /path/to/infra.json

Make sure that the infra you're importing is big enough.
The command fails with the following error message:

An internal diesel error occurred: 'number of parameters must be between 0 and 65535

Expected behavior

The command should work.

How to fix:

The functions persist_batch must handle vectors with a lot of elements by doing multiple pg queries.
@flomonster flomonster added kind:bug Something isn't working difficulty:good first issue Good for newcomers area:editoast Work on Editoast Service labels Nov 10, 2022
@Khoyo Khoyo self-assigned this Nov 14, 2022
Khoyo added a commit that referenced this issue Nov 14, 2022
@Khoyo
editoast: fix: diesel max params on big infra
38975eb 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
@Khoyo Khoyo mentioned this issue Nov 14, 2022
Merged
Khoyo added a commit that referenced this issue Nov 21, 2022
@Khoyo
editoast: fix: diesel max params on big infra
9c9def8 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
flomonster pushed a commit that referenced this issue Nov 23, 2022
@Khoyo @flomonster
editoast: fix: diesel max params on big infra
794dd6f 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
flomonster pushed a commit that referenced this issue Nov 23, 2022
@Khoyo @flomonster
editoast: fix: diesel max params on big infra
695e978 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
flomonster pushed a commit that referenced this issue Nov 23, 2022
@Khoyo @flomonster
editoast: fix: diesel max params on big infra
2a72c8d 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
flomonster pushed a commit that referenced this issue Nov 25, 2022
@Khoyo @flomonster
editoast: fix: diesel max params on big infra
b9923da 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
alexandredamiron pushed a commit that referenced this issue Jan 17, 2023
@Khoyo @alexandredamiron
editoast: fix: diesel max params on big infra
722fbf1 
Diesel only accepts 65535 parameters per query, so we need to chunk our
insertions when importing big infras.

Fixes #2328.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Khoyo Khoyo

Labels
area:editoast Work on Editoast Service difficulty:good first issue Good for newcomers kind:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

editoast: fix: diesel max params on big infra OpenRailAssociation/osrd
2 participants
@Khoyo @flomonster