Migration to 3.9.0.0 with errors

[miketwenty1]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.9.0.0

Current Behavior

We did a migration from 3.7.1.2 to 3.8.1.0 without error. We then went from 3.8.1.0 to 3.9.0.0 and got the follow output which includes some errors:

migrating ai-rate-limiting-advanced on database 'kong_z'...
ai-rate-limiting-advanced migrated up to: 001_370_to_380 (pending)
ai-rate-limiting-advanced migrated up to: 002_370_to_380 (pending)
migrating graphql-proxy-cache-advanced on database 'kong_z'...
graphql-proxy-cache-advanced migrated up to: 001_370_to_380 (pending)
graphql-proxy-cache-advanced migrated up to: 002_370_to_380 (pending)
migrating graphql-rate-limiting-advanced on database 'kong_z'...
graphql-rate-limiting-advanced migrated up to: 001_370_to_380 (pending)
graphql-rate-limiting-advanced migrated up to: 002_370_to_380 (pending)
migrating header-cert-auth on database 'kong_z'...
header-cert-auth migrated up to: 000_base_header_cert_auth (executed)
migrating openid-connect on database 'kong_z'...
openid-connect migrated up to: 004_370_to_380 (pending)
migrating saml on database 'kong_z'...
saml migrated up to: 001_370_to_380 (pending)
migrating enterprise on database 'kong_z'...
enterprise migrated up to: 021_3700_to_3800 (pending)
enterprise migrated up to: 021_3700_to_3800_1 (executed)
11 migrations processed
2 executed
9 pending

Database has pending migrations; run 'kong migrations finish' when ready





on 3.9.0.0


ong@b0965807910c:/$ kong migrations finish
migrating ai-rate-limiting-advanced on database 'kong_z'...
ai-rate-limiting-advanced migrated up to: 001_370_to_380 (executed)
ai-rate-limiting-advanced migrated up to: 002_370_to_380 (executed)
migrating graphql-proxy-cache-advanced on database 'kong_z'...
graphql-proxy-cache-advanced migrated up to: 001_370_to_380 (executed)
graphql-proxy-cache-advanced migrated up to: 002_370_to_380 (executed)
migrating graphql-rate-limiting-advanced on database 'kong_z'...
graphql-rate-limiting-advanced migrated up to: 001_370_to_380 (executed)
graphql-rate-limiting-advanced migrated up to: 002_370_to_380 (executed)
migrating openid-connect on database 'kong_z'...
openid-connect migrated up to: 004_370_to_380 (executed)
migrating saml on database 'kong_z'...
saml migrated up to: 001_370_to_380 (executed)
migrating enterprise on database 'kong_z'...
enterprise migrated up to: 021_3700_to_3800 (executed)
9 migrations processed
9 executed
Error: [PostgreSQL error] cluster_mutex callback threw an error: /usr/local/share/lua/5.1/kong/workspaces/counters.lua:172: receive_message: failed to get type: timeout
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/workspaces/counters.lua:172: in function 'postgres_run_query_in_transaction'
        /usr/local/share/lua/5.1/kong/workspaces/counters.lua:208: in function 'pg_initialize_counters_migration'
        /usr/local/share/lua/5.1/kong/workspaces/counters.lua:214: in function 'initialize_counters'
        /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:45: in function 'reinitialize_entity_counters'
        /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:186: in function </usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:165>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/db/init.lua:410: in function </usr/local/share/lua/5.1/kong/db/init.lua:360>
        [C]: in function 'pcall'
        /usr/local/share/lua/5.1/kong/concurrency.lua:66: in function 'cluster_mutex'
        /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:165: in function 'finish'
        /usr/local/share/lua/5.1/kong/cmd/migrations.lua:336: in function 'cmd_exec'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
        (command line -e):7: in function 'inline_gen'
        init_worker_by_lua(nginx.conf:128):44: in function <init_worker_by_lua(nginx.conf:128):43>
        [C]: in function 'xpcall'
        init_worker_by_lua(nginx.conf:128):52: in function <init_worker_by_lua(nginx.conf:128):50>

  Run with --v (verbose) or --vv (debug) for more details
kong@b0965807910c:/$ kong migrations finish
No pending migrations to finish










kong@c541995e074e:/$ kong migrations finish
migrating proxy-cache-advanced on database 'kong_z'...
proxy-cache-advanced migrated up to: 001_370_to_380 (executed)
proxy-cache-advanced migrated up to: 002_370_to_380 (executed)
migrating rate-limiting-advanced on database 'kong_z'...
rate-limiting-advanced migrated up to: 001_370_to_380 (executed)
rate-limiting-advanced migrated up to: 002_370_to_380 (executed)
4 migrations processed
4 executed
Error: [PostgreSQL error] cluster_mutex callback threw an error: /usr/local/share/lua/5.1/kong/workspaces/counters.lua:216: receive_message: failed to get type: timeout
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/workspaces/counters.lua:216: in function 'postgres_run_query_in_transaction'
        /usr/local/share/lua/5.1/kong/workspaces/counters.lua:252: in function 'pg_initialize_counters_migration'
        /usr/local/share/lua/5.1/kong/workspaces/counters.lua:258: in function 'initialize_counters'
        /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:45: in function 'reinitialize_entity_counters'
        /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:186: in function </usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:165>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/db/init.lua:410: in function </usr/local/share/lua/5.1/kong/db/init.lua:360>
        [C]: in function 'pcall'
        /usr/local/share/lua/5.1/kong/concurrency.lua:67: in function 'cluster_mutex'
        /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:165: in function 'finish'
        /usr/local/share/lua/5.1/kong/cmd/migrations.lua:336: in function 'cmd_exec'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
        (command line -e):5: in function 'inline_gen'
        init_worker_by_lua(nginx.conf:128):44: in function <init_worker_by_lua(nginx.conf:128):43>
        [C]: in function 'xpcall'
        init_worker_by_lua(nginx.conf:128):52: in function <init_worker_by_lua(nginx.conf:128):50>

  Run with --v (verbose) or --vv (debug) for more details
kong@c541995e074e:/$ kong migrations finish
No pending migrations to finish

Is this something we should be worried about or is it safe to run Kong in this state?
We are using ECS Fargate to run kong, and when a container boots up we will get:
<timestamp> \[error\] 2590#0: \*3236 \[kong\] exporter.lua:32 failed retrieving entity counts: receive\_message: failed to get type: timeout, context: ngx.timer

Expected Behavior

Migration without errors

Steps To Reproduce

We did a migration from 3.7.1.2 to 3.8.1.0 without error. We then went from 3.8.1.0 to 3.9.0.0 and got the follow output which includes some errors:

IMAGE_VERSION=<VERSION NUMBER> docker compose up -d

docker exec -it CONTAINER bash

# kong migrations up

# kong migrations finish

Anything else?

No response

[nowNick]

Hi @miketwenty1 !
Thank you for raising this issue. Could you share a bit more details regarding the Kong instance you are running? Approximately how many services / routes / plugin instances / consumers do you have?

Looking at the error message I suspect the reason you see this might be that the query timed out. Maybe the migration took to long and got killed. It would be helpful to know how many resources (roughly) are stored in the DB.

[miketwenty1]

@nowNick