Emptying Trash with a lot of offline images doesn't work

[daniel-freiermuth]

The bug

Background

Immich instance with ~80K pictures. I happened to add immich's library folder as an external library (by adding a wrong docker mount). This caused a lot of new images (~1000K) to be added to immich 🙈 When I realized my mistake the next morning, immich was already happily processing those. After fixing the mount problem, immich correctly recognized the newly added images as offline and put those into the trash. Now I'd like to empy the trash and that's where the problem starts.

The bug

In the Web UI, when clicking on the button "Empy Trash", the immediately says "Permanently deleted 0 assets." No logs are happening (docker-compose logs -f).

Selecting a few images (~200) and then deleting those works.

Selecting more (~3K) and then trying to delete those also brings the bubble with the correct count, but the images still seem to be there. (If I understood things correctly, immich will enqueue the images for deletion, so it makes sense that they are not gone right away?)

When trying to delete all or a lot of images, the immich_server container finally exits with the logs [1].

Another logged error that might or might be related is [2]. I couldn't figure out a causal connection to me using the Web UI.

The OS that Immich Server is running on

Docker on Fedora 40

Version of Immich Server

v1.126.1

Version of Immich Mobile App

n.a.

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload:z
      - /etc/localtime:/etc/localtime:ro,z
      - /var/lib/photos/input:/eos_photos/input:z
    env_file:
      - .env
    ports:
      - 2283:2283
      - 8081:8081
    depends_on:
      - redis
      - database

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    #image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    #image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:z
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=v1.126.1
IMMICH_TELEMETRY_INCLUDE=all

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=[redacted]

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Add a lot of images via external library.
2. Remove the files again.
3. Try to delete the now-offline images.

Relevant log output

[1]:
immich_server              | 
immich_server              | <--- Last few GCs --->
immich_server              | 
immich_server              | [18:0x32f40024000]   328067 ms: Mark-Compact 4069.5 (4143.8) -> 4068.8 (4144.6) MB, pooled: 0 MB, 4977.27 / 0.00 ms  (average mu = 0.217, current mu = 0.016) allocation failure; GC in old space requested
immich_server              | 
immich_server              | 
immich_server              | <--- JS stacktrace --->
immich_server              | 
immich_server              | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
immich_server              | ----- Native stack trace -----
immich_server              | 
immich_server              |  1: 0xe3811e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  2: 0x125fb70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  3: 0x125fe47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  4: 0x148d885  [immich-api]
immich_server              |  5: 0x14a70f9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api]
immich_server              |  6: 0x147b7c8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  7: 0x147c6f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  8: 0x14553ce v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api]
immich_server              |  9: 0x18b6b50 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api]
immich_server              | 10: 0x7f5ffedac476 
immich_server              | api worker exited with code null


and a slightly different one:

immich_server              | 
immich_server              | <--- Last few GCs --->
immich_server              | 
immich_server              | [17:0x43cd2024000]   451806 ms: Scavenge (interleaved) 4045.8 (4124.7) -> 4043.9 (4128.4) MB, pooled: 0 MB, 34.61 / 0.00 ms  (average mu = 0.674, current mu = 0.479) allocation failure; 
immich_server              | [17:0x43cd2024000]   451983 ms: Scavenge (interleaved) 4049.4 (4128.4) -> 4047.0 (4146.7) MB, pooled: 0 MB, 133.47 / 0.00 ms  (average mu = 0.674, current mu = 0.479) allocation failure; 
immich_server              | 
immich_server              | 
immich_server              | <--- JS stacktrace --->
immich_server              | 
immich_server              | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
immich_server              | ----- Native stack trace -----
immich_server              | 
immich_server              |  1: 0xe3811e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  2: 0x125fb70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  3: 0x125fe47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  4: 0x148d885  [immich-api]
immich_server              |  5: 0x14a70f9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api]
immich_server              |  6: 0x147b7c8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  7: 0x147c6f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  8: 0x14553ce v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api]
immich_server              |  9: 0x18b69cc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api]
immich_server              | 10: 0x7f750ceec476 
immich_server              | api worker exited with code null


——————————————————————————————————————————————————————————————————————
[2]:
immich_server              | [Nest] 18  - 02/24/2025, 7:24:43 PM   ERROR [Api:RangeError: Invalid string length
immich_server              |     at JSON.stringify (<anonymous>)
immich_server              |     at stringify (/usr/src/app/node_modules/express/lib/response.js:1008:12)
immich_server              |     at ServerResponse.json (/usr/src/app/node_modules/express/lib/response.js:234:14)
immich_server              |     at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:62:62)
immich_server              |     at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
immich_server              |     at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:176:48
immich_server              |     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
immich_server              |     at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13
immich_server              |     at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17~86qoqsji] Unknown error: RangeError: Invalid string length

Additional information

First of all, Thank you so much into looking this!! And of course for creating immich.

The issue points towards RAM. Can I increase the RAM availabe to immich somehow (my hardware certainly has enough)? But I guess, actually immich shouldn't take so much.

Please let me know any more logs I should catch or experiments I should perform.