[Bug] Unable to use AWS S3 as storage backend

[andres-tw]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	XP, Seven, 10, Ubuntu, ...
Virtualized Env.	True
Dedicated RAM	8 GB
vCPU	2
TheHive version / git hash	thehiveproject/thehive4:4.1.16-1 (image commit: 7b59e21f2442d077928d896dddcaf72fde657d61)
Package Type	Docker,
Database	Cassandra
Index type	Elasticsearch
Attachments storage	S3
Browser type & version	If applicable

Problem Description

When migrating data from The Hive 3.5.1 to 4.1.16 and using AWS S3 as the storage backend for the 4.1.16 cluster, file uploads fail with a 400 Bad Request due to missing uploadId.

Steps to Reproduce

1. Setup The Hive 4 with S3 storage backend. Relevant config section:

storage {
  provider: s3
  s3 {
    bucket = "bucket-name"
    readTimeout = 1 minute
    writeTimeout = 1 minute
    chunkSize = 1 MB
    endpoint = "https://s3.eu-central-1.amazonaws.com"
    region = "eu-central-1"
    accessKey = ""
    secretKey = ""
  }
}
alpakka.s3.aws.credentials.provider = default
alpakka.s3.access-style = virtual

NB! Using IAM role for auth, hence setting alpakka credentials provider to default

2. Start migration with: /opt/thehive/bin/migrate -d --output /etc/thehive/application.conf --main-organisation OrgName --input /etc/thehive/hive3.conf

3. Migration starts warning on each S3 upload with the following:

2022-01-09 19:01:22,218 [WARN] from org.thp.thehive.migration.th4.Output in TheHiveMigration-akka.actor.default-dispatcher-16 [|55ea1a97] Unable to set avatar to user [email protected]: akka.stream.alpakka.s3.FailedUpload: Upload part 1 request failed. Response header: (HttpResponse(400 Bad Request,List(x-amz-request-id: 1PWGZ6FZ51TC6PQE, x-amz-id-2: dgDz0+nxd6CkG3jtwgeoOhxrm2YlUhUhp3PQLzaS7NhsGvkcnF/Ps7H4zNZ22XQP7bdNEu1sF3g=, Date: Sun, 09 Jan 2022 19:01:21 GMT, Server: AmazonS3, Connection: close),HttpEntity.Chunked(application/xml),HttpProtocol(HTTP/1.1))), response body: (<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>This operation does not accept partNumber without uploadId</Message><ArgumentName>partNumber</ArgumentName><ArgumentValue>partNumber</ArgumentValue><RequestId>1PWGZ6FZ51TC6PQE</RequestId><HostId>xxx</HostId></Error>).

4. No files are uploaded to S3 bucket

Possible Solutions

• The Hive is currently using alpakka 2.0.2, but It seems that alpakka 3.0.4 introduces quite substantial changes to the AWS S3 library, which can possibly remedy the situation.
• Since the core problem seems to be a missing uploadId value for the MultiPartUpload and there seem to be references to it in alpakkas 2.0.2 version here then possibly setting the uploadId when calling the alpakka MultiPartUpload function could also fix it.