[breaking] core update-index do not stop at the first download error

[cmaglie]

Please check if the PR fulfills these requirements

• The PR has no duplicates (please search among the Pull Requests
  before creating one)
• The PR follows
  our contributing guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• UPGRADING.md has been updated with a migration guide (for breaking changes)

• What kind of change does this PR introduce?
  1. The core update-index do not stop anymore at the first error
  2. The UpdateIndex gRPC call has a new flag ignore_custom_package_indexes to allow skipping the update of custom package indexes on the first run.

• What is the current behavior?
  If there is a problem downloading a package index the core update-index command will stop and not complete the update of the remaining custom indexes url.

• What is the new behavior?
  The core update-index command will always complete the update of all indexes.

• Does this PR introduce a breaking change, and is titled accordingly?
  Yes

• Other information:
  Fix [gRPC] It should be possible to run the core update-index equivalent with and without the 3rd party URLs #1788

[codecov]

Codecov Report

Merging #1866 (9fddd77) into master (71cab65) will increase coverage by 0.02%.
The diff coverage is 29.48%.

@@            Coverage Diff             @@
##           master    #1866      +/-   ##
==========================================
+ Coverage   36.62%   36.65%   +0.02%     
==========================================
  Files         231      231              
  Lines       19610    19655      +45     
==========================================
+ Hits         7182     7204      +22     
- Misses      11600    11622      +22     
- Partials      828      829       +1     

Flag	Coverage Δ
unit	36.65% <29.48%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cli/core/search.go	0.00% <0.00%> (ø)
cli/core/update_index.go	0.00% <0.00%> (ø)
cli/update/update.go	0.00% <0.00%> (ø)
commands/daemon/daemon.go	0.00% <0.00%> (ø)
commands/instances.go	39.08% <18.42%> (-1.69%)	⬇️
cli/output/rpc_progress.go	68.75% <33.33%> (-5.80%)	⬇️
arduino/httpclient/httpclient.go	67.27% <100.00%> (ø)
cli/instance/instance.go	57.69% <100.00%> (+0.41%)	⬆️
internal/integrationtest/arduino-cli.go	83.73% <100.00%> (+0.54%)	⬆️
legacy/builder/hardware_loader.go	88.88% <0.00%> (-0.40%)	⬇️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[per1234]

Describe the problem

The value of the cc.arduino.cli.commands.v1.ArduinoCoreService.DownloadResult.successful field is set to true when a download fails.

To reproduce

Set up

$ arduino-cli version
arduino-cli.exe  Version: git-snapshot Commit: 732bdc54 Date: 2022-09-07T09:59:11Z

$ export ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS=http://example.com/package_foo_index.json

$ arduino-cli daemon
Daemon is now listening on 127.0.0.1:50051

Demo

Use grpcurl to run the following commands in another terminal.

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Create

{
  "instance": {
    "id": 1
  }
}

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Init

{
  "error": {
    "code": 9,
    "message": "Loading index file: loading json index file C:\\Users\\per\\AppData\\Local\\Arduino15\\package_foo_index.json: open C:\\Users\\per\\AppData\\Local\\Arduino15\\package_foo_index.json: The system cannot find the file specified."
  }
}

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.UpdateIndex

{
  "downloadProgress": {
    "url": "https://downloads.arduino.cc/packages/package_index.tar.bz2",
    "file": "Downloading index: package_index.tar.bz2",
    "totalSize": "43541"
  }
}
{
  "downloadProgress": {
    "downloaded": "43541"
  }
}
{
  "downloadProgress": {
    "completed": true
  }
}
{
  "downloadResult": {
    "url": "https://downloads.arduino.cc/packages/package_index.tar.bz2",
    "successful": true
  }
}
{
  "downloadProgress": {
    "url": "http://example.com/package_foo_index.json",
    "file": "Downloading index: package_foo_index.json",
    "totalSize": "-1"
  }
}
{
  "downloadProgress": {
    "downloaded": "1256"
  }
}
{
  "downloadResult": {
    "url": "http://example.com/package_foo_index.json",
    "error": "Error downloading index 'http://example.com/package_foo_index.json': Server responded with: 404 Not Found"
  }
}
{
  "downloadResult": {
    "url": "http://example.com/package_foo_index.json",
    "successful": true
  }
}
ERROR:
  Code: Internal
  Message: Some indexes could not be updated.

🐛 The value of the cc.arduino.cli.commands.v1.ArduinoCoreService.DownloadResult.successful field was true for the failed download of http://example.com/package_foo_index.json.

Expected behavior

The value of the cc.arduino.cli.commands.v1.ArduinoCoreService.DownloadResult.successful field is set to false when the download fails.

[cmaglie]

🐛 The value of the cc.arduino.cli.commands.v1.ArduinoCoreService.DownloadResult.successful field was true for the failed download of http://example.com/package_foo_index.json.

I've pushed the fix for this problem, and the other one #1866 (comment)

[per1234]

The bugs reported in my previous review have been resolved and it is working as expected now.

Thanks Cristian!