[API server] better error message on API version mismatch

[aylei]

close #4878

Tested locally by modifying the version constant:

# client too old
$ sky status
RuntimeError: Your SkyPilot client is too old: v1 (server version is v3). Please refer to the following link to upgrade your client:
https://docs.skypilot.co/en/latest/getting-started/installation.html

# remote server too old
$ sky status
RuntimeError: SkyPilot API server is too old: v3 (client version is v5). Please refer to the following link to upgrade your server:
https://docs.skypilot.co/en/latest/reference/api-server/api-server.html

# local server too old
$ sky status
RuntimeError: SkyPilot API server is too old: v3 (client version is v5). Please restart the SkyPilot API server with: sky api stop; sky api start

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual test see above, also refer to the new unit test case
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[romilbhardwaj]

Thanks for fixing this @aylei!

[romilbhardwaj]

Also we should probably print a warning in debug here that this is not expected, instead of silently assuming the server is newer :)

[aylei]

Good idea!

[aylei]

The comment is refined and a debug log is added.

[Michaelvll]

Thanks @aylei! A quick comment for the hinting we shown

[Michaelvll]

Although we allow vague versions with the internal version, we should not emphasize the internal version number here. We can make it easier for people to take actions, i.e. showing the version/commit of the SkyPilot package they should upgrade to. For example, we should check the server commit/version:

1. if the server version is something like 1.0.0.dev202xxxx, we should ask user to upgrade their local SkyPilot to skypilot-nightly==202xxx
2. if the server version is something like 0.x.x, we should ask user to upgrade local SkyPilot to skypilot==0.x.x
3. if the server version is something like 1.0.0.dev0 and the commit to be abcde, we should asks the user to upgrade local version to a specific commit pip install git+https://github.com...@abcde

[aylei]

Great idea! Then I think despite which one is older, we should always hint the user to upgrade/downgrade the client to the server version when there is a version mismatch, since the user may not have control over the server deployment. Wdyt?

[Michaelvll]

That sounds good to me!

In the server being older case, we can add another sentence, like "contact your admin to upgrade the API server or downgrade your local SkyPilot ... "