v1.0.0-rc9: Initial Import Of Users And Group Fails With 404

[mam8cc]

Describe the bug
Starting with an empty SSO directory running the below command to sync our AWS Power Users group:

❯ ./ssosync \
    --endpoint REDACTED \
    --access-token REDACTED \
    -c creds.json \
    -g 'email:aws-power-users*' \
    --google-admin [email protected] \
    --debug

INFO[0000] Syncing AWS users and groups from Google Workspace SAML Application
INFO[0000] syncing                                       sync_method=groups
DEBU[0000] get google groups                             query="email:aws-power-users*"
DEBU[0000] get google users and groups and its users
DEBU[0000] get group members from google                 group="AWS Power Users"
DEBU[0000] get users                                     group="AWS Power Users"
DEBU[0000] get user                                      group="AWS Power Users" [email protected]
...
DEBU[0016] get aws groups
INFO[0016] [DEBUG] GET REDACTED/scim/v2/Groups
DEBU[0016] get aws users
INFO[0016] [DEBUG] GET REDACTED/scim/v2/Users
DEBU[0016] get aws groups and its users
INFO[0016] creating user                                 [email protected]
INFO[0016] [DEBUG] POST REDACTED/scim/v2/Users
...
INFO[0023] creating group                                group="AWS Power Users"
INFO[0023] [DEBUG] POST REDACTED/scim/v2/Groups
DEBU[0023] finding user                                  group="AWS Power Users"
INFO[0023] [DEBUG] GET REDACTED/scim/v2/Users?filter=userName+eq+%22redacted%redacted.com%22
INFO[0023] adding user to group                          group="AWS Power Users" [email protected]
DEBU[0023] Group Change                                  group="AWS Power Users" operations=add [email protected]
INFO[0023] [DEBUG] PATCH REDACTED/scim/v2/Groups
FATA[0023] status of http response was 404

Checking in the AWS SSO console shows the group is created. A subsequent rerun of the same command yields a new error.

❯ ./ssosync \
    --endpoint REDACTED \
    --access-token REDACTED \
    -c creds.json \
    -g 'email:aws-power-users*' \
    --google-admin [email protected] \
    --debug

INFO[0000] Syncing AWS users and groups from Google Workspace SAML Application
INFO[0000] syncing                                       sync_method=groups
DEBU[0000] get google groups                             query="email:aws-power-users*"
DEBU[0000] get google users and groups and its users
DEBU[0000] get group members from google                 group="AWS Power Users"
DEBU[0000] get users                                     group="AWS Power Users"
DEBU[0000] get user                                      group="AWS Power Users" [email protected]
...
DEBU[0016] get aws groups
INFO[0016] [DEBUG] GET REDACTED/scim/v2/Groups
DEBU[0016] get aws users
INFO[0016] [DEBUG] GET REDACTED/scim/v2/Users
DEBU[0019] get group members from aws                    group="AWS Power Users"
DEBU[0019] checking if user is member of                 group="AWS Power Users"
INFO[0019] [DEBUG] GET REDACTED/scim/v2/Groups?filter=id+eq+%229a672df75b-0f856220-1288-46dc-b610-99ea77ef12eb%22+and+members+eq+%229a672df75b-12f494c8-893d-40f0-9fc0-cfc93e415ff2%22
...
INFO[0024] creating user                                 [email protected]
INFO[0024] [DEBUG] POST REDACTED/scim/v2/Users
ERRO[0024] error creating user                           [email protected]
FATA[0024] status of http response was 409

After checking the directory in the console, the group is empty.

Expected behavior
A group and its users should be created in a single pass.

Additional context
This does not happen consistently with all groups. I am able to sync another group without issues without problems.

[christiangda]

@mam8cc

This could be a nice improvement just to work as atomic operations in case the function reaches the AWS SSO-SCIM API limits before creating the groups or assigning users to their respective groups

I'll see how to implement this, but is a big refactor of the way the function SyncGroupsUsers works when --sync-method is groups

[joshuachong]

This does not happen consistently with all groups. I am able to sync another group without issues without problems.

@mam8cc are the membership size for these groups different?

[mam8cc]

Hey @joshuachong, thanks for getting back so quickly. Yes the AWS Admin group is 8 members, and AWS Power Users has around 60. It is worth noting that the 8 users in AWS Admin are also in AWS Power Users, but that wouldn't impact a clean run, I'd think.

[pkit]

Just FYI: 404 is always spitted when adding a first user to a group that did not exist prior to ssosync run.
It's probably not waiting for the group to properly start "existing" as a second ssosync run passes that point.
I.e. if you have 5 new groups you need to run ssosync 6 times and only the last one will succeed.

[badgerspoke]

Since we currently happen to have a scratch AWS organisation for testing, I have been able to poke around here.

With an empty SSO and 11 selected groups in Google (max of 22 users per filtered group) using the default --sync-method groups running from master we consistently encounter this 404 error when it tries to add a user to a newly created group so it feels like a consistency/timing issue in SSO:

2021-11-24T03:15:26.439Z {"level":"debug","msg":"creating aws groups added in google","time":"2021-11-24T03:15:26Z"}
2021-11-24T03:15:26.439Z {"group":"AWS Foo Group","level":"info","msg":"creating group","time":"2021-11-24T03:15:26Z"}
2021-11-24T03:15:26.513Z {"group":"AWS Foo Group","level":"debug","msg":"finding user","time":"2021-11-24T03:15:26Z"}
2021-11-24T03:15:26.557Z {"group":"AWS Foo Group","level":"info","msg":"adding user to group","time":"2021-11-24T03:15:26Z","user":"[email protected]"}
2021-11-24T03:15:26.557Z {"group":"AWS Foo Group","level":"debug","msg":"Group Change","operations":"add","time":"2021-11-24T03:15:26Z","user":"[email protected]"}
2021-11-24T03:15:26.582Z status of http response was 404: errorString
null

I can also confirm that with #45 we still encounter it (in case anyone is wondering if that PR helps - we need that fix too for other reasons).

However having applied #47 on top of that and having removed the groups and users from SSO to start clean, I am not seeing this 404 at all.

[ChrisPates]

I believe this issue has been resolved. by release v2.0.0. Please let me know if you are still seeing this.