feat: scim #566

raphael-cohere · 2024-08-01T15:51:51Z

This PR adds SCIM functionality to the project. It allows an organisation to sync their users and groups from their idp.

Allows synchronisation of users and groups from Okta to toolkit. While SCIM is an open standard, this implementation was only tested with Okta. If we want to add more providers we would need to retest, but as long as they are following the spec it should be fine (similar to OIDC).

General Notes

this only works when OIDC is enabled. We match the users from the OIDC request to the users that were synchronised. (via external id)
this feature should only be activated if there were no users created so far, otherwise there could be side effects

Resources

okta reference: https://developer.okta.com/docs/reference/scim/scim-20/#scim-user-operations
flask sample app: https://github.com/oktadev/okta-scim-flask-example/tree/master
original spec: https://www.rfc-editor.org/rfc/rfc7643.html#page-

PR Template

Thank you for contributing to the Cohere Toolkit!

PR title: "area: description"
- Where "area" is whichever of interface, frontend, model, tools, backend, etc. is being modified. Use "docs: ..." for purely docs changes, "infra: ..." for CI changes.
- Example: "deployment: add Azure model option"
PR message: Delete this entire checklist and replace with
- Description: a description of the change
- Issue: the issue # it fixes, if applicable
- Dependencies: any dependencies required for this change
Add tests and docs: Please include testing and documentation for your changes
Lint and test: Run make lint and make run-tests

AI Description

This PR introduces the System for Cross-domain Identity Management (SCIM) to the backend. SCIM is a standard for automating the exchange of user identity information between identity domains, such as between an enterprise identity management system and a service provider.

The changes include:

Adding SCIM configuration to the .env-template file.
Creating a new SCIMAuthValidation class in request_validators.py for SCIM authentication.
Adding SCIM-related imports and a new SCIM router in routers.py.
Adding SCIM-related fields to the secrets.template.yaml and settings.py files.
Creating new CRUD operations for groups and users in group.py and user.py, respectively.
Creating new SCIM exception handling and router in scim.py.
Adding SCIM-related imports and classes in schemas/scim.py.
Adding a ScimAuthValidation class in request_validators.py for SCIM authentication.
Updating the get_or_create_user function in utils.py to use the get_user_by_external_id function.
Adding SCIM-related tests in test_scim.py.
Adding SCIM-related configuration to the secrets.yaml file.

cla-assistant · 2024-08-01T15:52:04Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mtanzim
❌ raphael-cohere
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

src/backend/services/logger/middleware.py

# Conflicts: # src/backend/config/routers.py # src/backend/main.py # src/backend/tests/factories/__init__.py

# Conflicts: # src/backend/crud/user.py

src/backend/routers/scim.py

src/backend/services/auth/utils.py

codecov-commenter · 2024-08-08T08:53:16Z

Codecov Report

Attention: Patch coverage is 93.28358% with 36 lines in your changes missing coverage. Please review.

Project coverage is 83.50%. Comparing base (4edacee) to head (76eaf7d).
Report is 96 commits behind head on main.

Files	Patch %	Lines
src/backend/routers/scim.py	90.00%	13 Missing ⚠️
src/backend/tests/crud/test_group.py	79.48%	8 Missing ⚠️
src/backend/services/auth/utils.py	0.00%	6 Missing ⚠️
src/backend/alembic/versions/b6a70f628c24_.py	73.68%	5 Missing ⚠️
src/backend/services/auth/request_validators.py	91.66%	2 Missing ⚠️
src/backend/crud/group.py	97.14%	1 Missing ⚠️
src/backend/crud/user.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #566       /-   ##
==========================================
  Coverage   81.38%   83.50%    2.11%     
==========================================
  Files         199      243       44     
  Lines        8354    11037     2683     
==========================================
  Hits         6799     9216     2417     
- Misses       1555     1821      266

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

BeatrixCohere · 2024-08-08T09:59:24Z

src/backend/config/routers.py

+ Depends(get_session),
+ Depends(ScimAuthValidation()),
+ ],
+ "auth": [],


Why none in auth?

misunderstand how the dependency system work, also added to auth now

BeatrixCohere · 2024-08-08T10:12:52Z

src/backend/routers/scim.py

+async def get_users(
+ session: DBSessionDep,
+ count: int = 100,
+ start_index: int = 1,


Why 1 not 0?

This is in the spec 🤷

https://datatracker.ietf.org/doc/html/rfc7644
startIndex The 1-based index of the first result in the current set
of list results. REQUIRED when partial results are returned due
to pagination.

BeatrixCohere · 2024-08-08T10:13:04Z

src/backend/routers/scim.py

+) -> ListUserResponse:
+ if filter:
+ single_filter = filter.split(" ")
+ filter_value = single_filter[2].strip('"')


Should we do validation here?

BeatrixCohere · 2024-08-08T10:15:11Z

src/backend/routers/scim.py

+ session: DBSessionDep,
+ count: int = 100,
+ start_index: int = 1,
+ filter: Optional[str] = None,


Why have filter and the one below for username? Can you only filter by username?

For okta yes. That's all they support. Might be different for other providers.

BeatrixCohere · 2024-08-08T10:44:19Z

src/backend/routers/scim.py

+async def get_groups(
+ session: DBSessionDep,
+ count: int = 100,
+ start_index: int = 1,


Same questions about start index, fitler (just by username) etc

src/backend/routers/scim.py

raphael-cohere added 4 commits July 31, 2024 12:00

add groups

4601258

add externalId

e95f93b

implementation for scim users

d9f4517

implement filter

62aa4a9

mtanzim added 8 commits August 1, 2024 14:30

update migration for groups

5bbc314

group create

9098621

GET groups

892c9c9

get group

0076351

add delete group

a7fe424

finally pydantic is happy for now

73e3218

empty group patch working

4148df0

fix user errors

ba2806e

mtanzim changed the title ~~feat: slim~~ feat: scim Aug 1, 2024

raphael-cohere added 3 commits August 2, 2024 15:06

finish group implementation

7d6a9e8

make user_id, group_id unique

a5f99de

migration

7e6de14

mtanzim reviewed Aug 2, 2024

View reviewed changes

src/backend/services/logger/middleware.py Outdated Show resolved Hide resolved

mtanzim reviewed Aug 2, 2024

View reviewed changes

src/backend/services/logger/middleware.py Outdated Show resolved Hide resolved

raphael-cohere and others added 5 commits August 2, 2024 16:22

fix replace

b81bce0

feat: use user active column (#578)

a47d0c8

wip

79249c4

Merge branch 'main' into feat/scim

bd3b319

# Conflicts: # src/backend/config/routers.py # src/backend/main.py # src/backend/tests/factories/__init__.py

Merge remote-tracking branch 'origin/feat/scim' into feat/scim

5a77c51

# Conflicts: # src/backend/crud/user.py

raphael-cohere requested a deployment to development August 5, 2024 11:35 — with GitHub Actions Waiting

raphael-cohere added 4 commits August 5, 2024 13:40

merge migrations

8c01803

user group crud tests

d0dc545

fix group crud

f0ad80f

do not return full name

b2ab98b

raphael-cohere requested a deployment to development August 5, 2024 13:44 — with GitHub Actions Waiting

raphael-cohere requested a deployment to development August 7, 2024 12:03 — with GitHub Actions Waiting

get all users

f78074d

raphael-cohere requested a deployment to development August 7, 2024 13:01 — with GitHub Actions Waiting

raphael-cohere commented Aug 7, 2024

View reviewed changes

src/backend/routers/scim.py Outdated Show resolved Hide resolved

raphael-cohere commented Aug 7, 2024

View reviewed changes

src/backend/services/auth/utils.py Show resolved Hide resolved

move basic auth validation to router dependencies

0c4d2cd

raphael-cohere requested a deployment to development August 8, 2024 07:41 — with GitHub Actions Waiting

fix docs

d8a4292

raphael-cohere requested a deployment to development August 8, 2024 08:11 — with GitHub Actions Waiting

check for Bearer in header

9343e87

raphael-cohere requested a deployment to development August 8, 2024 08:39 — with GitHub Actions Waiting

Merge branch 'main' into feat/scim

76eaf7d

raphael-cohere temporarily deployed to development August 8, 2024 08:43 — with GitHub Actions Inactive

BeatrixCohere reviewed Aug 8, 2024

View reviewed changes

raphael-cohere added 5 commits August 9, 2024 10:40

add scim auth validation

3b356b0

refactor

a18a880

unique names

74c9ed0

Merge branch 'main' into feat/scim

30d10d4

migration

45e291c

raphael-cohere requested a deployment to development August 9, 2024 09:43 — with GitHub Actions Waiting

raphael-cohere requested a review from BeatrixCohere August 9, 2024 09:43

lint

9d5a8bd

raphael-cohere requested a deployment to development August 9, 2024 09:43 — with GitHub Actions Waiting

migration

ec28bb5

raphael-cohere requested a deployment to development August 9, 2024 09:46 — with GitHub Actions Waiting

can not be null

eb5e7ae

raphael-cohere requested a deployment to development August 9, 2024 09:46 — with GitHub Actions Waiting

remove conflict

45158d9

raphael-cohere requested a deployment to development August 9, 2024 12:19 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: scim #566

feat: scim #566

raphael-cohere commented Aug 1, 2024 •

edited by cohere-pr-pal bot

Loading

cla-assistant bot commented Aug 1, 2024 •

edited

Loading

codecov-commenter commented Aug 8, 2024

BeatrixCohere Aug 8, 2024

raphael-cohere Aug 9, 2024

BeatrixCohere Aug 8, 2024

raphael-cohere Aug 8, 2024

raphael-cohere Aug 9, 2024

BeatrixCohere Aug 8, 2024

BeatrixCohere Aug 8, 2024

raphael-cohere Aug 8, 2024

BeatrixCohere Aug 8, 2024

feat: scim #566

Are you sure you want to change the base?

feat: scim #566

Conversation

raphael-cohere commented Aug 1, 2024 • edited by cohere-pr-pal bot Loading

General Notes

Resources

PR Template

cla-assistant bot commented Aug 1, 2024 • edited Loading

codecov-commenter commented Aug 8, 2024

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raphael-cohere commented Aug 1, 2024 •

edited by cohere-pr-pal bot

Loading

cla-assistant bot commented Aug 1, 2024 •

edited

Loading