Cognition integration provider #302
Conversation
I see what you mean, I added an internal endpoint to refinery-gateway. Currently, SET_NULL - YES
|
Update is blocked on cognition-ui because we don't allow same integration names. This isn't the case with update, i.e. the "Update" button shouldn't be blocked if the integration name remains the same.
|
perf: rename internal delete endpoint
* Oidc identifier migrated to the users table * Search for user only if there is not oidc identifier * migrate * model * alembic merge * model * projects with access management * add access management attribute * deactivate mock up * deactivate access management * add groups/users to records * fix smaller issues * embeddings * model * update payloads * alembic fix * endpoint for sync internal * model * update logic * error handling * chore: update submodules * chore: update submodules * chore: update submodules --------- Co-authored-by: LennartSchmidtKern <[email protected]> Co-authored-by: andhreljaKern <[email protected]>
|
Ran the "test me" folder of the test sharepoint. it has ~ 140 documents. The project & records were created however the running_id seems to be created faulty:
|
|
I am able to manually add users to the project record permissions even with auto sync (groups=> permission view => edit access => select user & records)
|
|
Sync failed after changes were detected: the database table for sharepoint also only has 1 record, not sure if before or after the sync was triggered
|
| print(traceback.format_exc(), flush=True) | ||
| return ["tokenization failed"] | ||
| # TODO check if this is still needed for access management updates | ||
| if not only_access_management_update: |
There was a problem hiding this comment.
is this still needed?
| @@ -331,6 +336,100 @@ def delete_records( | |||
| __delete_records(project_id, record_ids) | |||
|
|
|||
|
|
|||
| def sync_access_groups_and_users_sharepoint(project_id: str, integration_id: str, permissions_users: Dict[str, Any], record_ids: Optional[List[str]]) -> None: | |||
| try: | |||
There was a problem hiding this comment.
long try: body
|
|
||
|
|
||
| def add_access_groups_or_users(project_id: str, record_ids: List[str], group_ids: Optional[List[str]] = None, user_ids: Optional[List[str]] = None) -> None: | ||
| try: |
| @@ -263,3 +263,15 @@ def check_user_exists(email: str) -> bool: | |||
| if i["traits"]["email"].lower() == email.lower(): | |||
| return True | |||
| return False | |||
|
|
|||
|
|
|||
| def get_user_from_search(email: str) -> bool: | |||
There was a problem hiding this comment.
can the existing function get_userid_from_mail be used to avoid code duplication?
| prev_filter_attributes = embedding_item.filter_attributes or [] | ||
| new_filter_attributes = list(set(prev_filter_attributes + filter_attributes)) | ||
| embedding_item.filter_attributes = new_filter_attributes | ||
| general.commit() |
There was a problem hiding this comment.
commit after the for loop? the if statement below also issues another commit
| return project.get_all_with_access_management(organization_id) | ||
|
|
||
|
|
||
| def activate_access_management(project_id): |
| str(embedding_item.id), | ||
| record_ids=changed_records_ids if partial_update else None, | ||
| ) | ||
| return errors |
|
|
||
| except Exception as e: | ||
| print(traceback.format_exc(), flush=True) | ||
| return [str(e)] |
| @@ -162,4 +162,20 @@ def __migrate_kratos_users(): | |||
| if user_database.sso_provider != sso_provider: | |||
| user_database.sso_provider = sso_provider | |||
|
|
|||
| if user_database.oidc_identifier is None: | |||
There was a problem hiding this comment.
how is the oidc_identifier used?
| @@ -70,6 +70,38 @@ def get_all_projects(request: Request) -> Dict: | |||
| return pack_json_result(projects) | |||
|
|
|||
|
|
|||
| # TO DO, some admin check should be added here | |||
| @router.get("/all-projects-with-access-management") | |||
| def get_all_projects_with_tokens(request: Request) -> Dict: | |||
There was a problem hiding this comment.
I could be mistaken, we don't usually add return typing in routes right?
|
I seem to be hanign an issue with the test me folder that an excel file is considered faulty:
With the error it didn't run through. maybe a try catch it missing somewhere to finalize the process if something doesn't work as expected |
|
The documents libary sync button should have "Uses the certificate, client & tenant id to collect all available document libraries"
|
This is the main PR.
Related PRs
New repository
Important
Retrieve:
Tenant and Client IDs in Azure Portal (@kern.ai) => Key Vaults => dev-krn-accompio => Objects -> Secrets => O365-TENANT-ID and O365-UNATTENDED-CLIENT-ID
Certificate .pfx file in Azure Portal (@kern.ai) => Key Vaults => dev-krn-admin => Objects -> Certificates => dev-accompio-certificate =>
Download in PFX/PEM format. The passphrase is an empty string. Use" "(space) as an environment variable value (reference).Document Library ID:
b!zzhsLojhuEaDy3fQIjUZxLC67xk1l9lFpzlKVAQ1-uDYfIQ6DMGYQrkemTRS4V0QUse dev-setup@cognition-integration-provider to run cognition-integration-provider (
bash start -a -b cognition-integration-provider)Tests
Tests were not developed for this container due to long running extraction and transformation tasks
Affected areas
dev-setup,deployment-cognition,deployment-managed-cognitionrefinery-submodule-modelcognition-task-masteradmin-dashboardcognition-uirefinery-uirefinery-gatewaycognition-gatewaycognition-integration-providerPerformance
MP - multiprocessing (# workers)
SP - singleprocessing