[Integration][HiBob] add initial version of HiBob integration #1273

thinkingabouther · 2024-12-27T17:56:04Z

TODO:

tests for new integration
example raw data

Description

What: Integration to fetch data from HiBob API - HR system that stores organization data such as employees' personal data and organizational structure, and facilitates performance reviews

Why: In our organization, the HR team manages employees' relations with their development teams in HiBob. By ingesting this data into Port, we will be able to set up Attribute-based Access Control for our Service Catalog by populating Port Teams to control who sees what and who does what. Generally, the new integration itself allows ingesting any HR-related data (e.g., office locations, performance metrics, even compensation-related information) into the development platform.

How: By using the Read the public profile section of all active employees API endpoint, it is possible to fetch employees' personal data (e.g. name, email, relation to their manager). By using the Get all company lists API endpoint, we fetch all custom company-wide lists (kind of enums) such as office locations, development teams, employment agreement types, etc. By merging the responses from the 1st and 2nd endpoint, it is possible to fetch most of the employees' HR data.

Type of change

New Integration (non-breaking change which adds a new integration)

All tests should be run against the port production environment(using a testing org).

Integration testing checklist (Awaiting Code Review resolution)

Integration able to create all default resources from scratch
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Resync finishes successfully
If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
If resource kind is updated, run the integration with the example data and check if the expected result is achieved
If new resource kind is added or updated, validate that live-events for that resource are working as expected
Docs PR link here

Preflight checklist

⚠️ Handled pagination: Unfortunately, HiBob API offers Pagination only for a specific set of endpoints, which are not used in the integration for now. As a result, both Profiles and Lists are fetched with a single request each.
⚠️ Handled rate limiting: As mentioned above, data is fetched with a single request, so it is highly unlikely that the current implementation of integration will ever hit the limits (Profiles allow 40 requests per minute by default, Lists allow 50/minute)
Implemented the code in async
⚠️ Support Multi account: I need help with this one since it is not clear what is meant by Mutli account.

Screenshots

HiBobUsers and HiBobTeams graph with default resources:

Employee profiles after merging them with Teams from Company Lists (mapping, blueprints and raw responses are in the examples folder)

API Documentation

Get Employees' Profiles: https://apidocs.hibob.com/reference/get_profiles
Get Company Lists: https://apidocs.hibob.com/reference/get_company-named-lists
Authorization in HiBob API: https://apidocs.hibob.com/reference/authorization

Tankilevitch

Thank you for your contribution @thinkingabouther!!!
Left some comments, I would love to assist with going over the reviews and solutions.

Tankilevitch · 2024-12-29T13:01:10Z

integrations/hibob/.port/resources/port-app-config.yml

+    selector:
+      query: .name | startswith("work.column_1659977338244")
+    port:


can you please elaborate on work.column_1659977338244 is that something specific to your org?

referenced it here: #1273 (comment)

Tankilevitch · 2024-12-30T14:52:30Z

integrations/hibob/.port/resources/blueprints.json

currently _team and _user is not available by default to all organizations, I would suggest defining default of HiBobTeam and HiBobUser at the start and then once its fully available we will adjust the defaults to point to the _team and _user blueprints

I've moved the _team and _user blueprint to examples, the default resources will use different identifiers as advised. Thank you!

integrations/hibob/.port/resources/blueprints.json

Tankilevitch · 2024-12-30T15:00:18Z

integrations/hibob/.port/resources/port-app-config.yml

+resources:
+  - kind: list
+    selector:
+      query: .name | <your-team-list-name>


can you elaborate on what you are trying achieve here? Ideally I would like this to be auto discoverable on setup either from the hibob api or configured by the user installing it

let me know if you would like to schedule a call to brainstorm

I'd like to have auto-discovery bit as well, yet I'm not sure it is possible that it is going to work with lists.

HiBob lists essentially store any custom enums defined in the HR system, such as types of employment agreements, locations, etc. Unfortunately, there’s no clear way to distinguish a list containing team names from any other list without knowing its exact list name. Moreover, the list names are assigned automatically by HiBob, which is why you noticed a name like column_1659977338244 in the initial draft.

However, the default /profile response includes a department field for each employee. In our case, the department represents a higher-level organizational unit, which we don’t plan to use in Port at this time.

Based on this, I propose the following approach:

Auto-populate HiBobTeam by configuring the default blueprints in port-app-config.yml to automatically populate the HiBobTeam with values from the department field in employee profiles. This setup ensures that it works out-of-the-box for most users, as some organizations may even prefer to use the department field for ABAC in Port.

Provide an example for custom mapping between lists and profiles by placing the current implementation (mapping a specific list to an employee field) in the _examples folder. This will serve as a reference for future users who might need a solution for the use cases similar to our organization.

Let me know your thoughts on this approach! 😄

Implemented the changes as mentioned, you are welcome to review again 😃

Tankilevitch · 2024-12-30T15:00:36Z

integrations/hibob/.port/resources/port-app-config.yml

+          identifier: .email
+          blueprint: '"_user"'
+          relations:
+            teams: <your-team-list-name>


proposed a solution here: #1273 (comment)

integrations/hibob/.port/spec.yaml

Tankilevitch · 2024-12-30T15:03:51Z

integrations/hibob/main.py

+def transform_to_array_of_lists(lists_object: dict[str, Any]) -> list[dict[str, Any]]:
+    array_of_lists = []
+    for list_name, list_content in lists_object.items():
+        transformed_entry = {
+            "name": list_name,
+            "values": list_content.get("values", []),
+            "items": list_content.get("items", []),
+        }
+        array_of_lists.append(transformed_entry)
+    return array_of_lists


what this is needed for? when implementing integrations we are thriving to not perform any data manipulation on raw and leaving it as is so it will be easy to reference to their api docs.

This is tricky, indeed.
The thing is, HiBob API wraps the list of lists into a top-level JSON object instead of an array.
So, this is the example structure for the lists response:

{ "list_name_1": { "name": "list_name_1" "items": [{...}, {...}] "values": [{...}, {...}] }, "list_name_2": { "name": "list_name_2" "items": [{...}, {...}] "values": [{...}, {...}] } }

To me, this structure seems less straightforward for future manipulation in Port. Therefore, I’ve opted to return the data in the following structure instead:

[ { "name": "list_name_1" "items": [{...}, {...}] "values": [{...}, {...}] }, { "name": "list_name_2" "items": [{...}, {...}] "values": [{...}, {...}] } ]

Rather than producing a single object of DictionaryOfLists kind, I’m returning an array of individual objects of List kind. From a usage perspective, I believe both implementations would work, but the latter is more concise and easier to manipulate.

Let me know your thoughts based on this input! I’m happy to adjust the logic if needed. 😃

P.S. Unfortunately, the easy to reference to their api docs won't work regardless of the implementation, since there are no examples provided in the API spec.

Thanks for the delayed answer @thinkingabouther !

I wonder if we will be able to pass the specific list which we want to query and only export it would it make this transformation redundant? I've commented above my suggestion

Co-authored-by: Tom Tankilevitch <[email protected]>

thinkingabouther · 2025-01-07T10:45:16Z

Thank you for your review @Tankilevitch! Please take a look at changes/comments when you have time.
By the way, what should I do so that my integration is available in Data Sources Port UI? At the moment, it is properly fetched from the Port API as seen on the screenshot, but is not present in the UI.

For your reference, the orgID I'm using for testing: org_FHEMbBtnzUFieJRE

Tankilevitch · 2025-01-14T22:34:54Z

hey @thinkingabouther, currently we are not supporting viewing not port official integrations in the UI, we are planning to add that capability to improve the development experience.

regarding the data model and api I wonder if something like the way fivetran had modeled the hibob integration with ERD can provide some guidance on the way the hibob api is structured. https://fivetran.com/docs/connectors/applications/hibob

Tankilevitch

Thank you @thinkingabouther looks very detailed PR!!!
I've left a few comments and suggestions, let me know what you think.
Feels like we are very close, in the mean time I'll get a hibob environment to try the integration as well :)

Tankilevitch · 2025-01-14T22:43:14Z

integrations/hibob/hibob/client.py

+        lists_responses = await self.client.get(
+            f"{self.api_url}/v1/company/named-lists"
+        )


can you reference the api docs for this endpoint?

Tankilevitch · 2025-01-14T22:47:03Z

integrations/hibob/examples/mapping.yml

+  - kind: list
+    selector:
+      query: .name | startswith("work.column_1659977338244") # Name of the List storing your teams' names (see lists.response.json)
+    port:


seems like what you actually trying to achieve is way to declare to the kind what list to query from the api.
we support ways to pass the api query params through the kind to the resync in order to perform more efficient queries.

see this pr for example on how we added support to pass query filters to integration that then are being applied to the api request.

Tankilevitch · 2025-01-14T22:49:06Z

integrations/hibob/main.py

+def transform_to_array_of_lists(lists_object: dict[str, Any]) -> list[dict[str, Any]]:
+    array_of_lists = []
+    for list_name, list_content in lists_object.items():
+        transformed_entry = {
+            "name": list_name,
+            "values": list_content.get("values", []),
+            "items": list_content.get("items", []),
+        }
+        array_of_lists.append(transformed_entry)
+    return array_of_lists


Thanks for the delayed answer @thinkingabouther !

I wonder if we will be able to pass the specific list which we want to query and only export it would it make this transformation redundant? I've commented above my suggestion

github-actions bot added the size/XXL label Dec 27, 2024

thinkingabouther marked this pull request as draft December 27, 2024 18:02

thinkingabouther force-pushed the hibob-integration branch from 9be84e2 to 06e9e8b Compare December 30, 2024 09:03

Tankilevitch requested changes Dec 30, 2024

View reviewed changes

Tankilevitch marked this pull request as ready for review December 30, 2024 15:52

Tankilevitch marked this pull request as draft December 30, 2024 15:53

a-neustroev added 3 commits January 2, 2025 15:00

[Integration][HiBob] add initial version of HiBob integration

68fe3cd

touch ups

d2d6f2b

add tests

52fc4a8

thinkingabouther force-pushed the hibob-integration branch from 06e9e8b to 52fc4a8 Compare January 2, 2025 07:00

thinkingabouther and others added 5 commits January 2, 2025 17:24

Update integrations/hibob/.port/resources/blueprints.json

b44de17

Co-authored-by: Tom Tankilevitch <[email protected]>

Update integrations/hibob/.port/resources/blueprints.json

a2c48a4

Co-authored-by: Tom Tankilevitch <[email protected]>

Update integrations/hibob/.port/resources/blueprints.json

2a2ae4b

Co-authored-by: Tom Tankilevitch <[email protected]>

Update integrations/hibob/.port/resources/blueprints.json

718a32d

Co-authored-by: Tom Tankilevitch <[email protected]>

set default value for API url

15ea0c0

thinkingabouther marked this pull request as ready for review January 7, 2025 05:44

a-neustroev added 2 commits January 7, 2025 17:15

add raw examples, mapping and fixed default mapping

69cbd9c

add contributing guide

792fe44

thinkingabouther force-pushed the hibob-integration branch from 792fe44 to 97a1469 Compare January 7, 2025 09:56

thinkingabouther requested a review from Tankilevitch January 12, 2025 14:36

fix default mapping

7a4cad1

thinkingabouther force-pushed the hibob-integration branch from 97a1469 to 7a4cad1 Compare January 14, 2025 05:29

Tankilevitch requested changes Jan 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Integration][HiBob] add initial version of HiBob integration #1273

[Integration][HiBob] add initial version of HiBob integration #1273

thinkingabouther commented Dec 27, 2024 •

edited

Loading

Tankilevitch left a comment

Tankilevitch Dec 29, 2024

thinkingabouther Jan 2, 2025

Tankilevitch Dec 30, 2024

thinkingabouther Jan 7, 2025

Tankilevitch Dec 30, 2024

Tankilevitch Dec 30, 2024

thinkingabouther Jan 2, 2025 •

edited

Loading

thinkingabouther Jan 7, 2025

Tankilevitch Dec 30, 2024

thinkingabouther Jan 2, 2025

Tankilevitch Dec 30, 2024

thinkingabouther Jan 2, 2025

Tankilevitch Jan 14, 2025

thinkingabouther commented Jan 7, 2025

Tankilevitch commented Jan 14, 2025

Tankilevitch left a comment

Tankilevitch Jan 14, 2025

Tankilevitch Jan 14, 2025

Tankilevitch Jan 14, 2025

[Integration][HiBob] add initial version of HiBob integration #1273

Are you sure you want to change the base?

[Integration][HiBob] add initial version of HiBob integration #1273

Conversation

thinkingabouther commented Dec 27, 2024 • edited Loading

Description

Type of change

All tests should be run against the port production environment(using a testing org).

Integration testing checklist (Awaiting Code Review resolution)

Preflight checklist

Screenshots

API Documentation

Tankilevitch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thinkingabouther Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thinkingabouther commented Jan 7, 2025

Tankilevitch commented Jan 14, 2025

Tankilevitch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thinkingabouther commented Dec 27, 2024 •

edited

Loading

thinkingabouther Jan 2, 2025 •

edited

Loading