UKG to CII SCIM User Sync

This project focuses on synchronizing user data from UKG (UltiPro) to Cisco Identity Intelligence (CII) SCIM integrations. While the principles may apply to other SCIM implementations, this tool has been specifically tested and optimized for CII's SCIM interface.

Customization

This tool was designed to work locally but is highly adaptable to different environments and requirements. You can use it as a core component and wrap it with whatever infrastructure is needed for your specific use case. You can customize various aspects to fit your specific:

• Organizational structure and user data model
• Field mappings between UKG and SCIM
• Execution environment (servers, containers, schedulers, cloud functions)
• Performance requirements via concurrency and batch size settings
• Error handling and logging preferences
• Integration with your existing automation workflows

Key customization points are highlighted throughout this documentation, especially in the "Field Mapping" and "Configuration" sections.

Key Information

• Daily Synchronization: This script should be configured to run daily to maintain proper synchronization between UKG and CII. Each run recreates all user records on CII, ensuring the most up-to-date information is always reflected. Note that this is a full refresh approach where users are entirely replaced rather than incrementally updated.

• Handling Deleted Users: Currently, this tool does not have a mechanism to detect and remove users that were completely deleted from UKG. If a user's record status (employeeStatusCode) in UKG changes from active ('A') or leave ('L') to terminated ('T'), the tool correctly updates their status on CII. However, if a user is completely removed from the UKG system, their record will persist in CII as this tool has no way to identify such deletions.

• CII Integration Setup: Before using this script, you must create a SCIM integration on your CII tenant. Use the integration API credentials (Base URL, Token URL, Client ID and Client Secret) to configure this script via the .env file. Important Security Note: The Client Secret is highly sensitive and should be treated as a credential. Never share it, commit it to public repositories, or expose it in logs.

• UKG Pro REST API: This project utilizes the UKG Pro REST API to fetch user data from UltiPro. You can learn more about this API, its endpoints, and authentication requirements at the UKG Pro API Documentation.

This tool provides an automated solution for synchronizing user data between systems with configurable settings for pagination, rate limiting, and concurrency.

Setup

1. Install dependencies:

pip install -r requirements.txt

2. Copy the environment template and configure your values:

cp .env.example .env

3. Edit .env file with your actual configuration values.

Configuration

The following environment variables need to be configured:

UKG API Configuration

• UKG_API_HOSTNAME: API endpoint hostname (default: useast1-gateway.ukg.net)
• UKG_USERNAME: Basic Auth username
• UKG_PASSWORD: Basic Auth password
• UKG_CUSTOMER_API_KEY: Customer API key header value
• UKG_PAGE_SIZE: Number of users to fetch per page (default: 100)
• UKG_RATE_LIMIT_SLEEP: Sleep duration in seconds after rate limit is hit (default: 0, i.e. rate limiting is disabled)
• UKG_RATE_LIMIT_CALLS: Number of API calls before sleeping (default: 10)
• UKG_MAX_WORKERS: Number of concurrent workers for UKG data enrichment (default: 10)

SCIM Configuration

• SCIM_BASE_URL: SCIM endpoint base URL
• SCIM_TOKEN_URL: OAuth token endpoint
• SCIM_CLIENT_ID: OAuth client ID
• SCIM_CLIENT_SECRET: OAuth client secret
• SCIM_BULK_BATCH_SIZE: Number of users to include in each SCIM Bulk operation (default: 100, as the current CII SCIM integrations limit)

Bulk User Synchronization

The application implements a comprehensive bulk synchronization process that handles large user datasets efficiently:

How It Works

1. Automatic Pagination: Starting from page 1, fetches users in batches using UKG_PAGE_SIZE
2. Page-by-Page Processing: For each page:
   • Fetches users from UKG API
   • Transforms users to SCIM format
   • Syncs the batch to SCIM endpoint
   • Reports success/failure counts for the page
3. Continuation Logic: Continues to next page automatically
4. Stop Condition: Stops when UKG API returns an empty result set
5. Final Reporting: Provides total success and failure statistics

UKG Data Collection Process

The tool implements a thorough data collection process to build complete user profiles from UKG:

Multi-Endpoint Data Collection

1. Base User Records: First, the tool fetches all employee records from the UKG employment-details endpoint in paginated batches (controlled by UKG_PAGE_SIZE).

2. Data Enrichment: For each employee record retrieved, the tool then makes additional API calls to:

   • person-details endpoint - To get personal information such as email addresses and phone numbers
   • user-profile-details endpoint - To get user account information like usernames and system preferences

3. Data Merging: The tool intelligently merges data from all three endpoints to create a comprehensive profile for each user.

Parallel Processing

To optimize performance, the data enrichment process uses parallel execution:

1. The main process fetches a page of employee records from the employment-details endpoint
2. These records are deduplicated by employee ID to ensure each user is only processed once
3. A thread pool (sized according to UKG_MAX_WORKERS) processes multiple users concurrently:
   • Each thread fetches the additional details for a specific user from both secondary endpoints
   • Rate limiting is applied across all threads to prevent overwhelming the UKG API

This approach significantly improves performance while still respecting API limits and ensures that the SCIM user records contain the most complete information possible from all available UKG data sources.

SCIM Bulk Operations

The application uses the SCIM /Bulk endpoint for efficient processing of large user sets:

1. Batched Processing: Users are processed in batches defined by SCIM_BULK_BATCH_SIZE
2. Efficient API Usage: Each batch is sent as a single API call, reducing network overhead
3. Error Handling: Individual operation failures are tracked and reported
4. Automatic Retries: Failed bulk requests are automatically retried up to 3 times
5. Configurable Batch Size: Adjust SCIM_BULK_BATCH_SIZE based on your SCIM provider's capabilities. The current CII limit is 100

Configuration Examples

# UKG pagination settings
UKG_PAGE_SIZE=100

# SCIM bulk operation settings
SCIM_BULK_BATCH_SIZE=100  # The current maximum value in CII
SCIM_BULK_BATCH_SIZE=50   # Smaller size for more conservative approach
SCIM_BULK_BATCH_SIZE=200  # Larger size for high-capacity SCIM endpoints

UKG Worker Configuration

The UKG_MAX_WORKERS environment variable controls the concurrency for UKG API operations:

Benefits

• Parallel Processing: Controls how many concurrent UKG API requests are made
• Performance Tuning: Adjust based on your UKG API rate limits and available resources
• Efficient Enrichment: Optimizes the data enrichment phase where additional user details are fetched

Performance Tuning Guidelines

# Conservative (safe for most APIs)
UKG_MAX_WORKERS=5

# Balanced (good for most scenarios)
UKG_MAX_WORKERS=10

# Aggressive (high-performance systems)
UKG_MAX_WORKERS=20

Rate Limiting

The UKG client implements intelligent rate limiting to prevent overwhelming the API:

How it works

1. Concurrent Processing: The client processes requests in parallel using thread pools
2. Call Counting: Each API call is counted across all threads using thread-safe counters
3. Automatic Throttling: After every N API calls (configurable), the system pauses for a specified duration

Configuration Examples

Conservative settings (slower but safer):

UKG_MAX_WORKERS=5
UKG_RATE_LIMIT_CALLS=5
UKG_RATE_LIMIT_SLEEP=1.0

Aggressive settings (faster but may hit rate limits):

UKG_MAX_WORKERS=20
UKG_RATE_LIMIT_CALLS=50
UKG_RATE_LIMIT_SLEEP=0.1

Disable rate limiting (not recommended):

UKG_RATE_LIMIT_SLEEP=0

Rate Limiting Best Practices

• Start with conservative settings and gradually increase if no rate limit errors occur
• Monitor API response times and error rates
• Set UKG_RATE_LIMIT_SLEEP=0 to disable sleeping (useful for testing)

Usage

Run the sync process:

python main.py

Or alternatively, you can run it as a module:

python -m main

The application will automatically:

1. Load configuration from environment variables
2. Initialize UKG and SCIM clients
3. Start bulk synchronization with automatic pagination
4. Process all users across all pages
5. Report final statistics and exit

Field Mapping

The application includes a default field mapping between UKG and SCIM formats. Note: You may need to modify the field mappings in transformer.py according to your specific use case and target SCIM endpoint requirements.

Important Field Requirements

• Mandatory Unique Fields: Both external_id and user_name are mandatory fields and must be unique across all users.
• User Name Matching: The user_name field is particularly important as it's used to match each UKG record with the corresponding user data in CII. It's critical to use the same values as already exist in CII.
• Email as User Name: In most CII tenants, user_name is set to the end user's email address. This is the recommended approach for ensuring consistent matching.

UKG Field	SCIM Field	Notes
employeeId	User.external_id	External reference
userName	User.user_name	In CII SCIM integrations, treated as 'login'
firstName	User.name.givenName
lastName	User.name.familyName
middleName	User.name.middleName
namePrefixCode	User.name.honorificPrefix
nameSuffixCode	User.name.honorificSuffix
preferredName	User.nickName
email	User.emails[0].value	Primary work email
emailAddressAlternate	User.emails[1].value	Secondary email
employeeStatusCode	User.active	Active when "A"
jobTitle/jobDescription	User.title	Uses jobTitle or falls back to jobDescription
employeeTypeCode	User.userType
languageCode	User.preferredLanguage

Customizing Field Mappings

To customize the field mappings:

1. Edit the transform_user method in transformer.py
2. Evaluate your source UKG schema and field usage patterns to determine which UKG fields should map to critical SCIM fields like id, external_id, and user_name
3. Consider your target SCIM endpoint's specific schema requirements
4. Test thoroughly with a small dataset before running full synchronization

Logging

Logs are written to both console and ukg2scim.log file with detailed progress information including:

• Page-by-page processing status
• Batch sizes and success/failure counts
• Final synchronization statistics
• Error details for troubleshooting

CII Data Processing

CII will automatically process new SCIM data pushed to it from the identity source every 24 hours on a regular schedule, but if you would like to manually trigger the processing of any data that has been pushed within the past 24 hours, you can use the Process New Data menu option in the Integrations page for that integration.

Testing

To run the test suite, ensure you have installed all dependencies from requirements-dev.txt, then execute the following command from the project's root directory:

pytest

This command will automatically discover and run all tests located in the tests/ directory. You should see output indicating the status of each test (e.g., passed, failed).

Distribution

If you need to package the project for distribution or deployment, you can create a zip archive containing all tracked files using git:

git ls-files | zip -@ ukg2scim.zip

This command will package all files tracked by git into a single zip file, making it easy to transfer the project to another environment or system.