Example of proper way to consume types in the callbacks?

[DillonSadofsky]

So I am trying to add SCIM support using SCIMMY and SCIMMYRouters to my express API. So far so good. Everything is hooked up, and my placeholder ingress/egress functions are being hit by my postman tests.

However, I saw that you had type definitions but I can't seem to find a reasonable way to use them in my modules. I was having trouble importing types from the module, possibly because the types aren't reexported at the top level of the module? IDK, I was able to get around that by typing inputs using typeof, but the objects passed to my callbacks don't seem to match these classes members, so its not quite right and would welcome input.

For example, for ingress, it is documented with a js example at https://scimmyjs.github.io/SCIMMY.Types.Resource.html#~IngressHandler (thanks for that) with the arguments types listed as SCIMMY.Types.Resource and SCIMMY.Types.Schema

I tried converting that literally:

	// Register the User resource
	SCIMMY.Resources.declare(SCIMMY.Resources.User, {})
	// Add support for the optional EnterpriseUser extension
	SCIMMY.Resources.User.extend(SCIMMY.Schemas.EnterpriseUser, false)
	// Add handlers for retrieval and submission
	SCIMMY.Resources.User.ingress(async (resource: typeof SCIMMY.Types.Resource, data: typeof SCIMMY.Resources.User.schema, _ctx: unknown): Promise<typeof SCIMMY.Schemas.User> => {
		console.log(`Request to create/edit user ${JSON.stringify(resource)}, ${JSON.stringify(data)}`)

                // For now, return a raw Promise, eventually we'll await database operations
		return await new Promise((resolve) => resolve({ ...data, id: '456' }))
	})
	SCIMMY.Resources.User.egress((resource: Record<string, unknown>) => {
		console.log(`Request to read user ${JSON.stringify(resource)}`)
	})
	SCIMMY.Resources.User.degress((resource: Record<string, unknown>) => {
		console.log(`Request to delete user ${JSON.stringify(resource)}`)
	})

But those types' members don't seem to match the members of the object resource/data. I also tried typing using SCIMMY.Resources.User and even SCIMMY.Resources.User.definition based on following steps through the constructors.

Anyway, I get that part of this is probably because the actual attributes on an object are defined by the schema, and can be optionally present, but it seemed like in schemas.js User.#definition did some magic, so I wasn't sure if there was a way for me to type the objects such that typescript would know what the possible (they could be missing on a given call I know) members supplied by the User or Group object would be? I assume that is kinda the point of these resources? To type them?

Maybe I'm missing something obvious, but I've been chasing my tail trying to use SCIMMY.Resources.User, SCIMMY.Schemas.User, SCIMMY.Types.User, etc but I can't seem to figure it out. Maybe its not possible and everyone just types the incoming data as unknown or Record<string, unknown> and test everything manually. Let me know if that is true, but it seems like this library validates the input before calling my handler, so it would suck to have to treat the object as an anonymous, untyped object when it has been verified to meet a, well, type.

Anyway, I'm liking the library so far, just hoping to add typing into my use case to add auto-complete and transpile-time error checking.

And if I am just missing something, I'd suggest changing the type of the handler argument to SCIMMY.Resources.User.ingress/etc. Currently it explicitly seems to take any, and it'd at least be cool if it typed its input as a callback that took the three arguments and returns a promise of an object of the requested shape. It'd take some of the guesswork out of making the handler implementation. I think it'd even help non-TS developers since VSCode will show hints based on types even in pure JS these days.

[DillonSadofsky]

In the end, I found this type magic to get me relatively close. Its been harder than consuming types from some libraries, and for a bit I thought it was impossible without modifying the .d.ts to explicitly export some more types, but this seems to be getting close (implementation and product specific details removed for space).

import type { Express } from 'express'
import SCIMMY from 'scimmy'
import SCIMMYRouters from 'scimmy-routers'
import { parse as SCIMParse, filter as SCIMFilter } from 'scim2-parse-filter'

type SCIMMYUserSchema = InstanceType<typeof SCIMMY.Schemas.User>
// An arbitrary object.  Technically its constrainted at runtime, but there doesn't seem to be a type for that?
type SCIMMYData = Record<string, any>
// The way the constructor of the User object works, it copies all the input data on to extend it
type SCIMMYUserInput = SCIMMYUserSchema | SCIMMYData

type SCIMMYResource = InstanceType<typeof SCIMMY.Types.Resource>
type SCIMMYConstraints = SCIMMYResource['constraints']
type SCIMMYFilter = SCIMMYResource['filter']

// As part of the RFC, the multi-objects have some basic properties
type SCIMMulti = {
	type: /*'home' | 'work' | 'other' |*/ string
	primary: boolean
	display: string
	value: string
}

type SCIMEmail = SCIMMulti
type Context = {
	// My product context
}
// Manages state, registers routes, contains the controller that performs product specific operations
class SCIMHandler<UserEntity, Controller extends SCIMController<UserEntity>> {
	controller: Controller

	applyFilter(data: Array<SCIMMYData>, filter: SCIMMYFilter) {
		return data.filter(SCIMFilter(SCIMParse(filter.expression)))
	}

	ingress = async (resource: SCIMMYResource, data: SCIMMYUserInput, ctx: Context): Promise<SCIMMYData> => {
		// Implementation removed for space
	}

	egress = async (resource: SCIMMYResource, ctx: Context) => {
		// Implementation removed for space
	}

	degress = (resource: SCIMMYResource, ctx: Context) => {
		// Implementation removed for space
	}

	// Main entrypoint for adding SCIM functionality to an Express server
	registerSCIMRoutes(app: Express) {
		const scimConfig = SCIMMY.Config.get() as Record<string, unknown>
		console.log(scimConfig)

		// Register the User resource
		SCIMMY.Resources.declare(SCIMMY.Resources.User, {})
		// Add support for the optional EnterpriseUser extension
		SCIMMY.Resources.User.extend(SCIMMY.Schemas.EnterpriseUser, false)
		// Add handlers for retrieval and submission
		SCIMMY.Resources.User.ingress(this.ingress)
		SCIMMY.Resources.User.egress(this.egress)
		SCIMMY.Resources.User.degress(this.degress)
	}
}

The types at least seem to line up for the resource.filter, resource.id, resource.constraints. Not so much for the actual User fields, so I've had to just use the names of the attributes at runtime or the RFC to build the marshall/unmarshall transforms for going back and forth between a raw SCIM object and one of my domain UserAccount entity objects. So I still welcome input if there is something I'm missing. And if any other TS users come through wanting to know how to type ingress/egress/etc maybe the above would be useful to them. I think the biggest issue was just that the types aren't reexported like the static objects are, so in some libraries one would do something like import type { UserResource } from 'scimmy' you can't do that here, and there are no 'hints' from the typing of the argument to when you register your ingress/egress since it takes argument any instead of something like (Resource, User, Ctx) => ?, which would flow to the function or give a consumer some idea of what to import, type-wise, to do it themselves.

[sleelin]

Hi @DillonSadofsky, apologies for the delay in addressing this.

Since the SCIMMY source code is written in vanilla JavaScript, the type declarations are generated at build time from the JSDoc comments using the process documented in the TypeScript handbook. Unfortunately, the TypeScript compiler is seemingly quite bad at this, and when it doesn't just go with the "any" type, the generated declaration file is littered with broken or incorrect types.

I'm working on modifying the build process to solve this issue by directly using the TypeScript compiler API to parse the source files and generate an accurate declaration file, which should eliminate the abundant "any" types and provide access to the correct types, however that still leaves the issue of typing the objects passed through to handlers.

As you correctly noted, SCIMMY does have various pieces of magic to validate schema conformance throughout the process. These schemas are defined at runtime - almost any attribute can be added or removed - making it very difficult if not impossible to accurately type the properties of each schema instance. In theory I could include types for the default shape of the schemas, but since almost all attributes are optional, they might end up looking more like the Record<string, unknown> type you originally mentioned. If you've had any further success in typing when moving between SCIM object shapes and your domain's object shapes without having to do so manually I'd love to know how, as it would be great to include this in a future version?

Thanks,
Sam

[DillonSadofsky]

As my integration has progressed, my type declarations have stuck pretty closely to the trimmed down prototype I posted above, so I can't say I've gotten any further.

I agree that process wise, everything is done at runtime, but technically the user does encode their opinions into the code by calling various .declare and .extend calls, which return the object. Theoretically, if the various objects were generics (https://www.typescriptlang.org/docs/handbook/2/generics.html), that information could 'follow' through those calls into the generic type argument of the return of those various objects, which could prime the object with an 'argument' that would tell it a bit about (at code time) what the valid objects will be at runtime. Maybe it could be solved somehow at the higher level of the namespaces that contain the various workhorse entities? The caller could keep an opinionated type they pass into a generic argument at a higher level, then when they do .Schema or whatever, it'd be part of that context. I'm not sure if that is even possible, I don't have much experience with that.

That said, this is, at its heart a wrapper around an HTTP API, and almost everything is optional, so ignoring the above which is probably a very advanced solution (could require a rethink of the structure of the project or rewriting it in TS to make it easy to manage), it would've helped me if the type could at least explicitly mention and type all optional inputs (some of which are defined by the RFC [attributes, filter, etc], and some of which would just be optional based on the common schemas). I know it wouldn't be perfect if someone customizes the schema, but if all the attributes are marked optional, then its not technically wrong if they're omitted/impossible. As for additional fields, if the type listed all explicit optional (expected) attributes, then unioned on a general one like Record, it wouldn't provide much error checking against misspellings (since any attribute is 'valid' at that point), but it'd give some auto-complete and explicit type information to modern IDEs. If the user does go with a custom schema, I guess that would be a benefit of the Generic type, as the generic arguments can have a default, so the User/Group could have defaults for the common case and the user could override if they wished. I honestly have no idea what is the most common, but I didn't plan to extend the schema, and I assume most users are just trying to add support for the base technology.

I apologize I don't have better suggestions. This request started out as confusion regarding to importing types nested within the namespaces (which as you mention may just be due to issues in the hand written type hints) and morphed into a bigger discussion on what is possible. I mostly just assumed I was doing something wrong/was confused since the same 'object' name is reused in several namespaces, and I was somewhat confused between the entity types (runtime objects) and the Types of the entities if that makes sense. I didn't know if that was throwing of VS code or if it was something else.

At this point, I've mostly figured things out by getting the basic stuff connected and setting breakpoints and just seeing what objects contained, but I guess my core complaint was that (as a new user who isn't yet super familiar with SCIM tech), I identified that overriding ingress/egress/degress with custom transforms was the thing todo, but the examples don't get much into what the arguments 'contain' and it took a bit of guess and check. I saw that your project shipped types, and even in non-TS projects, VS code and (presumably other IDEs) can read that to give hints, which are a kind of living documentation that ships with the code, so IMO, although you could improve the docs like https://scimmyjs.github.io/SCIMMY.Types.Resource.html#~IngressHandler I think more explicit shipped types help provide a bit of documentation to the consumer. Above, we were discussing how to make the types runtime-accurate, which is useful if you want types to constrain the code and detect bugs (automatically detect that .Id is bad while .id is good or whatever) and that is nice, but even if the types had those suggested properties and unioned Record, it could still serve the purpose of discoverability. That's my 2 cents at least.

[ymoriaud]

Hello here ! 👋

I was learning how to use the library, and I struggled with the typing of ingress/egress callback. I created a PR just to show a proposal to augment typing for the SCIMMY.Resources.User.ingress.

Since the User class extends Types.Resource but bring override, I notice that typescript handle it better if we declare the new type directly in the User class.

Do you think it's something that can work for the default types ? Or am I missing something ?

Great day to everyone !
Yann.

[sleelin]

Hi @DillonSadofsky and @ymoriaud,
Apologies for the delay in getting this fix over the line, I've just released v1.3.0 which should include a scimmy.d.ts file that isn't functionally useless! The types in the handler methods should now be correctly inferred, and I was even able to get partial type checking on the expected return values of the handlers. If possible, would you mind updating to this version and reporting back as to whether the types are now working as expected?

Thanks,
Sam

[ymoriaud]

Congrats again @sleelin for the release !
I can confirm that upgrading to v1.3.0 fix and improve a lots the typing. 👏

[...] I was even able to get partial type checking on the expected return values of the handlers [...]

When you say you were able to get partial type checking on the return values, is it the same partial type checking that in the example below ?

Example

Note

This example is only here to demonstrate typing check, it's not a "working" code.

You can see in the egress handler the three comments (1., 2., 3.). The comment 1. will raise a typing error, rather than the comments 2. and 3. won't. It looks like typescript having hard time with the union check with the EgressHandler define in the scimmy.d.ts file ? If I'm missing something, please let me know.

With comment 1.

With comment 2.

import SCIMMY from 'npm:[email protected]'

// Define the type for a User retrieved from the database
interface UserFromDatabase {
    id: string
    active: boolean
    userName: string,
    displayName: string,
    emails:{
        value: string,
        primary: boolean
    }[]
    meta: {
        created: Date,
        lastModified: Date,
        version: string
    }
}

// Fake function to retrieve a user from a database
function userFromDatabase(id: string, _ctx: unknown): Promise<UserFromDatabase> {
    return new Promise((resolve) => {
        resolve({
            id,
            active: true,
            userName: "[email protected]",
            displayName: "Nice user name",
            emails: [{
                value: "[email protected]",
                primary: true
            }],
            meta: {
                created: new Date("2024-12-09"),
                lastModified: new Date("2024-12-09"),
                version: "0001"
            }
        })
    })
}
// Fake function to retrieve multiple users from a database
function usersFromDatabase(_filter?: SCIMMY.Types.Filter, _constraints?: SCIMMY.Types.Resource<SCIMMY.Schemas.User>["constraints"], _ctx?: unknown): Promise<UserFromDatabase[]> {
    return new Promise((resolve) => {
        resolve([{
            id: "1234",
            active: true,
            userName: "[email protected]",
            displayName: "Nice user name",
            emails: [{
                value: "[email protected]",
                primary: true
            }],
            meta: {
                created: new Date("2024-12-09"),
                lastModified: new Date("2024-12-09"),
                version: "0001"
            }
        }])
    })
}

class ResourceController {
    static async findOne(id: string, ctx: unknown) {
        // Retrieve a specific user from the database
        return await userFromDatabase(id, ctx)
    }

    static async findMany(filter?: SCIMMY.Types.Filter, constraints?: SCIMMY.Types.Resource<SCIMMY.Schemas.User>["constraints"], ctx?: unknown) {
        // Faking retriving a list of users from the database
        return await usersFromDatabase(filter, constraints, ctx)
    }
}

SCIMMY.Resources.declare(SCIMMY.Resources.User)
    // Uncomment the lines below to see the typing error on the handler
    .egress(async (resource, ctx) => {
        // 1. This will raise a typing error on the handler
        if (resource.id) return await ResourceController.findOne(resource.id, ctx);
        else return await ResourceController.findMany(resource.filter, resource.constraints, ctx);
        
        // // 2. But if the handler return twice the same type, no typing error
        // if (resource.id) return await ResourceController.findOne(resource.id, ctx);
        // else return await ResourceController.findOne("1234", ctx);

        // // 3. But if the handler return twice the same type, no typing error
        // if (resource.id) return await ResourceController.findMany(resource.filter, resource.constraints, ctx);
        // else return await ResourceController.findMany(resource.filter, resource.constraints, ctx);
    })

[sleelin]

Hi @ymoriaud, thanks for testing this for me!
I can confirm that the partial type checking is on the return values of the ingress/egress handlers as in your examples.

I believe the issue in example 1 is because for some reason TypeScript seems to think that:

Promise<UserFromDatabase>|Promise<UserFromDatabase[]>

is somehow different to:

Promise<UserFromDatabase|UserFromDatabase[]>

I've updated the expected type in the JSDoc annotation for the EgressHandler callback to the latter format with the type union wrapped in a single Promise, and testing with your example code seems to show example 1 working as expected (allowing return of one or multiple users in the handler, instead of complaining about incompatible types).
In examples 2 and 3 the returned types are the same, so I suspect TypeScript was able to infer the return type matched one of the expected values (e.g. either the Promise<UserFromDatabase> or the Promise<UserFromDatabase[]> type, as opposed to in example 1 where it misunderstood the union).

I've now released v1.3.1 which should include the correct type, would you mind updating again and reporting whether it now behaves as expected?

Thanks,
Sam

[ymoriaud]

Awesome ! Using the v1.3.1 I don't have the typing "error" anymore. Thanks :)