Multi tenancy support on Document Objects

[harshalizode]

Hello,
I'm working on implementing the feature for the document method to execute operations on many databases at once.
In current implementation we are retrieve the collection object from the model class meta, before querying into database using the document object methods.
To implement the feature, I am overriding the following methods:

• _all(): This method is used to generate the Document object from the filtered results. So, while creating the Document object, we may specify the additional kwargs argument as from_collection. It is a nullable field in the from_row() method; if the from_collection is given, we can utilize that collection; otherwise, we can continue with the model class meta collection.

Is there another strategy to implementing this functionality that we can take?

[tarsil]

@harshalizode apologies for the delay but this week has been very busy. We will need to see what you mean with care and come back to you

[harshalizode]

Okay.

[harshalizode]

Hello I have implemented the logic partially can you check is I am implementing it in correct way.

firstly I have added the from_collection attribute in the meta field as:

class MetaInfo:
    __slots__ = (
        "pk",
        "id_attribute",
        "abstract",
        "fields",
        "registry",
        "collection",
        "indexes",
        "parents",
        "signals",
        "database",
        "manager",
        "autogenerate_index",
        "from_collection"
    )

then I have overrided the from_row() method in the Document as:

class BaseDocument(Document):
    """
    The base document for all documents using the `registry` registry.
    """

    @classmethod
    def from_row(
        cls: "Document",
        row: Dict[str, Any],
        is_only_fields: bool = False,
        is_defer_fields: bool = False,
        only_fields: Union[Sequence[str], None] = None,
        defer_fields: Union[Sequence[str], None] = None,
        from_collection:  Union[AsyncIOMotorCollection, None] = None  # type: ignore
    ) -> Union[Type["Document"], None]:
        """
        Class method to convert a dictionary row result into a Document row\
            type.
        :return: Document class.
        """
        model = super().from_row(row, is_only_fields=is_only_fields,
                                 is_defer_fields=is_defer_fields,
                                 only_fields=only_fields,
                                 defer_fields=defer_fields)
        model.Meta.from_collection = from_collection
        return model

Here, I have added the keyword argument from_collection and from here only we can set the collection object in the meta class of requested model.

In QueryManager we can override the method:

async def _all(self) -> List[T]:
        """
        Returns all the results for a given collection of a document
        """
        manager: "Manager" = self.clone()

        filter_query = Expression.compile_many(manager._filter)
        cursor = manager._collection.find(filter_query)

        if manager._sort:
            sort_query = [expr.compile() for expr in manager._sort]
            cursor = cursor.sort(sort_query)

        if manager._skip_count:
            cursor = cursor.skip(manager._skip_count)

        if manager._limit_count:
            cursor = cursor.limit(manager._limit_count)

        # For only fields
        is_only_fields = True if manager._only_fields else False
        is_defer_fields = True if manager._defer_fields else False

        results: List[T] = [
            manager.model_class.from_row(
                document,
                is_only_fields=is_only_fields,
                only_fields=manager._only_fields,
                is_defer_fields=is_defer_fields,
                defer_fields=manager._defer_fields,
                from_collection=manager._collection
            )
            async for document in cursor
        ]

        return results

Here, I have have passed the manager._collection to the from_collection while creating the document object. If any user change the database at the run time then we can store the collection instance to query on the same requested database, rather than taking the collection from the database.

Using this approach we can first check the from_collection meta variable has the collection object or not, if yes then we will use this to further query otherwise we can continue with the cls.meta.collection._collection. after using the from_collection we can garbage it or we can override in every request.

Is it right way?

Without this we need to pass the collection object to do as such operation on the requested database. Developer need to pass the collection objects to query on particular database.
Using this approach we can Implement the full multi tenancy support on each query, without passing the collection object.

[harshalizode]

With this implementation where we uses the cls.meta.collection._collection to query on database, where we need to apply the check for the from_collection if it is provided use this to query on database otherwise use the cls.meta.collection._collection.

[harshalizode]

@tarsil any suggestions on it? or did something need to be improved?

[harshalizode]

@tarsil can we create the pull request for it? or any improvement requires?

[tarsil]

@harshalizode apologies. For sure, open a pull request for it. Let me have a look. There is no specific direct API for this so it will be very interesting to see what you came out with.

[harshalizode]

@tarsil I added the pull request, and all of the test cases were passed.