The CRUD Service is a lightweight application that exposes an HTTP Interface to perform CRUD operations on MongoDB collections defined via JSON Schema, to make sure to have consistent data in every operation. Moreover, your documents will includes some metadata regarding the creation and the latest update of the document, and a state that will come in handy to better manage draft documents and virtual delete.
The CRUD Service can be launched with lc39, that will start a Fastify instance that will include the functionalities. It also includes a Swagger User Interface to be used to perform any operation for each of the collections defined.
Moreover, the CRUD Service can be configured used with the MongoDB Data Encryption functionality to automatically encrypt data stored in the Database.
To deploy the service in your local environment, you must have:
- Node at version 18 or superior,
- MongoDB instance at version 4.0 or superior
To setup Node, we suggest you to install nvm to have multiple node versions in your computer, and use the version suggested in the .nvmrc
file.
nvm install # <-- to be done only the first time
nvm use
About mongo, we suggest you to have it in a separate Docker container. You can follow this small guide or use the configuration you prefer:
docker pull mongo:6.0
docker volume create mongo
docker run --detach --name mongo -p 27017:27017 --mount source=mongo,target=/data/db mongo:6.0
In case you want to use the MongoDB Encryption functionality, you must install in your machine the MongoDB Enterprise Server and run the mongocryptd
file (its location may change based on the version you downloaded).
To run the CRUD Service, a *.env
file including the needed configuration is required. An example of this file is the default.env file. You want to copy the file in a different path to make sure your configuration is ignored by Git (we suggest the name local.env
):
cp ./default.env ./local.env
The default.env
only lists the necessary environment variables to be added to successfully run the CRUD Service in your local machine. A complete list of the environment variables can be found on the related page.
Once you have your env file, you might want to update the COLLECTION_DEFINITION_FOLDER
and VIEWS_DEFINITION_FOLDER
environment variables, which represent the folders where the definition of collections and views are found. The CRUD Service accepts absolute paths only, so please verify them. The default values are folders used in unit tests: if you plan to update the files inside for your needs, it's better to use another folder.
When everything is ready, you can run the service:
nvm use # <-- only if you use nvm
npm i
npm run start:local # <-- run the service using the 'local.env' file
Docker images of the CRUD Service are available on GitHub Registry and DockerHub. An image is released in those two registries for every tag created.
If you instead prefer to create your image (e.g. from your fork), you can use the Dockerfile
to generate your image:
docker build -t crud-service .
Thanks to the support of Docker BuildKit, you can also decide to create an image of the CRUD Service without including the mongocryptd
libraries:
DOCKER_BUILDKIT=1 docker build -t crud-service-no-encryption --target=crud-service-no-encryption .
If you are interested in it, you can get one and run it locally with these commands:
docker run --name crud-service \
--detach \
--env LOG_LEVEL=info \
--env MONGODB_URL=mongodb://<your-mongo-container-ip>/<database-name> \
--env COLLECTION_DEFINITION_FOLDER=/home/node/app/collections \
--env USER_ID_HEADER_KEY=userid \
--env CRUD_LIMIT_CONSTRAINT_ENABLED=true \
--env CRUD_MAX_LIMIT=200 \
--mount type=bind,source=$(pwd)/tests/collectionDefinitions,target=/home/node/app/collections \
--publish 3000:3000 \
<your-crud-service-image-name>:latest
Please note that you can mount the .env
file with your CRUD Service configuration instead of manually including the environment variables in the command.
You can access the Swagger User Interface, generally available at http://localhost:3000/documentation
(it depends on which port you deployed the service). From there you can verify the list of Collections and Views defined and execute HTTP requests to effectively use the service.
We use tap to test the CRUD Service. Once you have all the dependency in place, you can simply launch it with:
npm run coverage
It will run the tests with the coverage report that you can view as an HTML page in coverage/lcov-report/index.html
.
To run only one test:
env MONGO_HOST=127.0.0.1 TAP_BAIL=1 node tests/createIndexes.test.js
+----------------------+ +----------------------+ +----------------------+ +----------------------+
| | | | | | | |
| +-------------+ | | +-------------+ | | +-------------+ | | |
| | | | | | | | | | | | | +-------------+ |
| | CrudService | | | | CrudService | | | | CrudService | | | | | |
| | | | | | | | | | | | | | JoinService | |
| +-------------+ | | +-------------+ | | +-------------+ | | | | |
| +-------------+ | | +-------------+ | | +-------------+ | | +-------------+ |
| | | | | | | | | | | | | |
| | QueryParser | | | | QueryParser | | | | QueryParser | | | |
| | | | | | | | | | | | | |
| +-------------+ | | +-------------+ | | +-------------+ | | |
| +-------------+ | | +-------------+ | | +-------------+ | | |
| | | | | | | | | | | | | |
| |HTTPInterface| | | |HTTPInterface| | | |HTTPInterface| | | |
| | | | | | | | | | | | | |
| +-------------+ | | +-------------+ | | +-------------+ | | |
| | | | | | | |
| Crud on a collection | | Crud on a collection | | Crud on a collection | | JoinPlugin |
| | | | | | | |
+-----------^----------+ +-----------^----------+ +-----------^----------+ +-----------^----------+
| | | |
+-----------+-------------------------+-------------------------+-------------------------+----------+
| |
| +--------------------+ +--------------------+ |
| | | | | |
| | Model definitions | FASTIFY | Mongodb connection | |
| | | | | |
| +--------------------+ +--------------------+ |
| |
+----------------------------------------------------------------------------------------------------+
The CRUD Service application uses Fastify to connect to a MongoDB instance (thanks to the @fastify/mongodb plugin) and execute CRUD operations. At startup, it requires several details, such as the MongoDB URL to communicate with, but also the folders where the definitions of the collections and the views can be found. A complete and comprehensive list of the environment variables needed can be found on the Configuration page.
The collections and the views available in the service are the ones with Model definitions stored in the folders defined in, respectively, COLLECTION_DEFINITION_FOLDER
and VIEWS_DEFINITION_FOLDER
. In each folder, there will be one file per collection/view. Each file should be a JSON or a javascript file.
These models will be analyzed by the JSONSchemaGenerator class to be transformed to a JSON Schema that will be used to validate HTTP requests, to serialize the output and to return these information to the user as a documentation.
The service exposes several APIs to communicate with the collections. You can send requests as you prefer: you can use curl, use an application such as PostMan, or use the integrated Swagger API Interface accessible to http://{{url}}/documentation
.
When the Service is live, every HTTP request executed will be caught by the HTTP Interface that works as a communication channel with the CRUD Service. The data included with the request (query parameters, body, commands) is evaluated by the QueryParser
class to verify the query to make sure that every value associated to a specific field is in the correct type. Then it's forwarded to the CrudService
class to execute the query. Result of GET
requests are passed to the AdditionalCaster
class to casts GeoPoints and ObjectIds properties in the data.
The collections should be included in separate JSON or JavaScript files in the folder defined with the environment variable COLLECTION_DEFINITION_FOLDER
. Each collection object requires the following fields:
Name | Type | Required | Default value | Description |
---|---|---|---|---|
id | String | - | - | Additional identifier that can be associated to the collection definition. |
endpointBasePath | String | ✓ | - | The endpoint path, used as entry point to CRUD operations |
name | String | ✓ | - | The name of the collection on MongoDB. |
defaultState | String | - | DRAFT |
The default state assigned to a document when inserted. Can be one of the __STATE__ available values |
schema | JSONSchemaStandard | ✓ | - | The JSON Schema configuration of the fields to be included in the collection object. A complete description of its fields can be found in the schema section of the collection JSON Schema. |
indexes | Array of objects | ✓ | - | The list of indexes to be created when starting the service and initializing all the collections. A complete description of its fields can be found in the indexes section of the collection JSON Schema |
tags | Array of strings | - | [] | The list of tags to be associated to the collection's endpoints, useful to group different endpoint under the same section inside the swagger. |
WARNING: The definition of unique indexes makes the CRUD Service fail at startup if the database contains inconsistent documents (e.g. documents that have the same value for that key). Also documents without that key are all considered to have the same value (null), thus violating the uniqueness, and causing the index generation (and the CRUD Service) to fail at startup.
WARNING: every index that is not specified in the collection definition wil be dropped at startup of the application, unless its name starts with the preserve_
prefix.
Several examples of collections can be found in the Collections Definitions folder, whereas the schema that defines and validate the data model definition can be found here.
The collections should be included in separate JSON or JavaScript files in the folder defined with the environment variable COLLECTION_DEFINITION_FOLDER
. Each collection object requires the following fields:
Name | Type | Required | Default value | Description |
---|---|---|---|---|
id | String | - | - | Additional identifier that can be associated to the collection definition. |
endpointBasePath | String | ✓ | - | The endpoint path, used as entry point to CRUD operations |
name | String | ✓ | - | The name of the collection on MongoDB. |
defaultState | String | - | DRAFT |
The default state assigned to a document when inserted. Can be one of the __STATE__ available values |
fields | Array of objects | ✓ | - | The list of fields to be included in the collection object. A complete description of its fields can be found in the fields section of the collection JSON Schema. |
indexes | Array of objects | ✓ | - | The list of indexes to be created when starting the service and initializing all the collections. A complete description of its fields can be found in the indexes section of the collection JSON Schema |
tags | Array of strings | - | [] | The list of tags to be associated to the collection's endpoints, useful to group different endpoint under the same section inside the swagger. |
WARNING: The definition of unique indexes makes the CRUD Service fail at startup if the database contains inconsistent documents (e.g. documents that have the same value for that key). Also documents without that key are all considered to have the same value (null), thus violating the uniqueness, and causing the index generation (and the CRUD Service) to fail at startup.
WARNING: every index that is not specified in the collection definition wil be dropped at startup of the application, unless its name starts with the preserve_
prefix.
Several examples of collections can be found in the Collections Definitions folder, whereas the schema that defines and validate the data model definition can be found here.
The MongoDB Views should be included in separate JSON or JavaScript files in the folder defined with the environment variable VIEWS_DEFINITION_FOLDER
. Each collection object requires the following fields:
Name | Type | Required | Default value | Description |
---|---|---|---|---|
name | String | ✓ | - | The name of the view, used as identifier |
source | String | ✓ | - | The name of the collection to be used as source to generate the view |
type | view |
✓ | - | The type of MongoDB element, which is required by CRUD Service to understand which operations might be performed. It should be the value view . |
pipeline | Object | ✓ | - | The pipeline to aggregate the MongoDB View. It uses the same syntax of the Aggregation Pipeline |
Several examples of collections can be found in the Views Definitions folder, whereas the schema that defines and validate the data model definition can be found here (it is the same schema of the collection definition).
Note: __STATE__
field must be returned in each record eventually produced by the view aggregation pipeline.
On the contrary, records without the __STATE__
field would always be filtered out by the CRUD Service operations
(e.g. listing records via GET /<collection-name>
API method would not consider them in the result set).
The CRUD service offers the functionality to modify a view by editing the underlying collection. This enables clients to interact with a view as if it was an independent collection. Additionally, the service will expose additional routes that provide a comprehensive list of all possible values that can be included as lookup values, if any in the view.
To enable this feature, you need to include the enableLookup: true
property in the view configuration JSON. By default, this setting is set to false.
For more information on correctly configuring and understanding the capabilities of writable views, please refer to the writable views documentation.
Every HTTP request trusts in some headers:
- acl_rows allows us to limit the rows that the requester can see. The query to limit the rows is passed a stringified json.
- acl_read_columns allows us to limit the properties to be returned to the requester. The list of properties is passed as a stringified json.
- json-query-params-encoding allows us to use a different encoder for the data passed as query parameters (only supported value: base64)
- user_id includes the identifier of the user executing the request
When working with collections via CRUD Service, some fields will be automatically generated and updated. Those fields are the following:
- updaterId is the user id (string) that requests the last change successfully
- updatedAt is the date (date) of the request that has performed the last change
- creatorId is the user id (string) that creates this object
- createdAt is the date (date) of the request that has performed the object creation
- __STATE__ is the current state of the document, can be one of the following four:
PUBLIC
(by default, only data inpublic
will be shown viaGET
requests)DRAFT
TRASH
(a "soft delete" state, ideally to be visible in the Trash Bin of a backoffice application)DELETED
(marked as delete, it shouldn't be shown in GET requests and it shouldn't be modified with PUT operations)
We've just explained the difference between the four possible states of a document. This property can be set directly during an insert, and can be changed via REST API calls only in case of the following transformations:
- a document in
PUBLIC
can be moved toDRAFT
orTRASH
; - a document in
DRAFT
can be moved toPUBLIC
orTRASH
; - a document in
TRASH
can be moved toDRAFT
orDELETED
; - a document in
DELETED
can be moved to only toTRASH
;
Operations of hard delete are supported, although the permissions over this type of operations is defined via ACL.
This class makes the query to the mongodb collection, manages updaterId
, updatedAt
,
creatorId
, createdAt
, __STATE__
and checks whether operations are allowed.
This class casts the value from the query, the body and the commands in order to insert Date
and ObjectId
and perform other conversions such as GeoPoint
where needed
This class casts the value of the result from a query executed in order to have consistent data with the definition of the collection. It operates only on one type of properties: MongoDB's GeoPoint
.
This piece of code is a communication channel between HTTP and the CrudService.
It uses the QueryParser
to cast the value before forwarding the request to the crudService
, and the AdditionalCaster
to cast the GeoPoints and ObjectIds values inside the response to be returned.
The HTTPInterface includes by default different API methods for every kind of operation. The following are available for both Collections and Views:
Verb | API Method | Description |
---|---|---|
GET | {base URL}/{endpoint}/ | Returns a list of documents. |
GET | {base URL}/{endpoint}/export | Export the collection in different file formats. |
GET | {base URL}/{endpoint}/{id} | Returns the item with specific ID. |
GET | {base URL}/{endpoint}/count | Returns the number of items in the collection. |
For collections, also the following methods are available:
Verb | Method | Description |
---|---|---|
POST | {base URL}/{endpoint}/ | Add a new item to the collection. |
POST | {base URL}/{endpoint}/upsert-one | Update an item in the collection. If the item is not in the collection, it will be inserted. |
POST | {base URL}/{endpoint}/bulk | Insert new items in the collection. |
POST | {base URL}/{endpoint}/state | Change state of multiple items of the collection. |
POST | {base URL}/{endpoint}/{id}/state | Change state of the item with specific ID. |
POST | {base URL}/{endpoint}/validate | Verify if the body of the request is valid for an insertion in the collection. |
POST | {base URL}/{endpoint}/import | Inserts new items in the collection from file (json, ndjson and csv). |
PATCH | {base URL}/{endpoint}/ | Update the items of the collection that match the query. |
PATCH | {base URL}/{endpoint}/{id} | Update the item with specific ID in the collection. |
PATCH | {base URL}/{endpoint}/bulk | Update multiple items of the collection, each one with its own modifications |
PATCH | {base URL}/{endpoint}/import | Update the items in the collection from file (json, ndjson and csv), it must include the ID field |
DELETE | {base URL}/{endpoint}/ | Delete multiple items from the collection. |
DELETE | {base URL}/{endpoint}/{id} | Delete an item with specific ID from the collection. |
All these methods might include additional query parameters to refine the search. To have more detail, you can check the live documentation (available only when a service instance is started locally) or refer to the service documentation overview.
The CRUD Service includes also the join
feature, to join two different models. That feature is served on /join/<type>/:from/:to/export
, where:
- type:
one-to-one
orone-to-many
ormany-to-many
- from: the collection endpoint from which the join starts
- to: the collection endpoint which the join ends to
This API responses always in
application/application/x-ndjson
See the documentation to see which parameters are available.
To change the cryptd version, you can download the binary file from the MongoDB Repositories, then navigate to the following subpath: {version_wanted}/main/binary_amd64/mongodb-enterprise-cryptd_{version_wanted}_amd64.deb
.
For example, for version 5.0.14, the final url of the .deb
will be: https://repo.mongodb.com/apt/debian/dists/bullseye/mongodb-enterprise/5.0/main/binary-amd64/mongodb-enterprise-cryptd_5.0.14_amd64.deb
For leveraging the CRUD Service APIs in your services you can use one of the following libraries based on supported languages
CRUD Service is an active project developed and maintained by Mia Platform, it is distributed open source and it is Apache 2 licensed. You're more than welcome to partecipate by creating and/or partecipating to public discussion and actively partecipate to the development of the application.
The two main communication channel are:
- in the issues tab;
- in the official Mia Platform Community Discussion page (if you want to create a discussion here, please add the CRUD Service tag);
Please read CONTRIBUTING.md for further details about the process for submitting pull requests.
The CRUD Service can be used along with the MongoDB Data Encryption functionality, and the images available includes the libraries to use this feature.
However, the MongoDB Data Encryption should be used along with MongoDB Atlas or MongoDB Enterprise Advanced. When you use the CRUD Service, for personal projects or commercial applications, you should be aware of that and you will have to comply with MongoDB terms of use accordingly.
Mia Platform s.r.l. does not respond to any improper use of the MongoDB Data Encryption or any other MongoDB product.