-
Notifications
You must be signed in to change notification settings - Fork 59
Service contract design
#Contract design
This section talks about semantics and conventions that need to be used in designing contracts across the modules. As we break down all modules into μ-services, it would be useful to have guiding conventions to ensure that structure and thinking reflects into each module consistently.
Before we dive deeper into the conventions, it would be useful to be cognizant of existing constraints that may impact our thinking of contracts.
- In an ideal system that is broken down into μ-services it may be ideal to isolate reads from writes. While read logs are kept in sync with writes by streaming events. It is worth noting that we are constraining ourselves intentionally in that form of architecture by sharing a database to share state between the read and write components.
- Our message bus payload is a reflection of what we receive as the message on our REST web services. A copy of payload is put onto the message bus with minimal transformation instead of reflecting a delta write event.
- For developer productivity and in an attempt to reduce the steep learning curve for the team involved we have decided to use JSON to define message payload for both REST calls and message bus. Serialization and Deserialization are managed by individual services.
- We work with a strongly typed language, so it is not always possible to use deserialized payload as concrete objects and define a varying contract at the runtime both.
We have two distinct channels via which the individual services in our infrastructure communicate. It is important to build the differentiation between both.
REST services are one of the two entry points that are exposed in our architecture by different services both for internal and external consumers like desktop or mobile. While most REST endpoints will be consumed by external consumers, some services like Boundary service, which happens to be a stand alone service may expose an endpoint that can be used by other internal services to retrieve data synchronously.
In most REST services, the control of the payload downstream lies with the producer of the message. It may or may not have defined REST call payload as the contract for all the consumers downstream. If payload structure is not the same downstream, producer in REST API will consume incoming request and produce a message that is compliant with the payload contract on a specific message bus.
This is second of two dominant communication channel between services in the system. Most of the consumers are internal on this channel. However, the distinctive aspect is that each consumer within a module will only enrich/alter state in a message but will rarely modify the schema of the message for services downstream, as long as they are within the same module.
A service downstream that lies outside a module, may have published a completely separate contract and in instances like that, a consumer holds the responsibility of transforming and publishing new message.
This distinction is important to note as it would have a resulting impact on how messages are deserialized as we will observe in the section below.
Some features that we need to standardize across module to promote consistency around how each module deals with payload contracts for both REST and message bus consumers.
Currently, all services are being written in Java. This section will, therefore, focus on specific technology choice and will talk about how serialization/de-serialization will be managed for two types of consumers.
This is a consumer that exclusively consumes a message from another service. It is not responsible for delegating the message payload further to other downstream services. An example would be service that consumes "UserDetails" from UserService. User service may expose a REST endpoint to fetch details asynchronously. In an event, when consumer service is independent, the binding to contract published by User service should be loose and only in fields that are needed by the service. Any additional or missing fields should be ignored if they are not used by the consumer. An example to illustrate the intent.
UserService JSON response contract
{
"metadata": {
"ver":1,
"ts":"2017-01-23T18:25:43.511Z"
},
"user": {
"name": "Manjari",
"username": "[email protected]"
"age": 32,
"address": "palace grounds, bangalore, 560100"
}
}
An indicative parsing of response by consumer using Jackson JSON parser
public class UserResponse {
@JSONProperty("metadata")
private RequestInfo metadata;
@JSONProperty("name")
private String name;
@JSONProperty("username")
private String username;
@JSONProperty("name")
private Intger age;
@JSONProperty("address")
private String address;
// getters have been omitted...
}
In above parsing, all the fields in the response contracts have been bound by a local deserialized object. If the consumer is using only name and username fields in the previous objects then there is no need to bind to all the fields explicitly. The coupling for independent consumers needs to be with the minimum needed subset.
So a more flexible way to do this would be using Jackson JSON parser as follows:
@JsonIgnoreProperties(ignoreUnknown = true)
public class UserResponse {
@JSONProperty("name")
private String name;
@JSONProperty("username")
private String username;
// getters have been omitted...
}
This would ensure that any changes or deletions to remaining fields go unnoticed and contracts remain forward-compatible as long as the changes are not reflected in the fields that are being actively used.
We saw how contract association could be kept at a minimal subset. This is not necessarily true when the consumer is an intermediary one. It means that there are more consumers downstream that rely on original message payload. By binding to a contract that subscribes to a subset of fields, you loose the information that is present as part of the original message. Therefore, the binding cannot be with sub-set. Yet, if the binding is explicit then any change in contract affects everyone downstream. This would not be ideal.
In a case of an intermediary consumer, the approach would be to deserialize the JSON as a java.util.Map. A secondary conversion could be done from a Map to a statically typed object like UserResponse, however contents of the java.util.Map should be used to pass the message downstream. This would ensure that while individual consumers rely on specific fields in a contract, they are able to relay the information needed downstream.
The primary downside is that some of the Kafka consumer frameworks, which have a built-in support for deserializing a message into an object directly, can't be used. Since they would lose information as part of that conversion. So deserialization would become a two-step process, where Kafka consumers always return a java.util.Map and then a specific object is extracted from it for use. This trade-off is worth having to be able to remain forward-compatible with the implementation in future.
Services publish contracts as a means to establish communication with consumers. Service versions are a reflection of the evolution of contract. Internal changes in services that do not impact contract schema are not useful enough to version. A version is created when contract schema undergoes a modification.
Forward compatibility is an aspect of a consumer contract that allows consumers to remain unaffected by contract schema changes in future that they do not consume. We have already discussed two approaches in the previous section to indicate how we must remain forward-compatible for both independent consumers and intermediary consumers.
Jackson or any other JSON parser that allows for ignoring fields except the ones specified will allow us to maintain forward compatibility.
Backward compatibility refers to the ability to support the past and present versions of contract together at the service end. Backward compatibility may consider immediate past version or multiple past version. It is hard to conceive the changes that would require backward compatibility. So it is essential that we focus on support attributes that would allow us to manage different versions in future. The most crucial of those is version.
It is therefore recommended that each payload contains "version" header in the payload so that it can be leveraged in future to manage different versions that need to co-exist.
How backward compatibility will be managed, will be done on a case to case basis in future when the need arises. Version field will be used to distinguish payloads and act accordingly.
It is important to be able to communicate schema changes between different services. A combination of convention and tooling can make this a standard and reliable process in our architecture. All services should publish their payload schema definitions in root level contract folder. This can be generated using JSON schema specification here and available tools. The contracts should be versioned and available across all versions to consumers.
Each consumer can pull down a schema definition and ensure that based on their binding needs, the contract is still valid. These checks could be set up as part of unit testing for all the contracts across services that are consumed. Once a test fails, it indicates a breaking change and could prompt the team to either upgrade or manage contract dependencies accordingly.
To ensure that we have all JSON payloads produced and consumed consistently, following style guide should be mandated. This has been derived from the original JSON style guide published by google here.
- No comments in JSON payload
- Use double quotes only
- Choose meaningful property maps
- Arrays should have plural property names. All other should be singular.
- Avoid naming conflicts
- Naming should be done only in cameCase with the first alphabet in lower case. E.g.
apiVersion,magicField,keyetc.
- Values must be Unicode booleans, numbers, strings, objects, arrays, or null.
- Consider removing null values
- Enums should be sent as strings
- Dates should be formatted as recommended by RFC 3339.
- Time durations should be formatted as recommended by ISO 8601.
- Latitudes/Longitudes should be formatted as recommended by ISO 6709.