Service git repository structure

Git repository

Existing constraints

Since we are moving an existing monolith to a new μ-service architecture it is important to embrace the constraints we have during this migration period.

• Shared database: All modules across the monolith share database. While in short term explicit reference constraints can be removed, for a foreseeable future, the database within a module or across modules will still be shared. Each module will own tables and will hold an exclusive right to do a writes (and majority of reads) on tables. It is realistic to assume that within a module all services will write to the same database to tables owned by them. This will pose challenges around who will own the creation and migration of database?

• Read and REST services reside in the same service: In a pure μ-service architecture reads may be completely decoupled from writes. However, we will constrain ourselves to take a middle path and allow ingestion of write data and read via RESTful API in the same service. The database writes will be handed over to a stream which will decouple and do writes in the database by the owning service asynchronously. However, querying remains synchronous and resides alongside ingestion web services.

• Shared contracts: Within a module the RESTful web service's contract is shared by all the downstream workers as the payload is handed over to the streaming pipeline with minimal transformation. Therefore the message on the message bus is not a representation of delat change, instead, it is complete payload that ingestion web service receives.

Granularity

One of the key aspects to consider is to whether keep source code managed together in a code repository like git for a module or a service. Keeping code grouped together at a module has following benefits and drawbacks:

Benefits

• Common contract definitions can be easily shared and managed, since a module level build task can cross build dependent artifacts and dependencies both.
• Database creation and migration can be owned by a single module at repo level thereby removing any contention around who will initiate and manage tables within a module.
• Cross service testing will be relatively simpler to manage by building data seeds/reset at the module level.
• Limited pipelines to manage in CI since single build task across services for modules.

Drawbacks

• Versioning, patching and deployment of an individual service becomes difficult to manage. The structure will slowly enforce granularity of deployment at module level rather than service level.
• A breaking change within a single service in a module will prevent other services within a module from being deployed.
• There is a risk that grouped services may start sharing too much code via shared libs and may become tightly bound.
• Services required by multiple modules will inadvertently get grouped with one of the modules even though it may not belong there logically.

Recommendation

We should manage code at a granularity of a service. Shared libs, if any, should be published via build tools like maven with explicit version dependency specified in pom/build files. While this would mean that functional/behavior testing requires more setup and rigor, it would allow for a granularity of deployment that is service level. Multiple versions can be deployed without being polluted by commits from other services and can be rolled out in an independent manner. This is more in line with long-term vision and while it requires more effort, it prevents coupling and other anti-patterns from occurring. It would also allow for a datastore separation as a next logical step.

Since single service repo distributes the consensus that who will own database creation and migration, we will fallback on conventions and mandate that the RESTful interface exposed by the module will be used for all table creation and schema migrations until db is isolated by individual services.

Repo structure

/repo
    /src
    /test
    /contract
        contract.yml
        payload_def.schema
    /shared
    /libs
    /out
    /db
        /migrations
        /seed
    pom.xml
    Dockerfile
    service.properties

Repo naming conventions

Based on recommendation around granularity of repository, naming for a repository should be standardized so that logical inferences can be made from name itself and grouping of services is done by virtue of naming conventions. Here are the proposed naming considerations.

• Name for a repository is same as that for a service.
• Name should be able to indicate a module association if any. For independent services like user authentication this can be ignored.
• Name should be able to indicate the type of component the current repo represents. There are 3 primary type of components.
  • rest - A rest api that hands over writes to message bus and does reads synchronously.
  • enricher - A consumer downstream that enriches the message and pushes it back onto message bus
  • verb+common-noun - A consumer that does an action represented by a verb like index-complaint, send-email, send-sms, initiate-workflow etc. This type of component/service may be a end of life-cyle for a message on the message bus or it may be forwarded ahead to multiple downstream components.

Based on the guidelines, a naming scheme containing following elements might be a suitable choice:

<master/module_name>-<service-name>-<component-type>

Few examples to elaborate on the scheme that needs to be followed:

• Naming a service that provides boundary details as a stand alone service shared across modules. Has a REST api to fetch details and to create a new boundary.

master-boundary-service-rest

• A PGR module service that enriches boundary location on the seva request and pushes it downstream before it is used for assignment downstream.

pgr-boundary-info-enricher

Here `pgr` reflects that it is part of pgr module. `boundary-info` is an indication of that fact that it uses boundary service and `enricher` is to denote that it enriches the message and pushes it forward.

• A PGR module rest service that ingests all read and write requests. Reads are synchronous and writes are pushed to message bus.

pgr-rest

Here `pgr` reflects that it is part of pgr module. `rest` is indication of that fact that it represents rest interface of the module.

• A send sms service that is used across modules that takes a notification message and sends out a sms.

master-noitifications-send-sms

Here `master` reflects that it is a stand alone service used by modules. `notifications` is the module and `send-sms` is the capability it has. 

Challenges to address in future

• If there are more components types, we would need to identify them and incorporate into our naming conventions without disruption existing scheme too much.
• An implementation of continuous delivery for a single service across different environments may highlight additional challenges that we would need to solve as we move ahead. A lot of thinking is based on certain assumptions around how CI and CD will be done with kubernetes. The deployment section may attempt to address some of the challenges that may occur in the rollout process.
• Grouping database migrations in REST interface of the module would mean that any changes done to database for a service would need a rollout for the REST interface as well. This may be an acceptable tradeoff until we have truly isolated data stores for all the services.