Earthquakes - the gRPC service

A gRPC service for monitoring earthquakes based on data provided by the USGS.

Introduction

This repository contains gRPC message and service definitions for integrating a backend server (providing earthquake data) and frontend user interfaces (any client supporting gRPC).

It contains also a backend server implementation based on Go that fetches and caches data from the GeoJSON feed of the USGS Earthquake Hazards program.

USGS already provides RESTful or JSON services for earthquakes, so why bother to build a gRPC service for exactly same data? Well the main reasons for this repository are:

• Learning and evaluating possibilities of gRPC and Protocol Buffers.
• Developing an optimized backend for mobile clients.

Also some additional benefits for using gRPC:

• The API - both data model and service calls - is well defined using the service definition language that's easy to understand.
• With API definitions it's easy to generate compatible client and server stubs for different languages.
• Tools for gRPC are battle-tested and integrations for all relevant platforms exists.
• A service can be optimized as a mobile backend that sends only queried and filtered data to mobile clients.
• Efficient binary encoding for wire transfer as a bonus.

Running the quake-server

To run you need to have Go (at least version 1.13) installed on your system.

The quake-server has a main() function in the cmd/quake-server/main.go source file.

Under a local repository clone please compile and run:

$ cd cmd/quake-server
$ go build
$ ./quake-server

This runs a gRPC server instance listening on a local port 50051. By setting enviroment variable PORT you can modify a port to be listened.

Commands above create an executable file under a source folder. To clean up:

$ cd cmd/quake-server
$ go clean

Testing using the quake-client

The quake-client is a simple CLI command to test a gRPC service and it has a main() function in the cmd/quake-client/client.go source file.

Under a local repository clone please compile and run:

$ cd cmd/quake-client
$ go build
$ ./quake-client help

By default the client accesses localhost:50051 server address. This can be modified by setting a QUAKE_SERVICE environment variable.

Some examples for requesting earthquakes using command line:

$ ./quake-client ListEarthquakes significant 7days
$ ./quake-client ListEarthquakes 2.5 day 5 

Commands above create an executable file under a source folder. To clean up:

$ cd cmd/quake-client
$ go clean

gRPC definitions

The service - or the integration layer between a server and a client - is defined using Protocol Buffers with .proto files under the api/v1 folder.

Package names:

Platform	Package
Protocol buffers	quake.api.v1
Go	github.com/navibyte/quake/api/v1

The QuakeService provides following RPC methods:

Method	Description
ListEarthquakes	Get list of earthquakes for given period and magnitude.
GetEarthquake	Get an earthquake by id.

Service definition as a diagram:

Data structures used by the methods represent data based on the GeoJSON Summary Format of the USGS Earthquake Hazards program. Data structures are optimized for the wire usage.

The domain model defined as Protocol Buffer messages:

Message	Description
EarthquakeCollection	Contains a list of earthquakes, metadata and geographic bounds.
Earthquake	An earthquake with id, properties and geographic position with optional reference to detailed information (on EarthquakeDetails).
EarthquakeDetails	Detailed properties for an earthquake.
EarthquakeMetadata	Meta data for a set of earthquakes.

Location data is modeled as messages:

Message	Description
GeoBoundsE7	Geographical bounding box (min and max points) in E7 format.
GeoPointE7	Geographical point (latitude, longitude, height) in E7 format.

There are also some enums used by the domain model:

Enum	Description
Alert	An alert as suggested level of response for an earthquake occurred.
Status	Whether earthquake data is reviewed by a human or not.
Type	A type of a seismic event, like 'earthquake' or 'quarry'.

Message definitions as a diagram:

Go module

There is a standard Go module configuration with go.mod file on the repository root:

• Go version (minimum): 1.13
• Module name: github.com/navibyte/quake

Dependencies (see go.mod for version information):

• github.com/golang/protobuf
• github.com/stretchr/testify (indirect)
• google.golang.org/grpc
• github.com/tidwall/gjson

On this repository:

• generated Go code under api folder, please do not edit generated files by hand
• Go source code under cmd, internal and pkg folders structured by following guidelines from Standard Go Project Layout

Code generation

The repository contains .proto files for API definitions and also generated .go files for server interface code and client side stubs.

If you like or need to regenerate server interfaces and client stubs for Go then you need a development environment with following command line tools:

• protoc (Protocol Buffer v3)
• protoc-gen-go (protoc compiler extension for Go)

Please consult following quick start quide to setup your environment first:

• Go Quick Start for gRPC

When needing regeneration then on your development enviroment cd to the api/v1 folder and run make command to generate Go code. Generated code is stored on the same folder with .proto-files.

This process is defined on the Makefile. Please note that it assumes that the repository folder name on a local environment is quake - nothing else.

Go source and generated code

Package github.com/navibyte/quake/api/v1:

Source	Description
quake.go	Generated Go code for the domain model.
quake_api.go	Generated Go code for client stubs and abstract service classes of the QuakeService.

Package github.com/navibyte/quake/cmd/quake-client:

Source	Description
client.go	A test client for accessing a QuakeService based on gRPC.

Package github.com/navibyte/quake/cmd/quake-server:

Source	Description
main.go	main() for opening a TCP-listener and starting a gRPC-server.
mock.go	Mocks for creating mock earthquake objects for dev test purposes only.
server.go	The implementation for QuakeService delegating actual request processing for code available on the package github.com/navibyte/quake/pkg/earthquakes/usgs.

Package github.com/navibyte/quake/internal/geolib:

Source	Description
geo_e7.go	Helper functions to convert latitude and longitude between double and E7 integer representations. Also a method to calculate distances using the haversine formula.

Package github.com/navibyte/quake/internal/jsonlib:

Source	Description
cursor.go	A helper wrapper to gjson library that is a fast JSON parser for Go.

Package github.com/navibyte/quake/internal/mathlib:

Source	Description
math.go	Few simple math related helper functions.

Package github.com/navibyte/quake/pkg/earthquakes/usgs:

Source	Description
cache.go	A local in-memory-object cache for earthquake data fetched. Also synchronizes calls to remote service (see fetch.go) to ensure that only one fetch operation is active for each remote resource.
fetch.go	Calls the REST/JSON remote service (USGS) to fetch earthquake data.
parse.go	Parses GeoJSON data structures to domain model structures generated by a gRPC tool.
repository.go	Implements GetEarthquake and ListEarthquakes functions using caching, fetching and parsing functionality.

There are also unit tests (*_test.go) available for source code files on this usgs package testing caching, parsing and the whole repository.

Authors

This project is authored by Navibyte.

License

This project is licensed under the MIT License - see the LICENSE.

Links

Domain model

More information about domain model of USGS earthquake data:

• About USGS Earthquake Hazards program
• GeoJSON Summary Format
• ComCat Documentation - Event Terms
• ComCat Documentation - Metadata Terms
• PAGER Scientific Background

Schema language

External references consulted when designing protocol buffer schemas on this repository:

• gRPC - A high performance, open-source universal RPC framework
• Language Guide (proto3)
• Uber Protobuf Style Guide V2
• Buf - Style Guide
• Protocol Buffers - Style Guide

gRPC

Tutorials consulted when building the demo app:

• Go Quick Start