Shimmer is an application that makes it easy to pull health data from popular third-party APIs like Runkeeper and Fitbit. It converts that data into an Open mHealth compliant format, letting your application work with clean and clinically meaningful data.
We currently support the following APIs
This README should have everything you need to get started. If you have any questions, feel free to open an issue, email us, post on our form, or visit our website.
- Overview
- Installation
- Registering with third-party APIs
- Postman collection
- Authorizing access to a third-party user account
- Reading data
- Supported APIs and endpoints
- Contributing
Shimmer is made up of different components - individual shims, a resource server, and a console - which are each described below.
A shim is a library that can communicate with a specific third-party API, e.g. Fitbit. It handles the process of authenticating with the API, requesting data from it, and mapping that data into an Open mHealth compliant data format.
A shim generates data points, which are self-contained pieces of data that not only contain the health data of interest, but also include header information such as date of creation, acquisition provenance, and data source. This metadata helps describe the data and where it came from. The library is called a shim because such clean and clinically significant data is not provided natively by the third-party API.
The resource server exposes an API to retrieve data points. The server handles API requests by delegating them to the correct shim. As more and more shims are developed and added to the resource server, it becomes capable of providing data points from more and more third-party APIs. The resource server also manages third-party access tokens on behalf of shims.
The console provides a simple web interface that helps users interact with the resource server. It can trigger authentication flows, and request data using date pickers and drop downs.
There are two ways to install Shimmer.
- You can download and run pre-built Docker images.
- You can build all the code from source and run it natively or in Docker.
If you don't have Docker and Docker Compose, please set them up. (Docker for Mac, Docker for Windows).
Once you're set up, in a terminal
- Clone this Git repository.
- Download and start the containers using either
docker-compose up -d resourceserver- to bring up only the resource server
docker-compose up -d- to bring up the resource server and the console
- This will download up to 0.5 GB of Docker images if you don't already have them, the bulk of which are the underlying MongoDB, nginx and OpenJDK images.
- If you want to see logs and keep the containers in the foreground, omit the
-d.
- It can take up to a minute for the containers to start up. You can check their progress using
docker-compose logsif you started with-d. - The console container publishes port 8083 and the resource server container publishes port 8084.
- The console container proxies all API requests to the resource server container, so you can send API requests to port 8083 or port 8084.
- Visit
http://<shimmer-host>:8083in a browser to open the console.
If you prefer to build the code yourself,
- You must have a Java 8 or higher JDK installed. You can use either OpenJDK or the Oracle JDK.
- If you're building the optional console,
- You need Node.js.
- You need Xcode Command Line Tools if you're on a Mac.
- To run the code natively, you need a running MongoDB instance.
- To run the code in Docker, you need Docker and Docker Compose.
If you want to build and run the code natively, in a terminal
- Clone this Git repository.
- Run the
./run-natively.shscript and follow the instructions. - When the script blocks with the message
Started Application, the components are running.- Press Ctrl-C to stop them.
- The script creates a WAR file which you can alternatively drop into an application server. This issue has details.
- Visit
http://<shimmer-host>:8083in a browser to open the console.
If you want to build and run the code in Docker, in a terminal
- Clone this Git repository.
- Run the
./run-dockerized.shscript and follow the instructions.- The containers should now be running on your Docker host and expose ports 8083 and 8084.
- It can take up to a minute for the containers to start up.
- Visit
http://<shimmer-host>:8083in a browser to open the console.
If you can't run the Bash scripts on your system, open them and take a look at the commands they run. The important commands are marked with a "#CMD" comment.
To get data from a third-party API, you need to visit the developer website of that API and register a client application. The registration information that you give to the third-party lets them show relevant information to their end users about your application, and lets them manage other operational concerns like authorization and rate limits.
You will be given a set of client credentials, usually an OAuth client ID and client secret, for each application you register. You may also need to enter a redirect URL, which is the URL a user is sent to after granting your application access to their data.
The following table contains a link to the developer portal of each API and information about redirect URL restrictions. The restrictions can be good to know about during development, but TLS and full URLs should be used during production.
| API | requires TLS | allows non-FQDN hostname | allows IP addresses | allows localhost | requires URL path | example |
|---|---|---|---|---|---|---|
| Fitbit1 | false | true | true | true | true | http://localhost:8083/authorize/fitbit/callback |
| Google Fit | false | false | false | true | ? | http://localhost:8083/authorize/googlefit/callback |
| iHealth2 | ? | ? | ? | ? | false | http://localhost:8083/authorize/ihealth/callback |
| Jawbone UP | false 3 | ? | ? | ? | ? | http://localhost:8083/authorize/jawbone/callback |
| Misfit | ? | ? | ? | ? | ? | http://localhost:8083/authorize/misfit/callback |
| Moves | ? | ? | ? | ? | ? | http://localhost:8083/authorize/moves/callback |
| RunKeeper | ? | ? | ? | ? | ? | http://localhost:8083/authorize/runkeeper/callback |
| Withings | ? | true | ? | true | ? | http://localhost:8083/authorize/withings/callback |
1 Fitbit has deprecated OAuth 1.0a authorization in favour of OAuth 2.0. You will need OAuth 2.0 credentials.
2 You'll need to copy the iHealth SC and SV values found via the application management page
into the application.yaml or resource-server.env file.
3 The documentation states TLS is required, but authorization does work without it.
If any of the links or fields are incorrect or out of date, please submit an issue to let us know. This table will be fully populated in in the coming days.
Visit the links to register and configure your application for each of the APIs you want to use. Once credentials are
obtained for a particular API, you can either set the corresponding values in the application.yaml file and rebuild, or if you're running using Docker, set the corresponding values in the resource-server.env file.
A Postman collection is provided that makes it easy to experiment with Shimmer's API. An environment is also provided that contains settings for making API requests, such as Shimmer's host, port, and request parameters.
To set up the Postman environment,
- Click the cog wheel in the top right.
- Choose Manage Environments.
- Click the Import button and choose the file called
resources/postman/postman-environment.jsonfrom this repo. - Close the Manage Environments modal.
- Select the environment you just created from the drop-down in the top right.
To import the Postman collection,
- Click the Import button in the top left.
- In the Import File tab, click Choose Files and choose the file called
resources/postman/postman-collection.jsonfrom this repo. - If you already have a collection with the same name, replace it.
The data produced by a third-party API belongs to some user account registered on the third-party system. To allow a shim to read that data, you'll need to initiate an authorization process. This process lets the user account holder explicitly grant the shim access to their data.
To initiate the authorization process using Postman,
- Click the Environment quick look button in the top right of Postman and click Edit to edit the environment.
- Set the
usernamevalue to any unique identifier you'd like to use to identify the user. - Set the
shim-keyvalue to one of the keys listed below, e.g.fitbit. - Run the Authorization\Initiate authorization request.
- Find the
authorizationUrlvalue in the returned JSON response and load the URL in a browser. You will land on the third-party website where you can login and authorize access to your third-party user account. You should then be automatically redirected back to Shimmer where the OAuth flow will complete.
To initiate the authorization process programmatically,
- Make a GET request to
http://<shimmer-host>:8083/authorize/{shimKey}?username={userId}- Use port 8084 if you're not running the console container.
- The
shimKeypath parameter should be one of the keys listed below, e.g.fitbit. - The
usernamequery parameter can be set to any unique identifier you'd like to use to identify the user.
- Find the
authorizationUrlvalue in the returned JSON response and redirect your user to this URL. Your user will land on the third-party website where they can login and authorize access to their third-party user account. - Once authorized, they will be redirected to
http://<<shimmer-host>:8083/authorize/{shimKey}/callback.
To initiate the authorization process from the console,
- Type in an arbitrary user handle. This handle can be anything, it's just your way of referring to a user.
- Press Find and the console will show you a Connect button for each API with configured authentication credentials.
- Click Connect and a pop-up will open.
- Follow the authorization prompts.
- After following the prompts, you should see an
authorization successfulresponse in the pop-up. - The pop-up will then automatically close.
A shim can produce JSON data that is either normalized to Open mHealth schemas or in the raw format produced by the third-party API. Raw data is passed through from the third-party API. Normalized data conforms to Open mHealth schemas.
The following is an example of a normalized step count data point retrieved from Jawbone:
{
"header": {
"id": "243c773b-8936-407e-9c23-270d0ea49cc4",
"creation_date_time": "2015-09-10T12:43:39.138-06:00",
"acquisition_provenance": {
"source_name": "Jawbone UP API",
"modality": "sensed",
"source_updated_date_time": "2015-09-10T18:43:39Z"
},
"schema_id": {
"namespace": "omh",
"name": "step-count",
"version": "1.0"
}
},
"body": {
"effective_time_frame": {
"time_interval": {
"start_date_time": "2015-08-06T05:11:09-07:00",
"end_date_time": "2015-08-06T23:00:36-06:00"
}
},
"step_count": 7939
}
}To pull data from a third-party API using Postman,
- If you need to modify the environment, click the Environment quick look button in the top right of Postman and click Edit to edit the environment.
- Fill in the date range you're interested in by setting the
start-dateandend-datevalues. - Set the
normalizedvalue totruefor data that has been converted to an Open mHealth compliant format, orfalsefor raw data. - Run the Data points request you're interested in.
Most requests also have sample responses you can look at by clicking the Examples button in Postman. Please let us know if any examples you need are missing, or provide pull requests to contribute.
To pull data from a third-party API programmatically, make requests in the format
http://<<shimmer-host>>:8083/data/{shimKey}/{endpoint}?username={userId}&dateStart=yyyy-MM-dd&dateEnd=yyyy-MM-dd&normalize={true|false}
Use port 8084 if you're not running the console container.
The URL can be broken down as follows
- The
shimKeyandusernamepath variables are the same as above. - The
endpointpath variable corresponds to the type of data to retrieve. There's a table of these below. - The
normalizeparameter controls whether the shim returns data in a raw third-party API format (false) or in an Open mHealth compliant format (true).
N.B. This API may change significantly in the future to provide greater consistency across Open mHealth applications and to improve expressivity and ease of use. The data points it returns will not be affected, only the URLs used to request data and perhaps some book-keeping information at the top level of the response.
To pull data from a third-party API using the console,
- Click the name of the connected third-party API.
- Fill in the date range you're interested in.
- Press the Raw button for raw data, or the Normalized button for data that has been converted to an Open mHealth compliant data format.
The following is a table of the currently supported shims, their endpoints, the Open mHealth compliant data produced, and the corresponding mapper. The values in the shim key and endpoint columns are the values for the parameters of the same names used in programmatic access of the API.
The currently supported shims are:
1 The Fitbit API doesn't provide time zone information for the data points it returns. Furthermore, it is not possible to infer the time zone from any of the information provided. Because Open mHealth schemas require timestamps to have a time zone, we need to assign a time zone to timestamps. We set the time zone of all timestamps to UTC for consistency, even if the data may not have occurred in that time zone. This means that unless the event actually occurred in UTC, the timestamps will contain an incorrect time zone. Please consider this when working with data normalized into OmH schemas that are retrieved from the Fitbit shim. We will fix this as soon as Fitbit makes changes to their API to provide time zone information.
2 The configuration file controls whether to serve Fitbit intraday or summary data and at what granularity (see application.yaml or resource-server.env for details). Intraday activity requests are limited to 24 hours worth of data per request. Fitbit must enable intraday access explicitly for your application (click the endpoint link for details). Attempting to generate normalized data with the intraday access property set to true, but when your API credentials have not been granted intraday access, will result in an error.
3 The heart rate mapper has not been tested on real data from Jawbone devices. They have been tested on example data provided in Jawbone API documentation. Please help us out by testing Shimmer with real-world data of one of these types from a Jawbone device and letting us know whether or not it works correctly.
4 Moves time zone handling needs to be tested further, as it's not clear if the time zone assumptions in the mappers are correct.
5 The Withings configuration controls whether to serve intraday or summary data (see application.yaml or resource-server.env for details). Intraday activity requests are limited to 24 hours worth of data per request.
6 Sleep data has not been tested using real data directly from a device. It has been tested with example data provided in the Withings API documentation. Please help us out by testing real-world Withings sleep data with Shimmer and letting us know whether or not it works correctly.
The list of supported third-party APIs will grow over time as more shims are added. If you'd like to contribute a shim to work with your API or a third-party API, or contribute any other code,
- Open an issue to let us know what you're going to work on.
- This lets us give you feedback early and lets us put you in touch with people who can help.
- Fork this repository.
- Create your feature branch from the
developbranch. - Commit and push your changes to your fork.
- Create a pull request.