Skip to content

Release 0.1.0

Compare
Choose a tag to compare
@nivemaham nivemaham released this 10 Sep 13:12
· 434 commits to master since this release
d7fc0af

A generalised REST source connector:

  • Requests are generated by a request generator
  • A request generator can have multiple request routes (e.g. URL's), that are themselves request generators
  • A specialisation of a route is a polling route, that is expected to regularly poll a URL.
  • Use OkHttp3 for HTTP handling

Made a generalised Fitbit source connector:

  • Fitbit user registration is done outside of the connector, managed by a FitbitUserRepository.
    • Its default implementation just reads and writes data from a directory using YAML files. It also refreshes any OAuth2 tokens when necessary.
  • With a FitbitRequestGenerator that has 4 routes:
    1. intraday heart rate
    2. intraday steps
    3. sleep
    4. time zone
  • Each of the routes is a FitbitPollingRoute, with a separate polling counter per Fitbit user. It features a full historical pull for the first iteration, after which only the most recent history is polled. If data is added more recently than before-yesterday, no new data is expected for earlier than before-yesterday. If no data was added since 14 days, earlier days are also not reinspected.
  • Data for each of the route is converted to Avro using RADAR Schemas.
  • The connector backs off for a given user if the server indicates that too many requests are being made.
  • Offsets are committed to the offset file, which is also read on startup.

Available under open source Apache License version 2.0