Release 0.1.0
A generalised REST source connector:
- Requests are generated by a request generator
- A request generator can have multiple request routes (e.g. URL's), that are themselves request generators
- A specialisation of a route is a polling route, that is expected to regularly poll a URL.
- Use OkHttp3 for HTTP handling
Made a generalised Fitbit source connector:
- Fitbit user registration is done outside of the connector, managed by a
FitbitUserRepository
.- Its default implementation just reads and writes data from a directory using
YAML
files. It also refreshes any OAuth2 tokens when necessary.
- Its default implementation just reads and writes data from a directory using
- With a
FitbitRequestGenerator
that has 4 routes:- intraday heart rate
- intraday steps
- sleep
- time zone
- Each of the routes is a
FitbitPollingRoute
, with a separate polling counter per Fitbit user. It features a full historical pull for the first iteration, after which only the most recent history is polled. If data is added more recently than before-yesterday, no new data is expected for earlier than before-yesterday. If no data was added since 14 days, earlier days are also not reinspected. - Data for each of the route is converted to Avro using RADAR Schemas.
- The connector backs off for a given user if the server indicates that too many requests are being made.
- Offsets are committed to the offset file, which is also read on startup.
Available under open source Apache License version 2.0