Allow app to sync data across devices

[iSoron]

It would be nice if Loop could synchronize your habit data across multiple devices. This feature has been one of the most frequently requested so far. It also opens many interesting possibilites, such as syncing data with a future desktop or web client.

Since this feature raises many privacy and security concerns, here is my non-comprehensive list of requirements:

1. It should be completely opt-in, and the app should still be fully functional with this feature disabled.
2. If there is software running on a server to make this feature work, this software should be fullly open-source and self-hostable.
3. It goes without saying, but the whole thing should just work. In particular, it must gracefully handle the case where multiple devices make changes while offline, then go online. The solution must be able to merge the offline changes, not simply discard all the changes made all clients, except one.

Because of requirement 2, most commonly used solutions such as Parse, Firebase or Play Games Services are out of the picture. Requirement 3 probably eliminates file-based sync solutions, such as Syncthing, Google Drive or Dropbox.

[iSoron]

I am currently experimenting with custom back-end solution, implemented in node.js, on the branch feature/sync, but it's not usable yet. I will post here once it works somewhat properly.

Other ideas are very welcome, by the way.

[NiklasMM]

Do you want to host that backend yourself and make it available to all users? Or would that be a self-hosted solution, which everyone could set-up for themselves?

[iSoron]

Hi @NiklasMM, I will probably host an instance myself and make it available for all users. I might start accepting donations to keep the server running. In any case, you will also be able to run a private instance of the server yourself.

[quark6789]

Would it be possible to use git for this? Have each device make its own little repo with the data in it, and then either have each sync with a server, or possibly sync directly with each other (decentralized).

For merging, only conflicts for the same habit and day would be an issue. I think choosing the most recent change would be a fine solution.

[colonelpanic8]

THIS. git is quite good at managing conflicts as well.

[iSoron]

@PunctualEmoticon If you are suggesting to create a local git repository on each phone with the database file, then I think we would run into the same problems as using Syncthing or Dropbox; namely, there is not enough granularity for git to merge things automatically. Only if we split the database into multiple files, would git (or Syncthing/Dropbox) be able to help us, but this doesn't strike me as a very good solution.

[quark6789]

Yes, that's what I was suggesting.
But unlike those file syncing services, git isn't limited to just checking files as a whole. It looks at the contents of the files (if they're ultimately text files), and can resolve changes automatically, unless the same line has been modified on both branches since the branches last split.
Source: http://www.drdobbs.com/tools/three-way-merging-a-look-under-the-hood/240164902

[xdmx]

You may consider the ability to sync it with a nextcloud/owncloud server, that'd be nice

[fwuensche]

I believe you don't want to have any server structure on your side. So, as you suggested above, maybe we could try to use Dropbox.

Take a look at 1password for example. It works like a charm across my devices (mobile and laptop) and it's always in sync by using Dropbox as "database". Do you believe that might be feasible?

[LeonardoGentile]

Another nice thing about 1password is the ability to sync over wifi.
I don't use dropbox to sync, but the app has a self-hosted sync server embedded in the desktop app. When connected to the same wifi I can sync from desktop to mobile and vice-versa

[kyriakosbrastianos]

What about webdav? I would like to share and use data with apps like nextcloud tasks & calendar and qownnotes: https://github.com/pbek/QOwnNotes
QOwnNotes is great on desktop and syncs great with nextcloud tasks so u get web interface too and loop habits is amazing on android. A collaboration will be great with these services/apps.

[omarbenhamid]

What about using cloudant, it is à cloud service based on opensource couchdb technology ?
We could use https://github.com/cloudant/sync-android library. It should transparently fit the 3 requirements.

[MaluNoPeleke]

An optional Play Games Services support would be nice, even if it doesn't meet all requirements.

[recheej]

What about a web based solution? Send data back and forth between the web server. The web server stores the information in a database.

[mr-flibble]

Hello, just an idea:
Since now application uses sqlite, would it not be too time-consuming to allow a user to attach an application to their own mysql postgresql server? The application would be in the online only mode but would be available from multiple devices.

[jharder01]

I'm in favor of self hosted options as they are in-line with the spirit of this project. Granted, a shared, public sync server may help instead awareness of the app.

Additionally, I would love to be able to sync my Android Loop data locally to a personal Grafana/InfluxDB server for data logging and visualization.

[namdnguyen]

Where is the application sqlite database, not backup files, stored on Android? I'm having trouble finding it.

[TheHolyLoli]

nextcloud/owncloud/webdav support is also good.

[raffh]

Indeed, being able to auto-sync with Nextcloud would be amazing.

[oakbrad]

CalDAV / WebDAV sync seems like the most obvious and widely supported solution. Both calendar entries and tasks are obviously natively supported by iOS/Android, the syncing is already built into your phone and software, and the server backend is already easily and common self hosted if you don't trust cloud.

Even just having Loop output a read-only ICS would be useful, but CalDAV sync as the backend would really be amazing as it would make Loop's information available in everyday productivity software.

[ramzan]

What about Etebase? It's the backend used in Etesync.

[SkypLabs]

I was about to suggest this option.

I think Etebase ticks most of the boxes:

• Privacy-friendly (end-to-end encryption).
• Possibility to deploy your own server or to use someone else's without compromising your privacy thanks to E2E.
• Cross-device synchronisation with conflict handling and versioning.
• Offline support.
• Etebase back-ends can be mutualised to store and synchronise the data of different projects. No need to set up a server specifically for uhabits.
• Project widely audited.

[iSoron]

@ramzan Thank you for the link. That actually looks quite promising. We should look into it.

[iSoron]

A partial implementation of this feature has been merged into the dev branch and will probably be available in Loop 2.0, unless we find serious bugs or limitations in our implementation. (UPDATE: This feature still requires a bit of work, so we are delaying it to Loop 2.1)

If you would like to try out an early work-in-progress version of this feature and give us some feedback, it is also available in v2.0.0-alpha. Please note that alpha releases may contain serious bugs that can cause you to lose your entire data. You should create a full database backup before trying it out.

[snpefk]

I tried to open link from sync option (e.g. https://loophabits.org/sync/<some hash>) and got 404 😢

[iSoron]

@snpefk Are you trying to open this link on a phone that has Loop 2.0.0-alpha installed? You should have been offered the option to "Open with Habits". Feel free to open a bug report.

[snpefk]

Oh, so it's an app link? I thought that was a just link for download a sync archive. I wanted to use it in my scripts that load data to my Grafana backend.

[iSoron]

@snpefk Yes, it's an app link; but even if you were able to download the database from the sync server, it would be unreadable (it's an encrypted file, with the decryption key stored only in your phone).

[snpefk]

but even if you were able to download the database from the sync server, it would be unreadable

Can we mitigate this? For example with chechbox (and warning) for disabling encryption. I want to sync my data with my PC, move csv file from time to time seems clumsy.

[lkraav]

A partial implementation of this feature has been merged into the dev branch and will probably be available in Loop 2.0, unless we find serious bugs or limitations in our implementation.

Tyvm 🙏 Is there a short summary of how it technically works, somewhere, or gotta dig through code?

[iSoron]

Not at this time, since it's very experimental and subject to change, but I am planning to document it when the protocol gets more stable. For now, please see the source code.

[olegivo]

I think should be a core functional with high level sync logic and "plugin" modules for sync with concrete data sources such as WebDav or Dropbox or custom backend and so on.

[spiraldancing]

Quick thoughts on this feature idea. Anki / Ankiweb might offer a good model to base the design on. Also, uHabit already allows for data backup/restore ... a simple(r) first-draft solution would be expanding that tool to handle a relatively seamless behind-the-scenes automatic backup-restore process through a Cloud storage location of the user's choice, be it Dropbox or a self-hosted Nextcloud or whatever. If it isn't already, data backup file should/oughta be encrypted for this.

... my dos centavos.

[VA-MMXXII]

There appear to be a number of great synching options out there. I've used ​the open source Tasks.org app on Android (https://play.google.com/store/apps/details?id=org.tasks&hl=en_US&gl=US) and it offers a number of ways to synch. Maybe one or more of them would allow Loop users to use their phones or their tablets (or maybe even their desktop) to check off habits throughout the day?

[Improver2000]

Same here! Tasks sync works perfect! Implementation would be wonderful :-)

[fel1]

I've noticed that many of the Habits that I've added are more like to-do's/chores, which are repeatable. I like the app functionality and simplicity so much, that now many of my daily basic to-do's also live here. I.e. "Take out the trash", "Feed the dog" etc.

It would be more than amazing to be able to share some habits (tasks) with my wife so either of us could mark them as done on any of our devices. This would need some sort of indicator of who (which user) ticked off the habit. If some sort of "Users" function could be added then a file-based sync solution might work well, depending on the implementation. What do you think?

[recheej]

Hey @iSoron just checking the status of this one. Are you still looking for help with it?

[woodsea9]

I would love this to be done so my habits could be shared

[pflajszer]

+1 from me. I'd be very keen on either an exposed API, webhooks to subscribe to, or even some automated db exports.

[SimonDedman]

Hi, idk how useful this is but I recently noticed Joplin uses 3rd party services for backend sync Inc Dropbox, nextcloud, WebDAV, and many others. I wonder if you could copy their approach/code, they're open source?

[HelloLudger]

https://remotestorage.io/ could be used for this feature .

[AnsahMohammad]

Hey any update on this ?,

About crossplatform, i was thinking how about writing Flutter code which can be used to build for other platforms (ios, linux, windows, web etc..), and it'll be a lot easier. Native for android can be maintained along with the flutter for rest.

About the database i think "supabase" is a good open source alternative