Push sync for calendars, tasks, and contacts (CalDAV and CardDAV)

[aniqueta]

Is your feature request related to a problem? Please describe.
Nextcloud is clearly maturing fast. Thanks to everyone for the hard work. To bring it to parity with other first class cloud services, it would be awesome if Nextcloud could provide push notifications for CalDAV and CardDAV. Changes to calendars and tasks would benefit from faster propagation without needing to resort to rapid fetch intervals, which are resource intensive. It's frustrating when changes across devices are not in sync and create version control issues when editing on multiple devices.

It appears to be possible to enable push for CalDAV and CardDAV on Apple iOS, iPadOS, and macOS devices. Not sure about other devices.

Describe the solution you'd like
Implement something like this from FastMail. Unclear what the technical backend is. Perhaps they would be willing to share. For iOS, iPadOS, and macOS, it appears this involved collaboration with Apple, although not sure.
https://fastmail.blog/2016/10/10/push-sync-for-contacts-calendars-now-available-on-ios/

Describe alternatives you've considered
Faster fetch intervals are imperfect, drain resources (especially for battery powered devices. Z-push is an option but clunky and also resource intensive. It is inefficient, since it just converts CalDAV and CardDAV to Microsoft ActiveSync. This also creates reliance on a new protocol.

Thanks again, and completely understand that this is "nice to have" not "need to have." But, given how Nextcloud is maturing, it would be awesome to bring it to parity with cloud services others offer.

[tcitworld]

Not sure about what Apple's push for *DAV refers to but the only close thing to work with that we've got is this 3-year old first version of an IETF draft : https://tools.ietf.org/html/draft-gajda-dav-push-00.

Maybe it refers to this Apple's CalendarServer extension for Discovery of CalDAV Push-Notification Settings, but since Apple abandoned CalendarServer, not sure this is a safe bet.

Asking Fastmail for their technical details would indeed be a first step, but I'm thinking since they are working on JMAP Calendars, not sure they did go this way.

Anyway, I've not heard of any other clients supporting this, so it would be quite useless for non-Apple users for a while.

[jakobroehrl]

I don't like pull sync as it is now neither.
For android you need Davx5 to sync CalDAV and CardDAV. And there a Live/Push sync is not possible:
https://forums.bitfire.at/topic/2201/live-sync-possible/2?_=1598341471130

[rfc2822]

In theory, that could be done over OpenPush (see https://gitlab.com/bitfireAT/davx5-ose/-/wikis/Roadmap#other-ideas). But then

• the OpenPush app would have to be ready,
• Nextcloud would need to support this protocol,
• DAVx5 would need to support this protocol.

So I'd not expect this for the near future...

[szaimen]

@nextcloud/server-triage is this feasible?

[tcitworld]

The development of OpenPush seems to have stalled, and none of the other things mentioned did change, including no client changes.

Note that there's now a JMAP app for Nextcloud https://github.com/audriga/nextcloud-jmap.

[jvican]

I don't think Fastmail is using JMAP Calendars to send notifications to iOS because the Apple integration uses CalDAV and I doubt Fastmail's CalDAV server speaks JMAP with Apple servers.

I'm going to reach out to Fastmail to learn more about how they are actually doing it. Will report back when I know more about it, in case this information is still of interest to you.

[aniqueta]

@jvican Thanks for thinking of that. Sorry I neglected to reply here, but I worked through Fastmail support to reach a dev. Here's what they shared.

We had some personal contacts at Apple from our standards work. They were able to organise a meeting for us, and from there we were given the necessary certificates and protocol details to implement IMAP push. Later, when implemented CalDAV and CardDAV, we asked and were given access to push for those too. Unfortunately I can't give you any names, though I can say that most of them have since moved on from Apple.

The actual protocol we implement is documented at https://github.com/apple/ccs-calendarserver/blob/master/doc/Extensions/caldav-pubsubdiscovery.txt. I understand that it was at one time possible to extract the necessary certs from Apple's server products, though I've never seen it done. I'm not sure now if there's another way to get hold of them except directly from Apple.

There are still (unsupported) ways to get the necessary push notification certificates from Apple. For example:
scintill/macos-server-apns-certs
freswa/dovecot-xaps-daemon

@tcitworld Is it possible to reopen this issue and explore some of the above? I think there are no other viable paths forward for push notification for Apple platforms otherwise?

[tcitworld]

That's a great piece of information !

We shouldn't need mangling with Apple's certificate, and possibly could just reuse the push proxy used by the Nextcloud notification app (which, if we use it, will need adjustments to send custom payloads nextcloud/notifications#1055, otherwise put this directly in dav).

The form of the registration is flexible enough to have transports of other types, for instance DAVdroid could support type="FCM" (Google Firebase) or type="OpenPush" or type="UnifiedPush" (see https://unifiedpush.org). @rfc2822

Even better, we could do WebPush in the browser to tell the calendar to refresh itself (instead of nextcloud/calendar#31).

<push-transports xmlns='http://calendarserver.org/ns/'>
  <transport type='APSD'>
    <subscription-url>
      <href xmlns='DAV:'>https://server.example.com:8443/apns</href>
    </subscription-url>
    <apsbundleid>com.apple.calendar.XServer.934668ca-125e-4246-afee-8cf2df37aab8</apsbundleid>
    <env>PRODUCTION</env>
    <refresh-interval>172800</refresh-interval>
  </transport>
</push-transports>

There's more investigation needed, and the lack of documentation will be a pain in the ass, but this is exciting.

[rfc2822]

@tcitworld We're very interested in getting push (over both Google and open protocols) to CalDAV/CardDAV/DAVx5. I will have a a look at this.

[ghost]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

[karmanyaahm]

The form of the registration is flexible enough to have transports of other types, for instance DAVdroid could support type="FCM" (Google Firebase) or type="OpenPush" or type="UnifiedPush" (see https://unifiedpush.org). @rfc2822

Even better, we could do WebPush in the browser to tell the calendar to refresh itself (instead of nextcloud/calendar#31).

WebPush and UnifiedPush are (practically) API compatible, so supporting WebPush (using one of the many existing libraries) for browsers means automatically supporting UnifiedPush for Android clients on the server-side

[aniqueta]

@tcitworld I was excited to see this get tagged as a feature to develop about a year ago. Is this still in something we can expect? Or better to look elsewhere for CalDAV (and CardDAV) push? Thanks.

[tcitworld]

Well, a long as someone's not paying Nextcloud GmbH for development, this depends on volunteer work (possibly mine). So it could come anytime between soon or never. There's still interest for this from a user and developer point of vue anyway.

Or better to look elsewhere for CalDAV (and CardDAV) push?

I don't know of any other FLOSS project working on this, but feel free to check. Fastmail could well also be the only service in the world doing push over CalDAV, as the rest of the ecosystem may have moved to private APIs and/or JMAP. Maybe some of the people at CalConnect know.

[shane64]

Just an orphaned former Mac OS Server user without the skills or resources to engage in this work hoping to find a way to continue to self-host calendar data (with push notifications to my iOS devices) and hoping that connecting someone here on the NextCloud CalDAV side with someone familiar with Apple Push Notifications side might see some magic happen.

freswa/dovecot-xaps-daemon#20 (comment)

Thanks.

[devvv4ever]

We planning to start working on this pretty soon! For this we've teamed up with @verdigado who are interested in bringing this to life on the Nextcloud side, and we will implement it in DAVx5 for Android. We've also had a basic talk about this with @ChristophWurst and @st3iny at Nextcloud conf'22 and it seems that there is general interest in extending Nextclouds PUSH capabilities :)

[aniqueta]

Great to hear @devvv4ever. A clarifying question from your message in November: Is the focus of the near-term work just Android (DAVx5)? Thanks.

[devvv4ever]

The current project plan is:

• Develop a generalized WebDAV Push standard that also covers CardDAV and CalDAV
• Implement this on the Nextcloud server side (as real-world sample implementation on the server side)
• Implement this in DAVx5 using UnifiedPush (as a real-world sample implementation on the client side)

Working on the standard will probably start in March (this is the hardest of all these parts since it has to be developed very well and many parties are involved)

[aniqueta]

Thanks. So iOS/iPadOS and macOS are not in the first phase of this plan?

[devvv4ever]

Thats something we can't directly influence. We'll try however to activate connections to the Apple people (via CalConnect) to get them in on this during creating/forming the standard (which is the requirement for all other steps). They need to implement the standard on their iClouds CalDAV/CardDAV connectors so that this can be of use then. If that happens you should also be able to receive instant push notifications on Android via DAVx5 when a collection on the server (for example a calendar on the icloud server) has changed.

[aniqueta]

I see. So the approach described above for Apple devices won't be pursued or that's part of this approach? #22379 (comment)

Would Z-Push be the best path forward for push on Apple devices then?

[shane64]

What about combining the approach for push notifications originally taken in Apple's original open source CalDAV server (http://www.calendarserver.org/) and with the approach taken to patch the dovecot code to use Apple's new HTTP/2 push notification API (https://github.com/freswa/dovecot-xaps-daemon)? Automatic APNS certificate generation and renewal (that used to be taken care of by macOS Server) is included in the second project (which will give you native push notifications with iOS's calendar app (and the macOS one). If it can be done for dovecot, why not nextcloud's CalDAV implementation?

[WhyNotHugo]

There's been some discussion on creating an extension to caldav for push mechanisms on a separate issue tracker.

This separate discussion originally started out between myself (vdirsyncer maintainer) and some of the DAVx5 devs, since we both want push updates on the client side (although on different platforms).

IMHO, it seems that using caldav-pubsubdiscovery.txt as a base with Web Push is the best fit. Replacing APN-specific properties with WebPush-specific ones should suffice for this use case, while using APN (apple's notification protocol) as a non-normative/de-facto variation will result in things working properly on Apple's ecosystem.

If someone is willing to work on a server-side of this, I'm pretty confident that I can address a client side implementation (e.g.: for vdirsyncer) in the coming months.

[tcitworld]

I have started to look at details of the implementation in Nextcloud.
The aim is have at least WebPush (all of RFC 8030, RFC 8291 and RFC 8292) for the NC Web Calendar app and another push provider (probably UnifiedPush as it's closest) working.

Implementation details are:

• For the WebPush part
  • Push Notification interfaces (especially IProvider and IManager) in OCP, and their implementations
  • a bundled WebPush provider in a dedicated app that registers to the manager
  • possibility for other providers (apns - Apple¹, FCM - Google¹, UnifiedPush) to register as providers through dedicated apps as well
  • (possibly, at some point and if feasible) refactor the notification app to use the Nextcloud Push Proxy as just another push provider²
• For the DAV part
  • Generic support to expose <push-transports> and <pushkey> through a Sabre Plugin, loading registered providers
  • the bundled WebPush provider also registers it's <transport> (with it's own custom XML serializer to represent possible extra fields)
  • other providers register their own <transport> (with their own custom XML serializer to represent possible extra fields)
• For the Calendar part
  • Fetch new properties in calendar-js
  • Subscribe in browser to WebPush
  • Reload calendar data when receiving a push (fixes Auto-refresh  calendar#31)

On collection changes, we add a new listener for already existing events that lists registered providers for the user and sends notifications to the subscription URLs.

Other than that, we also need to explore how handle expired subscriptions (make sure every provider have a receipts concept and return appropriate error messages if subscription is expired).

¹ Obviously, you'll still need API keys (and eventually payment) for those.
² Meaning possibly bypassing the NC Push Proxy for regular if you have another provider… and client apps are made compatible

[provokateurin]

There is already work on the webpush part: nextcloud/notifications#1225 cc @karmanyaahm @Zocker1999NET