calendar-contacts-server: Submission of Apple Calendar and Contacts Server configuration

[essandess]

calendar-contacts-server: Submission of Apple Calendar and Contacts Server configuration

• Port of Apple's ccs-calendarserver
• Configuration independent of Server.app's Calendar and Contacts Server
• Standards-compliant server implementing the CalDAV and CardDAV protocols
• iMIP capability via port:mail-server

Description

Type(s)

• submission
• bugfix
• enhancement
• security fix

Tested on

macOS 10.14.6 18G87
Xcode 10.3 10G8

Verification

Have you

• followed our Commit Message Guidelines?
• squashed and minimized your commits?
• checked that there aren't other open pull requests for the same change?
• referenced existing tickets on Trac with full URL?

• checked your Portfile with port lint?
• tried existing tests with sudo port test?
• tried a full install with sudo port -vst install?
• tested basic functionality of all binary files?

[macportsbot]

Travis Build #7640 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[macportsbot]

Travis Build #7644 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[macportsbot]

Travis Build #7646 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[mf2k]

So this is fetching from the head of the git repository? I'm not a fan of that because we won't have reproducible builds. Is the latest release 9.0 not good enough to use? Can a GitHub tag be used at least?

[essandess]

@mf2k Yes, I did that on purpose because:

• Apple fixed several issues between the released 9.0 and the current 9.3 at HEAD.
• Apple has given the appearance of doing very, very little work on the deprecated parts of Server.app, Calendar server included. I wouldn’t hold my breath waiting for an “official” release of 9.3.
• This raises questions about how ccs-calendarserver’s reliance on EOL’d Python 2.7 will play out, but that’s an issue for another day because I haven’t seen how macOS itself will deal with this.

Given all that and the desire for reproducible builds, would you please post the necessary Portfile code to tie the build to a specific github commit hash? I don’t know how to do that.

[essandess]

@mf2k I need some help with this buildbot issue please.

The problem is that the equivalent of sudo easy_install pip isn’t working on the buildbots. It works fine on a local box as written in the Portfile.

DEBUG: system: /usr/bin/easy_install pip
error: can't create or remove files in install directory
 
The following error occurred while trying to add or remove files in the
installation directory:
 
    [Errno 1] Operation not permitted: '/Library/Python/2.7/site-packages/test-easy-install-46129.pth'
 
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
 
    /Library/Python/2.7/site-packages/

See https://paste.z0k.xyz/c8f9a2b4a84f

Background:

• I tried to get a working build from MacPorts tools, but was unsuccessful.
• The ccs-calendarserver code has a lot of details that are coupled to macOS system stuff.
• When I set up everything to use macOS system calls, including Python and easy_install, everything works consistently.
• Don’t even ask about destroot.

That’s the reason for using native macOS easy_install to install pip, which then installs and builds all the necessary packages correctly.

But the buildbot fails when I try to install pip from easy_install.

[essandess]

P.S. on my local box pip gets installed to /usr/local/bin. Perhaps this must be set explicitly in the easy_install command?

[mf2k]

@mf2k Yes, I did that on purpose because:

• Apple fixed several issues between the released 9.0 and the current 9.3 at HEAD.
• Apple has given the appearance of doing very, very little work on the deprecated parts of Server.app, Calendar server included. I wouldn’t hold my breath waiting for an “official” release of 9.3.

Given all that and the desire for reproducible builds, would you please post the necessary Portfile code to tie the build to a specific github commit hash? I don’t know how to do that.

Just specify a tag instead of a version number in the github.setup. Then add a version line with the date of the commit that is being tagged, in YYYYMMDD format, to make it more meaningful and so future updates increase monotonically.

See section 5.9.2.4. here:
https://guide.macports.org/chunked/reference.portgroup.html#reference.portgroup.github

[mf2k]

Hopefully someone more knowledgeable than me can help you with the python builder issue.

[essandess]

Implemented @mf2k's version suggestion. Trying putting the easy_install into post-activate…

[macportsbot]

Travis Build #7655 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[macportsbot]

Travis Build #7656 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

Port calendar-contacts-server's dependencies fail on xcode7.3. Log

The build timed out.

[mojca]

I really admire the amount of effort that went into creating this support, but there are many somewhat dirty hacks with practices against the general philosophy of macports, like installing stuff to system directories and never cleaning it up afterwards, requiring a particular port select for software that has seen two newer releases since, bundling some python with virtual env straight in the package, a gazillion of shell commands. Since we support very old systems, it would also be helpful to know the minimum OS version required to make this work. And someone would have to check that portindex doesn't choke on this Portfile when used under Linux.

With 1300 lines of code it's also probably something that only you would probably be capable of modifying and maintaining it in the future. It would be helpful to have at least one other developer fully understand what each of the steps is doing and whether it can be made more macports-friendly.

I'm not sure if I can give competent advice about how to go to improve this though, neither have the time to do extensive testing before or after, so I hope that someone else steps in.

[essandess]

@mojca Thanks for taking the time to review this. You found the dirty hacks!

Unfortunately, I found that these are all necessary to get Apple’s Calendar and Contacts Server up and running. This is the very same code Apple used to package up Calendar and Contacts Server for Server.app, so they didn’t care so much about how to deal with package managers, and it shows. See apple/ccs-calendarserver#540

After Server.app deprecated its Calender server, I honestly looked at migrating to sabre/dav, inverse-inc/sogo, or Kozea/Radicale for a calDAV+cardDAV server. But AFAICT only apple/ccs-calendarserver and sabre/dav support iMIP, and I've been using apple/ccs-calendarserver's iMIP for years and know it works. And sabre/dav is PHP-based.

So apple/ccs-calendarserver looks to be the main open source option of choice. This project is the only one I'm aware of that has iMIP, DKIM integration, and a decent code base (not PHP, thank you…) to make a MacPorts port (and later python3 migration) worth the effort.

As for hacking MacPorts to install it, none of these hacks are security related, and I did my best to isolate them, constrained to writing a Portfile that actually builds a running ccs-calendarserver. I have this deployed with my own mail server and it’s solid just like the old Server.app version.

Server.app has been around a long time, so I’d expect this will work across many macOS versions, but I’ve only deployed it on Mojave. Also, people have ported ccs-calendarserver to Linux and dockerized containers, but I don’t know how this port would play in Linux. I isolated everything to ${prefix}, except for personal user data, which by security/safety design is placed outside in /var/calendarserver/Library/CalendarServer/Data. AFAIK this should all work across multiple platforms.

Now that Server.app is dead, if you want to run a Calendar server on macOS, ccs-calendarserver is really the only open source option, and aside from just running the install code yourself, MacPorts is really the only package manager option—Brew’s security model is incompatible with running a secure mail server (or a secure much of anything).

I did my best to get ccs-calendarserver ported into MacPorts with a minimum of compromises. It works, and it’s secure to the best of my knowledge. There are already a “Known issues” section in the port notes in case others adopt this and can identify solutions to the issues you’ve raised.

[macportsbot]

Travis Build #7668 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[macportsbot]

Travis Build #7669 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[macportsbot]

Travis Build #7670 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

The build timed out.

[essandess]

@mojca I was able to eliminate most of the dirty hacks, and I got this working with MacPorts Python too.

It should be is ready to merge.

[mf2k]

@mojca @ryandesign @essandess : What needs to happen to get this committed?

[essandess]

All set to merge on my end.

[essandess]

I got APNS working on Calendar and Contacts Server, and added the instructions to notes.

I believe this is ready to commit if there's no other feedback.

[macportsbot]

Travis Build #9135 Errored.

Lint results

--->  Verifying Portfile for calendar-contacts-server
--->  0 errors and 0 warnings found.

Port calendar-contacts-server's dependencies fail on xcode7.3. Log

The build timed out.

[essandess]

@mojca @ryandesign @essandess : What needs to happen to get this committed?

@mf2k I have no more changes to add and believe that this is ready to merge.

[mf2k]

I am waiting for approval from @ryandesign and @mojca.

[ryandesign]

Oh you can go ahead without me. I still find the initialize variant weird (why would I want to recompile the entire project from source just to initialize my settings, and then reinstall it again without the variant so that the settings don't get initialized again the next time I upgrade?) but I'm not the user, so let's see what actual users say. And if it's not intended for users but is intended so that developers can have a clean slate every time they upgrade the port, then I can see how that would be useful, I would just be mindful that users don't want to see a lot of developer-centric cruft.

[essandess]

@mf2k @mojca May we please merge?

I and others rely every hour of every day on the awesomeness of this port and can testify to its robustness.