inotify-wait unrecognized option

[g105b]

Describe the bug
I have just cloned the git repository and run the install shell script. I have made no modifications to the source. I am running on Ubuntu 20.10.

To Reproduce
Steps to reproduce the behavior:

1. Setup osync with the attached config file (bottom of issue)
2. Run osync with following parameters: systemctl restart [email protected]
3. The logs show a successful sync, which I can confirm on the remote server, but the following error is left in the logs: Thu Apr 22 16:33:48 BST 2021 - (ERROR):#### inotify error detected, waiting 60 seconds before running next sync.

Expected behavior
inotify should trigger a sync when a file changes.

** Deviated behavior**
Syncing works, but only after 1 minute of polling. There is an issue with inotifywait's arguments.

Logs

TIME: 0 - Script begin, logging to [/var/log/osync.sync_Documents.log].
TIME: 0 - Local OS: [Linux 5.11.0-7612-generic x86_64 x86_64 GNU/Linux].
TIME: 0 - -------------------------------------------------------------
TIME: 0 - Thu Apr 22 16:40:10 BST 2021 - osync 1.2 script begin.
TIME: 0 - -------------------------------------------------------------
TIME: 0 - Sync task [sync_Documents] launched as root@hyperion (PID 285064)
TIME: 2 - Remote OS: [main: line 26: GetConfFileValue: command not found
Linux 5.4.0-51-generic x86_64 x86_64 GNU/Linux ( )
main: line 27: GetConfFileValue: command not found].
TIME: 2 - Checking minimum disk space in local replica [/home/g105b/Documents/].
TIME: 0 - Checking minimum disk space in remote replica [/home/g105b/Documents/].
TIME: 4 - Locked local initiator replica in [/home/g105b/Documents/.osync_workdir/lock].
TIME: 0 - Locked local target replica in [/home/g105b/Documents/.osync_workdir/lock].
TIME: 6 - Starting synchronization task.
TIME: 6 - Creating initiator replica file list [/home/g105b/Documents/].
TIME: 6 - Creating target replica file list [/home/g105b/Documents/].
TIME: 6 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -rltD -8 -zz --skip-compress=gz/xz/lz/lzma/lzo/rz/jpg/mp3/mp4/7z/bz2/rar/zip/sfark/s7z/ace/apk/arc/cab/dmg/jar/kgb/lzh/lha/lzx/pak/sfx --no-whole-file  -p -o -g --executability  --exclude ".osync_workdir"  --exclude-from="/etc/osync/exclude.list"  --list-only "/home/g105b/Documents/" 2> "/tmp/osync.treeList.initiator.error.285064.20210422T164010.722274414" | (grep -E "^-|^d|^l" || :) | (awk '{$1=$2=$3=$4="" ;print substr($0,5)}' || :) | (awk 'BEGIN { FS=" -> " } ; { print $1 }' || :) | (grep -v "^\.$" || :) | sort > "/tmp/osync.treeList.initiator.285064.20210422T164010.722274414"
TIME: 6 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -rltD -8 -zz --skip-compress=gz/xz/lz/lzma/lzo/rz/jpg/mp3/mp4/7z/bz2/rar/zip/sfark/s7z/ace/apk/arc/cab/dmg/jar/kgb/lzh/lha/lzx/pak/sfx --no-whole-file  -p -o -g --executability  --exclude ".osync_workdir"  --exclude-from="/etc/osync/exclude.list"  -e "/usr/bin/ssh -C -i /home/g105b/.ssh/id_rsa  -p 22" --list-only [email protected]:"/home/g105b/Documents/" 2> "/tmp/osync.treeList.target.error.285064.20210422T164010.722274414" | (grep -E "^-|^d|^l" || :) | (awk '{$1=$2=$3=$4="" ;print substr($0,5)}' || :) | (awk 'BEGIN { FS=" -> " } ; { print $1 }' || :) | (grep -v "^\.$" || :) | sort > "/tmp/osync.treeList.target.285064.20210422T164010.722274414"
TIME: 8 - Creating initiator replica deleted file list.
TIME: 8 - Creating target replica deleted file list.
TIME: 8 - Launching command [comm -23 "/home/g105b/Documents/.osync_workdir/state/initiator-tree-after-sync_Documents" "/home/g105b/Documents/.osync_workdir/state/initiator-tree-current-sync_Documents" > "/home/g105b/Documents/.osync_workdir/state/initiator-deleted-list-sync_Documents"].
TIME: 8 - Launching command [comm -23 "/home/g105b/Documents/.osync_workdir/state/target-tree-after-sync_Documents" "/home/g105b/Documents/.osync_workdir/state/target-tree-current-sync_Documents" > "/home/g105b/Documents/.osync_workdir/state/target-deleted-list-sync_Documents"].
TIME: 8 - Updating target replica.
TIME: 8 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -rltD -8 -zz --skip-compress=gz/xz/lz/lzma/lzo/rz/jpg/mp3/mp4/7z/bz2/rar/zip/sfark/s7z/ace/apk/arc/cab/dmg/jar/kgb/lzh/lha/lzx/pak/sfx --no-whole-file   -p -o -g --executability  -u -i -e "/usr/bin/ssh -C -i /home/g105b/.ssh/id_rsa  -p 22" --backup --backup-dir=".osync_workdir/backup" --exclude ".osync_workdir"  --exclude-from="/etc/osync/exclude.list"  --exclude-from="/home/g105b/Documents/.osync_workdir/state/initiator-deleted-list-sync_Documents" --exclude-from="/home/g105b/Documents/.osync_workdir/state/target-deleted-list-sync_Documents" "/home/g105b/Documents/" [email protected]:"/home/g105b/Documents/" >> /tmp/osync.update.target.285064.20210422T164010.722274414 2>&1
TIME: 10 - :List:

TIME: 10 - Updating target replica succeded.
TIME: 10 - Updating initiator replica.
TIME: 10 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -rltD -8 -zz --skip-compress=gz/xz/lz/lzma/lzo/rz/jpg/mp3/mp4/7z/bz2/rar/zip/sfark/s7z/ace/apk/arc/cab/dmg/jar/kgb/lzh/lha/lzx/pak/sfx --no-whole-file   -p -o -g --executability  -u -i -e "/usr/bin/ssh -C -i /home/g105b/.ssh/id_rsa  -p 22" --backup --backup-dir=".osync_workdir/backup" --exclude ".osync_workdir"  --exclude-from="/etc/osync/exclude.list"  --exclude-from="/home/g105b/Documents/.osync_workdir/state/initiator-deleted-list-sync_Documents" --exclude-from="/home/g105b/Documents/.osync_workdir/state/target-deleted-list-sync_Documents" [email protected]:"/home/g105b/Documents/" "/home/g105b/Documents/" >> /tmp/osync.update.initiator.285064.20210422T164010.722274414 2>&1
TIME: 12 - :List:

TIME: 12 - Updating initiator replica succeded.
TIME: 12 - Propagating deletions to initiator replica.
TIME: 12 - Propagating deletions to target replica.
TIME: 12 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -e "/usr/bin/ssh -C -i /home/g105b/.ssh/id_rsa  -p 22" "/home/g105b/Documents/.osync_workdir/state/initiator-deleted-list-sync_Documents" [email protected]:"/home/g105b/Documents/.osync_workdir/state/" >> /tmp/osync._deleteRemote.precopy.285064.20210422T164010.722274414 2>&1
TIME: 14 - RSYNC_CMD: /usr/bin/rsync -r --rsync-path="__(o_O)__ rsync" -e "/usr/bin/ssh -C -i /home/g105b/.ssh/id_rsa  -p 22" --include ".osync_workdir" --include ".osync_workdir/state" --include ".osync_workdir/state/target-failed-delete-sync_Documents" --include ".osync_workdir/state/target-success-delete-sync_Documents" --exclude='*' [email protected]:"/home/g105b/Documents/" "/home/g105b/Documents/" > "/tmp/osync.remote_failed_deletion_list_copy.285064.20210422T164010.722274414"
TIME: 16 - Creating initiator replica file list [/home/g105b/Documents/].
TIME: 16 - Creating target replica file list [/home/g105b/Documents/].
TIME: 16 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -rltD -8 -zz --skip-compress=gz/xz/lz/lzma/lzo/rz/jpg/mp3/mp4/7z/bz2/rar/zip/sfark/s7z/ace/apk/arc/cab/dmg/jar/kgb/lzh/lha/lzx/pak/sfx --no-whole-file  -p -o -g --executability  --exclude ".osync_workdir"  --exclude-from="/etc/osync/exclude.list"  --list-only "/home/g105b/Documents/" 2> "/tmp/osync.treeList.initiator.error.285064.20210422T164010.722274414" | (grep -E "^-|^d|^l" || :) | (awk '{$1=$2=$3=$4="" ;print substr($0,5)}' || :) | (awk 'BEGIN { FS=" -> " } ; { print $1 }' || :) | (grep -v "^\.$" || :) | sort > "/tmp/osync.treeList.initiator.285064.20210422T164010.722274414"
TIME: 16 - RSYNC_CMD: /usr/bin/rsync --rsync-path="__(o_O)__ rsync" -rltD -8 -zz --skip-compress=gz/xz/lz/lzma/lzo/rz/jpg/mp3/mp4/7z/bz2/rar/zip/sfark/s7z/ace/apk/arc/cab/dmg/jar/kgb/lzh/lha/lzx/pak/sfx --no-whole-file  -p -o -g --executability  --exclude ".osync_workdir"  --exclude-from="/etc/osync/exclude.list"  -e "/usr/bin/ssh -C -i /home/g105b/.ssh/id_rsa  -p 22" --list-only [email protected]:"/home/g105b/Documents/" 2> "/tmp/osync.treeList.target.error.285064.20210422T164010.722274414" | (grep -E "^-|^d|^l" || :) | (awk '{$1=$2=$3=$4="" ;print substr($0,5)}' || :) | (awk 'BEGIN { FS=" -> " } ; { print $1 }' || :) | (grep -v "^\.$" || :) | sort > "/tmp/osync.treeList.target.285064.20210422T164010.722274414"
TIME: 18 - Finished synchronization task.
TIME: 18 - Running conflict backup cleanup.
TIME: 18 - :The initiator replica dir [/home/g105b/Documents/.osync_workdir/backup] does not exist. Skipping cleaning of old files.
TIME: 18 - Removing files older than 30 days on target replica for conflict backup deletion.
TIME: 19 - Cleanup complete on target replica.
TIME: 19 - :Command output:
The target replica dir [/home/g105b/Documents/.osync_workdir/backup] does not exist. Skipping cleaning of old files
TIME: 19 - Running soft deletion cleanup.
TIME: 19 - Removing files older than 30 days on initiator replica for softdelete deletion.
TIME: 19 - :Command output:

TIME: 19 - File cleanup complete on initiator replica.
TIME: 19 - :Command output:

TIME: 19 - Directory cleanup complete on initiator replica.
TIME: 20 - Removing files older than 30 days on target replica for softdelete deletion.
TIME: 21 - Cleanup complete on target replica.
TIME: 21 - :Command output:

TIME: 21 - Removed local initiator replica lock.
TIME: 23 - Removed remote target replica lock.
TIME: 23 - osync finished.
Thu Apr 22 16:40:33 BST 2021 - (ERROR):#### inotify error  detected, waiting 60 seconds before running next sync.

Environment (please complete the following information):

• Full osync version (including build): osync 1.2 2017032101
• OS: [e.g. iOS]

DISTRIB_ID=Pop
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
NAME="Pop!_OS"
VERSION="20.10"
ID=pop
ID_LIKE="ubuntu debian"

• Bitness: x64 or x86
• Shell: bash

Additional context

The version of inotifywait that's shipped with Ubuntu 20.10 is inotifywait 3.14.

I have performed my own debugging on this, and I can see the following command is being run that is exiting with a non-zero status:

# I have extracted these variables from my own debugging inspection.
RSYNC_PATTERNS='--exclude-from="/etc/osync/exclude.list"'
RSYNC_PARTIAL_EXCLUDE=""
OSYNC_DIR=".osync_workdir"
INITIATOR_SYNC_DIR="/home/g105b/Documents/"

# I have formatted this to be more readable, and removed the -qq option.
inotifywait \
        $RSYNC_PATTERNS \
        $RSYNC_PARTIAL_EXCLUDE \
        --exclude "$OSYNC_DIR" \
        -r \
        -e create \
        -e modify \
        -e delete \
        -e move \
        -e attrib \
        --timeout "$MAX_WAIT" \
        "$INITIATOR_SYNC_DIR"

When I run the above command in bash on my computer, I get the following error:

inotifywait: unrecognized option '--exclude-from="/etc/osync/exclude.list"'

I will investigate why this would be causing an issue and then attempt to create a fix.

Log file: /etc/osync/Documents.conf

#!/usr/bin/env bash

###### osync - Rsync based two way sync engine with fault tolerance
###### (C) 2013-2017 by Orsiris de Jong (www.netpower.fr)
###### osync v1.1x / v1.2x config file rev 2017020801

## ---------- GENERAL OPTIONS

## Sync job identification
INSTANCE_ID="sync_Documents"

## Directories to synchronize.
## Initiator is the system osync runs on. The initiator directory must be a local path.
INITIATOR_SYNC_DIR="/home/g105b/Documents"

## Target is the system osync synchronizes to (can be the same system as the initiator in case of local sync tasks). The target directory can be a local or remote path.
TARGET_SYNC_DIR="ssh://[email protected]:22//home/g105b/Documents"
#TARGET_SYNC_DIR="ssh://[email protected]:22//home/git/osync/dir2"

## If the target system is remote, you can specify a RSA key (please use full path). If not defined, the default ~/.ssh/id_rsa will be used. See documentation for further information.
SSH_RSA_PRIVATE_KEY="/home/g105b/.ssh/id_rsa"

## Alternatively, you may specify an SSH password file (less secure). Needs sshpass utility installed.
#SSH_PASSWORD_FILE=""

## When using ssh filter, you must specify a remote token matching the one setup in authorized_keys
#_REMOTE_TOKEN=SomeAlphaNumericToken9

## Create sync directories if they do not exist
CREATE_DIRS=yes

## Log file location. Leaving this empty will create a logfile at /var/log/osync_version_SYNC_ID.log (or current directory if /var/log doesn't exist)
LOGFILE=""

## Generate an alert if initiator or target replicas have less free space than given value in KB. Set this to zero to skip disk space tests. 
MINIMUM_SPACE=10240

## Bandwidth limit Kbytes / second. Leave 0 to disable limitation
BANDWIDTH=0

## If enabled, synchronization on remote system will be processed as superuser. See documentation for /etc/sudoers file configuration.
SUDO_EXEC=no
## Paranoia option. Don't change this unless you read the documentation.
RSYNC_EXECUTABLE=rsync
## Remote rsync executable path. Leave this empty in most cases
RSYNC_REMOTE_PATH=""

## Rsync exclude / include order (the option set here will be set first, eg: include will make include then exclude patterns)
RSYNC_PATTERN_FIRST=include

## List of files / directories to incldue / exclude from sync on both sides (see rsync patterns, wildcards work).
## Paths are relative to sync dirs. List elements are separated by a semicolon.
RSYNC_INCLUDE_PATTERN=""
RSYNC_EXCLUDE_PATTERN=""
#RSYNC_EXCLUDE_PATTERN="tmp;archives"

## Files that contains lists of files / directories to include / exclude from sync on both sides. Leave this empty if you don't want to use an exclusion file.
## This file has to be in the same directory as the config file
## Paths are relative to sync dirs. One element per line.
RSYNC_INCLUDE_FROM=""
#RSYNC_EXCLUDE_FROM=""
RSYNC_EXCLUDE_FROM="exclude.list"

## List elements separator char.  You may set an alternative separator char for your directories lists above.
PATH_SEPARATOR_CHAR=";"

## ---------- REMOTE SYNC OPTIONS

## ssh compression should be used unless your remote connection is good enough (LAN)
SSH_COMPRESSION=yes

## Ignore ssh known hosts. DANGER WILL ROBINSON DANGER ! This can lead to security issues. Only enable this if you know what you're doing.
SSH_IGNORE_KNOWN_HOSTS=no

## Check for connectivity to remote host before launching remote sync task. Be sure the hosts responds to ping. Failing to ping will stop sync.
REMOTE_HOST_PING=no

## Check for internet access by pinging one or more 3rd party hosts before remote sync task. Leave empty if you don't want this check to be be performed. Failing to ping will stop sync.
## If you use this function, you should set more than one 3rd party host, and be sure you can ping them.
## Be aware some DNS like opendns redirect false hostnames. Also, this adds an extra execution time of a bit less than a minute.
REMOTE_3RD_PARTY_HOSTS="www.kernel.org www.google.com"

## ---------- MISC OPTIONS

## Optional arguments passed to rsync executable. The following are already managed by the program and shoul never be passed here
## -r -l -p -t -g -o -D -E - u- i- n --executability -A -X -L -K -H -8 -zz –skip-compress –checksum –bwlimit –partial –partial-dir –no-whole-file –whole-file –backup –backup-dir –suffix
## --exclude --exclude-from --include --include-from --list-only --stats
RSYNC_OPTIONAL_ARGS=""

## Preserve basic linux permissions
PRESERVE_PERMISSIONS=yes
PRESERVE_OWNER=yes
PRESERVE_GROUP=yes
## On MACOS X, does not work and will be ignored
PRESERVE_EXECUTABILITY=yes

## Preserve ACLS. Make sure source and target FS can handle ACL. Disabled on Mac OSX.
PRESERVE_ACL=no
## Preserve Xattr. Make sure source and target FS can manage identical XATTRS. Disabled on Mac OSX.
PRESERVE_XATTR=no
## Transforms symlinks into referent files/dirs. Be careful as symlinks without referrent will break sync as if standard files could not be copied.
COPY_SYMLINKS=no
## Treat symlinked dirs	as dirs. CAUTION: This also follows symlinks outside of the replica root.
KEEP_DIRLINKS=no
## Preserve hard links. Make sure source and target FS can manage hard links or you will lose them.
PRESERVE_HARDLINKS=no
## Do a full checksum on all files that have identical sizes, they are checksummed to see if they actually are identical. This can take a long time.
CHECKSUM=no

## Let RSYNC compress file transfers. Do not use this if both initator and target replicas are on local system. Also,  do not use this if you already enabled SSH compression.
RSYNC_COMPRESS=yes

## Maximum execution time (in seconds) for sync process. Set these values zero will disable max execution times.
## Soft exec time only generates a warning. Hard exec time will generate a warning and stop sync process.
SOFT_MAX_EXEC_TIME=7200
HARD_MAX_EXEC_TIME=10600

## Log a message every KEEP_LOGGING seconds just to know the task is still alive
KEEP_LOGGING=1801

## Minimum time (in seconds) in file monitor /daemon mode between modification detection and sync task in order to let copy operations finish.
MIN_WAIT=60

## Maximum time (in seconds) waiting in file monitor / daemon mode. After this time, sync is run.
## Use 0 to wait indefinitely.
MAX_WAIT=7200

## ---------- BACKUP AND DELETION OPTIONS

## Enabling this option will keep a backup of a file on the target replica if it gets updated from the source replica. Backups will be made to .osync_workdir/backups
CONFLICT_BACKUP=yes
## Keep multiple backup versions of the same file. Warning, This can be very space consuming.
CONFLICT_BACKUP_MULTIPLE=no
## Osync will clean backup files after a given number of days. Setting this to 0 will disable cleaning and keep backups forever. Warning: This can be very space consuming.
CONFLICT_BACKUP_DAYS=30
## If the same file exists on both replicas, newer version will be synced. However, if both files have the same timestamp but differ, CONFILCT_PREVALANCE sets winner replica.
CONFLICT_PREVALANCE=initiator

## On deletion propagation to the target replica, a backup of the deleted files can be kept. Deletions will be kept in .osync_workdir/deleted
SOFT_DELETE=yes
## Osync will clean deleted files after a given number of days. Setting this to 0 will disable cleaning and keep deleted files forever. Warning: This can be very space consuming.
SOFT_DELETE_DAYS=30

## Optional deletion skip on replicas. Valid values are "initiator", "target", or "initiator,target"
SKIP_DELETION=

## ---------- RESUME OPTIONS

## Try to resume an aborted sync task
RESUME_SYNC=yes
## Number maximum resume tries before initiating a fresh sync.
RESUME_TRY=2
## When a pidlock exists on slave replica that does not correspond to the initiator's instance-id, force pidlock removal. Be careful with this option if you have multiple initiators.
FORCE_STRANGER_LOCK_RESUME=no

## Keep partial uploads that can be resumed on next run, experimental feature
PARTIAL=no

## Use delta copy algortithm (usefull when local paths are network drives), defaults to yes
DELTA_COPIES=yes

## ---------- ALERT OPTIONS

## List of alert mails separated by spaces
## Most Unix systems (including Win10 bash) have mail support out of the box
## Just make sure that the current user has enough privileges to use mail / mutt / sendmail and that the mail system is configured to allow outgoing mails
## on pfSense platform, smtp support needs to be configured in System > Advanced > Notifications
DESTINATION_MAILS=""

## Optional change of mail body encoding (using iconv)
## By default, all mails are sent in UTF-8 format without header (because of maximum compatibility of all platforms)
## You may specify an optional encoding here (like "ISO-8859-1" or whatever iconv can handle)
MAIL_BODY_CHARSET=""

## Additional mail parameters needed for Android / Busybox / Cygwin / MSYS
## Android & Busybox use sendmail (and openssl if encryption is needed)
## MSYS & Cygwin Windows mail support relies on mailsend.exe from muquit, http://github.com/muquit/mailsend which needs to be in %PATH% environment variable

SENDER_MAIL="[email protected]"
SMTP_SERVER=smtp.your.isp.tld
SMTP_PORT=25
# encryption can be tls, ssl or none
SMTP_ENCRYPTION=none
SMTP_USER=
SMTP_PASSWORD=

## ---------- EXECUTION HOOKS

## Commands can will be run before and / or after sync process (remote execution will only happen if REMOTE_OPERATION is set).
LOCAL_RUN_BEFORE_CMD=""
LOCAL_RUN_AFTER_CMD=""

REMOTE_RUN_BEFORE_CMD=""
REMOTE_RUN_AFTER_CMD=""

## Max execution time of commands before they get force killed. Leave 0 if you don't wan't this to happen. Time is specified in seconds.
MAX_EXEC_TIME_PER_CMD_BEFORE=0
MAX_EXEC_TIME_PER_CMD_AFTER=0

## Stops osync execution if one of the above commands fail
STOP_ON_CMD_ERROR=yes

## Run local and remote after sync commands even on failure
RUN_AFTER_CMD_ON_ERROR=no

[deajan]

Looks like those versions of inotifywait that don't accept more than one --exclude option.
Can you confirm by making a test with inotifywait -r -e create /somedir and add one than two --exclude arguments ?
If so, can you give me the inotifywait version please ?

[g105b]

Hi @deajan,

My inotifywait version is inotifywait 3.14, it's whatever installed with the default packages on Ubuntu 20.10.

I can confirm that your theory is correct - zero or one --exclude arguments works, multiple cause the error.

However, I think there is one deeper issue here, that when tracking file exclusions in a separate file, inotifywait is actually provided the --exclude-from argument, and I can't find any documentation of this usage (I think --exclude-from is a rsync argument).

Either way, I've combined my exclusions into a single pattern as a solution for now, and I'm happy to say that osync is working perfectly if I ignore my original issue.

Do you think that the bash script should load the separate exclusions (either from the file, or from the semicolon delimited list), and build up a single regex to pass to intofiywait? Is that a good solution for backwards compatibility?

Cheers,
Greg.