error: failed to push some refs for large file

[anythingthatworks]

I'm pushing a very large file(7gigs, thousands of files) into my Dropbox, and my pushes fail partway through every time. I've tried pushing a couple different times to check where the progress stops, but it's inconsistent.

the error I normally get is: "error: failed to push some refs to Dropbox:///Git-Dropbox:Speeder"

I started using git-remote-dropbox in order to store an entire unity game project I'm working on, including assets in order to quickly be able to share between team members and also track updates to all the files.

A bit of research says that this looks similar to issue issues 15 , 36, and 57, but I couldn't find solutions in those issues.

git version 2.22
windows 10 v1903
python 3.8
git-remote-dropbox latest release

running the program with -v gives

Traceback (most recent call last):
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1043, in _send_output
self.send(chunk)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 965, in send
self.sock.sendall(data)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1204, in sendall
debug: writing: /git-dropbox:speeder/objects/2c/5bf73c65039d992b554a6c683046fcae495417
v = self.send(byte_view[count:])
File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\util\retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1043, in _send_output
self.send(chunk)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 965, in send
self.sock.sendall(data)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\0rson\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\0rson\appdata\local\programs\python\python38\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\0rson\AppData\Local\Programs\Python\Python38\Scripts\git-remote-Dropbox.exe_main.py", line 9, in
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\git_remote_dropbox\cli\helper.py", line 23, in main
helper.run()
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\git_remote_dropbox\helper.py", line 108, in run
self._do_push(line)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\git_remote_dropbox\helper.py", line 152, in _do_push
self._push(src, dst)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\git_remote_dropbox\helper.py", line 215, in _push
for done, _ in enumerate(res, 1):
File "c:\users\0rson\appdata\local\programs\python\python38\lib\multiprocessing\pool.py", line 865, in next
raise value
File "c:\users\0rson\appdata\local\programs\python\python38\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\git_remote_dropbox\util.py", line 103, in call
return method(self._obj, *args)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\git_remote_dropbox\helper.py", line 287, in _put_object
self._connection().files_upload(data, path, mode, mute=True)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\dropbox\base.py", line 2762, in files_upload
r = self.request(
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\dropbox\dropbox.py", line 269, in request
res = self.request_json_string_with_retry(host,
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\dropbox\dropbox.py", line 360, in request_json_string_wi
return self.request_json_string(host,
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\dropbox\dropbox.py", line 444, in request_json_string
r = self._session.post(url,
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\requests\sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))
error: failed to push some refs to 'Dropbox:///Git-Dropbox:Speeder'

[anythingthatworks]

The title also recommended I check out 64 but that looks resolved.

[anishathalye]

Yeah, I agree, there seems to be some issue with pushing repos with lots of large files. Or even a small number of large files (though I see such issues randomly, it's not consistent).

I didn't have any repos quite as large as yours, so I made a synthetic one with 1000 1M files, 100 10M files, 10 100M files, and 1 1G file with the following script:

for i in $(seq 1000); do
    dd if=/dev/urandom of=file-1M-$i.bin bs=1M count=1
done

for i in $(seq 100); do
    dd if=/dev/urandom of=file-10M-$i.bin bs=1M count=10
done

for i in $(seq 10); do
    dd if=/dev/urandom of=file-100M-$i.bin bs=1M count=100
done

for i in $(seq 1); do
    dd if=/dev/urandom of=file-1000M-$i.bin bs=1M count=1000
done

After this, I created a git repo, added all the files, and pushed it to dropbox. I got a slightly different error than yours:

...
debug: writing: /big-files/objects/69/7aaf196a5642650502f0d40f19c2d8f7c25b7e (size 10 MB)
debug: writing: /big-files/objects/eb/e2cc3f1c2b556d204fa26925eba9fb96e054d0 (size 10 MB)
debug: writing: /big-files/objects/c6/dd8888eba473c57aaf1ab7d9b39ac9fe5ed1b2 (size 10 MB)
debug: writing: /big-files/objects/69/60bc14dcaacf4b153d4eb527f3f8d9484fa1ca (size 10 MB)
debug: writing: /big-files/objects/7d/d6344e6cf0dc3cbe1f843854d4466b6a08a88e (size 10 MB)
debug: writing: /big-files/objects/30/7897d0c185a84e43498d474879b61133d5fc14 (size 10 MB)
Traceback (most recent call last):
  File "/home/anish/.local/bin/git-remote-dropbox", line 11, in <module>
    load_entry_point('git-remote-dropbox', 'console_scripts', 'git-remote-dropbox')()
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/cli/helper.py", line 23, in main
    helper.run()
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 108, in run
    self._do_push(line)
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 152, in _do_push
    self._push(src, dst)
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 215, in _push
    for done, _ in enumerate(res, 1):
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 673, in next
    raise value
requests.exceptions.ConnectionError: ('Connection aborted.', error("(32, 'EPIPE')",))
Exception in thread Thread-20 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 754, in run
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/util.py", line 103, in __call__
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 278, in _put_object
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/git.py", line 95, in encode_object
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/git.py", line 80, in object_data
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/git.py", line 12, in command_output
  File "/usr/lib/python2.7/subprocess.py", line 217, in check_output
  File "/usr/lib/python2.7/subprocess.py", line 480, in communicate
  File "/usr/lib/python2.7/subprocess.py", line 1099, in wait
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'waitpid'
error: failed to push some refs to 'dropbox:///big-files'

The failures are also nondeterministic. I tried a similar experiment as above, but with only 10 100M files and 1 1G file, and sometimes the push worked, and sometimes it failed. Sometimes, it failed with a different error:

debug: writing: /big-files/objects/ac/aa333ca0f93885f2cf872a3d04f6f5a65e6bd7 (size 1 MB)
Traceback (most recent call last):
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 316, in _put_object
    conn.files_upload_session_finish(chunk, cursor, commit_info)
  File "/home/anish/.local/lib/python2.7/site-packages/dropbox/base.py", line 2306, in files_upload_session_finish
debug: writing: /big-files/objects/b1/84940e6c51735c7933af51818f1821429be37d (size 1 MB)
    f,
  File "/home/anish/.local/lib/python2.7/site-packages/dropbox/dropbox.py", line 296, in request
    user_message_locale)
ApiError: ApiError('71252a8652b3eb530ddc9f3f3512b613', UploadSessionFinishError(u'lookup_failed', UploadSessionLookupError(u'incorrect_offset', UploadSessionOffsetError(correct_offset=1048895882))))
Traceback (most recent call last):
  File "/home/anish/.local/bin/git-remote-dropbox", line 11, in <module>
    load_entry_point('git-remote-dropbox', 'console_scripts', 'git-remote-dropbox')()
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/cli/helper.py", line 23, in main
    helper.run()
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 108, in run
    self._do_push(line)
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 152, in _do_push
    self._push(src, dst)
  File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 215, in _push
    for done, _ in enumerate(res, 1):
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 673, in next
    raise value
dropbox.exceptions.ApiError: ApiError('71252a8652b3eb530ddc9f3f3512b613', UploadSessionFinishError(u'lookup_failed', UploadSessionLookupError(u'incorrect_offset', UploadSessionOffsetError(correct_offset=1048895882))))
error: failed to push some refs to 'dropbox:///big-files'

As a temporary workaround, here's a suggestion you can try. If you have many commits, rather than trying to push the big repository at once, you can do it in pieces, so you can "checkpoint" progress. You can push older commits, e.g. with git push origin <commit hash of some ancestor of master>:master. And you can keep doing this with newer and newer ancestors, so you'll push partial data, and finally you can git push -u origin master to sync up Dropbox.

[anishathalye]

And actually, even the above example, with the huge number of files (1000 x 1M, ..., 1 x 1G) worked for me just now (pushing the entire repo at once).

[anythingthatworks]

coming back to this because I'm still having issues. I was able to make it work before just by continuously trying, but I've run into the same problem again.

For reference I'm pushing 14000 objects, so that could very well be the issue.

[anythingthatworks]

I have programming experience in C# but not in python, and I don't fully know what this code is doing, but here's what I've done and where I'm stuck.

so as per the advice of this, I've changed socket timeout to none.

gone into helper.py and changed
90 return dropbox.Dropbox(self._token)
to
90 return dropbox.Dropbox(self._token, timeout=None))

This manages to finish writing all the objects for my project, except then I get issues in UploadSessionOffsetError.

tracking down this error leads me to:
link1
link2

I can see that understanding and solving this issue will take some time for me, but I think I'm going to keep working on it because it would be really elegant if I could get this program working for my use case.

I'd be grateful for help or advice.

[echuber2]

I have no idea if it will help, but look at constants.py. Try reducing processes to 10 or fewer, increasing max retries, reducing (or increasing) chunk size.

[echuber2]

Also try setting a larger finite timeout as suggested here (instead of None, which may just cause problems, conceivably):

#36 (comment)

[anythingthatworks]

I was able to get the entire project pushed after trying several times, changing the constants.py settings between pushes. I also added a GitIgnore to the unity project in order to stop syncing non important files.

I solved my problem, but the issue is one change to constants worked for one project, but not for another, and having to go through that makes me uncomfortable using this with collaborators.

I've looked into UploadSessionOffsetError and found this. I think line endings could be the issue, might explain discrepancies in file size and I have been getting alerts that my line endings have been getting changed. I'll check this out later.

[Yukiya4]

Hello, i have the same issues with a project in Unreal Engine where i have to push 400 files for my first push.
I installed Git LFS for letting me to push heavy files as fbx or psd files.
i got the following errors :
"SSH: Could Not Resolve Hostname dropbox: no such host is known status 255"
and at last line the
"error: failed to push some refs for large file"

the link of my repo is "dropbox:///Git" which is neither a https nor a ssh in my opinion that's probably why it's bugging.
I uninstalled LFS and it's working again.

is it possible to get a solution for keeping your dropbox module at same time using LFS ?
Thank you.

ps: i tried again with another commit having only few files by letting LFS and it worked. i don't know why i had that error tho, but removing LFS let me to push my 400 files though.

[anishathalye]

git-remote-dropbox does not currently have any integration with Git LFS.