Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

git reports files changed when there are no changes #127

Open
brianmay opened this issue Oct 14, 2021 · 28 comments
Open

git reports files changed when there are no changes #127

brianmay opened this issue Oct 14, 2021 · 28 comments

Comments

@brianmay
Copy link

I have had this often happen. And it kind of makes transcrypt very difficult to use, as git thinks the work tree is not clean.

$ git checkout .
$ git status
[ reports files as being modified ]
$ git diff
[ reports no changes ]
$ git checkout .
[ no change ]

The files in question were never modified locally.

A work around seems to be to commit the non-existent changes anyway, but that doesn't appear to be a good a long term solution. Eventually it will happen again.
@marcelcremer
Copy link

This one occured to us since about two or three weeks ago.

  • we have mac and WSL/Ubuntu in use
  • I thought it might have something to do with different openssl versions (1.1.1f in ubuntu vs. LibreSSL 2.8.3 on mac), but even after installing and using LibreSSL via homebrew it still shows changes where no changes are present
  • transcrypt is for all of my team members 2.1.0
  • resetting or checking out changes from the "original" branch does not change anything (still showing changes, where no changes are present)

@andreineculau
Copy link
Collaborator

Which git version? I'm guessing #37 is back maybe due to a newer git.

Also could you please copy-paste an exact output of transcrypt? It's always good to iron out that we're taking about the same thing.

@andreineculau
Copy link
Collaborator

I misread. You're saying it's 100% git.
So you're saying that you have cloned, worked etc and then all of a sudden you get dirty files though not dirty? Assuming the files in question are transcrypted, and they haven't been changed between checkouts i.e. empty diff pre-checkout-HEAD..HEAD

@coxjc
Copy link

coxjc commented Dec 15, 2021

FWIW, I'm getting same the issue. I'm checking out #37 now.

Git version: git version 2.24.3 (Apple Git-128)
On the latest Transcrypt.

@coxjc
Copy link

coxjc commented Dec 15, 2021

I keep getting the following anytime I run a git command:

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

I also can't get unchanged files to say they're unchanged.

@coxjc
Copy link

coxjc commented Dec 15, 2021

UPDATE: I resolved this by overwriting the Xcode-packaged git w/ Homebrew. Now I'm on git 2.34.1 and all is good.

@brianmay
Copy link
Author

@coxjc See #59 concerning those (annoying) warnings. I suspect fixing this might cause breakage.

I have had this problem - changes when there are no changed files - occur through a number of versions of git. Typically after I pull changes from other developers. It might be caused be differences between development environments between different developers.

Once I fix the problem - e.g. by committing the "unchanged" files I can't get it to reoccur again on demand.

@jmurty
Copy link
Collaborator

jmurty commented Dec 15, 2021

Could anyone experiencing this issue try the latest pre-release version of transcrypt from the main branch?

The fix for #37 (PR #109) was merged back in April 2021 but we haven't cut a release since 2.1 in September 2020, so there's a chance the pre-release version on main will fix it.

Or manually run git update-index -q --really-refresh in the repo to see if that fixes the incorrect stale file status: that's what the #109 fix does.

@brianmay
Copy link
Author

@jmurty I think when I last experienced this I did have that change. From the git repo. git version aea3ff8 to be precise.

Will keep your suggestions in mind if I can reproduce the problem.

@brianmay
Copy link
Author

brianmay commented Jan 11, 2022 •
edited
Loading

I have the following version of transcrypt, which I believe is the latest in main:

commit aea3ff83c7aea95b18e18839d5beb6da29b7548c (HEAD -> main, origin/main, origin/HEAD)
Author: Aaron Bull Schaefer <[email protected]>
Date:   Thu Apr 29 16:25:56 2021 -0700

    Remove Ubuntu 16.04 LTS from test matrix (#123)
    
    The "Xenial Xerus" version of Ubuntu is EOL (end-of-life) as of 2021-04-30.
    
    See:
    - https://help.ubuntu.com/community/EOL

git pull on transcrypt dir does nothing:

$ git pull
Already up to date.

git status on my repo reports files changed:

brian@debian:~/tree/ea/infra/terra/values$ git status
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
On branch master
Your branch is behind 'origin/master' by 45 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   titles/central/aws/shared/net/rr3vpngw/psk.auto.tfvars
	modified:   titles/central/aws/shared/net/sfpvpngw/psk.auto.tfvars
	modified:   titles/nfs/aws/prod/services/aurora/env.auto.tfvars
	modified:   titles/nfs/aws/prod/services/coda/env.auto.tfvars
	modified:   titles/nfs/aws/stage/services/rds/env.auto.tfvars
	modified:   titles/rr3/aws/prod/services/aurora/env.auto.tfvars
	modified:   titles/rr3/aws/stage/services/rds/env.auto.tfvars
	modified:   titles/sfp/aws/prod/services/aurora/env.auto.tfvars
	modified:   titles/sfp/aws/stage/services/rds/env.auto.tfvars

Untracked files: [...]

no changes added to commit (use "git add" and/or "git commit -a")

I removed the untracked files from the output, these are expected and not related.

git diff reports no changes:

brian@debian:~/tree/ea/infra/terra/values$ git diff | cat
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

If I run the suggested command I get:

brian@debian:~/tree/ea/infra/terra/values$ git update-index -q --really-refresh
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

But the output of git status and git diff has not changed and is the same as above.

In this particular case it means I can't run the "git pull" on the repo, because git incorrectly thinks the local files conflict:

brian@debian:~/tree/ea/infra/terra/values$ git pull
Updating c3bba7b..2ed9510
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
error: Your local changes to the following files would be overwritten by merge:
	titles/central/aws/shared/net/rr3vpngw/psk.auto.tfvars
	titles/central/aws/shared/net/sfpvpngw/psk.auto.tfvars
Please commit your changes or stash them before you merge.

Similarly, I cannot run other commands such as rebase.

git stash will stash away something, but the phantom changes remain.

Same goes with git checkout . - it completes without errors, but the changes remain.

I have kept a copy of the working dir, so can run any other commands you want me to.

My theory is that the files were committed by another developer, and as a result there might be something different about the encryption or encoding or something that is confusing transcrypt.

@jmurty
Copy link
Collaborator

jmurty commented Jan 15, 2022 •
edited
Loading

Thanks for the detailed report @brianmay. I appreciate all the reports and feedback here into what is an annoying and difficult to reproduce issue.

Unfortunately I only have some guesswork to offer in response, but it's worth looking into especially since it aligns with your suspicions about there being "something different" about commits from another developer.

In issue #118 @Aramgutang found, documented, and fixed an edge case bug in Transcrypt where a partial commit of a transcrypt-ed file on one machine would cause these kinds of symptoms on another machine. For Git client tools that permit a partial commit of transcrypt-ed files, the bug would cause the original committing machine to generate an incorrect salt from that machine's file on disk, instead of from the partially-changed file bytes actually being committed. Most Git clients do not (and should not) allow partial commits of the secret files, but some do, in particular the very popular VSCode.

The explanation in issue #118 is much better, I encourage you to go and read it.

Even though you are indeed using the latest main branch version of Transcrypt yourself, if someone else is using a version of Transcrypt without the fix (which is not in the latest released version) and they make a partial commit on a secret file using a client that permits such a thing, the salt will be incorrect by the time you pull the commit.

Can you check with the developer who committed the changes causing your problem whether:

If both of these are the case, it could explain why you keep seeing this problem. And if that bug is indeed the cause, getting all devs on your project to use the main branch version could solve the problem, pending an official release that includes the fix.

@brianmay
Copy link
Author

Other developer on leave until February, will follow-up then.

@fannytavart
Copy link

fannytavart commented Jan 19, 2022 •
edited
Loading

I had the same issue, I was able to fix it by uninstalling OpenSSL (my version was OpenSSL 3.0.1 14 Dec 2021) and installing LibreSSL (brew install libressl) instead.

@brianmay
Copy link
Author

brianmay commented Mar 6, 2022

I have a suspicion this might be still occurring for me too, but my coworker is rather busy, and haven't been able to confirm what version he is using yet.

@jmurty
Copy link
Collaborator

jmurty commented Jul 9, 2022

To follow up on this: until recently Transcypt was not compatible with OpenSSL version 3 (see #133).

We just released the latest official version 2.2.0 with a compatibility fix.

@emcniece
Copy link

emcniece commented Feb 8, 2023

I've been encountering this for a few weeks now and finally associated the phantom changes with Transcrypt, which lead me here. Investigation ongoing, I too seem to only encounter this when files are changed by other developers.

$ sw_vers
ProductName:		macOS
ProductVersion:		13.1
BuildVersion:		22C65

$ which transcrypt
/opt/homebrew/bin/transcrypt

$ transcrypt --version
transcrypt 2.2.0

$ which openssl
/usr/bin/openssl

$ openssl version
LibreSSL 3.3.6

$ which git
/opt/homebrew/bin/git

$ git --version
git version 2.37.3

@brianmay
Copy link
Author

brianmay commented Feb 8, 2023

@emcniece Are you able to confirm if the other developers are using the same version?

@emcniece
Copy link

emcniece commented Feb 8, 2023

Indeed, they report:

  • transcrypt 2.2.0
  • LibreSSL 2.8.3
  • git version 2.38.1

Fairly sure they're on OSX 12.

@jmurty
Copy link
Collaborator

jmurty commented Feb 8, 2023 •
edited
Loading

Hi @emcniece sorry for your problems. If you're on Mac OS 13 (Ventura) this is the likely culprit #147 and we don't yet have a fix that works reliably.

In the meantime downgrading to version 2.1.0 on systems running MacOS 13 is probably the best work-around.

@jmurty
Copy link
Collaborator

jmurty commented Feb 9, 2023

Hi, if anyone here is suffering from the compatibility bug with MacOS Ventura and has some time, I'd appreciate it if you could run some tests for me on your system(s) and give me feedback to help isolate the lingering bugs – please see instructions in my comment #147 (comment)

Thanks!

@jmurty
Copy link
Collaborator

jmurty commented Feb 11, 2023

For anyone experiencing phantom file changes recently – and where anyone committing to the repository has been using transcrypt version 2.2.0 with MacOS 13 Ventura (or with LibreSSL 3+ in general) – please upgrade to the new 2.2.1 release of transcrypt across all your repositories.

You may still see phantom changes after the upgrade, but re-committing the phantom changed file should fix it once and for all. More info here (#147)

@twrally
Copy link

twrally commented Nov 15, 2024

we are seeing this with transcrypt 2.3.0. We are upgrading from transcrypt 1.1.0 and everyone uses LibreSSL 3. I can upgrade the repo to 2.3.0 but the next person that clones the repo and runs transcrypt 2.3.0 to setup the repo on their machine can see some of the phantom files. Recommitting them does seem to work for their cloned repo, but then the next person has the same issue, etc. Is there a recommended path we should be doing this maybe? It doesnt always show up, but sometimes. I am worried we have branches open with secrets from v1.1.0 when we upgrade to v2.3.0 that we will see this. Should we be closing those branches maybe and not try to use them in 2.3.0? I was hoping everyone could just do a transcrypt --upgrade in their repos, but it seems like pulling a fresh after its been re-encrypted to 2.3.0 might be our best option to avoid these phantom files if possible. Suggestions? Can I help debug this in some way?

@tfriedel
Copy link

I got fed up with this issue after not finding a permanent solution for many years. In the end I switched to git-crypt which works fine.

@brianmay
Copy link
Author

brianmay commented Nov 16, 2024 •
edited
Loading

I think I wouldn't trust the security of transcrypt anymore, see #126 #162 etc.

transcrypt is good in that it will work with any file type, and produce good diffs. But not so good in that secrets are left in plain text in the working tree. And uses known insecure encryption.

Some alternatives, depending on what you are trying to do, include:

Another approach entirely is to store the secrets on a server, and retrieve them as required.

  • helm-secrets (supports vault IIRC).
  • vault
  • openbao (fork of vault)
  • bitwarden
  • 1password

@jmurty
Copy link
Collaborator

jmurty commented Nov 16, 2024

Hi @twrally I think you are experiencing problems due to upgrading major versions from 1.x to 2.x.

There was a bug fix for salt generation in the version 2.0.0 release from over 5 years ago that requires re-encryption and a simultaneous upgrade by all users to avoid issues with persistent changed files. This fix was a breaking change and more painful that the usual upgrade path, as you are finding.

These steps should fix the issue and get your team back to a consistent, working setup:

  • have all users upgrade to latest stable version of transcrypt and follow the steps for the v2.0.0 release notes to flush and reapply the transcrypt configuration in their local repositories
  • have one person commit and push the re-encrypted secret files on a primary branch
  • have users on all other branches obtain the re-encryption commit, either by merging from the primary branch or by cherry-picking the re-encryption commit.

Unfortunately transcrypt's --upgrade option despite the name is not helpful for migrating from versions 1.x to 2.x because it does not change (or fix) the transcrypt configuration in the local repository, it only upgrades the scripts that do the work. This limitation is not at all clear in the description of this command, so I'm sorry it has sent you down the wrong path.

@twrally
Copy link

twrally commented Nov 18, 2024 via email

@twrally
Copy link

twrally commented Nov 19, 2024 via email

@twrally
Copy link

twrally commented Nov 19, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@jmurty @brianmay @andreineculau @emcniece @tfriedel @marcelcremer @coxjc @fannytavart @twrally