As you can see, I'm running a pretty recent build of git-annex (ac799c3f363e0008b23e9c174e6fedc35e6fa92a),

$ git annex version
git-annex version: 3.20120630
local repository version: 3
default repository version: 3
supported repository versions: 3
upgrade supported from repository versions: 0 1 2

We have a file here which isn't currently available yet isn't currently available (the link is shown in red),

$ ls -l plot.py
lrwxrwxrwx 1 ben ben 77 Jul  6 14:01 plot.py -> ../.git/annex/objects/WORM:1301941019:720:plot.py/WORM:1301941019:720:plot.py
$ 

Yet git-annex should be able to tell us where it is,

$ git-annex whereis plot.py
$ 

Hmm, well that's strange. What's happening here,

$ git-annex whereis plot.py -d
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","show-ref","git-annex"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","show-ref","--hash","refs/heads/git-annex"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","log","refs/heads/git-annex..d5582e05f41011b571a17003934fe9e40859e4be","--oneline","-n1"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","cat-file","--batch"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","ls-files","--cached","-z","--","plot.py"]
$ 

Alright, well maybe git-annex get will work,

$ git annex get plot.py -d
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","ls-files","--cached","-z","--","plot.py"]
$ ls -l plot.py
lrwxrwxrwx 1 ben ben 77 Jul  6 14:01 plot.py -> ../.git/annex/objects/WORM:1301941019:720:plot.py/WORM:1301941019:720:plot.py

Nope, the link is still shown in red.

Alright, what about git-annex copy?

$ git annex copy plot.py --from=goldnerlab --to=here -d 
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","show-ref","git-annex"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","show-ref","--hash","refs/heads/git-annex"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","log","refs/heads/git-annex..d5582e05f41011b571a17003934fe9e40859e4be","--oneline","-n1"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","cat-file","--batch"]
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","ls-files","--cached","-z","--","plot.py"]
$ ls -l plot.py
lrwxrwxrwx 1 ben ben 77 Jul  6 14:01 plot.py -> ../.git/annex/objects/WORM:1301941019:720:plot.py/WORM:1301941019:720:plot.py

Still red.

Alright, what if I just try to get a non-existent file?

$ git annex get adsflkah -d
git ["--git-dir=/home/ben/lori/analysis/data/.git","--work-tree=/home/ben/lori/analysis/data","ls-files","--cached","-z","--","adsflkah"]
$

Alright, it didn't fail with an error, that's very strange. What is going on here?

git ls-files is not listing your file. Perhaps your file is not checked into git?
Comment by http://joeyh.name/ Mon Jul 9 23:16:32 2012

Not really sure what to say about that other than,

$ git --git-dir=/home/ben/lori/analysis/data/.git --work-tree=/home/ben/lori/analysis/data ls-files --cached -- plot.py
plot.py
$ 

What does

git-annex status

Show?

Do you have any remotes configured? It looks like you don't somehow.

$ git annex status
supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL
supported remote types: git S3 bup directory rsync web hook
trusted repositories: 0
semitrusted repositories: 3
        00000000-0000-0000-0000-000000000001 -- web
        02e4ea72-a77c-11e1-bbd7-0749b04e4b59 -- goldnerlab (Data for Goldner)
        3c1fd026-c794-11e1-8ebb-dbe8684e8a73 -- here
untrusted repositories: 0
dead repositories: 0
transfers in progress: none
available local disk space: 16 gigabytes (+1 megabyte reserved)
local annex keys: 0
local annex size: 0 bytes
known annex keys: 0
known annex size: 0 bytes
bloom filter size: 16 mebibytes (0% full)
backend usage: 
$ git remote 
goldnerlab
$ git remote show goldnerlab
* remote goldnerlab
  Fetch URL: goldnerlab:data
  Push  URL: goldnerlab:data
  HEAD branch (remote HEAD is ambiguous, may be one of the following):
    master
    synced/master
  Remote branches:
    git-annex     tracked
    master        tracked
    synced/master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local refs configured for 'git push':
    git-annex     pushes to git-annex     (up to date)
    master        pushes to master        (up to date)
    synced/master pushes to synced/master (up to date)

Well that's odd. You have remotes but no annexed files..

Can you post the commands you used to arrive at this situation? I'm not sure how you would have done that.. Maybe you just need a

git-annex sync

to get things going?

I think somehow you cloned the git repo but not the annex stuff.

I can easily reproduce the issue as follows,

$ git clone goldnerlab:data                                                                                                                           
Cloning into 'data'...                                                                                                                                
remote: Counting objects: 61902, done.                                                                                                                
remote: Compressing objects: 100% (61354/61354), done.                                                                                                
remote: Total 61902 (delta 356), reused 61902 (delta 356)                                                                                             
Receiving objects: 100% (61902/61902), 5.50 MiB | 894 KiB/s, done.                                                                                    
Resolving deltas: 100% (356/356), done.                                                                                                               
$ cd data                                                                                                                                             
$ git annex sync                                                                                                                                      
(merging origin/git-annex into git-annex...)                                                                                                          
commit                                                                                                                                                
(Recording state in git...)                                                                                                                           
# On branch master                                                                                                                                    
nothing to commit (working directory clean)                                                                                                           
ok                                                                                                                                                    
pull origin                                                                                                                                           
ok                                                                                                                                                    
push origin                                                                                                                                           
Counting objects: 8, done.                                                                                                                            
Delta compression using up to 2 threads.                                                                                                              
Compressing objects: 100% (5/5), done.                                                                                                                
Writing objects: 100% (6/6), 726 bytes, done.                                                                                                         
Total 6 (delta 1), reused 1 (delta 0)                                                                                                                 
Auto packing the repository for optimum performance.                                                                                                  
warning: There are too many unreachable loose objects; run 'git prune' to remove them.                                                                
To goldnerlab:data                                                                                                                                    
   d5582e0..aaddf3c  git-annex -> git-annex                                                                                                           
ok                                                                                                                                                    

Everything looks good so far. I verify that alex/plot.py doesn't exist. Now let's try getting it,

$ git annex get alex/plot.py -d                                                                                                                       
git ["--git-dir=/home/ben/data/.git","--work-tree=/home/ben/data","ls-files","--cached","-z","--","alex/plot.py"]                                     

Uh oh. ls confirms that get was unsucessful.

But how was the goldnerlab:data repository created? That looks to be where the problem is..

I have a slightly older version, but in general it should work the same.. you can see right away, when I do git annex status it shows "known annex keys: 1". if you do git annex status on goldnerlab, does it say you have any annex keys?

$ git-annex version
git-annex version: 3.20120614~bpo60+1
$ mkdir a
$ cd a
$ git init
Initialized empty Git repository in /tmp/a/.git/
$ git annex init a
init a ok
(Recording state in git...)
$ echo hi > file
$ git annex add file
add file (checksum...) ok
(Recording state in git...)
$ git commit -m added
fatal: No HEAD commit to compare with (yet)
fatal: No HEAD commit to compare with (yet)
[master (root-commit) cfa9049] added
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 120000 file
$ cd ..
$ git clone a a_clone
Cloning into a_clone...
done.
$ cd a_clone
$ git annex status
(merging origin/git-annex into git-annex...)
supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL
supported remote types: git bup directory rsync web hook
trusted repositories: 0
semitrusted repositories: 3
        00000000-0000-0000-0000-000000000001 -- web
        445d616e-ca8b-11e1-b170-ff8b03c54243 -- origin (a)
        5d3db51c-ca8b-11e1-bbc3-039dd06ab47b -- here
untrusted repositories: 0
dead repositories: 0
available local disk space: 63 megabytes (+1 megabyte reserved)
local annex keys: 0
local annex size: 0 bytes
known annex keys: 1
known annex size: 3 bytes
backend usage:
        SHA256: 1
(Recording state in git...)
$ ls
file
$ cat file
cat: file: No such file or directory
$ git annex get file
get file (from origin...) ok
(Recording state in git...)
$ cat file
hi

On goldnerlab,

$ git annex status
supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL
supported remote types: git S3 bup directory rsync web hook
trusted repositories: 0
semitrusted repositories: 4
    00000000-0000-0000-0000-000000000001 -- web
    02e4ea72-a77c-11e1-bbd7-0749b04e4b59 -- here (Data for Goldner)
    351f3ddc-ca3e-11e1-a3fc-6338ef4724a7
    3c1fd026-c794-11e1-8ebb-dbe8684e8a73
untrusted repositories: 0
dead repositories: 0
transfers in progress: none
available local disk space: 2 terabytes (+1 megabyte reserved)
local annex keys: 19101
local annex size: 41 gigabytes
known annex keys: 19122
known annex size: 41 gigabytes
bloom filter size: 16 mebibytes (3.8% full)
backend usage:
        WORM: 38223

Can you run the series of commands I had above on your two machines? I figure there are two possibilities:

  1. There is something wrong with the git-annex versions you are using.
  2. There is something wrong with your repository. ("warning: There are too many unreachable loose objects"?)

so if you can make a temp repository on goldnerlab, then clone it on the other machine and see where it fails, that would be helpful.

after cloning git-annex status should hopefully say that you have 1 known key, not 0.

Obviously this won't fix the problem, but it will at least narrow it down.

Hmm, the commands above seem to have worked on both machines (both running 3.20120630). I guess I should probably just try rebuilding my data/ repository from scratch, eh?

I suppose.. joey can probably help you investigate exactly what went wrong. You might want to save an empty clone of the git repository for later..

The easiest way to fix the data is probably to run a git annex uninit in the old repository which will put the files back how they were before and then git-annex import them into a new repository.

Looking at this a leetle more closely, you had:

lrwxrwxrwx 1 ben ben 77 Jul  6 14:01 plot.py -> ../.git/annex/objects/WORM:1301941019:720:plot.py/WORM:1301941019:720:plot.py

Well, that is not how a git-annex symlink currently looks, so it ignores it.

Apparenly this repository was created with an old version of git-annex, possibly version 1, and you've dropped in the current version, but the normal upgrade machinery failed. This could happen if you made a new clone of a version 1 bare repository.

I suggest you first find out what version of git-annex was originally used to create this repository (ie, version 0, 1, or 2 ... probably 1). Then make a clone, and "git config annex.version $N" where N=the version used). Then "git annex upgrade" and you should be good to go. Remember to push or sync the upgrade back to the bare repo so you don't need to do this again.

Comment by http://joeyh.name/ Wed Jul 11 23:23:04 2012
Comments on this page are closed.