4

According to this tutorial when I update file I should remove file from under DVC control first (i.e. execute dvc unprotect <myfile>.dvc or dvc remove <myfile>.dvc) and then add it again via dvc add <mifile>. However It's not clear if I should apply the same workflow for the directories.

I have the directory under DVC control with the following structure:

data/
    1.jpg
    2.jpg

Should I run dvc unprotect data every time the directory content is updated?

More specifically I'm interested if I should run dvc unprotect data in the following use cases:

  • New file is added. For example if I put 3.jpg image in the data dir
  • File is deleted. For example if I delete 2.jpg image in the data dir
  • File is updated. For example if I edit 1.jpg image via graphic editor.
  • A combination of the previous use cases (i.e. some files are updated, other deleted and new files are added)
desertnaut
  • 57,590
  • 26
  • 140
  • 166
NShiny
  • 1,046
  • 1
  • 10
  • 19

2 Answers2

2

Only when file is updated - i.e. edit 1.jpg with your editor AND only if hadrlink or symlink cache type is enabled.

Please, check this link:

updating tracked files has to be carried out with caution to avoid data corruption when the DVC config option cache.type is set to hardlink or/and symlink

I would strongly recommend reading this document: Performance Optimization for Large Files it explains benefits of using hardlinks/symlinks.

Shcheklein
  • 5,979
  • 7
  • 44
  • 53
  • How can I check if cach.type is hardlinks/symlinks? Executing 'dvc config --local cache.type' gives me "config file error: section 'cache' doesn't exist". Same happens with "--system" and "--global" options. – NShiny May 24 '19 at 11:14
  • Default value is `reflink,copy`, so it does not use hardlinks, symlinks. There is no option to get current effective values for the config - you need to check docs here https://dvc.org/doc/commands-reference/config#configuration-sections .. Could you file a feature request on github to being able to print current configuration? – Shcheklein May 24 '19 at 18:39
0

Links above do not work anymore -> here is the up-to-date link and also pasting the instructions here:

Modifying content

Unlink the file with dvc unprotect. This will make train.tsv safe to edit:

dvc unprotect train.tsv

Then edit the content of the file, for example with:

echo "new data item" >> train.tsv

Add the new version of the file back with DVC:

dvc add train.tsv
git add train.tsv.dvc
git commit -m "modify train data"

If you have remote storage and/or an upstream repo:

dvc push
git push

Replacing files

If you want to replace the file altogether, you can take the following steps.

First, stop tracking the file by using dvc remove on the .dvc file. This will remove train.tsv from the workspace (and unlink it from the cache):

dvc remove train.tsv.dvc

Next, replace the file with new content:

echo new > train.tsv

And start tracking it again:

dvc add train.tsv
git add train.tsv.dvc .gitignore
git commit -m "new train data"

If you have remote storage and/or an upstream repo:

dvc push
git push
Henryk Borzymowski
  • 988
  • 1
  • 10
  • 22
  • Thanks for the detailed steps , But I wanted to replace a binary file and wanted to know that once I follow the mentioned steps will it remove the old file from remote storage(i.e. S3 bucket)? – sohel shaikh Oct 20 '22 at 05:21