Switch LFS to DVC #25
Conversation
README.md
Outdated
| If using a data repository, push the newly created profiles to DVC and the .dvc files and other files to GitHub as follows | ||
|
|
||
| ```bash | ||
| dvc add profiles/${BATCH} |
There was a problem hiding this comment.
You can send files or folders to DVC but whatever unit you send is tracked as that full unit. So, if you add profiles/BATCH then you can only download that whole folder from DVC. So options for DVC usage are:
dvc addall the profiles in a batch as one unit. This is the simplest and least error prone, but it means that anyone using the profiles will have to download them all as one large unit.dvc addin more granular units. Makes it a bit more complicated to give explicit instructions, but means that profilers can access just that granular unit.
@shntnu @bethac07 Do you have opinions on whether we should do bulk or granular? (And if granular, how granular?)
There was a problem hiding this comment.
Thanks for thinking this through.
Do you oppose --recursive and just tracking them all?
There was a problem hiding this comment.
@shntnu I think --recursive is a great solution for both granular and easy :)
Thanks @niranjchandrasekaran Co-authored-by: Niranj Chandrasekaran <[email protected]>
|
I believe this is good to go @shntnu. |
|
@ErinWeisbart would you like to go ahead and merge? |
|
@shntnu I think it's fine to merge. As I mentioned, I haven't run through my instructions start to finish since all the final edits, but I suppose if someone finds something unclear/wrong with a run-through, they can always start a new issue/PR. |
|
Thanks, Erin! I've made you a maintainer now FWIW |
Update README.md and environment.yaml so that large file storage uses DVC not Git LFS.