-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch LFS to DVC #25
Conversation
README.md
Outdated
If using a data repository, push the newly created profiles to DVC and the .dvc files and other files to GitHub as follows | ||
|
||
```bash | ||
dvc add profiles/${BATCH} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can send files or folders to DVC but whatever unit you send is tracked as that full unit. So, if you add profiles/BATCH then you can only download that whole folder from DVC. So options for DVC usage are:
dvc add
all the profiles in a batch as one unit. This is the simplest and least error prone, but it means that anyone using the profiles will have to download them all as one large unit.dvc add
in more granular units. Makes it a bit more complicated to give explicit instructions, but means that profilers can access just that granular unit.
@shntnu @bethac07 Do you have opinions on whether we should do bulk or granular? (And if granular, how granular?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for thinking this through.
Do you oppose --recursive
and just tracking them all?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shntnu I think --recursive
is a great solution for both granular and easy :)
Thanks @niranjchandrasekaran Co-authored-by: Niranj Chandrasekaran <[email protected]>
I believe this is good to go @shntnu. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@ErinWeisbart would you like to go ahead and merge? |
@shntnu I think it's fine to merge. As I mentioned, I haven't run through my instructions start to finish since all the final edits, but I suppose if someone finds something unclear/wrong with a run-through, they can always start a new issue/PR. |
Thanks, Erin! I've made you a maintainer now FWIW |
Update README.md and environment.yaml so that large file storage uses DVC not Git LFS.