Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

What's the purpose of these lines? #25

Open
XavierXiao opened this issue Feb 1, 2023 · 3 comments
Open

What's the purpose of these lines? #25

XavierXiao opened this issue Feb 1, 2023 · 3 comments

Comments

@XavierXiao
Copy link

Thanks for your amazing work! I have one quick question: what is the purpose of these lines, in the modified CrossAttention's forward function? It seems like you disable the gradient of the first token in the embedding? Can you explain a bit?

Thanks!
@nupurkmr9
Copy link
Collaborator

Hi,

Since the first start of the sentence token is always fixed, I noticed a small improvement when detaching it during the training. I guess this helps in better association between the "V* category" and the target image and thus improved generation on inference time prompt.

Thanks.

@XavierXiao
Copy link
Author

thanks! Another possible issue I spotted is here, where it always assume the --freeze_model is 'crossattn_kv', and if I set this argument to 'crossattn' this line will disregard it.

@nupurkmr9
Copy link
Collaborator

Ohh yeah. Thanks so much for catching it!!
I have corrected it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nupurkmr9 @XavierXiao