New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请教高手：关于./model/basic_var.py 实现的SelfAttention的两个问题 #140

Open

mswwd opened this issue Feb 26, 2025 · 0 comments

mswwd commented Feb 26, 2025

第一，我可以理解“attn_bias is None during inference”，但是kv cache和“attn_bias is None“之间的因果关系是什么？
第二，为什么单单令计算K的bias为0（即self.zero_k_bias）？
还请高手指点一二，万分感谢！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment