Feat/lazy init #1539
Feat/lazy init #1539
Conversation
nathanielsimard
force-pushed
the
feat/lazy-init
branch
from
895f3e0
to
e9f32da
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1539 /- ##
==========================================
Coverage 86.30% 86.48% 0.17%
==========================================
Files 684 684
Lines 78216 78093 -123
==========================================
Hits 67502 67535 33
Misses 10714 10558 -156 ☔ View full report in Codecov by Sentry. |
| /// To avoid creating locks on already initialized parameter, we wrap the lock inside an | ||
| /// Option, the inner option is required for resetting the state onced initialized. | ||
| /// TLDR: RwLock(None) only happens on the param reference that is lazy, but was initialized, | ||
| /// all other parameters |
There was a problem hiding this comment.
I feel like the last sentence is incomplete 😅
all other parameters [blank].. what?
| /// To avoid creating locks on already initialized parameter, we wrap the lock inside an | ||
| /// Option, the inner option is required for resetting the state onced initialized. | ||
| /// TLDR: RwLock(None) only happens on the param reference that is lazy, but was initialized, | ||
| /// all other parameters |
There was a problem hiding this comment.
I feel like the last sentence is incomplete 😅
all other parameters [blank].. what?
| pub fn init_with<B: Backend>(&self, record: BenchmarkModuleRecord<B>) -> BenchmarkModule<B> { | ||
| BenchmarkModule { | ||
| linears: record | ||
| .linears | ||
| .into_iter() | ||
| .map(|record| nn::Linear { | ||
| weight: record.weight, | ||
| bias: record.bias, | ||
| }) | ||
| .collect(), | ||
| } | ||
| } |
There was a problem hiding this comment.
Getting rid off init_with and adding in other places? ;-)
| let running_mean = Tensor::zeros([self.num_features], device); | ||
| let running_var = Tensor::ones([self.num_features], device); | ||
|
|
There was a problem hiding this comment.
Any reason why it's different for running*?
| You need two things in order to load weights for a model: the model's weights and the model's | ||
| config. Since parameters in Burn are lazy initialized, no allocation and GPU/CPU kernels are executed by |
There was a problem hiding this comment.
Rephrase to avoid using "weights" too many times?
Two things are required to load a model's weights: the model's record and the model's config.
| @@ -40,29 40,18 @@ fn main() { | |||
|
|
|||
| ## Good practices | |||
|
|
|||
| By using the Config pattern it is easy to create instances from this | |||
| config. Therefore, initialization methods should be implemented on the config struct. | |||
| By using the config type it is easy to create new module instances from it. The initialization | |||
There was a problem hiding this comment.
create new module instances from "it" sounds a bit weird. I think we can simply leave this out.
By using the config type it is easy to create new module instances.
Fix #1329