Bert #158
Merged
Bert #158
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit ae86109.
ScoThunder
commented
Jul 19, 2023
| @@ -83,9 83,6 @@ def evaluate(self, trainer): | |||
| total_masked = num_masked | |||
| #torch.cuda.synchronize() | |||
| dist_pytorch.barrier(config.vendor) | |||
| if config.vendor == 'kunlunxin': | |||
| import torch_xmlir.core.xpu_model as xm | |||
| xm.mark_step() | |||
There was a problem hiding this comment.
之前跑图模式需要这个xm.mark_step，现在是eager模式，不需要了
| from apex.parallel import DistributedDataParallel as APEX_DDP | ||
| from apex.parallel.distributed import flat_dist_call | ||
| except ImportError: | ||
| print("import apex error") |
There was a problem hiding this comment.
kunlunxin的机器没有apex，import会报错，加入try catch
| from torch.cuda.amp import GradScaler | ||
| from torch.nn.parallel import DistributedDataParallel as NativeDDP | ||
| from torch.optim import Optimizer | ||
|
|
||
| import utils | ||
| import config | ||
| #from converter import convert_model | ||
| from .distributed_fused_lamb import _pipeline_block_reductions_patched, _pipeline_step_patched |
| from torch.optim import Optimizer | ||
| from torch_xmlir.optimizer import Lamb | ||
| from torch_xmlir.optimizer import FusedLAMB |
| # e5m2_allgather=config.dwu_e5m2_allgather) | ||
| #optimizer.set_global_scale(float(os.getenv("INIT_LOSS_SCALE", 2 ** 20))) | ||
| else: | ||
| optimizer = Lamb(optimizer_grouped_parameters, |
| use_ddp = dist.is_initialized() | ||
| if use_ddp and config.use_xpu: | ||
| from torch_xmlir.distributed import DistributedDataParallel as DDP | ||
| model = DDP(model) |
There was a problem hiding this comment.
替换为torch原生的ddp，而不是自定的ddp
| optimizer, | ||
| delay_overflow_check=self.config. | ||
| allreduce_post_accumulation) as scaled_loss: | ||
| scaled_loss.backward() |
|
|
||
| update_step = step % config.gradient_accumulation_steps == 0 | ||
| if update_step: | ||
| update_model_params(loss, optimizer, grad_scaler) | ||
| else: | ||
| xm.mark_step() |
There was a problem hiding this comment.
eager模式不用xm.mark_step
| param.grad = None | ||
| else: | ||
| xm.optimizer_step(optimizer, barrier=True) | ||
|
|
shh2000
approved these changes
Jul 20, 2023
shh2000
approved these changes
Jul 20, 2023
yuzhou03
approved these changes
Jul 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
日志：root@p-kunlunxin-r480-005:/data/dufeilei/dev/code/FlagPerf/training/benchmarks/bert/pytorch/log/train_1x8.log
数据集：root@p-kunlunxin-r480-005:/data/datasets_ckpt/bert/train