Continue reading...
Alternating the GPUs each layer is on didn’t fix it, but it did produce an interesting result! It took longer to OOM. The memory started increasing on gpu 0, then 1, then 2, …, until eventually it came back around and OOM. This means memory is accumulating as the forward pass goes on. With each layer more memory is allocated and not freed. This could happen if we’re saving activations or gradients. Let’s try wrapping with torch.no_grad and make required_grad=False even for the LoRA.
。立即前往 WhatsApp 網頁版对此有专业解读
That being said, a few points have been raised that are worth noting here.，更多细节参见传奇私服新开网｜热血传奇SF发布站｜传奇私服网站
If this is in error and you're using a current version of your