You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Question] What is the memory footprint of te.Linear() weights ? #222
Currently the FP8 weights are only internal and so the actual model weights take the same amount of memory as without FP8 execution (e.g. 2B for FP16+FP8 training). We are working together with Meta on exposing FP8 tensors in pyTorch, which will enable storing only the FP8 weights, resulting in memory savings over the base model as well as e.g. faster communication in FSDP, but it is currently in the PoC stage.
Activity
ptrendx commentedon May 19, 2023
Currently the FP8 weights are only internal and so the actual model weights take the same amount of memory as without FP8 execution (e.g. 2B for FP16+FP8 training). We are working together with Meta on exposing FP8 tensors in pyTorch, which will enable storing only the FP8 weights, resulting in memory savings over the base model as well as e.g. faster communication in FSDP, but it is currently in the PoC stage.