Skip to content

[Question] What is the memory footprint of te.Linear() weights ? #222

Closed

Description

@vince62s

Does each parameter take 1 byte, 2 bytes, 4 bytes ?

it does not seem so clear in terms of memory efficiency.

Activity

  1. ptrendx commented on May 19, 2023

    @ptrendx
    Member

    Currently the FP8 weights are only internal and so the actual model weights take the same amount of memory as without FP8 execution (e.g. 2B for FP16+FP8 training). We are working together with Meta on exposing FP8 tensors in pyTorch, which will enable storing only the FP8 weights, resulting in memory savings over the base model as well as e.g. faster communication in FSDP, but it is currently in the PoC stage.

  2. locked and limited conversation to collaborators on May 22, 2023
  3. converted this issue into a discussion #239 on May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions