-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature 1.13 llm update #5063
Feature 1.13 llm update #5063
Conversation
Signed-off-by: weijingchen <talkingwallace@sohu.com>
support ipr fix typo Signed-off-by: weijingchen <talkingwallace@sohu.com>
1. Update homonn framework, support arbiter side model 2. Fix bug & log format 3. Update aggregator framework Signed-off-by: weijingchen <talkingwallace@sohu.com>
Signed-off-by: weijingchen <talkingwallace@sohu.com>
self._updated = {'trainer': False, 'dataset': False, | ||
'torch_seed': False, 'loss': False, 'optimizer': False, 'model': False} | ||
'torch_seed': False, 'loss': False, 'optimizer': False, 'model': False, 'ds_config': False, 'server_init': False} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
self._updated = {'trainer': False, 'dataset': False, | |
'torch_seed': False, 'loss': False, 'optimizer': False, 'model': False} | |
'torch_seed': False, 'loss': False, 'optimizer': False, 'model': False, 'ds_config': False, 'server_init': False} | |
self._updated = { | |
'trainer': False, | |
'dataset': False, | |
'torch_seed': False, | |
'loss': False, | |
'optimizer': False, | |
'model': False, | |
'ds_config': False, | |
'server_init': False} |
|
||
def __init__(self) -> None: | ||
super().__init__() | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
@property | ||
def embeded_param(self): | ||
return None | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
|
||
""" | ||
Recursively replaces the LayerNorm layers of a given module with SignatureLayerNorm layers. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
layer_name_set (set[str], optional): A set of layer names to be replaced. If None, | ||
all LayerNorm layers in the module will be replaced. | ||
""" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
@property | ||
def embeded_param(self): | ||
return self.ln.weight | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
return self.ln.weight | ||
|
||
def embeded_param_num(self): | ||
return self._embed_param_num |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
return self._embed_param_num | |
return self._embed_param_num |
conv = SignatureConv(3, 384, 3, 1, 1) | ||
layer_norm = SignatureLayerNorm((768, )) | ||
layer_norm_2 = SignatureLayerNorm.from_layer_norm_layer(layer_norm.ln) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
Signed-off-by: weijingchen <talkingwallace@sohu.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Chen <cwjghglbdcj@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Chen <cwjghglbdcj@gmail.com>
@@ -82,7 +86,9 @@ def __init__(self, | |||
loss=None, | |||
optimizer: OptimizerType = None, | |||
ds_config: dict = None, | |||
model: Sequential = None, **kwargs): | |||
model: Sequential = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[autopep8] reported by reviewdog 🐶
model: Sequential = None, | |
model: Sequential = None, |
Signed-off-by: weijingchen <talkingwallace@sohu.com>
Add support for FATE-LLM 1.3:
update nn framework
fix log & bug in nn