huawei
/
mindspore2022
0
0
Fork 53
Code Issues Pull Requests 29 Projects Releases Wiki Activity

花园宝宝战队 ----- 一阶段代码注释成果 #10

Open
bjutsecurity22 wants to merge 139 commits from bjutsecurity22/mindspore2022:master into master
pull from: bjutsecurity22/mindspore2022:master
merge into: huawei:master
Conversation 0 Commits 139 Files Changed 106 +20431 -10034
106 changed files with 20431 additions and 10034 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1954
mindspore/python/mindspore/Yolo v7 code comments for the blog of netural network learning （yolo v7 学习神经网络模型构建用代码评注）.py Normal file
View File

File diff suppressed because it is too large Load Diff

566
mindspore/python/mindspore/_checkparam.py
View File

File diff suppressed because it is too large Load Diff

18
mindspore/python/mindspore/dataset/audio/__init__.py
View File

@ -12,6 +12,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# 解释音频增强模块的结构和功能
"""
This module is to support audio augmentations.
It includes two parts: transforms and utils.
@ -30,5 +32,21 @@ Descriptions of common data processing terms are as follows:
- TensorOperation, the base class of all data processing operations implemented in C++.
- AudioTensorOperation, the base class of all audio processing operations. It is a derived class of TensorOperation.
"""
"""
该模块用于支持音频增强。
它包括两个部分：
transforms和utils。
transforms是一个高性能处理模块，其中包含常见的音频操作。
utils提供了一些用于音频处理的通用方法。
与相关API示例中常用的导入模块如下：
导入mindspore.dataset as ds
导入mindspore.dataset.audio.transforms as audio
常见的数据处理术语的描述如下：
TensorOperation是所有在C++中实现的数据处理操作的基类。
AudioTensorOperation是所有音频处理操作的基类，它是TensorOperation的派生类。
"""
# 从当前目录下的子目录"./"中导入名为"transforms"的模块
from . import transforms
# 从当前目录下的子目录"./"中导入名为"utils"的模块，并将其中的所有内容（包括函数、类、变量等）导入到当前模块中
from .utils import *

1546
mindspore/python/mindspore/dataset/audio/transforms.py
View File

File diff suppressed because it is too large Load Diff

112
mindspore/python/mindspore/dataset/audio/utils.py
View File

@ -21,18 +21,16 @@ import mindspore._c_dataengine as cde
from mindspore.dataset.core.validator_helpers import check_non_negative_float32, check_non_negative_int32, \
check_pos_float32, check_pos_int32, type_check
# 填充模式，可能的值为BorderType.CONSTANT, BorderType.EDGE, BorderType.REFLECT, BorderType.SYMMETRIC
class BorderType(str, Enum):
"""
Padding Mode, BorderType Type.
Possible enumeration values are: BorderType.CONSTANT, BorderType.EDGE, BorderType.REFLECT, BorderType.SYMMETRIC.
- BorderType.CONSTANT: means it fills the border with constant values.
- BorderType.EDGE: means it pads with the last value on the edge.
- BorderType.REFLECT: means it reflects the values on the edge omitting the last value of edge.
- BorderType.SYMMETRIC: means it reflects the values on the edge repeating the last value of edge.
BorderType.CONSTANT - 使用常量值填充。
BorderType.EDGE - 使用各边的边界像素值填充。
BorderType.REFLECT - 以各边的边界为轴进行镜像填充，忽略边界像素值。 例如，向 [1, 2, 3, 4] 的两边分别填充2个元素，结果为 [3, 2, 1, 2, 3, 4, 3, 2]。
BorderType.SYMMETRIC - 以各边的边界为轴进行对称填充，包括边界像素值。 例如，向 [1, 2, 3, 4] 的两边分别填充2个元素，结果为 [2, 1, 1, 2, 3, 4, 4, 3]
Note: This class derived from class str to support json serializable.
"""
CONSTANT: str = "constant"
@ -40,7 +38,7 @@ class BorderType(str, Enum):
REFLECT: str = "reflect"
SYMMETRIC: str = "symmetric"
# 密度函数类型，可能的值为DensityFunction.TPDF, DensityFunction.RPDF, DensityFunction.GPDF.
class DensityFunction(str, Enum):
"""
Density Functions.
@ -48,15 +46,15 @@ class DensityFunction(str, Enum):
Possible enumeration values are: DensityFunction.TPDF, DensityFunction.RPDF,
DensityFunction.GPDF.
- DensityFunction.TPDF: means triangular probability density function.
- DensityFunction.RPDF: means rectangular probability density function.
- DensityFunction.GPDF: means gaussian probability density function.
DensityFunction.TPDF：三角概率密度函数。
DensityFunction.RPDF：矩形概率密度函数。
DensityFunction.GPDF：高斯概率密度函数。
"""
TPDF: str = "TPDF"
RPDF: str = "RPDF"
GPDF: str = "GPDF"
# 淡入淡出形状，可能的值为FadeShape.QUARTER_SINE, FadeShape.HALF_SINE, FadeShape.LINEAR, FadeShape.LOGARITHMIC, FadeShape.EXPONENTIAL.
class FadeShape(str, Enum):
"""
Fade Shapes.
@ -64,11 +62,11 @@ class FadeShape(str, Enum):
Possible enumeration values are: FadeShape.QUARTER_SINE, FadeShape.HALF_SINE, FadeShape.LINEAR,
FadeShape.LOGARITHMIC, FadeShape.EXPONENTIAL.
- FadeShape.QUARTER_SINE: means the fade shape is quarter_sine mode.
- FadeShape.HALF_SINE: means the fade shape is half_sine mode.
- FadeShape.LINEAR: means the fade shape is linear mode.
- FadeShape.LOGARITHMIC: means the fade shape is logarithmic mode.
- FadeShape.EXPONENTIAL: means the fade shape is exponential mode.
FadeShape.QUARTER_SINE：表示淡入淡出形状为四分之一正弦模式。
FadeShape.HALF_SINE：表示淡入形状为半正弦模式。
FadeShape.LINEAR：表示淡入淡出形状为线性模式。
FadeShape.LOGARITHMIC：表示淡入淡出形状为对数模式。
FadeShape.EXPONENTIAL：表示淡入淡出形状为指数模式。
"""
QUARTER_SINE: str = "quarter_sine"
HALF_SINE: str = "half_sine"
@ -76,100 +74,100 @@ class FadeShape(str, Enum):
LOGARITHMIC: str = "logarithmic"
EXPONENTIAL: str = "exponential"
# 增益类型，可能的值为GainType.AMPLITUDE, GainType.POWER, GainType.DB
class GainType(str, Enum):
""""
Gain Types.
Possible enumeration values are: GainType.AMPLITUDE, GainType.POWER, GainType.DB.
- GainType.AMPLITUDE: means input gain type is amplitude.
- GainType.POWER: means input gain type is power.
- GainType.DB: means input gain type is decibel.
GainType.AMPLITUDE - 表示输入增益类型为振幅。
GainType.POWER - 表示输入增益类型为功率。
GainType.DB - 表示输入增益类型为分贝。
"""
AMPLITUDE: str = "amplitude"
POWER: str = "power"
DB: str = "db"
# 音频波形的插值模式，可能的值为Interpolation.LINEAR, Interpolation.QUADRATIC
class Interpolation(str, Enum):
"""
Interpolation Type.
Possible enumeration values are: Interpolation.LINEAR, Interpolation.QUADRATIC.
- Interpolation.LINEAR: means input interpolation type is linear.
- Interpolation.QUADRATIC: means input interpolation type is quadratic.
Interpolation.LINEAR - 插值模式为线性。
Interpolation.QUADRATIC - 插值模式为二次型。
"""
LINEAR: str = "linear"
QUADRATIC: str = "quadratic"
# 梅尔标度实现类型，可能的值为MelType.HTK, MelType.SLANEY
class MelType(str, Enum):
"""
Mel Types.
Possible enumeration values are: MelType.HTK, MelType.SLANEY.
- MelType.NONE: scale the input data with htk.
- MelType.ORTHO: scale the input data with slaney.
MelType.HTK - 隐马尔可夫工具包（HTK）实现，参考 HTK 。
MelType.SLANEY - MATLAB听觉工具包的Slaney实现， 参考 Auditory Toolbox 。
"""
HTK: str = "htk"
SLANEY: str = "slaney"
# 调制类型，可能的值为Modulation.SINUSOIDAL, Modulation.TRIANGULAR
class Modulation(str, Enum):
"""
Modulation Type.
Possible enumeration values are: Modulation.SINUSOIDAL, Modulation.TRIANGULAR.
- Modulation.SINUSOIDAL: means input modulation type is sinusoidal.
- Modulation.TRIANGULAR: means input modulation type is triangular.
Modulation.SINUSOIDAL - 表示输入调制类型为正弦。
Modulation.TRIANGULAR - 表示输入调制类型为三角形。
"""
SINUSOIDAL: str = "sinusoidal"
TRIANGULAR: str = "triangular"
# 标准化模式，可能的值为NormMode.ORTHO, NormMode.NONE
class NormMode(str, Enum):
"""
Norm Types.
Possible enumeration values are: NormMode.ORTHO, NormMode.NONE.
- NormMode.ORTHO: means the mode of input audio is ortho.
- NormMode.NONE: means the mode of input audio is none.
NormMode.ORTHO - 使用正交标准化的DCT基。
NormMode.NONE - 不使用标准化。
"""
ORTHO: str = "ortho"
NONE: str = "none"
# 标准化类型，可能的值为NormType.SLANEY, NormType.NONE
class NormType(str, Enum):
"""
Norm Types.
Possible enumeration values are: NormType.SLANEY, NormType.NONE.
- NormType.SLANEY: norm the input data with slaney.
- NormType.NONE: norm the input data with none.
NormType.SLANEY - 使用面积标准化。
NormType.None - 不使用标准化。
"""
SLANEY: str = "slaney"
NONE: str = "none"
# 音频标度枚举类，可能的值为ScaleType.POWER, ScaleType.MAGNITUDE
class ScaleType(str, Enum):
"""
Scale Types.
Possible enumeration values are: ScaleType.POWER, ScaleType.MAGNITUDE.
- ScaleType.POWER: means the scale of input audio is power.
- ScaleType.MAGNITUDE: means the scale of input audio is magnitude.
ScaleType.MAGNITUDE - 表示输入音频的标度为振幅。
ScaleType.POWER - 表示输入音频的标度为功率。
"""
POWER: str = "power"
MAGNITUDE: str = "magnitude"
# 窗函数类型，可能的值为WindowType.BARTLETT, WindowType.BLACKMAN, WindowType.HAMMING, WindowType.HANN, WindowType.KAISER
class WindowType(str, Enum):
"""
Window Function types,
@ -177,11 +175,11 @@ class WindowType(str, Enum):
Possible enumeration values are: WindowType.BARTLETT, WindowType.BLACKMAN, WindowType.HAMMING, WindowType.HANN,
WindowType.KAISER.
- WindowType.BARTLETT: means the type of window function is Bartlett.
- WindowType.BLACKMAN: means the type of window function is Blackman.
- WindowType.HAMMING: means the type of window function is Hamming.
- WindowType.HANN: means the type of window function is Hann.
- WindowType.KAISER: means the type of window function is Kaiser, currently not supported on macOS.
WindowType.BARTLETT - Bartlettc窗函数。
WindowType.BLACKMAN - Blackman窗函数。
WindowType.HAMMING - Hamming窗函数。
WindowType.HANN - Hann窗函数。
WindowType.KAISER - Kaiser窗函数。当前，不支持在macOS上使用。
"""
BARTLETT: str = "bartlett"
BLACKMAN: str = "blackman"
@ -193,15 +191,15 @@ class WindowType(str, Enum):
DE_C_NORM_MODE = {NormMode.ORTHO: cde.NormMode.DE_NORM_MODE_ORTHO,
NormMode.NONE: cde.NormMode.DE_NORM_MODE_NONE}
# 使用n_mels和n_mfcc创建一个dct矩阵
def create_dct(n_mfcc, n_mels, norm=NormMode.NONE):
"""
Create a DCT transformation matrix with shape (n_mels, n_mfcc), normalized depending on norm.
Args:
n_mfcc (int): Number of mfc coefficients to retain, the value must be greater than 0.
n_mels (int): Number of mel filterbanks, the value must be greater than 0.
norm (NormMode): Normalization mode, can be NormMode.NONE or NormMode.ORTHO (default=NormMode.NONE).
n_mfcc (int): MFCC特征的维度, 参数必须大于0.
n_mels (int): Mel频谱的维度, 参数必须大于0.
norm (NormMode): 归一化模式, 见上。
Returns:
numpy.ndarray, the transformation matrix, to be right-multiplied to row-wise data of size (n_mels, n_mfcc).
@ -212,6 +210,7 @@ def create_dct(n_mfcc, n_mels, norm=NormMode.NONE):
>>> dct = create_dct(100, 200, NormMode.NONE)
"""
# 判断n_mfcc, n_mels, norm的类型是否符合要求，否则抛出异常
if not isinstance(n_mfcc, int):
raise TypeError("n_mfcc with value {0} is not of type {1}, but got {2}.".format(
n_mfcc, int, type(n_mfcc)))
@ -240,13 +239,13 @@ def melscale_fbanks(n_freqs, f_min, f_max, n_mels, sample_rate, norm=NormType.NO
Create a frequency transformation matrix with shape (n_freqs, n_mels).
Args:
n_freqs (int): Number of frequency.
f_min (float): Minimum of frequency in Hz.
f_max (float): Maximum of frequency in Hz.
n_mels (int): Number of mel filterbanks.
sample_rate (int): Sample rate.
norm (NormType, optional): Norm to use, can be NormType.NONE or NormType.SLANEY (Default: NormType.NONE).
mel_type (MelType, optional): Scale to use, can be MelType.HTK or MelType.SLANEY (Default: NormType.SLANEY).
n_freqs (int): 特征频率数量
f_min (float): 特征频率最小值
f_max (float): 特征频率最大值
n_mels (int): 相邻频谱的频率数量
sample_rate (int): 采样率
norm (NormType, optional): 归一化模式, 见上
mel_type (MelType, optional): 梅尔标度实现类型，见上
Returns:
numpy.ndarray, the frequency transformation matrix.
@ -257,6 +256,7 @@ def melscale_fbanks(n_freqs, f_min, f_max, n_mels, sample_rate, norm=NormType.NO
>>> fbanks = melscale_fbanks(n_freqs=4096, f_min=0, f_max=8000, n_mels=40, sample_rate=16000)
"""
# 检查下列参数的形式是否正确
type_check(n_freqs, (int,), "n_freqs")
check_non_negative_int32(n_freqs, "n_freqs")

3
mindspore/python/mindspore/dataset/callback/__init__.py
View File

@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# 回调函数的初始化文件
"""init file for Python callback"""
# 从当前目录下的子目录"./"中导入名为"ds_callback"的模块，并导入该模块中的名为"DSCallback"和"WaitedDSCallback"的类
from .ds_callback import DSCallback, WaitedDSCallback
# __all__是一个特殊属性，用于指定模块中需要导出的类和函数。在这里，将"DSCallback"和"WaitedDSCallback"定义为需要导出的类
__all__ = ["DSCallback", "WaitedDSCallback"]

73
mindspore/python/mindspore/dataset/callback/ds_callback.py
View File

@ -49,6 +49,10 @@ class DSCallback:
@check_callback
def __init__(self, step_size=1):
'''
初始化DSCallback类
:param step_size: 步长
'''
self.step_size = step_size
def ds_begin(self, ds_run_context):
@ -98,31 +102,45 @@ class DSCallback:
Returns:
_c_dataengine.PyDSCallback.
"""
'''
创建运行对象
'''
# 创建PyDSCallback对象
c_cb = PyDSCallback(self.step_size)
at_least_one = False
if self.__class__.ds_begin != DSCallback.ds_begin:
# 如果父类的ds_begin方法不等于DSCallback的ds_begin方法，则设置begin方法
if self.__class__.ds_begin!= DSCallback.ds_begin:
c_cb.set_begin(self.ds_begin)
at_least_one = True
if self.__class__.ds_epoch_begin != DSCallback.ds_epoch_begin:
# 如果父类的ds_epoch_begin方法不等于DSCallback的ds_epoch_begin方法，则设置epoch_begin方法
if self.__class__.ds_epoch_begin!= DSCallback.ds_epoch_begin:
c_cb.set_epoch_begin(self.ds_epoch_begin)
at_least_one = True
if self.__class__.ds_epoch_end != DSCallback.ds_epoch_end:
# 如果父类的ds_epoch_end方法不等于DSCallback的ds_epoch_end方法，则设置epoch_end方法
if self.__class__.ds_epoch_end!= DSCallback.ds_epoch_end:
c_cb.set_epoch_end(self.ds_epoch_end)
at_least_one = True
if self.__class__.ds_step_begin != DSCallback.ds_step_begin:
# 如果父类的ds_step_begin方法不等于DSCallback的ds_step_begin方法，则设置step_begin方法
if self.__class__.ds_step_begin!= DSCallback.ds_step_begin:
c_cb.set_step_begin(self.ds_step_begin)
at_least_one = True
if self.__class__.ds_step_end != DSCallback.ds_step_end:
# 如果父类的ds_step_end方法不等于DSCallback的ds_step_end方法，则设置step_end方法
if self.__class__.ds_step_end!= DSCallback.ds_step_end:
c_cb.set_step_end(self.ds_step_end)
at_least_one = True
# 如果没有至少覆盖一个方法，则抛出异常
if not at_least_one:
raise AttributeError(
"Inheriting Callback class without overriding any methods, check the usage of user defined Callback.")
# 返回PyDSCallback对象
return c_cb
@ -218,15 +236,26 @@ class WaitedDSCallback(Callback, DSCallback):
>>> model.train(2, data, dataset_sink_mode=False, callbacks=[my_cb2, my_cb1])
"""
'''
训练过程中等待DSCallback回调函数
:param step_size: 步长
:param step_event: 步长事件
:param step_run_context: 步长运行上下文
:param epoch_event: epoch事件
:param epoch_run_context: epoch运行上下文
'''
def __init__(self, step_size=1):
super().__init__()
self.step_size = step_size
self.step_event = threading.Event()
self.step_run_context = None
self.epoch_event = threading.Event()
self.epoch_run_context = None
# 是否已经结束
self.training_ended = False
def sync_epoch_begin(self, train_run_context, ds_run_context):
@ -246,7 +275,7 @@ class WaitedDSCallback(Callback, DSCallback):
train_run_context: Include some information of the model with feedback from the previous step.
ds_run_context: Include some information of the data pipeline.
"""
def epoch_end(self, run_context):
"""
Internal method, do not call/override. Defines epoch_end of Callback to release the wait in ds_epoch_begin.
@ -254,9 +283,11 @@ class WaitedDSCallback(Callback, DSCallback):
Args:
run_context: Include some information of the model.
"""
# 将epoch_run_context赋值给self.epoch_run_context
self.epoch_run_context = run_context
# 将epoch_event设置为True
self.epoch_event.set()
def ds_epoch_begin(self, ds_run_context):
"""
Internal method, do not call/override. Define mindspore.dataset.DSCallback.ds_epoch_begin
@ -265,13 +296,20 @@ class WaitedDSCallback(Callback, DSCallback):
Args:
ds_run_context: Include some information of the data pipeline.
"""
# 如果当前epoch数大于1
if ds_run_context.cur_epoch_num > 1:
# 如果没有训练结束
if not self.training_ended:
# 等待epoch_event超时
success = self.epoch_event.wait(timeout=ds.config.get_callback_timeout())
# 清除epoch_event
self.epoch_event.clear()
# 如果超时
if not success:
# 抛出运行时错误
raise RuntimeError(f"ds_epoch_begin timed out after {ds.config.get_callback_timeout()} second(s).")
# by the time this thread wakes up, self.epoch_run_context is already available
# 如果训练结束
self.sync_epoch_begin(self.epoch_run_context, ds_run_context)
def step_end(self, run_context):
@ -281,7 +319,9 @@ class WaitedDSCallback(Callback, DSCallback):
Args:
run_context: Include some information of the model.
"""
# 将run_context赋值给self.step_run_context
self.step_run_context = run_context
# 设置self.step_event为True
self.step_event.set()
def ds_step_begin(self, ds_run_context):
@ -292,13 +332,18 @@ class WaitedDSCallback(Callback, DSCallback):
Args:
ds_run_context: Include some information of the data pipeline.
"""
# 如果当前步数大于step_size，则等待
if ds_run_context.cur_step_num > self.step_size:
# 如果training_ended为False，则等待
if not self.training_ended:
success = self.step_event.wait(timeout=ds.config.get_callback_timeout())
# 清除step_event
self.step_event.clear()
# 如果等待超时，则抛出异常
if not success:
raise RuntimeError(f"ds_step_begin timed out after {ds.config.get_callback_timeout()} second(s).")
# by the time this thread wakes up, self.epoch_run_context is already available
# 如果等待成功，则调用sync_step_begin
self.sync_step_begin(self.step_run_context, ds_run_context)
def create_runtime_obj(self):
@ -308,21 +353,26 @@ class WaitedDSCallback(Callback, DSCallback):
Returns:
_c_dataengine.PyDSCallback.
"""
# 创建一个PyDSCallback对象，用于设置步长
c_cb = PyDSCallback(self.step_size)
at_least_one = False
if self.__class__.sync_step_begin != WaitedDSCallback.sync_step_begin:
# 如果sync_step_begin不等于WaitedDSCallback.sync_step_begin，则设置step_begin
if self.__class__.sync_step_begin!= WaitedDSCallback.sync_step_begin:
c_cb.set_step_begin(self.ds_step_begin)
at_least_one = True
if self.__class__.sync_epoch_begin != WaitedDSCallback.sync_epoch_begin:
# 如果sync_epoch_begin不等于WaitedDSCallback.sync_epoch_begin，则设置epoch_begin
if self.__class__.sync_epoch_begin!= WaitedDSCallback.sync_epoch_begin:
c_cb.set_epoch_begin(self.ds_epoch_begin)
at_least_one = True
# 如果没有至少覆盖一个方法，则抛出异常
if not at_least_one:
raise AttributeError(
"Inheriting Callback class without overriding any methods, check the usage of user defined Callback.")
# 返回PyDSCallback对象
return c_cb
def end(self, run_context):
@ -332,6 +382,9 @@ class WaitedDSCallback(Callback, DSCallback):
Args:
run_context: Include some information of the model.
"""
# 调用epoch_end函数
self.epoch_end(run_context)
# 调用step_end函数
self.step_end(run_context)
# 将training_ended设置为True
self.training_ended = True

16
mindspore/python/mindspore/dataset/engine/__init__.py
View File

@ -13,6 +13,8 @@
# limitations under the License.
# ==============================================================================
# 模块dataset/engine。它提供了一个高性能的数据集引擎，用于加载和处理各种格式的数据集，如ImageNet、TFData、MNIST、Cifar10/100、Manifest、MindRecord等。
# 该引擎支持各种数据处理操作，如乱序、批次、重复、映射和组合等
"""
Introduction to dataset/engine:
@ -22,21 +24,35 @@ high performance and parse data precisely. It also provides the following
operations for users to preprocess data: shuffle, batch, repeat, map, and zip.
"""
# 回调类，用于在数据处理过程中提供通知和控制
from ..callback import DSCallback, WaitedDSCallback
from ..core import config
# 数据缓存客户端，用于缓存数据以提高数据加载速度
from .cache_client import DatasetCache
# 包含各种数据集类的模块，如ImageNet、TFData、MNIST、Cifar10/100、Manifest、MindRecord等
from .datasets import *
# 包含用于处理图像数据集的类
from .datasets_vision import *
# 包含用于处理文本数据集的类
from .datasets_text import *
# 包含用于处理音频数据集的类
from .datasets_audio import *
# 包含用于处理标准数据集格式的类
from .datasets_standard_format import *
# 包含用于处理用户自定义数据集的类
from .datasets_user_defined import *
# Graphdate是用于处理图数据的数据结构,从中导入了用于定义图数据的采样策略，和用于定义图数据输出的格式
from .graphdata import GraphData, SamplingStrategy, OutputFormat
# 用于迭代数据集的类
from .iterators import *
# 用于处理MindRecord数据集的类
from .obs.obs_mindrecord_dataset import *
# 用于从数据集中采样数据的类
from .samplers import *
# 用于数据序列化和反序列化的函数
from .serializer_deserializer import compare, deserialize, serialize, show
# 用于方便地调用这些类
__all__ = ["Caltech101Dataset", # Vision
"Caltech256Dataset", # Vision
"CelebADataset", # Vision

12
mindspore/python/mindspore/dataset/engine/cache_admin.py
View File

@ -24,24 +24,34 @@ import mindspore
def main():
# 主要用于启动缓存服务
"""Entry point for cache service"""
# 获取mindspore的bin目录
cache_admin_dir = os.path.join(os.path.dirname(mindspore.__file__), "bin")
# 设置当前工作目录为mindspore的bin目录
os.chdir(cache_admin_dir)
# 获取mindspore的cache_admin目录
cache_admin = os.path.join(cache_admin_dir, "cache_admin")
# 如果mindspore的cache_admin目录不存在，则抛出异常
if not os.path.exists(cache_admin):
raise RuntimeError("Dataset cache is not supported on your mindspore version.")
# 获取mindspore的cache_server目录
cache_server = os.path.join(cache_admin_dir, "cache_server")
# 设置mindspore的cache_admin目录为可读写，以允许用户操作该目录
os.chmod(cache_admin, stat.S_IRWXU)
# 设置mindspore的cache_server目录为可读写，以允许用户操作该目录
os.chmod(cache_server, stat.S_IRWXU)
# set LD_LIBRARY_PATH for libpython*.so
# 设置LD_LIBRARY_PATH环境变量，以指定libpython*.so库的路径
python_lib_dir = os.path.join(os.path.dirname(mindspore.__file__), "../../..")
os.environ['LD_LIBRARY_PATH'] = python_lib_dir + ":" + os.environ.get('LD_LIBRARY_PATH')
# LD_PRELOAD libnnacl.so
# 设置LD_PRELOAD环境变量，以指定nnacl.so库的路径。
nnacl_lib = os.path.join(os.path.dirname(mindspore.__file__), "lib/libnnacl.so")
os.environ['LD_PRELOAD'] = nnacl_lib
# 运行cache_admin命令，传入命令行参数，并将结果作为退出代码
sys.exit(subprocess.call([cache_admin] + sys.argv[1:], shell=False, env=os.environ))

35
mindspore/python/mindspore/dataset/engine/cache_client.py
View File

@ -53,6 +53,18 @@ class DatasetCache:
def __init__(self, session_id, size=0, spilling=False, hostname=None, port=None, num_connections=None,
prefetch_size=None):
'''
参数：
session_id (整数)：缓存会话ID。
size (整数)：缓存大小，默认为0。
spilling (布尔值)：是否进行溢出处理，默认为False。
hostname (字符串)：缓存服务器的hostname，默认为None。
port (整数)：缓存服务器的端口号，默认为None。
num_connections (整数)：缓存服务器的连接数，默认为None。
prefetch_size (整数)：预取大小，默认为None。
'''
# 检查参数类型和范围。
check_pos_uint32(session_id, "session_id")
type_check(size, (int,), "size")
if size != 0:
@ -76,24 +88,43 @@ class DatasetCache:
self.port = port
self.prefetch_size = prefetch_size
self.num_connections = num_connections
# 创建一个CacheClient对象，用于与缓存服务器进行通信。
self.cache_client = CacheClient(session_id, size, spilling, hostname, port, num_connections, prefetch_size)
def get_stat(self):
"""Get the statistics from a cache."""
'''
这个方法用于获取cache缓存的统计信息，如缓存中数据集的数量、已缓存的数据量等。
'''
return self.cache_client.GetStat()
def __deepcopy__(self, memodict):
'''
这个函数是一个深拷贝函数，用于在内存中深拷贝对象。当对DatasetCache对象进行深拷贝时，会同时复制其所有属性。
'''
# 如果id(self)在memodict字典中，函数直接返回memodict[id(self)]，表示已经存在缓存结果，直接返回。
if id(self) in memodict:
return memodict[id(self)]
cls = self.__class__
# 如果不存在id(self)在memodict字典中，那么创建一个新的类实例new_cache，这个实例的类名是cls。
new_cache = cls.__new__(cls)
# 将self对象的属性复制到new_cache中
memodict[id(self)] = new_cache
# 将self的session_id复制到新的类中
new_cache.session_id = copy.deepcopy(self.session_id, memodict)
# 将self的spilling复制到新的类中
new_cache.spilling = copy.deepcopy(self.spilling, memodict)
# 将self的size复制到新的类中
new_cache.size = copy.deepcopy(self.size, memodict)
# 将self的hostname复制到新的类中
new_cache.hostname = copy.deepcopy(self.hostname, memodict)
# 将self的port复制到新的类中
new_cache.port = copy.deepcopy(self.port, memodict)
# 将self的prefetch_size复制到新的类中
new_cache.prefetch_size = copy.deepcopy(self.prefetch_size, memodict)
# 将self的num_connections复制到新的类中
new_cache.num_connections = copy.deepcopy(self.num_connections, memodict)
# 将self的cache_client复制到新的类中
new_cache.cache_client = self.cache_client
return new_cache
# 这样，在后续遇到相同self对象时，可以直接从字典中获取缓存结果，而不需要重新计算耗时函数调用。
# 返回新的类
return new_cache

1562
mindspore/python/mindspore/dataset/engine/datasets.py
View File

File diff suppressed because it is too large Load Diff

523
mindspore/python/mindspore/dataset/engine/datasets_audio.py
View File

@ -34,33 +34,26 @@ from ..core.validator_helpers import replace_none
class CMUArcticDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses CMUArctic dataset.
用于读取和解析 CMUArctic 数据集。
The generated dataset has four columns: :py:obj:`["waveform", "sample_rate", "transcript", "utterance_id"]`.
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is of a scalar of uint32 type.
The tensor of column :py:obj:`transcript` is of a scalar of string type.
The tensor of column :py:obj:`utterance_id` is of a scalar of string type.
The tensor of column :py:obj:`waveform` 浮点数类型
The tensor of column :py:obj:`sample_rate` 无符号32位整数类型
The tensor of column :py:obj:`transcript` 符串类型
The tensor of column :py:obj:`utterance_id` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
name (str, optional): Part of this dataset, can be 'aew', 'ahw', 'aup', 'awb', 'axb', 'bdl',
'clb', 'eey', 'fem', 'gka', 'jmk', 'ksp', 'ljm', 'lnh', 'rms', 'rxr', 'slp' or 'slt'
(default=None, equal 'aew').
num_samples (int, optional): The number of audio to be included in the dataset
(default=None, will read all audio).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, will use value set in the config).
shuffle (bool, optional): Whether or not to perform shuffle on the dataset
(default=None, expected order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
dataset_dir (str): 指向包含数据集根目录的路径。
name (str, optional): 部分数据集，可以是'aew'、'ahw'、'aup'、'awb'、'axb'、'bdl'、'clb'、'eey'、'fem'、'gka'、
'jmk'、'ksp'、'ljm'、'lnh'、'rms'、'rxr'或'slt'，默认为None，等于'aew'。
num_samples (int, optional): 要包含在数据集中的音频数量（默认为None，表示读取所有音频）。
num_parallel_workers (int, optional): 用于读取数据的worker数量（默认为None，将使用配置文件中的值）。
shuffle (bool, optional): 是否对数据集进行shuffle（默认为None，按照预期顺序处理数据）。
sampler (Sampler, optional): 用于从数据集中选择样本的对象（默认为None，按照预期顺序处理数据）。
num_shards (int, optional): 数据集将被分为的shard数量（默认为None）。当此参数指定时，num_samples表示每个shard中
的最大样本数量。
shard_id (int, optional): shard ID，范围为0到num_shards - 1（默认为None）。当num_shards也指定时，此参数才有效。
cache (DatasetCache, optional): 用于加速数据集处理的张量缓存对象（默认为None，不使用缓存）。
Raises:
RuntimeError: If source raises an exception during execution.
@ -84,24 +77,12 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> cmu_arctic_dataset_directory = "/path/to/cmu_arctic_dataset_directory"
@ -114,12 +95,9 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset):
About CMUArctic dataset:
The CMU arctic databases are designed for the purpose of speech synthesis research.
These single speaker speech databases have been carefully recorded under studio conditions
and consist of approximately 1200 phonetically balanced English utterances. In addition to wavefiles,
the databases provide complete support for the Festival Speech Synthesis System, including pre-built
voices that may be used as is. The entire package is distributed as free software, without restriction
on commercial or non-commercial use.
CMU arctic databases 是一个用于语音合成研究的数据集。这个数据集由 John Kominek 和 Alan W Black 于 2003 年创
建，用于收集和发布预先录制好的单声道语音。此外，还提供了一个完整的支持语音合成系统的预先构建的音色库。该数据集完全
以免费形式提供，不限制商业和非商业使用。
You can construct the following directory structure from CMUArctic dataset and read by MindSpore's API.
@ -171,31 +149,27 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset):
class GTZANDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses GTZAN dataset.
读取和解析GTZAN数据集中的音频文件。
The generated dataset has three columns: :py:obj:`["waveform", "sample_rate", "label"]`.
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is of a scalar of uint32 type.
The tensor of column :py:obj:`label` is of a scalar of string type.
The tensor of column :py:obj:`waveform` 数据类型是float32
The tensor of column :py:obj:`sample_rate` 数据类型是uint32类型的标量
The tensor of column :py:obj:`label` 数据类型是字符串类型的标量
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be 'train', 'valid', 'test' or 'all'
(default=None, all samples).
num_samples (int, optional): The number of audio to be included in the dataset
(default=None, will read all audio).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, will use value set in the config).
shuffle (bool, optional): Whether or not to perform shuffle on the dataset
(default=None, expected order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
dataset_dir (str): 数据集的根目录路径。
usage (str, optional): 数据集的使用情况，可以是"train"（训练集）、"valid"（验证集）、"test"
（测试集）或"all"（所有样本）。
num_samples (int, optional): 要包含在数据集中的音频数量（默认值为None，表示读取所有音频）。
num_parallel_workers (int, optional): 用于读取数据的worker数量（默认值为None，将使用配置文件
中设置的值）。
shuffle (bool, optional): 是否对数据集进行shuffle（默认值为None，按照预期顺序读取数据）。
sampler (Sampler, optional): 用于从数据集中选择样本的对象（默认值为None，按照预期顺序读取数据）。
num_shards (int, optional): 将数据集划分为的shard数量（默认值为None）。当此参数指定时，
num_samples表示每个shard中的最大样本数量。
shard_id (int, optional): 当前shard的ID（默认值为None），当num_shards也被指定时，此参数必须同时
指定。
cache (DatasetCache, optional): 用于加速数据集处理的张量缓存对象（默认值为None，表示不使用缓存）。
Raises:
RuntimeError: If source raises an exception during execution.
@ -219,24 +193,12 @@ class GTZANDataset(MappableDataset, AudioBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> gtzan_dataset_directory = "/path/to/gtzan_dataset_directory"
@ -249,11 +211,9 @@ class GTZANDataset(MappableDataset, AudioBaseDataset):
About GTZAN dataset:
The GTZAN dataset appears in at least 100 published works and is the most commonly used
public dataset for evaluation in machine listening research for music genre recognition.
It consists of 1000 audio tracks, each of which is 30 seconds long. It contains 10 genres (blues,
classical, country, disco, hiphop, jazz, metal, pop, reggae and reggae), each of which is
represented by 100 tracks. The tracks are all 22050Hz Mono 16-bit audio files in .wav format.
GTZAN数据集在至少100篇论文中出现，是最常用的用于机器听觉研究的音乐类别识别评估数据集。
GTZAN数据集包含1000个音频文件，每个文件长度为30秒。它包含10个类别（蓝调、古典、乡村、迪斯科、爵士、金属、流
行、 Reggae 和 Reggae），每个类别都有100个音轨。音频文件是22050Hz Mono 16-bit音频文件，格式为.wav。
You can construct the following directory structure from GTZAN dataset and read by MindSpore's API.
@ -301,36 +261,33 @@ class GTZANDataset(MappableDataset, AudioBaseDataset):
class LibriTTSDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses the LibriTTS dataset.
用于读取和处理LibriTTS数据集。
The generated dataset has seven columns :py:obj:`['waveform', 'sample_rate', 'original_text', 'normalized_text',
'speaker_id', 'chapter_id', 'utterance_id']`.
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is of a scalar of uint32 type.
The tensor of column :py:obj:`original_text` is of a scalar of string type.
The tensor of column :py:obj:`normalized_text` is of a scalar of string type.
The tensor of column :py:obj:`speaker_id` is of a scalar of uint32 type.
The tensor of column :py:obj:`chapter_id` is of a scalar of uint32 type.
The tensor of column :py:obj:`utterance_id` is of a scalar of string type.
The tensor of column :py:obj:`waveform` 浮点数类型
The tensor of column :py:obj:`sample_rate` 无符号整数（uint32）
The tensor of column :py:obj:`original_text` 字符串类型
The tensor of column :py:obj:`normalized_text` 字符串类型
The tensor of column :py:obj:`speaker_id` 无符号整数（uint32）
The tensor of column :py:obj:`chapter_id` 无符号整数（uint32）
The tensor of column :py:obj:`utterance_id` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Part of this dataset, can be 'dev-clean', 'dev-other', 'test-clean', 'test-other',
'train-clean-100', 'train-clean-360', 'train-other-500', or 'all' (default=None, equal 'all').
num_samples (int, optional): The number of images to be included in the dataset
(default=None, will read all audio).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, will use value set in the config).
shuffle (bool, optional): Whether or not to perform shuffle on the dataset
(default=None, expected order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
dataset_dir (str): 指向包含数据集根目录的路径字符串。
usage (str, optional): 一个可选参数，表示要使用的数据集部分，可以是 'dev-clean'、'dev-other'、'test-clean'、
'test-other'、'train-clean-100'、'train-clean-360'、'train-other-500' 或 'all'（默认值为 None，等于 'all'）。
num_samples (int, optional): 一个可选参数，表示要包含在数据集中的音频数量（默认值为 None，表示读取所有音频）。
num_parallel_workers (int, optional): 一个可选参数，表示读取数据的 worker 数量（默认值为 None，表示使用配置文
件中设置的值）。
shuffle (bool, optional): 一个可选参数，表示是否对数据进行 shuffle（默认值为 None，表示按照预期顺序读取数据）。
sampler (Sampler, optional): 一个可选参数，表示用于从数据集中选择样本的对象（默认值为 None，表示按照预期顺序读
取数据）。
num_shards (int, optional): 一个可选参数，表示数据集将被分为的 shard 数量（默认值为 None）。当此参数指定时，
num_samples 表示每个 shard 中的最大样本数量。
shard_id (int, optional): 一个可选参数，表示当前 shard 的 ID（默认值为 None），当 num_shards 也指定时，此参数只
能被指定。
cache (DatasetCache, optional): 一个可选参数，用于加速数据集处理的可选张量缓存对象（默认值为 None，表示不使用缓存）。
Raises:
RuntimeError: If source raises an exception during execution.
@ -351,27 +308,12 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset):
:widths: 25 25 50
:header-rows: 1
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> libri_tts_dataset_dir = "/path/to/libri_tts_dataset_directory"
@ -384,10 +326,9 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset):
About LibriTTS dataset:
LibriTTS is a multi-speaker English corpus of approximately 585 hours of read English speech at 24kHz
sampling rate, prepared by Heiga Zen with the assistance of Google Speech and Google Brain team members.
The LibriTTS corpus is designed for TTS research. It is derived from the original materials (mp3 audio
files from LibriVox and text files from Project Gutenberg) of the LibriSpeech corpus.
这段代码描述了 LibriTTS 数据集，一个多说话者英语语料库，包含大约 585 小时以 24kHz 采样率读取的读音文。prepared
by Heiga Zen with the assistance of Google Speech 和 Google Brain 团队成员。LibriTTS 数据集设计用于 TTS 研
究。它是从原始资料（LibriVox 中的 mp3 音频文件和 Project Gutenberg 中的文本文件）中提取的。
You can construct the following directory structure from LibriTTS dataset and read by MindSpore's API.
@ -451,31 +392,29 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset):
class LJSpeechDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses LJSpeech dataset.
用于读取和解析 LJSpeech 数据集。
The generated dataset has four columns :py:obj:`[waveform, sample_rate, transcription, normalized_transcript]`.
The tensor of column :py:obj:`waveform` is a tensor of the float32 type.
The tensor of column :py:obj:`sample_rate` is a scalar of the int32 type.
The tensor of column :py:obj:`transcription` is a scalar of the string type.
The tensor of column :py:obj:`normalized_transcript` is a scalar of the string type.
The tensor of column :py:obj:`waveform` 浮点数类型
The tensor of column :py:obj:`sample_rate` 无符号整数（uint32）
The tensor of column :py:obj:`transcription` 字符串类型
The tensor of column :py:obj:`normalized_transcript` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
num_samples (int, optional): The number of audios to be included in the dataset
(default=None, all audios).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (bool, optional): Whether to perform shuffle on the dataset (default=None, expected
order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided
into (default=None). When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
dataset_dir (str): 指向包含数据集根目录的路径字符串。
num_samples (int, optional): 可选参数，表示要包含在数据集中的音频数量（默认值为 None，表示所有音频）。
num_parallel_workers (int, optional): 可选参数，表示用于读取数据的工人数（默认值为 None，表示根据配
置文件设置的工人数）。
shuffle (bool, optional): 可选参数，表示是否对数据进行洗牌（默认值为 None，表示根据预期顺序进行顺序读
取）。
sampler (Sampler, optional): 可选参数，表示用于从数据集中选择样本的对象（默认值为 None，表示不使
用采样器）。
num_shards (int, optional): 可选参数，表示将数据集分为的 shard 数量（默认值为 None）。当此参数指定时，
num_samples 表示每个 shard 中的最大样本数量。
shard_id (int, optional): 可选参数，表示 shard 的 ID（默认值为 None）。当 num_shards 也指定时，此参
数只能用于指定 shard ID。
cache (DatasetCache, optional): 可选参数，表示使用张量缓存服务加速数据处理（默认值为 None，表示不使用缓
存）。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -497,24 +436,12 @@ class LJSpeechDataset(MappableDataset, AudioBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> lj_speech_dataset_dir = "/path/to/lj_speech_dataset_directory"
@ -533,12 +460,11 @@ class LJSpeechDataset(MappableDataset, AudioBaseDataset):
About LJSPEECH dataset:
This is a public domain speech dataset consisting of 13,100 short audio clips of a single speaker
reading passages from 7 non-fiction books. A transcription is provided for each clip.
Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.
The texts were published between 1884 and 1964, and are in the public domain.
The audio was recorded in 2016-17 by the LibriVox project and is also in the public domain.
数据集包含 13100 个短音频片段，由一个 single 说话者阅读 7 本书中的文本。对于每个片段，提供了一个文本翻译。
音频片段的 lengths 范围从 1 到 10 秒，总长度approximately 24 小时。
文本内容随时间变化，公开domain。
音频片段由 The LibriVox 项目在 2016-17 年记录，也属于公开domain。
Here is the original LJSPEECH dataset structure.
You can unzip the dataset files into the following directory structure and read by MindSpore's API.
@ -587,34 +513,26 @@ class LJSpeechDataset(MappableDataset, AudioBaseDataset):
class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses the SpeechCommands dataset.
用于读取和解析 SpeechCommands 数据集。
The generated dataset has five columns :py:obj:`[waveform, sample_rate, label, speaker_id, utterance_number]`.
The tensor of column :py:obj:`waveform` is a vector of the float32 type.
The tensor of column :py:obj:`sample_rate` is a scalar of the int32 type.
The tensor of column :py:obj:`label` is a scalar of the string type.
The tensor of column :py:obj:`speaker_id` is a scalar of the string type.
The tensor of column :py:obj:`utterance_number` is a scalar of the int32 type.
The tensor of column :py:obj:`waveform` 浮点型数据
The tensor of column :py:obj:`sample_rate` 无符号整数（uint32）
The tensor of column :py:obj:`label` 字符串类型
The tensor of column :py:obj:`speaker_id` 字符串类型
The tensor of column :py:obj:`utterance_number` 无符号整数（uint32）
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be 'train', 'test', 'valid' or 'all'. 'train'
will read from 84,843 samples, 'test' will read from 11,005 samples, 'valid' will read from 9,981
test samples and 'all' will read from all 105,829 samples (default=None, will read all samples).
num_samples (int, optional): The number of samples to be included in the dataset
(default=None, will read all samples).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, will use value set in the config).
shuffle (bool, optional): Whether or not to perform shuffle on the dataset
(default=None, expected order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the dataset
(default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This argument can only be specified
when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
dataset_dir (str): SpeechCommands 数据集的根目录。
usage (str, optional): 用于指定要读取的数据集部分，可以是 'train'、'test'、'valid' 或 'all'。
num_samples (int, optional): 表示要包含在 dataset 中的样本数量，默认为 None，表示读取所有样本。
num_parallel_workers (int, optional): 表示用于读取数据的 worker 数量，默认为 None，表示使用配置中的值。
shuffle (bool, optional): 表示是否对 dataset 进行 shuffle，默认为 None，表示按照输入顺序读取。
sampler (Sampler, optional): 表示用于从 dataset 中选择样本的 sampler 对象，默认为 None，表示不使用 sampler。
num_shards (int, optional): N表示将 dataset 分为的 shard 数量，默认为 None。当这个参数被指定时，num_samples
表示每个 shard 中的最大样本数量。
shard_id (int, optional): 表示当前 shard 的 ID，当 num_shards 也指定时，这个参数必须被指定。
cache (DatasetCache, optional): 表示用于加速 dataset 处理的 tensor 缓存对象，默认为 None，表示不使用缓存。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -636,24 +554,12 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> speech_commands_dataset_dir = "/path/to/speech_commands_dataset_directory"
@ -666,8 +572,7 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
About SpeechCommands dataset:
The SpeechCommands is database for limited_vocabulary speech recognition, containing 105,829 audio samples of
'.wav' format.
SpeechCommands 是一个用于有限词汇语音识别的数据库，包含 105,829 个 .wav 格式的音频样本。
Here is the original SpeechCommands dataset structure.
You can unzip the dataset files into this directory structure and read by MindSpore's API.
@ -712,46 +617,33 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
class TedliumDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses Tedlium dataset.
The columns of generated dataset depend on the source SPH files and the corresponding STM files.
用于读取并解析 Tedlium 数据集。
生成的数据集的列取决于源 SPH 文件和对应的 STM 文件。
The generated dataset has six columns :py:obj:`[waveform, sample_rate, transcript, talk_id, speaker_id,
identifier]`.
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is a scalar of the int32 type.
The tensor of column :py:obj:`transcript` is a scalar of the string type.
The tensor of column :py:obj:`talk_id` is a scalar of the string type.
The tensor of column :py:obj:`speaker_id` is a scalar of the string type.
The tensor of column :py:obj:`identifier` is a scalar of the string type.
The tensor of column :py:obj:`waveform` 浮点型数据
The tensor of column :py:obj:`sample_rate` 无符号整数（uint32）
The tensor of column :py:obj:`transcript` 无符号整数（uint32）
The tensor of column :py:obj:`talk_id` 无符号整数（uint32）
The tensor of column :py:obj:`speaker_id` 无符号整数（uint32）
The tensor of column :py:obj:`identifier` 无符号整数（uint32）
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
release (str): Release of the dataset, can be 'release1', 'release2', 'release3'.
usage (str, optional): Usage of this dataset.
For release1 or release2, can be 'train', 'test', 'dev' or 'all'.
'train' will read from train samples,
'test' will read from test samples,
'dev' will read from dev samples,
'all' will read from all samples.
For release3, can only be 'all', it will read from data samples (default=None, all samples).
extensions (str): Extensions of the SPH files, only '.sph' is valid.
(default=None, ".sph").
num_samples (int, optional): The number of audio samples to be included in the dataset
(default=None, all samples).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (bool, optional): Whether to perform shuffle on the dataset (default=None, expected
order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided
into (default=None). When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
dataset_dir (str): 表示数据集的根目录路径。
release (str): 表示数据集的版本，可以是 'release1'、'release2'、'release3' 等。
usage (str, optional): 表示要使用的数据集类型。
对于 'release1' 或 'release2'，可以是 'train'、'test'、'dev' 或 'all'。
对于 'release3'，只能为 'all'。
extensions (str): 表示 SPH 文件的扩展名，只支持 '.sph'。
num_samples (int, optional): 表示要包含在数据集中的音频样本数量。
num_parallel_workers (int, optional): 表示用于读取数据的worker数量。
shuffle (bool, optional): 表示是否对数据集进行shuffle。
sampler (Sampler, optional): 用于从数据集中选择样本的对象。
num_shards (int, optional): 表示数据集将被分为的shard数量。当此参数指定时，num_samples 表示每个shard中的最大样本数量。
shard_id (int, optional): 表示当前shard的ID，当num_shards也被指定时，此参数才有效。
cache (DatasetCache, optional): 用于加速数据集处理的张量缓存对象。
Raises:
RuntimeError: If `dataset_dir` does not contain stm files.
@ -773,24 +665,12 @@ class TedliumDataset(MappableDataset, AudioBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> # 1) Get all train samples from TEDLIUM_release1 dataset in sequence.
@ -810,22 +690,18 @@ class TedliumDataset(MappableDataset, AudioBaseDataset):
About TEDLIUM_release1 dataset:
The TED-LIUM corpus is English-language TED talks, with transcriptions, sampled at 16kHz.
It contains about 118 hours of speech.
TED-LIUM 语料库是一个英文的 TED talk 语料库，采样率为 16kHz。它包含大约 118 小时的语言数据。
About TEDLIUM_release2 dataset:
This is the TED-LIUM corpus release 2, licensed under Creative Commons BY-NC-ND 3.0. All talks and text are
property of TED Conferences LLC. The TED-LIUM corpus was made from audio talks and their transcriptions available
on the TED website. We have prepared and filtered these data in order to train acoustic models to participate to
the International Workshop on Spoken Language Translation 2011 (the LIUM English/French SLT system reached the
first rank in the SLT task).
这是 TED-LIUM 语料库的第二个版本，根据 Creative Commons BY-NC-ND 3.0 许可发布。所有talk和文本属 TED Conferences LLC
所有。TED-LIUM 语料库是从 TED 网站上的音频talk和它们的翻译文本中准备和过滤而来的数据。我们已准备并过滤了这些数据，以便训
练参与国际语音翻译比赛（2011 年国际语音翻译比赛第一名的 LIUM 英法双语系统）。
About TEDLIUM_release-3 dataset:
This is the TED-LIUM corpus release 3, licensed under Creative Commons BY-NC-ND 3.0. All talks and text are
property of TED Conferences LLC. This new TED-LIUM release was made through a collaboration between the Ubiqus
company and the LIUM (University of Le Mans, France).
这是 TED-LIUM 语料库的第三个版本，根据 Creative Commons BY-NC-ND 3.0 许可发布。所有talk和文本属 TED Conferences LLC
所有。这个新的 TED-LIUM 发布是通过与 Ubiqus 公司和 LIUM（法属勒马大学）之间的协作开发的。
You can unzip the dataset files into the following directory structure and read by MindSpore's API.
@ -924,30 +800,24 @@ class TedliumDataset(MappableDataset, AudioBaseDataset):
class YesNoDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses the YesNo dataset.
用于读取并解析 YesNo 数据集，生成相应的数据集。
The generated dataset has three columns :py:obj:`[waveform, sample_rate, labels]`.
The tensor of column :py:obj:`waveform` is a vector of the float32 type.
The tensor of column :py:obj:`sample_rate` is a scalar of the int32 type.
The tensor of column :py:obj:`labels` is a scalar of the int32 type.
The tensor of column :py:obj:`waveform` 浮点型数据
The tensor of column :py:obj:`sample_rate` 无符号整数（uint32）
The tensor of column :py:obj:`labels` 无符号整数（uint32）
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
num_samples (int, optional): The number of images to be included in the dataset
(default=None, will read all images).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, will use value set in the config).
shuffle (bool, optional): Whether or not to perform shuffle on the dataset
(default=None, expected order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This argument can only
be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
dataset_dir (str): 指向数据集根目录的路径。这个目录应该包含数据集的图像或其他文件。
num_samples (int, optional): 要包含在数据集中的图像数量（默认为None，表示读取所有图像）。
num_parallel_workers (int, optional): 用于读取数据的worker数量（默认为None，表示使用配置文件中的值）。
shuffle (bool, optional): 是否对数据进行shuffle（默认为None，表示按照预期顺序读取数据，即不shuffle）。
sampler (Sampler, optional): 用于从数据集中选择样本的对象（默认为None，表示使用配置文件中的值）。
num_shards (int, optional): 将数据集划分为shard的数量（默认为None）。当这个参数被指定时，num_samples表
示每个shard中的最大样本数量。
shard_id (int, optional): 在num_shards中的shard ID（默认为None）。当这个参数也被指定时，shard_id必须
小于num_shards。
cache (DatasetCache, optional): 用于加速数据集处理的张量缓存对象（默认为None，表示不使用缓存）。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
ValueError: If `num_parallel_workers` exceeds the max thread numbers.
@ -968,24 +838,12 @@ class YesNoDataset(MappableDataset, AudioBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> yes_no_dataset_dir = "/path/to/yes_no_dataset_directory"
@ -997,9 +855,8 @@ class YesNoDataset(MappableDataset, AudioBaseDataset):
About YesNo dataset:
Yesno is an audio dataset consisting of 60 recordings of one individual saying yes or no in Hebrew; each
recording is eight words long. It was created for the Kaldi audio project by an author who wishes to
remain anonymous.
这个数据集包含60个录音，每个录音由一个个人 saying yes或no 组成，每个录音长度为8个单词。这个数据集是为了
Kaldi音频项目而创建的，由一个匿名作者创建。
Here is the original YesNo dataset structure.
You can unzip the dataset files into this directory structure and read by MindSpore's API.

256
mindspore/python/mindspore/dataset/engine/datasets_standard_format.py
View File

@ -40,40 +40,33 @@ from . import samplers
class CSVDataset(SourceDataset, UnionBaseDataset):
"""
A source dataset that reads and parses comma-separated values
用于从CSV文件中读取数据并将其作为数据集。
`(CSV) <http://en.volupedia.org/wiki/Comma-separated_values>`_ files as dataset.
The columns of generated dataset depend on the source CSV files.
Args:
dataset_files (Union[str, list[str]]): String or list of files to be read or glob strings to search
for a pattern of files. The list will be sorted in a lexicographical order.
field_delim (str, optional): A string that indicates the char delimiter to separate fields (default=',').
column_defaults (list, optional): List of default values for the CSV field (default=None). Each item
in the list is either a valid type (float, int, or string). If this is not provided, treats all
columns as string type.
column_names (list[str], optional): List of column names of the dataset (default=None). If this
is not provided, infers the column_names from the first row of CSV file.
num_samples (int, optional): The number of samples to be included in the dataset
(default=None, will include all images).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_files (Union[str, list[str]]): 一个字符串或列表，表示要读取或搜索的CSV文件。如果这是一个列表，它将
按字母顺序进行排序。
field_delim (str, optional): 一个字符串，表示用于分隔CSV文件中的字段的分隔符（默认值为','）。
column_defaults (list, optional): 一个列表，表示CSV文件中字段的默认值。每个项都是一个有效的类型（浮点数、整
数或字符串）。如果没有提供，则将所有列视为字符串类型。
column_names (list[str], optional): 一个列表，表示CSV文件中列的名称。如果没有提供，将根据第一个行的内容推断
列名称。
num_samples (int, optional): 一个整数，表示要包含在数据集中的样本数量。如果没有提供，将包含所有样本。
num_parallel_workers (int, optional): 一个整数，表示用于读取数据的 worker 数量。如果没有提供，将使用配置中
的最大线程数。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
有以下三种选项：
Shuffle.GLOBAL：既洗牌文件又洗牌样本。
Shuffle.FILES：只洗牌文件。
Shuffle.NONE：不洗牌。
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
num_shards (int, optional): 一个整数，表示将数据集分为多少个分片。如果没有提供，则不进行分片。
shard_id (int, optional): 一个整数，表示当前分片的ID。如果提供了num_shards，则必须提供shard_id。
cache (DatasetCache, optional): 一个个DatasetCache对象，用于加速数据处理。如果没有提供，则不使用缓存。
Raises:
RuntimeError: If dataset_files are not valid or do not exist.
@ -106,45 +99,35 @@ class CSVDataset(SourceDataset, UnionBaseDataset):
class MindDataset(MappableDataset, UnionBaseDataset):
"""
A source dataset that reads and parses MindRecord dataset.
用于从MindRecord文件中读取数据并将其作为数据集。
The columns of generated dataset depend on the source MindRecord files.
Args:
dataset_files (Union[str, list[str]]): If dataset_file is a str, it represents for
a file name of one component of a mindrecord source, other files with identical source
in the same path will be found and loaded automatically. If dataset_file is a list,
it represents for a list of dataset files to be read directly.
columns_list (list[str], optional): List of columns to be read (default=None).
num_parallel_workers (int, optional): The number of readers (default=None).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
dataset_files (Union[str, list[str]]): 一个字符串或列表，表示要读取或搜索的MindRecord文件。如果这
是一个列表，它将按字母顺序进行排序。
columns_list (list[str], optional): 一个列表，表示要读取的列名。如果没有提供，将根据第一个行的内容推断列名称。
num_parallel_workers (int, optional): 一个整数，表示用于读取数据的 worker 数量。如果没有提供，将使用配置中的
最大线程数。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Global shuffle of all rows of data in dataset, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
- Shuffle.FILES: Shuffle the file sequence but keep the order of data within each file.
- Shuffle.INFILE: Keep the file sequence the same but shuffle the data within each file.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, 'num_samples' reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, sampler is exclusive
with shuffle and block_reader). Support list: SubsetRandomSampler,
PkSampler, RandomSampler, SequentialSampler, DistributedSampler.
padded_sample (dict, optional): Samples will be appended to dataset, where
keys are the same as column_list.
num_padded (int, optional): Number of padding samples. Dataset size
plus num_padded should be divisible by num_shards.
num_samples (int, optional): The number of samples to be included in the dataset
(default=None, all samples).
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
num_shards (int, optional): 一个整数，表示将数据集分为多少个分片。当这个参数指定时，num_samples表示每个分片的
最大样本数。
shard_id (int, optional): 一个整数，表示当前分片的ID。如果提供了num_shards，则必须提供shard_id。
sampler (Sampler, optional): 一个Sampler对象，用于选择样本。sampler和shuffle是互斥的，如果提供了sampler，则
shuffle参数将被忽略。
padded_sample (dict, optional): 一个字典，用于填充样本，其中键是列名，值是填充的值。
num_padded (int, optional): 一个整数，表示要填充的样本数量。数据集大小加上num_padded应该可以被num_shards整除。
num_samples (int, optional): 一个整数，表示要包含在数据集中的样本数量。如果没有提供，将包含所有样本。
cache (DatasetCache, optional): 个DatasetCache对象，用于加速数据处理。如果没有提供，则不使用缓存。
Raises:
ValueError: If dataset_files are not valid or do not exist.
@ -164,24 +147,12 @@ class MindDataset(MappableDataset, UnionBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> mind_dataset_dir = ["/path/to/mind_dataset_file"] # contains 1 or multiple MindRecord files
@ -197,7 +168,9 @@ class MindDataset(MappableDataset, UnionBaseDataset):
shard_id=None, sampler=None, padded_sample=None, num_padded=None, num_samples=None, cache=None):
super().__init__(num_parallel_workers=num_parallel_workers, sampler=sampler, num_samples=num_samples,
shuffle=shuffle_to_bool(shuffle), num_shards=num_shards, shard_id=shard_id, cache=cache)
# 检查num_samples和shuffle是否同时指定
if num_samples and shuffle in (Shuffle.FILES, Shuffle.INFILE):
# 指定了Shuffle.FILES或Shuffle.INFILE策略和num_samples，则会引发一个错误
raise ValueError("'Shuffle.FILES' or 'Shuffle.INFILE' and 'num_samples' "
"cannot be specified at the same time.")
self.shuffle_option = shuffle
@ -206,29 +179,37 @@ class MindDataset(MappableDataset, UnionBaseDataset):
self.load_dataset = False
self.dataset_files = dataset_files
# 检查操作系统是否为"Windows"
if platform.system().lower() == "windows":
if isinstance(dataset_files, list):
file_tuple = []
for item in dataset_files:
# 对dataset_files中的路径进行替换（将"\"替换为"/"）
item.replace("\\", "/")
file_tuple.append(item)
self.dataset_files = file_tuple
else:
self.dataset_files = dataset_files.replace("\\", "/")
# 无论dataset_files是一个列表还是一个字符串，它都会被处理为一个包含正确路径的字符串列表。
self.columns_list = replace_none(columns_list, [])
if sampler is not None:
# 检查sampler是否是一个支持抽样的抽样器对象
if isinstance(sampler, (
samplers.SubsetRandomSampler, samplers.SubsetSampler, samplers.PKSampler,
samplers.DistributedSampler,
samplers.RandomSampler, samplers.SequentialSampler)) is False:
# 抛出一个错误
raise ValueError("The sampler is not supported yet.")
#初始化一个名为padded_sample的字典，用于存储填充后的样本
self.padded_sample = padded_sample
self.num_padded = replace_none(num_padded, 0)
# 初始化一个名为new_padded_sample的字典，用于存储填充后的样本
self.new_padded_sample = {}
# 如果padded_sample不为空，则将padded_sample中的键值对转换为字节串，并将其添加到new_padded_sample中。
if padded_sample:
for k, v in padded_sample.items():
if isinstance(v, np.ndarray):
@ -239,43 +220,34 @@ class MindDataset(MappableDataset, UnionBaseDataset):
class TFRecordDataset(SourceDataset, UnionBaseDataset):
"""
A source dataset that reads and parses datasets stored on disk in TFData format.
用于读取和解析存储在磁盘上的TFData格式数据集。
The columns of generated dataset depend on the source TFRecord files.
Args:
dataset_files (Union[str, list[str]]): String or list of files to be read or glob strings to search for a
pattern of files. The list will be sorted in a lexicographical order.
schema (Union[str, Schema], optional): Path to the JSON schema file or schema object (default=None).
If the schema is not provided, the meta data from the TFData file is considered the schema.
columns_list (list[str], optional): List of columns to be read (default=None, read all columns).
num_samples (int, optional): The number of samples (rows) to be included in the dataset (default=None).
If num_samples is None and numRows(parsed from schema) does not exist, read the full dataset;
If num_samples is None and numRows(parsed from schema) is greater than 0, read numRows rows;
If both num_samples and numRows(parsed from schema) are greater than 0, read num_samples rows.
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_files (Union[str, list[str]]): 一个字符串或一个包含字符串的列表，表示要读取或搜索的TFRecord文件。如
果列表包含多个文件，它们将按字典顺序排序。
schema (Union[str, Schema], optional): 一个字符串或一个Schema对象，表示要使用的JSONschema文件或schema对象。
如果没有提供schema，则从TFData文件中使用元数据作为schema。
columns_list (list[str], optional): 一个字符串列表，表示要读取的列名。如果没有提供，将读取所有列。
num_samples (int, optional): 一个整数，表示要包含在数据集中的样本数量。如果没有提供，或者提供的数量为0，将读取
整个数据集。如果提供了num_samples且numRows（从schema中解析）大于0，将读取num_samples个样本。
num_parallel_workers (int, optional): 一个整数，表示用于读取数据的worker数量。如果没有提供，将使用配置中的最
大线程数。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided
into (default=None). When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
shard_equal_rows (bool, optional): Get equal rows for all shards(default=False). If shard_equal_rows
is false, number of rows of each shard may be not equal, and may lead to a failure in distributed training.
When the number of samples of per TFRecord file are not equal, it is suggested to set to true.
This argument should only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
num_shards (int, optional): 一个整数，表示将数据集分为的shard数量。如果提供了num_shards，且没有提供shard_id，
将引发一个RuntimeError。
shard_id (int, optional): T一个整数，表示当前shard的ID。如果提供了shard_id，且没有提供num_shards，将引发一个RuntimeError。
shard_equal_rows (bool, optional): 一个布尔值，表示每个shard是否具有相等的行数。如果设置为False，可能会导致在分
布式训练中的失败。当num_shards不等于1时，建议设置为True。
cache (DatasetCache, optional): 个DatasetCache对象，用于加速数据处理。如果没有提供，则不使用缓存。
Raises:
ValueError: If dataset_files are not valid or do not exist.
@ -327,39 +299,32 @@ class TFRecordDataset(SourceDataset, UnionBaseDataset):
class OBSMindDataset(GeneratorDataset):
"""
A source dataset that reads and parses MindRecord dataset which stored in OBS.
用于从OBS中读取和解析MindRecord数据集。
The columns of generated dataset depend on the source MindRecord files.
Args:
dataset_files (list[str]): List of files in OBS to be read and file path is in
the format of s3://.
server (str): Endpoint for accessing OBS. For example: <https://your-endpoint:9000>.
ak (str): Access key ID of OBS.
sk (str): Secret key ID of OBS.
sync_obs_path (str): OBS dir path used for synchronization, users need to
create it on OBS in advance. Path is in the format of s3://.
columns_list (list[str], optional): List of columns to be read (default=None, read all columns).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
dataset_files (list[str]): 一个字符串列表，表示要读取和搜索的MindRecord文件。每个文件路径都用s3://开头。
server (str): OBS的终端地址，例如：https://your-endpoint:9000。
ak (str): OBS的访问密钥ID。
sk (str): OBS的密钥ID。
sync_obs_path (str): OBS同步目录路径，用于同步数据。路径用s3://开头。
columns_list (list[str], optional): 一个字符串列表，表示要读取的列名。如果没有提供，将读取所有列。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Global shuffle of all rows of data in dataset, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
- Shuffle.FILES: Shuffle the file sequence but keep the order of data within each file.
- Shuffle.INFILE: Keep the file sequence the same but shuffle the data within each file.
num_shards (int, optional): Number of shards that the dataset will be divided
into (default=None).
shard_id (int, optional): The shard ID within num_shards (default=None). This
argument can only be specified when num_shards is also specified.
shard_equal_rows (bool, optional): Get equal rows for all shards(default=True). If shard_equal_rows
is false, number of rows of each shard may be not equal, and may lead to a failure in distributed training.
When the number of samples of per MindRecord file are not equal, it is suggested to set to true.
This argument should only be specified when num_shards is also specified.
num_shards (int, optional): 一个整数，表示将数据集分为的shard数量。如果提供了num_shards，且没有提供shard_id，
将引发一个RuntimeError。
shard_id (int, optional): T一个整数，表示当前shard的ID。如果提供了shard_id，且没有提供num_shards，
将引发一个RuntimeError。
shard_equal_rows (bool, optional): 一个布尔值，表示每个shard是否具有相等的行数。如果设置为True，可能会导致在
分布式训练中的失败。当num_shards不等于1时，建议设置为True。
Raises:
RuntimeError: If `sync_obs_path` do not exist.
@ -394,35 +359,48 @@ class OBSMindDataset(GeneratorDataset):
shard_id=None,
shard_equal_rows=True):
# 从config_loader模块中导入config对象
from .obs.config_loader import config
# 对config对象的AK、SK、SERVER和SYNC_OBS_PATH属性进行赋值
config.AK = ak
config.SK = sk
config.SERVER = server
config.SYNC_OBS_PATH = sync_obs_path
# 判断shuffle参数的类型是否为bool或Shuffle枚举值
if shuffle is not None and not isinstance(shuffle, (bool, Shuffle)):
# 抛出一个TypeError异常
raise TypeError("shuffle must be of boolean or enum of 'Shuffle' values like 'Shuffle.GLOBAL' or "
"'Shuffle.FILES'.")
# 将num_shards、shard_id和shuffle参数替换为None的值，以确保它们是整数类型
self.num_shards = replace_none(num_shards, 1)
self.shard_id = replace_none(shard_id, 0)
self.shuffle = replace_none(shuffle, True)
# 创建一个MindRecordFromOBS实例
dataset = MindRecordFromOBS(dataset_files, columns_list, shuffle, self.num_shards, self.shard_id,
shard_equal_rows, config.DATASET_LOCAL_PATH)
# 判断columns_list参数是否为None，如果为None，则将dataset.get_col_names()赋值给columns_
if not columns_list:
columns_list = dataset.get_col_names()
else:
# 使用dataset.get_col_names()方法获取数据集中的所有列名
full_columns_list = dataset.get_col_names()
# 判断columns_list是否是full_columns_list的子集
if not set(columns_list).issubset(full_columns_list):
raise ValueError("columns_list: {} can not found in MindRecord fields: {}".format(columns_list,
full_columns_list))
super().__init__(source=dataset, column_names=columns_list, num_shards=None, shard_id=None, shuffle=False)
# 用于向OBSMindDataset类添加一个新的采样器
# 但是，这个函数没有被实现，即没有实现向OBSMindDataset类添加采样器的功能。当调用add_sampler函数时，
# 会抛出一个NotImplementedError异常，提示add_sampler没有被实现。
def add_sampler(self, new_sampler):
raise NotImplementedError("add_sampler is not supported for OBSMindDataset.")
# 用于向OBSMindDataset类使用一个新的采样器
# 但是，这个函数没有被实现，即没有实现从OBSMindDataset类中使用采样器的功能。当调用use_sampler函数时，
# 会抛出一个NotImplementedError异常，提示use_sampler没有被实现。
def use_sampler(self, new_sampler):
raise NotImplementedError("use_sampler is not supported for OBSMindDataset.")

530
mindspore/python/mindspore/dataset/engine/datasets_text.py
View File

@ -35,37 +35,38 @@ from .validators import check_imdb_dataset, check_iwslt2016_dataset, check_iwslt
from ..core.validator_helpers import replace_none
# 本文件中代码为用于读取与解析各种文本类数据集的类
class AGNewsDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses AG News datasets.
用于读取和解析AG新闻数据集。
生成一个包含三个列的dataset。
The generated dataset has three columns: :py:obj:`[index, title, description]`.
The tensor of column :py:obj:`index` is of the string type.
The tensor of column :py:obj:`title` is of the string type.
The tensor of column :py:obj:`description` is of the string type.
The tensor of column :py:obj:`index` 字符串类型
The tensor of column :py:obj:`title` 字符串类型
The tensor of column :py:obj:`description` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Acceptable usages include 'train', 'test' and 'all' (default=None, all samples).
num_samples (int, optional): Number of samples (rows) to read (default=None, reads the full dataset).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_dir (str): 路径到包含数据集的根目录。
usage (str, optional): 可接受的使用范围，包括'train'（训练集）、'test'（测试集）和'all'（所有样本），默认
为None，表示读取所有样本。
num_samples (int, optional): 要读取的样本数量（行数），默认为None，表示读取整个数据集。
num_parallel_workers (int, optional): 用于读取数据的worker数量，默认为None，表示根据配置文件设置的worker数量。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, 'num_samples' reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
num_shards (int, optional): 将数据集划分为多少个分片，默认为None，表示不进行分片。
shard_id (int, optional): 当前分片的ID，当num_shards指定时，此参数必须指定。
cache (DatasetCache, optional): 使用张量缓存服务加速数据处理，默认为None，表示不使用缓存。
Examples:
>>> ag_news_dataset_dir = "/path/to/ag_news_dataset_file"
@ -73,16 +74,12 @@ class AGNewsDataset(SourceDataset, TextBaseDataset):
About AGNews dataset:
AG is a collection of over 1 million news articles. The news articles were collected
by ComeToMyHead from over 2,000 news sources in over 1 year of activity. ComeToMyHead
is an academic news search engine that has been in operation since July 2004.
The dataset is provided by academics for research purposes such as data mining
(clustering, classification, etc.), information retrieval (ranking, searching, etc.),
xml, data compression, data streaming, and any other non-commercial activities.
AG's news topic classification dataset was constructed by selecting the four largest
classes from the original corpus. Each class contains 30,000 training samples and
1,900 test samples. The total number of training samples in train.csv is 120,000
and the number of test samples in test.csv is 7,600.
AGNews是一个包含1亿多条新闻文章的集合，这些文章是从2004年开始从2000多个新闻来源中收集的。ComeToMyHead是一个学术性的新闻搜
索引擎，已经运行了1年。
AGNews数据集提供给研究目的，包括数据挖掘（聚类、分类等）、信息检索（排序、搜索等）、XML、数据压缩、数据流等非商业activities。
AG新闻主题分类数据集是从原始语料中选择四个最大的类别，每个类别包含30000训练样本和1900测试样本。训练样本的总数为120000，测试
样本的总数为7600。
You can unzip the dataset files into the following structure and read by MindSpore's API:
@ -112,54 +109,49 @@ class AGNewsDataset(SourceDataset, TextBaseDataset):
@check_ag_news_dataset
def __init__(self, dataset_dir, usage=None, num_samples=None,
num_parallel_workers=None, shuffle=Shuffle.GLOBAL, num_shards=None, shard_id=None, cache=None):
# 调用父类的初始化方法
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
# 将self.dataset_dir赋值给dataset_dir
self.dataset_dir = dataset_dir
# 将usage赋值给usage，将其替换为"all"
self.usage = replace_none(usage, "all")
def parse(self, children=None):
# 返回cde.AGNewsNode
return cde.AGNewsNode(self.dataset_dir, self.usage, self.num_samples, self.shuffle_flag, self.num_shards,
self.shard_id)
class AmazonReviewDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses Amazon Review Polarity and Amazon Review Full datasets.
用于读取和解析Amazon Review Polarity和Amazon Review Full数据集。
The generated dataset has three columns: :py:obj:`[label, title, content]`.
The tensor of column :py:obj:`label` is of the string type.
The tensor of column :py:obj:`title` is of the string type.
The tensor of column :py:obj:`content` is of the string type.
The tensor of column :py:obj:`label` 字符串类型
The tensor of column :py:obj:`title` 字符串类型
The tensor of column :py:obj:`content` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the Amazon Review Polarity dataset
or the Amazon Review Full dataset.
usage (str, optional): Usage of this dataset, can be 'train', 'test' or 'all' (default= 'all').
For Polarity dataset, 'train' will read from 3,600,000 train samples,
'test' will read from 400,000 test samples,
'all' will read from all 4,000,000 samples.
For Full dataset, 'train' will read from 3,000,000 train samples,
'test' will read from 650,000 test samples,
'all' will read from all 3,650,000 samples (default=None, all samples).
num_samples (int, optional): Number of samples (rows) to be read (default=None, reads the full dataset).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_dir (str): 数据集的根目录，其中包含Amazon Review Polarity dataset或Amazon Review Full dataset。
usage (str, optional): 数据集的用途，可以是'train'（训练集）、'test'（测试集）或'all'（全部样本），默认为'all'。
对于Polarity dataset，'train'将读取3,600,000训练样本，'test'将读取400,000测试样本，'all'将读取全部4,000,000样本。
对于Full dataset，'train'将读取3,000,000训练样本，'test'将读取650,000测试样本，'all'将读取全部3,650,000样本。
num_samples (int, optional): 要读取的样本数量（行数），默认为None，表示读取整个数据集。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the mindspore.dataset.config).
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
num_shards (int, optional): 将数据集划分为多个分片（shard），默认为None。当这个参数指定时，num_samples表示每个分
片的最大样本数。
shard_id (int, optional): 在num_shards中的分片ID，当num_shards也指定时，这个参数必须被指定。
num_parallel_workers (int, optional): 用于读取数据的 worker 数量，默认为None，表示使用系统设置的 worker 数量。
cache (DatasetCache, optional): 使用张量缓存服务加速数据集处理，默认为None，表示不使用缓存。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -173,12 +165,12 @@ class AmazonReviewDataset(SourceDataset, TextBaseDataset):
About AmazonReview Dataset:
The Amazon reviews full dataset consists of reviews from Amazon. The data span a period of 18 years, including ~35
million reviews up to March 2013. Reviews include product and user information, ratings, and a plaintext review.
The dataset is mainly used for text classification, given the content and title, predict the correct star rating.
Amazon Reviews Full dataset是一个包含来自Amazon的评论的完整数据集，时间跨度为18年，包括3500万条评论，
直到2013年3月。这些评论包括产品信息和用户信息、评分和纯文本评论。这个数据集主要用于文本分类，给定内容和标
题，预测正确的星级评分。
The Amazon reviews polarity dataset is constructed by taking review score 1 and 2 as negative, 4 and 5 as positive.
Samples of score 3 is ignored. In the dataset, class 1 is the negative and class 2 is the positive.
Amazon Reviews Polarity dataset是Amazon Reviews Full dataset的子集，将评分1和2视为负向，4和5视为正向。
在数据集中，类1表示负向，类2表示正向。
The Amazon Reviews Polarity and Amazon Reviews Full datasets have the same directory structures.
You can unzip the dataset files into the following structure and read by MindSpore's API:
@ -208,50 +200,50 @@ class AmazonReviewDataset(SourceDataset, TextBaseDataset):
@check_amazon_review_dataset
def __init__(self, dataset_dir, usage=None, num_samples=None, num_parallel_workers=None, shuffle=Shuffle.GLOBAL,
num_shards=None, shard_id=None, cache=None):
# 调用父类的初始化方法
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
# 将self.dataset_dir赋值给dataset_dir
self.dataset_dir = dataset_dir
# 将usage赋值给usage，将其替换为"all"
self.usage = replace_none(usage, 'all')
def parse(self, children=None):
# 返回cde.AmazonReviewNode
return cde.AmazonReviewNode(self.dataset_dir, self.usage, self.num_samples, self.shuffle_flag, self.num_shards,
self.shard_id)
class CLUEDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses CLUE datasets.
Supported CLUE classification tasks: 'AFQMC', 'TNEWS', 'IFLYTEK', 'CMNLI', 'WSC' and 'CSL'.
用于读取和解析CLUE数据集。
CLUE是一个用于文本分类的任务集合，包括"AFQMC"、"TNEWS"、"IFLYTEK"、"CMNLI"和"WSC"等分类任务。
Args:
dataset_files (Union[str, list[str]]): String or list of files to be read or glob strings to search for
a pattern of files. The list will be sorted in a lexicographical order.
task (str, optional): The kind of task, one of 'AFQMC', 'TNEWS', 'IFLYTEK', 'CMNLI', 'WSC' and 'CSL'.
(default=AFQMC).
usage (str, optional): Specify the 'train', 'test' or 'eval' part of dataset (default='train').
num_samples (int, optional): The number of samples to be included in the dataset
(default=None, will include all images).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_files (Union[str, list[str]]): 用于读取的文件或文件glob字符串列表，会按字典序排序。
task (str, optional): 任务类型，可选值为"AFQMC"、"TNEWS"、"IFLYTEK"、"CMNLI"和"WSC"，默认为"AFQMC"。
usage (str, optional): 指定数据集的"train"、"test"或"eval"部分，默认为"train"。
num_samples (int, optional): 要包含在数据集中的样本数量，默认为None，表示包含所有样本。
num_parallel_workers (int, optional): 用于读取数据的worker数量，默认为None，表示使用系统设置的worker数量。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
num_shards (int, optional): 数据集将被划分为多个分片（shard），默认为None。当这个参数指定时，num_samples表
示每个分片的最大样本数。
shard_id (int, optional): 在num_shards中的分片ID，当num_shards也指定时，这个参数必须被指定。
cache (DatasetCache, optional): 使用张量缓存服务加速数据集处理，默认为None，表示不使用缓存。
The generated dataset with different task setting has different output columns:
task：表示要处理的任务类型，包括"AFQMC"、"TNEWS"、"IFLYTEK"、"CMNLI"、"WSC"和"CSL"。
usage：表示任务的使用方式，包括"train"和"test"。
Output column：表示函数的输出列，不同的任务可能有不同的输出列。例如，对于文本分类任务，输出列可能包括"label"和"dtype=string"。
+-------------------------+------------------------------+-----------------------------+
| `task` | `usage` | Output column |
+=========================+==============================+=============================+
@ -407,9 +399,7 @@ class CLUEDataset(SourceDataset, TextBaseDataset):
About CLUE dataset:
CLUE, a Chinese Language Understanding Evaluation benchmark. It contains multiple
tasks, including single-sentence classification, sentence pair classification, and machine
reading comprehension.
CLUE，汉语理解能力评估基准。它包含多个任务，包括单句分类、句对分类和机器阅读理解。
You can unzip the dataset files into the following structure and read by MindSpore's API,
such as afqmc dataset:
@ -441,50 +431,49 @@ class CLUEDataset(SourceDataset, TextBaseDataset):
@check_cluedataset
def __init__(self, dataset_files, task='AFQMC', usage='train', num_samples=None, num_parallel_workers=None,
shuffle=Shuffle.GLOBAL, num_shards=None, shard_id=None, cache=None):
# 调用父类的初始化方法
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
self.dataset_files = self._find_files(dataset_files)
# 根据dataset_files中的文件路径，查找数据集文件
self.usage = replace_none(usage, 'train')
# 替换None值为'train'
self.task = replace_none(task, 'AFQMC')
def parse(self, children=None):
# 返回cde.CLUENode
return cde.CLUENode(self.dataset_files, self.task, self.usage, self.num_samples, self.shuffle_flag,
self.num_shards, self.shard_id)
class CoNLL2000Dataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses CoNLL2000 dataset.
用于处理CoNLL2000数据集，这是一个关于自然语言处理任务的数据集。
The generated dataset has three columns: :py:obj:`[word, pos_tag, chunk_tag]`.
The tensor of column :py:obj:`word` is of the string type.
The tensor of column :py:obj:`pos_tag` is of the string type.
The tensor of column :py:obj:`chunk_tag` is of the string type.
The tensor of column :py:obj:`word` 字符串类型
The tensor of column :py:obj:`pos_tag` 字符串类型
The tensor of column :py:obj:`chunk_tag` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be 'train', 'test', or 'all'. 'train' will read from
8936 train samples, 'test' will read from 2,012 test samples,
'all' will read from all 1,0948 samples (default=None, all samples).
num_samples (int, optional): Number of samples (rows) to read (default=None, reads the full dataset).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_dir (str): 用于处理CoNLL2000数据集，这是一个关于自然语言处理任务的数据集。
usage (str, optional): 数据集的使用情况，可选值为'train'、'test'或'all'。'train'表示读取训练集，'test'表
示读取测试集，'all'表示读取所有数据集。默认为None，表示读取所有数据集。
num_samples (int, optional): 读取的样本数量，默认为None，表示读取所有样本。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
num_shards (int, optional): 数据集分片数量，默认为None。当这个参数指定时，num_samples表示每个分片的最大样本数量。
shard_id (int, optional): 当前分片ID，默认为None。当这个参数指定时，需要同时指定num_shards参数。
num_parallel_workers (int, optional): 读取数据的worker数量，默认为None，由配置文件设置。
cache (DatasetCache, optional): 是否使用tensor缓存服务加速数据集处理，默认为None，表示不使用缓存。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -500,51 +489,47 @@ class CoNLL2000Dataset(SourceDataset, TextBaseDataset):
@check_conll2000_dataset
def __init__(self, dataset_dir, usage=None, num_samples=None, shuffle=Shuffle.GLOBAL, num_shards=None,
shard_id=None, num_parallel_workers=None, cache=None):
# 调用父类函数进行初始化
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
self.dataset_dir = dataset_dir
self.usage = replace_none(usage, 'all')
def parse(self, children=None):
# 返回cde.CONLL2000Node
return cde.CoNLL2000Node(self.dataset_dir, self.usage, self.num_samples, self.shuffle_flag, self.num_shards,
self.shard_id)
class DBpediaDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses the DBpedia dataset.
用于读取和解析DBpedia数据集。
The generated dataset has three columns :py:obj:`[class, title, content]`.
The tensor of column :py:obj:`class` is of the string type.
The tensor of column :py:obj:`title` is of the string type.
The tensor of column :py:obj:`content` is of the string type.
The tensor of column :py:obj:`class` 字符串类型
The tensor of column :py:obj:`title` 字符串类型
The tensor of column :py:obj:`content` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be 'train', 'test' or 'all'.
'train' will read from 560,000 train samples,
'test' will read from 70,000 test samples,
'all' will read from all 630,000 samples (default=None, all samples).
num_samples (int, optional): The number of samples to be included in the dataset
(default=None, will include all text).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_dir (str): 数据集的根目录，用于存储所有数据文件。
usage (str, optional): 指定要使用的数据集部分，可以是'train'（训练集）、'test'（测试集）或
'all'（所有数据集）。默认情况下，使用所有数据集。
num_samples (int, optional): 指定要包含在数据集中的样本数量。如果没有指定，将包含所有样本。
num_parallel_workers (int, optional): 指定读取数据的并发worker数量。如果没有指定，将使用配置中的worker数量。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
num_shards (int, optional): 定将数据集划分为的shard数量。如果没有指定，将使用配置中的shard数量。
shard_id (int, optional): 指定当前shard的ID，当num_shards也指定时，才能使用此参数。
cache (DatasetCache, optional): 指定使用tensor缓存服务以加速数据集处理。如果没有指定，将不使用缓存。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -564,10 +549,8 @@ class DBpediaDataset(SourceDataset, TextBaseDataset):
About DBpedia dataset:
The DBpedia dataset consists of 630,000 text samples in 14 classes, there are 560,000 samples in the train.csv
and 70,000 samples in the test.csv.
The 14 different classes represent Company, EducationaInstitution, Artist, Athlete, OfficeHolder,
MeanOfTransportation, Building, NaturalPlace, Village, Animal, Plant, Album, Film, WrittenWork.
DBpedia是一个大型语言模型训练数据集，包含630,000条文本样本，分为14个类别，包括公司、教育机构、艺术家、
运动员、政府官员、交通工具、建筑、自然地名、村庄、动物、植物、音乐专辑、电影和小说等。
Here is the original DBpedia dataset structure.
You can unzip the dataset files into this directory structure and read by Mindspore's API.
@ -596,44 +579,44 @@ class DBpediaDataset(SourceDataset, TextBaseDataset):
@check_dbpedia_dataset
def __init__(self, dataset_dir, usage=None, num_samples=None, num_parallel_workers=None, shuffle=Shuffle.GLOBAL,
num_shards=None, shard_id=None, cache=None):
# 调用父类构造函数
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
self.dataset_dir = dataset_dir
self.usage = replace_none(usage, "all")
def parse(self, children=None):
# 返回cde.DBpediaNode
return cde.DBpediaNode(self.dataset_dir, self.usage, self.num_samples, self.shuffle_flag, self.num_shards,
self.shard_id)
class EnWik9Dataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses EnWik9 dataset.
用于读取EnWik9数据集，并将其解析为文本数据。
The generated dataset has one column :py:obj:`[text]` with type string.
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
num_samples (int, optional): The number of samples to be included in the dataset
(default=None, will include all samples).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=True).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_dir (str): 数据集的根目录，用于存储所有数据文件。
usage (str, optional): 指定要使用的数据集部分，可以是'train'（训练集）、'test'（测试集）或
'all'（所有数据集）。默认情况下，使用所有数据集。
num_samples (int, optional): 指定要包含在数据集中的样本数量。如果没有指定，将包含所有样本。
num_parallel_workers (int, optional): 指定读取数据的并发worker数量。如果没有指定，将使用配置中的worker数量。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
num_shards (int, optional): 定将数据集划分为的shard数量。如果没有指定，将使用配置中的shard数量。
shard_id (int, optional): 指定当前shard的ID，当num_shards也指定时，才能使用此参数。
cache (DatasetCache, optional): 指定使用tensor缓存服务以加速数据集处理。如果没有指定，将不使用缓存。
Examples:
>>> en_wik9_dataset_dir = "/path/to/en_wik9_dataset"
@ -642,13 +625,10 @@ class EnWik9Dataset(SourceDataset, TextBaseDataset):
About EnWik9 dataset:
The data of EnWik9 is UTF-8 encoded XML consisting primarily of English text. It contains 243,426 article titles,
of which 85,560 are #REDIRECT to fix broken links, and the rest are regular articles.
The data is UTF-8 clean. All characters are in the range U'0000 to U'10FFFF with valid encodings of 1 to
4 bytes. The byte values 0xC0, 0xC1, and 0xF5-0xFF never occur. Also, in the Wikipedia dumps,
there are no control characters in the range 0x00-0x1F except for 0x09 (tab) and 0x0A (linefeed).
Linebreaks occur only on paragraph boundaries, so they always have a semantic purpose.
EnWik9数据集是一个英文的UTF-8编码的XML文件，其中包含243,426个文章标题，其中85,560个是#REDIRECT，其余的是
regular articles。数据是干净的，所有字符都在U'0000到U'10FFFF的范围内，并且没有控制字符（除了0x09（制表符）
和0x0A（换行符））。在Wikipedia数据集中，没有控制字符在0x00-0x1F范围内，除了0x09（制表符）和0x0A（换行符）。
换行符只在段落边界出现，并且具有语义目的。
You can unzip the dataset files into the following directory structure and read by MindSpore's API.
@ -673,42 +653,45 @@ class EnWik9Dataset(SourceDataset, TextBaseDataset):
@check_en_wik9_dataset
def __init__(self, dataset_dir, num_samples=None, num_parallel_workers=None, shuffle=True,
num_shards=None, shard_id=None, cache=None):
# 调用父类构造函数
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
self.dataset_dir = dataset_dir
def parse(self, children=None):
# 返回cde.EnWik9Node
return cde.EnWik9Node(self.dataset_dir, self.num_samples, self.shuffle_flag, self.num_shards,
self.shard_id)
class IMDBDataset(MappableDataset, TextBaseDataset):
"""
A source dataset that reads and parses Internet Movie Database (IMDb).
用于读取Internet Movie Database (IMDb)数据集，并将其解析为文本数据。
The generated dataset has two columns: :py:obj:`[text, label]`.
The tensor of column :py:obj:`text` is of the string type.
The tensor of column :py:obj:`label` is of a scalar of uint32 type.
The tensor of column :py:obj:`text` 字符串类型
The tensor of column :py:obj:`label` 无符号整数（uint32）
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be 'train', 'test' or 'all'
(default=None, will read all samples).
num_samples (int, optional): The number of images to be included in the dataset
(default=None, will read all samples).
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, set in the config).
shuffle (bool, optional): Whether or not to perform shuffle on the dataset
(default=None, expected order behavior shown in the table).
sampler (Sampler, optional): Object used to choose samples from the
dataset (default=None, expected order behavior shown in the table).
num_shards (int, optional): Number of shards that the dataset will be divided
into (default=None). When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing
(default=None, which means no cache is used).
dataset_dir (str): 数据集的根目录，用于存储所有数据文件。
usage (str, optional): 指定要使用的数据集部分，可以是'train'（训练集）、'test'（测试集）或
'all'（所有数据集）。默认情况下，使用所有数据集。
num_samples (int, optional): 指定要包含在数据集中的样本数量。如果没有指定，将包含所有样本。
num_parallel_workers (int, optional): 指定读取数据的并发worker数量。如果没有指定，将使用配置中的worker数量。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
(default=None, performs global shuffle).
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
有以下三种选项：
Shuffle.GLOBAL：全局洗牌，即洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.INFILE：保留文件顺序但洗牌数据内部。
num_shards (int, optional): 定将数据集划分为的shard数量。如果没有指定，将使用配置中的shard数量。
shard_id (int, optional): 指定当前shard的ID，当num_shards也指定时，才能使用此参数。
cache (DatasetCache, optional): 指定使用tensor缓存服务以加速数据集处理。如果没有指定，将不使用缓存。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -731,24 +714,12 @@ class IMDBDataset(MappableDataset, TextBaseDataset):
* - Parameter `sampler`
- Parameter `shuffle`
- Expected Order Behavior
* - None
- None
- random order
* - None
- True
- random order
* - None
- False
- sequential order
* - Sampler object
- None
- order defined by sampler
* - Sampler object
- True
- not allowed
* - Sampler object
- False
- not allowed
* - 当sampler为None，shuffle为None时，数据集的顺序是随机的。
当sampler为None，shuffle为True时，数据集的顺序是随机的。
当sampler为None，shuffle为False时，数据集的顺序是顺序处理的。
* - 当sampler是一个sampler对象，shuffle为None时，数据集的顺序由sampler定义。
当sampler是一个sampler对象，shuffle为True时，不允许使用sampler，因为可能会导致顺序不一致。
当sampler是一个sampler对象，shuffle为False时，不允许使用sampler，因为可能会导致顺序不一致。
Examples:
>>> imdb_dataset_dir = "/path/to/imdb_dataset_directory"
@ -761,10 +732,9 @@ class IMDBDataset(MappableDataset, TextBaseDataset):
About IMDBDataset:
The IMDB dataset contains 50, 000 highly polarized reviews from the Internet Movie Database (IMDB). The dataset
was divided into 25 000 comments for training and 25 000 comments for testing, with both the training set and test
set containing 50% positive and 50% negative comments. Train labels and test labels are all lists of 0 and 1, where
0 stands for negative and 1 for positive.
IMDB数据集包含了50, 000条极性化的评论，这些评论来自IMDB。IMDB数据集被划分为25, 000条训练评论和25, 000条测
试评论，其中训练集和测试集分别包含50%的正极性和50%的负极性评论。这意味着训练标签和测试标签都是一个包含0和1的
列表，其中0表示负极性，1表示正极性。
You can unzip the dataset files into this directory structure and read by MindSpore's API.
@ -813,6 +783,7 @@ class IMDBDataset(MappableDataset, TextBaseDataset):
@check_imdb_dataset
def __init__(self, dataset_dir, usage=None, num_samples=None, num_parallel_workers=None, shuffle=None, sampler=None,
num_shards=None, shard_id=None, cache=None):
# 调用父类构造函数
super().__init__(num_parallel_workers=num_parallel_workers, sampler=sampler, num_samples=num_samples,
shuffle=shuffle, num_shards=num_shards, shard_id=shard_id, cache=cache)
@ -820,47 +791,45 @@ class IMDBDataset(MappableDataset, TextBaseDataset):
self.usage = replace_none(usage, "all")
def parse(self, children=None):
# 返回cde.IMDBNode
return cde.IMDBNode(self.dataset_dir, self.usage, self.sampler)
class IWSLT2016Dataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses IWSLT2016 datasets.
用于读取和解析IWSLT2016数据集。
The generated dataset has two columns: :py:obj:`[text, translation]`.
The tensor of column :py:obj: `text` is of the string type.
The tensor of column :py:obj: `translation` is of the string type.
The tensor of column :py:obj: `text` 字符串类型
The tensor of column :py:obj: `translation` 无符号整数（uint32）
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Acceptable usages include 'train', 'valid', 'test' and 'all' (default=None, all samples).
language_pair (sequence, optional): Sequence containing source and target language, supported values are
('en', 'fr'), ('en', 'de'), ('en', 'cs'), ('en', 'ar'), ('fr', 'en'), ('de', 'en'), ('cs', 'en'),
('ar', 'en') (default=('de', 'en')).
valid_set (str, optional): A string to identify validation set, when usage is valid or all, the validation set
of valid_set type will be read, supported values are 'dev2010', 'tst2010', 'tst2011', 'tst2012', 'tst2013'
and 'tst2014' (default='tst2013').
test_set (str, optional): A string to identify test set, when usage is test or all, the test set of test_set
type will be read, supported values are 'dev2010', 'tst2010', 'tst2011', 'tst2012', 'tst2013' and 'tst2014'
(default='tst2014').
num_samples (int, optional): Number of samples (rows) to read (default=None, reads the full dataset).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
dataset_dir (str): IWSLT2016数据集的根目录。
usage (str, optional): 可接受的使用类型，包括'train', 'valid', 'test'和'all'（默认为None，表示读取所有样本）。
language_pair (sequence, optional): 包含源语言和目标语言的元组，支持值有('en', 'fr'), ('en', 'de'), ('en', 'cs'),
('en', 'ar'), ('fr', 'en'), ('de', 'en'), ('cs', 'en'), ('ar', 'en')（默认为('de', 'en')）。
valid_set (str, optional): 用于验证的集类型，当usage为'valid'或'all'时，将读取相应的验证集，支持值有'dev2010',
'tst2010', 'tst2011', 'tst2012', 'tst2013'和'tst2014'（默认为'tst2013'）。
test_set (str, optional): 用于测试的集类型，当usage为'test'或'all'时，将读取相应的测试集，支持值有'dev2010',
'tst2010', 'tst2011', 'tst2012', 'tst2013'和'tst2014'（默认为'tst2014'）。
num_samples (int, optional): 要读取的样本数量（默认为None，表示读取所有样本）。
shuffle (Union[bool, Shuffle level], optional): 一个整数，表示用于读取数据的 worker 数量。如果没有提供，将使用配置中
的最大线程数。
shuffle (Union[bool, Shuffle level], optional): 一个枚举值，表示是否对数据进行洗牌。
shuffle值为False，不进行洗牌
shuffle值为True，进行全局洗牌
有以下三种选项：
Shuffle.GLOBAL：既洗牌文件又洗牌样本。
Shuffle.FILES：只洗牌文件。
Shuffle.NONE：不洗牌。
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
num_shards (int, optional): 表示每个shard的最大样本数量。
shard_id (int, optional): 当前shard的ID（默认为None，表示使用MindSpore的配置文件设置的shard ID）。
num_parallel_workers (int, optional): 用于读取数据的worker数量（默认为None，表示使用MindSpore的配置文件设置
的worker数量）。
cache (DatasetCache, optional): 使用tensor缓存服务加速数据处理（默认为None，表示不使用缓存）。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -875,10 +844,8 @@ class IWSLT2016Dataset(SourceDataset, TextBaseDataset):
About IWSLT2016 dataset:
IWSLT is an international oral translation conference, a major annual scientific conference dedicated to all aspects
of oral translation. The MT task of the IWSLT evaluation activity constitutes a dataset, which can be publicly
obtained through the WIT3 website wit3.fbk.eu. The IWSLT2016 dataset includes translations from English to Arabic,
Czech, French, and German, and translations from Arabic, Czech, French, and German to English.
IWSLT（国际语音翻译）是一个国际语音翻译会议，是的重要年度科学会议之一， dedicates to all aspects of oral translation。IWSLT2016
数据集包括从英语到阿拉伯语、捷克语、法语、德语的翻译，以及从阿拉伯语、捷克语、法语和德语到英语的翻译。
You can unzip the original IWSLT2016 dataset files into this directory structure and read by MindSpore's API. After
decompression, you also need to decompress the dataset to be read in the specified folder. For example, if you want
@ -936,6 +903,7 @@ class IWSLT2016Dataset(SourceDataset, TextBaseDataset):
def __init__(self, dataset_dir, usage=None, language_pair=None, valid_set=None, test_set=None,
num_samples=None, shuffle=Shuffle.GLOBAL, num_shards=None, shard_id=None, num_parallel_workers=None,
cache=None):
# 调用父类构造函数
super().__init__(num_parallel_workers=num_parallel_workers, num_samples=num_samples, shuffle=shuffle,
num_shards=num_shards, shard_id=shard_id, cache=cache)
self.dataset_dir = dataset_dir
@ -945,43 +913,37 @@ class IWSLT2016Dataset(SourceDataset, TextBaseDataset):
self.test_set = replace_none(test_set, 'tst2014')
def parse(self, children=None):
# 返回cde.IWSLT2016Node
return cde.IWSLT2016Node(self.dataset_dir, self.usage, self.language_pair, self.valid_set, self.test_set,
self.num_samples, self.shuffle_flag, self.num_shards, self.shard_id)
class IWSLT2017Dataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses IWSLT2017 datasets.
用于读取和解析IWSLT2017（国际文摘语言翻译任务）数据集。
The generated dataset has two columns: :py:obj:`[text, translation]`.
The tensor of column :py:obj:`text` is of the string type.
The tensor of column :py:obj:`translation` is of the string type.
The tensor of column :py:obj:`text` 字符串类型
The tensor of column :py:obj:`translation` 字符串类型
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Acceptable usages include 'train', 'valid', 'test' and 'all' (default=None, all samples).
language_pair (list, optional): List containing src and tgt language, supported values are ('en', 'nl'),
dataset_dir (str): 数据集的根目录。
usage (str, optional): 可接受的使用方式包括'train'（训练集）、'valid'（验证集）、'test'（测试集）和
'all'（全部样本），默认为None，表示读取所有样本。
language_pair (list, optional): 包含源语言和目标语言的列表，支持值有('en', 'nl'),
('en', 'de'), ('en', 'it'), ('en', 'ro'), ('nl', 'en'), ('nl', 'de'), ('nl', 'it'), ('nl', 'ro'),
('de', 'en'), ('de', 'nl'), ('de', 'it'), ('de', 'ro'), ('it', 'en'), ('it', 'nl'), ('it', 'de'),
('it', 'ro'), ('ro', 'en'), ('ro', 'nl'), ('ro', 'de'), ('ro', 'it') (default=('de', 'en')).
num_samples (int, optional): Number of samples (rows) to read (default=None, reads the full dataset).
shuffle (Union[bool, Shuffle level], optional): Perform reshuffling of the data every epoch
(default=Shuffle.GLOBAL).
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
- Shuffle.GLOBAL: Shuffle both the files and samples, same as setting shuffle to True.
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into (default=None).
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards` (default=None). This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data
(default=None, number set in the config).
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing.
(default=None, which means no cache is used).
num_samples (int, optional): 要读取的样本数量（行数），默认为None，表示读取整个数据集。
shuffle (Union[bool, Shuffle level], optional): 是否进行随机洗牌，有三种级别：
Shuffle.GLOBAL：同时洗牌文件和样本。
Shuffle.FILES：只洗牌文件。
Shuffle.NONE：不进行洗牌。
num_shards (int, optional): 整数，表示数据集将被划分为的份数。当这个参数被指定时，num_samples表示每个分区
的最大样本数。
shard_id (int, optional): 整数，表示当前分区的ID，当num_shards也被指定时，这个参数才有效。
num_parallel_workers (int, optional): 整数，表示用于读取数据的 worker 数量。
cache (DatasetCache, optional): DatasetCache类的实例，用于加速数据集处理。
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
@ -995,10 +957,8 @@ class IWSLT2017Dataset(SourceDataset, TextBaseDataset):
About IWSLT2017 dataset:
IWSLT is an international oral translation conference, a major annual scientific conference dedicated to all aspects
of oral translation. The MT task of the IWSLT evaluation activity constitutes a dataset, which can be publicly
obtained through the WIT3 website wit3.fbk.eu. The IWSLT2017 dataset involves German, English, Italian, Dutch, and
Romanian. The dataset includes translations in any two different languages.
IWSLT（国际语音翻译）是一个国际语音翻译会议，是针对语音翻译任务举办的重要年度科学会议。IWSLT2017数据集是一个包含德语、英语、意大
利语、荷兰语和罗马尼亚语的翻译数据集，其中包括两种不同的语言的翻译。
You can unzip the original IWSLT2017 dataset files into this directory structure and read by MindSpore's API. You
need to decompress the dataset package in texts/DeEnItNlRo/DeEnItNlRo directory to get the DeEnItNlRo-DeEnItNlRo

443
mindspore/python/mindspore/dataset/engine/datasets_user_defined.py
View File

File diff suppressed because it is too large Load Diff

2396
mindspore/python/mindspore/dataset/engine/datasets_vision.py
View File

File diff suppressed because it is too large Load Diff

118
mindspore/python/mindspore/dataset/engine/graphdata.py
View File

@ -16,22 +16,34 @@
graphdata.py supports loading graph dataset for GNN network training,
and provides operations related to graph data.
"""
# 导入atexit库，用于在程序退出时执行一些清理工作
import atexit
# 导入time库，用于处理时间相关的操作
import time
# 导入IntEnum类，用于定义整数枚举类型
from enum import IntEnum
# 导入numpy库，用于处理和计算数组和矩阵
import numpy as np
# 下面五个import用于处理图数据的部分
# 导入mindspore._c_dataengine模块中的GraphDataClient
from mindspore._c_dataengine import GraphDataClient
# 导入mindspore._c_dataengine模块中的GraphDataServer
from mindspore._c_dataengine import GraphDataServer
# 导入mindspore._c_dataengine模块中的Tensor
from mindspore._c_dataengine import Tensor
# 导入mindspore._c_dataengine模块中的SamplingStrategy（采样策略）
from mindspore._c_dataengine import SamplingStrategy as Sampling
# 导入mindspore._c_dataengine模块中的OutputFormat（输出格式）
from mindspore._c_dataengine import OutputFormat as Format
# 导入一些验证函数，这些函数用于检查GNN数据处理过程中的参数是否合法
from .validators import check_gnn_graphdata, check_gnn_get_all_nodes, check_gnn_get_all_edges, \
check_gnn_get_nodes_from_edges, check_gnn_get_edges_from_nodes, check_gnn_get_all_neighbors, \
check_gnn_get_sampled_neighbors, check_gnn_get_neg_sampled_neighbors, check_gnn_get_node_feature, \
check_gnn_get_edge_feature, check_gnn_random_walk
# 是一个整数枚举类型，用于定义图神经网络（GNN）中采样策略的类型
# 在这个函数中有两个常量RANDOM和EDGE_WEIGHT
# 其中RANDOM：表示随机游走采样策略；EDGE_WEIGHT：表示基于边权重的采样策略。
class SamplingStrategy(IntEnum):
"""
Specifies the sampling strategy when execute `get_sampled_neighbors`.
@ -39,16 +51,21 @@ class SamplingStrategy(IntEnum):
- RANDOM: Random sampling with replacement.
- EDGE_WEIGHT: Sampling with edge weight as probability.
"""
# 定义RANDOM为0
RANDOM = 0
# 定义EDGE_WEIGHT为1
EDGE_WEIGHT = 1
# 定义一个字典DE_C_INTER_SAMPLING_STRATEGY，用于将SamplingStrategy枚举类型映射到Sampling枚举类型
DE_C_INTER_SAMPLING_STRATEGY = {
# 表示随机游走采样策略，将其映射到Sampling.DE_SAMPLING_RANDOM
SamplingStrategy.RANDOM: Sampling.DE_SAMPLING_RANDOM,
# 表示基于边权重的采样策略，将其映射到Sampling.DE_SAMPLING_EDGE_WEIGHT
SamplingStrategy.EDGE_WEIGHT: Sampling.DE_SAMPLING_EDGE_WEIGHT,
}
# 一个整数枚举类型，用于定义图神经网络（GNN）中输出数据的格式
# NODE：表示输出节点特征；EDGE：表示输出边特征；GRAPH：表示输出图结构
class OutputFormat(IntEnum):
"""
Specifies the output storage format when execute `get_all_neighbors`.
@ -57,18 +74,25 @@ class OutputFormat(IntEnum):
- COO: COO format.
- CSR: CSR format.
"""
# 定义NORMAL为0
NORMAL = 0
# 定义COO为1
COO = 1
# 定义CSR为2
CSR = 2
# 定义一个字典DE_C_INTER_OUTPUT_FORMAT，用于将OutputFormat枚举类型映射到Format枚举类型
DE_C_INTER_OUTPUT_FORMAT = {
# 表示输出格式为正常格式，将其映射到Format.DE_FORMAT_NORMAL
OutputFormat.NORMAL: Format.DE_FORMAT_NORMAL,
# 表示输出格式为COO格式，将其映射到Format.DE_FORMAT_COO
OutputFormat.COO: Format.DE_FORMAT_COO,
# 表示输出格式为CSR格式，将其映射到Format.DE_FORMAT_CSR
OutputFormat.CSR: Format.DE_FORMAT_CSR,
}
# 用于表示图数据
# GraphData类定义了图数据的属性，如节点数量、边数量、节点特征、边特征和邻接矩阵等，并提供了一些方法，用于获取和修改图数据的相关属性
class GraphData:
"""
Reads the graph dataset used for GNN training from the shared file and database.
@ -121,31 +145,50 @@ class GraphData:
"""
@check_gnn_graphdata
# 参数
# dataset_file：表示图数据文件路径，用于指定图数据文件的位置，以便从文件中读取图数据
# num_parallel_workers：表示并行工作线程数量，用于指定在读取图数据时使用的并行线程数量，以提高读取速度
# working_mode：表示工作模式，用于指定图数据处理的工作模式，如本地模式、集群模式等
# hostname：表示主机名，用于指定分布式模式下服务器的地址
# port：表示服务器的端口号，用于指定分布式模式下服务器的端口号
# num_client：表示客户端数量，用于指定分布式模式下客户端的数量
# auto_shutdown：表示是否自动关闭，用于指定在完成图数据处理后是否自动关闭服务器
def __init__(self, dataset_file, num_parallel_workers=None, working_mode='local', hostname='127.0.0.1', port=50051,
num_client=1, auto_shutdown=True):
self._dataset_file = dataset_file
self._working_mode = working_mode
# 如果num_parallel_workers参数为None
if num_parallel_workers is None:
# 设置num_parallel_workers参数为1
num_parallel_workers = 1
def stop():
# 调用self._graph_data.stop()在完成数据处理后关闭服务器
self._graph_data.stop()
# 如果working_mode为'local'或'client'
if working_mode in ['local', 'client']:
# 创建一个GraphDataClient对象，并将其赋值给self._graph_data
self._graph_data = GraphDataClient(dataset_file, num_parallel_workers, working_mode, hostname, port)
# 用atexit模块注册一个stop函数，该函数在完成图数据处理后关闭服务器
atexit.register(stop)
# 如果working_mode为'server'
if working_mode == 'server':
# 创建一个GraphDataServer对象，并将其赋值给self._graph_data
self._graph_data = GraphDataServer(
dataset_file, num_parallel_workers, hostname, port, num_client, auto_shutdown)
# 用atexit模块注册一个stop函数，该函数在完成图数据处理后关闭服务器
atexit.register(stop)
# 使用try捕获异常
try:
# 循环检查图数据服务器是否已经停止
while self._graph_data.is_stopped() is not True:
time.sleep(1)
except KeyboardInterrupt:
# 抛出异常
raise Exception("Graph data server receives KeyboardInterrupt.")
@check_gnn_get_all_nodes
# 用于获取图中所有指定类型的节点
def get_all_nodes(self, node_type):
"""
Get all nodes in the graph.
@ -162,11 +205,16 @@ class GraphData:
Raises:
TypeError: If `node_type` is not integer.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.get_all_nodes(node_type)获取图中所有指定类型的节点
# 然后使用as_array()方法将节点转换为数组，并返回该数组
return self._graph_data.get_all_nodes(node_type).as_array()
@check_gnn_get_all_edges
# 用于获取图中所有指定类型的边
def get_all_edges(self, edge_type):
"""
Get all edges in the graph.
@ -183,11 +231,16 @@ class GraphData:
Raises:
TypeError: If `edge_type` is not integer.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.get_all_nodes(node_type)获取图中所有指定类型的节点
# 然后使用as_array()方法将节点转换为数组，并返回该数组
return self._graph_data.get_all_edges(edge_type).as_array()
@check_gnn_get_nodes_from_edges
# 用于根据给定的边列表获取该边所连接的节点
def get_nodes_from_edges(self, edge_list):
"""
Get nodes from the edges.
@ -201,11 +254,16 @@ class GraphData:
Raises:
TypeError: If `edge_list` is not list or ndarray.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.get_all_nodes(node_type)获取图中所有指定类型的节点
# 然后使用as_array()方法将节点转换为数组，并返回该数组
return self._graph_data.get_nodes_from_edges(edge_list).as_array()
@check_gnn_get_edges_from_nodes
# 用于根据给定的节点列表获取与这些节点相连的所有边
def get_edges_from_nodes(self, node_list):
"""
Get edges from the nodes.
@ -222,11 +280,16 @@ class GraphData:
Raises:
TypeError: If `edge_list` is not list or ndarray.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.get_all_nodes(node_type)获取图中所有指定类型的节点
# 然后使用as_array()方法将节点转换为数组，并返回该数组
return self._graph_data.get_edges_from_nodes(node_list).as_array()
@check_gnn_get_all_neighbors
# 用于获取与给定节点列表中的每个节点相邻的指定类型的邻居节点
def get_all_neighbors(self, node_list, neighbor_type, output_format=OutputFormat.NORMAL):
"""
Get `neighbor_type` neighbors of the nodes in `node_list`.
@ -342,17 +405,26 @@ class GraphData:
TypeError: If `node_list` is not list or ndarray.
TypeError: If `neighbor_type` is not integer.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 调用图数据对象的get_all_neighbors方法，传入节点列表node_list、邻居类型neighbor_type和输出格式output_format作为参数
result_list = self._graph_data.get_all_neighbors(node_list, neighbor_type,
DE_C_INTER_OUTPUT_FORMAT[output_format]).as_array()
# 如果输出格式为OutputFormat.CSR
if output_format == OutputFormat.CSR:
# 如果输出格式为OutputFormat.CSR，将结果列表的第一个部分（即偏移表）赋值给变量offset_table
offset_table = result_list[:len(node_list)]
# 如果输出格式为OutputFormat.CSR，将结果列表的第二个部分（即邻居表）赋值给变量neighbor_table
neighbor_table = result_list[len(node_list):]
# 返回offset_table和neighbor_table
return offset_table, neighbor_table
# 返回result_list
return result_list
@check_gnn_get_sampled_neighbors
# 用于获取与给定节点列表中的每个节点相邻的指定类型的随机采样邻居节点
def get_sampled_neighbors(self, node_list, neighbor_nums, neighbor_types, strategy=SamplingStrategy.RANDOM):
"""
Get sampled neighbor information.
@ -386,14 +458,20 @@ class GraphData:
TypeError: If `neighbor_nums` is not list or ndarray.
TypeError: If `neighbor_types` is not list or ndarray.
"""
# 如果输入的策略strategy不是SamplingStrategy枚举类型
if not isinstance(strategy, SamplingStrategy):
# 抛出异常
raise TypeError("Wrong input type for strategy, should be enum of 'SamplingStrategy'.")
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.get_sampled_neighbors
return self._graph_data.get_sampled_neighbors(
node_list, neighbor_nums, neighbor_types, DE_C_INTER_SAMPLING_STRATEGY[strategy]).as_array()
@check_gnn_get_neg_sampled_neighbors
# 用于获取与给定节点列表中的每个节点相邻的随机负采样邻居节点
def get_neg_sampled_neighbors(self, node_list, neg_neighbor_num, neg_neighbor_type):
"""
Get `neg_neighbor_type` negative sampled neighbors of the nodes in `node_list`.
@ -416,12 +494,16 @@ class GraphData:
TypeError: If `neg_neighbor_num` is not integer.
TypeError: If `neg_neighbor_type` is not integer.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.get_neg_sampled_neighbors
return self._graph_data.get_neg_sampled_neighbors(
node_list, neg_neighbor_num, neg_neighbor_type).as_array()
@check_gnn_get_node_feature
# 用于获取给定节点列表的指定特征值
def get_node_feature(self, node_list, feature_types):
"""
Get `feature_types` feature of the nodes in `node_list`.
@ -441,16 +523,22 @@ class GraphData:
TypeError: If `node_list` is not list or ndarray.
TypeError: If `feature_types` is not list or ndarray.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 如果node_list为列表类型
if isinstance(node_list, list):
# 将node_list转换为numpy数组类型，并指定数据类型为int32
node_list = np.array(node_list, dtype=np.int32)
# 返回用图数据对象的get_node_feature方法，传入节点特征类型列表feature_types和节点列表node_list作为参数
return [
t.as_array() for t in self._graph_data.get_node_feature(
Tensor(node_list),
feature_types)]
@check_gnn_get_edge_feature
# 用于获取给定边列表的指定特征值
def get_edge_feature(self, edge_list, feature_types):
"""
Get `feature_types` feature of the edges in `edge_list`.
@ -470,15 +558,20 @@ class GraphData:
TypeError: If `edge_list` is not list or ndarray.
TypeError: If `feature_types` is not list or ndarray.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 如果edge_list为列表类型
if isinstance(edge_list, list):
# 将edge_list转换为numpy数组类型，并指定数据类型为int32
edge_list = np.array(edge_list, dtype=np.int32)
# 返回用图数据对象的get_edge_feature方法，传入节点特征类型列表feature_types和节点列表edge_list作为参数
return [
t.as_array() for t in self._graph_data.get_edge_feature(
Tensor(edge_list),
feature_types)]
# 用于获取图的基本信息，如节点数量、边数量、最大节点ID等
def graph_info(self):
"""
Get the meta information of the graph, including the number of nodes, the type of nodes,
@ -488,11 +581,15 @@ class GraphData:
dict, meta information of the graph. The key is node_type, edge_type, node_num, edge_num,
node_feature_type and edge_feature_type.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 调用图数据对象的graph_info方法，获取图的基本信息，并返回这些信息
return self._graph_data.graph_info()
@check_gnn_random_walk
# 用于执行随机游走算法，根据给定的目标节点、元路径和参数，生成目标节点的随机游走路径
def random_walk(self, target_nodes, meta_path, step_home_param=1.0, step_away_param=1.0, default_node=-1):
"""
Random walk in nodes.
@ -516,7 +613,10 @@ class GraphData:
TypeError: If `target_nodes` is not list or ndarray.
TypeError: If `meta_path` is not list or ndarray.
"""
# 如果self._working_mode为'server'
if self._working_mode == 'server':
# 抛出异常
raise Exception("This method is not supported when working mode is server.")
# 返回self._graph_data.random_walk
return self._graph_data.random_walk(target_nodes, meta_path, step_home_param, step_away_param,
default_node).as_array()

85
mindspore/python/mindspore/dataset/engine/iterators.py
View File

@ -31,28 +31,37 @@ _ITERATOR_CLEANUP = False
def _set_iterator_cleanup():
global _ITERATOR_CLEANUP
# 将全局变量 _ITERATOR_CLEANUP 的值设置为 True
# 在后续的迭代中，会自动清理迭代器，避免资源泄漏。
_ITERATOR_CLEANUP = True
def _unset_iterator_cleanup():
global _ITERATOR_CLEANUP
# 将全局变量 _ITERATOR_CLEANUP 的值设置为 False
# 在后续的迭代中，不会自动清理迭代器，可能会导致资源泄漏
_ITERATOR_CLEANUP = False
# 检查 _ITERATOR_CLEANUP 的值
def check_iterator_cleanup():
global _ITERATOR_CLEANUP
return _ITERATOR_CLEANUP
# 定义一个全局便令ITERATORS_LIST，用于存储所有创建的迭代器，以便在后续的迭代中释放它们
ITERATORS_LIST = list()
def _cleanup():
"""Release all the Iterator."""
# 将全局变量 _ITERATOR_CLEANUP 的值设置为 True
_set_iterator_cleanup()
# 遍历全局变量 ITERATORS_LIST 的逆序，清空每个迭代器
for itr_ref in reversed(ITERATORS_LIST):
# 使用 itr_ref() 获取迭代器的值
itr = itr_ref()
# 检查它是否为 None
if itr is not None:
# 释放资源
itr.release()
@ -67,7 +76,7 @@ class Iterator:
def __init__(self, dataset, num_epochs=-1, output_numpy=False, do_copy=True):
self._col_names = None
# create a copy of tree and work on it.
# 创建一个副本，并将其转换为 IR 树
self.__ori_dataset = dataset
self.ir_tree, self.dataset = dataset.create_ir_tree()
@ -79,97 +88,143 @@ class Iterator:
self._runtime_context.AssignConsumer(consumer)
self._iterator = self._runtime_context.GetConsumer()
# 用于将张量转换为 NumPy 数组
self._transform_tensor = lambda t: t.as_array()
# 当 output_numpy 为 False 时
if not output_numpy:
# 对张量进行转换
# do_copy 表示是否对张量进行复制
def _transform(t, do_copy):
array = t.as_array()
# 将 dtype 转换为 np.str_
if array.dtype.type is np.bytes_:
array = array.astype(np.str_)
if do_copy:
return Tensor(array)
# 直接返回张量的 NumPy 数组
return Tensor.from_numpy(array)
self._transform_tensor = lambda t: _transform(t, do_copy)
self.__index = 0
self.offload_model = None
# 获取一个 offload_model 对象，包含了识别到的计算图中需要 offloaded 操作的信息。
offload_model = offload.GetOffloadModel(consumer, self.__ori_dataset.get_col_names())
# See if GetOffloadModel identified any operations set to be offloaded.
# 检查 offload_model 的 transform_list 是否为空
if offload_model.transform_list != []:
# 如果不为空，说明有操作被识别为可以 offloaded
# 检查数据集中是否存在需要 concat 或 zip 的操作
offload.check_concat_zip_dataset(self.__ori_dataset)
self.offload_model = offload_model
# 将当前对象的引用添加到 ITERATORS_LIST 中
ITERATORS_LIST.append(weakref.ref(self))
# 调用 _unset_iterator_cleanup 函数取消迭代器的自动清理
_unset_iterator_cleanup()
def __iter__(self):
return self
# 定义stop函数，用于手动终止Python迭代器，而不是依靠外部销毁
def stop(self):
"""
Manually terminate Python iterator instead of relying on out of scope destruction.
"""
# 判断是否有_runtime_context属性，且不为空
if hasattr(self, '_runtime_context') and self._runtime_context:
# 判断是否有_iterator属性，且不为空
if hasattr(self, '_iterator') and self._iterator:
# 终止运行时上下文
self._runtime_context.Terminate()
# 删除_iterator
del self._iterator
# 删除_runtime_context
del self._runtime_context
# 删除dataset
del self.dataset
# get weakref which is dead
# 创建dead_iterator列表
dead_iterator = []
# 遍历ITERATORS_LIST
for index, item in enumerate(ITERATORS_LIST):
# item() == None indicate the object is dead
# id(item()) == id(self) indicate del self
if item() is None or id(item()) == id(self):
# 将dead_iterator中的索引添加到dead_iterator列表中
dead_iterator.append(index)
# del dead weakref
for index in reversed(dead_iterator):
# 从ITERATORS_LIST中删除dead_iterator中的索引
ITERATORS_LIST.pop(index)
# 调用 stop 方法停止迭代
def release(self):
self.stop()
# 在对象被销毁时释放当前迭代器
def __del__(self):
self.release()
@abstractmethod
# 用于获取下一个元素
def _get_next(self):
# 抛出一个 RuntimeError 异常
# 应该使用子类 DatasetIterator 的 get_next 方法来获取下一个元素
raise RuntimeError("Calling base class Iterator's get_next is invalid.")
def __next__(self):
# 检查 self._runtime_context 是否为 None
if not self._runtime_context:
# 如果为 None，说明当前迭代器没有正在运行的 C++ 管道，日志警告会被输出
# 表明迭代器可能已经被停止或者 C++ 管道崩溃
logger.warning("Iterator does not have a running C++ pipeline." +
"It might because Iterator stop() had been called, or C++ pipeline crashed silently.")
raise RuntimeError("Iterator does not have a running C++ pipeline.")
# 从 self._get_next() 方法中获取下一个元素
data = self._get_next()
# 如果 data 为 None，说明已经到达了数据集的末尾
if not data:
# 如果这是第一次遍历，则日志警告可能会被输出，表明没有可用的记录
if self.__index == 0:
logger.warning("No records available.")
# 如果数据集没有提供 dataset_size 属性，那么将其设置为当前索引 self.__index
if self.__ori_dataset.dataset_size is None:
self.__ori_dataset.dataset_size = self.__index
# 迭代结束
raise StopIteration
# 遍历次数 + 1
self.__index += 1
# 检查 self.offload_model 是否为 None
if self.offload_model is not None:
# 如果不为 None，说明存在需要 offloaded 操作
data = offload.apply_offload_iterators(data, self.offload_model)
# 返回处理后的数据集元素
return data
def __deepcopy__(self, memo):
return self
# 用于获取管道信息
def _getters(self):
"""
Get pipeline information.
"""
# 初始化一个对象
getter = cde.TreeGetters()
# 初始化树
getter.Init(self.ir_tree)
# 将获取器分配给 self._runtime_context 的消费者
self._runtime_context.AssignConsumer(getter)
# 从获取器中获取列名并将其存储在 self._col_names 变量中
self._col_names = getter.GetColumnNames()
# 用于获取数据集的列名
def get_col_names(self):
"""
Get names of the columns in the dataset
@ -178,6 +233,7 @@ class Iterator:
self._getters()
return self._col_names
# 用于将迭代器重置到给定的步数
def _reset(self, step):
"""
Reset the iterator to the given step number.
@ -191,8 +247,10 @@ class Iterator:
class DictIterator(Iterator):
"""
The derived class of Iterator with dict type.
用于将数据集中的元素转换为字典类型
"""
# 定义一个函数_get_next，用于返回数据集中下一个记录作为字典
def _get_next(self):
"""
Returns the next record in the dataset as dictionary
@ -201,25 +259,30 @@ class DictIterator(Iterator):
Dict, the next record in the dataset.
"""
try:
# 使用self._iterator.GetNextAsMap()获取下一个记录，并将其中的每个元素转换为字典
return {k: self._transform_tensor(t) for k, t in self._iterator.GetNextAsMap().items()}
except RuntimeError as err:
# maybe "Out of memory" / "MemoryError" error
# 捕获可能出现的“Out of memory” /“MemoryError”错误
err_info = str(err)
if err_info.find("Out of memory") >= 0 or err_info.find("MemoryError") >= 0:
logger.critical("Memory error occurred, process will exit.")
os.kill(os.getpid(), signal.SIGKILL)
raise err
class TupleIterator(Iterator):
"""
The derived class of Iterator with list type.
用于将数据集中的元素转换为元组类型
"""
def __init__(self, dataset, columns=None, num_epochs=-1, output_numpy=False, do_copy=True):
# 如果columns不为空，则判断是否为list类型，不是则转换为list
if columns is not None:
if not isinstance(columns, list):
columns = [columns]
# 调用父类构造函数，传入dataset和columns
dataset = dataset.project(columns)
super().__init__(dataset, num_epochs, output_numpy, do_copy)
@ -231,12 +294,15 @@ class TupleIterator(Iterator):
List, the next record in the dataset.
"""
# 调用父类的_get_next方法，返回list类型的数据
return [self._transform_tensor(t) for t in self._iterator.GetNextAsList()]
class DummyIterator:
"""
A DummyIterator only work when env MS_ROLE="MS_PSERVER" or MS_ROLE="MS_SCHED"
用于模拟数据集中的迭代器。
这个类主要用于在特定环境中工作，即 MS_ROLE="MS_PSERVER" 或 MS_ROLE="MS_SCHED"。
"""
def __init__(self, dataset, mode):
@ -246,19 +312,30 @@ class DummyIterator:
self.fetched_first = False
def __get_tensor(self):
# 定义一个空列表，用于存放tensor
tensor_row = []
# 遍历shapes和types，将每个shape和type对应的数据转换为tensor
for np_shape, np_type in zip(self.shapes, self.types):
# 创建一个指定shape和type的numpy数组
input_np = np.zeros(np_shape, np_type)
# 将numpy数组转换为tensor
tensor = Tensor(input_np)
# 将tensor添加到tensor_row列表中
tensor_row.append(tensor)
# 返回tensor_row列表
return tensor_row
def __iter__(self):
return self
def __next__(self):
# 当 mode 为 "tuple" 时
if self.mode == "tuple":
# 检查 fetched_first 是否为 True
if not self.fetched_first:
# 调用 __get_tensor 方法获取一个包含多个张量元组的列表
# 每个张量元组表示数据集中的一个记录，并将 fetched_first 设置为 True
self.fetched_first = True
return self.__get_tensor()
# 否则，抛出 StopIteration 异常
raise StopIteration()

177
mindspore/python/mindspore/dataset/engine/offload.py
View File

@ -25,50 +25,71 @@ from mindspore.ops import operations as P
from mindspore.ops.primitive import constexpr
# 定义一个函数，检查是否需要将数据集卸载，并应用转换
def check_add_offload_sink_mode(dataset, dataset_helper, network):
"""
Check if any map operations were removed to be offloaded and apply the transforms if so.
检查是否需要将数据集卸载，并应用转换
"""
if hasattr(dataset, '__no_send__'):
# Dataset was not sent to device. Skip adding offload.
# 数据集未被发送到设备。跳过添加卸载
return network
offload_model = dataset.__transfer_dataset__.get_offload_model()
# See if the offload pass identified any operations to be offloaded
# 检查是否 identified any operations to be offloaded
if offload_model.transform_list != []:
# 检查是否需要将zip和concat数据集卸载
check_concat_zip_dataset(dataset.__transfer_dataset__)
# 应用转换
network = ApplyPreTransform(offload_model, network)
return network
# 定义一个函数，检查传入的dataset是否是拼接或者压缩的
def check_concat_zip_dataset(dataset):
"""
Check if dataset is concatenated or zipped.
检查传入的dataset是否是拼接或者压缩的
"""
# 遍历 dataset 的子节点，直到 dataset 为空为止
while dataset:
# 如果子节点的数量大于1
if len(dataset.children) > 1:
# 抛出一个 RuntimeError 异常，表示不支持拼接或压缩的dataset。
raise RuntimeError("Offload module currently does not support concatenated or zipped datasets.")
if dataset.children:
# 将 dataset 设置为第一个子节点，并继续遍历
dataset = dataset.children[0]
continue
dataset = dataset.children
# 定义一个函数，用于获取输入列的索引
def get_col_idxs(node_cols, ds_cols):
"""
Get the index(es) of the input column(s) from the dataset
用于获取输入列的索引
"""
col_idxs = []
non_exist_cols = []
# temporary error if multiple node columns
# 处理多个节点列的错误
if len(node_cols) > 1:
# node_cols 的数量大于1时，它会抛出一个 RuntimeError 异常，表示不支持多个节点列的映射操作。
raise RuntimeError(
"Offload hardware accelerator currently does not support map operations with multiple input columns")
# 遍历节点列
for node_col in node_cols:
# 如果节点列在数据集中
if node_col in ds_cols:
# 将节点列的索引添加到col_idxs中
col_idxs.append(ds_cols.index(node_col))
else:
# 否则将节点列添加到non_exist_cols中
non_exist_cols.append(node_col)
# 如果non_exist_cols不为空
if non_exist_cols:
# 抛出运行时错误
raise RuntimeError(
("The following input column(s) for an offloaded map operation "
"do not exist: {}").format(non_exist_cols))
@ -76,65 +97,90 @@ def get_col_idxs(node_cols, ds_cols):
return col_idxs
# 定义一个函数，用于应用非sink模式管道中的offload
def apply_offload_iterators(data, offload_model):
"""
Apply offload for non sink mode pipeline.
用于应用非sink模式管道中的offload
"""
# 创建一个空列表，用于存储非张量索引
non_tensor_idxs = []
# 遍历data中的每一个元素
for i, _ in enumerate(data):
# 如果data中的元素不是张量，则将其转换为张量，并将索引添加到non_tensor_idxs中
if not isinstance(data[i], Tensor):
data[i] = Tensor(data[i], dtype=mstype.float32)
non_tensor_idxs.append(i)
# 将data传入offload_model中，并将返回值转换为列表
data = offload_model(data)
data = list(data)
# 遍历non_tensor_idxs中的每一个索引
for idx in non_tensor_idxs:
# 将data中的元素转换为numpy，并将转换后的结果赋值给data
data[idx] = data[idx].asnumpy()
# 返回data
return data
@constexpr
# 检查输入的维度是否符合操作的要求
def check_input_dims(x_shape, required_dim, offload_op_name):
"""
Check if input has the required number of dimensions for the operation.
检查输入的维度是否符合操作的要求
"""
input_dim = len(x_shape)
# 当输入数据的维度与所需维度不相同时
if input_dim is not required_dim:
raise ValueError("For %s offload operation, the dimension of input should be %d, but got %d." %
(offload_op_name, required_dim, input_dim))
# 定义一个函数，用于调整输入参数的参数范围
def assign_min_max_params(in_params, center=1):
"""
Adjust input parameters for ops.
用于调整输入参数的参数范围
"""
# 判断输入参数是否为列表或元组
if isinstance(in_params, (list, tuple)):
# 将输入参数的第一个元素赋值给min_param
min_param = in_params[0]
# 将输入参数的第二个元素赋值给max_param
max_param = in_params[1]
else:
# 将输入参数的绝对值赋值给min_param
min_param = max(0, center - in_params)
# 将输入参数加上中心参数赋值给max_param
max_param = center + in_params
# 返回min_param和max_param
return min_param, max_param
class ApplyPreTransform(nn.Cell):
"""
Concatenates offload model with network.
用于将预处理模型与网络连接起来
"""
def __init__(self, transform, model):
# 调用父类构造函数
super(ApplyPreTransform, self).__init__(auto_prefix=False, flags=model.get_flags())
self.transform = transform
self.model = model
def construct(self, *x):
data = []
# 将输入数据分成多个数据列
for data_col in x:
data.append(data_col)
# 将每个数据列传递给预处理模型
data = self.transform(data)
# 将预处理模型的输出传递给网络
data = self.model(*data)
return data
@ -143,19 +189,22 @@ class ApplyPreTransform(nn.Cell):
class IdentityCell(nn.Cell):
"""
Applies identity transform on given input tensors.
用于应用恒等变换
"""
def __init__(self):
# 调用父类构造
super(IdentityCell, self).__init__()
self.identity = P.Identity()
def construct(self, x):
# 用于组合恒等变换。它接收一个输入张量 x，并返回 x 本身
return self.identity(x)
class RandomHorizontalFlip(nn.Cell):
"""
Applies Random Horizontal Flip transform on given input tensors.
# 用于应用随机水平翻转变换
"""
def __init__(self, prob):
@ -171,24 +220,34 @@ class RandomHorizontalFlip(nn.Cell):
def construct(self, x):
# 将x转换为float32类型
x = self.cast(x, mstype.float32)
# 获取x的形状
x_shape = self.shape(x)
# 检查输入维度是否为4
check_input_dims(x_shape, 4, 'RandomHorizontalFlip')
# 获取x的bs，h，w，c
bs, h, w, c = x_shape
# 生成一个bs*1的随机数，范围为[0, 1]
flip_rand_factor = Tensor(np.random.uniform(size=(bs, 1)), dtype=mstype.float32)
# 将随机数转换为float32类型，并将其转换为bs*h*w*c的形状
flip_rand_factor = self.cast((self.prob > flip_rand_factor), mstype.float32)
flip_rand_factor = self.reshape(C.repeat_elements(flip_rand_factor, rep=(h*w*c)), (bs, h, w, c))
# 将x水平翻转
x_flip = self.h_flip(x)
# 将x_flip和flip_rand_factor相乘，再加上x和(1 - flip_rand_factor)相乘的结果
x = self.mul(x_flip, flip_rand_factor) + self.mul((1 - flip_rand_factor), x)
# 返回x
return x
class RandomVerticalFlip(nn.Cell):
"""
Applies Random Vertical Flip transform on given input tensors.
用于应用随机垂直翻转变
"""
def __init__(self, prob):
@ -204,24 +263,34 @@ class RandomVerticalFlip(nn.Cell):
def construct(self, x):
# 将x转换为float32类型
x = self.cast(x, mstype.float32)
# 获取x的形状
x_shape = self.shape(x)
# 检查输入的维度是否为4
check_input_dims(x_shape, 4, 'RandomVerticalFlip')
# 获取x的bs, h, w, c
bs, h, w, c = x_shape
# 生成一个bs*1的随机数组，范围为0-1
flip_rand_factor = Tensor(np.random.uniform(size=(bs, 1)), dtype=mstype.float32)
# 将随机数组转换为float32类型，并将其转换为bs*h*w*c的形状
flip_rand_factor = self.cast((self.prob > flip_rand_factor), mstype.float32)
flip_rand_factor = self.reshape(C.repeat_elements(flip_rand_factor, rep=(h*w*c)), (bs, h, w, c))
# 将x水平翻转
x_flip = self.h_flip(x)
# 将x_flip和flip_rand_factor相乘，再加上x和(1-flip_rand_factor)相乘的结果
x = self.mul(x_flip, flip_rand_factor) + self.mul((1 - flip_rand_factor), x)
# 返回x
return x
class GenerateRandBatch(nn.Cell):
"""
Generate batch with random values uniformly selected from [degree_min, degree_max].
用于生成具有随机值的批次。
"""
def __init__(self):
@ -232,11 +301,17 @@ class GenerateRandBatch(nn.Cell):
def construct(self, degree_min, degree_max, check_rand, shape):
# 获取输入的shape
bs, h, w, c = shape
# 生成一个bs大小的随机数组
rand_factor = Tensor(np.random.uniform(size=(bs, 1)), dtype=mstype.float32)
# 将随机数组归一化到[degree_min, degree_max]
rand_factor = degree_min + (degree_max - degree_min)*rand_factor
# 生成一个bs大小的全为degree_min的数组
degree_factor = degree_min * self.ones((bs, 1), mstype.float32)
# 将随机数组和全为degree_min的数组进行比较，取其中随机数组
rand_factor = (check_rand * degree_factor) + (~check_rand * rand_factor)
# 将随机数组扩展到(bs, h, w, c)的形状
rand_factor = self.reshape(C.repeat_elements(rand_factor, rep=(h*w*c)), (bs, h, w, c))
return rand_factor
@ -245,6 +320,7 @@ class GenerateRandBatch(nn.Cell):
class RandomColorAdjust(nn.Cell):
"""
Applies Random Color Adjust transform on given input tensors.
用于应用随机颜色调整变换。
"""
def __init__(self, brightness, contrast, saturation, hue):
@ -255,11 +331,13 @@ class RandomColorAdjust(nn.Cell):
self.sa_min, self.sa_max = assign_min_max_params(saturation)
self.hue_min, self.hue_max = assign_min_max_params(hue)
# 检查随机亮度、对比度、饱和度和色调是否相等
self.check_rand_br = Tensor(self.br_min == self.br_max)
self.check_rand_cont = Tensor(self.cont_min == self.cont_max)
self.check_rand_sa = Tensor(self.sa_min == self.sa_max)
self.check_rand_hue = Tensor(self.hue_min == self.hue_max)
# 定义操作
self.cast = P.Cast()
self.shape = P.Shape()
self.reshape = P.Reshape()
@ -267,6 +345,7 @@ class RandomColorAdjust(nn.Cell):
self.expand_dims = P.ExpandDims()
self.mul = P.Mul()
# 定义操作
self.mean = P.ReduceMean()
self.argmaxvalue = P.ArgMaxWithValue(axis=3, keep_dims=False)
self.argminvalue = P.ArgMinWithValue(axis=3, keep_dims=False)
@ -284,64 +363,94 @@ class RandomColorAdjust(nn.Cell):
self.ones = P.Ones()
self.reshape = P.Reshape()
# 生成随机batch
self.generate_rand_batch = GenerateRandBatch()
# 用于组合随机颜色调整变换。
def construct(self, x):
x = self.cast(x, mstype.float32)
x_shape = self.shape(x)
# 检查维度是否与所需维度相等
check_input_dims(x_shape, 4, 'RandomColorAdjust')
bs, h, w, c = x_shape
# 生成随机亮度
br_rand_factor = self.generate_rand_batch(self.br_min, self.br_max, self.check_rand_br, x_shape)
# 生成随机对比度
cont_rand_factor = self.generate_rand_batch(self.cont_min, self.cont_max, self.check_rand_cont, x_shape)
# 生成随机饱和度
sat_rand_factor = self.generate_rand_batch(self.sa_min, self.sa_max, self.check_rand_sa, x_shape)
# 将输入图像拆分成RGB三个通道
r, g, b = self.unstack(x)
# 将RGB转换成灰度图
x_gray = 0.2989 * r + 0.587 * g + 0.114 * b
# 计算灰度图的均值
x_gray_mean = self.expand_dims(self.mean(x_gray, (1, 2)) + 0.5, -1)
# 将灰度图的均值扩展到RGB三个通道
x_gray_mean = self.reshape(C.repeat_elements(x_gray_mean, rep=(h*w*c)), (bs, h, w, c))
# 将输入图像扩展到RGB三个通道
x_gray = C.repeat_elements(self.expand_dims(x_gray, -1), rep=c, axis=-1)
# Apply brightness
# 将输入张量 x 与亮度随机因子相乘
x = self.mul(x, br_rand_factor)
# 应用裁剪操作，以确保输出的张量值在 0 到 255 之间
x = C.clip_by_value(x, 0.0, 255.0)
# Apply contrast
# 将输入张量 x 与对比度随机因子相乘
x = self.mul(x, cont_rand_factor) + self.mul((1 - cont_rand_factor), x_gray_mean)
# 应用裁剪操作，以确保输出的张量值在 0 到 255 之间
x = C.clip_by_value(x, 0.0, 255.0)
# Apply saturation
# 将输入张量 x 与饱和度随机因子相乘
x = self.mul(x, sat_rand_factor) + self.mul((1 - sat_rand_factor), x_gray)
# 应用裁剪操作，以确保输出的张量值在 0 到 255 之间
x = C.clip_by_value(x, 0.0, 255.0)
# Apply Hue Transform
# Convert tensor from rgb to hsv
# 用于应用随机颜色调整变换中的颜色哈希变换
# 将输入张量 x 从 RGB 格式转换为 HSV 格式
r, g, b = self.unstack(x)
# 计算最大值、最小值和 HSV 值的占比
max_c, max_v = self.argmaxvalue(x)
_, min_v = self.argminvalue(x)
hsv_denum = max_v - min_v + self.epsilon
# 根据这些值计算 HSV 空间的三个分量
h1 = self.fmod(((b - g) * 60 / hsv_denum), 360)
h2 = (g - r) * 60 / hsv_denum + 120
h3 = (r - g) * 60 / hsv_denum + 240
# 将这三个分量组合成一个新的张量
hue = self.squeeze(self.gatherd(self.stack((h1, h2, h3)), 0, self.expand_dims(max_c, 0)))
s = self.cast((max_v > 0), mstype.float32) * (1 - min_v / (max_v + self.epsilon))
v = self.cast(max_v, mstype.float32)
# Adjust hue
# 调整亮度
# 用于根据随机生成的 hue_rand_factor 调整输入张量的颜色
# 创建一个范围在 hue_min 和 hue_max 之间的随机张量
hue_rand_factor = Tensor(np.random.uniform(size=(bs, 1)), dtype=mstype.float32)
hue_rand_factor = self.hue_min + (self.hue_max - self.hue_min)*hue_rand_factor
degree_factor = self.hue_min * self.ones((bs, 1), mstype.float32)
# 将其与 degree_factor（全为 1 的张量）相加
hue_rand_factor = (self.check_rand_hue * degree_factor) + (~self.check_rand_hue * hue_rand_factor)
hue_rand_factor = self.reshape(C.repeat_elements(hue_rand_factor, rep=(h*w)), (bs, h, w))
# 最后将结果与原始的 hue 值相加
hue = hue + (hue_rand_factor * 360.0)
# Convert tensor from hsv to rgb
# 用于将输入张量从 HSV 格式转换回 RGB 格式
# 计算 HSV 空间的三个分量（色调、饱和度和明度）
h_ = (hue - self.floor(hue / 360.0) * 360.0) / 60.0
c = self.mul(s, v)
x_ = self.mul(c, (1 - self.abs(self.fmod(h_, 2) - 1)))
zero_tensor = self.zeros_like(c)
# 根据这些值计算 RGB 空间的三个分量
y = self.stack((self.stack_axis_1((c, x_, zero_tensor)), self.stack_axis_1((x_, c, zero_tensor)),
self.stack_axis_1((zero_tensor, c, x_)), self.stack_axis_1((zero_tensor, x_, c)),
self.stack_axis_1((x_, zero_tensor, c)), self.stack_axis_1((c, zero_tensor, x_)),
@ -349,8 +458,10 @@ class RandomColorAdjust(nn.Cell):
index = self.expand_dims(self.floor(h_), 1)
index = self.expand_dims(C.repeat_elements(index, 3, 1), 0)
index = self.cast(index, mstype.int32)
x = self.squeeze(self.gatherd(y, 0, index))
x = x + self.reshape(C.repeat_elements((v - c), rep=(3)), self.shape(x))
# 将结果乘以 255 并裁剪到 0 到 255 之间，返回转换后的张量
x = self.transpose(x, (0, 2, 3, 1)) * 255.0
x = C.clip_by_value(x, 0.0, 255.0)
@ -360,6 +471,7 @@ class RandomColorAdjust(nn.Cell):
class RandomSharpness(nn.Cell):
"""
Applies Random Sharpness transform on given input tensors.
用于应用随机锐化变换。
"""
def __init__(self, degrees):
@ -372,6 +484,7 @@ class RandomSharpness(nn.Cell):
self.degree_min = max(0, 1 - degrees)
self.degree_max = 1 + degrees
# 定义操作
self.cast = P.Cast()
self.shape = P.Shape()
self.ones = P.Ones()
@ -380,54 +493,75 @@ class RandomSharpness(nn.Cell):
self.mul = P.Mul()
self.transpose = P.Transpose()
# 检查输入的度数是否相等
self.check_rand = Tensor(self.degree_min == self.degree_max)
# 初始化权重
self.weight = np.array([[1, 1, 1], [1, 5, 1], [1, 1, 1]])/13.0
self.weight = np.repeat(self.weight[np.newaxis, :, :], 3, axis=0)
self.weight = np.repeat(self.weight[np.newaxis, :, :], 3, axis=0)
self.weight = Tensor(self.weight, mstype.float32)
# 初始化卷积层
self.filter = P.Conv2D(out_channel=3, kernel_size=(3, 3), pad_mode='pad', pad=1)
def construct(self, x):
# 将输入x转换为float32类型
x = self.cast(x, mstype.float32)
# 获取输入x的形状
x_shape = self.shape(x)
# 检查输入x的形状是否符合要求
check_input_dims(x_shape, 4, 'RandomSharpness')
# 获取输入x的batch_size，h，w，c
bs, h, w, c = x_shape
# 生成一个bs*1的随机数，范围在[0, 1]
degree_rand_factor = Tensor(np.random.uniform(size=(bs, 1)), dtype=mstype.float32)
# 将degree_rand_factor的值设置为[self.degree_min, self.degree_max]
degree_rand_factor = self.degree_min + (self.degree_max - self.degree_min)*degree_rand_factor
# 将degree_rand_factor的值设置为[self.degree_min, self.degree_min]
degree_factor = self.degree_min * self.ones((bs, 1), mstype.float32)
# 将degree_rand_factor的值设置为[self.degree_min, self.degree_max]
degree_rand_factor = (self.check_rand * degree_factor) + (~self.check_rand * degree_rand_factor)
# 将degree_rand_factor的值重复h*w*c次，并转换为bs*h*w*c的形状
degree_rand_factor = self.reshape(C.repeat_elements(degree_rand_factor, rep=(h*w*c)), (bs, h, w, c))
# 将输入x转换为bs*c*h*w的形状，并使用self.weight对输入x进行卷积操作
x_sharp = self.filter(self.transpose(x, (0, 3, 1, 2)), self.weight)
# 将x_sharp转换为bs*h*w*c的形状
x_sharp = self.transpose(x_sharp, (0, 2, 3, 1))
# 将输入x和x_sharp进行乘法操作，并使用degree_rand_factor和1-degree_rand_factor进行加法操作
x = self.mul(x, degree_rand_factor) + self.mul((1 - degree_rand_factor), x_sharp)
# 将x的值限制在[0, 255]之间
x = C.clip_by_value(x, 0.0, 255.0)
# 返回处理后的x
return x
class Rescale(nn.Cell):
"""
Applies Rescale transform on given input tensors.
用于应用缩放变换。
"""
def __init__(self, rescale, shift):
super(Rescale, self).__init__()
# 将 rescale 和 shift 转换为张量
self.rescale = Tensor(rescale, dtype=mstype.float32)
self.shift = Tensor(shift, dtype=mstype.float32)
# 定义操作
self.cast = P.Cast()
self.mul = P.Mul()
def construct(self, x):
# 将x转换为float32类型
x = self.cast(x, mstype.float32)
# 将x乘以rescale，再加上shift
x = x * self.rescale + self.shift
return x
@ -436,47 +570,60 @@ class Rescale(nn.Cell):
class HwcToChw(nn.Cell):
"""
Applies Channel Swap transform on given input tensors.
用于交换输入张量的通道。
"""
def __init__(self):
super(HwcToChw, self).__init__()
# 定义操作
self.trans = P.Transpose()
self.shape = P.Shape()
def construct(self, x):
# 获取输入x的形状
x_shape = self.shape(x)
# 检查输入x的形状是否符合要求
check_input_dims(x_shape, 4, 'HwcToChw')
# 返回转置后的x
return self.trans(x, (0, 3, 1, 2))
class Normalize(nn.Cell):
"""
Applies Normalize transform on given input tensors.
用于应用归一化变换。
"""
def __init__(self, mean, std):
super(Normalize, self).__init__()
# 将 mean 和 std 转换为张量
self.mean = Tensor(mean, mstype.float32)
self.std = Tensor(std, mstype.float32)
# 定义操作
self.sub = P.Sub()
self.div = P.Div()
self.cast = P.Cast()
def construct(self, x):
# 将x转换为float32类型
x = self.cast(x, mstype.float32)
# 将x减去均值
x = self.sub(x, self.mean)
# 将x除以标准差
x = self.div(x, self.std)
# 返回处理后的x
return x
class TypeCast(nn.Cell):
"""
Applies TypeCast transform on given input tensors.
用于应用类型转换。
"""
def __init__(self, data_type_str):
super(TypeCast, self).__init__()
# 定义操作
self.cast = P.Cast()
self.data_type = mstype.typing.str_to_type(data_type_str)
@ -485,13 +632,16 @@ class TypeCast(nn.Cell):
return self.cast(x, self.data_type)
# 定义一个OffloadModel类
class OffloadModel():
# 初始化函数，参数func为要调用的函数，args_names为函数参数的名称
def __init__(self, func, args_names=None):
self.func = func
self.args_names = args_names
# Dictionary connecting operation name to model
# 定义了一个名为 op_to_model 的字典，用于将操作名称映射到对应的模型。
op_to_model = {
"HWC2CHW": OffloadModel(HwcToChw),
"HwcToChw": OffloadModel(HwcToChw),
@ -508,6 +658,7 @@ op_to_model = {
class GetModelFromJson2Col(nn.Cell):
"""
Generates offload ME model from offload JSON file for a single map op.
于从Offload JSON文件中生成Offload ME模型。
"""
def __init__(self, json_offload, col_idxs):
@ -515,21 +666,27 @@ class GetModelFromJson2Col(nn.Cell):
self.col_idxs = col_idxs
self.me_ops = []
self.input_cols = []
# 如果json_offload不为空，则从offload_hw_accelerator中获取操作
if json_offload is not None:
offload_ops = json_offload["operations"]
# 遍历操作，获取操作模型
for op in offload_ops:
name = op["tensor_op_name"]
args = op["tensor_op_params"]
op_model = op_to_model[name]
op_model_inputs = []
# 如果操作模型有参数，则获取参数
if op_model.args_names is not None:
for arg_key in op_model.args_names:
op_model_inputs.append(args[arg_key])
# 获取操作模型
self.me_ops.append(op_model.func(*op_model_inputs))
else:
# 如果offload_hw_accelerator为空，则抛出异常
raise RuntimeError("Offload hardware accelarator cannot be applied for this pipeline.")
# 创建序列模型
self.cell = nn.SequentialCell(self.me_ops)
def construct(self, x):
@ -543,20 +700,30 @@ class GetModelFromJson2Col(nn.Cell):
class GetOffloadModel(nn.Cell):
"""
Generates offload ME model.
用于生成Offload ME模型。
"""
def __init__(self, dataset_consumer, ds_cols):
super(GetOffloadModel, self).__init__()
self.transform_list = []
# 从dataset_consumer中获取offload信息
json_offload = json.loads(dataset_consumer.GetOffload())
# 如果offload信息不为空
if json_offload is not None:
# 遍历offload信息中的每一个节点
for node in json_offload:
# 如果节点的操作类型为Map
if node["op_type"] == 'Map':
# 获取输入列的索引
ds_col_idxs = get_col_idxs(node["input_columns"], ds_cols)
# 将节点和输入列索引添加到transform_list中
self.transform_list.append(GetModelFromJson2Col(node, ds_col_idxs))
# 将transform_list反转
self.transform_list.reverse()
def construct(self, x):
# 遍历transform_list中的每一个transform，对x进行转换
for transform in self.transform_list:
x = transform(x)
# 返回转换后的x
return x

70
mindspore/python/mindspore/dataset/engine/queue.py
View File

@ -30,26 +30,30 @@ from ..transforms.py_transforms_util import ExceptionHandler
class _SharedQueue(multiprocessing.queues.Queue):
"""
Class to implement a queue using shared memory for better performance.
提高队列操作的性能，通过使用共享内存来减少进程之间的数据传输。
Args:
size: Number of elements in the queue.
copy_out: Flag to indidcate whether an extra copy should be done before returning. If data will immediately be
copied before returning, then this can be set to False.
max_rowsize: Maximum size of any element in the Queue in MB.
size: 队列中元素的个数。
copy_out: 一个标志位，表示在返回数据之前是否需要进行一次额外的复制。如果数据立即被复制，可以设置为False。
max_rowsize: 队列中任何元素的最大大小（以MB为单位）。
"""
def __init__(self, size, copy_out=False, max_rowsize=6):
# 调用父类构造函数
super().__init__(size, ctx=multiprocessing.get_context())
# 将copy_out标志位设置为False
self.copy_out = copy_out
# change max_rowsize in MB into bytes
# 计算seg_size，将其设置为max_rowsize（以MB为单位）乘以1024*1024
self.seg_size = max_rowsize * 1024 * 1024
##pipe can hold up to 65,636 bytes at a time
# 计算min_shared_mem，将其设置为10000字节
self.min_shared_mem = 10000
self.shm_list = []
self.shm_list = [] # 用于存储共享内存段
self.seg_pos = 0
# num_seg has to be 2 more than the queue size. We can have remote worker filling a buffer, main process
# reading a buffer and also have a full queue of buffers in the meta-data queue
# num_seg需要设置为size加2。这样，我们可以有远程工作进程填充缓冲区，主进程读取缓冲区，以及一个完整的缓冲区队列在元数据队列中。
self.num_seg = size + 2
self.data_immediate = 0
self.data_shared = 1
@ -57,9 +61,12 @@ class _SharedQueue(multiprocessing.queues.Queue):
try:
for _ in range(self.num_seg):
# 分配num_seg个元素，每个元素的大小为seg_size
a = multiprocessing.Array("b", self.seg_size)
# 将分配的元素添加到shm_list中
self.shm_list.append(a)
except Exception:
# 如果分配失败，抛出异常
raise RuntimeError(
"_SharedQueue: Error allocating "
+ str(self.seg_size)
@ -68,41 +75,57 @@ class _SharedQueue(multiprocessing.queues.Queue):
+ " elements."
+ " This might be caused by insufficient shm, and the recommended shm size is at least 5 GB."
)
def put(self, data, timeout=None):
if isinstance(data, ExceptionHandler): # pylint: disable=too-many-nested-blocks
# 检查data是否是一个ExceptionHandler对象
if isinstance(data, ExceptionHandler):
# 如果是，则调用父类的put方法
super().put(data, timeout=timeout)
else:
# 否则，将数据处理成一个新的列表name_list
name_list = []
count = 0
start_bytes = 0
# 如果data不是一个tuple类型也不是np.ndarray类型
if not isinstance(data, tuple) and not isinstance(data, np.ndarray):
# 抛出一个类型异常
raise TypeError("return value of user defined python function in GeneratorDataset or"
" map should be numpy array or tuple of numpy array.")
# 如果data是一个np.ndarray类型
if isinstance(data, np.ndarray):
# 添加到name_list中，其中第一个元素是self.data_immediate，第二个元素是np.array(data)
name_list.append((self.data_immediate, np.array(data)))
else:
# 遍历data中的每个元素r
for r in data:
# the map:pyfunc is a yield generator which can't be serialize
# 如果r是一个types.GeneratorType对象抛出一个类型错误，因为无法将生成器对象序列化。
if isinstance(r, types.GeneratorType):
# 抛出一个类型异常，因为无法将生成器对象序列化。
raise TypeError("Can not pickle {} object, please verify pyfunc return with numpy array"
.format(type(r)))
# 检查r是否是一个np.ndarray类型，并且其大小是否大于min_shared_mem，并且start_bytes加上r.nbytes小于seg_size
if (isinstance(r, np.ndarray) and r.size > self.min_shared_mem
and start_bytes + r.nbytes < self.seg_size):
# need to convert start_bytes to offset in array
# 将start_bytes转换为相对于数组偏移量
start_offset = start_bytes
# 创建一个新的np.ndarray对象dest
dest = np.ndarray(r.shape, r.dtype, buffer=self.shm_list[self.seg_pos].get_obj(),
offset=start_offset)
# 将r的值复制到dest中
np.copyto(dest, r)
byte = r.nbytes
byte = 8 * ((byte + 7) // 8)
start_bytes += byte
# 将处理后的name_list添加到name_list中
name_list.append((self.data_shared, self.seg_pos, byte, r.dtype, r.shape))
count += 1
else:
# 如果r是一个np.ndarray类型，并且其大小大于min_shared_mem
if isinstance(r, np.ndarray) and r.size >= self.min_shared_mem:
# Only print out error the first time it happens
if self.print_error:
# 打印一条警告信息，只在第一次发生时打印
logger.warning(
"Using shared memory queue, but rowsize is larger than allocated memory "
+ "max_rowsize "
@ -110,47 +133,76 @@ class _SharedQueue(multiprocessing.queues.Queue):
+ " current rowsize "
+ str(start_bytes + r.nbytes)
)
# 将print_error设置为False
self.print_error = False
# 将处理后的r添加到name_list中
name_list.append((self.data_immediate, r))
super().put(name_list, timeout=timeout)
# note above could generate a queue full exception. It will be handled by teh caller
# only increment seg_pos after successfully adding to metadata queue
# 在处理数据时，可能会遇到队列满的情况。在这种情况下，只有成功添加到metadata_queue后，才会增加seg_pos。
# 如果start_bytes大于0
if start_bytes > 0:
# 将seg_pos加1，然后对num_seg取模
# 确保seg_pos始终在0到num_seg-1之间
self.seg_pos = (self.seg_pos + 1) % self.num_seg
# 从队列中获取数据，并将获取到的数据转换为np.ndarray对象
def get(self, timeout=None):
# 调用父类queue.Queue的get方法，获取数据
result = super().get(timeout=timeout)
# 如果获取到的数据是一个ExceptionHandler对象
if isinstance(result, ExceptionHandler):
# 直接返回该对象
return result
# 创建一个空列表r，用于存储从队列中获取到的数据。
r = []
start_bytes = 0
# 遍历result，处理每个数据
for x in result:
# 如果数据是以self.data_shared为前缀的，那么说明它是一个共享内存中的数据
if x[0] == self.data_shared:
# 获取数据
seg_pos = x[1]
byte = x[2]
dtype = x[3]
shape = x[4]
start_offset = start_bytes
b = self.shm_list[seg_pos]
# 获取buffer对象，并将其转换为numpy数组
data = np.ndarray(shape, dtype, buffer=b.get_obj(), offset=start_offset)
# 更新字节数
start_bytes += byte
# 判断是否需要复制输出
if self.copy_out:
# 复制输出
data2 = np.copy(data)
# 将复制后的数据添加到列表中
r.append(data2)
else:
# 将原始数据添加到列表中
r.append(data)
elif x[0] == self.data_immediate:
# 从result列表中获取数据，并将其添加到r列表中
r.append(x[1])
else:
# 抛出一个运行时异常
raise RuntimeError("SharedQueue, invalid entry in metadata.")
# 将r列表转换为元组，作为返回值
return tuple(r)
def __del__(self):
# 获取shm_list的长度
shm_list_len = len(self.shm_list)
# 从后往前遍历shm_list，删除每一个元素
for idx in range(shm_list_len):
del self.shm_list[shm_list_len - idx - 1]
# 删除shm_list
del self.shm_list
# 关闭文件
self.close()
# 结束线程
self.join_thread()

225
mindspore/python/mindspore/dataset/engine/samplers.py
View File

@ -29,6 +29,7 @@ from ..core import validator_helpers as validator
def select_sampler(num_samples, input_sampler, shuffle, num_shards, shard_id):
"""
Create sampler based on user input.
根据用户输入创建一个采样器。
Args:
num_samples (int): Number of samples.
@ -40,8 +41,16 @@ def select_sampler(num_samples, input_sampler, shuffle, num_shards, shard_id):
Returns:
Sampler, sampler selected based on user input.
"""
# 根据输入的参数，函数会根据不同的情况选择不同的采样器类型。
if input_sampler is not None:
# 当用户输入给定的采样器时，我们不再选择采样器类型。
# 意味着以下参数都为空： num_shards, shard_id, shuffle, num_samples
# 下面这个例子为：
# sampler = ds.DistributedSampler(num_shards=8, shard_id=3, shuffle=shuffle)
# data1 = ds.VOCDataset(voc_dir, decode=True, sampler=sampler, num_shards=4, shard_id=1)
# 在这种情况下，用户给出了与样本相关的不同参数，这些参数相互矛盾。
# 为了防止这种情况，如果这些参数都是None，只允许用户手动指定采样器。
# If the user provided a sampler, then it doesn't matter what the other args are because
# we are being asked specifically to use the given sampler.
# That means the following arguments: num_shards, shard_id, shuffle, num_samples should all
@ -50,35 +59,56 @@ def select_sampler(num_samples, input_sampler, shuffle, num_shards, shard_id):
# data1 = ds.VOCDataset(voc_dir, decode=True, sampler=sampler, num_shards=4, shard_id=1)
# In this case, the user has given different sample-related arguments that contradict each other.
# To prevent this, only allow the user to manually specify the sampler if those arguments are all None
# 检查input_sampler是否为内置的采样器类型，并且检查其他参数（如num_shards、shard_id和shuffle）是否为None。
if (isinstance(input_sampler, BuiltinSampler) and
(any(arg is not None for arg in [num_shards, shard_id, shuffle, num_samples]))):
# 如果时内置采样器类型且参数不为None，抛出一个异常
raise ValueError(
'Conflicting arguments during sampler assignments. num_samples: {}, num_shards: {},'
' shard_id: {}, shuffle: {}.'.format(num_samples, num_shards, shard_id, shuffle))
# 检查input_sampler是否为内置的采样器类型
if isinstance(input_sampler, BuiltinSampler):
return input_sampler
# 如果input_sampler不是str类型，是np.ndarray,list或tuple类型
if not isinstance(input_sampler, str) and isinstance(input_sampler, (np.ndarray, list, tuple)):
# 创建一个SubsetSampler对象，用于实现从给定序列的索引中采样的元素的功能
return SubsetSampler(input_sampler, num_samples)
if not isinstance(input_sampler, str) and validator.is_iterable(input_sampler):
# in this case, the user passed in their own sampler object that's not of type BuiltinSampler
# 返回一个IterSampler对象。这种情况下用户提供的采样器是一个可迭代的对象，但不继承自Sampler类。
# 这意味着这个对象可能不是我们期望的采样器类型，因此我们需要对其进行进一步处理。
return IterSampler(input_sampler, num_samples)
# 如果input_sampler是整型
if isinstance(input_sampler, int):
# 返回SubsetSampler对象
return SubsetSampler([input_sampler])
raise TypeError('Unsupported sampler object of type ({})'.format(type(input_sampler)))
# 检查shuffle是否为None。如果是，那么它会检查num_shards是否为None
if shuffle is None:
# 如果num_shards不为None，那么说明用户已经指定了分片采样
if num_shards is not None:
# If shuffle is not specified, sharding enabled, use distributed random sampler
shuffle = True
# 函数会启用分片随机采样并返回一个DistributedSampler对象。
return DistributedSampler(num_shards, shard_id, shuffle=shuffle, num_samples=num_samples)
# If shuffle is not specified, sharding disabled, use random sampler
# 检查num_samples是否不为None且不为0
if num_samples is not None and num_samples != 0:
# replacement设置为True。采样器会重复采样，直到采样完指定的数量
return RandomSampler(replacement=True, num_samples=num_samples)
# 返回一个只采样一次的RandomSampler对象
return RandomSampler(num_samples=num_samples)
# 根据shuffle和num_shards的值来决定使用哪种采样器
# 检查shuffle是否为True
if shuffle is True:
# 检查num_shards是否为None
if num_shards is not None:
# If shuffle enabled, sharding enabled, use distributed random sampler
# 如果num_shards不为None，那么说明用户已经指定了分片采样，因此函数会启用分片随机采样并返回一个DistributedSampler对象
return DistributedSampler(num_shards, shard_id, shuffle=shuffle, num_samples=num_samples)
# If shuffle enabled, sharding disabled, use random sampler
# 检查num_samples是否不为None且不为0。如果是，那么它返回一个RandomSampler对象
if num_samples is not None:
return RandomSampler(replacement=True, num_samples=num_samples)
return RandomSampler(num_samples=num_samples)
@ -86,14 +116,17 @@ def select_sampler(num_samples, input_sampler, shuffle, num_shards, shard_id):
# If shuffle disabled, sharding enabled, use distributed sequential sampler
return DistributedSampler(num_shards, shard_id, shuffle=shuffle, num_samples=num_samples)
# If shuffle disabled, sharding disabled, use sequential sampler
# 返回一个只采样一次的SequentialSampler对象
return SequentialSampler(num_samples=num_samples)
class BuiltinSampler:
"""
Base class for BuiltinSampler.
用于处理内置的采样器。
User should not extend this class.
用户不应该扩展这个类。
"""
def __init__(self, num_samples=None):
@ -107,6 +140,9 @@ class BuiltinSampler:
"""
Add a sub-sampler for given sampler. The parent will receive all data from the
output of sub-sampler sampler and apply its sample logic to return new samples.
用于向现有的采样器添加一个子采样器。采样器的子采样器可以是一个内置的采样器。
允许一个采样器接收另一个采样器的结果，并将它的采样逻辑应用到新样本上。这样，用户可以创
建一个复杂的采样器网络，以便根据需要灵活地采样数据。
Args:
sampler (Sampler): Object used to choose samples from the dataset. Only builtin
@ -118,13 +154,16 @@ class BuiltinSampler:
>>> sampler.add_child(ds.RandomSampler(num_samples=4))
>>> dataset = ds.Cifar10Dataset(cifar10_dataset_dir, sampler=sampler)
"""
# 如果self.child_sampler已经有一个子采样器
if self.child_sampler is not None:
# 抛出一个异常
raise RuntimeError("Cannot add child sampler, this sampler already has a child.")
self.child_sampler = sampler
def get_child(self):
"""
Get the child sampler of given sampler.
用于从给定的采样器中获取子采样器。
Returns:
Sampler, The child sampler of given sampler.
@ -137,23 +176,35 @@ class BuiltinSampler:
return self.child_sampler
def parse_child(self):
""" Parse the child sampler. """
""" Parse the child sampler.
解析子采样器。
"""
c_child_sampler = None
# 检查self.child_sampler是否为None
if self.child_sampler is not None:
# 解析子采样器
c_child_sampler = self.child_sampler.parse()
# 返回解析后的子采样器
return c_child_sampler
def parse_child_for_minddataset(self):
""" Parse the child sampler for MindRecord. """
""" Parse the child sampler for MindRecord.
与上面的parse_child方法类似，但是它主要用于处理MindDataset
"""
c_child_sampler = None
# 检查self.child_sampler是否为None
if self.child_sampler is not None:
# 解析子采样器
c_child_sampler = self.child_sampler.parse_for_minddataset()
# 返回解析后的子采样器
return c_child_sampler
# 用于检查采样器是否已经打乱数据，未实现的方法
def is_shuffled(self):
""" Not implemented. """
raise NotImplementedError("Sampler must implement is_shuffled.")
# 用于检查采样器是否已经分片，未实现的方法
def is_sharded(self):
""" Not implemented. """
raise NotImplementedError("Sampler must implement is_sharded.")
@ -212,17 +263,23 @@ class BuiltinSampler:
if child_samples is not None:
return min(self.num_samples, child_samples)
# 如果子采样器没有返回样本数，则返回父采样器设置的样本数
return self.num_samples
# 如果父采样器没有设置样本数，则返回子采样器返回的样本数
return child_samples
# 如果没有父采样器，则返回父采样器设置的样本数
return self.num_samples
class Sampler(BuiltinSampler):
"""
Base class for user defined sampler.
通用的采样器类。
A user defined sampler can be used with any existing dataset with sampler support.
用户可以继承这个类来创建自己的采样器。
A required _iter_() method should by overridden by the user for sample index generation.
An optional reset() method can be overridden for per repeat reset,
@ -239,6 +296,7 @@ class Sampler(BuiltinSampler):
"""
def __init__(self, num_samples=None):
# 调用父类构造函数
super().__init__(num_samples)
self.dataset_size = 0
self.child_sampler = None
@ -246,8 +304,11 @@ class Sampler(BuiltinSampler):
def __iter__(self):
"""
抽象方法
User defined iterator, must be overridden.
用户定义迭代器时，必须重写此方法。
_handshake is guaranteed to be called prior to iterator construction.
_handshake是一个私有方法，会在构建迭代器之前被调用
"""
raise NotImplementedError
@ -257,63 +318,92 @@ class Sampler(BuiltinSampler):
"""
# Initialization handshake callback
# 初始化 handshake 回调
# Do not override this method!
# 不要重写这个方法！
def _handshake(self, ds_size, num_samples):
# 设置数据集大小
self.dataset_size = ds_size
# 设置样本数量
self.num_samples = num_samples
# Indices fetcher
# Do not override this method!
# # 不要重写这个方法！
# pylint: disable=missing-docstring
# 定义一个函数_get_indices，用于获取索引
def _get_indices(self):
# 创建一个迭代器
sampler_iter = iter(self)
# 创建一个空列表
ret = []
# 循环num_samples次
for _ in range(self.num_samples):
try:
# 获取下一个索引
idx = next(sampler_iter)
# 将索引添加到ret中
ret.append(idx)
# 如果迭代结束，抛出StopIteration异常
except StopIteration:
break
# 将ret转换为numpy数组
indices = np.array(ret)
# 如果转换后的数组类型为object，抛出RuntimeError异常
if indices.dtype == object:
raise RuntimeError("Fetched indices can not be converted to a valid ndarray.")
# 返回转换后的数组
return indices
# Instance fetcher
# Do not override this method!
# 定义一个函数parse，用于解析sampler
def parse(self):
""" Parse the sampler."""
# 如果num_samples不为None，则将num_samples赋值给num_samples，否则赋值为0
num_samples = self.num_samples if self.num_samples is not None else 0
# 创建一个PreBuiltSamplerObj对象，参数为num_samples和self
c_sampler = cde.PreBuiltSamplerObj(num_samples, self)
# 解析子sampler
c_child_sampler = self.parse_child()
# 将子sampler添加到c_sampler中
c_sampler.add_child(c_child_sampler)
# 返回c_sampler
return c_sampler
# 用于将一个子采样器添加到当前采样器中
def add_child(self, sampler):
self.child_sampler = sampler
# 用于实现多级放样（MCMC）算法中的辅助采样器
def get_child(self):
return self.child_sampler
# 解析子采样器
def parse_child(self):
c_child_sampler = None
# 如果子采样器不为空
if self.child_sampler is not None:
# 解析子采样器
c_child_sampler = self.child_sampler.parse()
return c_child_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
if self.child_sampler is None:
return False
return self.child_sampler.is_shuffled()
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return False
return self.child_sampler.is_sharded()
# 返回此数据集的样本数量
def get_num_samples(self):
if self.num_samples is None:
return None
@ -323,6 +413,7 @@ class Sampler(BuiltinSampler):
class DistributedSampler(BuiltinSampler):
"""
A sampler that accesses a shard of the dataset, it helps divide dataset into multi-subset for distributed training.
用于分布式训练的子采样器。
Args:
num_shards (int): Number of shards to divide the dataset into.
@ -354,22 +445,29 @@ class DistributedSampler(BuiltinSampler):
"""
def __init__(self, num_shards, shard_id, shuffle=True, num_samples=None, offset=-1):
# 检查num_shards是否为整数
if not isinstance(num_shards, int):
raise TypeError("num_shards must be integer but was: {}.".format(num_shards))
# 检查shard_id是否为整数
if not isinstance(shard_id, int):
raise TypeError("shard_id must be integer but was: {}.".format(shard_id))
# 检查shuffle是否为布尔值
if not isinstance(shuffle, bool):
raise TypeError("shuffle must be a boolean value but was: {}.".format(shuffle))
# 检查num_samples是否为整数
if num_samples is not None:
# 判断num_samples是否为整数
if not isinstance(num_samples, int):
raise TypeError("num_samples must be integer but was: {}.".format(num_samples))
# 判断num_samples是否小于0或者大于INT64_MAX
if num_samples < 0 or num_samples > validator.INT64_MAX:
raise ValueError("num_samples exceeds the boundary between {} and {}(INT64_MAX)!"
.format(0, validator.INT64_MAX))
# 检查offset是否为整数
if not isinstance(offset, int):
raise TypeError("offset must be integer but was: {}.".format(offset))
@ -382,40 +480,61 @@ class DistributedSampler(BuiltinSampler):
def parse(self):
""" Parse the sampler."""
# 获取num_samples，如果没有指定，则默认为0
num_samples = self.num_samples if self.num_samples is not None else 0
# 获取shuffle，如果没有指定，则默认为True
shuffle = self.shuffle if self.shuffle is not None else True
# 获取offset，如果没有指定，则默认为-1
offset = self.offset if self.offset is not None else -1
# each time user calls create_dict_iterator() (to do repeat) sampler would get a different seed to shuffle
# 每次调用create_dict_iterator()（重复）时，sampler会得到一个不同的种子来shuffle
self.seed += 1
# 创建一个DistributedSamplerObj实例，参数分别为num_shards，shard_id，shuffle，num_samples，seed，offset，True
c_sampler = cde.DistributedSamplerObj(self.num_shards, self.shard_id,
shuffle, num_samples, self.seed, offset, True)
# 解析子采样器
c_child_sampler = self.parse_child()
# 将子采样器添加到父采样器中
c_sampler.add_child(c_child_sampler)
# 返回父采样器
return c_sampler
def parse_for_minddataset(self):
""" Parse the sampler for MindRecord."""
""" Parse the sampler for MindRecord.
用于为MindRecord数据集创建一个分布式采样器
"""
# 从self对象中获取num_samples和shuffle属性
# 如果num_samples为None，则将其设置为0
num_samples = self.num_samples if self.num_samples is not None else 0
# 如果shuffle为None，则将其设置为True
shuffle = self.shuffle if self.shuffle is not None else True
# 创建一个MindrecordDistributedSampler对象，参数为总分区数，当前分区ID，是否打乱数据，随机数种子，采样数量和采样偏移量
c_sampler = cde.MindrecordDistributedSampler(self.num_shards, self.shard_id, shuffle,
self.seed, num_samples, self.offset)
# 解析子采样器
c_child_sampler = self.parse_child_for_minddataset()
# 将子采样器添加到分布式采样器的children列表中
c_sampler.add_child(c_child_sampler)
# 设置分布式采样器的num_samples属性
c_sampler.set_num_samples(num_samples)
# 返回创建的分布式采样器对象
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
if self.child_sampler is None:
return self.shuffle
return self.child_sampler.is_shuffled()
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return self.num_shards > 1
return self.child_sampler.is_sharded()
# 用于设置分布式采样器的采样偏移量
def set_offset(self, offset):
self.offset = offset
return self
@ -423,7 +542,7 @@ class DistributedSampler(BuiltinSampler):
class PKSampler(BuiltinSampler):
"""
Samples K elements for each P class in the dataset.
用于从MindRecord数据集中对于每个P类，采样K个元素。
Args:
num_val (int): Number of elements to sample for each class.
@ -451,21 +570,28 @@ class PKSampler(BuiltinSampler):
"""
def __init__(self, num_val, num_class=None, shuffle=False, class_column='label', num_samples=None):
# 检查num_val的类型是否为int型
if not isinstance(num_val, int):
raise TypeError("num_val must be integer but was: {}.".format(num_val))
# 检查num_class是否为None
if num_class is not None:
raise NotImplementedError("Not supported to specify num_class for PKSampler.")
# 检查shuffle的类型是否为bool型
if not isinstance(shuffle, bool):
raise TypeError("shuffle must be a boolean value but was: {}.".format(shuffle))
# 检查class_column的类型是否为str型
if not isinstance(class_column, str):
raise TypeError("class_column must be a str value but was: {}.".format(class_column))
# 检查num_samples是否为None
if num_samples is not None:
# 检查num_samples的类型是否为int型
if not isinstance(num_samples, int):
raise TypeError("num_samples must be integer but was: {}.".format(num_samples))
# 检查num_samples是否在有效范围内
if num_samples < 0 or num_samples > validator.INT64_MAX:
raise ValueError("num_samples exceeds the boundary between {} and {}(INT64_MAX)!"
.format(0, validator.INT64_MAX))
@ -477,19 +603,27 @@ class PKSampler(BuiltinSampler):
def parse(self):
""" Parse the sampler."""
# 如果num_samples不为空，则取num_samples，否则取0
num_samples = self.num_samples if self.num_samples is not None else 0
# 如果shuffle不为空，则取shuffle，否则取False
shuffle = self.shuffle if self.shuffle is not None else False
# 创建一个PKSamplerObj对象
c_sampler = cde.PKSamplerObj(self.num_val, shuffle, num_samples)
# 解析子采样器
c_child_sampler = self.parse_child()
# 将子采样器添加到PKSamplerObj对象中
c_sampler.add_child(c_child_sampler)
# 返回PKSamplerObj对象
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
if self.child_sampler is None:
return self.shuffle
return self.child_sampler.is_shuffled()
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return False
@ -497,21 +631,33 @@ class PKSampler(BuiltinSampler):
return self.child_sampler.is_sharded()
def parse_for_minddataset(self):
"""Parse the sampler for MindRecord."""
"""Parse the sampler for MindRecord.
用于为MindRecord数据集创建一个解析器
"""
# 检查class_column是否为空字符串或非字符串类型
if not self.class_column or not isinstance(self.class_column, str):
# 如果是，则抛出一个异常
raise ValueError("class_column should be a not empty string value, \
but got class_column: {}.".format(self.class_column))
# 从self对象中获取num_samples属性
# 如果num_samples为None，则将其设置为0
num_samples = self.num_samples if self.num_samples is not None else 0
# 创建一个MindrecordPkSampler对象，参数为验证集数量，类别列名，是否打乱数据，采样数量
c_sampler = cde.MindrecordPkSampler(self.num_val, self.class_column, self.shuffle, num_samples)
# 解析子解析器
c_child_sampler = self.parse_child_for_minddataset()
# 将子解析器添加到解析器的children列表中
c_sampler.add_child(c_child_sampler)
# 设置解析器的num_samples属性
c_sampler.set_num_samples(num_samples)
# 返回创建的解析器对象
return c_sampler
class RandomSampler(BuiltinSampler):
"""
Samples the elements randomly.
用于从数据集中随机采样元素。
Args:
replacement (bool, optional): If True, put the sample ID back for the next draw (default=False).
@ -531,12 +677,17 @@ class RandomSampler(BuiltinSampler):
"""
def __init__(self, replacement=False, num_samples=None):
# 检查replacement的类型是否为bool型
if not isinstance(replacement, bool):
raise TypeError("replacement must be a boolean value but was: {}.".format(replacement))
# 检查num_samples是否为None
if num_samples is not None:
# 检查num_samples的类型是否为int型
if not isinstance(num_samples, int):
raise TypeError("num_samples must be integer but was: {}.".format(num_samples))
# 检查num_samples是否在有效范围内
if num_samples < 0 or num_samples > validator.INT64_MAX:
raise ValueError("num_samples exceeds the boundary between {} and {}(INT64_MAX)!"
.format(0, validator.INT64_MAX))
@ -548,25 +699,39 @@ class RandomSampler(BuiltinSampler):
def parse(self):
""" Parse the sampler."""
# 如果num_samples不为空，则取num_samples，否则取0
num_samples = self.num_samples if self.num_samples is not None else 0
# 如果replacement不为空，则取replacement，否则取False
replacement = self.replacement if self.replacement is not None else False
# 创建一个RandomSamplerObj对象
c_sampler = cde.RandomSamplerObj(replacement, num_samples, self.reshuffle_each_epoch)
# 解析子采样器
c_child_sampler = self.parse_child()
# 将子采样器添加到PKSamplerObj对象中
c_sampler.add_child(c_child_sampler)
# 返回RandomSamplerObj对象
return c_sampler
def parse_for_minddataset(self):
"""Parse the sampler for MindRecord."""
# 如果num_samples不为空，则取num_samples，否则取0
num_samples = self.num_samples if self.num_samples is not None else 0
# 创建一个MindrecordRandomSampler对象
c_sampler = cde.MindrecordRandomSampler(num_samples, self.replacement, self.reshuffle_each_epoch)
# 解析子解析器
c_child_sampler = self.parse_child_for_minddataset()
# 将子解析器添加到解析器的children列表中
c_sampler.add_child(c_child_sampler)
# 设置解析器的num_samples属性
c_sampler.set_num_samples(num_samples)
# 返回MindrecordRandomSampler对象
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
return True
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return False
@ -577,6 +742,7 @@ class RandomSampler(BuiltinSampler):
class SequentialSampler(BuiltinSampler):
"""
Samples the dataset elements sequentially that is equivalent to not using a sampler.
用于从数据集中按顺序采样元素。
Args:
start_index (int, optional): Index to start sampling at. (default=None, start at first ID)
@ -597,6 +763,7 @@ class SequentialSampler(BuiltinSampler):
"""
def __init__(self, start_index=None, num_samples=None):
# 检查参数类型
if start_index is not None and not isinstance(start_index, int):
raise TypeError("start_index must be integer but was: {}.".format(start_index))
@ -612,29 +779,44 @@ class SequentialSampler(BuiltinSampler):
def parse(self):
""" Parse the sampler."""
# 如果start_index不为空，则取start_index，否则取False
start_index = self.start_index if self.start_index is not None else 0
# 如果num_samples不为空，则取num_samples，否则取0
num_samples = self.num_samples if self.num_samples is not None else 0
# 创建一个SequentialSamplerObj对象
c_sampler = cde.SequentialSamplerObj(start_index, num_samples)
# 解析子采样器
c_child_sampler = self.parse_child()
# 将子解析器添加到解析器的children列表中
c_sampler.add_child(c_child_sampler)
# 返回SequentiaSamplerObj对象
return c_sampler
def parse_for_minddataset(self):
"""Parse the sampler for MindRecord."""
# 如果start_index不为空，则取start_index，否则取False
start_index = self.start_index if self.start_index is not None else 0
# 如果num_samples不为空，则取num_samples，否则取0
num_samples = self.num_samples if self.num_samples is not None else 0
# 创建一个MindrecordSequentialSampler对象
c_sampler = cde.MindrecordSequentialSampler(num_samples, start_index)
# 解析子解析器
c_child_sampler = self.parse_child_for_minddataset()
# # 将子解析器添加到解析器的children列表中
c_sampler.add_child(c_child_sampler)
# 设置解析器的num_samples属性
c_sampler.set_num_samples(num_samples)
# 返回MindrecordSequentialSampler对象
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
if self.child_sampler is None:
return False
return self.child_sampler.is_shuffled()
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return False
@ -698,16 +880,20 @@ class SubsetSampler(BuiltinSampler):
super().__init__(num_samples)
def parse(self):
""" Parse the sampler."""
""" Parse the sampler.
代码含义与之前相似
"""
num_samples = self.num_samples if self.num_samples is not None else 0
c_sampler = cde.SubsetSamplerObj(self.indices, num_samples)
c_child_sampler = self.parse_child()
c_sampler.add_child(c_child_sampler)
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
return False
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return False
@ -715,7 +901,9 @@ class SubsetSampler(BuiltinSampler):
return self.child_sampler.is_sharded()
def parse_for_minddataset(self):
"""Parse the sampler for MindRecord."""
"""Parse the sampler for MindRecord.
代码含义与之前相似
"""
c_sampler = cde.MindrecordSubsetSampler(self.indices)
c_child_sampler = self.parse_child_for_minddataset()
c_sampler.add_child(c_child_sampler)
@ -723,6 +911,9 @@ class SubsetSampler(BuiltinSampler):
return c_sampler
def get_num_samples(self):
'''
返回此数据集的样本数
'''
num_samples = super().get_num_samples()
if num_samples is None:
return len(self.indices)
@ -752,18 +943,23 @@ class SubsetRandomSampler(SubsetSampler):
"""
def parse(self):
""" Parse the sampler."""
""" Parse the sampler.
代码含义与之前相似
"""
num_samples = self.num_samples if self.num_samples is not None else 0
c_sampler = cde.SubsetRandomSamplerObj(self.indices, num_samples)
c_child_sampler = self.parse_child()
c_sampler.add_child(c_child_sampler)
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
return True
def parse_for_minddataset(self):
"""Parse the sampler for MindRecord."""
"""Parse the sampler for MindRecord.
代码含义与之前相似
"""
c_sampler = cde.MindrecordSubsetSampler(self.indices, ds.config.get_seed())
c_child_sampler = self.parse_child_for_minddataset()
c_sampler.add_child(c_child_sampler)
@ -774,6 +970,7 @@ class SubsetRandomSampler(SubsetSampler):
class IterSampler(Sampler):
"""
User provided an iterable object without inheriting from our Sampler class.
允许用户提供一个没有继承自我们的Sampler类的迭代对象。
Note:
This class exists to allow handshake logic between dataset operators and user defined samplers.
@ -856,7 +1053,9 @@ class WeightedRandomSampler(BuiltinSampler):
super().__init__(num_samples)
def parse(self):
""" Parse the sampler."""
""" Parse the sampler.
代码含义与之前相似
"""
num_samples = self.num_samples if self.num_samples is not None else 0
replacement = self.replacement if self.replacement is not None else True
c_sampler = cde.WeightedRandomSamplerObj(self.weights, num_samples, replacement)
@ -864,9 +1063,11 @@ class WeightedRandomSampler(BuiltinSampler):
c_sampler.add_child(c_child_sampler)
return c_sampler
# 用于检查采样器是否已经打乱数据
def is_shuffled(self):
return True
# 用于检查采样器是否已经分片
def is_sharded(self):
if self.child_sampler is None:
return False

19
mindspore/python/mindspore/dataset/engine/serializer_deserializer.py
View File

@ -25,6 +25,7 @@ from . import datasets as de
def serialize(dataset, json_filepath=""):
"""
Serialize dataset pipeline into a JSON file.
用于将数据集管道序列化为一个JSON文件。
Note:
Currently some Python objects are not supported to be serialized.
@ -48,12 +49,14 @@ def serialize(dataset, json_filepath=""):
>>> # serialize it to JSON file
>>> serialized_data = ds.serialize(dataset, json_filepath="/path/to/mnist_dataset_pipeline.json")
"""
# 返回一个字典，其中包含序列化后的数据集图。
return dataset.to_json(json_filepath)
def deserialize(input_dict=None, json_filepath=None):
"""
Construct dataset pipeline from a JSON file produced by de.serialize().
用于从JSON文件中构造数据集管道。
Note:
Currently Python function deserialization of map operator are not supported.
@ -82,17 +85,24 @@ def deserialize(input_dict=None, json_filepath=None):
"""
data = None
# 判断input_dict是否为空
if input_dict:
# 将input_dict转换为de.Dataset对象
data = de.DeserializedDataset(input_dict)
# 如果json_filepath不为空
if json_filepath:
# 将json_filepath转换为de.Dataset对象
data = de.DeserializedDataset(json_filepath)
return data
# 定义一个函数expand_path，用于将相对路径转换为绝对路径
def expand_path(node_repr, key, val):
"""Convert relative to absolute path."""
"""Convert relative to absolute path.用于将相对路径转换为绝对路径。"""
# 如果val是一个列表
if isinstance(val, list):
# 那么将其中的每个文件路径转换为绝对路径，并添加到node_repr字典中
node_repr[key] = [os.path.abspath(file) for file in val]
else:
node_repr[key] = os.path.abspath(val)
@ -101,6 +111,7 @@ def expand_path(node_repr, key, val):
def show(dataset, indentation=2):
"""
Write the dataset pipeline graph to logger.info file.
用于将数据集管道图形写入logger.info文件。
Args:
dataset (Dataset): The starting node.
@ -114,14 +125,16 @@ def show(dataset, indentation=2):
>>> dataset = dataset.batch(batch_size=10, drop_remainder=True)
>>> ds.show(dataset)
"""
# 使用json.dumps函数将其转换为格式化的字符串
pipeline = dataset.to_json()
# 写入logger.info文件
logger.info(json.dumps(pipeline, indent=indentation))
def compare(pipeline1, pipeline2):
"""
Compare if two dataset pipelines are the same.
用于比较两个数据集管道的是否相同。
Args:
pipeline1 (Dataset): a dataset pipeline.
@ -135,5 +148,5 @@ def compare(pipeline1, pipeline2):
>>> pipeline2 = ds.Cifar10Dataset(cifar10_dataset_dir, num_samples=100)
>>> res = ds.compare(pipeline1, pipeline2)
"""
# 返回一个bool值
return pipeline1.to_json() == pipeline2.to_json()

46
mindspore/python/mindspore/dataset/text/__init__.py
View File

@ -11,6 +11,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# mindspore.dataset.text。它提供了两个部分：transforms和utils。transforms是一个高性能的自然语言处理（NLP）文本处理模块，它使用ICU4C和cppjieba进行开发。utils提供了用于NLP文本处理的通用方法
"""
This module is to support text processing for NLP. It includes two parts:
transforms and utils. transforms is a high performance
@ -29,6 +31,34 @@ Descriptions of common data processing terms are as follows:
- TensorOperation, the base class of all data processing operations implemented in C++.
- TextTensorOperation, the base class of all text processing operations. It is a derived class of TensorOperation.
"""
"""
该模块中包含了以下内容：
Lookup：用于查找字典中的单词。
JiebaTokenizer：使用结巴分词库进行分词的类。
UnicodeCharTokenizer：用于将Unicode字符切分为字符的类。
Ngram：用于生成n-gram的类。
WordpieceTokenizer：用于对单词进行分块的类。
TruncateSequencePair：用于截断序列对的类。
ToNumber：用于将文本转换为数字的类。
SlidingWindow：用于生成滑动窗口的类。
SentencePieceTokenizer：使用SentencePiece进行分词的类。
PythonTokenizer：使用Python进行分词的类。
ToVectors：用于将文本向量化的类。
to_str：将字节转换为字符串的函数。
to_bytes：将字符串转换为字节的函数。
JiebaMode：结巴分词的模式。
Vocab：词汇表类。
NormalizeForm：文本正则化模式。
SentencePieceVocab：SentencePiece词汇表类。
SentencePieceModel：SentencePiece模型类。
SPieceTokenizerOutType：SentencePiece分词器的输出类型。
SPieceTokenizerLoadType：SentencePiece分词器的加载类型。
Vectors：向量类。
FastText：FastText向量类。
GloVe：GloVe向量类。
CharNGram：CharNGram向量类
"""
import platform
from .transforms import Lookup, JiebaTokenizer, UnicodeCharTokenizer, Ngram, WordpieceTokenizer, \
TruncateSequencePair, ToNumber, SlidingWindow, SentencePieceTokenizer, PythonTokenizer, ToVectors
@ -43,9 +73,23 @@ __all__ = [
"GloVe", "CharNGram"
]
# 这段代码是用于判断当前操作系统是否为Windows
# 如果不是Windows，则从mindspore.dataset.text.transforms模块中导入相应的数据处理操作类
# 其中，platform.system()函数用于获取当前操作系统的名称，并将其转换为小写
if platform.system().lower() != 'windows':
from .transforms import UnicodeScriptTokenizer, WhitespaceTokenizer, CaseFold, NormalizeUTF8, \
RegexReplace, RegexTokenizer, BasicTokenizer, BertTokenizer
# 如果当前操作系统不是Windows（即platform.system().lower() != 'windows'），则从transforms模块中导入以下数据处理操作类：
"""
UnicodeScriptTokenizer：用于将Unicode脚本切分为单词的类。
WhitespaceTokenizer：用于将空格分隔的文本切分为单词的类。
CaseFold：用于将文本转换为小写的类。
NormalizeUTF8：用于对UTF-8编码的文本进行正则化处理的类。
RegexReplace：用于使用正则表达式替换文本的类。
RegexTokenizer：使用正则表达式切分文本的类。
BasicTokenizer：用于对文本进行基本切分的类。
BertTokenizer：用于对BERT模型进行分词的类。
"""
# 同时，将导入的类添加到__all__列表中，以便在其他模块中使用这些类
__all__.extend(["UnicodeScriptTokenizer", "WhitespaceTokenizer", "CaseFold", "NormalizeUTF8",
"RegexReplace", "RegexTokenizer", "BasicTokenizer", "BertTokenizer"])

1010
mindspore/python/mindspore/dataset/text/transforms.py
View File

File diff suppressed because it is too large Load Diff

132
mindspore/python/mindspore/dataset/text/utils.py
View File

@ -30,7 +30,7 @@ __all__ = [
"Vocab", "SentencePieceVocab", "to_str", "to_bytes", "Vectors", "FastText", "GloVe", "CharNGram"
]
# 用于查找单词的Vocab对象
class Vocab:
"""
Vocab object that is used to save pairs of words and ids.
@ -39,6 +39,7 @@ class Vocab:
"""
def __init__(self):
# 初始化词汇表
self.c_vocab = None
def vocab(self):
@ -52,7 +53,9 @@ class Vocab:
>>> vocab = text.Vocab.from_list(["word_1", "word_2", "word_3", "word_4"])
>>> vocabory_dict = vocab.vocab()
"""
# 检查c_vocab是否有效
check_vocab(self.c_vocab)
# 返回c_vocab的vocab
return self.c_vocab.vocab()
@check_tokens_to_ids
@ -71,11 +74,16 @@ class Vocab:
>>> vocab = text.Vocab.from_list(["w1", "w2", "w3"], special_tokens=["<unk>"], special_first=True)
>>> ids = vocab.tokens_to_ids(["w1", "w3"])
"""
# 检查c_vocab中是否存在tokens
check_vocab(self.c_vocab)
# 如果tokens是ndarray类型，则将其转换为列表
if isinstance(tokens, np.ndarray):
tokens = tokens.tolist()
# 如果tokens是字符串类型，则将其转换为列表
if isinstance(tokens, str):
tokens = [tokens]
# 返回c_vocab中tokens转换为id的结果
return self.c_vocab.tokens_to_ids(tokens)
@check_ids_to_tokens
@ -94,6 +102,8 @@ class Vocab:
>>> vocab = text.Vocab.from_list(["w1", "w2", "w3"], special_tokens=["<unk>"], special_first=True)
>>> token = vocab.ids_to_tokens(0)
"""
# 检查c_vocab中是否存在ids
check_vocab(self.c_vocab)
if isinstance(ids, np.ndarray):
ids = ids.tolist()
@ -141,7 +151,18 @@ class Vocab:
>>> dataset = dataset.map(operations=text.Lookup(vocab, "<unk>"), input_columns=["text"])
"""
'''
创建一个词汇表，用于将数据集中的所有列转换为词汇表
:param dataset: 数据集
:param columns: 列名，默认为None
:param freq_range: 频率范围，默认为None
:param top_k: 根据频率范围设置的最大词汇数，默认为None
:param special_tokens: 特殊标记，默认为None
:param special_first: 是否将特殊标记放在第一个词汇表的第一个位置，默认为True
:return: 词汇表
'''
vocab = cls()
# 创建一个vocab实例
vocab.c_vocab = dataset.build_vocab(columns, freq_range, top_k, special_tokens, special_first)
return vocab
@ -166,9 +187,13 @@ class Vocab:
"""
if special_tokens is None:
# 如果special_tokens为空，则将其设置为空列表
special_tokens = []
# 创建一个Vocab对象
vocab = Vocab()
# 将word_list中的元素添加到vocab中，并将special_tokens设置为special_first
vocab.c_vocab = cde.Vocab.from_list(word_list, special_tokens, special_first)
# 返回vocab
return vocab
@classmethod
@ -207,9 +232,20 @@ class Vocab:
>>> vocabulary = vocab.vocab()
"""
'''
从文件中读取词汇表
:param file_path: 包含词汇表的文件路径
:param delimiter: 分隔符
:param vocab_size: 从文件路径读出的词汇表大小
:param special_tokens: 以逗号分隔的特殊标记
:param special_first: 是否按照特殊标记的顺序排序
:return: 词汇表
'''
# 如果词汇表大小为空，则把大小设置为-1
if vocab_size is None:
vocab_size = -1
if special_tokens is None:
# 如果special_tokens为空，则将其设置为空列表
special_tokens = []
vocab = cls()
vocab.c_vocab = cde.Vocab.from_file(file_path, delimiter, vocab_size, special_tokens, special_first)
@ -217,6 +253,7 @@ class Vocab:
@classmethod
@check_from_dict
# 从字典中创建词汇表
def from_dict(cls, word_dict):
"""
Build a vocab object from a dict.
@ -236,7 +273,7 @@ class Vocab:
vocab.c_vocab = cde.Vocab.from_dict(word_dict)
return vocab
# 用于执行分词的SentencePiece对象
class SentencePieceVocab:
"""
SentencePiece object that is used to do words segmentation.
@ -281,10 +318,13 @@ class SentencePieceVocab:
... SentencePieceModel.UNIGRAM, {})
"""
# 创建句子段落词汇表
sentence_piece_vocab = cls()
# 使用数据集构建句子段落词汇表
sentence_piece_vocab.c_sentence_piece_vocab = dataset.build_sentencepiece_vocab(col_names, vocab_size,
character_coverage,
model_type, params)
# 返回句子段落词汇表
return sentence_piece_vocab
@classmethod
@ -322,7 +362,17 @@ class SentencePieceVocab:
... SentencePieceModel.UNIGRAM, {})
"""
'''
从文件中创建SentencePieceVocab对象
:param file_path: 文件路径
:param vocab_size: 词汇表大小
:param character_coverage: 字符覆盖率
:param model_type: 模型类型
:param params: 参数
:return: SentencePieceVocab对象
'''
sentence_piece_vocab = cls()
# 从文件中加载SentencePieceVocab
sentence_piece_vocab.c_sentence_piece_vocab = \
cde.SentencePieceVocab.from_file(file_path, vocab_size, character_coverage,
DE_C_INTER_SENTENCEPIECE_MODE[model_type], params)
@ -330,6 +380,7 @@ class SentencePieceVocab:
@classmethod
@check_save_model
# 将模型保存至给定路径
def save_model(cls, vocab, path, filename):
"""
Save model into given filepath.
@ -348,7 +399,7 @@ class SentencePieceVocab:
cde.SentencePieceVocab.save_model(vocab.c_sentence_piece_vocab, path, filename)
# 将数组转换为字符串
def to_str(array, encoding='utf8'):
"""
Convert NumPy array of `bytes` to array of `str` by decoding each element based on charset `encoding`.
@ -372,7 +423,7 @@ def to_str(array, encoding='utf8'):
return np.char.decode(array, encoding)
# 将数组转化成bytes
def to_bytes(array, encoding='utf8'):
"""
Convert NumPy array of `str` to array of `bytes` by encoding each element based on charset `encoding`.
@ -397,35 +448,34 @@ def to_bytes(array, encoding='utf8'):
return np.char.encode(array, encoding)
# mindspore.dataset.text.JiebaTokenizer 的枚举值，可能的值为JiebaMode.MIX, JiebaMode.MP, JiebaMode.HMM
class JiebaMode(IntEnum):
"""
An enumeration for JiebaTokenizer.
Possible enumeration values are: JiebaMode.MIX, JiebaMode.MP, JiebaMode.HMM.
- JiebaMode.MIX: tokenize with a mix of MPSegment and HMMSegment algorithm.
- JiebaMode.MP: tokenize with MPSegment algorithm.
- JiebaMode.HMM: tokenize with Hidden Markov Model Segment algorithm.
JiebaMode.MIX - 使用最大概率法和隐马尔可夫模型算法混合进行分词。
JiebaMode.MP - 使用最大概率法算法进行分词。
JiebaMode.HMM - 使用隐马尔可夫模型算法进行分词。
"""
MIX = 0
MP = 1
HMM = 2
# Unicode规范化模式 枚举类,可能的值为NormalizeForm.NONE, NormalizeForm.NFC, NormalizeForm.NFKC, NormalizeForm.NFD, NormalizeForm.NFKD
class NormalizeForm(IntEnum):
"""
Enumeration class for `Unicode normalization forms <http://unicode.org/reports/tr15/>`_ .
Possible enumeration values are: NormalizeForm.NONE, NormalizeForm.NFC, NormalizeForm.NFKC, NormalizeForm.NFD
and NormalizeForm.NFKD.
Possible enumeration values are: .
- NormalizeForm.NONE: no normalization.
- NormalizeForm.NFC: Canonical Decomposition, followed by Canonical Composition.
- NormalizeForm.NFKC: Compatibility Decomposition, followed by Canonical Composition.
- NormalizeForm.NFD: Canonical Decomposition.
- NormalizeForm.NFKD: Compatibility Decomposition.
NormalizeForm.NONE - 不进行规范化处理。
NormalizeForm.NFC - 先以标准等价方式分解，再以标准等价方式重组。
NormalizeForm.NFKC - 先以兼容等价方式分解，再以标准等价方式重组。
NormalizeForm.NFD - 以标准等价方式分解。
NormalizeForm.NFKD - 以兼容等价方式分解。
"""
NONE = 0
@ -434,7 +484,7 @@ class NormalizeForm(IntEnum):
NFD = 3
NFKD = 4
# SentencePiece分词方法的枚举类，可能的值为SentencePieceModel.UNIGRAM, SentencePieceModel.BPE, SentencePieceModel.CHAR, SentencePieceModel.WORD
class SentencePieceModel(IntEnum):
"""
An enumeration for SentencePieceModel.
@ -442,12 +492,10 @@ class SentencePieceModel(IntEnum):
Possible enumeration values are: SentencePieceModel.UNIGRAM, SentencePieceModel.BPE, SentencePieceModel.CHAR,
SentencePieceModel.WORD.
- SentencePieceModel.UNIGRAM: Unigram Language Model means the next word in the sentence is assumed to be
independent of the previous words generated by the model.
- SentencePieceModel.BPE: refers to byte pair encoding algorithm, which replaces the most frequent pair of bytes in
a sentence with a single, unused byte.
- SentencePieceModel.CHAR: refers to char based sentencePiece Model type.
- SentencePieceModel.WORD: refers to word based sentencePiece Model type.
SentencePieceModel.UNIGRAM - Unigram语言模型意味着句子中的下一个单词被假定为独立于模型生成的前一个单词。
SentencePieceModel.BPE - 指字节对编码算法，它取代了最频繁的句子对中的字节数，其中包含一个未使用的字节。
SentencePieceModel.CHAR - 引用基于字符的SentencePiece模型类型。
SentencePieceModel.WORD - 引用基于单词的SentencePiece模型类型。
"""
UNIGRAM = 0
@ -463,35 +511,35 @@ DE_C_INTER_SENTENCEPIECE_MODE = {
SentencePieceModel.WORD: cde.SentencePieceModel.DE_SENTENCE_PIECE_WORD
}
# mindspore.dataset.text.SentencePieceTokenizer 输出类型的枚举值，可能的值为SPieceTokenizerOutType.STRING, SPieceTokenizerOutType.INT
class SPieceTokenizerOutType(IntEnum):
"""
An enumeration for SPieceTokenizerOutType.
Possible enumeration values are: SPieceTokenizerOutType.STRING, SPieceTokenizerOutType.INT.
- SPieceTokenizerOutType.STRING: means output type of SentencePiece Tokenizer is string.
- SPieceTokenizerOutType.INT: means output type of SentencePiece Tokenizer is int.
SPieceTokenizerOutType.STRING - 表示SentencePiece分词器的输出类型为string。
SPieceTokenizerOutType.INT - 表示SentencePiece分词器的输出类型为int。
"""
STRING = 0
INT = 1
# mindspore.dataset.text.SentencePieceTokenizer 加载类型的枚举值，可能的值为SPieceTokenizerLoadType.FILE, SPieceTokenizerLoadType.MODEL
class SPieceTokenizerLoadType(IntEnum):
"""
An enumeration for loading type of SentencePieceTokenizer.
Possible enumeration values are: SPieceTokenizerLoadType.FILE, SPieceTokenizerLoadType.MODEL.
- SPieceTokenizerLoadType.FILE: Load SentencePiece tokenizer from a Vocab file.
- SPieceTokenizerLoadType.MODEL: Load SentencePiece tokenizer from a SentencePieceVocab object.
SPieceTokenizerLoadType.FILE - 从词典文件中加载SentencePiece分词器。
SPieceTokenizerLoadType.MODEL - 从 mindspore.dataset.text.SentencePieceVocab 对象中加载SentencePiece分词器。
"""
FILE = 0
MODEL = 1
# 用于将tokens映射到向量的Vectors对象
class Vectors(cde.Vectors):
"""
Vectors object that is used to map tokens into vectors.
@ -514,14 +562,17 @@ class Vectors(cde.Vectors):
>>> vector = text.Vectors.from_file("/path/to/vectors/file", max_vectors=None)
"""
'''
从文件中加载词向量
:param file_path: 文件路径
:param max_vectors: 最大词向量数量，默认为0
:return: 词向量
'''
max_vectors = max_vectors if max_vectors is not None else 0
return super().from_file(file_path, max_vectors)
# 用于将tokens映射到向量的FastText对象
class FastText(cde.FastText):
"""
FastText object that is used to map tokens into vectors.
"""
@classmethod
@check_from_file_vectors
@ -541,14 +592,13 @@ class FastText(cde.FastText):
>>> fast_text = text.FastText.from_file("/path/to/fast_text/file", max_vectors=None)
"""
'''从文件中读取数据'''
max_vectors = max_vectors if max_vectors is not None else 0
return super().from_file(file_path, max_vectors)
# 用于将tokens映射到向量的GloVe对象
class GloVe(cde.GloVe):
"""
GloVe object that is used to map tokens into vectors.
"""
@classmethod
@check_from_file_vectors
@ -571,11 +621,9 @@ class GloVe(cde.GloVe):
max_vectors = max_vectors if max_vectors is not None else 0
return super().from_file(file_path, max_vectors)
# CharNGram对象，用于将 tokens 映射到预训练的向量中
class CharNGram(cde.CharNGram):
"""
CharNGram object that is used to map tokens into pre-trained vectors.
"""
@classmethod
@check_from_file_vectors

5
mindspore/python/mindspore/dataset/transforms/__init__.py
View File

@ -11,6 +11,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# 这段代码是Python中的一个模块，名为mindspore.dataset.transforms。它提供了两个部分：c_transforms和py_transforms。c_transforms是一个高性能的数据增强模块，它使用C++进行开发。py_transforms提供了Python和NumPy实现的可选实现
"""
This module is to support common augmentations. C_transforms is a high performance
augmentation module which is developed by C++. Py_transforms provides an optional
@ -31,6 +33,9 @@ Descriptions of common data processing terms are as follows:
- TensorOperation, the base class of all data processing operations implemented in C++.
- PyTensorOperation, the base class of all data processing operations implemented in Python.
"""
# 包含与图像处理相关的函数
from .. import vision
# 包含C++实现的数据增强操作
from . import c_transforms
# 包含Python和NumPy实现的可选数据增强操作
from . import py_transforms

261
mindspore/python/mindspore/dataset/transforms/py_transforms.py
View File

@ -87,297 +87,104 @@ class PyTensorOperation:
class OneHotOp(PyTensorOperation):
"""
Apply one hot encoding transformation to the input label, make label be more smoothing and continuous.
Args:
num_classes (int): Number of classes of objects in dataset.
It should be larger than the largest label number in the dataset.
smoothing_rate (float, optional): Adjustable hyperparameter for label smoothing level.
(Default=0.0 means no smoothing is applied.)
Raises:
TypeError: `num_classes` is not of type int.
TypeError: `smoothing_rate` is not of type float.
ValueError: `smoothing_rate` is not in range [0.0, 1.0].
Supported Platforms:
``CPU``
Examples:
>>> # Assume that dataset has 10 classes, thus the label ranges from 0 to 9
>>> transforms_list = [py_transforms.OneHotOp(num_classes=10, smoothing_rate=0.1)]
>>> transform = py_transforms.Compose(transforms_list)
>>> mnist_dataset = mnist_dataset.map(input_columns=["label"], operations=transform)
"""
# 定义OneHotOp类，参数num_classes, smoothing_rate
@check_one_hot_op
def __init__(self, num_classes, smoothing_rate=0.0):
# 初始化num_classes和smoothing_rate
self.num_classes = num_classes
self.smoothing_rate = smoothing_rate
self.random = False
def __call__(self, label):
"""
Call method.
Args:
label (numpy.ndarray): label to be applied label smoothing.
Returns:
label (numpy.ndarray), label after being Smoothed.
"""
# 返回one_hot_encoding函数的结果
return util.one_hot_encoding(label, self.num_classes, self.smoothing_rate)
# 将多个数据增强操作组合使用
class Compose(PyTensorOperation):
"""
Compose a list of transforms.
.. Note::
Compose takes a list of transformations either provided in py_transforms or from user-defined implementation;
each can be an initialized transformation class or a lambda function, as long as the output from the last
transformation is a single tensor of type numpy.ndarray. See below for an example of how to use Compose
with py_transforms classes and check out FiveCrop or TenCrop for the use of them in conjunction with lambda
functions.
Args:
transforms (list): List of transformations to be applied.
Raises:
TypeError: If `transforms` is not of type list.
ValueError: If `transforms` is empty.
TypeError: If transformations in `transforms` are not Python callable objects.
Supported Platforms:
``CPU``
Examples:
>>> image_folder_dataset_dir = "/path/to/image_folder_dataset_directory"
>>> # create a dataset that reads all files in dataset_dir with 8 threads
>>> image_folder_dataset = ds.ImageFolderDataset(image_folder_dataset_dir, num_parallel_workers=8)
>>> # create a list of transformations to be applied to the image data
>>> transform = py_transforms.Compose([py_vision.Decode(),
... py_vision.RandomHorizontalFlip(0.5),
... py_vision.ToTensor(),
... py_vision.Normalize((0.491, 0.482, 0.447), (0.247, 0.243, 0.262)),
... py_vision.RandomErasing()])
>>> # apply the transform to the dataset through dataset.map function
>>> image_folder_dataset = image_folder_dataset.map(operations=transform, input_columns=["image"])
>>>
>>> # Compose is also be invoked implicitly, by just passing in a list of ops
>>> # the above example then becomes:
>>> transforms_list = [py_vision.Decode(),
... py_vision.RandomHorizontalFlip(0.5),
... py_vision.ToTensor(),
... py_vision.Normalize((0.491, 0.482, 0.447), (0.247, 0.243, 0.262)),
... py_vision.RandomErasing()]
>>>
>>> # apply the transform to the dataset through dataset.map()
>>> image_folder_dataset_1 = image_folder_dataset_1.map(operations=transforms_list, input_columns=["image"])
>>>
>>> # Certain C++ and Python ops can be combined, but not all of them
>>> # An example of combined operations
>>> arr = [0, 1]
>>> dataset = ds.NumpySlicesDataset(arr, column_names=["cols"], shuffle=False)
>>> transformed_list = [py_transforms.OneHotOp(2), c_transforms.Mask(c_transforms.Relational.EQ, 1)]
>>> dataset = dataset.map(operations=transformed_list, input_columns=["cols"])
>>>
>>> # Here is an example of mixing vision ops
>>> import numpy as np
>>> op_list=[c_vision.Decode(),
... c_vision.Resize((224, 244)),
... py_vision.ToPIL(),
... np.array, # need to convert PIL image to a NumPy array to pass it to C++ operation
... c_vision.Resize((24, 24))]
>>> image_folder_dataset = image_folder_dataset.map(operations=op_list, input_columns=["image"])
"""
@check_compose_list
def __init__(self, transforms):
'''
参数：
transforms：PyTensorOperation对象列表
'''
self.transforms = transforms
if all(hasattr(transform, "random") and not transform.random for transform in self.transforms):
self.random = False
@check_compose_call
def __call__(self, *args):
"""
Call method.
Returns:
lambda function, Lambda function that takes in an args to apply transformations on.
"""
# 返回：组合后的PyTensorOperation对象
return util.compose(self.transforms, *args)
@staticmethod
def reduce(operations):
"""
Wraps adjacent Python operations in a Compose to allow mixing of Python and C++ operations.
Args:
operations (list): list of tensor operations.
Returns:
list, the reduced list of operations.
"""
# import nn and ops locally for type check
from mindspore import nn, ops
# 检查输入的运算是否包含像mindspore.nn或mindspore.ops的运算
for item in operations:
if isinstance(item, (nn.Cell, ops.Primitive)):
raise ValueError("Input operations should not contain network computing operator like in "
"mindspore.nn or mindspore.ops, got operation:", str(item))
# 检查输入的运算是否为空
if len(operations) == 1:
# 检查输入的运算是否为张量运算，如果是返回运算
if str(operations).find("c_transform") >= 0 or isinstance(operations[0], TensorOperation):
return operations
# 如果不是，返回util.FuncWrapper的运算结果
return [util.FuncWrapper(operations[0])]
# 重新设置数值
new_ops, start_ind, end_ind = [], 0, 0
# 循环遍历运算
for i, op in enumerate(operations):
# 检查当前运算为 Compose
if str(op).find("c_transform") >= 0:
# reset counts
if start_ind != end_ind:
# 为new_ops添加新的数值
if start_ind!= end_ind:
new_ops.append(Compose(operations[start_ind:end_ind]))
# 添加当前运算符
new_ops.append(op)
start_ind, end_ind = i + 1, i + 1
# 检查当前运算为 Compose
else:
# 让end_ind 加一
end_ind += 1
# do additional check in case the last operation is a Python operation
if start_ind != end_ind:
# 额外检查以防最后一个运算为Python运算
if start_ind!= end_ind:
new_ops.append(Compose(operations[start_ind:end_ind]))
return new_ops
# 指定一组数据增强处理及其被应用的概率，在运算时按概率随机应用其中的增强处理
class RandomApply(PyTensorOperation):
"""
Randomly perform a series of transforms with a given probability.
Args:
transforms (list): List of transformations to apply.
prob (float, optional): The probability to apply the transformation list (default=0.5).
Raises:
TypeError: If `transforms` is not of type list.
ValueError: If `transforms` is empty.
TypeError: If elements of `transforms` are neither Python callable objects nor data
processing operations in py_transforms.
TypeError: If `prob` is not of type float.
ValueError: If `prob` is not in range [0.0, 1.0].
Supported Platforms:
``CPU``
Examples:
>>> from mindspore.dataset.transforms.py_transforms import Compose
>>> transforms_list = [py_vision.RandomHorizontalFlip(0.5),
... py_vision.Normalize((0.491, 0.482, 0.447), (0.247, 0.243, 0.262)),
... py_vision.RandomErasing()]
>>> transforms = Compose([py_vision.Decode(),
... py_transforms.RandomApply(transforms_list, prob=0.6),
... py_vision.ToTensor()])
>>> image_folder_dataset = image_folder_dataset.map(operations=transforms, input_columns=["image"])
"""
# 定义RandomApply类，参数transforms和prob：概率
@check_random_apply
def __init__(self, transforms, prob=0.5):
self.prob = prob
self.transforms = transforms
def __call__(self, img):
"""
Call method.
Args:
img (PIL image): Image to be randomly applied a list transformations.
Returns:
img (PIL image), Transformed image.
"""
# 返回随机应用于img的结果
return util.random_apply(img, self.transforms, self.prob)
# 在一组数据增强中随机选择部分增强处理进行应用
class RandomChoice(PyTensorOperation):
"""
Randomly select one transform from a series of transforms and applies that on the image.
Args:
transforms (list): List of transformations to be chosen from to apply.
Raises:
TypeError: If `transforms` is not of type list.
TypeError: If elements of `transforms` are neither Python callable objects nor data
processing operations in py_transforms.
ValueError: If `transforms` is empty.
Supported Platforms:
``CPU``
Examples:
>>> from mindspore.dataset.transforms.py_transforms import Compose
>>> transforms_list = [py_vision.RandomHorizontalFlip(0.5),
... py_vision.Normalize((0.491, 0.482, 0.447), (0.247, 0.243, 0.262)),
... py_vision.RandomErasing()]
>>> transforms = Compose([py_vision.Decode(),
... py_transforms.RandomChoice(transforms_list),
... py_vision.ToTensor()])
>>> image_folder_dataset = image_folder_dataset.map(operations=transforms, input_columns=["image"])
"""
# 定义RandomChoice类，继承PyTensorOperation类
@check_transforms_list
def __init__(self, transforms):
# 初始化transforms参数
self.transforms = transforms
def __call__(self, img):
"""
Call method.
Args:
img (PIL image): Image to be applied transformation.
Returns:
img (PIL image), Transformed image.
"""
# 调用PyTensorOperation类的__call__方法，传入img参数
return util.random_choice(img, self.transforms)
# 给一个数据增强的列表，随机打乱数据增强处理的顺序
class RandomOrder(PyTensorOperation):
"""
Perform a series of transforms to the input PIL image in a random order.
Args:
transforms (list): List of the transformations to apply.
Raises:
TypeError: If `transforms` is not of type list.
TypeError: If elements of `transforms` are neither Python callable objects nor data
processing operations in py_transforms.
ValueError: If `transforms` is empty.
Supported Platforms:
``CPU``
Examples:
>>> from mindspore.dataset.transforms.py_transforms import Compose
>>> transforms_list = [py_vision.RandomHorizontalFlip(0.5),
... py_vision.Normalize((0.491, 0.482, 0.447), (0.247, 0.243, 0.262)),
... py_vision.RandomErasing()]
>>> transforms = Compose([py_vision.Decode(),
... py_transforms.RandomOrder(transforms_list),
... py_vision.ToTensor()])
>>> image_folder_dataset = image_folder_dataset.map(operations=transforms, input_columns=["image"])
"""
# 定义一个RandomOrder类，用于接收一个可调用的transforms列表
@check_transforms_list
def __init__(self, transforms):
self.transforms = transforms
def __call__(self, img):
"""
Call method.
Args:
img (PIL image): Image to apply transformations in a random order.
Returns:
img (PIL image), Transformed image.
"""
# 调用PyTensorOperation的__call__方法，传入img参数，并将返回值赋值给img
return util.random_order(img, self.transforms)

142
mindspore/python/mindspore/dataset/transforms/py_transforms_util.py
View File

@ -24,7 +24,11 @@ from ..core.py_util_helpers import is_numpy, ExceptionHandler
def all_numpy(args):
""" for multi-input lambdas"""
'''
判断传入的参数是否都是numpy的
:param args: 参数
:return: 是否都是numpy的
'''
if isinstance(args, tuple):
for value in args:
if not is_numpy(value):
@ -34,22 +38,21 @@ def all_numpy(args):
def compose(transforms, *args):
"""
Compose a list of transforms and apply on the image.
Args:
img (numpy.ndarray): An image in NumPy ndarray.
transforms (list): A list of transform Class objects to be composed.
Returns:
img (numpy.ndarray), An augmented image in NumPy ndarray.
"""
'''
将多个参数组合在一起
:param transforms: 可变参数列表
:param args: 可变参数列表
:return: 参数列表
'''
for transform in transforms:
try:
# 使用transform函数对args参数进行处理
args = transform(*args)
except Exception:
# 如果发生异常，则抛出异常处理结果
result = ExceptionHandler(where="in map(or batch) worker and execute Python function")
result.reraise()
# 如果args参数不是元组，则将其转换为元组
args = (args,) if not isinstance(args, tuple) else args
if all_numpy(args):
@ -58,50 +61,51 @@ def compose(transforms, *args):
def one_hot_encoding(label, num_classes, epsilon):
"""
Apply label smoothing transformation to the input label, and make label be more smoothing and continuous.
Args:
label (numpy.ndarray): label to be applied label smoothing.
num_classes (int): Num class of object in dataset, value should over 0.
epsilon (float): The adjustable Hyper parameter. Default is 0.0.
Returns:
img (numpy.ndarray), label after being one hot encoded and done label smoothed.
Examples:
>>> # assume num_classes = 5
>>> # 1) input np.array(3) output [0, 0, 0, 1, 0]
>>> # 2) input np.array([4, 2, 0]) output [[0, 0, 0, 0, 1], [0, 0, 1, 0, 0], [1, 0, 0, 0, 0]]
>>> # 3) input np.array([[4], [2], [0]]) output [[[0, 0, 0, 0, 1]], [[0, 0, 1, 0, 0][, [[1, 0, 0, 0, 0]]]
"""
'''
将label转换为one-hot编码
:param label: 数据
:param num_classes: 类别数
:param epsilon: 正则化阈值
:return: one-hot编码
'''
# 如果numpy不是（）或（1，)或(n, 1),抛出数值错误异常
if isinstance(label, np.ndarray): # the numpy should be () or (1, ) or shape: (n, 1)
if label.dtype not in [np.int8, np.int16, np.int32, np.int64,
np.uint8, np.uint16, np.uint32, np.uint64]:
raise ValueError('the input numpy type should be int, but the input is: ' + str(label.dtype))
raise ValueError('the input numpy type should be int, but the input is:'+ str(label.dtype))
# 如果label的维度为0，则初始化一个num_classes维的one_hot_label
if label.ndim == 0:
# 如果label大于num_classes,抛出数值错误异常
if label >= num_classes:
raise ValueError('the num_classes is smaller than the category number.')
# 初始化一个num_classes维的one_hot_label
one_hot_label = np.zeros((num_classes), dtype=int)
# 将label中的每一个元素都设置为1
one_hot_label[label] = 1
else:
# 将label转换为num_classes维
label_flatten = label.flatten()
# 对label_flatten进行遍历，如果元素大于num_classes，则抛出异常
for item in label_flatten:
if item >= num_classes:
raise ValueError('the num_classes:' + str(num_classes) +
' is smaller than the category number:' + str(item))
# 计算label_flatten的长度
num_elements = label_flatten.size
# 初始化一个num_elements*num_classes维的one_hot_label
one_hot_label = np.zeros((num_elements, num_classes), dtype=int)
# 对label_flatten进行遍历，将每一个元素都设置为1
for index in range(num_elements):
one_hot_label[index][label_flatten[index]] = 1
# 将label的形状转换为num_classes维
new_shape = []
for dim in label.shape:
new_shape.append(dim)
new_shape.append(num_classes)
# 将one_hot_label转换为num_classes维
one_hot_label = one_hot_label.reshape(new_shape)
else:
raise ValueError('the input is invalid, it should be numpy.ndarray.')
@ -110,88 +114,80 @@ def one_hot_encoding(label, num_classes, epsilon):
def random_order(img, transforms):
"""
Applies a list of transforms in a random order.
Args:
img: Image to be applied transformations in a random order.
transforms (list): List of the transformations to be applied.
Returns:
img, Transformed image.
"""
'''
随机打乱转换组
参数：
img：图片
transforms：转换组
返回：
转换后的图片
'''
# 随机打乱转换组
random.shuffle(transforms)
# 遍历转换组，将图片转换为参数
for transform in transforms:
img = transform(img)
# 返回转换后的图片
return img
def random_apply(img, transforms, prob):
"""
Apply a list of transformation, randomly with a given probability.
Args:
img: Image to be randomly applied a list transformations.
transforms (list): List of transformations to be applied.
prob (float): The probability to apply the transformation list.
Returns:
img, Transformed image.
"""
# 如果概率小于随机数，则直接返回图片
if prob < random.random():
return img
# 遍历可选参数
for transform in transforms:
# 调用可选参数的transform函数
img = transform(img)
# 返回调用可选参数的transform函数的结果
return img
def random_choice(img, transforms):
"""
Random selects one transform from a list of transforms and applies that on the image.
Args:
img: Image to be applied transformation.
transforms (list): List of transformations to be chosen from to apply.
Returns:
img, Transformed image.
"""
'''
随机选择一个转换函数
:param img: 图像
:param transforms: 转换函数列表
:return: 返回转换后的图像
'''
return random.choice(transforms)(img)
class FuncWrapper:
"""
Wrap function with try except logic, mainly for warping python function.
Args:
transform: Callable python function.
Returns:
result, data after apply transformation.
"""
# 初始化函数包装器
def __init__(self, transform):
# 如果输入操作不是可调用的python函数，抛出异常
if not callable(transform):
raise ValueError("Input operations should be callable python function, but got: " + str(transform))
# 将输入操作赋值给self.transform
self.transform = transform
# 尝试获取self.transform的random属性，如果没有，将self.random设置为True
try:
if hasattr(self.transform, "random") and not self.transform.random:
self.random = False
except KeyError:
self.random = True
# 实现__call__方法
def __call__(self, *args):
# 定义一个空的结果变量
result = None
# 尝试调用self.transform函数，如果出现异常，将异常赋值给result
try:
result = self.transform(*args)
except Exception:
result = ExceptionHandler(where="in map(or batch) worker and execute python function")
result.reraise()
# 返回result
return result
# 实现to_json方法
def to_json(self):
# 如果self.transform是FunctionType类型，则将其转换为json格式
if isinstance(self.transform, FunctionType):
json_obj = {}
json_obj["tensor_op_name"] = self.transform.__name__
json_obj["python_module"] = self.__class__.__module__
return json.dumps(json_obj)
# 否则，返回self.transform的to_json方法
return self.transform.to_json()

3
mindspore/python/mindspore/dataset/utils/__init__.py
View File

@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# mindspore.dataset.util。它提供了imshow_det_bbox函数，用于在图像上显示检测结果
"""init file for MindData utils"""
from .browse_dataset import imshow_det_bbox
# __all__是一个包含模块中可导出对象的列表。在这里，只有imshow_det_bbox函数被导出
__all__ = ["imshow_det_bbox"]

53
mindspore/python/mindspore/dataset/utils/browse_dataset.py
View File

@ -96,39 +96,63 @@ def imshow_det_bbox(image, bboxes, labels, segm=None, class_names=None, score_th
"""
try:
# 导入cv2模块
cv2 = importlib.import_module("cv2")
except ModuleNotFoundError:
# 如果导入失败，抛出异常
raise ImportError("Importing cv2 failed, try to install it by running `pip install opencv-python`.")
# validation
assert isinstance(image, np.ndarray) and image.ndim == 3 and (image.shape[0] == 3 or image.shape[2] == 3), \
"image must be a ndarray in (H, W, C) or (C, H, W) format."
if bboxes is not None:
# 断言函数，在条件不满足时直接返回错误
# 判断bboxes是否为ndarray类型，且ndim等于2，且shape[1]等于4或者shape[1]等于5
assert isinstance(bboxes, np.ndarray) and bboxes.ndim == 2 and (bboxes.shape[1] == 4 or bboxes.shape[1] == 5), \
"bboxes must be a ndarray in (N, 4) or (N, 5) format."
# 判断labels是否为ndarray类型，且ndim等于2，且shape[1]等于1，且shape[0]等于bboxes的行数
assert isinstance(labels, np.ndarray) and labels.ndim == 2 and labels.shape[1] == 1 and \
labels.shape[0] == bboxes.shape[
0], "labels must be a ndarray in (N, 1) format and has same N with bboxes."
if segm is not None:
# 断言函数，在条件不满足时直接返回错误
# 判断segm是否为ndarray，且ndim等于3
assert isinstance(segm, np.ndarray) and segm.ndim == 3, "segm must be a ndarray in (M, H, W) format."
# 获取图像的高度和宽度
H, W = (image.shape[0], image.shape[1]) if image.shape[2] == 3 else (image.shape[1], image.shape[2])
# 判断segm的高度和宽度是否相等
assert H == segm.shape[1] and W == segm.shape[2], "segm must has same height and width with image."
# 判断bboxes是否为None
if bboxes is not None:
# 判断bboxes的长度是否小于segm的长度
assert bboxes.shape[0] <= segm.shape[0], "number of segm masks must not be less than the number of bboxes."
# 断言，判断类名是否为tuple, list或dict格式
assert isinstance(class_names, (tuple, list, dict)), "class_names must be a list, tuple or dict."
# 断言，判断bbox线的颜色是否为（BGR）格式
assert isinstance(bbox_color, tuple) and len(bbox_color) == 3, \
"bbox_color must be a three tuple, formatted (B, G, R)."
# 断言，判断文本的颜色是否为（BGR）格式
assert isinstance(text_color, tuple) and len(text_color) == 3, \
"text_color must be a three tuple, formatted (B, G, R)."
# 断言，判断蒙层颜色是否为（BGR）格式
assert isinstance(mask_color, tuple) and len(mask_color) == 3, \
"mask_color must be a three tuple, formatted (B, G, R)."
# 断言，判断线宽度是否为整型
assert isinstance(thickness, int), "thickness must be an int."
# 断言，判断线宽度是否大于等于0
assert thickness >= 0, "thickness must be larger than or equal to zero."
# 断言，判断字体大小是否为整型或浮点型
assert isinstance(font_size, (int, float)), "font_size must be an int or float."
# 断言，判断字体大小是否大于等于0
assert font_size >= 0, "font_size must be larger than or equal to zero."
# 断言，判断展示图片是否为bool类型
assert isinstance(show, bool), "show must be a bool."
# 断言，判断窗口名是否为字符串
assert isinstance(win_name, str), "win_name must be a str."
# 断言，判断等待时间是否为整型
assert isinstance(wait_time, int), "wait_time must be an int."
# 断言，判断等待时间是否大于等于0
assert wait_time >= 0, "wait_time must be larger than or equal to zero."
if out_file is not None:
assert isinstance(out_file, str), "out_file must be a str."
@ -140,43 +164,70 @@ def imshow_det_bbox(image, bboxes, labels, segm=None, class_names=None, score_th
# image
if image.shape[0] == 3:
# 将图片转换为RGB格式
image = image.transpose((1, 2, 0))
# 将图片转换为BGR格式
draw_image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
if bboxes is not None:
# 获取bbox数量
bbox_num = bboxes.shape[0]
# 遍历bbox数组
for i in range(bbox_num):
# 获取bbox
draw_bbox = bboxes[i]
# 判断bbox数组是否有4个元素
if len(draw_bbox) > 4:
# 判断bbox的置信度是否小于阈值
if draw_bbox[4] < score_threshold:
continue
# bbox
# bbox，给图片边框赋值
x1, y1 = int(draw_bbox[0]), int(draw_bbox[1])
x2, y2 = int(draw_bbox[0] + draw_bbox[2]), int(draw_bbox[1] + draw_bbox[3])
# 画框
cv2.rectangle(draw_image, (x1, y1), (x2, y2), bbox_color, thickness)
# label
try:
draw_label = str(class_names[labels[i][0]]) if class_names is not None else f'class {labels[i][0]}'
except (IndexError, KeyError):
draw_label = f'class {labels[i][0]}'
# 判断bbox数组是否有4个元素
if len(draw_bbox) > 4:
# 判断bbox的置信度是否小于阈值
draw_label += f'|{draw_bbox[-1]:.02f}'
# 画标签
cv2.putText(draw_image, draw_label, (x1, y2), cv2.FONT_HERSHEY_SIMPLEX, font_size, text_color, thickness)
# 判断segm数组是否有值
if segm is not None:
# 获取segm数组
mask = segm[i].astype(bool)
# 画segm
draw_image[mask] = draw_image[mask] * 0.5 + np.array(mask_color) * 0.5
else:
# 判断segm数组是否有值
if segm is not None:
# 获取segm数组数量
segm_num = segm.shape[0]
# 遍历segm数组
for i in range(segm_num):
# 获取segm数组
mask = segm[i].astype(bool)
# 画segm
draw_image[mask] = draw_image[mask] * 0.5 + np.array(mask_color) * 0.5
# 是否显示
if show:
# 显示图片
cv2.imshow(win_name, draw_image)
# 等待按键
if cv2.waitKey(wait_time) == 27:
sys.exit()
# 是否保存
if out_file:
# 输出图片文件名
logger.info("Saving image file with name: " + out_file + "...")
# 保存图片
cv2.imwrite(out_file, draw_image)
# 更改文件权限
os.chmod(out_file, 0o600)
# 返回图片
return draw_image

5
mindspore/python/mindspore/dataset/vision/__init__.py
View File

@ -11,6 +11,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# mindspore.dataset.vision。它提供了两个部分：c_transforms和py_transforms。c_transforms是一个高性能的图像增强模块，它使用C++的OpenCV进行开发。py_transforms提供了更多种类的图像增强，它们使用Python的PIL进行开发
"""
This module is to support vision augmentations. It includes two parts:
c_transforms and py_transforms. C_transforms is a high performance
@ -31,6 +33,9 @@ Descriptions of common data processing terms are as follows:
- PyTensorOperation, the base class of all data processing operations implemented in Python.
- ImageTensorOperation, the base class of all image processing operations. It is a derived class of TensorOperation.
"""
# 包含C++实现的图像增强操作
from . import c_transforms
# 包含Python实现的图像增强操作
from . import py_transforms
# 包含一些通用方法，用于图像处理
from .utils import Inter, Border, ConvertMode, ImageBatchFormat, SliceMode, AutoAugmentPolicy

1896
mindspore/python/mindspore/dataset/vision/c_transforms.py
View File

File diff suppressed because it is too large Load Diff

1740
mindspore/python/mindspore/dataset/vision/py_transforms.py
View File

File diff suppressed because it is too large Load Diff

1055
mindspore/python/mindspore/dataset/vision/py_transforms_util.py
View File

File diff suppressed because it is too large Load Diff

55
mindspore/python/mindspore/nn/__init__.py
View File

@ -17,34 +17,49 @@ Neural Networks Cells.
Pre-defined building blocks or computing units to construct neural networks.
"""
from . import layer, loss, optim, metrics, wrap, grad, probability, sparse, dynamic_lr,\
reinforcement
from .learning_rate_schedule import *
from .dynamic_lr import *
from .cell import Cell, GraphCell
from .layer import *
from .loss import *
from .optim import *
from .metrics import *
from .wrap import *
from .grad import Jvp, Vjp
from .sparse import *
from .reinforcement import *
from .transformer import AttentionMask, VocabEmbedding, MultiHeadAttention, FeedForward, TransformerEncoder, \
TransformerDecoder, TransformerEncoderLayer, TransformerDecoderLayer, Transformer, TransformerOpParallelConfig, \
EmbeddingOpParallelConfig, TransformerRecomputeConfig, MoEConfig, OpParallelConfig
# 定义构建单元或计算单元，用于构建神经网络
from. import layer, loss, optim, metrics, wrap, probability, sparse, dynamic_lr
# 导入layer、loss、optim、metrics、wrap、probability、sparse、dynamic_lr模块
from.learning_rate_schedule import *
# 导入learning_rate_schedule模块
from.dynamic_lr import *
# 导入dynamic_lr模块
from.cell import Cell, GraphKernel, GraphCell
# 导入Cell、GraphKernel、GraphCell模块
from.layer import *
# 导入layer模块
from.loss import *
# 导入loss模块
from.optim import *
# 导入optim模块
from.metrics import *
# 导入metrics模块
from.wrap import *
# 导入wrap模块
from.sparse import *
__all__ = ["Cell", "GraphCell"]
# 导入sparse模块
__all__ = ["Cell", "GraphKernel", "GraphCell"]
# 定义构建单元或计算单元，用于构建神经网络
__all__.extend(layer.__all__)
# 向__all__中添加layer模块
__all__.extend(loss.__all__)
# 向__all__中添加loss模块
__all__.extend(optim.__all__)
# 向__all__中添加optim模块
__all__.extend(metrics.__all__)
# 向__all__中添加metrics模块
__all__.extend(wrap.__all__)
__all__.extend(grad.__all__)
# 向__all__中添加wrap模块
__all__.extend(sparse.__all__)
# 向__all__中添加sparse模块
__all__.extend(learning_rate_schedule.__all__)
# 向__all__中添加learning_rate_schedule模块
__all__.extend(dynamic_lr.__all__)
__all__.extend(reinforcement.__all__)
__all__.extend(transformer.__all__)
# 向__all__中添加dynamic_lr模块
__all__.sort()

663
mindspore/python/mindspore/nn/cell.py
View File

File diff suppressed because it is too large Load Diff

127
mindspore/python/mindspore/nn/dynamic_lr.py
View File

@ -13,12 +13,15 @@
# limitations under the License.
# ============================================================================
"""Dynamic Learning Rate"""
# 本文件为动态学习率定义，与LearningRateSchedule类相比可自定义参数更多
# 导入数学模块
import math
# 导入检查模块
from mindspore._checkparam import Validator as validator
def piecewise_constant_lr(milestone, learning_rates):
# 获取分段常量学习率
r"""
Get piecewise constant learning rate. The learning rate for each step will be stored in a list.
@ -55,41 +58,60 @@ def piecewise_constant_lr(milestone, learning_rates):
>>> print(output)
[0.1, 0.1, 0.05, 0.05, 0.05, 0.01, 0.01, 0.01, 0.01, 0.01]
"""
# 检查milestone和learning_rates的值类型是否正确
validator.check_value_type('milestone', milestone, (tuple, list))
validator.check_value_type('learning_rates', learning_rates, (tuple, list))
if len(milestone) != len(learning_rates):
# 检查milestone和learning_rates的长度是否相同
if len(milestone)!= len(learning_rates):
raise ValueError("For 'piecewise_constant_lr', "
"the size of 'milestone' must be same with the size of 'learning_rates', "
"but got 'milestone' size: {}, 'learning_rates' size: {}."
"the size of'milestone' must be same with the size of 'learning_rates', "
"but got'milestone' size: {}, 'learning_rates' size: {}."
.format(len(milestone), len(learning_rates)))
# 初始化lr
lr = []
# 设置last_item为0
last_item = 0
for i, item in enumerate(milestone):
# 检查milestone[i]是否为正整数
validator.check_positive_int(item, f'milestone[{i}]')
# 检查learning_rates[i]是否为浮点数
validator.check_is_float(learning_rates[i], f'learning_rates[{i}]')
# 如果milestone[i]小于last_item，抛出ValueError异常
if item < last_item:
raise ValueError(f"For 'piecewise_constant_lr', "
f"the value of milestone[{i}] must be greater than milestone[{i - 1}], "
f"but got milestone[{i}]: {milestone[i]}, "
f"milestone[{i - 1}]: {milestone[i - 1]}.")
# 将learning_rates[i]和milestone[i]拼接到lr中
lr += [learning_rates[i]] * (item - last_item)
# 记录last_item
last_item = item
return lr
def _check_inputs(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair):
'''检查输入参数是否合法'''
validator.check_positive_int(total_step, 'total_step')
validator.check_positive_int(step_per_epoch, 'step_per_epoch')
# 检查total_step是否大于0
validator.check_positive_int(step_per_epoch,'step_per_epoch')
# 检查step_per_epoch是否大于0
validator.check_positive_int(decay_epoch, 'decay_epoch')
# 检查decay_epoch是否大于0
validator.check_positive_float(learning_rate, 'learning_rate')
# 检查learning_rate是否大于0
validator.check_is_float(learning_rate, 'learning_rate')
# 检查learning_rate是否为浮点数
validator.check_positive_float(decay_rate, 'decay_rate')
# 检查decay_rate是否大于0
validator.check_is_float(decay_rate, 'decay_rate')
# 检查decay_rate是否为浮点数
validator.check_value_type('is_stair', is_stair, [bool])
# 检查is_stair是否为bool类型
def exponential_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair=False):
# 基于指数衰减函数计算学习率。
r"""
Calculates learning rate base on exponential decay function. The learning rate for each step will
be stored in a list.
@ -133,18 +155,27 @@ def exponential_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch,
>>> print(output)
[0.1, 0.1, 0.09000000000000001, 0.09000000000000001, 0.08100000000000002, 0.08100000000000002]
"""
# 检查输入参数
_check_inputs(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair)
# 定义一个函数，用于计算学习率，参数分别为学习率，衰减率，总步数，步数每次衰减的结束步数，衰减的结束第几次，是否为偶数
# 参数：learning_rate：学习率；decay_rate：衰减率；total_step：总步数；step_per_epoch：步数每次衰减的结束步数；decay_epoch：衰减的结束第几次；is_stair：是否为偶数
# 返回：学习率列表
lr = []
# 遍历总步数，计算学习率
for i in range(total_step):
# 如果is_stair为TRUE，则使用两次指数衰减函数计算学习率
if is_stair:
lr.append(learning_rate * decay_rate ** math.floor(math.floor(i / step_per_epoch) / decay_epoch))
# 否则，使用一次指数衰减函数计算学习率
else:
lr.append(learning_rate * decay_rate ** (math.floor(i / step_per_epoch) / decay_epoch))
# 返回学习率列表
return lr
def natural_exp_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair=False):
# 基于自然指数衰减函数计算学习率。
r"""
Calculates learning rate base on natural exponential decay function. The learning rate for each step will be
stored in a list.
@ -188,19 +219,26 @@ def natural_exp_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch,
>>> print(output)
[0.1, 0.1, 0.1, 0.1, 0.016529888822158657, 0.016529888822158657]
"""
# 检查输入参数，并返回学习率
_check_inputs(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair)
function = lambda x, y: x
# 如果为True，则学习率每 decay_epoch 次衰减一次。
if is_stair:
# 如果是，则使用指数衰减函数
function = lambda x, y: math.floor(x / y) * y
# 初始化学习率
lr = []
# 循环计算学习率
for i in range(total_step):
# 使用自然指数衰减函数计算学习率
lr.append(learning_rate * math.e ** (-decay_rate * function(math.floor(i / step_per_epoch), decay_epoch)))
# 返回学习率
return lr
def inverse_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair=False):
# 基于逆时衰减函数计算学习率。
r"""
Calculates learning rate base on inverse-time decay function. The learning rate for each step
will be stored in a list.
@ -246,16 +284,34 @@ def inverse_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, deca
"""
_check_inputs(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair)
# 定义一个函数，用于计算学习率
# learning_rate：学习率
# decay_rate：衰减率
# total_step：总步数
# step_per_epoch：每个epoch的步数
# decay_epoch：衰减的epoch
# is_stair：是否是偶数
# 返回：学习率
def _lr_calc(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair):
# 检查输入参数
_check_inputs(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair)
# 初始化学习率
lr = []
# 循环计算学习率
for i in range(total_step):
# 如果为TRUE，则用逆时间衰减函数每 decay_epoch 次衰减一次。
if is_stair:
lr.append(learning_rate / (1 + decay_rate * math.floor(math.floor(i / step_per_epoch) / decay_epoch)))
# 否则计算学习率
else:
lr.append(learning_rate / (1 + decay_rate * math.floor(i / step_per_epoch) / decay_epoch))
# 返回学习率
return lr
def cosine_decay_lr(min_lr, max_lr, total_step, step_per_epoch, decay_epoch):
# 基于余弦衰减函数计算学习率。
r"""
Calculates learning rate base on cosine decay function. The learning rate for each step will be stored in a list.
@ -300,27 +356,41 @@ def cosine_decay_lr(min_lr, max_lr, total_step, step_per_epoch, decay_epoch):
[0.1, 0.1, 0.05500000000000001, 0.05500000000000001, 0.01, 0.01]
"""
if not isinstance(min_lr, float):
raise TypeError("For 'cosine_decay_lr', the argument 'min_lr' must be type of float, "
"but got 'min_lr' type: {}.".format(type(min_lr)))
raise TypeError("For 'cosine_decay_lr', the argument'min_lr' must be type of float, "
"but got'min_lr' type: {}.".format(type(min_lr)))
# 检查min_lr的类型是否为float
validator.check_non_negative_float(min_lr, "min_lr", None)
validator.check_positive_float(max_lr, 'max_lr')
validator.check_is_float(max_lr, 'max_lr')
# 检查min_lr是否大于0
validator.check_positive_float(max_lr,'max_lr')
# 检查max_lr是否大于min_lr
validator.check_is_float(max_lr,'max_lr')
# 检查total_step是否大于0
validator.check_positive_int(total_step, 'total_step')
validator.check_positive_int(step_per_epoch, 'step_per_epoch')
# 检查step_per_epoch是否大于0
validator.check_positive_int(step_per_epoch,'step_per_epoch')
# 检查decay_epoch是否大于0
validator.check_positive_int(decay_epoch, 'decay_epoch')
# 检查min_lr是否小于max_lr
if min_lr >= max_lr:
raise ValueError("For 'cosine_decay_lr', the 'max_lr' should be greater than the 'min_lr', "
"but got 'max_lr' value: {}, 'min_lr' value: {}.".format(max_lr, min_lr))
raise ValueError("For 'cosine_decay_lr', the'max_lr' should be greater than the'min_lr', "
"but got'max_lr' value: {},'min_lr' value: {}.".format(max_lr, min_lr))
# 定义一个delta值，用于计算cosine的角度
delta = 0.5 * (max_lr - min_lr)
# 定义一个lr列表，用于存放cosine的角度
lr = []
# 遍历total_step
for i in range(total_step):
# 计算tmp_epoch
tmp_epoch = min(math.floor(i / step_per_epoch), decay_epoch)
# 将cosine的角度添加到lr列表中
lr.append(min_lr + delta * (1 + math.cos(math.pi * tmp_epoch / decay_epoch)))
# 返回lr列表
return lr
def polynomial_decay_lr(learning_rate, end_learning_rate, total_step, step_per_epoch, decay_epoch, power,
update_decay_epoch=False):
# 基于多项式衰减函数计算学习率。
r"""
Calculates learning rate base on polynomial decay function. The learning rate for each step
will be stored in a list.
@ -382,33 +452,48 @@ def polynomial_decay_lr(learning_rate, end_learning_rate, total_step, step_per_e
[0.1, 0.1, 0.07363961030678928, 0.07363961030678928, 0.01, 0.01]
"""
validator.check_positive_float(learning_rate, 'learning_rate')
# 检查learning_rate是否为正数
validator.check_is_float(learning_rate, 'learning_rate')
# 检查end_learning_rate的取值是否是浮点数
if not isinstance(end_learning_rate, float):
raise TypeError("For 'polynomial_decay_lr', the argument 'end_learning_rate' must be type of float, "
"but got 'end_learning_rate' type: {}.".format(type(end_learning_rate)))
validator.check_non_negative_float(end_learning_rate, "end_learning_rate", None)
# 检查end_learning_rate的取值是否大于0
validator.check_positive_float(power, 'power')
# 检查power的取值是否大于0
validator.check_is_float(power, 'power')
# 检查power的取值是否为float
validator.check_positive_int(total_step, 'total_step')
validator.check_positive_int(step_per_epoch, 'step_per_epoch')
# 检查total_step的取值是否大于0
validator.check_positive_int(step_per_epoch,'step_per_epoch')
# 检查step_per_epoch的取值是否大于0
validator.check_positive_int(decay_epoch, 'decay_epoch')
# 检查decay_epoch的取值是否大于0
validator.check_value_type('update_decay_epoch', update_decay_epoch, [bool])
origin_decay_epoch = decay_epoch
# 初始化函数
function = lambda x, y: (x, min(x, y))
# 如果更新梯度衰减周期
if update_decay_epoch:
# 用多项式衰减函数计算值
function = lambda x, y: (origin_decay_epoch * max(math.ceil(y / origin_decay_epoch), 1), y)
lr = []
# 计算函数的值
delta = learning_rate - end_learning_rate
for i in range(total_step):
current_epoch = math.floor(i / step_per_epoch)
# 计算学习路周期
decay_epoch, tmp_epoch = function(decay_epoch, current_epoch)
# 计算学习率
lr.append(delta * (1 - tmp_epoch / decay_epoch) ** power + end_learning_rate)
return lr
def warmup_lr(learning_rate, total_step, step_per_epoch, warmup_epoch):
# 预热学习率方法。
r"""
Gets learning rate warming up. The learning rate for each step will be stored in a list.
@ -447,21 +532,31 @@ def warmup_lr(learning_rate, total_step, step_per_epoch, warmup_epoch):
>>> print(output)
[0.0, 0.0, 0.05, 0.05, 0.1, 0.1]
"""
# 检查learning_rate是否为浮点数
if not isinstance(learning_rate, float):
raise TypeError("For 'warmup_lr', the argument 'learning_rate' must be type of float, "
"but got 'learning_rate' type: {}.".format(type(learning_rate)))
validator.check_non_negative_float(learning_rate, "learning_rate", None)
# 检查learning_rate是否为正数
validator.check_positive_int(warmup_epoch, 'warmup_epoch')
# 检查warmup_epoch是否为正数
validator.check_positive_int(total_step, 'total_step')
validator.check_positive_int(step_per_epoch, 'step_per_epoch')
# 检查total_step是否为正数
validator.check_positive_int(step_per_epoch,'step_per_epoch')
# 定义一个预热学习率函数，用于计算函数的值
function = lambda x, y: (x, min(x, y))
# 定义一个空列表，用于存放计算函数的值
lr = []
# 遍历total_step次，每次计算函数的值
for i in range(total_step):
# 计算当前epoch
current_epoch = math.floor(i / step_per_epoch)
# 计算warmup_epoch和当前epoch的最小值
warmup_epoch, tmp_epoch = function(warmup_epoch, current_epoch)
# 用函数计算学习率的值
lr.append(learning_rate * tmp_epoch / warmup_epoch)
# 返回计算后学习率的值
return lr

2
mindspore/python/mindspore/nn/grad/__init__.py
View File

@ -17,7 +17,7 @@ Grad
Cells of grad function. Calculate the gradient of input network or function.
"""
# 本文件为神经网络梯度的雅各比矩阵向量乘积计算操作的构建
from .cell_grad import Jvp, Vjp

107
mindspore/python/mindspore/nn/grad/cell_grad.py
View File

@ -13,54 +13,80 @@
# limitations under the License.
# ============================================================================
"""cell grad"""
from ..cell import Cell
from ...ops import composite as C
from ...ops import operations as P
from ...ops.primitive import Primitive
from ...common import dtype as mstype
from ...common.api import ms_function
# Cell神经网络基本单元的梯度计算
from..cell import Cell
from...ops import composite as C
from...ops import operations as P
from...ops.primitive import Primitive
from...common import dtype as mstype
from...common.api import ms_function
class _FirstGrad(Cell):
# 初始化梯度类
def __init__(self, fn):
# 初始化类
super(_FirstGrad, self).__init__()
# 初始化first_grad_op变量
self.first_grad_op = C.GradOperation(sens_param=True, get_all=True)
# 将fn变量赋值给self.fn
self.fn = fn
# 构建梯度
def construct(self, u, first_grad_input):
# 返回first_grad_op函数的输出
return self.first_grad_op(self.fn)(*first_grad_input, u)
class _JvpFirstGrad(Cell):
# 初始化雅可比矩阵与向量乘积的计算类
def __init__(self):
super(_JvpFirstGrad, self).__init__()
# 定义一个变量，用来存储梯度
self.first_grad_op = C.GradOperation(sens_param=True, get_all=True)
# 构建梯度
def construct(self, u, fn, first_grad_input):
# 返回梯度
return self.first_grad_op(fn)(*first_grad_input, u)
class _FirstGradSingleValue(Cell):
'''
计算第一次梯度单元
'''
def __init__(self, fn):
'''
初始化FirstGradSingleValue类
:param fn: 函数
:return: None
'''
super(_FirstGradSingleValue, self).__init__()
self.first_grad_single_value_op = C.GradOperation(sens_param=True)
self.fn = fn
def construct(self, u, first_grad_single_value_input):
'''
计算第一次梯度单元
'''
return self.first_grad_single_value_op(self.fn)(*first_grad_single_value_input, u)
class _JvpFirstGradSingleValue(Cell):
# 初始化雅可比矩阵与向量乘积的第一次梯度单值操作计算类
def __init__(self):
super(_JvpFirstGradSingleValue, self).__init__()
# 初始化第一次梯度单值操作
self.first_grad_single_value_op = C.GradOperation(sens_param=True)
# 构建梯度
def construct(self, u, fn, first_grad_single_value_input):
# 返回第一次梯度单值操作的梯度
return self.first_grad_single_value_op(fn)(*first_grad_single_value_input, u)
class Jvp(Cell):
# 实现雅可比矩阵与向量乘积的操作类
"""
Compute the jacobian-vector-product of the given fn. Jvp is equivalent to forward mode autodiff.
@ -99,81 +125,125 @@ class Jvp(Cell):
"""
def __init__(self, fn):
super(Jvp, self).__init__()
# 初始化函数
self.fn = fn
# 初始化OnesLike函数
self.oneslike = P.OnesLike()
# 初始化FirstGrad函数
self.first_grad = _FirstGrad(fn)
# 添加flags
self.first_grad.add_flags(enable_tuple_grad=True)
self.first_grad_single_value = _FirstGradSingleValue(fn)
self.first_grad_single_value.add_flags(enable_tuple_grad=True)
# 初始化C.GradOperation函数
self.second_grad_op = C.GradOperation(sens_param=True)
# 初始化IsSubClass函数
self.issubclass_ = P.IsSubClass()
# 初始化Primitive函数
self.typeof = Primitive('typeof')
# 初始化MakeTuple函数
self.make_tuple = Primitive('MakeTuple')
# 初始化tuple_len函数
self.tuple_len = Primitive("tuple_len")
@ms_function
def construct(self, *args):
# 获取输入参数
jvp_input = args[0:-1]
# 获取输出参数
v = args[-1]
# 调用函数
output = self.fn(*jvp_input)
# 判断输出类型
if self.issubclass_(self.typeof(output), mstype.tuple_):
# 初始化u
u = self.make_tuple()
# 遍历输出参数
for i in range(self.tuple_len(output)):
# 将OnesLike函数赋值给u
u = u + self.make_tuple(self.oneslike(output[i]))
else:
# 将OnesLike函数赋值给u
u = self.oneslike(output)
# 判断输入参数的长度
if self.tuple_len(jvp_input) == 1:
# 初始化second_gradient_net
second_gradient_net = self.second_grad_op(self.first_grad_single_value)
# 调用second_gradient_net函数
gradient_output = second_gradient_net(u, jvp_input, v)
else:
# 初始化second_gradient_net
second_gradient_net = self.second_grad_op(self.first_grad)
# 调用second_gradient_net函数
gradient_output = second_gradient_net(u, jvp_input, v)
# 返回输出参数和梯度输出参数
return output, gradient_output
class _JvpInner(Cell):
# 实现雅可比矩阵与向量乘积的内部过程类
"""
Compute the jacobian-vector-product of the given network. Jvp is equivalent to forward mode autodiff.
This class implements the inner process of function jvp.
"""
def __init__(self):
super(_JvpInner, self).__init__()
# 初始化一个OnesLike函数
self.oneslike = P.OnesLike()
# 初始化一个_JvpFirstGrad函数
self.first_grad = _JvpFirstGrad()
# 添加_JvpFirstGrad函数的标志位enable_tuple_grad=True
self.first_grad.add_flags(enable_tuple_grad=True)
# 初始化一个_JvpFirstGradSingleValue函数
self.first_grad_single_value = _JvpFirstGradSingleValue()
# 添加_JvpFirstGradSingleValue函数的标志位enable_tuple_grad=True
self.first_grad_single_value.add_flags(enable_tuple_grad=True)
# 初始化一个C.GradOperation函数的参数sens_param=True
self.second_grad_op = C.GradOperation(sens_param=True)
# 初始化一个P.IsSubClass函数
self.issubclass_ = P.IsSubClass()
# 初始化一个Primitive函数
self.typeof = Primitive('typeof')
# 初始化一个Primitive函数
self.make_tuple = Primitive('MakeTuple')
# 初始化一个Primitive函数
self.tuple_len = Primitive("tuple_len")
# 定义一个函数，用于构建JVP
def construct(self, *args):
# 获取参数
fn = args[0]
v = args[1]
jvp_input = args[2:]
# 调用函数
output = fn(*jvp_input)
# 如果输出是一个元组，则调用_JvpFirstGradSingleValue函数
if self.issubclass_(self.typeof(output), mstype.tuple_):
u = self.make_tuple()
for i in range(self.tuple_len(output)):
# 将output[i]的值设置为oneslike函数的返回值
u = u + self.make_tuple(self.oneslike(output[i]))
# 否则调用_JvpFirstGrad函数
else:
u = self.oneslike(output)
# 如果输入是一个元组，则调用_JvpFirstGrad函数
if self.tuple_len(jvp_input) == 1:
second_gradient_net = self.second_grad_op(self.first_grad_single_value)
gradient_output = second_gradient_net(u, fn, jvp_input, v)
# 否则调用_JvpFirstGrad函数
else:
second_gradient_net = self.second_grad_op(self.first_grad)
gradient_output = second_gradient_net(u, fn, jvp_input, v)
# 返回输出和梯度输出
return output, gradient_output
class Vjp(Cell):
# 计算向量`v`与给定fn的雅可比矩阵之间的点积。
"""
Computes the dot product between a vector `v` and the Jacobian of the given fn at the point
given by the inputs.
@ -216,26 +286,39 @@ class Vjp(Cell):
"""
def __init__(self, fn):
# 初始化Vjp类
super(Vjp, self).__init__()
# 将fn赋值给self.fn
self.fn = fn
# 创建一个GradOperation对象，获取所有参数和sens_param
self.grad = C.GradOperation(get_all=True, sens_param=True)
# 创建一个GradOperation对象，获取sens_param
self.grad_single_value = C.GradOperation(sens_param=True)
# 创建一个IsSubClass对象
self.issubclass_ = P.IsSubClass()
# 创建一个typeof和tuple_len Primitive
self.typeof = Primitive('typeof')
self.tuple_len = Primitive("tuple_len")
# 定义构造函数
@ms_function
def construct(self, *args):
# 获取前一个参数
front_input = args[0:-1]
# 调用fn函数
output = self.fn(*front_input)
# 如果前一个参数为tuple，则调用grad_single_value函数
if self.tuple_len(front_input) == 1:
gradient_output = self.grad_single_value(self.fn)(*args)
# 否则调用grad函数
else:
gradient_output = self.grad(self.fn)(*args)
# 返回输出和梯度输出
return output, gradient_output
class _VjpInner(Cell):
# 计算向量`v`与给定网络在输入给出的点处的雅可比矩阵之间的点积（内部过程）
"""
Computes the dot product between a vector `v` and the Jacobian of the given network at the point
given by the inputs. This class implements the inner process of function vjp.
@ -243,17 +326,27 @@ class _VjpInner(Cell):
def __init__(self):
super(_VjpInner, self).__init__()
# 创建一个梯度操作，获取每个输入的参数的梯度
self.grad = C.GradOperation(get_all=True, sens_param=True)
# 创建一个梯度操作，获取单个输入的梯度
self.grad_single_value = C.GradOperation(sens_param=True)
# 创建一个操作，获取输入的tuple长度
self.tuple_len = Primitive("tuple_len")
def construct(self, *args):
# 获取输入的函数
fn = args[0]
# 获取输入的前一个输入
front_input = args[1:-1]
# 获取输入的元组
input_with_v = args[1:]
# 计算输出和梯度
output = fn(*front_input)
if self.tuple_len(front_input) == 1:
# 若输入的tuple长度为1，则计算单个输入的梯度
gradient_output = self.grad_single_value(fn)(*input_with_v)
else:
# 若输入的tuple长度不为1，则计算多个输入的梯度
gradient_output = self.grad(fn)(*input_with_v)
# 返回输出和梯度
return output, gradient_output

20
mindspore/python/mindspore/nn/layer/__init__.py
View File

@ -11,7 +11,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# ============================================================================】
# 用于初始化layer
"""
Layer.
@ -19,22 +20,39 @@ The high-level components(Cells) used to construct the neural network.
"""
from . import activation, normalization, container, conv, basic, embedding, pooling, image, quant, math, \
combined, timedistributed, thor_layer, rnns, rnn_cells
# 包含激活函数（如ReLU、sigmoid等）
from .activation import *
# 包含数据归一化方法（如Batch Normalization、Layer Normalization等）
from .normalization import *
# 包含神经网络层容器（如Sequential、TimeDistributed等）
from .container import *
# 包含卷积层（如Conv1D、Conv2D等）
from .conv import *
# 包含所有RNN层（如SimpleRNN、GRU、LSTM等）
from .rnns import *
# 包含RNN单元（如SimpleRNNCell、GRUCell、LSTMCell等）
from .rnn_cells import *
# 包含基本层（如Dense、Flatten等）
from .basic import *
# 包含嵌入层（将文本数据转换为向量表示）
from .embedding import *
# 包含池化层（如MaxPooling1D、MaxPooling2D等）
from .pooling import *
# 包含图像层（如ResizeMethod、RandomCrop等）
from .image import *
# 包含量化层（将浮点数转换为定点数）
from .quant import *
# 包含数学运算层（如Add、Multiply等）
from .math import *
# 包含组合层（将多个层组合为一个层）
from .combined import *
# 包含时间序列层（如Bidirectional、TimeDistributed等）
from .timedistributed import *
# 包含Thor层（用于Thor框架）
from .thor_layer import DenseThor, Conv2dThor, EmbeddingThor, EmbeddingLookupThor
# 定义一个名为__all__的列表，并将多个模块中的类添加到该列表中
# 下面extend中和上面的import中包含的基本一致
__all__ = []
__all__.extend(activation.__all__)
__all__.extend(normalization.__all__)

112
mindspore/python/mindspore/nn/layer/activation.py
View File

@ -14,13 +14,17 @@
# ============================================================================
"""activation"""
import numpy as np
# 从mindspore._checkparam库中导入Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 从mindspore._extends库中导入cell_attr_register函数，用于注册Cell类的属性
from mindspore._extends import cell_attr_register
# 从mindspore.common库中导入dtype和Parameter类，用于处理数据类型和参数
from mindspore.common import dtype as mstype
# 从mindspore.common.tensor库中导入Tensor类，用于处理张量数据
from mindspore.common.parameter import Parameter
from mindspore.common.tensor import Tensor
from mindspore.ops import functional as F
# 从mindspore.ops库中导入functional和operations模块，用于定义一些基本的操作
from mindspore.ops import operations as P
from ..cell import Cell
@ -44,7 +48,8 @@ __all__ = ['Softmax',
'CELU',
]
#解决神经网络中梯度消失问题的激活函数
#优点是其没有梯度消失且附近所有平均激活为零，可以帮助我们加速并以其他方法改善学习
class CELU(Cell):
r"""
Continuously differentiable exponential linear units activation function.
@ -89,12 +94,14 @@ class CELU(Cell):
def __init__(self, alpha=1.0):
"""Initialize CELU."""
super(CELU, self).__init__()
#初始化CELU类
self.celu = P.CeLU(alpha=alpha)
def construct(self, x):
#定义构造函数，用于构造神经网络
return self.celu(x)
#输出值转化为概率分布
class Softmax(Cell):
r"""
Softmax activation function. It is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of
@ -142,12 +149,15 @@ class Softmax(Cell):
def __init__(self, axis=-1):
"""Initialize Softmax."""
super(Softmax, self).__init__()
#初始化Softmax类
self.softmax = P.Softmax(axis)
def construct(self, x):
#返回x
return self.softmax(x)
#计算log的Softmax函数
#计算概率分布的对数概率的归一化操作，其常用于计算神经网络的输出概率，确保概率和为1，有助于计算损失函数并且优化网络参数
class LogSoftmax(Cell):
r"""
LogSoftmax activation function.
@ -193,12 +203,15 @@ class LogSoftmax(Cell):
def __init__(self, axis=-1):
"""Initialize LogSoftmax."""
super(LogSoftmax, self).__init__()
#初始化LogSoftmax激活函数
self.log_softmax = P.LogSoftmax(axis)
def construct(self, x):
#返回x
return self.log_softmax(x)
#与前面的CELU函数类似
#用于替代传统的ReLU函数，在负数区域有着很好的表现，有利于帮助我们处理神经网络中的负值
class ELU(Cell):
r"""
Exponential Linear Unit activation function.
@ -246,12 +259,14 @@ class ELU(Cell):
def __init__(self, alpha=1.0):
"""Initialize ELU."""
super(ELU, self).__init__()
#初始化ELU激活函数
self.elu = P.Elu(alpha)
def construct(self, x):
#返回x值
return self.elu(x)
#一种用于替代传统sigmoid函数和tanh函数的激活函数，计算速度快但是收负数区限制
class ReLU(Cell):
r"""
Rectified Linear Unit activation function.
@ -291,12 +306,14 @@ class ReLU(Cell):
def __init__(self):
"""Initialize ReLU."""
super(ReLU, self).__init__()
#初始化ReLU激活函数
self.relu = P.ReLU()
def construct(self, x):
#返回x
return self.relu(x)
#为前面ReLU函数的变体
class ReLU6(Cell):
r"""
Compute ReLU6 activation function.
@ -335,12 +352,14 @@ class ReLU6(Cell):
def __init__(self):
"""Initialize ReLU6."""
super(ReLU6, self).__init__()
#初始化函数
self.relu6 = P.ReLU6()
def construct(self, x):
#返回x
return self.relu6(x)
#LeakyReLU函数不同于传统的ReLU函数，其优点是在负数区域也有着很好的表现，有利于处理负值
class LeakyReLU(Cell):
r"""
Leaky ReLU activation function.
@ -382,17 +401,24 @@ class LeakyReLU(Cell):
def __init__(self, alpha=0.2):
"""Initialize LeakyReLU."""
super(LeakyReLU, self).__init__()
#检查alpha的类型是否为float或int
validator.check_value_type('alpha', alpha, [float, int], self.cls_name)
#初始化greater_equal和mul
self.greater_equal = P.GreaterEqual()
self.mul = P.Mul()
#将alpha赋值给self.alpha
self.alpha = alpha
#初始化select_op
self.select_op = P.Maximum()
#如果alpha大于1，则将select_op赋值为Minimum
if self.alpha > 1:
self.select_op = P.Minimum()
def construct(self, x):
alpha_array = P.Cast()(F.scalar_to_array(self.alpha), P.DType()(x))
#将alpha_array转换为x的数据类型
out = self.select_op(alpha_array * x, x)
#返回select_op函数的输出
return out
@ -433,12 +459,15 @@ class Tanh(Cell):
def __init__(self):
"""Initialize Tanh."""
super(Tanh, self).__init__()
# 初始化Tanh函数
self.tanh = P.Tanh()
def construct(self, x):
#返回x
return self.tanh(x)
#GELU是一种可以一定程度上代替传统ReLU函数的函数
#在负数区域有很好表现，计算速度也更快
class GELU(Cell):
r"""
Gaussian error linear unit activation function.
@ -499,10 +528,14 @@ class GELU(Cell):
def __init__(self, approximate=True):
"""Initialize GELU."""
super(GELU, self).__init__()
#检查approximate参数
validator.check_bool(approximate, 'approximate', self.cls_name)
#将approximate参数设置为True
self.approximate = approximate
#如果approximate参数为True，则将gelu设置为P.GeLU()
if self.approximate:
self.gelu = P.GeLU()
#否则，将erf设置为P.Erf()，并将sqrt设置为P.Sqrt()，并将const0设置为Tensor(0.5, mstype.float32)，const1设置为Tensor(1.0, mstype.float32)，const2设置为Tensor(2.0, mstype.float32)
else:
self.erf = P.Erf()
self.sqrt = P.Sqrt()
@ -512,11 +545,13 @@ class GELU(Cell):
def construct(self, x):
if self.approximate:
#如果approximate为True，则返回gelu函数的输出
return self.gelu(x)
#否则，返回x的const0和const1的求和结果，并将const2的值转换为x的dtype
return x * F.cast(self.const0, x.dtype) * (F.cast(self.const1, x.dtype) + \
self.erf(x / self.sqrt(F.cast(self.const2, x.dtype))))
#为GELU活函数的变体
class FastGelu(Cell):
r"""
Fast Gaussian error linear unit activation function.
@ -559,12 +594,14 @@ class FastGelu(Cell):
def __init__(self):
"""Initialize FastGelu."""
super(FastGelu, self).__init__()
#初始化FastGelu类
self.fast_gelu = P.FastGeLU()
def construct(self, x):
return self.fast_gelu(x)
#Sigmoid 激活函数用于将神经网络的输出映射到0到1之间的实数
#0到1之间的实数有利于进行概率估计
class Sigmoid(Cell):
r"""
Sigmoid activation function.
@ -605,12 +642,13 @@ class Sigmoid(Cell):
def __init__(self):
"""Initialize Sigmoid."""
super(Sigmoid, self).__init__()
# 初始化Sigmoid类
self.sigmoid = P.Sigmoid()
def construct(self, x):
return self.sigmoid(x)
#增强版的ReLU函数，在负数区域有着更好的表现，计算速度更快
class PReLU(Cell):
r"""
PReLU activation function.
@ -668,45 +706,60 @@ class PReLU(Cell):
def __init__(self, channel=1, w=0.25):
"""Initialize PReLU."""
super(PReLU, self).__init__()
#检查channel是否为正整数
validator.check_positive_int(channel, 'channel', self.cls_name)
#如果w是一个float类型，则创建一个新的float32类型的数组
if isinstance(w, (float, np.float32)):
tmp = np.empty((channel,), dtype=np.float32)
#将w的值赋值给tmp
tmp.fill(w)
#将tmp转换为Tensor类型
w = Tensor(tmp, dtype=mstype.float32)
#如果w是一个list类型，则检查list的长度是否等于channel
elif isinstance(w, list):
if len(w) != channel:
if len(w)!= channel:
#如果长度不等于channel，则抛出ValueError异常
raise ValueError(f"For '{self.cls_name}', the length of 'w' should be equal to the 'channel' when "
f"the 'w' is a list, but got the length of 'w': {len(w)}, the 'channel': {channel}.")
for i in w:
#如果w不是float类型，抛出异常
if not isinstance(i, (float, np.float32)):
raise ValueError(f"For '{self.cls_name}', all elements in 'w' should be "
f"float when the 'w' is a list, but got {i}.")
#将w转换为float32类型
w = Tensor(w, dtype=mstype.float32)
elif isinstance(w, Tensor):
#如果w是Tensor类型，且dtype不是float16或float32，抛出异常
if w.dtype not in (mstype.float16, mstype.float32):
raise ValueError(f"For '{self.cls_name}', the dtype of 'w' should be float16 or "
f"float32 when the 'w' is a tensor, but got {w.dtype}.")
if len(w.shape) != 1 or w.shape[0] != channel:
#如果w的维度不是1，且w的元素个数不等于channel，抛出异常
if len(w.shape)!= 1 or w.shape[0]!= channel:
raise ValueError(f"For '{self.cls_name}', the dimension of 'w' should be 1, and the elements number "
f"should be equal to the 'channel' when the 'w' is a tensor, "
f"but got 'w' shape {w.shape}, the 'channel' {channel}.")
else:
#如果w不是float，list或tensor，抛出异常
raise TypeError(f"For '{self.cls_name}', the 'w' only supported float, list and tensor, "
f"but got {type(w).__name__}.")
self.w = Parameter(w, name='a')
#创建一个PReLU激活函数
self.prelu = P.PReLU()
#创建一个ReLU激活函数
self.relu = P.ReLU()
#创建一个Assign操作
self.assign = P.Assign()
def construct(self, x):
u = self.relu(self.w)
v = self.prelu(x, F.cast(u, x.dtype))
#如果训练，将w赋值给u
if self.training:
self.assign(self.w, u)
#返回v
return v
#一种可以替代ReUL的新激活函数，可以在
class HSwish(Cell):
r"""
Hard swish activation function.
@ -744,9 +797,11 @@ class HSwish(Cell):
def __init__(self):
"""Initialize HSwish."""
super(HSwish, self).__init__()
#初始化HSwish函数
self.hswish = P.HSwish()
def construct(self, x):
#返回x值
return self.hswish(x)
@ -784,12 +839,14 @@ class HSigmoid(Cell):
def __init__(self):
"""Initialize HSigmoid."""
super(HSigmoid, self).__init__()
#初始化HSigmoid类
self.hsigmoid = P.HSigmoid()
def construct(self, input_x):
#返回input_x
return self.hsigmoid(input_x)
#可以用于替代传统Sigmoid函数
#输出范围是[-inf, 0]，有助于提高神经网络的响应能力
class LogSigmoid(Cell):
r"""
Logsigmoid activation function.
@ -827,21 +884,29 @@ class LogSigmoid(Cell):
def __init__(self):
"""Initialize LogSigmoid."""
super(LogSigmoid, self).__init__()
#初始化mul和exp
self.mul = P.Mul()
self.exp = P.Exp()
#初始化add和rec和log
self.add = P.Add()
self.rec = P.Reciprocal()
self.log = P.Log()
def construct(self, input_x):
neg_input = self.mul(input_x, -1)
#计算负号的指数
exp_neg_input = self.exp(neg_input)
#计算负号的指数的平方
exp_neg_input_1 = self.add(exp_neg_input, 1)
#计算负号的指数的平方的1+
rec_exp_neg_input_1 = self.rec(exp_neg_input_1)
#计算负号的指数的平方的1+的逆
ret = self.log(rec_exp_neg_input_1)
#返回负号的指数的平方的1+的逆
return ret
#可以用于替代ReLU函数
#输出范围是 [-alpha, alpha]，有助于提高神经网络的响应能力
class SoftShrink(Cell):
r"""
Applies the SoftShrink function element-wise.
@ -884,13 +949,16 @@ class SoftShrink(Cell):
def __init__(self, lambd=0.5):
super(SoftShrink, self).__init__()
#初始化softshrink函数
self.softshrink = P.SoftShrink(lambd)
def construct(self, input_x):
output = self.softshrink(input_x)
#返回output
return output
#可以用于替代ReLU函数
#输出范围是 [-alpha, alpha]，有助于提高神经网络的响应能力
class HShrink(Cell):
r"""
Hard Shrink activation function. Calculates the output according to the input elements.
@ -935,6 +1003,7 @@ class HShrink(Cell):
def __init__(self, lambd=0.5):
super(HShrink, self).__init__()
# 初始化HShrink类的实例
self.hshrink = P.HShrink(lambd)
def construct(self, input_x):
@ -981,9 +1050,12 @@ def get_activation(name, prim_name=None):
Sigmoid<>
"""
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
#如果name为None，则返回None
if name is None:
return None
#如果name不在_activation中，则抛出KeyError异常
if name not in _activation:
raise KeyError(f"{msg_prefix} 'name' should be in {list(_activation.keys())}, but got {name}.")
#返回_activation中name对应的函数
return _activation[name]()

249
mindspore/python/mindspore/nn/layer/basic.py
View File

@ -14,22 +14,37 @@
# ============================================================================
"""basic"""
# 导入Python的math库，提供了一些数学函数，如 sin、cos、sqrt 等
import math
# 导入NumPy库
import numpy as np
# 从mindspore.common.dtype模块中导入DType类，用于表示数据类型
import mindspore.common.dtype as mstype
# 从mindspore.ops.composite.multitype_ops模块中导入_constexpr_utils模块，用于提供一些常量计算工具
from mindspore.ops.composite.multitype_ops import _constexpr_utils as const_utils
# 从mindspore.common.seed模块中导入_get_graph_seed函数，用于获取图的随机种子
from mindspore.common.seed import _get_graph_seed
# 从mindspore.common.tensor模块中导入Tensor类，用于表示张量（多维数组）
from mindspore.common.tensor import Tensor
# 从mindspore.common.initializer模块中导入initializer函数，用于创建初始化器（用于初始化张量的值）
from mindspore.common.initializer import initializer
# 从mindspore.ops模块中导入operations子模块，用于提供一些基本的操作（如矩阵乘法、加法等）
from mindspore.ops import operations as P
# 从mindspore.ops模块中导入functional子模块，用于提供一些基本的函数（如relu、sigmoid等）
from mindspore.ops import functional as F
# 从mindspore.ops.functional模块中导入identity函数，用于返回输入的副本
from mindspore.ops.functional import identity
# 从mindspore.ops.operations模块中导入_inner_ops子模块，用于提供一些内部操作（如逐元素操作等）
from mindspore.ops.operations import _inner_ops as inner
# 从mindspore.ops.primitive模块中导入constexpr和Primitive类，用于定义常量表达式和基本操作
from mindspore.ops.primitive import constexpr, Primitive
from mindspore.common.parameter import Parameter
# 从mindspore._extends模块中导入cell_attr_register函数，用于注册Cell类的属性
from mindspore._extends import cell_attr_register
from mindspore._checkparam import Rel, Validator
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
# 从当前模块（.activation）中导入get_activation函数，用于获取激活函数
from .activation import get_activation
__all__ = ['Dropout', 'Flatten', 'Dense', 'ClipByNorm', 'Norm', 'OneHot', 'Pad', 'Unfold', 'Tril', 'Triu',
@ -81,18 +96,26 @@ class L1Regularizer(Cell):
def __init__(self, scale):
"""Initialize L1Regularizer."""
super(L1Regularizer, self).__init__()
#检查scale的类型是否为整数或浮点数
Validator.check_value_type("scale", scale, [int, float], self.cls_name)
#如果scale小于等于0，抛出异常
if scale <= 0:
raise ValueError(f"For '{self.cls_name}', the 'scale' should be greater than 0, but got {scale}.")
raise ValueError(f"For '{self.cls_name}', the'scale' should be greater than 0, but got {scale}.")
#如果scale是INF或NAN，抛出异常
if math.isinf(scale) or math.isnan(scale):
raise ValueError(f"For '{self.cls_name}', the 'scale' can not be INF or NAN, but got {scale}.")
raise ValueError(f"For '{self.cls_name}', the'scale' can not be INF or NAN, but got {scale}.")
#初始化绝对值
self.abs = P.Abs()
#初始化平方和
self.reduce_sum = P.ReduceSum()
#将scale转换为Tensor类型
self.scale = Tensor(scale, dtype=mstype.float32)
def construct(self, weights):
const_utils.check_type_valid(F.dtype(weights), mstype.number_type, 'weights')
#计算weights的绝对值
l1_regularization = self.scale * self.reduce_sum(self.abs(weights))
#返回l1_regularization
return l1_regularization
@ -150,28 +173,40 @@ class Dropout(Cell):
def __init__(self, keep_prob=0.5, dtype=mstype.float32):
"""Initialize Dropout."""
super(Dropout, self).__init__()
#检查keep_prob的类型
Validator.check_value_type('keep_prob', keep_prob, [float], self.cls_name)
#检查keep_prob的值在0和1之间
if keep_prob <= 0 or keep_prob > 1:
raise ValueError(f"For '{self.cls_name}', the 'keep_prob' should be a number in range (0, 1], "
f"but got {keep_prob}.")
#检查dtype的类型
Validator.check_subclass("dtype", dtype, mstype.number_type, self.cls_name)
#将keep_prob赋值给self.keep_prob
self.keep_prob = keep_prob
#获取图的seed0和seed1
seed0, seed1 = _get_graph_seed(0, "dropout")
#将seed0赋值给self.seed0
self.seed0 = seed0
#将seed1赋值给self.seed1
self.seed1 = seed1
#将Dropout函数赋值给self.dropout
self.dropout = P.Dropout(keep_prob, seed0, seed1)
def construct(self, x):
#如果训练模式为False，则直接返回x
if not self.training:
return x
#如果keep_prob为1，则直接返回x
if self.keep_prob == 1:
return x
#否则，使用dropout操作，并返回结果
out, _ = self.dropout(x)
return out
def extend_repr(self):
#返回keep_prob的字符串表示
return 'keep_prob={}'.format(self.keep_prob)
@ -213,17 +248,20 @@ class Flatten(Cell):
super(Flatten, self).__init__()
def construct(self, x):
#将x的形状转换为(batch_size, -1)
return F.reshape(x, (F.shape(x)[0], -1))
@constexpr
def check_dense_input_shape(x, prim_name=None):
#检查输入是否为稠密的形状
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
if len(x) < 2:
#如果x的维度小于2，抛出ValueError异常
raise ValueError(f"{msg_prefix} dimension of 'x' should not be less than 2, but got {len(x)}.")
class Dense(Cell):
#全链接层
r"""
The dense connected layer.
@ -285,87 +323,122 @@ class Dense(Cell):
activation=None):
"""Initialize Dense."""
super(Dense, self).__init__()
#获取输入通道数
self.in_channels = Validator.check_positive_int(in_channels, "in_channels", self.cls_name)
#获取输出通道数
self.out_channels = Validator.check_positive_int(out_channels, "out_channels", self.cls_name)
#是否有偏差
self.has_bias = Validator.check_bool(has_bias, "has_bias", self.cls_name)
#初始化reshape
self.reshape = P.Reshape()
#获取shape
self.shape_op = P.Shape()
if isinstance(weight_init, Tensor):
if weight_init.ndim != 2 or weight_init.shape[0] != out_channels or \
weight_init.shape[1] != in_channels:
#判断weight_init的维度是否正确，且第一个维度和第二个维度是否相等
if weight_init.ndim!= 2 or weight_init.shape[0]!= out_channels or \
weight_init.shape[1]!= in_channels:
#如果不正确，抛出异常
raise ValueError(f"For '{self.cls_name}', weight init shape error. The ndim of 'weight_init' should "
f"be equal to 2, and the first dim should be equal to 'out_channels', and the "
f"second dim should be equal to 'in_channels'. But got 'weight_init': {weight_init}, "
f"'out_channels': {out_channels}, 'in_channels': {in_channels}.")
#创建一个参数，初始化为weight_init，并设置name为"weight"
self.weight = Parameter(initializer(weight_init, [out_channels, in_channels]), name="weight")
self.bias = None
if self.has_bias:
#如果有bias，则初始化bias
if isinstance(bias_init, Tensor):
if bias_init.ndim != 1 or bias_init.shape[0] != out_channels:
if bias_init.ndim!= 1 or bias_init.shape[0]!= out_channels:
raise ValueError(f"For '{self.cls_name}', bias init shape error. The ndim of 'bias_init' should "
f"be equal to 1, and the first dim should be equal to 'out_channels'. But got "
f"'bias_init': {bias_init}, 'out_channels': {out_channels}.")
#初始化bias
self.bias = Parameter(initializer(bias_init, [out_channels]), name="bias")
#初始化bias_add
self.bias_add = P.BiasAdd()
#初始化matmul
self.matmul = P.MatMul(transpose_b=True)
#如果activation不为空，且不是Cell或Primitive，则抛出异常
self.activation = get_activation(activation) if isinstance(activation, str) else activation
if activation is not None and not isinstance(self.activation, (Cell, Primitive)):
raise TypeError(f"For '{self.cls_name}', the 'activation' must be str or Cell or Primitive, but got "
f"{type(activation).__name__}.")
#如果activation不为空，且不是Cell或Primitive，则将activation赋值给activation_flag
self.activation_flag = self.activation is not None
def construct(self, x):
#获取x的形状
x_shape = self.shape_op(x)
#检查x的形状是否符合要求
check_dense_input_shape(x_shape, self.cls_name)
if len(x_shape) != 2:
#如果x的形状不是2维，则将x转换为2维
if len(x_shape)!= 2:
x = self.reshape(x, (-1, x_shape[-1]))
#将x与weight相乘，并将结果添加到x中
x = self.matmul(x, self.weight)
#如果有偏置，则将x与偏置相加，并将结果添加到x中
if self.has_bias:
x = self.bias_add(x, self.bias)
#如果有激活函数，则执行激活函数，并将结果添加到x中
if self.activation_flag:
x = self.activation(x)
if len(x_shape) != 2:
#如果x的形状不是2维，则将x转换为2维，并将结果添加到x中
if len(x_shape)!= 2:
out_shape = x_shape[:-1] + (-1,)
x = self.reshape(x, out_shape)
#返回x
return x
def extend_repr(self):
#扩展字符串
s = 'input_channels={}, output_channels={}'.format(self.in_channels, self.out_channels)
#如果有bias，则添加bias
if self.has_bias:
s += ', has_bias={}'.format(self.has_bias)
#如果有激活函数，则添加激活函数
if self.activation_flag:
s += ', activation={}'.format(self.activation)
#返回扩展字符串
return s
@constexpr
def _is_equal_one(x):
#如果x为None，返回False
if x is None:
return False
#返回布尔型的x
return bool(x.asnumpy().mean() == 1.0)
@constexpr
#检查x_dtype
def _dtype_check(x_dtype, prim_name=None):
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
#果x_dtype不是float32或float16，抛出异常
if x_dtype not in [mstype.float32, mstype.float16]:
raise TypeError(f"{msg_prefix} x_dtype must be float32 or float16, but got {x_dtype}.")
@constexpr
#判断是否为浮点型
def _is_float_dtype(dtype):
if dtype in [mstype.float32, mstype.float16]:
#如果dtype类型为float32或float16，返回True
return True
#否则，返回False
return False
@constexpr
#判断是否需要执行_reduce_all
def _need_reduce_all(axis):
if axis == ():
# 如果axis为空，则返回True
return True
#否则，返回False
return False
@ -413,47 +486,72 @@ class ClipByNorm(Cell):
def __init__(self, axis=None):
"""Initialize ClipByNorm."""
super(ClipByNorm, self).__init__()
#如果axis为空，则设置为元组类型
if axis is None:
axis = ()
#如果axis为元组类型，则遍历元组中的每一个元素，检查元素类型是否为int，若不是则抛出异常
if isinstance(axis, tuple):
for idx, item in enumerate(axis):
Validator.check_value_type("axis[%d]" % idx, item, [int], self.cls_name)
#将axis转换为int类型
self.axis = Validator.check_value_type('axis', axis, [int, tuple], self.cls_name)
#实例化ReduceSum类，keep_dims为True
self.reduce_sum = P.ReduceSum(keep_dims=True)
#将keep_dims设置为True
self.select_ = P.Select()
#将Select()函数设置为一个选择函数
self.greater_ = P.Greater()
#将Greater()函数设置为一个比较函数
self.cast = P.Cast()
#将Cast()函数设置为一个转换函数
self.sqrt = P.Sqrt()
#将Sqrt()函数设置为一个平方根函数
self.max_op = P.Maximum()
#将Maximum()函数设置为一个最大函数
self.shape = P.Shape()
#将Shape()函数设置为一个维度函数
self.reshape = P.Reshape()
#将Reshape()函数设置为一个变形函数
self.fill = P.Fill()
#将Fill()函数设置为一个填充函数
self.expand_dims = P.ExpandDims()
#将ExpandDims()函数设置为一个增加维度函数
self.dtype = P.DType()
def construct(self, x, clip_norm):
#计算x的平方
mul_x = F.square(x)
#计算x的平方和
l2sum = self.cast(self.reduce_sum(mul_x, self.axis), mstype.float32)
#如果l2sum大于0，则将l2sum赋值给cond，否则将l2sum赋值为ones_
cond = self.greater_(l2sum, 0)
ones_ = self.fill(self.dtype(cond), self.shape(cond), 1.0)
#将l2sum赋值给l2sum_safe，如果cond为True，则将l2sum赋值给l2sum_safe，否则将l2sum赋值为ones_
l2sum_safe = self.select_(cond, l2sum, self.cast(ones_, self.dtype(l2sum)))
#将l2sum_safe赋值给l2norm，如果cond为True，则将l2sum_safe赋值给l2norm，否则将l2sum赋值为l2sum
l2norm = self.select_(cond, self.sqrt(l2sum_safe), l2sum)
_dtype_check(self.dtype(x), self.cls_name)
#判断x的数据类型是否符合要求
if _is_equal_one(clip_norm):
#如果clip_norm为1，则直接返回x
intermediate = x
else:
#否则，将x乘以clip_norm
intermediate = x * clip_norm
#计算max_norm
max_norm = self.max_op(l2norm, clip_norm)
#如果clip_norm为1，则将max_norm转换为float32
if _need_reduce_all(self.axis):
max_norm = self.expand_dims(max_norm, -1)
#计算values_clip
values_clip = self.cast(intermediate, mstype.float32) / max_norm
#将values_clip转换为x的数据类型
values_clip = self.reshape(values_clip, self.shape(x))
#返回values_clip
values_clip = identity(values_clip)
return values_clip
class Norm(Cell):
r"""
Computes the norm of vectors, currently including Euclidean norm, i.e., :math:`L_2`-norm.
@ -515,21 +613,29 @@ class Norm(Cell):
def __init__(self, axis=(), keep_dims=False):
"""Initialize Norm."""
super(Norm, self).__init__()
#检查keep_dims的类型是否为布尔型
Validator.check_value_type("keep_dims", keep_dims, [bool], self.cls_name)
#将axis赋值给self.axis
self.axis = axis
#将keep_dims赋值给self.keep_dims
self.keep_dims = keep_dims
#初始化ReduceSum函数
self.reduce_sum = P.ReduceSum(True)
#初始化Sqrt函数
self.sqrt = P.Sqrt()
#初始化Squeeze函数
self.squeeze = P.Squeeze(self.axis)
def construct(self, x):
x = self.sqrt(self.reduce_sum(F.square(x), self.axis))
#如果keep_dims为False，则将x转置
if not self.keep_dims:
x = self.squeeze(x)
return x
#添加一个字符串，用于表示当前类的扩展信息
def extend_repr(self):
return 'axis={}, keep_dims={}'.format(self.axis, self.keep_dims)
@ -651,12 +757,17 @@ class OneHot(Cell):
def __init__(self, axis=-1, depth=1, on_value=1.0, off_value=0.0, dtype=mstype.float32):
"""Initialize OneHot."""
super(OneHot, self).__init__()
#初始化OneHot类
self.onehot = P.OneHot(axis)
#初始化axis参数
self.depth = depth
#初始化depth参数
self.dtype = dtype
#初始化dtype参数
self.on_value = on_value
#初始化on_value参数
self.off_value = off_value
#构建onehot编码
def construct(self, indices):
return self.onehot(indices, self.depth, F.cast(self.on_value, self.dtype), F.cast(self.off_value, self.dtype))
@ -799,26 +910,37 @@ class Pad(Cell):
def __init__(self, paddings, mode="CONSTANT"):
"""Initialize Pad."""
super(Pad, self).__init__()
#设置模式
self.mode = mode
#检查模式是否为CONSTANT、REFLECT、SYMMETRIC
Validator.check_string(self.mode, ["CONSTANT", "REFLECT", "SYMMETRIC"],'mode', self.cls_name)
#设置填充
self.paddings = paddings
Validator.check_string(self.mode, ["CONSTANT", "REFLECT", "SYMMETRIC"], 'mode', self.cls_name)
#检查mode是否为字符串，是否为CONSTANT、REFLECT或SYMMETRIC
if not isinstance(paddings, tuple):
raise TypeError(f"For '{self.cls_name}', the type of 'paddings' must be tuple, "
f"but got {type(paddings).__name__}.")
for item in paddings:
if len(item) != 2:
#如果item的长度不等于2，抛出异常
if len(item)!= 2:
raise ValueError(f"For '{self.cls_name}', the dimension of 'paddings' must be (n, 2), "
f"but got {paddings}.")
#如果paddings的长度大于4，抛出异常
if len(paddings) > 4:
raise ValueError(f"For '{self.cls_name}', only 'paddings' up to 4 dims is supported, but got "
f"{len(paddings)}.")
#如果mode的值为CONSTANT，则将paddings转换为Tensor类型
if mode == "CONSTANT":
self.pad = P.Pad(self.paddings)
else:
#否则将paddings转换为numpy数组类型，并赋值给self.paddings
self.paddings = Tensor(np.array(self.paddings), dtype=mstype.int64)
#将mode设置为MirrorPad模式
self.pad = P.MirrorPad(mode=mode)
#构建模式为CONSTANT的卷积层
def construct(self, x):
if self.mode == "CONSTANT":
x = self.pad(x)
else:
@ -830,21 +952,29 @@ class Pad(Cell):
def bilinear(shape, size, scale, align_corners, prim_name=None):
"""Check input and calculate shape"""
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
#检查align_corners的类型是否为布尔值
if not isinstance(align_corners, bool):
raise TypeError(f"{msg_prefix} type of 'align_corners' should be boolean, "
f"but got {type(align_corners).__name__}.")
#检查size和scale的值是否都不为空
if size is None and scale is None:
raise ValueError(f"{msg_prefix} 'size' and 'scale' both none.")
raise ValueError(f"{msg_prefix}'size' and'scale' both none.")
#检查size和scale的值是否都不为空
if size is not None and scale is not None:
raise ValueError(f"{msg_prefix} 'size' and 'scale' both not none.")
raise ValueError(f"{msg_prefix}'size' and'scale' both not none.")
if size is not None:
#如果size不是元组或列表，则抛出错误
if not isinstance(size, (tuple, list)):
raise ValueError(f"{msg_prefix} 'size' must be tuple or list or None, but got {type(size).__name__}.")
raise ValueError(f"{msg_prefix}'size' must be tuple or list or None, but got {type(size).__name__}.")
#检查size的长度是否为2，是否小于等于2，是否大于等于1
Validator.check_int(len(size), 2, Rel.EQ, "size", "bilinear")
Validator.check_int(size[0], 1, Rel.GE, "size[0]", "bilinear")
Validator.check_int(size[1], 1, Rel.GE, "size[1]", "bilinear")
#返回size
return size
#检查scale是否大于等于1
Validator.check_int(scale, 1, Rel.GE, "scale factor", "bilinear")
#返回scale乘以shape的第2列和第3列
ret = (scale * shape[2], scale * shape[3])
return ret
@ -915,11 +1045,13 @@ class ResizeBilinear(Cell):
self.half_pixel_centers = half_pixel_centers
def construct(self, x, size=None, scale_factor=None, align_corners=False):
#调用bilinear函数，计算shape
shape = bilinear(x.shape, size, scale_factor, align_corners, self.cls_name)
#调用ResizeBilinear函数，计算resize_bilinear
resize_bilinear = P.ResizeBilinear(shape, align_corners, self.half_pixel_centers)
#返回resize_bilinear函数的结果
return resize_bilinear(x)
class Unfold(Cell):
r"""
Extracts patches from images.
@ -982,10 +1114,13 @@ class Unfold(Cell):
super(Unfold, self).__init__()
def _check_tuple_or_list(arg_name, arg_val, prim_name):
#检查参数arg_name的类型是否为tuple或list
Validator.check_value_type(f"{arg_name}s", ksizes, [tuple, list], self.cls_name)
if len(arg_val) != 4 or arg_val[0] != 1 or arg_val[3] != 1:
#检查参数arg_val的长度是否为4，且参数arg_val的第一个元素是否为1，第四个元素是否为1
if len(arg_val)!= 4 or arg_val[0]!= 1 or arg_val[3]!= 1:
raise ValueError(f"For '{prim_name}' the format of '{arg_name}s' should be [1, {arg_name}_row, "
f"{arg_name}_col, 1], but got {arg_val}.")
#检查参数arg_val的第二个元素和第三个元素是否为正整数
if not isinstance(arg_val[1], int) or not isinstance(arg_val[2], int) or arg_val[1] < 1 or arg_val[2] < 1:
raise ValueError(f"For '{prim_name}' the {arg_name}_row and {arg_name}_col in '{arg_name}s' should be "
f"an positive integer number, but got {arg_name}_row is {arg_val[1]}, "
@ -994,16 +1129,22 @@ class Unfold(Cell):
_check_tuple_or_list("ksize", ksizes, self.cls_name)
_check_tuple_or_list("stride", strides, self.cls_name)
_check_tuple_or_list("rate", rates, self.cls_name)
# 将ksizes转换为元组，并将元组转换为列表
ksizes = ksizes[0], ksizes[3], ksizes[1], ksizes[2]
# 将strides转换为元组，并将元组转换为列表
strides = strides[0], strides[3], strides[1], strides[2]
# 将rates转换为元组，并将元组转换为列表
rates = rates[0], rates[3], rates[1], rates[2]
# 调用inner.ExtractImagePatches函数，传入ksizes、strides、rates、padding参数
self.extract_image_patches = inner.ExtractImagePatches(ksizes, strides, rates, padding)
#定义一个构造函数，用于提取图像特征
def construct(self, input_x):
#使用extract_image_patches函数提取图像特征
result = self.extract_image_patches(input_x)
#返回提取的特征
return result
@constexpr
def tril(x_shape, x_dtype, k):
Validator.check_int(len(x_shape), 1, Rel.GE, "x rank", "tril")
@ -1096,13 +1237,18 @@ class Tril(Cell):
def __init__(self):
"""Initialize Tril."""
super(Tril, self).__init__()
#初始化dtype
self.dtype = P.DType()
#初始化mul
self.mul = P.Mul()
#初始化cast
self.cast = P.Cast()
def construct(self, x, k=0):
assist = tril(x.shape, self.dtype(x), k)
#将x的行数和列数转换为float32类型
result = self.mul(self.cast(x, mstype.float32), self.cast(assist, mstype.float32))
#将result的类型转换为x的类型
return self.cast(result, self.dtype(x))
@ -1110,7 +1256,9 @@ class Tril(Cell):
def triu(x_shape, x_dtype, k):
Validator.check_int(len(x_shape), 1, Rel.GE, "x rank", "triu")
Validator.check_is_int(k, "k value", "triu")
# 创建一个mask，其中元素值为1，表示元素在对角线上，其余为0
mask = np.triu(np.ones(x_shape), k)
# 返回一个Tensor，其中元素值为mask，其类型为x_dtype
return Tensor(mask, x_dtype)
@ -1185,33 +1333,46 @@ class Triu(Cell):
[ 0 11 12 13]
[ 0 0 16 17]]
"""
#构造
def __init__(self):
"""Initialize Triu."""
super(Triu, self).__init__()
#初始化dtype
self.dtype = P.DType()
#初始化mul
self.mul = P.Mul()
#初始化cast
self.cast = P.Cast()
#构建或计算类的逻辑
def construct(self, x, k=0):
assist = triu(x.shape, self.dtype(x), k)
#将x的行和列都设置为0，并将其转换为float32类型
result = self.mul(self.cast(x, mstype.float32), self.cast(assist, mstype.float32))
#将result转换为x的类型
return self.cast(result, self.dtype(x))
@constexpr
#获取一个张量对角线上的值
def _get_matrix_diag_assist(x_shape, x_dtype):
Validator.check_int(len(x_shape), 1, Rel.GE, "x rank", "_get_matrix_diag_assist")
#创建一个eye矩阵，维度为x_shape[-1]，x_shape[-1]
base_eye = np.eye(x_shape[-1], x_shape[-1]).reshape(-1)
#将eye矩阵填充到x_shape中，每个元素都是base_eye
assist = np.tile(base_eye, x_shape[:-1]).reshape(x_shape + (x_shape[-1],))
#将eye矩阵和x_shape中的元素进行tile，每个元素都是base_eye
return Tensor(assist, x_dtype)
@constexpr
#获取一个张量对角线上的值
def _get_matrix_diag_part_assist(x_shape, x_dtype):
Validator.check_int(len(x_shape), 2, Rel.GE, "x rank", "_get_matrix_diag_part_assist")
#创建一个eye矩阵，输入维度为x_shape[-2]和x_shape[-1]，输出维度为x_shape
base_eye = np.eye(x_shape[-2], x_shape[-1]).reshape(-1)
#将eye矩阵填充到x_shape中，输出维度为x_shape
assist = np.tile(base_eye, x_shape[:-2]).reshape(x_shape)
#将eye矩阵填充到Tensor中，输出维度为x_shape
return Tensor(assist, x_dtype)
@ -1279,17 +1440,21 @@ class MatrixDiag(Cell):
def __init__(self):
"""Initialize MatrixDiag."""
super(MatrixDiag, self).__init__()
#初始化矩阵对角线
self.matrix_diag = inner.MatrixDiag()
#获取数据类型
self.dtype = P.DType()
def construct(self, input_x):
x_shape = F.shape(input_x)
x_dtype = self.dtype(input_x)
#获取矩阵对角线元素的辅助信息
assist = _get_matrix_diag_assist(x_shape, x_dtype)
#调用矩阵对角线元素的函数
out_matrix_diag = self.matrix_diag(input_x, assist)
#返回矩阵对角线元素
return out_matrix_diag
class MatrixDiagPart(Cell):
r"""
Returns the batched diagonal part of a batched tensor.
@ -1336,14 +1501,19 @@ class MatrixDiagPart(Cell):
def __init__(self):
"""Initialize MatrixDiagPart."""
super(MatrixDiagPart, self).__init__()
#初始化矩阵对角线部分
self.matrix_diag_part = inner.MatrixDiagPart()
#获取数据类型
self.dtype = P.DType()
def construct(self, input_x):
x_shape = F.shape(input_x)
x_dtype = self.dtype(input_x)
#获取输入x的对角线部分
assist = _get_matrix_diag_part_assist(x_shape, x_dtype)
#调用matrix_diag_part函数，获取输入x的对角线部分
out_matrix_diag_part = self.matrix_diag_part(input_x, assist)
#返回输出x的对角线部分
return out_matrix_diag_part
@ -1395,18 +1565,23 @@ class MatrixSetDiag(Cell):
def __init__(self):
"""Initialize MatrixSetDiag."""
super(MatrixSetDiag, self).__init__()
#初始化矩阵集合对角线
self.matrix_set_diag = inner.MatrixSetDiag()
#获取数据类型
self.dtype = P.DType()
def construct(self, input_x, diagonal):
x_shape = F.shape(input_x)
x_dtype = self.dtype(input_x)
#获取输入x的矩阵对角线部分
assist = _get_matrix_diag_part_assist(x_shape, x_dtype)
#调用matrix_set_diag函数，计算输入x的对角线元素
out_matrix_set_diag = self.matrix_set_diag(input_x, diagonal, assist)
return out_matrix_set_diag
@constexpr
#检查输入的维度是否在指定的范围内
def _check_input_dim(axis, dim, cls_name):
Validator.check_int_range(axis, -dim, dim, Rel.INC_LEFT, 'axis', cls_name)
@ -1459,35 +1634,53 @@ class Roll(Cell):
def __init__(self, shift, axis):
"""Initialize Roll"""
super(Roll, self).__init__()
#检查shift的类型是否为int, tuple, list
Validator.check_value_type("shift", shift, [int, tuple, list], self.cls_name)
#检查axis的类型是否为int, tuple, list
Validator.check_value_type("axis", axis, [int, tuple, list], self.cls_name)
#创建shape操作对象
self.shape_op = P.Shape()
#将shift赋值给self.shift
self.shift = shift
#将axis赋值给self.axis
self.axis = axis
#创建op_list列表
self.op_list = []
if not isinstance(self.axis, (list, tuple)):
#如果axis不是list或tuple类型，则添加一个Roll类型的op_list
self.op_list.append((inner.Roll(shift=self.shift, axis=0), self.axis))
else:
if len(self.shift) != len(self.axis):
raise ValueError(f"For '{self.cls_name}', the shape of 'shift' and the shape of 'axis' must be "
f"the same, but got the length of 'shift' {len(self.shift)} and the length of 'axis'"
#如果axis是list或tuple类型，则检查shift和axis的长度是否相等
if len(self.shift)!= len(self.axis):
raise ValueError(f"For '{self.cls_name}', the shape of'shift' and the shape of 'axis' must be "
f"the same, but got the length of'shift' {len(self.shift)} and the length of 'axis'"
f" {len(self.axis)}.")
#遍历axis，添加一个Roll类型的op_list
for idx, _ in enumerate(self.axis):
self.op_list.append((inner.Roll(shift=self.shift[idx], axis=0), self.axis[idx]))
def construct(self, input_x):
dim = len(self.shape_op(input_x))
#遍历op_list，检查输入的维度是否正确
for single_op_roll, single_axis in self.op_list:
#检查输入的维度是否正确
_check_input_dim(single_axis, dim, self.cls_name)
#如果输入的维度小于0，则将输入的维度翻转
if single_axis < 0:
single_axis += dim
#初始化transpose_perm
transpose_perm = []
#遍历维度
for i in range(dim):
#将维度添加到transpose_perm中
transpose_perm.append(i)
#将输入的维度翻转
transpose_perm[0], transpose_perm[single_axis] = single_axis, 0
input_x = input_x.transpose(transpose_perm)
#将输入x的维度进行转置
input_x = single_op_roll(input_x)
#对输入x进行单操作滑动
input_x = input_x.transpose(transpose_perm)
#返回转置后的输入x
return input_x

37
mindspore/python/mindspore/nn/layer/combined.py
View File

@ -12,13 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
#combined操作函数主要完成让我们简化模型。减少代码重复，提高代码可读性和可维护性
"""Combined cells."""
# 从mindspore库中导入nn模块，用于提供神经网络相关的功能，如模型、损失函数、优化器等
from mindspore import nn
# 从mindspore.ops.primitive模块中导入Primitive类，用于定义基本操作（如矩阵乘法、加法等）
from mindspore.ops.primitive import Primitive
# 从mindspore._checkparam模块中导入Validator类，用于验证参数的范围和类型
from mindspore._checkparam import Validator
# 从当前模块（.normalization）中导入BatchNorm2d和BatchNorm1d类，用于实现批量归一化层
from .normalization import BatchNorm2d, BatchNorm1d
# 从当前模块（.activation）中导入get_activation和LeakyReLU函数，用于获取激活函数和LeakyReLU激活函数
from .activation import get_activation, LeakyReLU
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
@ -27,6 +33,8 @@ __all__ = [
'DenseBnAct'
]
#用于实现卷积神经网络
#结合了卷积层、批量归一化层和激活函数（如ReLU、Swish或Mish等），通常用于构建深度学习模型
class Conv2dBnAct(Cell):
r"""
@ -127,28 +135,38 @@ class Conv2dBnAct(Cell):
has_bias=has_bias,
weight_init=weight_init,
bias_init=bias_init)
#判断是否有BN(Batch Normalization)层
self.has_bn = Validator.check_bool(has_bn, "has_bn", self.cls_name)
#判断是否有激活函数
self.has_act = activation is not None
#判断是否在fake前
self.after_fake = Validator.check_bool(after_fake, "after_fake", self.cls_name)
if has_bn:
#如果有batchnorm，则创建一个batchnorm
self.batchnorm = BatchNorm2d(out_channels, eps, momentum)
if activation == "leakyrelu":
#如果activation为leakyrelu，则创建一个leakyrelu
self.activation = LeakyReLU(alpha)
else:
#如果activation不为leakyrelu，则创建一个激活函数
self.activation = get_activation(activation) if isinstance(activation, str) else activation
#如果activation不为空，且不是Cell或Primitive，则抛出错误
if activation is not None and not isinstance(self.activation, (Cell, Primitive)):
raise TypeError(f"For '{self.cls_name}', the 'activation' must be str or Cell or Primitive, "
f"but got {type(activation).__name__}.")
def construct(self, x):
x = self.conv(x)
#如果有bn层，则添加bn层
if self.has_bn:
x = self.batchnorm(x)
#如果有激活函数，则添加激活函数
if self.has_act:
x = self.activation(x)
return x
#用于实现全连接神经网络
#结合了全连接层、归一化层和激活函数
class DenseBnAct(Cell):
r"""
A combination of Dense, Batchnorm, and the activation layer.
@ -218,23 +236,34 @@ class DenseBnAct(Cell):
weight_init,
bias_init,
has_bias)
#定义激活函数
self.has_bn = Validator.check_bool(has_bn, "has_bn", self.cls_name)
#判断是否有BN层
self.has_act = activation is not None
#判断是否有激活函数
self.after_fake = Validator.check_bool(after_fake, "after_fake", self.cls_name)
if has_bn:
#如果有BN层
self.batchnorm = BatchNorm1d(out_channels, eps, momentum)
if activation == "leakyrelu":
#如果激活函数为leakyrelu
self.activation = LeakyReLU(alpha)
else:
#如果激活函数不为leakyrelu
self.activation = get_activation(activation) if isinstance(activation, str) else activation
#如果激活函数不为空且不是Cell或Primitive
if activation is not None and not isinstance(self.activation, (Cell, Primitive)):
#如果激活函数不是Cell或Primitive，抛出异常
raise TypeError(f"For '{self.cls_name}', the 'activation' must be str or Cell or Primitive, "
f"but got {type(activation).__name__}.")
def construct(self, x):
# 将x转换为维度为[B, C, H, W]的矩阵
x = self.dense(x)
# 如果有bn层，则对矩阵进行操作
if self.has_bn:
x = self.batchnorm(x)
# 如果有act层，则对矩阵进行操作
if self.has_act:
x = self.activation(x)
return x
# 返回矩阵
return x

154
mindspore/python/mindspore/nn/layer/container.py
View File

@ -13,66 +13,92 @@
# limitations under the License.
# ============================================================================
"""container"""
# 从collections库中导入OrderedDict类，用于创建一个有序字典，用于存储网络层的参数
from collections import OrderedDict
# 从abc库中导入abstractmethod类，用于定义一个抽象方法，要求子类覆盖这个方法
from abc import abstractmethod
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
__all__ = ['SequentialCell', 'CellList']
#用于检测搜索的值和类型是否有效，从而避免在神经网络中出现错误
def _valid_index(cell_num, index, op_name=None):
"""Internal function, used to detect the value and type of index."""
msg_prefix = f"For '{op_name}', the" if op_name else "The"
#检查索引类型是否为int
if not isinstance(index, int):
# 如果不是，抛出类型错误
raise TypeError(f"{msg_prefix} type of 'index' should be int, but got {type(index).__name__}.")
#检查索引是否在cell_num范围内
if not -cell_num <= index < cell_num:
#如果不在，抛出索引错误
raise IndexError(f"{msg_prefix} value of 'index' should be a number in range [{-cell_num}, {cell_num}), "
f"but got {index}.")
#返回索引对应的整数
return index % cell_num
#用于检查输入的Cell是否为Cell子类，来处理卷积神经网络中Cell合并和连接
def _valid_cell(cell, op_name=None):
"""Internal function, used to check whether the input cell is a subclass of Cell."""
#判断cell是否是Cell的子类
if issubclass(cell.__class__, Cell):
# 如果是，返回True
return True
#如果不是，拼接提示信息
msg_prefix = f"For '{op_name}'," if op_name else ""
#抛出异常
raise TypeError(f'{msg_prefix} each cell should be subclass of Cell, but got {type(cell).__name__}.')
#获得字符串前缀和索引
def _get_prefix_and_index(cells):
"""get prefix and index of parameter name in sequential cell or cell list."""
#给prefix和index赋值
prefix = ""
index = 0
#如果cells为空，则返回prefix和index
if not cells:
return prefix, index
#将cells字典转换为列表
cell_list = list(cells.items())
#定义变量first_param，first_key，second_param，second_key
first_param, first_key = None, None
second_param, second_key = None, None
#遍历cells字典
for key, cell in cell_list:
try:
#获取cell中的参数和名称
_, param = next(cell.parameters_and_names())
except StopIteration:
#如果遍历完cells字典，则跳出循环
continue
#如果first_param为空，则将参数和名称赋值给first_param和first_key
if first_param is None:
first_param = param
first_key = key
continue
#如果second_param为空，则将参数和名称赋值给second_param和second_key
second_param = param
second_key = key
#跳出循环
break
#如果first_param为空，返回prefix和index
if first_param is None:
return prefix, index
split_names = first_param.name.split(".")
#遍历split_names，从第一个元素开始，拆分出每一个元素
for idx, name in enumerate(split_names):
#如果拆分出的元素与first_key相同，则将prefix设置为拆分出的元素
if name == first_key:
prefix = ".".join(split_names[:idx])
prefix = prefix + "." if prefix else prefix
index = idx
#如果second_param不为空，且拆分出的元素与second_key相同，则结束循环
if second_param is not None and second_param.name.split(".")[idx] == second_key:
break
#返回prefix和index
return prefix, index
@ -93,16 +119,18 @@ class _CellListBase:
@abstractmethod
def __len__(self):
#返回自身的长度
pass
@abstractmethod
def __getitem__(self, index):
#返回索引index对应的元素
pass
def construct(self):
raise NotImplementedError
#用于表示一个顺序存储的单元格列表，构造Cell顺序容器
#其继承了_CellListBase来实现相关方法
class SequentialCell(Cell):
"""
Sequential Cell container. For more details about Cell, please refer to
@ -162,82 +190,122 @@ class SequentialCell(Cell):
def __init__(self, *args):
"""Initialize SequentialCell."""
super(SequentialCell, self).__init__()
#初始化参数
self._is_dynamic_name = []
#初始化变量
if len(args) == 1:
cells = args[0]
#如果参数是列表，则将其转换为字典
if isinstance(cells, list):
for index, cell in enumerate(cells):
#将子类插入到cell中
self.insert_child_to_cell(str(index), cell)
#将cell的参数名更新为index+"."
cell.update_parameters_name(str(index) + ".")
#将变量添加到变量列表中
self._is_dynamic_name.append(True)
#如果参数是字典，则将其转换为列表
elif isinstance(cells, OrderedDict):
for name, cell in cells.items():
#将子类插入到cell中
self.insert_child_to_cell(name, cell)
#将cell的参数名更新为name+"."
cell.update_parameters_name(name + ".")
#将变量添加到变量列表中
self._is_dynamic_name.append(False)
else:
#返回错误
raise TypeError(f"For '{self.__class__.__name__}', the 'args[0]' must be list or orderedDict, "
f"but got {type(cells).__name__}")
else:
for index, cell in enumerate(args):
#将cell插入到cell_list中
self.insert_child_to_cell(str(index), cell)
#将cell的参数名添加到_is_dynamic_name中
cell.update_parameters_name(str(index) + ".")
self._is_dynamic_name.append(True)
#将cells转换为列表
self.cell_list = list(self._cells.values())
#在这里我们来解释一下slice类，其用于表示切片来用于从序列之中获取子序列的抽象概念
#用于实现索引访问单元列表操作
def __getitem__(self, index):
if isinstance(index, slice):
#如果index是一个slice，则返回一个新的OrderedDict，其中包含self._cells中的指定位置的元素
return self.__class__(
OrderedDict(list(self._cells.items())[index]))
#如果index是一个整数，则检查index是否在self._cells中，若在则返回self._cells中的指定位置的元素，否则抛出异常
index = _valid_index(len(self), index, self.__class__.__name__)
#返回list
return list(self._cells.values())[index]
#用于实现索引访问单元列表操作
def __setitem__(self, index, cell):
cls_name = self.__class__.__name__
#检查cell是否符合要求
if _valid_cell(cell, cls_name):
#获取cell的前缀和索引
prefix, _ = _get_prefix_and_index(self._cells)
#检查索引是否合法
index = _valid_index(len(self), index, cls_name)
#获取cell的键
key = list(self._cells.keys())[index]
#将cell添加到self._cells中
self._cells[key] = cell
#更新cell的参数名
cell.update_parameters_name(prefix + key + ".")
#更新self.cell_list
self.cell_list = list(self._cells.values())
#用于实现删除索引单元列表操作
def __delitem__(self, index):
cls_name = self.__class__.__name__
if isinstance(index, int):
#如果index是int类型，则将index转换为_valid_index函数的返回值
index = _valid_index(len(self), index, cls_name)
#获取key
key = list(self._cells.keys())[index]
#删除key
del self._cells[key]
#删除is_dynamic_name中index
del self._is_dynamic_name[index]
elif isinstance(index, slice):
#如果index是slice类型，则获取keys
keys = list(self._cells.keys())[index]
#遍历keys，删除key
for key in keys:
del self._cells[key]
#删除is_dynamic_name中index
del self._is_dynamic_name[index]
else:
#如果index不是int类型或者slice类型，则抛出TypeError异常
raise TypeError(f"For '{cls_name}', the type of index should be int type or slice type, "
f"but got {type(index).__name__}")
#获取prefix和key_index
prefix, key_index = _get_prefix_and_index(self._cells)
#创建一个临时字典
temp_dict = OrderedDict()
#遍历cells，将key和cell添加到temp_dict中
for idx, key in enumerate(self._cells.keys()):
cell = self._cells[key]
#如果is_dynamic_name中idx为True，则将cell添加到temp_dict中
if self._is_dynamic_name[idx]:
for _, param in cell.parameters_and_names():
param.name = prefix + str(idx) + "." + ".".join(param.name.split(".")[key_index+1:])
temp_dict[str(idx)] = cell
else:
temp_dict[key] = cell
#将temp_dict中的值赋值给cells
self._cells = temp_dict
#将cells中的值赋值给self.cell_list
self.cell_list = list(self._cells.values())
#获取长度
def __len__(self):
#返回长度
return len(self._cells)
#设置单元表格梯度
def set_grad(self, flag=True):
self.requires_grad = flag
#检查是否设置梯度
for cell in self._cells.values():
cell.set_grad(flag)
def append(self, cell):
"""
Appends a given Cell to the end of the list.
@ -264,16 +332,23 @@ class SequentialCell(Cell):
[[26.999863 26.999863]
[26.999863 26.999863]]]]
"""
#_valid_cell函数用于检查给定的单元是否有效
if _valid_cell(cell, self.__class__.__name__):
prefix, _ = _get_prefix_and_index(self._cells)
#将当前cell的名称添加到prefix中
cell.update_parameters_name(prefix + str(len(self)) + ".")
#将当前cell添加到self._cells中
self._is_dynamic_name.append(True)
#将当前cell添加到self._is_dynamic_name中
self._cells[str(len(self))] = cell
#将self._cells中的值赋值给self.cell_list
self.cell_list = list(self._cells.values())
#创建表格
def construct(self, input_data):
for cell in self.cell_list:
#调用cell函数，传入input_data参数
input_data = cell(input_data)
#返回input_data参数值
return input_data
@ -301,63 +376,84 @@ class CellList(_CellListBase, Cell):
>>> cell_ls.append(relu)
>>> cell_ls.extend([relu, relu])
"""
#初始化
def __init__(self, *args, **kwargs):
"""Initialize CellList."""
auto_prefix = kwargs["auto_prefix"] if "auto_prefix" in kwargs.keys() else True
#初始化CellListBase类
_CellListBase.__init__(self)
#初始化Cell类
Cell.__init__(self, auto_prefix)
#如果只有一个参数，则将其追加到CellListBase中
if len(args) == 1:
self.extend(args[0])
#调用
def __getitem__(self, index):
cls_name = self.__class__.__name__
#如果index是slice类型，则返回一个新的list，其中包含self._cells中的值
if isinstance(index, slice):
return self.__class__(list(self._cells.values())[index])
#如果index是int类型，则根据index的长度，获取对应的index，并返回self._cells中对应的值
if isinstance(index, int):
index = _valid_index(len(self), index, cls_name)
return self._cells[str(index)]
#如果index的类型不是int或slice，则抛出TypeError异常
raise TypeError(f"For '{cls_name}', the type of 'index' should be int or slice, "
f"but got {type(index).__name__}.")
#修改对象数值
def __setitem__(self, index, cell):
cls_name = self.__class__.__name__
#判断索引是否为整数，并且cell是否为有效cell
if not isinstance(index, int) and _valid_cell(cell, cls_name):
raise TypeError(f"For '{cls_name}', the type of 'index' should be int, "
f"but got {type(index).__name__}.")
#如果自动前缀为True，则获取前缀和索引
index = _valid_index(len(self), index, cls_name)
#如果自动前缀为True，则更新参数名
if self._auto_prefix:
prefix, _ = _get_prefix_and_index(self._cells)
cell.update_parameters_name(prefix + str(index) + ".")
#将索引和cell添加到self._cells中
self._cells[str(index)] = cell
#用于实现删除索引单元列表操作
def __delitem__(self, index):
cls_name = self.__class__.__name__
if isinstance(index, int):
#如果index是int类型，则将index转换为_valid_index函数的返回值
index = _valid_index(len(self), index, cls_name)
#将_valid_index函数的返回值赋值给index
del self._cells[str(index)]
elif isinstance(index, slice):
#如果index是slice类型，则遍历self._cells字典，将每一项删除
keys = list(self._cells.keys())[index]
for key in keys:
del self._cells[key]
else:
#如果index不是int类型或slice类型，则抛出TypeError异常
raise TypeError(f"For '{cls_name}', the type of 'index' should be int or slice, "
f"but got {type(index).__name__}.")
# adjust orderedDict
prefix, key_index = _get_prefix_and_index(self._cells)
#创建一个空，用于存储cell
temp_dict = OrderedDict()
#遍历cells中的每一个cell
for idx, cell in enumerate(self._cells.values()):
#如果自动前缀，则把cell中的参数名称添加到前缀中
if self._auto_prefix:
for _, param in cell.parameters_and_names():
param.name = prefix + str(idx) + "." + ".".join(param.name.split(".")[key_index+1:])
#添加cell
temp_dict[str(idx)] = cell
#更新cell
self._cells = temp_dict
#获取长度
def __len__(self):
#返回长度
return len(self._cells)
#返回迭代器
def __iter__(self):
return iter(self._cells.values())
#实现对象间加法
def __iadd__(self, cells):
self.extend(cells)
return self
@ -385,7 +481,7 @@ class CellList(_CellListBase, Cell):
self._cells[str(idx)] = cell
if self._auto_prefix:
cell.update_parameters_name(prefix + str(idx) + ".")
#迭代器，将 cells 中的每个元素添加到 MyContainer 对象的末尾
def extend(self, cells):
"""
Appends Cells from a Python iterable to the end of the list.
@ -397,15 +493,23 @@ class CellList(_CellListBase, Cell):
TypeError: If the argument cells are not a list of Cells.
"""
cls_name = self.__class__.__name__
#判断cells是否为list类型
if not isinstance(cells, list):
# 如果不是，抛出错误
raise TypeError(f"For '{cls_name}', the new cells wanted to append "
f"should be instance of list, but got {type(cells).__name__}.")
#获取cells的前缀和索引
prefix, _ = _get_prefix_and_index(self._cells)
#遍历cells
for cell in cells:
#判断cell是否有效
if _valid_cell(cell, cls_name):
#如果自动前缀，更新cell的参数名
if self._auto_prefix:
cell.update_parameters_name(prefix + str(len(self)) + ".")
#将cell添加到self._cells中
self._cells[str(len(self))] = cell
#返回self
return self
def append(self, cell):
@ -416,15 +520,21 @@ class CellList(_CellListBase, Cell):
cell(Cell): The subcell to be appended.
"""
if _valid_cell(cell, self.__class__.__name__):
#如果cell是有效的cell，则更新cell的参数名
if self._auto_prefix:
#获取cell的前缀和索引
prefix, _ = _get_prefix_and_index(self._cells)
#更新cell的参数名
cell.update_parameters_name(prefix + str(len(self)) + ".")
#将cell添加到self._cells中
self._cells[str(len(self))] = cell
#设置单元表格梯度
def set_grad(self, flag=True):
self.requires_grad = flag
#检查是否设置梯度
for cell in self._cells.values():
cell.set_grad(flag)
#创建表格
def construct(self, *inputs):
raise NotImplementedError

218
mindspore/python/mindspore/nn/layer/conv.py
View File

@ -13,21 +13,36 @@
# limitations under the License.
# ============================================================================
"""conv"""
# 导入numpy库，用于处理数值计算
import numpy as np
# 从mindspore库中导入log模块，用于提供日志记录功能
from mindspore import log as logger
# 从mindspore库中导入context模块，用于获取和设置运行环境
from mindspore import context
# 从mindspore.ops模块中导入operations模块，用于提供基本操作（如矩阵乘法、加法等）
from mindspore.ops import operations as P
# 从mindspore.ops.primitive模块中导入constexpr类，用于定义一个常量
from mindspore.ops.primitive import constexpr
# 从mindspore.common.parameter模块中导入Parameter类，用于创建一个参数
from mindspore.common.parameter import Parameter
# 从mindspore.common.initializer模块中导入initializer函数，用于初始化参数
from mindspore.common.initializer import initializer
# 从mindspore.common.tensor模块中导入Tensor类，用于表示张量
from mindspore.common.tensor import Tensor
# 从mindspore._checkparam模块中导入Validator、Rel、twice和_check_3d_int_or_tuple函数，用于验证参数的范围和类型
from mindspore._checkparam import Validator, Rel, twice, _check_3d_int_or_tuple
# 从mindspore._extends模块中导入cell_attr_register函数，用于注册Cell类的属性
from mindspore._extends import cell_attr_register
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
# P：用于定义MindSpore操作的模块（操作是可以在MindSpore设备上执行的函数）
# constexpr：装饰器，用于标记一个函数为常量表达式，将在编译时执行
# Parameter：用于定义模型参数的类
# initializer：用于定义初始化参数的模块
# Validator：用于验证参数的类
__all__ = ['Conv2d', 'Conv2dTranspose', 'Conv1d', 'Conv1dTranspose', 'Conv3d', 'Conv3dTranspose']
# 用于实现卷积层
class _Conv(Cell):
"""
Applies a N-D convolution over an input signal composed of several input planes.
@ -51,61 +66,80 @@ class _Conv(Cell):
super(_Conv, self).__init__()
self.in_channels = Validator.check_positive_int(in_channels, 'in_channels', self.cls_name)
self.out_channels = Validator.check_positive_int(out_channels, 'out_channels', self.cls_name)
# 定义卷积核的大小
self.kernel_size = kernel_size
# 定义步长
self.stride = stride
# 定义填充模式
self.pad_mode = pad_mode
# 定义权重初始化方式
self.weight_init = weight_init
# 定义偏置初始化方式
self.bias_init = bias_init
# 检查数据格式
self.format = Validator.check_string(data_format, ['NCHW', 'NHWC', 'NCDHW'], 'format', self.cls_name)
if context.get_context("device_target") != "GPU" and self.format == "NHWC":
# 检查设备目标是否为GPU，并且格式是否为NHWC
raise ValueError(f"For '{self.cls_name}', the \"NHWC\" format only support in GPU target, "
f"but got the 'format' is {self.format} and "
f"the platform is {context.get_context('device_target')}.")
if isinstance(padding, int):
# 检查padding是否为非负整数
Validator.check_non_negative_int(padding, 'padding', self.cls_name)
self.padding = padding
elif isinstance(padding, tuple):
# 检查padding是否为元组
for pad in padding:
Validator.check_non_negative_int(pad, 'padding item', self.cls_name)
self.padding = padding
else:
# 抛出类型错误
raise TypeError(f"For '{self.cls_name}', the type of 'padding' must be int or tuple(int), "
f"but got {type(padding).__name__}.")
self.dilation = dilation
self.group = Validator.check_positive_int(group)
self.has_bias = has_bias
# 检查kernel_size是否为正整数
for kernel_size_elem in kernel_size:
Validator.check_positive_int(kernel_size_elem, 'kernel_size item', self.cls_name)
# 检查stride是否为正整数
for stride_elem in stride:
Validator.check_positive_int(stride_elem, 'stride item', self.cls_name)
Validator.check_positive_int(stride_elem,'stride item', self.cls_name)
# 检查dilation是否为正整数
for dilation_elem in dilation:
Validator.check_positive_int(dilation_elem, 'dilation item', self.cls_name)
if in_channels % group != 0:
# 检查in_channels是否为偶数
if in_channels % group!= 0:
raise ValueError(f"For '{self.cls_name}', the attr 'in_channels' must be divisible by attr 'group', "
f"but got 'in_channels': {in_channels} and 'group': {group}.")
if out_channels % group != 0:
# 检查out_channels是否为偶数
if out_channels % group!= 0:
raise ValueError(f"For '{self.cls_name}', the 'out_channels' must be divisible by attr 'group', "
f"but got 'out_channels': {out_channels} and 'group': {group}.")
if transposed:
# 如果transposed为True，则设置weight的shape为[in_channels, out_channels//group, *kernel_size]
shape = [in_channels, out_channels // group, *kernel_size]
else:
# 如果transposed为False，则设置weight的shape为[out_channels, *kernel_size, in_channels//group]
shape = [out_channels, *kernel_size, in_channels // group] if self.format == "NHWC" else \
[out_channels, in_channels // group, *kernel_size]
# 将weight初始化为输入的weight_init
self.weight = Parameter(initializer(self.weight_init, shape), name='weight')
# 如果has_bias为True，则设置bias的shape为[out_channels]
if Validator.check_bool(has_bias, "has_bias", self.cls_name):
self.bias = Parameter(initializer(self.bias_init, [out_channels]), name='bias')
# 如果has_bias为False，则设置bias的shape为[out_channels]，并将bias的初始值设置为0
else:
if self.bias_init != 'zeros':
if self.bias_init!='zeros':
logger.warning("Value of 'has_bias' is False, value of 'bias_init' will be ignored.")
self.bias = None
# 如果没有子类覆盖这个方法则抛出一个未实现异常
def construct(self, *inputs):
"""Must be overridden by all subclasses."""
raise NotImplementedError
# 用于实现二维卷积操作
class Conv2d(_Conv):
r"""
2D convolution layer.
@ -240,17 +274,29 @@ class Conv2d(_Conv):
@cell_attr_register
def __init__(self,
# 输入通道数
in_channels,
# 输出通道数
out_channels,
# 卷积核大小
kernel_size,
# 步长为1
stride=1,
# 填充模式为'same'
pad_mode='same',
# 填充数量为0
padding=0,
# 空洞大小为1
dilation=1,
# 分组数为1
group=1,
# 权重初始化方式为False
has_bias=False,
# 权重初始化方式'normal'
weight_init='normal',
# 权重初始化方式'zeros'
bias_init='zeros',
# 数据格式为'NCHW'
data_format='NCHW'):
"""Initialize Conv2d."""
kernel_size = twice(kernel_size)
@ -258,17 +304,29 @@ class Conv2d(_Conv):
self._dilation = dilation
dilation = twice(dilation)
super(Conv2d, self).__init__(
# 输入通道数
in_channels,
# 输出通道数
out_channels,
# 卷积核大小
kernel_size,
# 步长
stride,
# 填充模式
pad_mode,
# 填充数量
padding,
# 空洞大小
dilation,
# 分组数
group,
# 是否包含偏置
has_bias,
# 权重初始化方式
weight_init,
# 偏置初始化方式
bias_init,
# 数据格式
data_format)
self.conv2d = P.Conv2D(out_channel=self.out_channels,
kernel_size=self.kernel_size,
@ -279,14 +337,16 @@ class Conv2d(_Conv):
dilation=self.dilation,
group=self.group,
data_format=self.format)
# 初始化卷积层
self.bias_add = P.BiasAdd(data_format=self.format)
def construct(self, x):
output = self.conv2d(x, self.weight)
# 如果有偏置，则在输出的后面添加偏置
if self.has_bias:
output = self.bias_add(output, self.bias)
# 返回输出
return output
# 扩展输出repr,添加卷积操作的额外信息
def extend_repr(self):
s = 'input_channels={}, output_channels={}, kernel_size={}, ' \
'stride={}, pad_mode={}, padding={}, dilation={}, ' \
@ -309,10 +369,12 @@ class Conv2d(_Conv):
@constexpr
def _check_input_3d(input_shape, op_name):
# 如果长度不是3
if len(input_shape) != 3:
#抛出异常
raise ValueError(f"For '{op_name}', the dimension of input should be 3d, but got {len(input_shape)}.")
# 用于实现一维卷积
class Conv1d(_Conv):
r"""
1D convolution layer.
@ -413,37 +475,60 @@ class Conv1d(_Conv):
@cell_attr_register
def __init__(self,
# 输入通道数
in_channels,
# 输出通道数
out_channels,
# 卷积核大小
kernel_size,
# 步长为1
stride=1,
# 填充模式为'same'
pad_mode='same',
# 填充数量为0
padding=0,
# 空洞大小为1
dilation=1,
# 分组数为1
group=1,
# 权重初始化方式为False
has_bias=False,
# 权重初始化方式'normal'
weight_init='normal',
bias_init='zeros'):
# 权重初始化方式'zeros'
bias_init='zeros',):
"""Initialize Conv1d."""
Validator.check_value_type("kernel_size", kernel_size, [int], self.cls_name)
Validator.check_value_type("stride", stride, [int], self.cls_name)
Validator.check_value_type("padding", padding, [int], self.cls_name)
Validator.check_value_type("dilation", dilation, [int], self.cls_name)
Validator.check_int(kernel_size, 1, Rel.GE, 'kernel_size', self.cls_name)
Validator.check_int(stride, 1, Rel.GE, 'stride', self.cls_name)
Validator.check_int(stride, 1, Rel.GE,'stride', self.cls_name)
Validator.check_non_negative_int(padding, 'padding', self.cls_name)
Validator.check_int(dilation, 1, Rel.GE, 'dilation', self.cls_name)
# 判断kernel_size是否为整数
kernel_size = (1, kernel_size)
# 判断stride是否为整数
stride = (1, stride)
# 判断dilation是否为整数
dilation = (1, dilation)
# 获取形状
get_shape = P.Shape()
# 获取数据类型
get_dtype = P.DType()
# 判断weight_init是否为Tensor类型
if isinstance(weight_init, Tensor):
# 获取weight_init的形状
weight_init_shape = get_shape(weight_init)
# 判断weight_init的形状是否为3维
Validator.check_equal_int(len(weight_init_shape), 3, 'weight_init_shape', self.cls_name)
# 获取weight_init的数据类型
weight_init_dtype = get_dtype(weight_init)
# 获取weight_init的值
weight_init_value = weight_init.asnumpy()
# 将weight_init的值按照2维拉直
weight_init_value = np.expand_dims(weight_init_value, 2)
# 将weight_init转换为Tensor类型
weight_init = Tensor(weight_init_value, weight_init_dtype)
super(Conv1d, self).__init__(
@ -458,8 +543,10 @@ class Conv1d(_Conv):
has_bias,
weight_init,
bias_init)
# 定义卷积层参数
self.padding = (0, 0, padding, padding)
Validator.check_string(pad_mode, ['valid', 'same', 'pad'], 'pad_mode', self.cls_name)
Validator.check_string(pad_mode, ['valid','same', 'pad'], 'pad_mode', self.cls_name)
# 定义卷积层
self.conv2d = P.Conv2D(out_channel=self.out_channels,
kernel_size=self.kernel_size,
mode=1,
@ -468,19 +555,28 @@ class Conv1d(_Conv):
stride=self.stride,
dilation=self.dilation,
group=self.group)
# 定义BiasAdd层
self.bias_add = P.BiasAdd()
# 定义ExpandDims层
self.expand_dims = P.ExpandDims()
# 定义Squeeze层
self.squeeze = P.Squeeze(2)
# 定义Shape层
self.shape = P.Shape()
def construct(self, x):
x_shape = self.shape(x)
# 检查输入的形状是否为3D
_check_input_3d(x_shape, self.cls_name)
# 将输入x展开为3D
x = self.expand_dims(x, 2)
# 将x和weight拼接起来
output = self.conv2d(x, self.weight)
# 如果有偏置，则在输出的维度上添加偏置
if self.has_bias:
output = self.bias_add(output, self.bias)
# 将输出梯度平滑
output = self.squeeze(output)
return output
@ -508,7 +604,8 @@ def _check_input_5dims(input_shape, op_name):
if len(input_shape) != 5:
raise ValueError(f"For '{op_name}', the dimension of input should be 5d, but got {len(input_shape)}.")
# 用于实现三维卷积
# 其中参数与Conv1d和Conv2d中基本一样
class Conv3d(_Conv):
r"""
3D convolution layer.
@ -663,10 +760,15 @@ class Conv3d(_Conv):
bias_init='zeros',
data_format='NCDHW'):
"""Initialize Conv3d."""
# 检查kernel_size参数是否为整数或元组并且进行类型转换
kernel_size = _check_3d_int_or_tuple("kernel_size", kernel_size, self.cls_name)
# 检查stride参数是否为整数或元组并且进行类型转换
stride = _check_3d_int_or_tuple("stride", stride, self.cls_name)
# 检查dilation参数是否为整数或元组并且进行类型转换
dilation = _check_3d_int_or_tuple("dilation", dilation, self.cls_name)
# 检查padding参数是否为整数或元组
Validator.check_value_type('padding', padding, (int, tuple), self.cls_name)
# 如果padding的类型是元组，检查padding的长度是否为6
if isinstance(padding, tuple):
Validator.check_equal_int(len(padding), 6, 'padding size', self.cls_name)
super(Conv3d, self).__init__(
@ -696,16 +798,19 @@ class Conv3d(_Conv):
def construct(self, x):
x_shape = self.shape(x)
# 检查输入的形状是否为5维
_check_input_5dims(x_shape, self.cls_name)
# 调用conv3d函数，将x转换为3维
output = self.conv3d(x, self.weight)
# 如果有偏置，调用bias_add函数，将偏置和输出添加到输出上
if self.has_bias:
output = self.bias_add(output, self.bias)
# 返回输出
return output
def extend_repr(self):
s = 'input_channels={}, output_channels={}, kernel_size={}, ' \
'stride={}, pad_mode={}, padding={}, dilation={}, ' \
'group={}, has_bias={}, ' \
'stride={}, pad_mode={}, padding={}, dilation={},'\
'group={}, has_bias={},'\
'weight_init={}, bias_init={}, format={}'.format(
self.in_channels,
self.out_channels,
@ -721,7 +826,8 @@ class Conv3d(_Conv):
self.format)
return s
# 实现三维卷积转置操作
# 其中参数也和前面一致
class Conv3dTranspose(_Conv):
r"""
3D transposed convolution layer.
@ -875,9 +981,12 @@ class Conv3dTranspose(_Conv):
kernel_size = _check_3d_int_or_tuple("kernel_size", kernel_size, self.cls_name)
stride = _check_3d_int_or_tuple("stride", stride, self.cls_name)
dilation = _check_3d_int_or_tuple("dilation", dilation, self.cls_name)
# 检查padding的类型是否为整数或元组
Validator.check_value_type('padding', padding, (int, tuple), self.cls_name)
# 如果padding是元组，检查padding的长度是否等于6
if isinstance(padding, tuple):
Validator.check_equal_int(len(padding), 6, 'padding size', self.cls_name)
# 检查output_padding的类型是否为整数或元组，且大于0
self.output_padding = _check_3d_int_or_tuple("output_padding", output_padding, self.cls_name,
greater_zero=False)
super(Conv3dTranspose, self).__init__(
@ -905,13 +1014,17 @@ class Conv3dTranspose(_Conv):
group=self.group,
output_padding=self.output_padding,
data_format=self.format)
# 定义卷积转换层的变换操作
self.bias_add = P.BiasAdd(data_format=self.format)
# 定义偏置和形状
self.shape = P.Shape()
def construct(self, x):
x_shape = self.shape(x)
_check_input_5dims(x_shape, self.cls_name)
# 调用conv3d_transpose函数，将x转换为3D维度
output = self.conv3d_transpose(x, self.weight)
# 如果有偏置，调用bias_add函数，将转换后的结果加到output上
if self.has_bias:
output = self.bias_add(output, self.bias)
return output
@ -937,20 +1050,31 @@ class Conv3dTranspose(_Conv):
def _deconv_output_length(is_valid, is_same, is_pad, input_length, filter_size, stride_size, dilation_size, padding):
"""Calculate the width and height of output."""
length = 0
# 计算滤波器的长度
filter_size = filter_size + (filter_size - 1) * (dilation_size - 1)
# 如果是有效的
if is_valid:
# 如果滤波器的长度大于等于输入长度的步长
if filter_size - stride_size > 0:
# 计算输入长度的步长
length = input_length * stride_size + filter_size - stride_size
# 如果滤波器的长度小于等于输入长度的步长
else:
# 计算输入长度的步长
length = input_length * stride_size
# 如果是填充的
elif is_same:
# 计算输入长度的步长
length = input_length * stride_size
# 如果是填充的
elif is_pad:
# 计算输入长度的步长
length = input_length * stride_size - padding + filter_size - stride_size
return length
# 实现三维卷积转置操作
# 其中参数也和前面一致
class Conv2dTranspose(_Conv):
r"""
2D transposed convolution layer.
@ -1083,7 +1207,9 @@ class Conv2dTranspose(_Conv):
kernel_size = twice(kernel_size)
stride = twice(stride)
dilation = twice(dilation)
# 检查padding的类型是否为int或tuple
Validator.check_value_type('padding', padding, (int, tuple), self.cls_name)
# 如果padding的类型是tuple，检查padding的长度是否为4
if isinstance(padding, tuple):
Validator.check_equal_int(len(padding), 4, 'padding size', self.cls_name)
# out_channels and in_channels swap.
@ -1106,11 +1232,14 @@ class Conv2dTranspose(_Conv):
self.in_channels = in_channels
self.out_channels = out_channels
self.shape = P.Shape()
Validator.check_string(pad_mode, ['valid', 'same', 'pad'], 'pad_mode', self.cls_name)
Validator.check_string(pad_mode, ['valid','same', 'pad'], 'pad_mode', self.cls_name)
# 判断pad_mode是否为valid，same，pad
self.is_valid = self.pad_mode == 'valid'
self.is_same = self.pad_mode == 'same'
self.is_same = self.pad_mode =='same'
self.is_pad = self.pad_mode == 'pad'
# 判断has_bias是否为布尔值
if Validator.check_bool(has_bias, "has_bias", self.cls_name):
# 初始化偏置参数
self.bias = Parameter(initializer(bias_init, [out_channels]), name='bias')
# cause Conv2DTranspose's out_channel refers to Conv2D's out_channel.
@ -1123,6 +1252,7 @@ class Conv2dTranspose(_Conv):
dilation=dilation,
group=group)
self.bias_add = P.BiasAdd()
# 如果padding的类型是int，则将padding设置为padding的值，否则设置为4倍padding值
if isinstance(self.padding, int):
self.padding_top, self.padding_bottom, self.padding_left, self.padding_right = (self.padding,) * 4
else:
@ -1134,10 +1264,12 @@ class Conv2dTranspose(_Conv):
def construct(self, x):
n, _, h, w = self.shape(x)
# 计算输出长度
h_out = _deconv_output_length(self.is_valid, self.is_same, self.is_pad, h, self.kernel_size[0],
self.stride[0], self.dilation[0], self.padding_top + self.padding_bottom)
w_out = _deconv_output_length(self.is_valid, self.is_same, self.is_pad, w, self.kernel_size[1],
self.stride[1], self.dilation[1], self.padding_left + self.padding_right)
# 如果有偏置，则在输出上添加偏置
if self.has_bias:
return self.bias_add(self.conv2d_transpose(x, self.weight, (n, self.out_channels, h_out, w_out)),
self.bias)
@ -1145,8 +1277,8 @@ class Conv2dTranspose(_Conv):
def extend_repr(self):
s = 'input_channels={}, output_channels={}, kernel_size={}, ' \
'stride={}, pad_mode={}, padding={}, dilation={}, ' \
'group={}, has_bias={}, ' \
'stride={}, pad_mode={}, padding={}, dilation={},'\
'group={}, has_bias={},'\
'weight_init={}, bias_init={}'.format(self.in_channels,
self.out_channels,
self.kernel_size,
@ -1160,7 +1292,9 @@ class Conv2dTranspose(_Conv):
self.bias_init)
return s
# 实现一维卷积转置操作
# 其中参数和前面一致
# 函数中逻辑与过程也一致
class Conv1dTranspose(_Conv):
r"""
1D transposed convolution layer.
@ -1264,25 +1398,39 @@ class Conv1dTranspose(_Conv):
weight_init='normal',
bias_init='zeros'):
"""Initialize Conv1dTranspose."""
# 检查kernel_size的类型是否为int型
Validator.check_value_type("kernel_size", kernel_size, [int], self.cls_name)
# 检查stride的类型是否为int型
Validator.check_value_type("stride", stride, [int], self.cls_name)
# 检查padding的类型是否为int型
Validator.check_value_type("padding", padding, [int], self.cls_name)
# 检查dilation的类型是否为int型
Validator.check_value_type("dilation", dilation, [int], self.cls_name)
# 验证kernel_size参数是否为正整数，且大于等于
Validator.check_int(kernel_size, 1, Rel.GE, 'kernel_size', self.cls_name)
Validator.check_int(stride, 1, Rel.GE, 'stride', self.cls_name)
# 验证stride参数是否为正整数，且大于等于
Validator.check_int(stride, 1, Rel.GE,'stride', self.cls_name)
# 验证padding参数是否为非负整数
Validator.check_non_negative_int(padding, 'padding', self.cls_name)
# 验证dilation参数是否为正整数，且大于等于1
Validator.check_int(dilation, 1, Rel.GE, 'dilation', self.cls_name)
kernel_size = (1, kernel_size)
stride = (1, stride)
dilation = (1, dilation)
# 获取形状和数据类型
get_shape = P.Shape()
get_dtype = P.DType()
# 判断权重初始化是否为Tensor
if isinstance(weight_init, Tensor):
# 获取权重初始化的形状和数据类型
weight_init_shape = get_shape(weight_init)
Validator.check_equal_int(len(weight_init_shape), 3, 'weight_init_shape', self.cls_name)
weight_init_dtype = get_dtype(weight_init)
# 获取权重初始化的值
weight_init_value = weight_init.asnumpy()
# 将值按照维度增加2
weight_init_value = np.expand_dims(weight_init_value, 2)
# 将值转换为Tensor
weight_init = Tensor(weight_init_value, weight_init_dtype)
# out_channels and in_channels swap.
# cause Conv2DBackpropInput's out_channel refers to Conv2D's out_channel,
@ -1301,13 +1449,15 @@ class Conv1dTranspose(_Conv):
bias_init,
transposed=True)
self.padding = (0, 0, padding, padding)
# 定义padding，可以是任意数量的值，如果是valid，则padding为0，same，则padding为padding，pad，则padding为(padding, padding)
self.in_channels = in_channels
self.out_channels = out_channels
self.shape = P.Shape()
Validator.check_string(pad_mode, ['valid', 'same', 'pad'], 'pad_mode', self.cls_name)
Validator.check_string(pad_mode, ['valid','same', 'pad'], 'pad_mode', self.cls_name)
self.is_valid = self.pad_mode == 'valid'
self.is_same = self.pad_mode == 'same'
self.is_same = self.pad_mode =='same'
self.is_pad = self.pad_mode == 'pad'
# 判断has_bias是否为布尔值，如果是，则初始化bias，并设置name
if Validator.check_bool(has_bias, "has_bias", self.cls_name):
self.bias = Parameter(initializer(bias_init, [out_channels]), name='bias')
@ -1320,8 +1470,11 @@ class Conv1dTranspose(_Conv):
stride=stride,
dilation=dilation,
group=group)
# 定义BiasAdd层
self.bias_add = P.BiasAdd()
# 定义ExpandDims层
self.expand_dims = P.ExpandDims()
# 定义Squeeze层
self.squeeze = P.Squeeze(2)
def shard(self, strategy):
@ -1331,21 +1484,26 @@ class Conv1dTranspose(_Conv):
def construct(self, x):
x_shape = self.shape(x)
_check_input_3d(x_shape, self.cls_name)
# 将x的形状和kernel_size和stride和dilation属性转换成一个元组
x = self.expand_dims(x, 2)
# 获取x的形状
n, _, h, w = self.shape(x)
# 计算h_out和w_out
h_out = _deconv_output_length(self.is_valid, self.is_same, self.is_pad, h, self.kernel_size[0],
self.stride[0], self.dilation[0], self.padding[0] + self.padding[1])
w_out = _deconv_output_length(self.is_valid, self.is_same, self.is_pad, w, self.kernel_size[1],
self.stride[1], self.dilation[1], self.padding[2] + self.padding[3])
# 将x和weight进行卷积转换
output = self.conv2d_transpose(x, self.weight, (n, self.out_channels, h_out, w_out))
# 如果有偏置，则在转换后的结果中加上偏置
if self.has_bias:
output = self.bias_add(output, self.bias)
# 将转换后的结果转换成一个元组
output = self.squeeze(output)
return output
def extend_repr(self):
s = 'input_channels={}, output_channels={}, kernel_size={}, ' \
'stride={}, pad_mode={}, padding={}, dilation={}, ' \

282
mindspore/python/mindspore/nn/layer/embedding.py
View File

@ -13,41 +13,67 @@
# limitations under the License.
# ============================================================================
"""embedding"""
# 从mindspore.common.dtype模块中导入mstype类，用于表示数据类型
import mindspore.common.dtype as mstype
# 从mindspore库中导入log模块，用于提供日志记录功能
from mindspore import log as logger
# 从mindspore.common.tensor模块中导入Tensor类，用于表示张量
from mindspore.common.tensor import Tensor
# 从mindspore.ops模块中导入operations模块，用于提供基本操作（如矩阵乘法、加法等）
from mindspore.ops import operations as P
# 从mindspore.ops.functional模块中导入functional类，用于提供一些基本功能，如条件判断、循环等
from mindspore.ops import functional as F
# 从mindspore.common.parameter模块中导入Parameter类，用于创建一个参数
from mindspore.common.parameter import Parameter
# 从mindspore.common.initializer模块中导入initializer函数，用于初始化参数
from mindspore.common.initializer import initializer
# mindspore.communication.management模块中导入get_group_size和get_rank函数，用于获取集群中组的大小和当前节点的排名
from mindspore.communication.management import get_group_size, get_rank
# 从mindspore.context模块中导入ParallelMode类，用于表示并行模式
from mindspore.context import ParallelMode
# 从mindspore.parallel._utils模块中导入_get_parallel_mode和_get_full_batch函数，用于获取并行模式和全批量大小
from mindspore.parallel._utils import _get_parallel_mode, _get_full_batch
# 从mindspore.parallel._ps_context模块中导入_is_role_worker和_get_ps_context函数
# 用于获取当前节点是否为工作节点和获取PS上下文
from mindspore.parallel._ps_context import _is_role_worker, _get_ps_context
# 从mindspore.parallel._ps_context模块中导入_insert_hash_table_size、_set_cache_enable和_set_rank_id函数
# 用于设置哈希表大小、缓存启用和节点排名
from mindspore.parallel._ps_context import _insert_hash_table_size, _set_cache_enable, _set_rank_id
# 从mindspore库中导入context模块，用于获取和设置运行环境
from mindspore import context
# 从mindspore._checkparam模块中导入Rel类，用于表示相对关系
from mindspore._checkparam import Rel
# 从mindspore._checkparam模块中导入Validator类，用于验证参数的范围和类型
from mindspore._checkparam import Validator as validator
# 从mindspore.ops.primitive模块中导入constexpr类，用于定义一个常量
from mindspore.ops.primitive import constexpr
# 从当前模块（..basic）中导入ClipByNorm类，用于裁剪张量范数
from .basic import ClipByNorm
# 从当前模块（..math）中导入Range类，用于创建一个范围
from .math import Range
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
__all__ = ['Embedding', 'EmbeddingLookup', 'MultiFieldEmbeddingLookup']
@constexpr
# 检查模型输入参数是否符合预期
def _check_input_2d(input_shape, param_name, func_name):
# 如果输入形状不是二维
if len(input_shape) != 2:
# 则抛出ValueError异常
raise ValueError(f"For '{func_name}', the dimension of '{param_name}' should be 2d, but got {len(input_shape)}")
return True
@constexpr
# 检查输入数据类型是否在允许的类型列表中
def _check_input_dtype(input_dtype, param_name, allow_dtypes, cls_name):
validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name)
# Embedding嵌入层
# 用于将连续输入数据转换为嵌入向量
# 将输入数据转换为一个固定大小的嵌入向量，其中每个嵌入向量表示输入数据中的一个特定特征
class Embedding(Cell):
r"""
A simple lookup table that stores embeddings of a fixed dictionary and size.
@ -97,6 +123,15 @@ class Embedding(Cell):
def __init__(self, vocab_size, embedding_size, use_one_hot=False, embedding_table='normal',
dtype=mstype.float32, padding_idx=None):
# 主要参数
# vocab_size (int): 嵌入向量的数量
# embedding_size (int): 嵌入向量的维度
# use_one_hot (bool): 是否使用独热编码，默认为False
# embedding_table (str, optional): 嵌入向量的初始化方式，默认为'normal'
# dtype (:class:`mindspore.dtype`, optional): 数据类型，默认为`mstype.float32
# padding_idx (int, optional):在输入数据中填充的索引，默认为None。
"""Initialize Embedding."""
super(Embedding, self).__init__()
self.vocab_size = validator.check_value_type('vocab_size', vocab_size, [int], self.cls_name)
@ -105,54 +140,81 @@ class Embedding(Cell):
validator.check_subclass("dtype", dtype, mstype.number_type, self.cls_name)
self.use_one_hot = use_one_hot
self.dtype = dtype
# 初始化embedding_table
self.init_tensor = initializer(embedding_table, [vocab_size, embedding_size])
# 如果padding_idx不为空，则将padding_idx转换为整数
self.padding_idx = padding_idx
if padding_idx is not None:
self.padding_idx = validator.check_int_range(padding_idx, 0, vocab_size, Rel.INC_BOTH,
"padding_idx", self.cls_name)
# 如果init_tensor是Tensor类型，则将其转换为numpy数组
if isinstance(self.init_tensor, Tensor) and self.init_tensor.init is not None:
self.init_tensor = self.init_tensor.init_data()
# 将padding_idx转换为numpy数组
self.init_tensor = self.init_tensor.asnumpy()
# 将padding_idx中的元素设置为0
self.init_tensor[self.padding_idx] = 0
# 将转换后的numpy数组转换为Tensor类型
self.init_tensor = Tensor(self.init_tensor)
self.embedding_table = Parameter(self.init_tensor, name='embedding_table')
# 初始化embedding_table参数
self.expand = P.ExpandDims()
# 将embedding_table参数放大一倍
self.reshape_flat = P.Reshape()
# 将embedding_table参数变形为一维
self.shp_flat = (-1,)
# 获取embedding_table参数的形状
self.gather = P.Gather()
# 从embedding_table参数中抽取元素
self.one_hot = P.OneHot()
# 将embedding_table参数转换为one_hot
self.on_value = Tensor(1.0, self.dtype)
# 将embedding_table参数的值设置为1.0
self.off_value = Tensor(0.0, self.dtype)
# 将embedding_table参数的值设置为0.0
self.array_mul = P.MatMul()
# 使用矩阵乘法
self.reshape = P.Reshape()
# 将embedding_table参数变形为一维
self.get_shp = P.Shape()
def construct(self, ids):
extended_ids = self.expand(ids, -1)
out_shape = self.get_shp(ids) + (self.embedding_size,)
# 将extended_ids转换为矩阵
flat_ids = self.reshape_flat(extended_ids, self.shp_flat)
# 如果使用one_hot
if self.use_one_hot:
# 将flat_ids转换为one_hot矩阵
one_hot_ids = self.one_hot(flat_ids, self.vocab_size, self.on_value, self.off_value)
# 将one_hot矩阵乘以embedding_table
output_for_reshape = self.array_mul(one_hot_ids, self.embedding_table)
else:
# 将embedding_table中的flat_ids按照索引获取
output_for_reshape = self.gather(self.embedding_table, flat_ids, 0)
# 将reshape后的output_for_reshape转换为矩阵
output = self.reshape(output_for_reshape, out_shape)
return output
def extend_repr(self):
# 返回一个字符串，用于表示给定的参数
s = 'vocab_size={}, embedding_size={}, use_one_hot={}, embedding_table={}, dtype={}, padding_idx={}'.format(
self.vocab_size, self.embedding_size, self.use_one_hot, self.embedding_table, self.dtype, self.padding_idx)
return s
@constexpr
# 生成一个从start到end的范围
def _make_axis_range(start, end):
axis = tuple(range(start, end))
return axis
# 嵌入查找层
# 参数与Embedding相同
# 除了Embedding中的一部分功能，在EmbeddingLookup中还可以用于将图像数据的整数张量转换为嵌入向量，以便于图像分类
class EmbeddingLookup(Cell):
r"""
Returns a slice of the input tensor based on the specified indices.
@ -230,152 +292,243 @@ class EmbeddingLookup(Cell):
self.vocab_cache_size = validator.check_non_negative_int(vocab_cache_size, 'vocab_cache_size')
self.target = target
self.sparse = sparse
# 检查target参数是否为CPU或DEVICE
self.cache_enable = self.vocab_cache_size > 0
# 检查vocab_cache_size参数是否大于0
validator.check_string(target, ['CPU', 'DEVICE'], 'target', self.cls_name)
# 检查target参数是否为CPU或DEVICE
self.forward_unique = False
# 如果vocab_cache_size大于0，则target参数为CPU或DEVICE，否则target参数为DEVICE
validator.check_string(target, ['CPU', 'DEVICE'], 'target', self.cls_name)
if not sparse and target == 'CPU':
raise ValueError(f"For '{self.cls_name}', 'sparse' must be True when 'target' is \"CPU\", "
f"but got 'sparse': {sparse} and 'target': {target}")
raise ValueError(f"For '{self.cls_name}','sparse' must be True when 'target' is \"CPU\", "
f"but got'sparse': {sparse} and 'target': {target}")
if sparse:
# 如果sparse为True，则使用SparseGatherV2
self.gatherv2 = P.SparseGatherV2()
else:
# 否则使用Gather
self.gatherv2 = P.Gather()
# 将EmbeddingLookup添加primitive_target属性，值为CPU
self.embeddinglookup = P.EmbeddingLookup().add_prim_attr('primitive_target', 'CPU')
# 获取enable_ps的上下文
enable_ps = _get_ps_context("enable_ps")
if enable_ps:
# 如果enable_ps为True，则处理词汇表缓存
self._process_vocab_cache(slice_mode)
self.embedding_size = validator.check_positive_int(embedding_size, 'embedding_size', self.cls_name)
# 创建embedding_table参数，初始化为初始化参数初始值，名字为embedding_table
self.embedding_table = Parameter(initializer(param_init, [self.vocab_size, self.embedding_size]),
name='embedding_table')
# 获取并行模式
parallel_mode = _get_parallel_mode()
# 是否自动平行
is_auto_parallel = parallel_mode in (ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL)
# 获取gather_revert函数
self.gather_revert = P.Gather()
# 获取reshape函数
self.reshape_first = P.Reshape()
# 获取reshape函数
self.reshape = P.Reshape()
# 获取unique函数
self.unique = P.Unique()
# 获取shape函数
self.shape = P.Shape()
# 如果自动平行，则设置unique函数的参数cache_enable为True
if is_auto_parallel:
self.unique = P.Unique().shard(((1,),))
# 如果缓存启用，则设置_set_voacb_cache_enable_for_ps函数
if self.cache_enable and enable_ps:
self._set_voacb_cache_enable_for_ps(vocab_cache_size, embedding_size, vocab_size)
# 如果自动平行，则设置unique函数的参数cache_enable为True
if is_auto_parallel:
self.unique.add_prim_attr('cache_enable', True)
# 维度大小为2
indices_shape_size = 2
# 检查'manual_shapes'是否不为None，当"field_mode"为"field_slice"
if slice_mode == "field_slice" and is_auto_parallel:
if not manual_shapes:
# 抛出一个异常
raise ValueError(f"For '{self.cls_name}', the 'manual_shapes' should not be none "
f"when the 'slice_mode' is \"filed_slice\", but got {manual_shapes}.")
# 检查 'manual_shapes' 是否是一个元组且每个维度都是一个正整数
if not isinstance(manual_shapes, tuple):
# 抛出一个异常
raise TypeError(f"For '{self.cls_name}', the type of 'manual_shapes' must be tuple(int), "
f"but got {type(manual_shapes).__name__}!")
# 遍历 'manual_shapes' 并检查每个维度是否是一个正整数
for dim in manual_shapes:
validator.check_positive_int(dim, 'manual shape dim', self.cls_name)
self.gatherv2.add_prim_attr("manual_split", manual_shapes)
self.embeddinglookup.add_prim_attr("manual_split", manual_shapes)
# 将manual_shapes添加到gatherv2和embeddinglookup中
self.gatherv2.shard(((get_group_size(), 1), (1, get_group_size())))
self.embeddinglookup.shard(((get_group_size(), 1), (1, get_group_size())))
elif slice_mode == "table_row_slice" and is_auto_parallel:
# 获取全部批次
full_batch = _get_full_batch()
# 如果target为DEVICE，且不是全部批次，或者cache_enable为True，且sparse为True
if (target == 'DEVICE' and not full_batch) or (self.cache_enable and enable_ps and sparse):
# 设置indices_shape_size为1
indices_shape_size = 1
# 调用gather_revert.shard方法
self.gather_revert.shard(((1, 1), (get_group_size(),)))
# 设置forward_unique为True
self.forward_unique = True
# 设置indices_strategy为(1,)*indices_shape_size
indices_strategy = (1,)*indices_shape_size
# 调用gatherv2.shard方法
self.gatherv2.shard(((get_group_size(), 1), indices_strategy))
# 调用embeddinglookup.shard方法
self.embeddinglookup.shard(((get_group_size(), 1), indices_strategy))
elif slice_mode == "table_column_slice" and is_auto_parallel:
# 如果target为DEVICE，则获取indices_shape_size
if target == 'DEVICE':
indices_shape_size = 1
# 将shard操作设置为True
self.gather_revert.shard(((1, get_group_size()), (1,)))
# 将forward_unique设置为True
self.forward_unique = True
indices_strategy = (1,)*indices_shape_size
# 分配给gatherv2的维度
self.gatherv2.shard(((1, get_group_size()), indices_strategy))
# 分配给embeddinglookup的维度
self.embeddinglookup.shard(((1, get_group_size()), indices_strategy))
# 如果slice_mode为batch_slice，且is_auto_parallel为True
elif slice_mode == "batch_slice" and is_auto_parallel:
# 获取每个维度的分区大小
indices_strategy = [get_group_size()]
indices_strategy.extend([1]*(indices_shape_size - 1))
indices_strategy = tuple(indices_strategy)
# 将gatherv2分区大小设置为(1,1)，以及每个维度的维度大小
self.gatherv2.shard(((1, 1), indices_strategy))
# 将embeddinglookup分区大小设置为(1,1)，以及每个维度的维度大小
self.embeddinglookup.shard(((1, 1), indices_strategy))
else:
# 如果是自动并行，则支持模式为['field_slice', 'table_row_slice', 'table_column_slice', 'batch_slice']
if is_auto_parallel:
support_mode = ["field_slice", "table_row_slice", "table_column_slice", "batch_slice"]
raise ValueError("For '{}', the 'slice_mode' must be in {}, "
# 抛出异常
raise ValueError("For '{}', the'slice_mode' must be in {}, "
"but got \"{}\".".format(self.cls_name, support_mode, slice_mode))
if self.cache_enable and not enable_ps:
# 如果缓存模式为True，并且不支持ps模式
raise ValueError(f"For '{self.cls_name}', haven't supported cache enable for not ps mode.")
self.embedding_table.unique = self.forward_unique
# 设置最大范数
self.max_norm = max_norm
# 如果最大范数不为None，则检查最大范数是否为正数
if self.max_norm is not None:
self.max_norm = validator.check_positive_float(self.max_norm, 'max_norm', self.cls_name)
# 检查最大范数是否为正数
self.max_norm = validator.check_positive_float(self.max_norm,'max_norm', self.cls_name)
# 将最大范数转换为Tensor类型
self.max_norm = Tensor(self.max_norm, dtype=mstype.float32)
# 用于处理词汇缓存
def _process_vocab_cache(self, slice_mode):
"""PS embeddingLookup cache check and process."""
self.cache_enable = False
# 如果self.vocab_cache_size > 0
if self.vocab_cache_size > 0:
# 如果目标为CPU
if self.target == 'CPU':
# 则发出警告，设置logger的warning信息
logger.warning("The configuration of 'vocab_cache_size' is valid only in 'DEVICE' target, "
"current target is CPU, so it will be ignored.")
# 如果vocab_cache_size大于0，则设置enable_ps为False
enable_ps = _get_ps_context("enable_ps")
if not enable_ps:
# 如果enable_ps为False，则设置logger的warning信息
logger.warning("The configuration of 'vocab_cache_size' is valid only in parameter server training "
"mode, current mode is not parameter server trainning mode, so it will be ignored.")
return
enable_ps = _get_ps_context("enable_ps")
# 如果enable_ps为False，则跳过
if not enable_ps:
logger.warning("The configuration of 'vocab_cache_size' is valid only in parameter server training "
"mode, current mode is not parameter server trainning mode, so it will be ignored.")
return
# 获取并行模式
parallel_mode = _get_parallel_mode()
# 判断并行模式是否为自动平行
is_auto_parallel = parallel_mode in (ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL)
if is_auto_parallel:
# 获取组的大小
rank_size = get_group_size()
# 获取当前进程的编号
rank_id = get_rank()
# 获取全部的批次
full_batch = _get_full_batch()
# 如果组的大小大于1且不是全部批次且slice_mode为行切片
if rank_size > 1 and not (full_batch and slice_mode == "table_row_slice"):
# 抛出错误
raise ValueError(f"For '{self.cls_name}', the cache of parameter server parallel should only be "
f"used in \"full_batch\" and the value of \"full_batch\" should be True. "
f"Meanwhile, the value of 'slice_mode' should be \"table_row_slice\"."
f"But got full_batch: {full_batch} and 'slice_mode': \"{slice_mode}\".")
f"Meanwhile, the value of'slice_mode' should be \"table_row_slice\"."
f"But got full_batch: {full_batch} and'slice_mode': \"{slice_mode}\".")
# 设置编号
self.vocab_cache_size = self.vocab_cache_size * rank_size
# 设置rank_id
_set_rank_id(rank_id)
# 设置缓存开关
self.cache_enable = True
if _is_role_worker():
# 如果是工作节点，则将vocab_size设置为vocab_cache_size
self.vocab_size = self.vocab_cache_size
if context.get_context("enable_sparse") != self.sparse:
raise ValueError(f"For '{self.cls_name}', the value of parameter 'sparse' must be same for all "
# 如果context中的enable_sparse不等于self.sparse
if context.get_context("enable_sparse")!= self.sparse:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the value of parameter'sparse' must be same for all "
f"kernels and equal the value of 'enable_sparse' in context setting in "
f"parameter server cache mode, but got value of parameter 'sparse': {self.sparse}"
f"parameter server cache mode, but got value of parameter'sparse': {self.sparse}"
f" and the 'enable_sparse' in context setting: "
f"{context.get_context('enable_sparse')}.")
# 在PS下设置词汇缓存的相关配置
def _set_voacb_cache_enable_for_ps(self, vocab_cache_size, embedding_size, vocab_size):
"""PS embeddingLookup cache enable set."""
# 设置self.embedding_table.cache_enable为True
self.embedding_table.cache_enable = True
# 设置self.embedding_table.is_param_ps为True
self.embedding_table.is_param_ps = True
_set_cache_enable(True)
if self.sparse:
# 如果sparse为True，则设置forward_unique为True
self.forward_unique = True
# 如果_is_role_worker()返回True
if _is_role_worker():
# 插入hash表大小
_insert_hash_table_size(self.embedding_table.name, vocab_cache_size, embedding_size, vocab_size)
# 根据给定的索引张量构建输出张量
def construct(self, indices):
if self.target == "CPU":
# 如果target为CPU，则将indices转换为embedding_table的维度，并计算embedding_lookup
out = self.embeddinglookup(self.embedding_table, indices, 0)
else:
# 如果target为GPU，则将indices转换为embedding_table的维度，并计算embedding_lookup
if self.forward_unique:
# 如果forward_unique为True，则将indices转换为shape+embedding_size的维度，并计算unique
shp = self.shape(indices) + (self.embedding_size,)
indices_flatten = self.reshape_first(indices, (-1,))
unique_id, unique_idx = self.unique(indices_flatten)
# 将unique_id和unique_idx转换为embedding_table的维度，并计算weight_unique
weight_unique = self.gatherv2(self.embedding_table, unique_id, 0)
# 将weight_unique和unique_idx转换为shape的维度，并计算weight_flatten
weight_flatten = self.gather_revert(weight_unique, unique_idx, 0)
# 将weight_flatten和shp转换为out的维度
out = self.reshape(weight_flatten, shp)
else:
# 如果forward_unique为False，则将indices转换为embedding_table的维度，并计算out
out = self.gatherv2(self.embedding_table, indices, 0)
# 如果max_norm不为None，则将out的维度转换为axis
if self.max_norm is not None:
axis = _make_axis_range(F.rank(indices), F.rank(out))
# 将out的维度转换为clip_by_norm的维度
clip_by_norm = ClipByNorm(axis)
# 将out转换为clip_by_norm的维度，并计算out
out = clip_by_norm(out, self.max_norm)
# 返回out
return out
# 根据指定的索引和字段ID，返回输入Tensor的切片
class MultiFieldEmbeddingLookup(EmbeddingLookup):
r"""
Returns a slice of input tensor based on the specified indices and the field ids. This operation
@ -453,6 +606,17 @@ class MultiFieldEmbeddingLookup(EmbeddingLookup):
def __init__(self, vocab_size, embedding_size, field_size, param_init='normal', target='CPU',
slice_mode='batch_slice', feature_num_list=None, max_norm=None, sparse=True, operator='SUM'):
# 参数
# vocab_size(int): 词汇表大小
# embedding_size(int): 嵌入维度
# field_size(int): 字段数量。
# param_init(str): 参数初始化方法，默认为 'normal'。
# target(str): 计算目标，默认为 'CPU'。
# slice_mode(str): 切片模式，默认为 'batch_slice'。
# feature_num_list(list): 字段中特征的数量列表，用于指定每个字段的特征数量。
# max_norm(float): 最大归一化值，用于对输出张量进行归一化。
# sparse(bool): 是否为稀疏张量，默认为 True。
# operator(str): 聚合操作，默认为 'SUM'。
"""Initialize MultiFieldEmbeddingLookup."""
super(MultiFieldEmbeddingLookup, self).__init__(vocab_size, embedding_size, param_init, target,
slice_mode, feature_num_list, max_norm, sparse)
@ -475,111 +639,177 @@ class MultiFieldEmbeddingLookup(EmbeddingLookup):
self.max_no_equal = P.NotEqual()
validator.check_string(operator, ['SUM', 'MAX', 'MEAN'], 'operator', self.cls_name)
# 如果operator的值为SUM，则将merge_op设置为UnsortedSegmentSum()
if operator == MultiFieldEmbeddingLookup.OPERATOR_SUM:
self.merge_op = P.UnsortedSegmentSum()
# 如果operator的值为MAX，则将merge_op设置为UnsortedSegmentMax()
elif operator == MultiFieldEmbeddingLookup.OPERATOR_MAX:
self.merge_op = P.UnsortedSegmentMax()
# 否则，将merge_op设置为UnsortedSegmentSum()
else:
self.merge_op = P.UnsortedSegmentSum()
parallel_mode = _get_parallel_mode()
# 判断是否为自动并行
is_auto_parallel = parallel_mode in (ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL)
# 判断是否为行切片模式
if slice_mode in ["table_row_slice", "batch_slice"] and is_auto_parallel:
# 将shard操作添加到merge_op中
self.merge_op.shard(((get_group_size(), 1, 1), (get_group_size(), 1)))
# 将shard操作添加到expand中
self.expand.shard(((get_group_size(),),))
self.bias_add.shard(((1, 1), (1, 1)))
# 将bias_add按照分区大小分割
self.mul.shard(((get_group_size(), 1, 1), (get_group_size(), 1, 1)))
# 将mul按照分区大小分割
self.count_op.shard(((get_group_size(), 1), (get_group_size(), 1)))
# 将count_op按照分区大小分割
self.add.shard(((get_group_size(),), (get_group_size(),)))
# 将add按照分区大小分割
self.div_no_nan.shard(((get_group_size(), 1), (get_group_size(), 1)))
# 将div_no_nan按照分区大小分割
self.max_mask_mul.shard(((get_group_size(), 1), (get_group_size(), 1)))
# 将max_mask_mul按照分区大小分割
self.max_no_equal.shard(((1,), ()))
# 如果操作为MultiFieldEmbeddingLookup.OPERATOR_MAX
if operator == MultiFieldEmbeddingLookup.OPERATOR_MAX:
# 将shard操作添加到equal和inf_mask_mul中
self.equal.shard(((get_group_size(), 1, 1), ()))
self.inf_mask_mul.shard(((get_group_size(), 1, 1), ()))
# 将shard操作添加到merge_op中
self.merge_op.shard(((get_group_size(), 1), (get_group_size(),)))
# 将shard操作添加到count_op中
self.count_op.shard(((get_group_size(),), (get_group_size(),)))
# 将shard操作添加到inf_add中
self.inf_add.shard(((get_group_size(), 1, 1), (get_group_size(), 1, 1)))
elif slice_mode == "table_column_slice" and is_auto_parallel:
self.merge_op.shard(((1, 1, get_group_size()), (1, 1)))
# 将div_no_nan的shard参数设置为(1, get_group_size())
self.div_no_nan.shard(((1, get_group_size()), (1, 1)))
# 将bias_add的shard参数设置为(1, 1)
self.bias_add.shard(((1, 1), (1, 1)))
# 将mul的shard参数设置为(1, 1, get_group_size())
self.mul.shard(((1, 1, 1), (1, 1, get_group_size())))
# 将count_op的shard参数设置为(1, 1)
self.count_op.shard(((1, 1), (1, 1)))
# 将add的shard参数设置为(1,)
self.add.shard(((1,), (1,)))
# 将max_mask_mul的shard参数设置为(1, get_group_size())
self.max_mask_mul.shard(((1, get_group_size()), (1, 1)))
# 将expand的shard参数设置为(1,)
self.expand.shard(((1,),))
# 将max_no_equal的shard参数设置为(1,)
self.max_no_equal.shard(((1,), ()))
if operator == MultiFieldEmbeddingLookup.OPERATOR_MAX:
# 将equal的shard参数设置为(1, 1, 1)
self.equal.shard(((1, 1, 1), ()))
# 将inf_mask_mul的shard参数设置为(1, 1, 1)
self.inf_mask_mul.shard(((1, 1, 1), ()))
# 将merge_op的shard参数设置为(1, get_group_size())
self.merge_op.shard(((1, get_group_size()), (1,)))
# 将count_op的shard参数设置为(1,)
self.count_op.shard(((1,), (1,)))
# 将inf_add的shard参数设置为(1, 1, get_group_size())
self.inf_add.shard(((1, 1, get_group_size()), (1, 1, 1)))
else:
if is_auto_parallel:
# 抛出异常
raise ValueError("For '{}', the 'slice_mode' should be in ['table_row_slice', 'batch_slice' and \
'table_column_slice'], but got {}".format(self.cls_name, str(slice_mode)))
# Min value for fp32
self.negative_inf_value = -3.402823466E+38
# 构建输出张量
def construct(self, input_indices, input_values, field_ids):
# 检查输入input_indices是否为2d
_check_input_2d(F.shape(input_indices), "input_indices", self.cls_name)
# 检查输入input_values是否为2d
_check_input_2d(F.shape(input_values), "input_values", self.cls_name)
# 检查输入field_ids是否为2d
_check_input_2d(F.shape(field_ids), "field_ids", self.cls_name)
# 检查输入input_indices是否符合允许类型
_check_input_dtype(F.dtype(input_indices), "input_indices", [mstype.int32, mstype.int64], self.cls_name)
# 检查输入input_values是否符合允许类型
_check_input_dtype(F.dtype(input_values), "input_values", [mstype.float32], self.cls_name)
# 检查输入field_ids是否符合允许类型
_check_input_dtype(F.dtype(field_ids), "field_ids", [mstype.int32], self.cls_name)
# 定义一个函数，计算输入的field_ids和bias
batch_size = self.shape(input_indices)[0]
# 计算输入索引的数量
num_segments = batch_size * self.field_size
# 计算bias
bias = Range(0, num_segments, self.field_size)()
# 将bias变换为batch_size*num_segments的形状
bias = self.reshape(bias, (batch_size, -1))
# 将field_ids和bias合并
field_ids = self.bias_add(field_ids, bias)
# 如果目标是CPU
if self.target == "CPU":
# 调用embeddinglookup函数
out = self.embeddinglookup(self.embedding_table, input_indices, 0)
else:
# 如果目标是GPU，则调用forward_unique参数为True时，调用unique函数，获取unique_id和unique_idx
if self.forward_unique:
# 计算输入索引形状，并扩展为包含嵌入维数的形状
shp = self.shape(input_indices) + (self.embedding_size,)
# 将输入索引展开成一位数组
indices_flatten = self.reshape(input_indices, (-1,))
# 调用unique函数，获取unique_id和unique_idx
unique_id, unique_idx = self.unique(indices_flatten)
# 调用gatherv2函数，获取embedding_table中unique_id对应的embedding
weight_unique = self.gatherv2(self.embedding_table, unique_id, 0)
# 调用gather函数，获取embedding_table中unique_idx对应的embedding
weight_flatten = self.gather_revert(weight_unique, unique_idx, 0)
out = self.reshape(weight_flatten, shp)
else:
# 否则，调用gatherv2函数
out = self.gatherv2(self.embedding_table, input_indices, 0)
# 如果max_norm不为None，则调用ClipByNorm函数进行裁剪，设置axis和clip_by_norm参数
if self.max_norm is not None:
# 计算裁剪轴的范围
axis = _make_axis_range(F.rank(input_indices), F.rank(out))
# 创建用于输出裁剪的对象
clip_by_norm = ClipByNorm(axis)
# 对输出裁剪
out = clip_by_norm(out, self.max_norm)
# 将输入值reshape成一个(input_indices)(索引形状)[1], 1)的矩阵
weights = self.reshape(input_values, (batch_size, self.shape(input_indices)[1], 1))
# 将输入值与输出值相乘
embedding = self.mul(weights, out)
# 检查self.operator是否为'MAX'，如果为'MAX'
if self.operator == 'MAX':
# Fill the padding value to -inf, so the padded value will not influence the results
# 创建一个码，在权重为0的位置填充-inf
negative_inf_mask = self.cast(self.equal(weights, 0), mstype.float32)
#用上行创建的码与-inf相乘，得到一个新的码
inf_mask = self.inf_mask_mul(negative_inf_mask, self.negative_inf_value)
# 相加
embedding = self.inf_add(embedding, inf_mask)
# reshape为一个(-1, self.embedding_size)的矩阵
embedding = self.reshape(embedding, (-1, self.embedding_size))
# 将field_ids(字段)reshape
field_ids = self.reshape(field_ids, (-1,))
merged_vectors = self.merge_op(embedding, field_ids, num_segments)
# 如果operator为'MAX'
if self.operator == 'MAX':
# 使用count_op函数计算字段ID中每个值的计数
value_count = self.count_op(self.abs(self.reshape(input_values, (-1,))), field_ids, num_segments)
# 创建一个元素值全为0的数组
value_zeros = self.cast(self.max_no_equal(value_count, 0.0), mstype.float32)
# 用extend进行全零数组扩充，成为与出入向量形状相同的数组
count = self.expand(value_zeros, -1)
# 进行乘法运算
merged_vectors = self.max_mask_mul(merged_vectors, count)
# 如果operator为'MEAN'
if self.operator == 'MEAN':
# 使用count_op函数计算字段ID中每个值的计数
value_count = self.count_op(self.abs(input_values), field_ids, num_segments)
# 用extend进行扩充，成为与出入向量形状相同的数组
value_count = self.expand(value_count, -1)
# 使用div_no_nan进行运算
merged_vectors = self.div_no_nan(merged_vectors, value_count)
# 将函数reshape为(batch_size, self.field_size, -1)的矩阵
merged_vectors = self.reshape(merged_vectors, (batch_size, self.field_size, -1))
return merged_vectors

197
mindspore/python/mindspore/nn/layer/image.py
View File

@ -13,23 +13,36 @@
# limitations under the License.
# ============================================================================
"""image"""
# 导入numbers库，用于处理整数和浮点数
import numbers
# 导入numpy库，用于处理数值计算
import numpy as np
# 从mindspore.common.dtype模块中导入mstype类，用于表示数据类型
import mindspore.common.dtype as mstype
# 从mindspore.common.tensor模块中导入Tensor类，用于表示张量
from mindspore.common.tensor import Tensor
# 从mindspore.ops模块中导入operations模块，用于提供基本操作（如矩阵乘法、加法等）
from mindspore.ops import operations as P
# 从mindspore.ops.functional模块中导入functional类，用于提供一些基本功能，如条件判断、循环等
from mindspore.ops import functional as F
# 从mindspore.ops.primitive模块中导入constexpr类，用于定义一个常量
from mindspore.ops.primitive import constexpr
# 从mindspore._checkparam模块中导入Rel类（用于表示相对关系）和Validator类（用于验证参数的范围和类型）
from mindspore._checkparam import Rel, Validator as validator
# 从当前模块（..conv）中导入Conv2d类，用于创建卷积层
from .conv import Conv2d
# 从当前模块（..container）中导入CellList类，用于创建Cell列表
from .container import CellList
# 从当前模块（..pooling）中导入AvgPool2d类，用于创建平均池化层
from .pooling import AvgPool2d
# 从当前模块（..activation）中导入ReLU类，用于创建ReLU激活函数
from .activation import ReLU
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
__all__ = ['ImageGradients', 'SSIM', 'MSSSIM', 'PSNR', 'CentralCrop']
# 用于计算图像梯度
class ImageGradients(Cell):
r"""
Returns two tensors, the first is along the height dimension and the second is along the width dimension.
@ -68,137 +81,203 @@ class ImageGradients(Cell):
[[[[1, 0],
[1, 0]]]]))
"""
# 计算图像梯度
def __init__(self):
super(ImageGradients, self).__init__()
def construct(self, images):
# 检查如数张量形状是否为四维
check = _check_input_4d(F.shape(images), "images", self.cls_name)
# 用depend函数将检查结果添加到计算图中
images = F.depend(images, check)
# 获取images的形状
batch_size, depth, height, width = P.Shape()(images)
# 如果images的高度为1，则将dy设置为0
if height == 1:
dy = P.Fill()(P.DType()(images), (batch_size, depth, 1, width), 0)
# 否则，将dy的第二个维度设置为images的第一个维度的值
else:
dy = images[:, :, 1:, :] - images[:, :, :height - 1, :]
dy_last = P.Fill()(P.DType()(images), (batch_size, depth, 1, width), 0)
dy = P.Concat(2)((dy, dy_last))
# 如果images的宽度为1，则将dx设置为0
if width == 1:
dx = P.Fill()(P.DType()(images), (batch_size, depth, height, 1), 0)
# 否则，将dx的第三个维度设置为images的第一个维度的值
else:
dx = images[:, :, :, 1:] - images[:, :, :, :width - 1]
dx_last = P.Fill()(P.DType()(images), (batch_size, depth, height, 1), 0)
dx = P.Concat(3)((dx, dx_last))
# 返回dy和dx
return dy, dx
# 将图像类型转换为float32，并且将最大值归一化到[0, 1]范围
def _convert_img_dtype_to_float32(img, max_val):
"""convert img dtype to float32"""
# Usually max_val is 1.0 or 255, we will do the scaling if max_val > 1.
# We will scale img pixel value if max_val > 1. and just cast otherwise.
# 将图像类型转换为float32
ret = F.cast(img, mstype.float32)
# 计算最大值
max_val = F.scalar_cast(max_val, mstype.float32)
# 如果大于1则
if max_val > 1.:
# 计算比例
scale = 1. / max_val
ret = ret * scale
return ret
@constexpr
# 获取数据中最大值
def _get_dtype_max(dtype):
"""get max of the dtype"""
# 将数据类型转换为numpy数据类型
np_type = mstype.dtype_to_nptype(dtype)
# 检查数据类型是否为整数类型，如果是整数类型
if issubclass(np_type, numbers.Integral):
# 获取最大值
dtype_max = np.float64(np.iinfo(np_type).max)
# 如果不是整型
else:
# 把最大值赋为1.0
dtype_max = 1.0
return dtype_max
@constexpr
# 检查输入的图像是否为四维
def _check_input_4d(input_shape, param_name, func_name):
# 如果输入的图像不是四维
if len(input_shape) != 4:
# 抛出异常
raise ValueError(f"For '{func_name}', the dimension of '{param_name}' should be 4d, "
f"but got {len(input_shape)}.")
return True
@constexpr
# 检查输入的图像尺寸、名称，滤波器尺寸和函数名称
def _check_input_filter_size(input_shape, param_name, filter_size, func_name):
# 检查输入的图像是否为四维
_check_input_4d(input_shape, param_name, func_name)
# 检查参数shape，size
validator.check(param_name + " shape[2]", input_shape[2], "filter_size", filter_size, Rel.GE, func_name)
# 检查参数shape，size
validator.check(param_name + " shape[3]", input_shape[3], "filter_size", filter_size, Rel.GE, func_name)
@constexpr
# 检查输入数据类型
def _check_input_dtype(input_dtype, param_name, allow_dtypes, cls_name):
validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name)
# 用于实现两个特征图片的二维卷积操作
# 输入输出通道、卷积核尺寸、权重、填充大小和填充模式
def _conv2d(in_channels, out_channels, kernel_size, weight, stride=1, padding=0):
# 返回处理完的图像
return Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride,
weight_init=weight, padding=padding, pad_mode="valid")
# 创建用于输入参数的高斯窗口
def _create_window(size, sigma):
x_data, y_data = np.mgrid[-size // 2 + 1:size // 2 + 1, -size // 2 + 1:size // 2 + 1]
# 将x_data和y_data拉直，并且添加一个维度
x_data = np.expand_dims(x_data, axis=-1).astype(np.float32)
x_data = np.expand_dims(x_data, axis=-1) ** 2
y_data = np.expand_dims(y_data, axis=-1).astype(np.float32)
y_data = np.expand_dims(y_data, axis=-1) ** 2
# 计算sigma的值
sigma = 2 * sigma ** 2
# 计算g的值
g = np.exp(-(x_data + y_data) / sigma)
# 返回g的值，并且排序，以便于计算
return np.transpose(g / np.sum(g), (2, 3, 0, 1))
# 将输入图像按给定分块尺寸进行分块
def _split_img(x):
# 获取输入张量的通道数
_, c, _, _ = F.shape(x)
# 创建P.Split操作符
img_split = P.Split(1, c)
# 输入x
output = img_split(x)
# 返回output和c
return output, c
# 用卷积神经网络计算输入的两张图片img1和img2之间的损失
def _compute_per_channel_loss(c1, c2, img1, img2, conv):
"""computes ssim index between img1 and img2 per single channel"""
dot_img = img1 * img2
# 计算图像1和图像2的乘积
mu1 = conv(img1)
# 计算图像1的均值
mu2 = conv(img2)
# 计算图像1和图像2的均值的平方
mu1_sq = mu1 * mu1
# 计算图像1和图像2的均值的平方
mu2_sq = mu2 * mu2
# 计算图像1和图像2的均值的平方和
mu1_mu2 = mu1 * mu2
# 计算图像1和图像2的方差
sigma1_tmp = conv(img1 * img1)
# 计算图像1和图像2的方差的平方
sigma1_sq = sigma1_tmp - mu1_sq
# 计算图像2和图像1的方差
sigma2_tmp = conv(img2 * img2)
# 计算图像2和图像1的方差的平方
sigma2_sq = sigma2_tmp - mu2_sq
# 计算图像1和图像2的点乘
sigma12_tmp = conv(dot_img)
# 计算图像1和图像2的点乘的平方
sigma12 = sigma12_tmp - mu1_mu2
# 计算图像1和图像2的平方根
a = (2 * mu1_mu2 + c1)
# 计算图像1和图像2的平方根的平方
b = (mu1_sq + mu2_sq + c1)
# 计算图像1和图像2的二次方差
v1 = 2 * sigma12 + c2
# 计算图像1和图像2的二次方差的平方
v2 = sigma1_sq + sigma2_sq + c2
# 计算图像1和图像2的平方根的二次方差
ssim = (a * v1) / (b * v2)
# 计算图像1和图像2的二次方差的平方根
cs = v1 / v2
# 返回ssim和cs
return ssim, cs
# 计算两个图像张量的损失
def _compute_multi_channel_loss(c1, c2, img1, img2, conv, concat, mean):
"""computes ssim index between img1 and img2 per color channel"""
# 初始化img1
split_img1, c = _split_img(img1)
# 初始化img2
split_img2, _ = _split_img(img2)
# 初始化每个通道的损失
multi_ssim = ()
multi_cs = ()
# 遍历每个通道
for i in range(c):
# 计算每个通道的损失
ssim_per_channel, cs_per_channel = _compute_per_channel_loss(c1, c2, split_img1[i], split_img2[i], conv)
# 将每个通道的损失添加到multi_ssim中
multi_ssim += (ssim_per_channel,)
# 将每个通道的损失添加到multi_cs中
multi_cs += (cs_per_channel,)
# 将multi_ssim和multi_cs合并
multi_ssim = concat(multi_ssim)
multi_cs = concat(multi_cs)
# 计算平均损失
ssim = mean(multi_ssim, (2, 3))
cs = mean(multi_cs, (2, 3))
# 返回平均损失
return ssim, cs
# 用于检测图像之间的结构相似性指数
# 通过比较图像的局部特征来衡量图像的相似程度，输出值通常在1到-1之间，越接近1越相似
class SSIM(Cell):
r"""
Returns SSIM index between two images.
@ -256,46 +335,64 @@ class SSIM(Cell):
>>> print(output)
[1.]
"""
# 参数
# max_val：最大值，用于归一化SSIM值。默认值为1.0
# filter_size：滤波器大小，用于计算SSIM的卷积核大小。默认值为11
# filter_sigma：滤波器sigma值，用于计算SSIM的卷积核sigma大小。默认值为1.5
# k1：权重参数k1，用于计算SSIM的公式。默认值为0.01
# k2：权重参数k2，用于计算SSIM的公式。默认值为0.03
def __init__(self, max_val=1.0, filter_size=11, filter_sigma=1.5, k1=0.01, k2=0.03):
super(SSIM, self).__init__()
# 检查max_val类型是否为int或float
validator.check_value_type('max_val', max_val, [int, float], self.cls_name)
# 检查max_val是否大于0.0
validator.check_number('max_val', max_val, 0.0, Rel.GT, self.cls_name)
self.max_val = max_val
# 检查filter_size是否大于等于1
self.filter_size = validator.check_int(filter_size, 1, Rel.GE, 'filter_size', self.cls_name)
# 检查filter_sigma是否大于等于0
self.filter_sigma = validator.check_positive_float(filter_sigma, 'filter_sigma', self.cls_name)
# 赋值k1、k2
self.k1 = validator.check_value_type('k1', k1, [float], self.cls_name)
self.k2 = validator.check_value_type('k2', k2, [float], self.cls_name)
# 创建计算滤波器窗口
window = _create_window(filter_size, filter_sigma)
# 创建一个二维卷积层
self.conv = _conv2d(1, 1, filter_size, Tensor(window))
self.conv.weight.requires_grad = False
self.reduce_mean = P.ReduceMean()
self.concat = P.Concat(axis=1)
# 将img1、img2这两个输入的图像张量传入SSIM进行比较
def construct(self, img1, img2):
# 检查img1的类型是否为float32或float16
_check_input_dtype(F.dtype(img1), "img1", [mstype.float32, mstype.float16], self.cls_name)
# 检查filter_size是否等于图像尺寸
_check_input_filter_size(F.shape(img1), "img1", self.filter_size, self.cls_name)
# 用SameTypeShape保证img1和img2形状相同
P.SameTypeShape()(img1, img2)
# 获取张量最大值
dtype_max_val = _get_dtype_max(F.dtype(img1))
max_val = F.scalar_cast(self.max_val, F.dtype(img1))
max_val = _convert_img_dtype_to_float32(max_val, dtype_max_val)
# 把img1和img2转化为float32数据类型
img1 = _convert_img_dtype_to_float32(img1, dtype_max_val)
img2 = _convert_img_dtype_to_float32(img2, dtype_max_val)
c1 = (self.k1 * max_val) ** 2
c2 = (self.k2 * max_val) ** 2
# 计算img1与img2之间的ssim损失
ssim_ave_channel, _ = _compute_multi_channel_loss(c1, c2, img1, img2, self.conv, self.concat, self.reduce_mean)
loss = self.reduce_mean(ssim_ave_channel, -1)
# 返回损失值
return loss
# 向下取样img1、img2
def _downsample(img1, img2, op):
a = op(img1)
b = op(img2)
return a, b
# MSSIM相比于SSIM对于不同分辨率都可以保持性能稳定性
class MSSSIM(Cell):
r"""
Returns MS-SSIM index between two images.
@ -351,43 +448,78 @@ class MSSSIM(Cell):
>>> print(output)
[1.]
"""
# 参数
# max_val：最大值，用于归一化SSIM值。默认值为1.0
# filter_size：滤波器大小，用于计算SSIM的卷积核大小。默认值为11
# filter_sigma：滤波器sigma值，用于计算SSIM的卷积核sigma大小。默认值为1.5
# k1：权重参数k1，用于计算SSIM的公式。默认值为0.01
# k2：权重参数k2，用于计算SSIM的公式。默认值为0.03
def __init__(self, max_val=1.0, power_factors=(0.0448, 0.2856, 0.3001, 0.2363, 0.1333), filter_size=11,
filter_sigma=1.5, k1=0.01, k2=0.03):
super(MSSSIM, self).__init__()
# 检查max_val类型是否为int或float
validator.check_value_type('max_val', max_val, [int, float], self.cls_name)
# 检查max_val是否大于0.0
validator.check_number('max_val', max_val, 0.0, Rel.GT, self.cls_name)
self.max_val = max_val
# 检查power_factors的类型，是否为tuple或者list，以及最大值是否大于0.0，最后赋值给power_factors
validator.check_value_type('power_factors', power_factors, [tuple, list], self.cls_name)
# 检查filter_size是否大于等于1
self.filter_size = validator.check_int(filter_size, 1, Rel.GE, 'filter_size', self.cls_name)
# 检查filter_sigma是否大于等于0
self.filter_sigma = validator.check_positive_float(filter_sigma, 'filter_sigma', self.cls_name)
# 检查参k1是否为float型
self.k1 = validator.check_value_type('k1', k1, [float], self.cls_name)
# 检查参k2是否为float型
self.k2 = validator.check_value_type('k2', k2, [float], self.cls_name)
# 创建计算滤波器窗口
window = _create_window(filter_size, filter_sigma)
# 计算level
self.level = len(power_factors)
self.conv = []
for i in range(self.level):
# 将卷积层添加到self.conv中
self.conv.append(_conv2d(1, 1, filter_size, Tensor(window)))
# 将卷积层的权重设置为不可求导
self.conv[i].weight.requires_grad = False
# 将self.conv添加到self.multi_convs_list中
self.multi_convs_list = CellList(self.conv)
# 将self.weight_tensor设置为power_factors的浮点型值
self.weight_tensor = Tensor(power_factors, mstype.float32)
# 创建平均池化层
self.avg_pool = AvgPool2d(kernel_size=2, stride=2, pad_mode='valid')
# 创建ReLU激活层
self.relu = ReLU()
# 创建平均池化层
self.reduce_mean = P.ReduceMean()
# 创建乘法层
self.prod = P.ReduceProd()
# 创建Pow层
self.pow = P.Pow()
# 创建stack层
self.stack = P.Stack(axis=-1)
# 创建concat层
self.concat = P.Concat(axis=1)
def construct(self, img1, img2):
# 检查img1是否为4d
_check_input_4d(F.shape(img1), "img1", self.cls_name)
# 检查img2是否为4d
_check_input_4d(F.shape(img2), "img2", self.cls_name)
valid_type = [mstype.float64, mstype.float32, mstype.float16, mstype.uint8]
# 检查输入的数据类型
_check_input_dtype(F.dtype(img1), 'img1', valid_type, self.cls_name)
# 对img1和img2进行同类型和形状的操作
P.SameTypeShape()(img1, img2)
# 获取img1的最大值
dtype_max_val = _get_dtype_max(F.dtype(img1))
# 设置最大值
max_val = F.scalar_cast(self.max_val, F.dtype(img1))
# 将最大值转换为float32类型
max_val = _convert_img_dtype_to_float32(max_val, dtype_max_val)
# 将img1转换为float32类型
img1 = _convert_img_dtype_to_float32(img1, dtype_max_val)
# 将img2转换为float32类型
img2 = _convert_img_dtype_to_float32(img2, dtype_max_val)
c1 = (self.k1 * max_val) ** 2
@ -395,7 +527,7 @@ class MSSSIM(Cell):
sim = ()
mcs = ()
# 循环遍历self.level的值
for i in range(self.level):
sim, cs = _compute_multi_channel_loss(c1, c2, img1, img2,
self.multi_convs_list[i], self.concat, self.reduce_mean)
@ -403,14 +535,18 @@ class MSSSIM(Cell):
img1, img2 = _downsample(img1, img2, self.avg_pool)
mcs = mcs[0:-1:1]
# 将mcs中的元素按照1:1:1的方式进行堆叠
mcs_and_ssim = self.stack(mcs + (self.relu(sim),))
# 将mcs_and_ssim中的元素乘以weight_tensor的结果
mcs_and_ssim = self.pow(mcs_and_ssim, self.weight_tensor)
# 将mcs_and_ssim中的元素乘以-1的结果
ms_ssim = self.prod(mcs_and_ssim, -1)
# 将ms_ssim中的元素乘以-1的结果并且求和
loss = self.reduce_mean(ms_ssim, -1)
# 返回loss值
return loss
# 用于计算图像修复和增强模型中的PSNR值
class PSNR(Cell):
r"""
Returns Peak Signal-to-Noise Ratio of two image batches.
@ -451,37 +587,48 @@ class PSNR(Cell):
>>> print(output)
[-6.0206]
"""
# 初始化最大值
def __init__(self, max_val=1.0):
super(PSNR, self).__init__()
# 检查max_val的类型是否为int或float
validator.check_value_type('max_val', max_val, [int, float], self.cls_name)
# 检查max_val的值是否大于0.0
validator.check_number('max_val', max_val, 0.0, Rel.GT, self.cls_name)
self.max_val = max_val
def construct(self, img1, img2):
# 检查img1是否为4d
_check_input_4d(F.shape(img1), "img1", self.cls_name)
# 检查img2是否为4d
_check_input_4d(F.shape(img2), "img2", self.cls_name)
P.SameTypeShape()(img1, img2)
dtype_max_val = _get_dtype_max(F.dtype(img1))
max_val = F.scalar_cast(self.max_val, F.dtype(img1))
# 将max_val转换为float32类型
max_val = _convert_img_dtype_to_float32(max_val, dtype_max_val)
# 将img1和img2转换为float32类型
img1 = _convert_img_dtype_to_float32(img1, dtype_max_val)
img2 = _convert_img_dtype_to_float32(img2, dtype_max_val)
# 计算MSE均方误差
mse = P.ReduceMean()(F.square(img1 - img2), (-3, -2, -1))
# 使用10 * P.Log函数计算psnr
psnr = 10 * P.Log()(F.square(max_val) / mse) / F.scalar_log(10.0)
# 返回psnr值
return psnr
@constexpr
def _raise_dims_rank_error(input_shape, param_name, func_name):
"""raise error if input is not 3d or 4d"""
# 抛出异常
raise ValueError(f"{func_name} {param_name} should be 3d or 4d, but got shape {input_shape}")
@constexpr
# 用于计算剪裁图像时的高度、宽度的起始位置和大小
def _get_bbox(rank, shape, central_fraction):
"""get bbox start and size for slice"""
# 如果为rank为3
if rank == 3:
c, h, w = shape
else:
@ -491,17 +638,17 @@ def _get_bbox(rank, shape, central_fraction):
bbox_w_start = int((float(w) - np.float32(w * central_fraction)) / 2)
bbox_h_size = h - bbox_h_start * 2
bbox_w_size = w - bbox_w_start * 2
# 如果rank为3
if rank == 3:
bbox_begin = (0, bbox_h_start, bbox_w_start)
bbox_size = (c, bbox_h_size, bbox_w_size)
else:
bbox_begin = (0, 0, bbox_h_start, bbox_w_start)
bbox_size = (n, c, bbox_h_size, bbox_w_size)
# 返回初始位置和大小
return bbox_begin, bbox_size
# 用于从图像中剪裁处中心部分
class CentralCrop(Cell):
"""
Crops the central region of the images with the central_fraction.
@ -532,20 +679,26 @@ class CentralCrop(Cell):
def __init__(self, central_fraction):
super(CentralCrop, self).__init__()
# 检查central_fraction是否为float型
validator.check_value_type("central_fraction", central_fraction, [float], self.cls_name)
# 检查central_fraction是否在0.0到1.0之间
validator.check_float_range(central_fraction, 0.0, 1.0, Rel.INC_RIGHT, 'central_fraction', self.cls_name)
self.central_fraction = central_fraction
self.slice = P.Slice()
def construct(self, image):
# 获取图像的形状(高度、宽度、通道数)
image_shape = F.shape(image)
rank = len(image_shape)
# 检查图像的形状是否为3维或4维
if not rank in (3, 4):
return _raise_dims_rank_error(image_shape, "image", self.cls_name)
if self.central_fraction == 1.0:
return image
# 获取图像的边界框
bbox_begin, bbox_size = _get_bbox(rank, image_shape, self.central_fraction)
# 将图像切片
image = self.slice(image, bbox_begin, bbox_size)
return image

293
mindspore/python/mindspore/nn/layer/math.py
View File

@ -13,14 +13,23 @@
# limitations under the License.
# ============================================================================
"""math"""
# 导入numpy库，用于处理数值计算
import numpy as np
# 从mindspore.ops模块中导入operations模块，用于提供基本操作（如矩阵乘法、加法等）
from mindspore.ops import operations as P
# 从mindspore.common.tensor模块中导入Tensor类，用于表示张量
from mindspore.common.tensor import Tensor
# 从mindspore.common._decorator模块中导入deprecated装饰器，用于标记已过时的功能
from mindspore.common._decorator import deprecated
# 从mindspore.ops.primitive模块中导入constexpr类，用于定义一个常量
from mindspore.ops.primitive import constexpr
# 从mindspore.ops.functional模块中导入functional类，用于提供一些基本功能，如条件判断、循环等
from mindspore.ops import functional as F
# 从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
# 从mindspore.common模块中导入dtype类，用于表示数据类型
from ...common import dtype as mstype
# 从mindspore._checkparam模块中导入Validator类，用于验证参数的范围和类型
from ..._checkparam import Validator as validator
__all__ = ['ReduceLogSumExp',
@ -34,7 +43,8 @@ __all__ = ['ReduceLogSumExp',
'MatInverse',
'MatDet',
]
# 该系数用于Lanczos重采样算法
# 用于将图像调整为不同分辨率，用于计算Lanczos核
_BASE_LANCZOS_COEFF = 0.99999999999980993227684700473478
_LANCZOS_COEFFICIENTS = [676.520368121885098567009190444019,
-1259.13921672240287047156078755283,
@ -47,10 +57,12 @@ _LANCZOS_COEFFICIENTS = [676.520368121885098567009190444019,
@constexpr
# 检查输入数据的类型是否为允许输入类型
def _check_input_dtype(param_name, input_dtype, allow_dtypes, cls_name):
validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name)
# 用于输入进行对数和归一化指数的降维的神经网络层
# 将输入张量的每个元素求对数和归一化指数
class ReduceLogSumExp(Cell):
r"""
Reduces a dimension of a tensor by calculating exponential for all elements in the dimension,
@ -99,20 +111,31 @@ class ReduceLogSumExp(Cell):
def __init__(self, axis, keep_dims=False):
"""Initialize ReduceLogSumExp."""
super(ReduceLogSumExp, self).__init__()
# 检查'axis'的类型是否为int, list, tuple
validator.check_value_type('axis', axis, [int, list, tuple], self.cls_name)
# 检查'keep_dims'的类型是否为bool
validator.check_value_type('keep_dims', keep_dims, [bool], self.cls_name)
# 赋值
self.axis = axis
# 创建Exp对象
self.exp = P.Exp()
# 创建ReduceSum对象
self.sum = P.ReduceSum(keep_dims)
# 创建Log对象
self.log = P.Log()
# 构造
def construct(self, x):
# 计算exp（x）
exp = self.exp(x)
# 计算sum(exp, self.axis)
sumexp = self.sum(exp, self.axis)
# 计算log(sumexp)
logsumexp = self.log(sumexp)
# 返回logsumexp
return logsumexp
# 用于创建一个表示单元格范围的类
# 主要作用是储存单元格范围的起始和结束单元格地址
class Range(Cell):
r"""
Creates a sequence of numbers in range [start, limit) with step size delta.
@ -143,23 +166,30 @@ class Range(Cell):
>>> print(output)
[1 3 5 7]
"""
# 初始化Range函数
def __init__(self, start, limit=None, delta=1):
"""Initialize Range."""
super(Range, self).__init__()
# 如果delta为0
if delta == 0:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'delta' can not be zero.")
# 创建一个单元格范围的数组
data = np.arange(start, limit, delta)
# 如果data的类型为float
if data.dtype == np.float:
# 设置ms_dtype为float32
self.ms_dtype = mstype.float32
else:
# 设置ms_dtype为int32
self.ms_dtype = mstype.int32
self.result_tensor = Tensor(data, dtype=self.ms_dtype)
def construct(self):
return self.result_tensor
# 用于计算自然对数的gamma函数值
# 通常用于计算概率分布的归一化因子
class LGamma(Cell):
r"""
Calculates LGamma using Lanczos' approximation referring to "A Precision Approximation of the Gamma Function".
@ -216,58 +246,91 @@ class LGamma(Cell):
super(LGamma, self).__init__()
# const numbers
self.k_lanczos_gamma = 7
# k_lanczos_gamma的值
self.k_base_lanczos_coeff = _BASE_LANCZOS_COEFF
# k_base_lanczos_coeff的值
self.k_lanczos_coefficients = _LANCZOS_COEFFICIENTS
# k_lanczos_coefficients的值
self.one_half = 0.5
# one_half的值
self.one = 1
# one的值
self.two = 2
# two的值
self.inf = np.inf
# inf的值
self.pi = np.pi
# pi的值
self.log_2 = np.log(self.two)
# log_2的值
self.log_pi = np.log(np.pi)
# log_pi的值
self.log_sqrt_two_pi = (self.log_2 + self.log_pi) / self.two
# log_sqrt_two_pi的值
self.lanczos_gamma_plus_one_half = self.k_lanczos_gamma + 0.5
# lanczos_gamma_plus_one_half的值
self.log_lanczos_gamma_plus_one_half = np.log(self.lanczos_gamma_plus_one_half)
# operations
self.log = P.Log()
# 计算log函数
self.log1p = P.Log1p()
# 计算log1p函数
self.abs = P.Abs()
# 计算绝对值函数
self.shape = P.Shape()
# 计算形状函数
self.dtype = P.DType()
# 计算数据类型函数
self.fill = P.Fill()
# 计算填充函数
self.floor = P.Floor()
# 计算小于函数
self.equal = P.Equal()
# 计算等于函数
self.greater = P.Greater()
# 计算大于函数
self.less = P.Less()
# 计算小于等于函数
self.lessequal = P.LessEqual()
# 计算小于等于等于函数
self.select = P.Select()
# 计算选择函数
self.sin = P.Sin()
# 计算sin函数
self.isfinite = P.IsFinite()
def construct(self, x):
input_dtype = self.dtype(x)
_check_input_dtype("x", input_dtype, [mstype.float16, mstype.float32], self.cls_name)
# 检查"x"类型是否为float16或float16
_check_input_dtype("x", input_dtype, [mstype.float16, mstype.float16], self.cls_name)
infinity = self.fill(input_dtype, self.shape(x), self.inf)
need_to_reflect = self.less(x, 0.5)
neg_input = -x
# 如果x小于0.5，则将-x放入z中，否则将x-1放入z中
z = self.select(need_to_reflect, neg_input, x - 1)
@constexpr
def _calculate_reflected_x(z, k_base_lanczos_coeff, k_lanczos_coefficients):
reflex_x = k_base_lanczos_coeff
for i in range(8):
# 计算k_lanczos_coefficients[i] / (z + i + 1)
product_ = k_lanczos_coefficients[i] / (z + i + 1)
# 将product_加到reflex_x上
reflex_x = product_ + reflex_x
return reflex_x
# 计算reflected_x
reflex_x = _calculate_reflected_x(z, self.k_base_lanczos_coeff, self.k_lanczos_coefficients)
# 计算t
t = z + self.lanczos_gamma_plus_one_half
log_t = self.log1p(z / self.lanczos_gamma_plus_one_half) + self.log_lanczos_gamma_plus_one_half
# 计算log_y
log_y = self.log(reflex_x) + (z + self.one_half - t / log_t) * log_t + self.log_sqrt_two_pi
# 计算abs_input
abs_input = self.abs(x)
abs_frac_input = abs_input - self.floor(abs_input)
x = self.select(self.lessequal(x, 0.0), self.select(self.equal(abs_frac_input, 0.0), infinity, x), x)
@ -275,10 +338,12 @@ class LGamma(Cell):
1 - abs_frac_input, abs_frac_input)
reflection_denom = self.log(self.sin(self.pi * reduced_frac_input))
# 计算reflection
reflection = self.select(self.isfinite(reflection_denom),
-reflection_denom - log_y + self.log_pi, # pylint: disable=invalid-unary-operand-type
-reflection_denom) # pylint: disable=invalid-unary-operand-type
# 计算result
result = self.select(need_to_reflect, reflection, log_y)
return self.select(self.isfinite(x), result, infinity)
@ -337,35 +402,54 @@ class DiGamma(Cell):
# operations
self.log1p = P.Log1p()
# 定义绝对值函数
self.abs = P.Abs()
# 定义获取shape函数
self.shape = P.Shape()
# 定义获取数据类型函数
self.dtype = P.DType()
# 定义填充函数
self.fill = P.Fill()
# 定义取整函数
self.floor = P.Floor()
# 定义比较函数
self.equal = P.Equal()
self.less = P.Less()
# 定义选择函数
self.select = P.Select()
# 定义正弦函数
self.sin = P.Sin()
# 定义余弦函数
self.cos = P.Cos()
# 定义逻辑与函数
self.logicaland = P.LogicalAnd()
def construct(self, x):
input_dtype = self.dtype(x)
# 检查"x"的类型是否为float16或float32
_check_input_dtype("x", input_dtype, [mstype.float16, mstype.float32], self.cls_name)
# 将x的值小于0.5的值设置为True，其余的设置为False
need_to_reflect = self.less(x, 0.5)
# 将x的值取反
neg_input = -x
# 根据need_to_reflect的值，将neg_input的值设置为x-1
z = self.select(need_to_reflect, neg_input, x - 1)
@constexpr
# 用于计算一个给定表达式的分子和分母
def _calculate_num_denom(z, k_base_lanczos_coeff, k_lanczos_coefficients):
num = 0
denom = k_base_lanczos_coeff
# 循环遍历
for i in range(8):
# 在每个迭代中，更新分子 by 减去k_lanczos_coefficients[i]除以(z + i + 1)乘以(z + i + 1)的值
num = num - k_lanczos_coefficients[i] / ((z + i + 1) * (z + i + 1))
# 在每个迭代中，更新分母 by 加上k_lanczos_coefficients[i]除以(z + i + 1)的值
denom = denom + k_lanczos_coefficients[i] / (z + i + 1)
# 返回分子分母
return num, denom
num, denom = _calculate_num_denom(z, self.k_base_lanczos_coeff, self.k_lanczos_coefficients)
# 计算t、y的值
t = z + self.lanczos_gamma_plus_one_half
log_t = self.log1p(z / self.lanczos_gamma_plus_one_half) + self.log_lanczos_gamma_plus_one_half
@ -382,13 +466,23 @@ class DiGamma(Cell):
eps_fp32 = Tensor(np.finfo(np.float32).eps, mstype.float32)
# 用于在满足给定条件cond时重复执行body中的代码
def _while_helper_func(cond, body, vals):
# 定义一个名为while_loop的函数，参数为cond和body
while cond(vals).any():
# 当cond函数返回值为True时，执行body函数
vals = body(vals)
# 返回vals
return vals
# 用于计算给定参数ax和x的逆元Gamma函数的series形式
# 参数
# logicaland：一个布尔函数，用于对两个布尔值进行逻辑与操作
# greater：一个比较函数，用于比较两个浮点数，并返回一个布尔值
# fill：一个填充函数，用于将一个数值填充到一个给定形状和数据类型的数组中
# shape：一个属性函数，用于获取一个数组的形状
# dtype：一个属性函数，用于获取一个数组的数据类型
# select：一个选择函数，用于根据条件选择一个值
def _igamma_series(ax, x, a, enabled):
"""Helper function for computing Igamma using a power series."""
@ -407,19 +501,32 @@ def _igamma_series(ax, x, a, enabled):
return enabled
def body(vals):
# 设置enabled为vals[0]
enabled = vals[0]
# 设置r为vals[1]
r = vals[1]
# 设置c为vals[2]
c = vals[2]
# 设置ans为vals[3]
ans = vals[3]
# 设置x为vals[4]
x = vals[4]
# 设置dc_da为vals[5]
dc_da = vals[5]
# 设置dans_da为vals[6]
dans_da = vals[6]
# 计算r
r = r + 1
# 计算dc_da
dc_da = dc_da * (x / r) + (-1 * c * x) / (r * r)
# 计算dans_da
dans_da = dans_da + dc_da
# 计算c
c = c * (x / r)
# 计算ans
ans = ans + c
# 计算条件
conditional = logicaland(enabled, greater(c / ans, epsilon))
return (conditional, select(enabled, r, vals[1]),
@ -427,15 +534,18 @@ def _igamma_series(ax, x, a, enabled):
select(enabled, x, vals[4]), select(enabled, dc_da, vals[5]),
select(enabled, dans_da, vals[6]))
# 初始化变量
ones = fill(dtype(a), shape(a), 1)
zeros = fill(dtype(a), shape(a), 0)
vals = (enabled, a, ones, ones, x, zeros, zeros)
# 调用_while_helper_func函数
vals = _while_helper_func(cond, body, vals)
# 计算ans
ans = vals[3]
# 返回结果
return (ans * ax) / a
def _igammac_continued_fraction(ax, x, a, enabled):
"""Helper function for computing Igammac using a continued fraction."""
@ -453,44 +563,74 @@ def _igammac_continued_fraction(ax, x, a, enabled):
epsilon = eps_fp32
def cond(vals):
# 返回一个布尔值，该布尔值表示c是否小于2000，且enabled为真
enabled = vals[0]
c = vals[5]
return logicaland(less(c, 2000), enabled)
def body(vals):
# 设置enabled为vals[0]
enabled = vals[0]
# 设置ans为vals[1]
ans = vals[1]
# 设置t为vals[2]
t = vals[2]
# 设置y为vals[3]
y = vals[3]
# 设置z为vals[4]
z = vals[4]
# 设置c为vals[5]
c = vals[5]
# 设置pkm1为vals[6]
pkm1 = vals[6]
# 设置qkm1为vals[7]
qkm1 = vals[7]
# 设置pkm2为vals[8]
pkm2 = vals[8]
# 设置qkm2为vals[9]
qkm2 = vals[9]
# 设置dpkm2_da为vals[10]
dpkm2_da = vals[10]
# 设置dqkm2_da为vals[11]
dqkm2_da = vals[11]
# 设置dpkm1_da为vals[12]
dpkm1_da = vals[12]
# 设置dqkm1_da为vals[13]
dqkm1_da = vals[13]
# 设置dans_da为vals[14]
dans_da = vals[14]
# c加1
c = c + 1
# y加1
y = y + 1
# z加2
z = z + 2
# yc等于y乘以c
yc = y * c
# pk等于pkm1乘以z减去pkm2乘以yc
pk = pkm1 * z - pkm2 * yc
# qk等于qkm1乘以z减去qkm2乘以yc
qk = qkm1 * z - qkm2 * yc
# qk_is_nonzero等于qk不等于0
qk_is_nonzero = notequal(qk, 0)
# r等于pk除以qk
r = pk / qk
# t等于select函数，当qk_is_nonzero为真时，abs_x函数的结果等于ans减去r除以r，否则等于fill函数
t = select(qk_is_nonzero, abs_x((ans - r) / r), fill(dtype(t), shape(t), 1))
# ans等于select函数，当qk_is_nonzero为真时，r，否则等于ans
ans = select(qk_is_nonzero, r, ans)
# dpk_da等于dpkm1_da乘以z减去pkm1减去dpkm2_da乘以yc加起来乘以c
dpk_da = dpkm1_da * z - pkm1 - dpkm2_da * yc + pkm2 * c
# dqk_da等于dqkm1_da乘以z减去qkm1减去dqkm2_da乘以yc加起来乘以c
dqk_da = dqkm1_da * z - qkm1 - dqkm2_da * yc + qkm2 * c
# dans_da_new等于select函数，当qk_is_nonzero为真时，dpk_da减去ans乘以dqk_da除以qk，否则等于dans_da
dans_da_new = select(qk_is_nonzero, (dpk_da - ans * dqk_da) / qk, dans_da)
# grad_conditional等于select函数，当qk_is_nonzero为真时，abs_x函数的结果等于dans_da_new减去dans_da，否则等于fill函数
grad_conditional = select(qk_is_nonzero,
abs_x(dans_da_new - dans_da),
fill(dtype(dans_da), shape(dans_da), 1))
@ -526,26 +666,44 @@ def _igammac_continued_fraction(ax, x, a, enabled):
select(enabled, dqkm2_da, vals[11]), select(enabled, dpkm1_da, vals[12]),
select(enabled, dqkm1_da, vals[13]), select(enabled, dans_da_new, vals[14]))
# 计算x的值
y = 1 - a
# 计算z的值
z = x + y + 1
# 初始化c的值
c = fill(dtype(x), shape(x), 0)
# 初始化pkm2的值
pkm2 = fill(dtype(x), shape(x), 1)
# 初始化qkm2的值
qkm2 = x
# 初始化pkm1的值
pkm1 = x + 1
# 初始化qkm1的值
qkm1 = z * x
# 计算ans的值
ans = pkm1 / qkm1
# 初始化t的值
t = fill(dtype(x), shape(x), 1)
# 初始化dpkm2_da的值
dpkm2_da = fill(dtype(x), shape(x), 0)
# 初始化dqkm2_da的值
dqkm2_da = fill(dtype(x), shape(x), 0)
# 初始化dpkm1_da的值
dpkm1_da = fill(dtype(x), shape(x), 0)
# 初始化dqkm1_da的值
dqkm1_da = -x
# 计算dans_da的值
dans_da = (dpkm1_da - ans * dqkm1_da) / qkm1
# 将enabled，ans，t，y，z，c，pkm1，qkm1，pkm2，qkm2，dpkm2_da，dqkm2_da，dpkm1_da，dqkm1_da，dans_da放入vals中
vals = (enabled, ans, t, y, z, c, pkm1, qkm1, pkm2, qkm2, dpkm2_da, dqkm2_da, dpkm1_da, dqkm1_da, dans_da)
# 调用_while_helper_func函数，计算vals的值
vals = _while_helper_func(cond, body, vals)
# 获取ans的值
ans = vals[1]
# 返回ans乘以ax的值
return ans * ax
# 用于创建一个计算逆元Gamma函数
class IGamma(Cell):
r"""
Calculates lower regularized incomplete Gamma function.
@ -619,27 +777,40 @@ class IGamma(Cell):
x_dtype = self.dtype(x)
_check_input_dtype("a", a_dtype, [mstype.float32], self.cls_name)
_check_input_dtype("x", x_dtype, a_dtype, self.cls_name)
# 检查输入的类型是否正确
domain_error = self.logicalor(self.less(x, 0), self.less(a, 0))
use_igammac = self.logicaland(self.greater(x, 1), self.greater(x, a))
# 计算a*log(x) - x - lgamma(a)
ax = a * self.log(x) - x - self.lgamma(a)
# 获取ax的形状
para_shape = self.shape(ax)
# 如果ax的形状不为空，则进行广播
if para_shape != ():
broadcastto = P.BroadcastTo(para_shape)
x = broadcastto(x)
a = broadcastto(a)
# 判断x是否为0
x_is_zero = self.equal(x, 0)
# 计算log_maxfloat
log_maxfloat = self.log_maxfloat32
# 判断ax是否小于-log_maxfloat
underflow = self.less(ax, self.neg(log_maxfloat))
# 计算ax
ax = self.exp(ax)
# 判断是否启用
enabled = self.logicalnot(self.logicalor(self.logicalor(x_is_zero, domain_error), underflow))
# 计算output
output = self.select(use_igammac,
1 - _igammac_continued_fraction(ax, x, a, self.logicaland(enabled, use_igammac)),
_igamma_series(ax, x, a, self.logicaland(enabled, self.logicalnot(use_igammac))))
# 判断x是否为0，如果是，则output为0
output = self.select(x_is_zero, self.zeroslike(output), output)
# 判断domain_error，如果是，则output为nan
output = self.select(domain_error, self.fill(self.dtype(a), self.shape(a), np.nan), output)
# 返回output
return output
# 用于创建一个计算二项式Beta函数
class LBeta(Cell):
r"""
This method avoids the numeric cancellation by explicitly
@ -684,6 +855,7 @@ class LBeta(Cell):
super(LBeta, self).__init__()
# const numbers
self.log_2pi = np.log(2 * np.pi)
# 用于计算给定参数ax和x的逆元Gamma函数的series形式
self.minimax_coeff = [-0.165322962780713e-02,
0.837308034031215e-03,
-0.595202931351870e-03,
@ -700,29 +872,39 @@ class LBeta(Cell):
self.dtype = P.DType()
self.lgamma = LGamma()
self.const = P.ScalarToTensor()
def construct(self, x, y):
x_dtype = self.dtype(x)
y_dtype = self.dtype(y)
# 检查"x"的类型是否为float16或float32
_check_input_dtype("x", x_dtype, [mstype.float16, mstype.float32], self.cls_name)
# 检查"y"的类型是否与"x"的类型相同
_check_input_dtype("y", y_dtype, x_dtype, self.cls_name)
# 求x与y的和并赋值给x_plus_y
x_plus_y = x + y
# 获取x_plus_y的形状
para_shape = self.shape(x_plus_y)
if para_shape != ():
#用P.BroadcastTo将x和y广播到相同的形状
broadcastto = P.BroadcastTo(para_shape)
x = broadcastto(x)
y = broadcastto(y)
# 比较x和y
comp_less = self.less(x, y)
x_min = self.select(comp_less, x, y)
y_max = self.select(comp_less, y, x)
@constexpr
# 用于计算给定参数x的逆元Gamma函数的log correction
def _log_gamma_correction(x, minimax_coeff):
# 计算inverse_x和inverse_x_squared
inverse_x = 1. / x
inverse_x_squared = inverse_x * inverse_x
accum = minimax_coeff[0]
# 循环遍历
for i in range(1, 6):
accum = accum * inverse_x_squared + minimax_coeff[i]
# 返回accum * inverse_x的值
return accum * inverse_x
log_gamma_correction_x = _log_gamma_correction(x_min, self.minimax_coeff)
@ -730,6 +912,7 @@ class LBeta(Cell):
log_gamma_correction_x_y = _log_gamma_correction(x_plus_y, self.minimax_coeff)
# Two large arguments case: y >= x >= 8.
# 计算log_beta_two_large的值
log_beta_two_large = self.const(0.5 * self.log_2pi, x_dtype) - 0.5 * self.log(y_max) \
+ log_gamma_correction_x + log_gamma_correction_y - log_gamma_correction_x_y \
+ (x_min - 0.5) * self.log(x_min / (x_min + y_max)) - y_max * self.log1p(x_min / y_max)
@ -750,29 +933,44 @@ class LBeta(Cell):
@constexpr
# 用于获取两个形状x_shape和y_shape的广播矩阵乘法形状
def get_broadcast_matmul_shape(x_shape, y_shape, prim_name=None):
"""get broadcast_matmul shape"""
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 检查x_shape和y_shape的长度是否小于二，如果小于二
if (len(x_shape) < 2) or (len(y_shape) < 2):
# 抛出异常
raise ValueError(f"{msg_prefix} length of 'x_shape' and 'y_shape' should be equal to or greater than 2, "
f"but got the length of 'x_shape': {len(x_shape)} and the length of 'y_shape': "
f"{len(y_shape)}.")
# 计算x_shape_batch和y_shape_batch
x_shape_batch = x_shape[:-2]
y_shape_batch = y_shape[:-2]
# 比较x与y
if x_shape_batch == y_shape_batch:
# 返回x_shape，y_shape
return x_shape, y_shape
x_len = len(x_shape)
y_len = len(y_shape)
length = x_len if x_len < y_len else y_len
broadcast_shape_back = []
# 循环遍历从-length到-2
for i in range(-length, -2):
# 如果x_shape[i]为1
if x_shape[i] == 1:
# 将y_shape[i]添加到broadcast_shape_back列表中
broadcast_shape_back.append(y_shape[i])
# 如果y_shape[i]为1
elif y_shape[i] == 1:
# 将x_shape[i]添加到broadcast_shape_back列表中
broadcast_shape_back.append(x_shape[i])
# 如果x_shape[i]与y_shape[i]相同
elif x_shape[i] == y_shape[i]:
# 将x_shape[i]添加到broadcast_shape_back列表中
broadcast_shape_back.append(x_shape[i])
# 否则
else:
# 抛出异常
raise ValueError(f"{msg_prefix} 'x_shape[{i}]' should be equal to 1, or the 'y_shape[{i}]' should be equal "
f"to 1, or the 'x_shape[{i}]' should be equal to 'y_shape[{i}]', but got "
f"'x_shape[{i}]': {x_shape[i]}, 'y_shape[{i}]': {y_shape[i]}.")
@ -784,42 +982,58 @@ def get_broadcast_matmul_shape(x_shape, y_shape, prim_name=None):
@constexpr
# 用于检查两个形状x1_shape和x2_shape是否为列向量或行向量
def check_col_row_equal(x1_shape, x2_shape, transpose_x1, transpose_x2, prim_name=None):
"""check col and row equal"""
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果x1_shape的长度为1
if len(x1_shape) == 1:
transpose_x1 = False
# 转化为单列二维形状
x1_shape = (1,) + x1_shape
# 如果x2_shape的长度为1
if len(x2_shape) == 1:
transpose_x2 = False
# 转化为单列二维形状
x2_shape = x2_shape + (1,)
x1_last = x1_shape[-2:]
x2_last = x2_shape[-2:]
# 用x1_col表示x1中矩阵维度的最后一列
x1_col = x1_last[not transpose_x1] # x1_col = x1_last[1] if (not transpose_a) else x1_last[0]
# 用x2_row表示x2中矩阵维度的最后一行
x2_row = x2_last[transpose_x2] # x2_row = x2_last[0] if (not transpose_b) else x2_last[1]
# 如果x1_col和x2_row不相等
if x1_col != x2_row:
# 抛出异常
raise ValueError(f"{msg_prefix} column of matrix dimensions of 'x1' should be equal to "
f"the row of matrix dimensions of 'x2', but got 'x1_col' {x1_col} and 'x2_row' {x2_row}.")
# 根据输入数据选择合适的矩阵乘法
def matmul_op_select(x1_shape, x2_shape, transpose_x1, transpose_x2):
"""select matmul op"""
x1_dim, x2_dim = len(x1_shape), len(x2_shape)
# 判断x1和x2的维度
if x1_dim == 1 and x2_dim == 1:
# 如果x1和x2的维度都为1，则使用乘法操作
matmul_op = P.Mul()
elif x1_dim <= 2 and x2_dim <= 2:
# 如果x1和x2的维度都小于等于2，则使用矩阵乘法操作
transpose_x1 = False if x1_dim == 1 else transpose_x1
transpose_x2 = False if x2_dim == 1 else transpose_x2
matmul_op = P.MatMul(transpose_x1, transpose_x2)
elif x1_dim == 1 and x2_dim > 2:
# 如果x1的维度为1，x2的维度大于2，则使用批量矩阵乘法操作
matmul_op = P.BatchMatMul(False, transpose_x2)
elif x1_dim > 2 and x2_dim == 1:
# 如果x1的维度大于2，x2的维度为1，则使用批量矩阵乘法操作
matmul_op = P.BatchMatMul(transpose_x1, False)
else:
# 如果x1和x2的维度都不满足上面的条件，则使用批量矩阵乘法操作
matmul_op = P.BatchMatMul(transpose_x1, transpose_x2)
# 返回矩阵乘法操作
return matmul_op
# 用于矩阵乘法计算图类
class MatMul(Cell):
r"""
The nn.MatMul interface is deprecated, please use the :class:`mindspore.ops.matmul` instead.
@ -829,14 +1043,19 @@ class MatMul(Cell):
"""
@deprecated('1.2', 'ops.matmul', False)
# 初始化函数
def __init__(self, transpose_x1=False, transpose_x2=False):
"""Initialize MatMul."""
super(MatMul, self).__init__()
# 检查'transpose_x1'是否为bool(布尔)型
validator.check_value_type('transpose_x1', transpose_x1, [bool], self.cls_name)
# 检查'transpose_x2'是否为bool(布尔)型
validator.check_value_type('transpose_x2', transpose_x2, [bool], self.cls_name)
# 用self.transpose_x1记录transpose_x1的值
self.transpose_x1 = transpose_x1
# 用self.transpose_x2记录transpose_x2的值
self.transpose_x2 = transpose_x2
# 定义操作
self.shape_op = P.Shape()
self.expand_op = P.ExpandDims()
self.squeeze_left_op = P.Squeeze(-2)
@ -846,37 +1065,52 @@ class MatMul(Cell):
def construct(self, x1, x2):
x1_shape = self.shape_op(x1)
x2_shape = self.shape_op(x2)
# 检查x1和x2的形状是否相等
check_col_row_equal(x1_shape, x2_shape, self.transpose_x1, self.transpose_x2, self.cls_name)
# 根据x1和x2的形状，选择矩阵乘法操作
matmul_op = matmul_op_select(x1_shape, x2_shape, self.transpose_x1, self.transpose_x2)
# 获取x1和x2的维度
x1_dim, x2_dim = len(x1_shape), len(x2_shape)
# 如果x1和x2的维度都为1
if x1_dim == x2_dim and x2_dim == 1:
# 返回矩阵乘法的结果
return self.reduce_sum_op(matmul_op(x1, x2), -1)
# 如果x1的维度为1
if x1_dim == 1:
# 将x1扩展到x1_shape
x1 = self.expand_op(x1, 0)
# 获取x1的形状
x1_shape = self.shape_op(x1)
# 如果x2的维度为1
if x2_dim == 1:
# 将x2扩展到x2_shape
x2 = self.expand_op(x2, 1)
# 获取x2的形状
x2_shape = self.shape_op(x2)
# 通过 get_broadcast_matmul_shape 函数获取两个张量x1与x2的广播形状
x1_broadcast_shape, x2_broadcast_shape = get_broadcast_matmul_shape(x1_shape, x2_shape)
x1_broadcast_to = P.BroadcastTo(x1_broadcast_shape)
x2_broadcast_to = P.BroadcastTo(x2_broadcast_shape)
# 检查x1和x2广播形状是否与原始形状相同
if x1_broadcast_shape != x1_shape:
x1 = x1_broadcast_to(x1)
if x2_broadcast_shape != x2_shape:
x2 = x2_broadcast_to(x2)
matmul_broadcast = matmul_op(x1, x2)
# 如果x1是1维
if x1_dim == 1:
# 对x1进行squeeze操作
matmul_broadcast = self.squeeze_left_op(matmul_broadcast)
# 如果x2是1维
if x2_dim == 1:
# 对x2进行squeeze操作
matmul_broadcast = self.squeeze_right_op(matmul_broadcast)
# 返回matmul_broadcast
return matmul_broadcast
# 用于计算张量各个分量统计指标
class Moments(Cell):
"""
Calculate the mean and variance of the input `x` along the specified `axis`.
@ -943,15 +1177,19 @@ class Moments(Cell):
def __init__(self, axis=None, keep_dims=None):
"""Initialize Moments."""
super(Moments, self).__init__()
# 检查axis的类型是否为tuple
if axis is None:
axis = ()
if isinstance(axis, tuple):
for idx, item in enumerate(axis):
validator.check_value_type("axis[%d]" % idx, item, [int], self.cls_name)
# 检查axis的类型是否为int或tuple
self.axis = validator.check_value_type('axis', axis, [int, tuple], self.cls_name)
# 检查keep_dims的类型是否为bool
if keep_dims is None:
keep_dims = False
self.keep_dims = validator.check_value_type('keep_dims', keep_dims, [bool], self.cls_name)
# 初始化cast，reduce_mean，square_diff，squeeze
self.cast = P.Cast()
self.reduce_mean = P.ReduceMean(keep_dims=True)
self.square_diff = P.SquaredDifference()
@ -962,18 +1200,22 @@ class Moments(Cell):
_check_input_dtype("input x", tensor_dtype, [mstype.float16, mstype.float32], self.cls_name)
if tensor_dtype == mstype.float16:
x = self.cast(x, mstype.float32)
# 计算x的均值
mean = self.reduce_mean(x, self.axis)
# 计算x的方差
variance = self.reduce_mean(self.square_diff(x, F.stop_gradient(mean)), self.axis)
# 如果keep_dims为False，则删除mean和variance的维度
if not self.keep_dims:
mean = self.squeeze(mean)
variance = self.squeeze(variance)
# 如果x的类型为float16，则将mean和variance的类型转换为float16
if tensor_dtype == mstype.float16:
mean = self.cast(mean, mstype.float16)
variance = self.cast(variance, mstype.float16)
return mean, variance
return mean, variance
# 用于计算矩阵的逆的函数
class MatInverse(Cell):
"""
Calculates the inverse of Positive-Definite Hermitian matrix using Cholesky decomposition.
@ -1003,18 +1245,20 @@ class MatInverse(Cell):
def __init__(self):
"""Initialize MatInverse."""
super(MatInverse, self).__init__()
# 初始化dtype、choleskytrsm、matmul
self.dtype = P.DType()
self.choleskytrsm = P.CholeskyTrsm()
self.matmul = MatMul(transpose_x1=True)
def construct(self, a):
input_dtype = self.dtype(a)
# 检查"input_a"是否为float16或float32
_check_input_dtype("input_a", input_dtype, [mstype.float16, mstype.float32], self.cls_name)
l_inverse = self.choleskytrsm(a)
a_inverse = self.matmul(l_inverse, l_inverse)
return a_inverse
# 用于创建一个计算矩阵行列式
class MatDet(Cell):
"""
Calculates the determinant of Positive-Definite Hermitian matrix using Cholesky decomposition.
@ -1039,9 +1283,11 @@ class MatDet(Cell):
>>> print(output)
35.999996
"""
# 初始化
def __init__(self):
"""Initialize MatDet."""
super(MatDet, self).__init__()
# 设置属性
self.dtype = P.DType()
self.cholesky = P.Cholesky()
self.det_triangle = P.DetTriangle()
@ -1049,6 +1295,7 @@ class MatDet(Cell):
def construct(self, a):
input_dtype = self.dtype(a)
# 检查"input_a"是否为float16或float32
_check_input_dtype("input_a", input_dtype, [mstype.float16, mstype.float32], self.cls_name)
l = self.cholesky(a)
l_det = self.det_triangle(l)

207
mindspore/python/mindspore/nn/layer/normalization.py
View File

@ -15,23 +15,37 @@
"""normalization"""
import itertools
import numbers
# 从mindspore.ops模块中导入operations模块，用于提供基本操作（如矩阵乘法、加法等）
from mindspore.ops import operations as P
# 从mindspore.ops.functional模块中导入functional类，用于提供一些基本功能，如条件判断、循环等
from mindspore.ops import functional as F
# 从mindspore.ops.operations模块中导入_inner_ops模块，用于提供内部操作
from mindspore.ops.operations import _inner_ops as inner
# 从mindspore.common.parameter模块中导入Parameter类，用于创建一个参数
from mindspore.common.parameter import Parameter
# 从mindspore.common.initializer模块中导入initializer函数（用于初始化参数）和Initializer类（用于创建初始化器）
from mindspore.common.initializer import initializer, Initializer
# 从mindspore.common.tensor模块中导入Tensor类，用于表示张量
from mindspore.common.tensor import Tensor
# 从mindspore.common._decorator模块中导入deprecated装饰器，用于标记已过时的功能
from mindspore.common._decorator import deprecated
from mindspore.ops.primitive import constexpr
import mindspore.context as context
# 从mindspore._checkparam模块中导入Rel类（用于表示相对关系）
from mindspore._checkparam import Rel
# 从mindspore._checkparam模块中导入Validator类（用于验证参数的范围和类型）
from mindspore._checkparam import Validator as validator
# 从mindspore._extends模块中导入cell_attr_register函数，用于注册Cell类的属性
from mindspore._extends import cell_attr_register
# mindspore.communication.management模块中导入get_group_size和get_rank函数，用于获取集群中组的大小和当前节点的排名
from mindspore.communication.management import get_group_size, get_rank
# 从mindspore.communication模块中导入management类，用于处理通信相关功能
from mindspore.communication import management
# 从mindspore.common模块中导入dtype类，用于表示数据类型
from mindspore.common import dtype as mstype
# 从mindspore.parallel._utils模块中导入_is_in_auto_parallel_mode函数，用于判断当前是否为自动并行模式
from mindspore.parallel._utils import _is_in_auto_parallel_mode
# from ..cell import Cell：从当前模块（..cell）中导入Cell类，用于创建自定义Cell
from ..cell import Cell
__all__ = ['BatchNorm1d', 'BatchNorm2d', 'BatchNorm3d', 'LayerNorm', 'GroupNorm',
@ -39,11 +53,26 @@ __all__ = ['BatchNorm1d', 'BatchNorm2d', 'BatchNorm3d', 'LayerNorm', 'GroupNorm'
SYNC_BN_GROUP_NAME = ""
# 用于实现批量归一化操作
class _BatchNorm(Cell):
"""Batch Normalization base class."""
@cell_attr_register
# 初始化
# 参数
# num_features：输入数据的特征数量（通道数）
# eps：防止除以零的值，默认值为1e-5
# momentum：用于更新移动平均和方差的动量系数。默认值为0.9
# affine：一个布尔值，表示是否对输入数据进行缩放和平移，默认值为True
# gamma_init：缩放参数（gamma）的初始值，可以是'ones'、'zeros'或一个张量，默认值为'ones'
# beta_init：偏移参数（beta）的初始值，可以是'zeros'或一个张量，默认值为'zeros'
# moving_mean_init：移动平均的初始值，可以是'zeros'或一个张量，默认值为'zeros'
# moving_var_init：移动方差的初始值，可以是'ones'或一个张量，默认值为'ones'
# use_batch_statistics：一个布尔值，表示是否使用批量统计信息进行归一化，如果设置为None，则使用全局批量统计信息，默认值为None
# device_num_each_group：每个进程组的设备数量，默认值为1
# process_groups：进程组的数量，默认值为0
# input_dims：输入数据的维度，可以是'1d'、'2d'或'3d'，默认值为'2d'
# data_format：输入数据的格式，可以是'NCHW'或'NHWC'，默认值为'NCHW'
def __init__(self,
num_features,
eps=1e-5,
@ -60,11 +89,15 @@ class _BatchNorm(Cell):
data_format='NCHW'):
"""Initialize _BatchNorm."""
super(_BatchNorm, self).__init__()
# 检查'num_features'的类型是否为int
validator.check_value_type('num_features', num_features, [int], self.cls_name)
# 如果num_features小于1
if num_features < 1:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'num_features' must be at least 1, but got {num_features}.")
# 如果momentum小于0或大于1
if momentum < 0 or momentum > 1:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'momentum' should be a number in range [0, 1], "
f"but got {momentum}.")
self.input_dims = input_dims
@ -78,19 +111,28 @@ class _BatchNorm(Cell):
f" but got {use_batch_statistics}.")
self.num_features = num_features
self.eps = eps
# 初始化moving_mean
self.moving_mean = Parameter(initializer(
moving_mean_init, num_features), name="mean", requires_grad=False)
# 初始化moving_variance
self.moving_variance = Parameter(initializer(
moving_var_init, num_features), name="variance", requires_grad=False)
# 初始化gamma
self.gamma = Parameter(initializer(
gamma_init, num_features), name="gamma", requires_grad=affine)
# 初始化beta
self.beta = Parameter(initializer(
beta_init, num_features), name="beta", requires_grad=affine)
# 检查device_num_each_group是否为正整数
self.group_device_num = validator.check_positive_int(device_num_each_group, "device_num_each_group",
self.cls_name)
# 初始化process_groups
self.process_groups = process_groups
# 初始化is_global
self.is_global = False
# 获取并行模式
self.parallel_mode = context.get_auto_parallel_context("parallel_mode")
# 初始化SYNC_BN_GROUP_NAME
global SYNC_BN_GROUP_NAME
# for GlobalBatchNorm
if self.group_device_num != 1:
@ -102,24 +144,40 @@ class _BatchNorm(Cell):
self._create_global_groups()
# for SyncBatchNorm
if self.process_groups != 0:
# 获取当前进程的rank_id
self.rank_id = get_rank()
# 获取当前进程的rank_size
self.rank_size = get_group_size()
# 如果process_groups不为空
if self.process_groups is not None:
# 检查process_groups是否为list
validator.check_isinstance("process_groups", self.process_groups, list)
# 检查rank_ids是否正确
self._check_rank_ids(self.process_groups, self.rank_size)
# 创建同步组
self._create_sync_groups()
# 如果process_groups为空，且rank_size大于1
elif self.rank_size > 1:
# 设置is_global为True
self.is_global = True
# 设置group_device_num为rank_size
self.group_device_num = self.rank_size
# 设置device_list
self.device_list = [i for i in range(0, self.rank_size)]
# 如果设备类型为Ascend
if context.get_context("device_target") == "Ascend":
# 如果SYNC_BN_GROUP_NAME为空
if SYNC_BN_GROUP_NAME == "":
# 设置SYNC_BN_GROUP_NAME
SYNC_BN_GROUP_NAME = "sync_bn_group0"
# 创建同步组
management.create_group(SYNC_BN_GROUP_NAME, self.device_list)
# 如果设备类型为GPU
elif context.get_context("device_target") == "GPU":
# 如果SYNC_BN_GROUP_NAME为空
if SYNC_BN_GROUP_NAME == "":
# 设置SYNC_BN_GROUP_NAME
SYNC_BN_GROUP_NAME = "nccl_world_group"
self.shape = P.Shape()
self.reduce_mean = P.ReduceMean(keep_dims=True)
self.square = P.Square()
@ -135,57 +193,74 @@ class _BatchNorm(Cell):
else:
self.is_ge_backend = False
# 初始化训练模式下的batchnorm
self.bn_train = P.BatchNorm(is_training=True,
epsilon=self.eps,
momentum=self.momentum,
data_format=self.format)
# 如果是全局模式，则使用同步batchnorm
if self.is_global:
self.bn_train = inner.SyncBatchNorm(epsilon=self.eps,
momentum=self.momentum,
group=SYNC_BN_GROUP_NAME,
device_num=self.group_device_num)
# 初始化测试模式下的batchnorm
self.bn_infer = P.BatchNorm(is_training=False, epsilon=self.eps, data_format=self.format)
# 如果是自动并行模式，则设置数据并行策略
if _is_in_auto_parallel_mode():
data_parallel_strategy = ((1,), (1,))
data_parallel_strategy_one = ((1,), ())
else:
data_parallel_strategy = None
data_parallel_strategy_one = None
# 初始化减法操作
self.sub_mean = P.Sub().shard(data_parallel_strategy)
self.sub_var = P.Sub().shard(data_parallel_strategy)
# 初始化乘法操作
self.mul_mean = P.Mul().shard(data_parallel_strategy_one)
self.mul_var = P.Mul().shard(data_parallel_strategy_one)
# 初始化减法赋值操作
self.assign_sub_mean = P.AssignSub().shard(data_parallel_strategy)
self.assign_sub_var = P.AssignSub().shard(data_parallel_strategy)
# 检查输入数据x的维度
def _check_data_dim(self, x):
# 抛出异常
raise NotImplementedError
# 用于根据进程号和进程组大小获取进程组中的所有进程
def list_group(self, world_rank, group_size):
""" Check whether world_rank and group_size are valid. """
# 如果进程组的进程数量大于本地进程数量
if group_size > get_group_size():
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'device_num_each_group' cannot be greater than "
f"local rank size, but got 'device_num_each_group': {group_size}, "
f"local rank size: {get_group_size()}.")
# 如果world_rank列表的长度不可以被group_size整除
if len(world_rank) % group_size != 0:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the dimension of device_list should be divisible by "
f"'device_num_each_group', but got the length of device_list: {len(world_rank)}, "
f"'device_num_each_group': {group_size}.")
# 将world_rank按照group_size分组，并将其转换为list
world_rank_list = zip(*(iter(world_rank),) * group_size)
group_list = [list(i) for i in world_rank_list]
return group_list
# 用于检查进程组和本地进程数量是否匹配
def _check_rank_ids(self, process_groups, rank_size):
seen = set()
for rid in itertools.chain(*process_groups):
# 检查rank id是否在0到rank_size之间
validator.check_int_range(rid, 0, rank_size, Rel.INC_LEFT, "rank id in process_groups", self.cls_name)
# 检查process_groups中是否有重复的rank id
if rid in seen:
#抛出异常
raise ValueError(f"For '{self.cls_name}', rank id in 'process_groups' should not be duplicated, "
f"but got {process_groups}.")
seen.add(rid)
# 用于创建一个包含所有进程的全局进程组
def _create_global_groups(self):
# 循环遍历self.rank_list_idx
for i in range(self.rank_list_idx):
if self.rank_id in self.rank_list[i]:
self.is_global = True
@ -193,34 +268,46 @@ class _BatchNorm(Cell):
if SYNC_BN_GROUP_NAME == "":
SYNC_BN_GROUP_NAME = "sync_bn_group%d" % i
management.create_group(SYNC_BN_GROUP_NAME, self.rank_list[i])
# 用于创建一个包含所有进程的同步进程组
def _create_sync_groups(self):
for i in range(len(self.process_groups)):
# 遍历process_groups列表
for i in range(len(self.process_groups)):
# 检查"process_groups[%d]"
validator.check_isinstance("process_groups[%d]" % i, self.process_groups[i], list)
# 获取process_groups列表中每一个元素的长度
self.group_device_num = len(self.process_groups[i])
# 检查rank_id是否在process_groups列表中，并且列表中元素的长度大于1
if self.rank_id in self.process_groups[i] and self.group_device_num > 1:
# 设置is_global为True
self.is_global = True
# 设置SYNC_BN_GROUP_NAME
global SYNC_BN_GROUP_NAME
if SYNC_BN_GROUP_NAME == "":
SYNC_BN_GROUP_NAME = "sync_bn_group%d" % i
# 创建一个组
management.create_group(SYNC_BN_GROUP_NAME, self.process_groups[i])
def construct(self, x):
# 检查输入张量的形状是否与Batch Normalization层的参数匹配
_shape_check_bn(self.shape(x), self.input_dims, self.cls_name)
# 如果self.use_batch_statistics为None，则根据self.training的值来决定是训练模式还是推理模式
if self.use_batch_statistics is None:
if self.training:
# 如果是训练模式，则调用bn_train函数
return self.bn_train(x,
self.gamma,
self.beta,
self.moving_mean,
self.moving_variance)[0]
if not self.training:
# 如果是推理模式，则调用bn_infer函数
return self.bn_infer(x,
self.gamma,
self.beta,
self.moving_mean,
self.moving_variance)[0]
# 如果self.use_batch_statistics为True，则调用bn_train函数
if self.use_batch_statistics:
return self.bn_train(x,
self.gamma,
@ -228,6 +315,7 @@ class _BatchNorm(Cell):
self.moving_mean,
self.moving_variance)[0]
# 如果self.use_batch_statistics为False，则调用bn_infer函数
return self.bn_infer(x,
self.gamma,
self.beta,
@ -240,47 +328,65 @@ class _BatchNorm(Cell):
@constexpr
# 检查输入通道数是否与num_channel相等
def _channel_check(channel, num_channel, prim_name=None):
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果不相等
if channel != num_channel:
# 抛出异常
raise ValueError(f"{msg_prefix} channel(the second dim of the input 'x') should be equal to num_channels, "
f"but got channel: {channel}, num_channels: {num_channel}.")
@constexpr
# 检查输入shape是否为4维
def _shape_check(in_shape, prim_name=None):
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果不是4维
if len(in_shape) != 4:
# 抛出异常
raise ValueError(f"{msg_prefix} in_shape must has 4 dims, but got the length of in_shape: {len(in_shape)}.")
@constexpr
# 检查输入维度是否符合batch norm的要求
def _shape_check_bn(in_shape, in_dims, prim_name=None):
"""check input dims of batch norm."""
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
dim = len(in_shape)
# 如果in_dims是1维，dims不是2维
if in_dims == '1d' and dim != 2:
# 抛出异常
raise ValueError(f"{msg_prefix} in_shape must have 2 dims, but got {len(in_shape)}.")
# 如果in_dims是2维，dims不是4维
if in_dims == '2d' and dim != 4:
# 抛出异常
raise ValueError(f"{msg_prefix} in_shape must have 4 dims, but got {len(in_shape)}.")
# 如果in_dims是3维，dims不是5维
if in_dims == '3d' and dim != 5:
# 抛出异常
raise ValueError(f"{msg_prefix} in_shape must have 5 dims, but got {len(in_shape)}.")
# 如果in_dims是both并且dim不是2维和4维
if in_dims == 'both' and dim != 2 and dim != 4:
# 抛出异常
raise ValueError(f"{msg_prefix} in_shape must have 2 dims or 4 dims, but got {len(in_shape)}.")
@constexpr
# 定义一个函数，用于推断输入x的形状和axes
def _shape_infer(x_shape, num_feature):
"""global Batch Normalization shape and axes infer"""
# 如果输入x的形状为4，则axes为(0,2,3)，re_shape为(1,num_feature,1,1)
if len(x_shape) == 4:
axes = (0, 2, 3)
re_shape = (1, num_feature, 1, 1)
# 如果输入x的形状不为4，则axes为(0,)，re_shape为(1,num_feature)
else:
axes = (0,)
re_shape = (1, num_feature)
# 返回axes和re_shape
return axes, re_shape
# 在二维或三维输入（mini-batch 一维输入或二维输入）上应用批归一化（Batch Normalization Layer），避免内部协变量偏移
class BatchNorm1d(_BatchNorm):
r"""
Batch Normalization layer over a 2D input.
@ -373,7 +479,7 @@ class BatchNorm1d(_BatchNorm):
if x.ndim != 2:
pass
# 在四维输入（具有额外通道维度的小批量二维输入）上应用批归一化处理（Batch Normalization Layer），以避免内部协变量偏移
class BatchNorm2d(_BatchNorm):
r"""
Batch Normalization layer over a 4D input.
@ -459,7 +565,7 @@ class BatchNorm2d(_BatchNorm):
[[ 0.999995 0.999995 ]
[ 0.999995 0.999995 ]]]]
"""
# 参数与第一个类中的init类似
def __init__(self,
num_features,
eps=1e-5,
@ -490,18 +596,22 @@ class BatchNorm2d(_BatchNorm):
@constexpr
# 检查输入张量是否为5维
def _check_3d_shape(input_shape, prim_name=None):
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果不是5维
if len(input_shape) != 5:
# 抛出异常
raise ValueError(f"{msg_prefix} input_shape must be 5-dimensional, but got the length of input_shape: "
f"{len(input_shape)}.")
@constexpr
def _check_dtype(dtype, valid_dtypes, args_name, prim_name=None):
# 检查args_name的类型是否为dtype，并且是否在valid_dtypes中
validator.check_type_name(args_name, dtype, valid_dtypes, prim_name)
# 在二维或三维输入（mini-batch 一维输入或二维输入）上应用批归一化（Batch Normalization Layer），避免内部协变量偏移
class BatchNorm3d(Cell):
r"""
Batch Normalization layer over a 5D input.
@ -566,7 +676,7 @@ class BatchNorm3d(Cell):
>>> print(output.shape)
(16, 3, 10, 32, 32)
"""
# 参数与第一个类中的init类似
def __init__(self,
num_features,
eps=1e-5,
@ -595,13 +705,17 @@ class BatchNorm3d(Cell):
def construct(self, input_x):
x_shape = F.shape(input_x)
# 检查x的形状是否是3维
_check_3d_shape(x_shape, self.cls_name)
input_x = self.reshape(input_x, (x_shape[0], x_shape[1], x_shape[2] * x_shape[3], x_shape[4]))
# 用bn2d来处理input_x
bn2d_out = self.bn2d(input_x)
# 用reshape方法将张量转换为原来形状
bn3d_out = self.reshape(bn2d_out, x_shape)
# 返回bn3d_out
return bn3d_out
# 用于处理全局输入的张量
class GlobalBatchNorm(_BatchNorm):
r"""
The GlobalBatchNorm interface is deprecated, please use the :class:`mindspore.nn.SyncBatchNorm` instead.
@ -609,7 +723,8 @@ class GlobalBatchNorm(_BatchNorm):
Supported Platforms:
deprecated
"""
# 参数与第一个类中相似
@deprecated("1.2", "SyncBatchNorm", True)
def __init__(self,
num_features,
@ -634,17 +749,22 @@ class GlobalBatchNorm(_BatchNorm):
use_batch_statistics,
device_num_each_group,
input_dims='both')
# 检查device_num_each_group是否为正数
self.group_device_num = validator.check_positive_int(device_num_each_group, "device_num_each_group",
self.cls_name)
# 如果device_num_each_group小于等于1
if self.group_device_num <= 1:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'device_num_each_group' must be greater than 1, "
f"but got {self.group_device_num}.")
def _check_data_dim(self, x):
# 检查输入维度
if x.dim == 0:
pass
# 主要用于处理分布式输入的张量
# 在N维输入上进行跨设备同步批归一化（Batch Normalization，BN）
class SyncBatchNorm(_BatchNorm):
r"""
Sync Batch Normalization layer over a N-dimension input.
@ -733,7 +853,7 @@ class SyncBatchNorm(_BatchNorm):
[[ 0.999995 0.999995 ]
[ 0.999995 0.999995 ]]]]
"""
# 参数和第一个类相似
def __init__(self,
num_features,
eps=1e-5,
@ -762,7 +882,8 @@ class SyncBatchNorm(_BatchNorm):
if x.dim == 0:
pass
# 用于处理输入张量，并实现层归一化
# 在mini-batch输入上应用层归一化（Layer Normalization）
class LayerNorm(Cell):
r"""
Applies Layer Normalization over a mini-batch of inputs.
@ -827,17 +948,21 @@ class LayerNorm(Cell):
):
"""Initialize LayerNorm."""
super(LayerNorm, self).__init__()
# 检查normalized_shape是否为元组或列表
if not isinstance(normalized_shape, (tuple, list)):
# 抛出异常
raise TypeError(f"For '{self.cls_name}', the type of 'normalized_shape' should be tuple[int] or list[int], "
f"but got {normalized_shape} and the type is {type(normalized_shape)}.")
self.normalized_shape = normalized_shape
self.begin_norm_axis = begin_norm_axis
self.begin_params_axis = begin_params_axis
self.epsilon = epsilon
# 初始化gamma和beta
self.gamma = Parameter(initializer(
gamma_init, normalized_shape), name="gamma")
self.beta = Parameter(initializer(
beta_init, normalized_shape), name="beta")
# 初始化LayerNorm
self.layer_norm = P.LayerNorm(begin_norm_axis=self.begin_norm_axis,
begin_params_axis=self.begin_params_axis,
epsilon=self.epsilon)
@ -850,7 +975,7 @@ class LayerNorm(Cell):
return 'normalized_shape={}, begin_norm_axis={}, begin_params_axis={}, gamma{}, beta={}'.format(
self.normalized_shape, self.begin_norm_axis, self.begin_params_axis, self.gamma, self.beta)
# 该层在四维输入（带有额外通道维度的mini-batch二维输入）上应用实例归一化
class InstanceNorm2d(Cell):
r"""
Instance Normalization layer over a 4D input.
@ -931,9 +1056,13 @@ class InstanceNorm2d(Cell):
beta_init='zeros'):
"""Initialize InstanceNorm2d."""
super(InstanceNorm2d, self).__init__()
# 检查'num_features'是否为int型
validator.check_value_type('num_features', num_features, [int], self.cls_name)
# 检查'eps'是否为float型
validator.check_value_type('eps', eps, [float], self.cls_name)
# 检查'momentum'是否为float型
validator.check_value_type('momentum', momentum, [float], self.cls_name)
# 检查'affine'是否为bool型
validator.check_value_type('affine', affine, [bool], self.cls_name)
args_input = {"gamma_init": gamma_init, "beta_init": beta_init}
self.check_types_valid(args_input, 'InstanceNorm2d')
@ -946,21 +1075,27 @@ class InstanceNorm2d(Cell):
self.num_features = num_features
self.eps = eps
self.input_dims = '2d'
# 初始化moving_mean
self.moving_mean = Parameter(initializer('zeros', num_features), name="mean", requires_grad=False)
# 初始化moving_variance
self.moving_variance = Parameter(initializer('ones', num_features), name="variance", requires_grad=False)
# 初始化gamma
self.gamma = Parameter(initializer(
gamma_init, num_features), name="gamma", requires_grad=affine)
# 初始化beta
self.beta = Parameter(initializer(
beta_init, num_features), name="beta", requires_grad=affine)
self.shape = P.Shape()
self.momentum = momentum
self.instance_bn = P.InstanceNorm(epsilon=self.eps, momentum=self.momentum)
# 检查输入张量的维度是否为1
def _check_data_dim(self, x):
# 抛出异常NotImplementedError
raise NotImplementedError
def construct(self, x):
# 检查输入张量的形状是否与LayerNorm类的input_dims参数匹配
_shape_check_bn(self.shape(x), self.input_dims, self.cls_name)
return self.instance_bn(x,
self.gamma,
@ -973,16 +1108,21 @@ class InstanceNorm2d(Cell):
self.num_features, self.eps, self.momentum, self.gamma, self.beta, self.moving_mean, self.moving_variance)
def check_types_valid(self, args_dict, name):
for key, _ in args_dict.items():
# 遍历
for key, _ in args_dict.items():
val = args_dict[key]
# 检查每个键值的类型是否为Tensor、numbers.Number、str或Initializer
if not isinstance(val, (Tensor, numbers.Number, str, Initializer)):
# 抛出异常
raise TypeError(f"For '{self.cls_name}', the type of '{key}' should be in "
f"[Tensor, numbers.Number, str, Initializer], but got type {type(val).__name__}.")
# 检查了Tensor类型的值是否为float32
if isinstance(val, Tensor) and val.dtype != mstype.float32:
# 抛出异常
raise TypeError(f"For '{self.cls_name}', the type of '{key}' should be float32, "
f"but got {val.dtype}.")
# 主要用于处理输入张量，并实现组归一化
# 在mini-batch输入上进行组归一化
class GroupNorm(Cell):
r"""
Group Normalization over a mini-batch of inputs.
@ -1050,12 +1190,15 @@ class GroupNorm(Cell):
self.eps = validator.check_value_type('eps', eps, (float,), type(self).__name__)
self.affine = validator.check_bool(affine, arg_name="affine", prim_name=self.cls_name)
# 初始化gamma和beta
self.gamma = Parameter(initializer(
gamma_init, num_channels), name="gamma", requires_grad=affine)
self.beta = Parameter(initializer(
beta_init, num_channels), name="beta", requires_grad=affine)
# 初始化shape和reshape
self.shape = F.shape
self.reshape = F.reshape
# 初始化reduce_mean、square、reduce_sum和sqrt
self.reduce_mean = P.ReduceMean(keep_dims=True)
self.square = F.square
self.reduce_sum = P.ReduceSum(keep_dims=True)
@ -1063,19 +1206,29 @@ class GroupNorm(Cell):
def _cal_output(self, x):
"""calculate groupnorm output"""
# 检查输入的形状
batch, channel, height, width = self.shape(x)
# 检查输入的通道数
_channel_check(channel, self.num_channels, self.cls_name)
# 将输入的形状转换为(batch, num_groups, -1)
x = self.reshape(x, (batch, self.num_groups, -1))
# 计算均值
mean = self.reduce_mean(x, 2)
# 计算方差
var = self.reduce_sum(self.square(x - mean), 2) / (channel * height * width / self.num_groups)
# 计算标准差
std = self.sqrt(var + self.eps)
# 将均值和标准差应用到输入中
x = (x - mean) / std
# 将输入的形状转换为(batch, channel, height, width)
x = self.reshape(x, (batch, channel, height, width))
# 计算输出
output = x * self.reshape(self.gamma, (-1, 1, 1)) + self.reshape(self.beta, (-1, 1, 1))
return output
def construct(self, x):
_shape_check(self.shape(x), self.cls_name)
# 检查x.dtype是否为float16或float32
_check_dtype(x.dtype, [mstype.float16, mstype.float32], "input", self.cls_name)
output = self._cal_output(x)
return output

77
mindspore/python/mindspore/nn/layer/pooling.py
View File

@ -13,45 +13,68 @@
# limitations under the License.
# ============================================================================
"""pooling"""
# 从mindspore.ops模块中导入operations（操作）类，用于定义各种计算图操作
from mindspore.ops import operations as P
# 从mindspore.ops模块中导入functional（功能）类，用于提供一些实用的功能函数，如F.reduce_sum、F.shape等
from mindspore.ops import functional as F
# 从mindspore._checkparam模块中导入Rel和Validator类，用于验证参数的合法性
from mindspore._checkparam import Rel, Validator as validator
# 从mindspore.ops.primitive模块中导入constexpr函数，用于在编译时计算常量表达式
from mindspore.ops.primitive import constexpr
# 从mindspore.context模块中导入context类，用于设置计算图的运行环境
import mindspore.context as context
# 从..cell模块中导入Cell类，用于创建自定义的神经网络层
from ..cell import Cell
__all__ = ['AvgPool2d', 'MaxPool2d', 'AvgPool1d', 'MaxPool1d']
# 用于实现各种池化操作，如最大池化、平均池化、L2池化等
class _PoolNd(Cell):
"""N-D AvgPool"""
def __init__(self, kernel_size, stride, pad_mode, data_format="NCHW"):
"""Initialize _PoolNd."""
super(_PoolNd, self).__init__()
# 检查'pad_mode'的类型是否为str
validator.check_value_type('pad_mode', pad_mode, [str], self.cls_name)
self.pad_mode = validator.check_string(pad_mode.upper(), ['VALID', 'SAME'], 'pad_mode', self.cls_name)
# 检查输入的data_format是否为NCHW或NHWC
self.format = validator.check_string(data_format, ['NCHW', 'NHWC'], 'format', self.cls_name)
# 如果设备目标不是GPU，而data_format为NHWC
if context.get_context("device_target") != "GPU" and self.format == "NHWC":
# 抛出异常
raise ValueError(f"For '{self.cls_name}, the 'NHWC' format only support in GPU target, but got device "
f"target {context.get_context('device_target')}.")
def _check_int_or_tuple(arg_name, arg_value):
# 检查arg_name和arg_value是否为int和tuple
validator.check_value_type(arg_name, arg_value, [int, tuple], self.cls_name)
error_msg = f"For '{self.cls_name}', the '{arg_name}' should be an positive int number or " \
f"a tuple of two positive int numbers, but got {arg_value}"
# 检查arg_value是否为int
if isinstance(arg_value, int):
# 如果arg_value小于等于0
if arg_value <= 0:
# 抛出异常
raise ValueError(error_msg)
# 如果arg_value的长度等于2
elif len(arg_value) == 2:
# 循环遍历
for item in arg_value:
# 如果都大于0
if isinstance(item, int) and item > 0:
# 则继续
continue
# 否则抛出异常
raise ValueError(error_msg)
else:
# 抛出异常
raise ValueError(error_msg)
# 返回arg_value
return arg_value
# 检查kernel_size和stride的值是否为正整数或者元组
self.kernel_size = _check_int_or_tuple('kernel_size', kernel_size)
self.stride = _check_int_or_tuple('stride', stride)
@ -68,7 +91,8 @@ def _shape_check(in_shape, prim_name=None):
if len(in_shape) != 3:
raise ValueError(f"{msg_prefix} input must has 3 dim, but got {len(in_shape)}")
# 在一个输入Tensor上应用1D最大池化运算，该Tensor可被视为一维平面的组合
# 用于实现2维最大池化操作
class MaxPool2d(_PoolNd):
r"""
2D max pooling operation for temporal data.
@ -130,7 +154,11 @@ class MaxPool2d(_PoolNd):
>>> print(output.shape)
(1, 2, 2, 2)
"""
# 参数
# kernel_size (int): 池化核大小，即池化操作的窗口大小，默认值为 1
# stride (int): 池化步长，即在输入特征图上每次滑动的距离，默认值为 1
# pad_mode (str): 填充模式，表示在边界处如何填充，可选值有 "valid"（有效池化）、"same"（等效池化）和 "pad"（填充池化），默认值为 "valid"
# data_format (str): 数据格式，表示输入特征图的数据 layout。可选值有 "NCHW"（通道在前，深度在后）和 "NHWC"（通道在后，深度在前），默认值为 "NCHW"
def __init__(self, kernel_size=1, stride=1, pad_mode="valid", data_format="NCHW"):
"""Initialize MaxPool2d."""
super(MaxPool2d, self).__init__(kernel_size, stride, pad_mode, data_format)
@ -140,10 +168,14 @@ class MaxPool2d(_PoolNd):
data_format=self.format)
def construct(self, x):
# 将输入特征图x传递给max_pool
out = self.max_pool(x)
# 返回得到out
return out
# 在一个输入Tensor上应用1D LP池化运算，可被视为组成一个1D输入平面
# 用于实现1维最大池化操作
class MaxPool1d(_PoolNd):
r"""
1D max pooling operation for temporal data.
@ -203,30 +235,42 @@ class MaxPool1d(_PoolNd):
def __init__(self, kernel_size=1, stride=1, pad_mode="valid"):
"""Initialize MaxPool1d."""
super(MaxPool1d, self).__init__(kernel_size, stride, pad_mode)
# 检查'kernel_size'是否为int
validator.check_value_type('kernel_size', kernel_size, [int], self.cls_name)
# 检查'stride'是否为int
validator.check_value_type('stride', stride, [int], self.cls_name)
# 检查pad_mode'是否为str
validator.check_value_type('pad_mode', pad_mode, [str], self.cls_name)
self.pad_mode = validator.check_string(pad_mode.upper(), ['VALID', 'SAME'], 'pad_mode', self.cls_name)
# 检查kernel_size和stride是否为正整数
validator.check_int(kernel_size, 1, Rel.GE, "kernel_size", self.cls_name)
validator.check_int(stride, 1, Rel.GE, "stride", self.cls_name)
# 设置卷积核宽高为1
self.kernel_size = (1, kernel_size)
# 设置步长为1
self.stride = (1, stride)
# 创建MaxPool层
self.max_pool = P.MaxPool(kernel_size=self.kernel_size,
strides=self.stride,
pad_mode=self.pad_mode)
# 设置shape来获取张量形状
self.shape = F.shape
# 创建一个ReduceMean层
self.reduce_mean = P.ReduceMean(keep_dims=True)
# 创建一个ExpandDims层
self.expand = P.ExpandDims()
# 创建一个Squeeze层
self.squeeze = P.Squeeze(2)
def construct(self, x):
# 检查形状
_shape_check(self.shape(x), self.cls_name)
x = self.expand(x, 2)
output = self.max_pool(x)
output = self.squeeze(output)
return output
# 在输入Tensor上应用2D平均池化运算，可视为二维输入平面的组合
# 用于实现2维平均池化的类
class AvgPool2d(_PoolNd):
r"""
2D average pooling for temporal data.
@ -289,7 +333,11 @@ class AvgPool2d(_PoolNd):
>>> print(output.shape)
(1, 2, 2, 2)
"""
# 参数
# kernel_size：表示卷积核的宽度和高度，默认为1
# stride：表示卷积核在宽度和高度方向上的步长，默认为1
# pad_mode：表示填充模式，默认为"valid"，表示不填充，还可以选择"same"或"ceil"模式
# data_format：表示输入张量的数据格式，默认为"NCHW"，表示通道在第一个维度，高度和宽度在第二个维度
def __init__(self,
kernel_size=1,
stride=1,
@ -305,7 +353,9 @@ class AvgPool2d(_PoolNd):
def construct(self, x):
return self.avg_pool(x)
# 在一个输入Tensor上应用1D平均池化运算，可被视为组成一个1D输入平面
# 用于实现1维平均池化
class AvgPool1d(_PoolNd):
r"""
1D average pooling for temporal data.
@ -368,6 +418,7 @@ class AvgPool1d(_PoolNd):
stride=1,
pad_mode="valid"):
"""Initialize AvgPool1d."""
# 该模块注释与MaxPool1d(_PoolNd)中init类似
validator.check_value_type('kernel_size', kernel_size, [int], self.cls_name)
validator.check_value_type('stride', stride, [int], self.cls_name)
validator.check_value_type('pad_mode', pad_mode, [str], self.cls_name)
@ -388,14 +439,24 @@ class AvgPool1d(_PoolNd):
def construct(self, x):
x = F.depend(x, _shape_check(self.shape(x), self.cls_name))
# 获取输入x的batch, channel, width
batch, channel, width = self.shape(x)
# 如果输入x的宽度等于卷积核的宽度
if width == self.kernel_size[1]:
# 将输入x的宽度进行平均池化
x = self.reduce_mean(x, 2)
# 如果输入x的宽度减去卷积核的宽度小于步长
elif width - self.kernel_size[1] < self.stride[1]:
# 将输入x的宽度进行切片，取卷积核的宽度
x = self.slice(x, (0, 0, 0), (batch, channel, self.kernel_size[1]))
# 将输入x的宽度进行平均池化
x = self.reduce_mean(x, 2)
else:
# 将输入x的宽度进行扩展
x = self.expand(x, 2)
# 将输入x的宽度进行平均池化
x = self.avg_pool(x)
# 将输入x的宽度进行 squeezing
x = self.squeeze(x)
# 返回输入x
return x

382
mindspore/python/mindspore/nn/layer/quant.py
View File

File diff suppressed because it is too large Load Diff

104
mindspore/python/mindspore/nn/layer/rnn_cells.py
View File

@ -13,63 +13,93 @@
# limitations under the License.
# ============================================================================
'''RNN Cells module, include RNNCell, GRUCell, LSTMCell'''
# 导入math模块，用于处理数学计算
import math
# 导入numpy模块，用于处理数值计算
import numpy as np
# 从mindspore.ops模块中导入P类，用于定义各种计算图操作
import mindspore.ops as P
# 从mindspore.common.dtype模块中导入mstype类，用于处理数据类型
import mindspore.common.dtype as mstype
# 从mindspore模块中导入logger函数，用于记录日志
from mindspore import log as logger
# 从mindspore.common.tensor模块中导入Tensor类，用于表示神经网络中的张量
from mindspore.common.tensor import Tensor
# 从mindspore.common.parameter模块中导入Parameter类，用于表示神经网络中的参数
from mindspore.common.parameter import Parameter
# 从mindspore.common.initializer模块中导入initializer函数和Uniform函数，用于初始化神经网络参数
from mindspore.common.initializer import initializer, Uniform
# 从mindspore.ops.primitive模块中导入constexpr函数，用于在编译时计算常量表达式
from mindspore.ops.primitive import constexpr
# 从mindspore.nn.cell模块中导入Cell类，用于创建自定义的神经网络层
from mindspore.nn.cell import Cell
# 从mindspore._checkparam模块中导入Validator类，用于验证参数的合法性
from mindspore._checkparam import Validator as validator
__all__ = ['LSTMCell', 'GRUCell', 'RNNCell']
@constexpr
# 检查输入_dtype是否有效，根据allow_dtypes列表
# 如果输入_dtype不在allow_dtypes列表中，它会引发一个ValueError，带有自定义错误消息
def _check_input_dtype(input_dtype, param_name, allow_dtypes, cls_name):
validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name)
@constexpr
# 用于检查输入数据是否为张量
def _check_is_tensor(param_name, input_data, cls_name):
"""Internal function, used to check whether the input data is Tensor."""
# # 检查输入数据是否为张量
if input_data is not None and not isinstance(P.typeof(input_data), mstype.tensor_type):
# 抛出异常
raise TypeError(f"For '{cls_name}', the '{param_name}' should be '{mstype.tensor_type}', "
f"but got '{P.typeof(input_data)}'")
@constexpr
# 用于检查输入数据是否为元组
def _check_is_tuple(param_name, input_data, cls_name):
"""Internal function, used to check whether the input data is Tensor."""
# 检查输入数据是否为元组
if input_data is not None and not isinstance(P.typeof(input_data), mstype.Tuple):
# 抛出异常
raise TypeError(f"For '{cls_name}', the '{param_name}' should be '{mstype.Tuple}', "
f"but got '{P.typeof(input_data)}'")
@constexpr
# 用于检查输入数据是否为指定长度的元组
def _check_tuple_length(param_name, input_data, length, cls_name):
"""Internal function, used to check whether the input data is Tensor."""
# 如果输入数据不是指定长度的元组
if input_data is not None and len(input_data) != length:
# 抛出异常
raise TypeError(f"For '{cls_name}', the length of '{param_name}' should be '{length}', "
f"but got '{len(input_data)}'")
@constexpr
# 用于检查两个batch size是否相等
def _check_batch_size_equal(batch_size_x, batch_size_hx, cls_name):
# 如果batch_size_x不等于batch_size_hx
if batch_size_x != batch_size_hx:
# 抛出异常
raise ValueError(f"For '{cls_name}' batch size of x and hx should be equal, but got {batch_size_x} of x "
f"and {batch_size_hx} of hx.")
# 用于检查old MindSpore版本的LSTMCell初始化
# 在old MindSpore版本中，`nn.LSTMCell`是一个完整的LSTM层，而不是单个LSTM单元
# 为了兼容新版本的MindSpore，这个装饰器会在检测到old版本的`nn.LSTMCell`初始化时发出警告，并建议使用`nn.LSTM`代替
# 如果仍然需要使用单层LSTM，请确保使用`nn.LSTM`而不是`nn.LSTMCell`
def _check_lstmcell_init(func):
def wrapper(*args, **kwargs):
logger.warning(f"LSTMCell has been changed from 'single LSTM layer' to 'single LSTM cell', "
f"if you still need use single LSTM layer, please use `nn.LSTM` instead.")
# 如果args长度大于4或者'batch_size' in kwargs、'dropout' in kwargs或'bidirectional' in kwargs
if len(args) > 4 or 'batch_size' in kwargs or \
'dropout' in kwargs or 'bidirectional' in kwargs:
# 抛出异常
raise ValueError(f"The arguments of `nn.LSTMCell` from old MindSpore version(<1.6) are detected, "
f"if you still need use single LSTM layer, please use `nn.LSTM` instead.")
return func(*args, **kwargs)
@ -78,48 +108,57 @@ def _check_lstmcell_init(func):
def _rnn_tanh_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
'''RNN cell function with tanh activation'''
# 如果b_ih为None，则使用inputs和w_ih进行矩阵乘法，否则使用inputs和w_ih加b_ih进行矩阵乘法
if b_ih is None:
igates = P.MatMul(False, True)(inputs, w_ih)
hgates = P.MatMul(False, True)(hidden, w_hh)
else:
igates = P.MatMul(False, True)(inputs, w_ih) + b_ih
hgates = P.MatMul(False, True)(hidden, w_hh) + b_hh
# 返回计算激活函数
return P.Tanh()(igates + hgates)
def _rnn_relu_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
'''RNN cell function with relu activation'''
# 如果b_ih为None，则使用inputs和w_ih进行矩阵乘法，否则使用inputs和w_ih加b_ih进行矩阵乘法
if b_ih is None:
igates = P.MatMul(False, True)(inputs, w_ih)
hgates = P.MatMul(False, True)(hidden, w_hh)
else:
igates = P.MatMul(False, True)(inputs, w_ih) + b_ih
hgates = P.MatMul(False, True)(hidden, w_hh) + b_hh
# 返回计算激活函数
return P.ReLU()(igates + hgates)
# 用于实现LSTM单元
def _lstm_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
'''LSTM cell function'''
hx, cx = hidden
# 如果b_ih为None
if b_ih is None:
gates = P.MatMul(False, True)(inputs, w_ih) + P.MatMul(False, True)(hx, w_hh)
else:
gates = P.MatMul(False, True)(inputs, w_ih) + P.MatMul(False, True)(hx, w_hh) + b_ih + b_hh
# 将gates拆分为输入门、遗忘门、单元门和输出门
ingate, forgetgate, cellgate, outgate = P.Split(1, 4)(gates)
# 计算输入门、遗忘门、单元门和输出门
ingate = P.Sigmoid()(ingate)
forgetgate = P.Sigmoid()(forgetgate)
cellgate = P.Tanh()(cellgate)
outgate = P.Sigmoid()(outgate)
# 计算当前状态
cy = (forgetgate * cx) + (ingate * cellgate)
hy = outgate * P.Tanh()(cy)
# 返回hy和cy
return hy, cy
def _gru_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
'''GRU cell function'''
# 如果b_ih为None
if b_ih is None:
gi = P.MatMul(False, True)(inputs, w_ih)
gh = P.MatMul(False, True)(hidden, w_hh)
@ -129,40 +168,55 @@ def _gru_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
i_r, i_i, i_n = P.Split(1, 3)(gi)
h_r, h_i, h_n = P.Split(1, 3)(gh)
# 计算重置门、输入门和遗忘门
resetgate = P.Sigmoid()(i_r + h_r)
inputgate = P.Sigmoid()(i_i + h_i)
newgate = P.Tanh()(i_n + resetgate * h_n)
# 计算隐藏状态
hy = newgate + inputgate * (hidden - newgate)
return hy
# 用于创建RNN单元的基本类
class RNNCellBase(Cell):
'''Basic class for RNN Cells'''
# input_size：输入数据的特征大小。
# hidden_size：隐藏层的特征大小。
# has_bias：是否使用偏置。
# num_chunks：分块数量，用于实现门控循环单元（GRU）。
def __init__(self, input_size: int, hidden_size: int, has_bias: bool, num_chunks: int):
super().__init__()
# 检查has_bias是否为bool型
validator.check_value_type("has_bias", has_bias, [bool], self.cls_name)
# 检查hidden_size是否为正整数
validator.check_positive_int(hidden_size, "hidden_size", self.cls_name)
# 检查input_size是否为正整数
validator.check_positive_int(input_size, "input_size", self.cls_name)
# 初始化LSTM的参数
self.input_size = input_size
self.hidden_size = hidden_size
self.has_bias = has_bias
# 初始化输入门参数
self.weight_ih = Parameter(Tensor(np.random.randn(num_chunks * hidden_size, input_size).astype(np.float32)))
# 初始化遗忘门参数
self.weight_hh = Parameter(Tensor(np.random.randn(num_chunks * hidden_size, hidden_size).astype(np.float32)))
# 初始化输出门参数
if has_bias:
self.bias_ih = Parameter(Tensor(np.random.randn(num_chunks * hidden_size).astype(np.float32)))
self.bias_hh = Parameter(Tensor(np.random.randn(num_chunks * hidden_size).astype(np.float32)))
else:
self.bias_ih = None
self.bias_hh = None
# 初始化参数
self.reset_parameters()
# 重置网络参数
def reset_parameters(self):
stdv = 1 / math.sqrt(self.hidden_size)
for weight in self.get_parameters():
weight.set_data(initializer(Uniform(stdv), weight.shape))
# 用于实现RNN（递归神经网络）的通用基础类
# RNNCell类主要用于创建和初始化RNNcell，包括输入门、输出门和cell状态。它还包含用于计算隐藏状态和输出值的函数
class RNNCell(RNNCellBase):
r"""
An Elman RNN cell with tanh or ReLU non-linearity.
@ -212,24 +266,35 @@ class RNNCell(RNNCellBase):
def __init__(self, input_size: int, hidden_size: int, has_bias: bool = True, nonlinearity: str = "tanh"):
super().__init__(input_size, hidden_size, has_bias, num_chunks=1)
# 检查"nonlinearity类型是否为str
validator.check_value_type("nonlinearity", nonlinearity, [str], self.cls_name)
# 检查nonlinearity参数是否为有效的字符串
validator.check_string(nonlinearity, self._non_linearity, "nonlinearity", self.cls_name)
self.nonlinearity = nonlinearity
def construct(self, x, hx):
# 检查'x'是否为tenser(张量)
_check_is_tensor('x', x, self.cls_name)
# 检查'hx'是否为tenser
_check_is_tensor('hx', hx, self.cls_name)
# 检查"x"是否为float32或float16
_check_input_dtype(x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name)
# 检查"hx"是否为float32或float16
_check_input_dtype(hx.dtype, "hx", [mstype.float32, mstype.float16], self.cls_name)
# 用于检查两个x和hx的批量大小是否相同
_check_batch_size_equal(x.shape[0], hx.shape[0], self.cls_name)
# 如果nonlinearity为tanh
if self.nonlinearity == "tanh":
# 在tanh非线性情况下，它调用_rnn_tanh_cell函数
ret = _rnn_tanh_cell(x, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
else:
# 在relu非线性情况下，它调用_rnn_relu_cell函数
ret = _rnn_relu_cell(x, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
return ret
# 用于创建和初始化LSTMCell类
# LSTM（Long Short-Term Memory）是一种常用的递归神经网络类型，用于处理序列数据
class LSTMCell(RNNCellBase):
r"""
A LSTM (Long Short-Term Memory) cell.
@ -290,20 +355,33 @@ class LSTMCell(RNNCellBase):
self.support_non_tensor_inputs = True
def construct(self, x, hx):
# 检查'x'是否为tenser
_check_is_tensor('x', x, self.cls_name)
# 检查'hx'是否为tuple
_check_is_tuple('hx', hx, self.cls_name)
# 检查hx是否为一个长度为2的元组
_check_tuple_length('hx', hx, 2, self.cls_name)
# 检查hx[0]是否为tenser
_check_is_tensor('hx[0]', hx[0], self.cls_name)
# 检查hx[1]是否为tenser
_check_is_tensor('hx[1]', hx[1], self.cls_name)
# 检查x类型是否为float32或float16
_check_input_dtype(x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name)
# 检查hx[0]类型是否为float32或float16
_check_input_dtype(hx[0].dtype, "hx[0]", [mstype.float32, mstype.float16], self.cls_name)
# 检查hx[1+]类型是否为float32或float16
_check_input_dtype(hx[1].dtype, "hx[1]", [mstype.float32, mstype.float16], self.cls_name)
# 检查x与hx[0]的批量大小是否相同
# 这步操作对于创建LSTMCell十分重要。因为隐藏状态从批量大小通常与输入值的批量大小相同
_check_batch_size_equal(x.shape[0], hx[0].shape[0], self.cls_name)
# 检查x与hx[1]的批量大小是否相同
_check_batch_size_equal(x.shape[0], hx[1].shape[0], self.cls_name)
return _lstm_cell(x, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
# 用于检查在创建LSTM细胞时传入的输入参数是否有效
def _check_construct_args(self, *inputs, **kwargs):
# 如果长度为4
if len(inputs) == 4:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the number of input args of construct is {len(inputs)}, if you "
f"are using the implementation of `nn.LSTMCell` from old MindSpore version(<1.6), "
f"please notice that: LSTMCell has been changed from 'single LSTM layer' to "
@ -311,7 +389,9 @@ class LSTMCell(RNNCellBase):
f"please use `nn.LSTM` instead.")
return super()._check_construct_args(*inputs, **kwargs)
# 用于创建和初始化GRUCell的类
# GRU（Gated Recurrent Unit）是一种常用的递归神经网络类型，用于处理序列数据
# 与LSTM不同，GRUCell具有更新门和重置门，可以更有效地处理长期依赖关系
class GRUCell(RNNCellBase):
r"""
A GRU(Gated Recurrent Unit) cell.
@ -372,9 +452,15 @@ class GRUCell(RNNCellBase):
super().__init__(input_size, hidden_size, has_bias, num_chunks=3)
def construct(self, x, hx):
# 检查x类型是否为tenser
_check_is_tensor('x', x, self.cls_name)
# 检查hx类型是否为tenser
_check_is_tensor('hx', hx, self.cls_name)
# 检查x类型是否为float32或float16
_check_input_dtype(x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name)
# 检查hx类型是否为float32或float16
_check_input_dtype(hx.dtype, "hx", [mstype.float32, mstype.float16], self.cls_name)
# 用于检查x的批量大小是否与hx的批量大小相等
_check_batch_size_equal(x.shape[0], hx.shape[0], self.cls_name)
# 返回由_gru_cell函数计算得到的输出值
return _gru_cell(x, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)

26
mindspore/python/mindspore/nn/layer/rnn_utils.py
View File

@ -13,15 +13,24 @@
# limitations under the License.
# ============================================================================
'''Utils for RNNs CPU version, like Reverse operators'''
# 导入numpy模块，用于处理数值计算
import numpy as np
# 从mindspore.common.dtype模块中导入mstype类，用于处理数据类型
import mindspore.common.dtype as mstype
# 从mindspore.ops模块中导入P类，用于定义各种计算图操作
import mindspore.ops as P
# 从mindspore.ops.primitive模块中导入constexpr函数，用于在编译时计算常量表达式
from mindspore.ops.primitive import constexpr
# 从mindspore.nn.cell模块中导入Cell类，用于创建自定义的神经网络层
from mindspore.nn.cell import Cell
# 从mindspore.common.tensorTensor类，用于表示神经网络中的张量
from mindspore.common.tensor import Tensor
@constexpr
# 用于生成一个指定范围、步长的等差数列
# 其中三个参数分别是start（起始值），stop（结束值），和 step（步长）
# 当 start 和 stop 相等时，arange 函数返回一个空列表
def arange(start, stop, step):
return Tensor(np.arange(start, stop, step), mstype.int32)
@ -33,14 +42,18 @@ class _Reverse(Cell):
self.dim = dim
def construct(self, input_x):
dim_size = input_x.shape[self.dim]
# 定义一个reversed_indexes函数，用arange函数生成一个等差数列
reversed_indexes = arange(dim_size-1, -1, -1)
output = P.Gather()(input_x, reversed_indexes, self.dim)
# 返回output(反向索引张量)
return output
class _ReverseSequence(Cell):
"""Reverse sequence operator, like ReverseSequenceV2 in mindspore"""
# 初始化函数
def __init__(self, seq_dim, batch_dim=0):
super().__init__()
self.seq_dim = seq_dim
@ -48,40 +61,51 @@ class _ReverseSequence(Cell):
def construct(self, x, seq_lengths):
"""Defines the ReverseSequence operator computation performed."""
# 定义一个函数，用于实现序列的反向解码
batch_size = x.shape[self.batch_dim]
max_seq_len = x.shape[self.seq_dim]
seq_lens_type = seq_lengths.dtype
# 计算序列长度
back = P.Sub()(seq_lengths, P.OnesLike()(seq_lengths))
# 创建batch_idx和forward_idx
batch_idx = self.make_shape((batch_size, max_seq_len), seq_lens_type, 0)
forward_idx = self.make_shape((batch_size, max_seq_len), seq_lens_type, 1)
# 计算back
back = back.view(-1, 1)
reverse_idx = P.Sub()(back, forward_idx)
# 判断reverse_idx是否小于0
condition = P.Less()(reverse_idx, P.ZerosLike()(reverse_idx))
reverse_idx = P.Select()(condition, forward_idx, reverse_idx)
# 将reverse_idx和batch_idx扩展维度
reverse_idx = P.ExpandDims()(reverse_idx, 2)
batch_idx = P.ExpandDims()(batch_idx, 2)
# 如果batch_dim大于seq_dim，则对x，batch_idx，reverse_idx进行转置
if self.batch_dim > self.seq_dim:
batch_idx = P.Transpose()(batch_idx, (1, 0, 2))
reverse_idx = P.Transpose()(reverse_idx, (1, 0, 2))
x = P.Transpose()(x, (1, 0, 2))
# 计算start_indices
start_indices = P.Concat(2)((batch_idx, reverse_idx))
# 计算输出
output = P.GatherNd()(x, start_indices)
# 返回output
return output
@staticmethod
def make_shape(shape, dtype, range_dim):
"""Calculates the shape according by the inputs."""
# 定义一个函数，用于计算输入shape的输出，并使用mstype.float32和dtype作为输出类型
output = P.Ones()(shape, mstype.float32)
output = P.CumSum()(output, range_dim)
output = P.Cast()(output, dtype)
output = output - 1
# 返回output
return output

326
mindspore/python/mindspore/nn/layer/rnns.py
View File

@ -13,106 +13,154 @@
# limitations under the License.
# ============================================================================
'''RNN operators module, include RNN, GRU'''
# 导入数学库，提供了一些基本的数学函数，如 sin、cos 等
import math
# 导入NumPy库，提供了一个强大的数组操作库
import numpy as np
# 导入MindSpore的神经网络模块
import mindspore.nn as nn
# 导入MindSpore的操作模块
import mindspore.ops as P
# 导入MindSpore的上下文模块，用于设置计算环境
import mindspore.context as context
# 导入MindSpore的数据类型模块
import mindspore.common.dtype as mstype
# 从MindSpore的操作模块中导入constexpr函数，用于在编译时计算常量
from mindspore.ops.primitive import constexpr
# 从MindSpore的张量模块中导入Tensor类，用于创建和操作张量
from mindspore.common.tensor import Tensor
# 从MindSpore的参数模块中导入ParameterTuple和Parameter类，用于创建和操作参数
from mindspore.common.parameter import ParameterTuple, Parameter
# 从MindSpore的神经网络模块中导入Cell类，用于创建和操作神经网络层
from mindspore.nn.cell import Cell
# 导入log模块，用于记录日志
from mindspore import log as logger
# 导入Validator类，用于验证参数
from mindspore._checkparam import Validator as validator
# 导入CudnnGRU类，用于实现CudnnGRU操作
from mindspore.ops.operations._rl_inner_ops import CudnnGRU
from .rnn_cells import _rnn_relu_cell, _rnn_tanh_cell, _gru_cell, _lstm_cell
# 导入_rnn_relu_cell、_rnn_tanh_cell、_gru_cell和_lstm_cell等函数，用于实现常见的神经网络层
from .rnn_utils import _Reverse, _ReverseSequence
__all__ = ['LSTM', 'GRU', 'RNN']
@constexpr
# 用于生成一个指定范围、步长和数据类型的等差数列
# 四个参数，start（起始值），stop（结束值），step（步长）和 dtype（数据类型）
def arange(start, stop, step, dtype):
# 返回生成数
return Tensor(np.arange(start, stop, step), dtype)
@constexpr
# 用于初始化循环神经网络的状态
def _init_state(shape, dtype, is_lstm):
# 创建一个名为hx的具有指定形状的零张量
hx = Tensor(np.zeros(shape), dtype)
# 创建一个名为cx的具有指定形状的零张量
cx = Tensor(np.zeros(shape), dtype)
# 如果is_lstm参数为True
if is_lstm:
# 返回hx和cx
return (hx, cx)
# 返回hx
return hx
@constexpr
def _check_input_dtype(input_dtype, param_name, allow_dtypes, cls_name):
# 检查输入张量的数据类型是否在允许的范围内
validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name)
@constexpr
# 用于检查输入张量的数据类型是否相同且在允许的范围内
def _check_input_dtype_same_and_valid(args_name, args_value, valid_values, cls_name):
# args 的字典，其中包含输入参数的名称和值
# 使用列表推导式遍历 args_name 和 args_value 列表，并将每个名称和值作为键值对添加到字典中
args = {args_name[i]: args_value[i] for i in range(len(args_value))}
# 检查输入张量的数据类型是否在允许的范围内
validator.check_types_same_and_valid(args, valid_values, cls_name)
@constexpr
# 检查是否为tensor
def _check_is_tensor(param_name, input_data, cls_name):
"""Internal function, used to check whether the input data is Tensor."""
# 如果input_data不为None并且P.typeof(input_data)不是mstype.tensor_type
if input_data is not None and not isinstance(P.typeof(input_data), mstype.tensor_type):
# 抛出异常
raise TypeError(f"For '{cls_name}', the '{param_name}' should be '{mstype.tensor_type}', "
f"but got '{P.typeof(input_data)}'")
@constexpr
# 检查是否为tuple
def _check_is_tuple(param_name, input_data, cls_name):
"""Internal function, used to check whether the input data is Tensor."""
# 如果input_data不为None并且P.typeof(input_data)不是mstype.tensor_type
if input_data is not None and not isinstance(P.typeof(input_data), mstype.Tuple):
# 抛出异常
raise TypeError(f"For '{cls_name}', the '{param_name}' should be '{mstype.Tuple}', "
f"but got '{P.typeof(input_data)}'")
@constexpr
# 检查tuple长度
def _check_tuple_length(param_name, input_data, length, cls_name):
"""Internal function, used to check whether the input data is Tensor."""
# 如果input_data不为None并且input_data 的长度（len(input_data)不等于给定的length
if input_data is not None and len(input_data) != length:
# 抛出异常
raise TypeError(f"For '{cls_name}', the length of '{param_name}' should be '{length}', "
f"but got '{len(input_data)}'")
@constexpr
# 检查长度和大小
def _check_seq_length_size(batch_size_x, seq_length_size, cls_name):
# 如果batch_size_x不等于seq_length_size
if batch_size_x != seq_length_size:
# 抛出异常
raise ValueError(f"For '{cls_name}' batch size of x and seq_length should be equal, "
f"but got {batch_size_x} of x and {seq_length_size} of seq_length.")
# 接受两个参数lengths 和 maxlen
# lengths 是一个整数列表，表示每个序列的长度；maxlen 是一个整数
def sequence_mask(lengths, maxlen):
"""generate mask matrix by seq_length"""
# 使用arange创建一个从0到maxlen的整数张量
range_vector = arange(0, maxlen, 1, lengths.dtype)
result = range_vector < lengths.view(lengths.shape + (1,))
# 输出转化为int型的mstype
return result.astype(mstype.int32)
# 两个参数，inputs是一个张量，表示输入序列；mask 是一个布尔张量
def select_by_mask(inputs, mask):
"""mask hiddens by mask matrix"""
# 返回选择的元素
return mask.view(mask.shape + (1,)).swapaxes(0, 1) \
.expand_as(inputs).astype(mstype.bool_) * inputs
# 两个参数output是一个张量，表示隐藏层的状态；seq_length是一个整数列表
def get_hidden(output, seq_length):
"""get hidden state by seq_length"""
# 用arange创建一个从0到seq_length.shape[0]的整数张量
batch_index = arange(0, seq_length.shape[0], 1, seq_length.dtype)
# 用concat将seq_length合并堆叠
indices = P.Concat(1)((seq_length.view(-1, 1) - 1, batch_index.view(-1, 1)))
# 返回合并后的结果
return P.GatherNd()(output, indices)
# 用于实现动态循环神经网络（如 LSTM、GRU 等）
class _DynamicRNNBase(Cell):
'''Dynamic RNN module to compute RNN cell by timesteps'''
def __init__(self, mode):
super().__init__()
# 根据mode给cell匹配与mode相同的参数
if mode == "RNN_RELU":
cell = _rnn_relu_cell
elif mode == "RNN_TANH":
@ -122,84 +170,132 @@ class _DynamicRNNBase(Cell):
elif mode == "GRU":
cell = _gru_cell
else:
# 抛出异常
raise ValueError("Unrecognized RNN mode: " + mode)
self.cell = cell
self.is_lstm = mode == "LSTM"
# 参数
# x：张量，表示输入序列
# h_0：张量，表示初始隐藏状态
# w_ih：张量，表示输入到隐藏层的权重
# w_hh：张量，表示隐藏层到隐藏层的权重
# b_ih：张量，表示输入层偏置
# b_hh：张量，表示隐藏层偏置
def recurrent(self, x, h_0, w_ih, w_hh, b_ih, b_hh):
'''recurrent steps without sequence length'''
time_step = x.shape[0]
outputs = []
t = 0
h = h_0
# 当t<time_step，循环计算t与time_step
while t < time_step:
x_t = x[t:t + 1:1]
x_t = P.Squeeze(0)(x_t)
h = self.cell(x_t, h, w_ih, w_hh, b_ih, b_hh)
if self.is_lstm:
# 初始化空列表outputs，用于储存隐藏层状态
outputs.append(h[0])
else:
outputs.append(h)
t += 1
outputs = P.Stack()(outputs)
return outputs, h
# 用于实现动态循环神经网络的递归计算
def variable_recurrent(self, x, h, seq_length, w_ih, w_hh, b_ih, b_hh):
'''recurrent steps with sequence length'''
# 定义一个函数，用于计算时间步长
time_step = x.shape[0]
# 定义一个变量，用于存储隐藏状态
h_t = h
# 判断是否是LSTM模型
if self.is_lstm:
# 获取隐藏状态的尺寸
hidden_size = h[0].shape[-1]
# 定义一个变量，用于存储零矩阵
zero_output = P.ZerosLike()(h_t[0])
# 否则
else:
# 获取隐藏状态的尺寸
hidden_size = h.shape[-1]
# 定义一个变量，用于存储零矩阵
zero_output = P.ZerosLike()(h_t)
# 将seq_length转换为float32类型
seq_length = P.Cast()(seq_length, mstype.float32)
# 将seq_length转换为hidden_size的维度
seq_length = P.BroadcastTo((hidden_size, -1))(seq_length)
# 将seq_length转换为int32类型
seq_length = P.Cast()(seq_length, mstype.int32)
# 将seq_length转换为转置的维度
seq_length = P.Transpose()(seq_length, (1, 0))
outputs = []
state_t = h_t
t = 0
# 当t<time_step，循环计算t与time_step
# 定义一个函数，用于计算LSTM的输出和状态
# 参数：
# x：输入张量，形状为[seq_len, batch_size, input_size]
# init_state：初始状态，形状为[batch_size, num_hidden]
# w_ih：输入门权重，形状为[num_hidden, input_size]
# w_hh：隐藏状态权重，形状为[num_hidden, num_hidden]
# b_ih：输入门偏置，形状为[num_hidden, ]
# b_hh：隐藏状态偏置，形状为[num_hidden, ]
# seq_length：序列长度，形状为[batch_size, ]
# time_step：时间步数
# zero_output：零输出，形状为[batch_size, num_hidden]
while t < time_step:
# 获取输入张量x的t时刻的值
x_t = x[t:t + 1:1]
# 将输入张量x的t时刻的值转换为标量
x_t = P.Squeeze(0)(x_t)
# 计算LSTM的输出和状态
h_t = self.cell(x_t, state_t, w_ih, w_hh, b_ih, b_hh)
# 判断序列长度是否大于t
seq_cond = seq_length > t
# 如果是LSTM，则计算输出和状态
if self.is_lstm:
# 计算输出和状态
state_t_0 = P.Select()(seq_cond, h_t[0], state_t[0])
state_t_1 = P.Select()(seq_cond, h_t[1], state_t[1])
output = P.Select()(seq_cond, h_t[0], zero_output)
state_t = (state_t_0, state_t_1)
# 如果不是LSTM，则计算输出和状态
else:
state_t = P.Select()(seq_cond, h_t, state_t)
output = P.Select()(seq_cond, h_t, zero_output)
# 将输出和状态添加到输出列表中
outputs.append(output)
# 更新t
t += 1
outputs = P.Stack()(outputs)
return outputs, state_t
# 将输出列表中的元素堆叠起来
outputs = P.Stack()(outputs)
# 返回输出和状态
return outputs, state_t
def construct(self, x, h, seq_length, w_ih, w_hh, b_ih, b_hh):
x_dtype = x.dtype
w_ih = w_ih.astype(x_dtype)
w_hh = w_hh.astype(x_dtype)
# 如果b_ih是not None
if b_ih is not None:
b_ih = b_ih.astype(x_dtype)
b_hh = b_hh.astype(x_dtype)
# 如果seq_length是None
if seq_length is None:
# 返回recurrent
return self.recurrent(x, h, w_ih, w_hh, b_ih, b_hh)
# 返回
return self.variable_recurrent(x, h, seq_length, w_ih, w_hh, b_ih, b_hh)
# 用于实现具有 ReLU 激活函数的动态循环神经网络
class _DynamicRNNRelu(_DynamicRNNBase):
'''Dynamic RNN module with Relu activation'''
# 初始化
def __init__(self):
mode = 'RNN_RELU'
super().__init__(mode)
# 用于实现具有 Tanh 激活函数的动态循环神经网络
class _DynamicRNNTanh(_DynamicRNNBase):
'''Dynamic RNN module with Tanh activation'''
@ -207,26 +303,42 @@ class _DynamicRNNTanh(_DynamicRNNBase):
mode = 'RNN_TANH'
super().__init__(mode)
# 用于在CPU和GPU上实现动态 GRU 循环神经网络
class _DynamicGRUCPUGPU(Cell):
'''Dynamic GRU module on CPU and GPU'''
def __init__(self):
super().__init__()
# 创建Concat
self.concat = P.Concat()
# 检查当前输出设备是否为GPU
self.is_gpu = context.get_context("device_target") == "GPU"
def construct(self, x, h_0, seq_length, w_ih, w_hh, b_ih, b_hh):
# 定义一个函数，用于计算GRU的输出和隐藏状态
# 参数：
# w_ih：输入权重，形状为[gate_size, input_size]
# w_hh：输入权重，形状为[gate_size, hidden_size]
# b_ih：输入偏置，形状为[gate_size, 1]
# b_hh：输入偏置，形状为[gate_size, 1]
# x：输入张量值
# h_0：初始隐藏状态
# seq_length：序列长度，如果为None，则使用x的第一个维度
# 获取门张量大小和隐藏层大小
gate_size, input_size = w_ih.shape
hidden_size = gate_size // 3
if self.is_gpu and seq_length is None:
# 如果输入是GPU，并且没有指定序列长度
if b_ih is None:
# 如果没有指定偏置
weights = self.concat((
w_ih.view(-1, 1, 1),
w_hh.view(-1, 1, 1)
))
has_bias = False
else:
# 如果有指定偏置
has_bias = True
weights = self.concat((
w_ih.view(-1, 1, 1),
@ -234,6 +346,7 @@ class _DynamicGRUCPUGPU(Cell):
b_ih.view(-1, 1, 1),
b_hh.view(-1, 1, 1)
))
# 调用CudnnGRU函数
output, h_n, _, _ = CudnnGRU(input_size, hidden_size, 1, has_bias, False, 0.0)(
x,
h_0.view(1, *h_0.shape),
@ -244,50 +357,76 @@ class _DynamicGRUCPUGPU(Cell):
return output, h_n
# 用于在 Ascend 设备上实现动态 GRU 循环神经网络
class _DynamicGRUAscend(Cell):
'''Dynamic GRU module on Ascend'''
# 用于构建动态 GRU 神经网络
def __init__(self):
super().__init__()
self.gru = P.DynamicGRUV2(gate_order='rzh')
self.transpose = P.Transpose()
self.dtype = mstype.float16
# 用于实现 GRU 神经网络的递归计算
# x：输入张量，形状为[seq_length, batch_size, input_size]
# w_ih：输入权重，形状为[num_directions, 4*hidden_size, input_size]
# w_hh：输入权重，形状为[num_directions, 4*hidden_size, hidden_size]
# b_ih：输入偏置，形状为[num_directions, 4*hidden_size]
# b_hh：输入偏置，形状为[num_directions, 4*hidden_size]
# seq_length：序列长度，形状为[batch_size]
# h_0：初始隐藏状态，形状为[num_directions, batch_size, hidden_size]
def construct(self, x, h_0, seq_length, w_ih, w_hh, b_ih, b_hh):
# 如果b_ih为None，则将b_ih设置为全0矩阵，b_hh设置为全0矩阵
if b_ih is None:
b_ih = P.Zeros()(w_ih.shape[0], w_ih.dtype)
b_hh = P.Zeros()(w_ih.shape[0], w_ih.dtype)
# 调用gru函数，计算输出，状态，梯度
outputs, _, _, _, _, _ = self.gru(self.cast(x, self.dtype), \
self.cast(self.transpose(w_ih, (1, 0)), self.dtype), \
self.cast(self.transpose(w_hh, (1, 0)), self.dtype), \
self.cast(b_ih, self.dtype), \
self.cast(b_hh, self.dtype), \
None, self.cast(h_0, self.dtype))
self.cast(self.transpose(w_ih, (1, 0)), self.dtype), \
self.cast(self.transpose(w_hh, (1, 0)), self.dtype), \
self.cast(b_ih, self.dtype), \
self.cast(b_hh, self.dtype), \
None, self.cast(h_0, self.dtype))
# 判断seq_length是否为None
if seq_length is not None:
# 获取outputs中每一行中seq_length个元素
h = get_hidden(outputs, seq_length)
# 根据seq_length生成mask
mask = sequence_mask(seq_length, x.shape[0])
# 根据mask从outputs中取出每一行
outputs = select_by_mask(outputs, mask)
else:
# 如果seq_length为None，则h为outputs中最后一行
h = outputs[-1]
# 返回outputs值
return outputs, h
# 用于在CPU和GPU上实现动态 LSTM 循环神经网络
class _DynamicLSTMCPUGPU(Cell):
'''Dynamic LSTM module on CPU and GPU'''
def __init__(self):
super().__init__()
# 创建Concat
self.concat = P.Concat()
# 如果输出设备为GPU
self.is_gpu = context.get_context("device_target") == "GPU"
# 参数：
# w_ih：输入权重，形状为[gate_size, input_size]
# w_hh：输入权重，形状为[gate_size, hidden_size]
# b_ih：输入偏置，形状为[gate_size, 1]
# b_hh：输入偏置，形状为[gate_size, 1]
# x：输入张量值
# h_0：初始隐藏状态
# seq_length：序列长度，如果为None，则使用x的第一个维度
def construct(self, x, h_0, seq_length, w_ih, w_hh, b_ih, b_hh):
gate_size, input_size = w_ih.shape
hidden_size = gate_size // 4
if seq_length is not None:
output, (h_n, c_n) = _DynamicRNNBase('LSTM')(x, h_0, seq_length, w_ih, w_hh, b_ih, b_hh)
else:
# 判断是否有偏置
if b_ih is None:
# 将权重和偏置拼接
weights = self.concat((
w_ih.view(-1, 1, 1),
w_hh.view(-1, 1, 1)
@ -295,7 +434,9 @@ class _DynamicLSTMCPUGPU(Cell):
has_bias = False
else:
has_bias = True
# 判断是否是GPU
if self.is_gpu:
# 将权重和偏置拼接
weights = self.concat((
w_ih.view(-1, 1, 1),
w_hh.view(-1, 1, 1),
@ -303,107 +444,154 @@ class _DynamicLSTMCPUGPU(Cell):
b_hh.view(-1, 1, 1)
))
else:
# 计算偏置
bias = b_ih + b_hh
# 将权重和偏置拼接
weights = self.concat((
w_ih.view(-1, 1, 1),
w_hh.view(-1, 1, 1),
bias.view(-1, 1, 1)
))
# 计算LSTM
output, h_n, c_n, _, _ = P.LSTM(input_size, hidden_size, 1, has_bias, False, 0.0)(
x,
P.ExpandDims()(h_0[0], 0),
P.ExpandDims()(h_0[1], 0),
weights.astype(x.dtype)
)
# 返回output
return output, (h_n, c_n)
# 用于在Ascend设备上实现动态LSTM循环神经网络
class _DynamicLSTMAscend(Cell):
'''Dynamic LSTM module on Ascend'''
def __init__(self):
super().__init__()
# 定义LSTM网络
self.lstm = P.DynamicRNN()
# 将输入的维度拼接
self.concat_dim1 = P.Concat(axis=1)
# 将输入的维度拼接
self.concat_dim0 = P.Concat(axis=0)
# 转置输入
self.transpose = P.Transpose()
# 将输入的类型转换为float16
self.cast = P.Cast()
# 将输入按照axis=0分割，输出4个
self.split = P.Split(axis=0, output_num=4)
self.dtype = mstype.float16
# 参数：
# w_ih：输入权重，形状为[gate_size, input_size]
# w_hh：输入权重，形状为[gate_size, hidden_size]
# b_ih：输入偏置，形状为[gate_size, 1]
# b_hh：输入偏置，形状为[gate_size, 1]
# x：输入张量值
# h_0：初始隐藏状态
# seq_length：序列长度，如果为None，则使用x的第一个维度
def construct(self, x, h_0, seq_length, w_ih, w_hh, b_ih, b_hh):
# 定义LSTM函数，参数x为输入，w_ih为输入权重，w_hh为隐藏状态权重，b_ih为输入偏置，b_hh为隐藏状态偏置，h_0为初始隐藏状态，seq_length为序列长度
w_ih_i, w_ih_f, w_ih_g, w_ih_o = self.split(w_ih)
w_hh_i, w_hh_f, w_hh_g, w_hh_o = self.split(w_hh)
w_ih = self.concat_dim0((w_ih_i, w_ih_g, w_ih_f, w_ih_o))
w_hh = self.concat_dim0((w_hh_i, w_hh_g, w_hh_f, w_hh_o))
weight = self.concat_dim1((w_ih, w_hh))
# 如果b_ih是None
if b_ih is None:
# 全部设置为0
bias = P.Zeros()(w_ih.shape[0], w_ih.dtype)
# 否则
else:
# 首先对输入参数b_ih和b_hh拆分为四个部分
b_ih_i, b_ih_f, b_ih_g, b_ih_o = self.split(b_ih)
b_hh_i, b_hh_f, b_hh_g, b_hh_o = self.split(b_hh)
# 最后将结果拼接在一起
bias = self.concat_dim0((b_ih_i + b_hh_i, \
b_ih_g + b_hh_g, \
b_ih_f + b_hh_f, \
b_ih_o + b_hh_o))
# 运算outputs输出量
outputs, h, c, _, _, _, _, _ = self.lstm(self.cast(x, self.dtype), \
self.cast(self.transpose(weight, (1, 0)), self.dtype), \
self.cast(bias, self.dtype), None, \
self.cast(P.ExpandDims()(h_0[0], 0), self.dtype), \
self.cast(P.ExpandDims()(h_0[1], 0), self.dtype))
# 如果seq_length是not None
if seq_length is not None:
# 用get_hidden函数对于隐藏状态张量h和工作记忆张量c进行切片
h = get_hidden(h, seq_length)
c = get_hidden(c, seq_length)
# 根据输入长度创建mask
mask = sequence_mask(seq_length, x.shape[0])
# 使用 select_by_mask 函数根据掩码张量 mask 对输出张量进行切片
outputs = select_by_mask(outputs, mask)
else:
h = h[-1]
c = c[-1]
# 返回处理后的输出张量、隐藏状态张量h和工作记忆张量c
return outputs, (h, c)
# 用于是先RNN循环神经网络的基本类
class _RNNBase(Cell):
'''Basic class for RNN operators'''
def __init__(self, mode, input_size, hidden_size, num_layers=1, has_bias=True,
batch_first=False, dropout=0., bidirectional=False):
super().__init__()
# 检查hidden_size是否为正整数
validator.check_positive_int(hidden_size, "hidden_size", self.cls_name)
# 检查input_size是否为正整数
validator.check_positive_int(input_size, "input_size", self.cls_name)
# 检查num_layers是否为正整数
validator.check_positive_int(num_layers, "num_layers", self.cls_name)
# 检查dropout类型是否为float型
validator.check_is_float(dropout, "dropout", self.cls_name)
# 检查has_bias类型是否为bool型
validator.check_value_type("has_bias", has_bias, [bool], self.cls_name)
# 检查batch_first类型是否为bool型
validator.check_value_type("batch_first", batch_first, [bool], self.cls_name)
# 检查bidirectional类型是否为bool型
validator.check_value_type("bidirectional", bidirectional, [bool], self.cls_name)
# 如果dropout的值在0和1中间
if not 0 <= dropout < 1:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'dropout' should be a number in range [0, 1) "
f"representing the probability of an element being zeroed, but got {dropout}.")
# 如果dropout大于0且num_layers为1
if dropout > 0 and num_layers == 1:
# 给出一个警告
logger.warning("dropout option adds dropout after all but last "
"recurrent layer, so non-zero dropout expects "
"num_layers greater than 1, but got dropout={} and "
"num_layers={}".format(dropout, num_layers))
# 如果接受输出装置为Ascend
is_ascend = context.get_context("device_target") == "Ascend"
# 如果mode等于"LSTM"
if mode == "LSTM":
# 计算门*4大小
gate_size = 4 * hidden_size
self.rnn = _DynamicLSTMAscend() if is_ascend else _DynamicLSTMCPUGPU()
# 如果mode等于"GRU"
elif mode == "GRU":
# 如果is_ascend和hidden_size整除16不等于0
if is_ascend and hidden_size % 16 != 0:
# 抛出异常
raise ValueError(f"GRU on ascend do not support hidden size that is not divisible by 16, "
f"but get hidden size {hidden_size}, please reset the argument.")
gate_size = 3 * hidden_size
gate_size = 3 * hidden_size
self.rnn = _DynamicGRUAscend() if is_ascend else _DynamicGRUCPUGPU()
# 如果mode等于"RNN_TANH"
elif mode == "RNN_TANH":
gate_size = hidden_size
# 用_DynamicRNNTanh函数
self.rnn = _DynamicRNNTanh()
# 如果mode等于"RNN_RELU"
elif mode == "RNN_RELU":
gate_size = hidden_size
# 用_DynamicRNNRelu函数
self.rnn = _DynamicRNNRelu()
else:
# 抛出异常
raise ValueError(f"For '{self.cls_name}', the 'mode' should be in ['RNN_RELU', 'RNN_TANH', 'LSTM', 'GRU'], "
f"but got {mode}.")
@ -413,53 +601,81 @@ class _RNNBase(Cell):
else:
self.reverse = P.ReverseV2([0])
self.reverse_sequence = P.ReverseSequence(0, 1)
# 初始化隐藏层大小
self.hidden_size = hidden_size
# 初始化batch_first
self.batch_first = batch_first
# 初始化层数
self.num_layers = num_layers
# 初始化dropout
self.dropout = dropout
# 初始化dropout操作
self.dropout_op = nn.Dropout(float(1 - dropout))
# 初始化双向性
self.bidirectional = bidirectional
# 初始化是否有偏置
self.has_bias = has_bias
# 初始化方向数
num_directions = 2 if bidirectional else 1
# 初始化是否为LSTM
self.is_lstm = mode == "LSTM"
# 用于存储输入门权重
self.w_ih_list = []
# 用于存储隐藏门权重
self.w_hh_list = []
# 用于存储输入门偏置
self.b_ih_list = []
# 用于存储隐藏门偏置
self.b_hh_list = []
stdv = 1 / math.sqrt(self.hidden_size)
# 遍历隐藏层数量（num_layers）和方向数量（num_directions）
for layer in range(num_layers):
for direction in range(num_directions):
# 获取当前层的输入大小（input_size if layer == 0 else hidden_size * num_directions）
layer_input_size = input_size if layer == 0 else hidden_size * num_directions
# 添加后缀（'_reverse' if direction == 1 else '')
suffix = '_reverse' if direction == 1 else ''
# 创建当前层的输入权重
self.w_ih_list.append(Parameter(
Tensor(np.random.uniform(-stdv, stdv, (gate_size, layer_input_size)).astype(np.float32)),
name='weight_ih_l{}{}'.format(layer, suffix)))
# 创建当前层的隐藏权重
self.w_hh_list.append(Parameter(
Tensor(np.random.uniform(-stdv, stdv, (gate_size, hidden_size)).astype(np.float32)),
name='weight_hh_l{}{}'.format(layer, suffix)))
# 如果存在偏置项
if has_bias:
# 创建当前层的输入偏置
self.b_ih_list.append(Parameter(
Tensor(np.random.uniform(-stdv, stdv, (gate_size)).astype(np.float32)),
name='bias_ih_l{}{}'.format(layer, suffix)))
# 创建当前层的隐藏偏置
self.b_hh_list.append(Parameter(
Tensor(np.random.uniform(-stdv, stdv, (gate_size)).astype(np.float32)),
name='bias_hh_l{}{}'.format(layer, suffix)))
# 将列表转换为ParameterTuple
self.w_ih_list = ParameterTuple(self.w_ih_list)
self.w_hh_list = ParameterTuple(self.w_hh_list)
self.b_ih_list = ParameterTuple(self.b_ih_list)
self.b_hh_list = ParameterTuple(self.b_hh_list)
# 用于实现一个堆叠的双向LSTM（双向Long Short-Term Memory）网络
def _stacked_bi_dynamic_rnn(self, x, h, seq_length):
# 参数
# self：指向包含此函数的类的实例。
# x：输入张量，形状为[batch_size, max_time, input_size]。
# h：初始隐藏状态张量，形状为[num_layers * 2, batch_size, hidden_size]。
# seq_length：一个整数张量，表示每个序列的实际长度。形状为[batch_size]
"""stacked bidirectional dynamic_rnn"""
pre_layer = x
h_n = ()
c_n = ()
output = 0
# 从0开始遍历LSTM层的数量
for i in range(self.num_layers):
offset = i * 2
if self.has_bias:
# 获取LSTM层的权重矩阵、隐藏状态矩阵和偏置项，分别赋值给w_f_ih、w_f_hh、b_f_ih和b_f_hh
w_f_ih, w_f_hh, b_f_ih, b_f_hh = \
self.w_ih_list[offset], self.w_hh_list[offset], \
self.b_ih_list[offset], self.b_hh_list[offset]
@ -501,62 +717,88 @@ class _RNNBase(Cell):
return output, (h_n.view(h[0].shape), c_n.view(h[1].shape))
h_n = P.Concat(0)(h_n)
return output, h_n.view(h.shape)
# 实现多层RNN的计算，其中每一层都使用动态规划算法来计算隐藏层的状态
def _stacked_dynamic_rnn(self, x, h, seq_length):
"""stacked mutil_layer dynamic_rnn"""
pre_layer = x
h_n = ()
c_n = ()
output = 0
# 从0开始循环遍历多层RNN的每一层
# 遍历每一层
for i in range(self.num_layers):
# 如果有偏置
if self.has_bias:
# 获取权重和偏置
w_ih, w_hh, b_ih, b_hh = self.w_ih_list[i], self.w_hh_list[i], self.b_ih_list[i], self.b_hh_list[i]
else:
# 获取权重
w_ih, w_hh = self.w_ih_list[i], self.w_hh_list[i]
# 偏置为None
b_ih, b_hh = None, None
# 如果是LSTM
if self.is_lstm:
# 获取输入的h和c
h_i = (h[0][i], h[1][i])
else:
# 获取输入的h
h_i = h[i]
# 运行RNN
output, h_t = self.rnn(pre_layer, h_i, seq_length, w_ih, w_hh, b_ih, b_hh)
# 如果是LSTM，将输出和h_t的第一个元素和c_t的第一个元素拼接
pre_layer = self.dropout_op(output) if (self.dropout != 0 and i < self.num_layers - 1) else output
if self.is_lstm:
h_n += (h_t[0],)
c_n += (h_t[1],)
else:
h_n += (h_t,)
if self.is_lstm:
h_n = P.Concat(0)(h_n)
c_n = P.Concat(0)(c_n)
h_n = h_n.view(h[0].shape)
c_n = c_n.view(h[1].shape)
# 如果是LSTM，将h_n和c_n拼接，并转换为h的形状
if self.is_lstm:
h_n = P.Concat(0)(h_n)
c_n = P.Concat(0)(c_n)
h_n = h_n.view(h[0].shape)
c_n = c_n.view(h[1].shape)
return output, (h_n.view(h[0].shape), c_n.view(h[1].shape))
# 将h_n拼接，并转换为h的形状
h_n = P.Concat(0)(h_n)
return output, h_n.view(h.shape)
def construct(self, x, hx=None, seq_length=None):
'''Defines the RNN like operators performed'''
# 获取输入数据batch的大小
max_batch_size = x.shape[0] if self.batch_first else x.shape[1]
# 如果使用双向RNN，那么每一层的隐藏层状态需要保留两个方向（前向和后向）的隐藏层状态
num_directions = 2 if self.bidirectional else 1
# 检查x是否为tensor
_check_is_tensor("x", x, self.cls_name)
x_dtype = x.dtype
if hx is not None:
if not self.is_lstm:
# 检查h的类型是否为tensor
_check_is_tensor("h", hx, self.cls_name)
# 检查输入数据的类型是否相同且有效
_check_input_dtype_same_and_valid(['x', 'hx'], [x_dtype, hx.dtype], \
[mstype.float32, mstype.float16], self.cls_name)
else:
# 检查hx的类型是否为tuple
_check_is_tuple('hx', hx, self.cls_name)
# 检查hx的长度是否为2
_check_tuple_length('hx', hx, 2, self.cls_name)
# 检查hx[0]的类型是否为tensor
_check_is_tensor('hx[0]', hx[0], self.cls_name)
# 检查hx[1]的类型是否为tensor
_check_is_tensor('hx[1]', hx[1], self.cls_name)
# 检查输入数据的类型是否相同且有效，类型是否为float32或float16
_check_input_dtype_same_and_valid(['x', 'hx[0]', 'hx[1]'], [x_dtype, hx[0].dtype, hx[1].dtype], \
[mstype.float32, mstype.float16], self.cls_name)
else:
# 使用_init_state函数根据给定的隐藏层状态类型、批量大小和隐藏层大小初始化隐藏层状态
hx = _init_state((self.num_layers * num_directions, max_batch_size, self.hidden_size), \
x_dtype, self.is_lstm)
if seq_length is not None:
# 检查seq_length的类型是否为int32或int64
_check_input_dtype(seq_length.dtype, "seq_length", [mstype.int32, mstype.int64], self.cls_name)
# 检查seq_length的形状是否与max_batch_size相匹配
_check_seq_length_size(max_batch_size, seq_length.shape[0], self.cls_name)
if self.batch_first:
x = P.Transpose()(x, (1, 0, 2))
@ -570,7 +812,7 @@ class _RNNBase(Cell):
return x_n.astype(x_dtype), hx_n.astype(x_dtype)
return x_n.astype(x_dtype), (hx_n[0].astype(x_dtype), hx_n[1].astype(x_dtype))
# 用于实现简单循环神经网络（RNN）神经网络的类
class RNN(_RNNBase):
r"""
Stacked Elman RNN layers.
@ -638,6 +880,7 @@ class RNN(_RNNBase):
"""
def __init__(self, *args, **kwargs):
# 判断kwargs中是否包含nonlinearity参数，如果包含则根据nonlinearity参数的值设置mode的值，并从kwargs中删除nonlinearity参数
if 'nonlinearity' in kwargs:
if kwargs['nonlinearity'] == 'tanh':
mode = 'RNN_TANH'
@ -650,9 +893,10 @@ class RNN(_RNNBase):
else:
mode = 'RNN_TANH'
# 调用父类的构造函数，传入mode和args,kwargs参数
super(RNN, self).__init__(mode, *args, **kwargs)
# 用于实现门控循环神经网络（GRU）神经网络的类
class GRU(_RNNBase):
r"""
Stacked GRU (Gated Recurrent Unit) layers.
@ -739,7 +983,7 @@ class GRU(_RNNBase):
mode = 'GRU'
super(GRU, self).__init__(mode, *args, **kwargs)
# 用于实现长短期记忆（LSTM）神经网络的类
class LSTM(_RNNBase):
r"""
Stacked LSTM (Long Short-Term Memory) layers.

577
mindspore/python/mindspore/nn/layer/thor_layer.py
View File

File diff suppressed because it is too large Load Diff

58
mindspore/python/mindspore/nn/layer/timedistributed.py
View File

@ -13,9 +13,11 @@
# limitations under the License.
# ============================================================================
"""Time Distributed."""
# 从mindspore.ops.primitive中导入了一些常量、原始操作和辅助函数
from mindspore.ops.primitive import constexpr, Primitive
# 从mindspore.ops中导入了一些操作，如Reshape、Transpose、Stack和Unstack
from mindspore.ops import Reshape, Transpose, Stack, Unstack
# 从mindspore.common中导入了一个Tensor类
from mindspore.common import Tensor
from mindspore._checkparam import Validator
from ..cell import Cell
@ -24,9 +26,12 @@ __all__ = ['TimeDistributed']
@constexpr
# 用于检查reshape_pos是否是一个有效的reshape位置
def _check_reshape_pos(reshape_pos, inputs_shape, outputs_shape, prim_name=None):
# 定义了一个变量msg_prefix，用于在生成错误消息时使用
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
if reshape_pos >= len(outputs_shape) or inputs_shape[reshape_pos] != outputs_shape[reshape_pos]:
# 抛出异常
raise ValueError(f"{msg_prefix} 'reshape_with_axis' is invalid in the input and output. "
f"The 'reshape_pos' should be less than the length of 'outputs_shape', and the "
f"'inputs_shape[reshape_pos]' should be equal to 'outputs_shape[reshape_pos]', but got "
@ -36,33 +41,48 @@ def _check_reshape_pos(reshape_pos, inputs_shape, outputs_shape, prim_name=None)
@constexpr
def _check_expand_dims_axis(time_axis, ndim, prim_name=None):
# 定义了一个变量msg_prefix，用于在生成错误消息时使用
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果time_axis大于ndim
if time_axis > ndim:
# 抛出异常
raise ValueError(f"{msg_prefix} value of 'time_axis' should be in range of [{-ndim - 1}, {ndim}], "
f"but got {time_axis}.")
@constexpr
# 用于生成一个元组，表示对某个张量的轴进行重新排序
def _generate_perm(axis_a, axis_b, length):
# 创建了一个包含从0到length - 1的整数序列的元组
perm = tuple(range(length))
# 根据axis_a和axis_b的值是否小于等于交换它们的位置
axis_a, axis_b = (axis_a, axis_b) if axis_a < axis_b else (axis_b, axis_a)
# 返回并计算
return perm[:axis_a] + (perm[axis_b],) + perm[axis_a: axis_b] + perm[axis_b + 1:]
@constexpr
# 用于检查输入数据是否符合预期的格式
def _check_data(flag, prim_name=None):
# 定义了一个变量msg_prefix，用于在生成错误消息时使用
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果是not
if not flag:
# 返回异常
raise TypeError(f"{msg_prefix} inputs and outputs should be a Tensor.")
@constexpr
# 检查输入张量的形状是否符合预期的格式
def _check_inputs_dim(shape, prim_name=None):
# 定义了一个变量msg_prefix，用于在生成错误消息时使用
msg_prefix = f"For '{prim_name}', the" if prim_name else "The"
# 如果长度小于3
if len(shape) < 3:
# 抛出异常
raise ValueError(f"{msg_prefix} inputs shape should be at least 3D, but got {len(shape)}.")
# 用于实现时间分布模型的神经网络层
class TimeDistributed(Cell):
r"""
The time distributed layer.
@ -103,41 +123,70 @@ class TimeDistributed(Cell):
def __init__(self, layer, time_axis, reshape_with_axis=None):
"""Initialize TimeDistributed."""
# 如果layer不是一个Primitive对象
if not isinstance(layer, (Cell, Primitive)):
raise TypeError(f"For '{self.cls_name}', the 'layer' should be Cell or Primitive instance, "
f"but got type: {type(layer).__name__}.")
super(TimeDistributed, self).__init__()
# 检查time_axis类型是否为int
Validator.check_is_int(time_axis, "time_axis", self.cls_name)
# 如果reshape_with_axis为None
if reshape_with_axis is not None:
# 检查reshape_with_axis的类型是否为int型
Validator.check_is_int(reshape_with_axis, "reshape_with_axis", self.cls_name)
# 将layer的值赋值给self.layer
self.layer = layer
# 将time_axis的值赋值给self.time_axis
self.time_axis = time_axis
# reshape_with_axis的值赋值给self.reshape_with_axis
self.reshape_with_axis = reshape_with_axis
# 创建Transpose对象
self.transpose = Transpose()
# 创建Reshape
self.reshape = Reshape()
# 用于初始化层的状态
def construct(self, inputs):
# 检查输入数据是否是一个Tensor对象
_check_data(isinstance(inputs, Tensor), self.cls_name)
# 检查输入张量的形状是否符合时间分布模型的要求
_check_inputs_dim(inputs.shape, self.cls_name)
# 计算时间轴线位置
time_axis = self.time_axis % len(inputs.shape)
# 如果reshape_with_axis是not None
if self.reshape_with_axis is not None:
# 计算余数
reshape_with_axis = self.reshape_with_axis % len(inputs.shape)
# 将inputs.shape张量的值赋给inputs_shape变量
inputs_shape = inputs.shape
# 检查reshape_with_axis是否等于inputs_shape张量的长度减1
# 如果是，那么就将time_axis_new设置为len(inputs_shape) - 2
time_axis_new = len(inputs_shape) - 2 if reshape_with_axis == len(inputs_shape) - 1 \
else (reshape_with_axis + 1 if time_axis > reshape_with_axis else
reshape_with_axis - 1)
# 计算reshape_pos的值，它等于time_axis_new如果time_axis_new小于reshape_with_axis
reshape_pos = time_axis_new if time_axis_new < reshape_with_axis else reshape_with_axis
# 生曾一个排列perm
perm = _generate_perm(time_axis_new, time_axis, len(inputs_shape))
# 将inputs张量转换
inputs = self.transpose(inputs, perm)
# 变为新shape并且赋值
inputs_shape_new = inputs.shape
# 使用inputs_shape_new[: reshape_pos] + (-1,) + inputs_shape_new[reshape_pos + 2:]作为参数调用self.reshape函数，将inputs张量重塑为新的形状
inputs = self.reshape(inputs, inputs_shape_new[: reshape_pos] + (-1,) + inputs_shape_new[reshape_pos + 2:])
# 得到outputs
outputs = self.layer(inputs)
# 检查outputs的类型是否为tensor
_check_data(isinstance(outputs, Tensor), self.cls_name)
# 检查reshape_pos的值是否在允许的范围内
_check_reshape_pos(reshape_pos, inputs.shape, outputs.shape, self.cls_name)
# 计算outputs张量的新的形状
outputs_shape_new = outputs.shape[:reshape_pos] + inputs_shape_new[reshape_pos: reshape_pos + 2]
# 如果reshape_pos+1小于outputs的长度
if reshape_pos + 1 < len(outputs.shape):
# 将outputs张量中从reshape_pos + 1开始的所有轴的形状添加到outputs_shape_new中
outputs_shape_new += outputs.shape[reshape_pos + 1:]
outputs_shape_new = (-1,) + outputs_shape_new[1:]
# 返回新的outputs和outputs_shape_new
return self.reshape(outputs, outputs_shape_new)
unstack = Unstack(time_axis)
@ -145,8 +194,11 @@ class TimeDistributed(Cell):
y = ()
for item in inputs:
outputs = self.layer(item)
# 检查outputs的类型是否为tensor
_check_data(isinstance(outputs, Tensor), self.cls_name)
# 检查time_axis是否在允许的范围内
_check_expand_dims_axis(time_axis, outputs.ndim, self.cls_name)
y += (outputs,)
y = Stack(time_axis)(y)
# 返回y
return y

237
mindspore/python/mindspore/nn/learning_rate_schedule.py
View File

@ -13,17 +13,22 @@
# limitations under the License.
# ============================================================================
"""Learning rate schedule."""
# 本文件为动态学习率的定义
# 导入数学模块
import math
# 导入数据类型模块
from ..common import dtype as mstype
# 导入ops算子
from ..ops import operations as P
# 导入神经网络基本单元Cell
from .cell import Cell
# 导入检查模块
from .._checkparam import Validator as validator
class LearningRateSchedule(Cell):
"""Basic class of learning rate schedule."""
# LearningRateSchedule的基本类，所有动态学习率均继承自此类
def __init__(self):
super(LearningRateSchedule, self).__init__()
@ -37,15 +42,21 @@ class LearningRateSchedule(Cell):
The output must be a Tensor of scalar.
Inputs:
- **global_step** (Tensor) - The current step number.
Inputs:
Tensor. Learning rate at current step with shape :math:`()`.
Tensor. The current step number.
"""
raise NotImplementedError
def _check_inputs(learning_rate, decay_rate, decay_steps, is_stair, cls_name):
'''
检查输入参数是否合法
:param learning_rate: 学习率
:param decay_rate: 平滑系数
:param decay_steps: 时间步长
:param is_stair: 是否平滑
:param cls_name: 名称
:return:
'''
validator.check_positive_int(decay_steps, 'decay_steps', cls_name)
validator.check_positive_float(learning_rate, 'learning_rate', cls_name)
validator.check_is_float(learning_rate, 'learning_rate', cls_name)
@ -55,13 +66,14 @@ def _check_inputs(learning_rate, decay_rate, decay_steps, is_stair, cls_name):
class ExponentialDecayLR(LearningRateSchedule):
# 基于指数衰减函数计算学习率。
r"""
Calculates learning rate based on exponential decay function.
Calculates learning rate base on exponential decay function.
For current step, the formula of computing decayed learning rate is:
For the i-th step, the formula of computing decayed_learning_rate[i] is:
.. math::
decayed\_learning\_rate = learning\_rate * decay\_rate^{p}
decayed\_learning\_rate[i] = learning\_rate * decay\_rate^{p}
Where :
@ -76,14 +88,14 @@ class ExponentialDecayLR(LearningRateSchedule):
Args:
learning_rate (float): The initial value of learning rate.
decay_rate (float): The decay rate.
decay_steps (int): Number of steps to decay over.
decay_steps (int): A value used to calculate decayed learning rate.
is_stair (bool): If true, learning rate is decayed once every `decay_steps` time. Default: False.
Inputs:
- **global_step** (Tensor) - The current step number.
Tensor. The current step number.
Outputs:
Tensor. The learning rate value for the current step with shape :math:`()`.
Tensor. The learning rate value for the current step.
Raises:
TypeError: If `learning_rate` or `decay_rate` is not a float.
@ -95,19 +107,23 @@ class ExponentialDecayLR(LearningRateSchedule):
``Ascend`` ``GPU`` ``CPU``
Examples:
>>> import mindspore
>>> from mindspore import Tensor, nn
>>>
>>> learning_rate = 0.1
>>> decay_rate = 0.9
>>> decay_steps = 4
>>> global_step = Tensor(2, mindspore.int32)
>>> global_step = Tensor(2, mstype.int32)
>>> exponential_decay_lr = nn.ExponentialDecayLR(learning_rate, decay_rate, decay_steps)
>>> result = exponential_decay_lr(global_step)
>>> print(result)
0.09486833
"""
def __init__(self, learning_rate, decay_rate, decay_steps, is_stair=False):
'''
初始化ExponentialDecayLR类，并使用_check_inputs方法检查参数
:param learning_rate: 学习率
:param decay_rate: 平滑系数
:param decay_steps: 时间步长
:param is_stair: 是否平滑
'''
super(ExponentialDecayLR, self).__init__()
_check_inputs(learning_rate, decay_rate, decay_steps, is_stair, self.cls_name)
self.learning_rate = learning_rate
@ -118,20 +134,28 @@ class ExponentialDecayLR(LearningRateSchedule):
self.cast = P.Cast()
def construct(self, global_step):
'''
构建ExponentialDecayLR类
:param global_step: 步骤
:return:
'''
p = self.cast(global_step, mstype.float32) / self.decay_steps
# 如果是在梯度下降时，则使用Floor函数
if self.is_stair:
p = P.Floor()(p)
# 返回指数衰减学习率
return self.learning_rate * self.pow(self.decay_rate, p)
class NaturalExpDecayLR(LearningRateSchedule):
# 基于自然指数衰减函数计算学习率。
r"""
Calculates learning rate base on natural exponential decay function.
For current step, the formula of computing decayed learning rate is:
For the i-th step, the formula of computing decayed_learning_rate[i] is:
.. math::
decayed\_learning\_rate= learning\_rate * e^{-decay\_rate * p}
decayed\_learning\_rate[i] = learning\_rate * e^{-decay\_rate * p}
Where :
@ -146,14 +170,14 @@ class NaturalExpDecayLR(LearningRateSchedule):
Args:
learning_rate (float): The initial value of learning rate.
decay_rate (float): The decay rate.
decay_steps (int): Number of steps to decay over.
decay_steps (int): A value used to calculate decayed learning rate.
is_stair (bool): If true, learning rate is decayed once every `decay_steps` time. Default: False.
Inputs:
- **global_step** (Tensor) - The current step number.
Tensor. The current step number.
Outputs:
Tensor. The learning rate value for the current step with shape :math:`()`.
Tensor. The learning rate value for the current step.
Raises:
TypeError: If `learning_rate` or `decay_rate` is not a float.
@ -165,19 +189,23 @@ class NaturalExpDecayLR(LearningRateSchedule):
``Ascend`` ``GPU`` ``CPU``
Examples:
>>> import mindspore
>>> from mindspore import Tensor, nn
>>>
>>> learning_rate = 0.1
>>> decay_rate = 0.9
>>> decay_steps = 4
>>> global_step = Tensor(2, mindspore.int32)
>>> global_step = Tensor(2, mstype.int32)
>>> natural_exp_decay_lr = nn.NaturalExpDecayLR(learning_rate, decay_rate, decay_steps, True)
>>> result = natural_exp_decay_lr(global_step)
>>> print(result)
0.1
"""
def __init__(self, learning_rate, decay_rate, decay_steps, is_stair=False):
'''
初始化一个NaturalExpDecayLR类，并使用_check_inputs方法检查参数
:param learning_rate: 学习率
:param decay_rate: 分段衰减率
:param decay_steps: 时间步长
:param is_stair: 是否变为折线图
'''
super(NaturalExpDecayLR, self).__init__()
_check_inputs(learning_rate, decay_rate, decay_steps, is_stair, self.cls_name)
self.learning_rate = learning_rate
@ -189,20 +217,28 @@ class NaturalExpDecayLR(LearningRateSchedule):
self.cast = P.Cast()
def construct(self, global_step):
'''
构建NaturalExpDecayLR类
:param global_step: 步骤
:return:
'''
p = self.cast(global_step, mstype.float32)
# 如果是在梯度下降模式下，则将p乘以decay_steps，并将p除以decay_steps
if self.is_stair:
p = P.FloorDiv()(p, self.decay_steps) * self.decay_steps
# 返回learning_rate乘以pow(math_e, -decay_rate * p)
return self.learning_rate * self.pow(self.math_e, -self.decay_rate * p)
class InverseDecayLR(LearningRateSchedule):
# 基于逆时衰减函数计算学习率。
r"""
Calculates learning rate base on inverse-time decay function.
For current step, the formula of computing decayed learning rate is:
For the i-th step, the formula of computing decayed_learning_rate[i] is:
.. math::
decayed\_learning\_rate = learning\_rate / (1 + decay\_rate * p)
decayed\_learning\_rate[i] = learning\_rate / (1 + decay\_rate * p)
Where :
@ -217,14 +253,14 @@ class InverseDecayLR(LearningRateSchedule):
Args:
learning_rate (float): The initial value of learning rate.
decay_rate (float): The decay rate.
decay_steps (int): Number of steps to decay over.
decay_steps (int): A value used to calculate decayed learning rate.
is_stair (bool): If true, learning rate decay once every `decay_steps` times. Default: False.
Inputs:
- **global_step** (Tensor) - The current step number.
Tensor. The current step number.
Outputs:
Tensor. The learning rate value for the current step with shape :math:`()`.
Tensor. The learning rate value for the current step.
Raises:
TypeError: If `learning_rate` or `decay_rate` is not a float.
@ -236,19 +272,23 @@ class InverseDecayLR(LearningRateSchedule):
``Ascend`` ``GPU`` ``CPU``
Examples:
>>> import mindspore
>>> from mindspore import Tensor, nn
>>>
>>> learning_rate = 0.1
>>> decay_rate = 0.9
>>> decay_steps = 4
>>> global_step = Tensor(2, mindspore.int32)
>>> global_step = Tensor(2, mstype.int32)
>>> inverse_decay_lr = nn.InverseDecayLR(learning_rate, decay_rate, decay_steps, True)
>>> result = inverse_decay_lr(global_step)
>>> print(result)
0.1
"""
def __init__(self, learning_rate, decay_rate, decay_steps, is_stair=False):
'''
初始化InverseDecayLR，并使用_check_inputs方法检查参数
:param learning_rate: 学习率
:param decay_rate: 平滑系数
:param decay_steps: 时间步长
:param is_stair: 是否平滑
'''
super(InverseDecayLR, self).__init__()
_check_inputs(learning_rate, decay_rate, decay_steps, is_stair, self.cls_name)
self.learning_rate = learning_rate
@ -258,33 +298,41 @@ class InverseDecayLR(LearningRateSchedule):
self.cast = P.Cast()
def construct(self, global_step):
'''
计算每次迭代的学习率
:param global_step: 所有epoch的迭代次数
:return: 每次迭代的学习率
'''
p = self.cast(global_step, mstype.float32) / self.decay_steps
# 如果是在梯度下降过程中，则将p转换为浮点数
if self.is_stair:
p = P.Floor()(p)
# 返回学习率的除法结果，使用指数衰减
return self.learning_rate / (1 + self.decay_rate * p)
class CosineDecayLR(LearningRateSchedule):
# 基于余弦衰减函数计算学习率。
r"""
Calculates learning rate based on cosine decay function.
Calculates learning rate base on cosine decay function.
For current step, the formula of computing decayed learning rate is:
For the i-th step, the formula of computing decayed_learning_rate[i] is:
.. math::
decayed\_learning\_rate = min\_lr + 0.5 * (max\_lr - min\_lr) *
decayed\_learning\_rate[i] = min\_learning\_rate + 0.5 * (max\_learning\_rate - min\_learning\_rate) *
(1 + cos(\frac{current\_step}{decay\_steps}\pi))
Args:
min_lr (float): The minimum value of learning rate.
max_lr (float): The maximum value of learning rate.
decay_steps (int): Number of steps to decay over.
decay_steps (int): A value used to calculate decayed learning rate.
Inputs:
- **global_step** (Tensor) - The current step number.
Tensor. The current step number.
Outputs:
Tensor. The learning rate value for the current step with shape :math:`()`.
Tensor. The learning rate value for the current step.
Raises:
TypeError: If `min_lr` or `max_lr` is not a float.
@ -296,52 +344,67 @@ class CosineDecayLR(LearningRateSchedule):
``Ascend`` ``GPU``
Examples:
>>> import mindspore
>>> from mindspore import Tensor, nn
>>>
>>> min_lr = 0.01
>>> max_lr = 0.1
>>> decay_steps = 4
>>> global_steps = Tensor(2, mindspore.int32)
>>> global_steps = Tensor(2, mstype.int32)
>>> cosine_decay_lr = nn.CosineDecayLR(min_lr, max_lr, decay_steps)
>>> result = cosine_decay_lr(global_steps)
>>> print(result)
0.055
"""
def __init__(self, min_lr, max_lr, decay_steps):
'''
初始化CosineDecayLR类，并检查参数
:param min_lr: 最小学习率
:param max_lr: 最大学习率
:param decay_steps: 时间步数
'''
super(CosineDecayLR, self).__init__()
if not isinstance(min_lr, float):
raise TypeError("For 'CosineDecayLR', the argument 'min_lr' must be type of float, "
"but got 'min_lr' type: {}.".format(type(min_lr)))
raise TypeError("min_lr must be float.")
validator.check_non_negative_float(min_lr, "min_lr", self.cls_name)
validator.check_positive_float(max_lr, 'max_lr', self.cls_name)
validator.check_is_float(max_lr, 'max_lr', self.cls_name)
validator.check_positive_float(max_lr,'max_lr', self.cls_name)
validator.check_is_float(max_lr,'max_lr', self.cls_name)
validator.check_positive_int(decay_steps, "decay_steps", self.cls_name)
if min_lr >= max_lr:
raise ValueError("For 'CosineDecayLR', the 'max_lr' should be greater than the 'min_lr', "
"but got 'max_lr' value: {}, 'min_lr' value: {}.".format(max_lr, min_lr))
raise ValueError('`max_lr` should be greater than `min_lr`.')
# 将min_lr和max_lr赋值给变量min_lr和max_lr
self.min_lr = min_lr
self.max_lr = max_lr
# 将decay_steps赋值给变量decay_steps
self.decay_steps = decay_steps
# 将math.pi赋值给变量math_pi
self.math_pi = math.pi
# 将delta赋值给变量delta
self.delta = 0.5 * (max_lr - min_lr)
# 创建一个Cos函数
self.cos = P.Cos()
# 创建一个Minimum函数
self.min = P.Minimum()
# 创建一个Cast函数
self.cast = P.Cast()
def construct(self, global_step):
'''
构建CosineDecayLR
:param global_step: 总共的时间步数
:return:
'''
p = self.cast(self.min(global_step, self.decay_steps), mstype.float32)
# 计算p的值，并将其转换为float32类型
return self.min_lr + self.delta * (1.0 + self.cos(self.math_pi * p / self.decay_steps))
class PolynomialDecayLR(LearningRateSchedule):
# 基于多项式衰减函数计算学习率。
r"""
Calculates learning rate base on polynomial decay function.
For current step, the formula of computing decayed learning rate is:
For the i-th step, the formula of computing decayed_learning_rate[i] is:
.. math::
decayed\_learning\_rate = (learning\_rate - end\_learning\_rate) *
decayed\_learning\_rate[i] = (learning\_rate - end\_learning\_rate) *
(1 - tmp\_step / tmp\_decay\_steps)^{power} + end\_learning\_rate
Where :
@ -357,15 +420,15 @@ class PolynomialDecayLR(LearningRateSchedule):
Args:
learning_rate (float): The initial value of learning rate.
end_learning_rate (float): The end value of learning rate.
decay_steps (int): Number of steps to decay over.
power (float): The power of polynomial. It must be greater than 0.
decay_steps (int): A value used to calculate decayed learning rate.
power (float): A value used to calculate decayed learning rate. This parameter must be greater than 0.
update_decay_steps (bool): If true, learning rate is decayed once every `decay_steps` time. Default: False.
Inputs:
- **global_step** (Tensor) - The current step number.
Tensor. The current step number.
Outputs:
Tensor. The learning rate value for the current step with shape :math:`()`.
Tensor. The learning rate value for the current step.
Raises:
TypeError: If `learning_rate`, `end_learning_rate` or `power` is not a float.
@ -377,28 +440,30 @@ class PolynomialDecayLR(LearningRateSchedule):
``Ascend`` ``GPU``
Examples:
>>> import mindspore
>>> from mindspore import Tensor, nn
>>>
>>> learning_rate = 0.1
>>> end_learning_rate = 0.01
>>> decay_steps = 4
>>> power = 0.5
>>> global_step = Tensor(2, mindspore.int32)
>>> global_step = Tensor(2, mstype.int32)
>>> polynomial_decay_lr = nn.PolynomialDecayLR(learning_rate, end_learning_rate, decay_steps, power)
>>> result = polynomial_decay_lr(global_step)
>>> print(result)
0.07363961
"""
def __init__(self, learning_rate, end_learning_rate, decay_steps, power, update_decay_steps=False):
'''
初始化PolynomialDecayLR类，并检查参数
:param learning_rate: 学习率
:param end_learning_rate: 终止学习率
:param decay_steps: 时间步数
:param power: 指数
:param update_decay_steps: 是否更新指数步长
'''
super(PolynomialDecayLR, self).__init__()
validator.check_positive_float(learning_rate, 'learning_rate')
validator.check_is_float(learning_rate, 'learning_rate')
if not isinstance(end_learning_rate, float):
raise TypeError("For 'PolynomialDecayLR', the argument 'end_learning_rate' "
"must be type of float, but got 'end_learning_rate' type: {}."
.format(type(end_learning_rate)))
raise TypeError("end_learning_rate must be float.")
validator.check_non_negative_float(end_learning_rate, "end_learning_rate", self.cls_name)
validator.check_positive_int(decay_steps, 'decay_steps', self.cls_name)
validator.check_value_type('update_decay_steps', update_decay_steps, [bool], self.cls_name)
@ -417,29 +482,41 @@ class PolynomialDecayLR(LearningRateSchedule):
self.max = P.Maximum()
def construct(self, global_step):
'''
构建PolynomialDecayLR类
:param global_step: 步数
:return: 学习率
'''
tmp_global_step = P.Cast()(global_step, mstype.float32)
# 将训练步数转换为浮点数
tmp_decay_step = self.decay_steps
# 如果update_decay_steps为True，则tmp_decay_step乘以max(ceil(tmp_global_step / tmp_decay_step), 1)
if self.update_decay_steps:
tmp_decay_step = tmp_decay_step * self.max(self.ceil(tmp_global_step / tmp_decay_step), 1)
# 否则，tmp_global_step小于tmp_decay_step，则tmp_global_step等于tmp_decay_step
else:
tmp_global_step = self.min(tmp_global_step, tmp_decay_step)
# tmp_global_step / tmp_decay_step
p = tmp_global_step / tmp_decay_step
# 将p的值转换为float32类型，并乘以diff_learning_rate，加上end_learning_rate
lr = self.diff_learning_rate * self.pow(1.0 - p, self.power) + self.end_learning_rate
# 返回lr
return lr
class WarmUpLR(LearningRateSchedule):
# 预热学习率。
r"""
Gets learning rate warming up.
For current step, the formula of computing warmup learning rate is:
For the i-th step, the formula of computing warmup_learning_rate[i] is:
.. math::
warmup\_learning\_rate = learning\_rate * tmp\_step / warmup\_steps
warmup\_learning\_rate[i] = learning\_rate * tmp\_step / warmup\_steps
Where :
.. math::
.. math:
tmp\_step=min(current\_step, warmup\_steps)
Args:
@ -447,10 +524,10 @@ class WarmUpLR(LearningRateSchedule):
warmup_steps (int): The warm up steps of learning rate.
Inputs:
- **global_step** (Tensor) - The current step number.
Tensor. The current step number.
Outputs:
Tensor. The learning rate value for the current step with shape :math:`()`.
Tensor. The learning rate value for the current step.
Raises:
TypeError: If `learning_rate` is not a float.
@ -462,22 +539,23 @@ class WarmUpLR(LearningRateSchedule):
``Ascend`` ``GPU``
Examples:
>>> import mindspore
>>> from mindspore import Tensor, nn
>>>
>>> learning_rate = 0.1
>>> warmup_steps = 2
>>> global_step = Tensor(2, mindspore.int32)
>>> global_step = Tensor(2, mstype.int32)
>>> warmup_lr = nn.WarmUpLR(learning_rate, warmup_steps)
>>> result = warmup_lr(global_step)
>>> print(result)
0.1
"""
def __init__(self, learning_rate, warmup_steps):
'''
初始化WarmUpLR类，并检查输入参数
:param learning_rate: 学习率
:param warmup_steps: 过去warmup_steps步后的学习率
'''
super(WarmUpLR, self).__init__()
if not isinstance(learning_rate, float):
raise TypeError("For 'WarmUpLR', the argument 'learning_rate' must be type of float, "
"but got 'learning_rate' type: {}.".format(type(learning_rate)))
raise TypeError("learning_rate must be float.")
validator.check_non_negative_float(learning_rate, "learning_rate", self.cls_name)
validator.check_positive_int(warmup_steps, 'warmup_steps', self.cls_name)
self.warmup_steps = warmup_steps
@ -486,10 +564,15 @@ class WarmUpLR(LearningRateSchedule):
self.cast = P.Cast()
def construct(self, global_step):
warmup_percent = self.cast(self.min(global_step, self.warmup_steps), mstype.float32) / self.warmup_steps
'''
计算学习率
:param global_step: 步数
:return: 学习率
'''
warmup_percent = self.cast(self.min(global_step, self.warmup_steps), mstype.float32)/ self.warmup_steps
# 返回预热学习率，乘以warmup_percent
return self.learning_rate * warmup_percent
__all__ = [
'ExponentialDecayLR',
'NaturalExpDecayLR',

2
mindspore/python/mindspore/nn/loss/__init__.py
View File

@ -18,7 +18,7 @@ Loss.
Cells of loss function. Loss function in machine learning is the target of the model.
It shows how well the model works on a dataset and the optimization target which the optimizer is searching.
"""
#本函数为loss函数的初始化函数
from .loss import LossBase, L1Loss, MSELoss, SmoothL1Loss, SoftMarginLoss, FocalLoss,\
SoftmaxCrossEntropyWithLogits, BCELoss, CosineEmbeddingLoss, \
SampledSoftmaxLoss, DiceLoss, BCEWithLogitsLoss, MultiClassDiceLoss,\

2414
mindspore/python/mindspore/nn/loss/loss.py
View File

File diff suppressed because it is too large Load Diff

49
mindspore/python/mindspore/nn/metrics/__init__.py
View File

@ -12,32 +12,58 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 定义了一个名为Metrics的类，用于提供各种评估指标，用于衡量机器学习模型的性能
"""
Metrics.
Functions to measure the performance of the machine learning models
on the evaluation dataset. It's used to choose the best model.
"""
# 这个类包含了以下模块：
# 1.计算模型在分类问题中的准确率
from .accuracy import Accuracy
# 2.计算模型在分类问题中的汉氏距离
from .hausdorff_distance import HausdorffDistance
# 3.计算模型在分类问题中的均方误差（MSE）和平均绝对误差（MAE）
from .error import MAE, MSE
# 4.一个抽象类，用于封装评估指标的计算
from .metric import Metric, rearrange_inputs
# 5.计算模型在多标签分类问题中的精确度
from .precision import Precision
# 6.计算模型在多标签分类问题中的召回率
from .recall import Recall
# 7.计算模型在多标签分类问题中的F分数
from .fbeta import Fbeta, F1
# 8.计算模型在多标签分类问题中的Dice系数
from .dice import Dice
# 9.计算模型在二分类问题中的ROC曲线
from .roc import ROC
# 10.计算模型在二分类问题中的AUC值
from .auc import auc
# 11.计算模型在多分类问题中的Top-K准确率
# 12.计算模型在多分类问题中的Top-1准确率
# 13.计算模型在多分类问题中的Top-5准确率
from .topk import TopKCategoricalAccuracy, Top1CategoricalAccuracy, Top5CategoricalAccuracy
# 14.计算模型在回归问题中的损失函数
from .loss import Loss
# 15.计算模型在三维形状上的平均表面距离
from .mean_surface_distance import MeanSurfaceDistance
# 16.计算模型在三维形状上的根均方距离
from .root_mean_square_surface_distance import RootMeanSquareDistance
# 17.计算模型在文本生成问题中的BLEU分数
from .bleu_score import BleuScore
# 18.计算模型在文本生成问题中的余弦相似度
from .cosine_similarity import CosineSimilarity
# 19.计算模型在图像上进行遮罩测试时的敏感度
from .occlusion_sensitivity import OcclusionSensitivity
# 20.计算模型在文本生成问题中的困惑度
from .perplexity import Perplexity
# 21.计算模型在分类问题中的混淆矩阵
# 22.计算模型在分类问题中的混淆矩阵
from .confusion_matrix import ConfusionMatrixMetric, ConfusionMatrix
# __all__是一个包含类或函数名称的列表，用于告诉Python解释器这些名称可以被导入。
# 这样，当用户从Metrics类中导入这些名称时，不会导入整个类，而是只导入所需的名称。这有助于减少导入时间，提高代码的可读性和性能
__all__ = [
"names",
"get_metric_fn",
@ -66,6 +92,7 @@ __all__ = [
"ConfusionMatrixMetric",
]
# 名为__factory__的静态方法。这个方法允许在创建类实例时提供一个工厂函数，用于根据参数创建类的具体实现。这在创建具有不同功能的子类时非常有用，例如在框架中
__factory__ = {
'accuracy': Accuracy,
'acc': Accuracy,
@ -94,6 +121,7 @@ __factory__ = {
def names():
# 用于获取Metrics类中所有指标方法的名称
"""
Gets all names of the metric methods.
@ -103,10 +131,13 @@ def names():
Supported Platforms:
``Ascend`` ``GPU`` ``CPU``
"""
# 首先，通过__factory__字典获取所有指标方法的名称，然后对名称进行排序，最后返回排序后的名称列表
return sorted(__factory__.keys())
# 这样，用户可以通过调用names函数获取Metrics类中所有可用的指标方法名称，以便根据需要选择和使用它们
def get_metric_fn(name, *args, **kwargs):
# 用于根据输入的名称获取Metrics类中对应的方法
"""
Gets the metric method based on the input name.
@ -126,13 +157,18 @@ def get_metric_fn(name, *args, **kwargs):
>>> from mindspore import nn
>>> metric = nn.get_metric_fn('precision', eval_type='classification')
"""
# 首先，通过__factory__字典检查输入的名称是否在字典中
if name not in __factory__:
# 如果不存在，则抛出一个KeyError异常
raise KeyError(f"For 'get_metric_fn', unsupported metric {name}, please refer to official website "
f"for more details about supported metrics.")
# 如果存在，则根据输入的名称和参数创建一个Metrics类中的具体实现，并返回这个实现类的实例
return __factory__[name](*args, **kwargs)
# 在示例中，用户可以通过调用get_metric_fn函数来获取不同指标方法的实例，然后使用这个实例来计算评估结果
def get_metrics(metrics):
# 用于根据输入的指标名称获取Metrics类中对应的方法。函数接受一个字典或集合作为输入，集合中的指标名称将自动映射到对应的Metrics类实例
"""
Get metrics used in evaluation.
@ -146,20 +182,33 @@ def get_metrics(metrics):
Raises:
TypeError: If the type of argument 'metrics' is not None, dict or set.
"""
# 首先检查输入的指标名称是否为None
if metrics is None:
# 如果是，则直接返回metrics
return metrics
# 然后，检查输入的指标名称是否为字典或集合
if isinstance(metrics, dict):
# 如果是，则遍历指标名称和对应的指标方法,将它们存储在一个新的字典中，并返回这个字典
for name, metric in metrics.items():
# 如果输入的指标名称不是字典或集合
if not isinstance(name, str) or not isinstance(metric, Metric):
# 则抛出一个TypeError异常
raise TypeError(f"For 'get_metrics', if 'metrics' is dict, the key in 'metrics' must be string and "
f"value in 'metrics' must be Metric, but got key:{type(name)}, value:{type(metric)}.")
# 返回指标字典
return metrics
# 首先检查输入的指标名称是否为字典或集合
if isinstance(metrics, set):
# 新建空字典存储指标信息
out_metrics = {}
# 如果是，则遍历指标名称和对应的指标方法
for name in metrics:
# 将它们存储在一个新的字典中
out_metrics[name] = get_metric_fn(name)
# 并返回这个字典
return out_metrics
# 如果输入的指标名称不是字典或集合，则抛出一个TypeError异常
raise TypeError("For 'get_metrics', the argument 'metrics' should be None, dict or set, "
"but got {}".format(metrics))

52
mindspore/python/mindspore/nn/metrics/accuracy.py
View File

@ -13,11 +13,24 @@
# limitations under the License.
# ============================================================================
"""Accuracy."""
# 名为Accuracy的Python模块，其中包含了与准确率相关的功能
# 首先，导入了numpy库，用于处理数值计算
import numpy as np
# 然后，从metric模块中导入了一个名为EvaluationBase的类，以及一个名为rearrange_inputs的函数，以及一个名为_check_onehot_data的函数。这些类和函数用于计算和处理准确率相关的内容
# EvaluationBase，是metric模块中的一个基类，用于实现评估指标的基本功能
# rearrange_inputs，该函数用于重新排列输入数据，以便在计算评估指标时可以正确地处理数据。具体来说，它会检查输入数据是否是一个多维数组，
# 如果是，它会将其转换为一个二维数组，其中每一行表示一个样本，每一列表示一个特征
# _check_onehot_data，该函数用于检查输入数据是否符合独热编码规则。具体来说，它会检查输入数据是否是一个二维数组，其中每一行表示一个样本，
# 每一列表示一个类别。此外，它还检查输入数据中的每个类别是否是唯一的，如果不是，则会抛出一个异常
from .metric import EvaluationBase, rearrange_inputs, _check_onehot_data
class Accuracy(EvaluationBase):
# Accuracy类，它是EvaluationBase类的子类，用于计算分类和多标签数据的精度。
# 在计算精度时，它会创建两个本地变量，即正确数量的计数器和总数量的计数器。然后，它使用这些变量计算准确率，
# 具体来说，它是正确数量的除数，加上正确数量的除数和假正例数量的除数和假负例数量的除数
r"""
Calculates the accuracy for classification and multilabel data.
@ -51,17 +64,27 @@ class Accuracy(EvaluationBase):
0.6666666666666666
"""
def __init__(self, eval_type='classification'):
'''
初始化准确率
:param eval_type: 评估类型，默认为分类
'''
super(Accuracy, self).__init__(eval_type)
# 调用子类的clear方法，用于清除评估结果
self.clear()
def clear(self):
# 清空内部评估结果
"""Clears the internal evaluation result."""
# 正确项计数器
self._correct_num = 0
# 总数
self._total_num = 0
# 类数量
self._class_num = 0
@rearrange_inputs
def update(self, *inputs):
# 更新本地变量。对于二分类任务，如果预测值的索引与标签相匹配，则认为预测结果正确。对于多标签任务，如果预测值与标签相匹配，则认为预测结果正确
"""
Updates the local variables. For 'classification', if the index of the maximum of the predict value
matches the label, the predict result is correct. For 'multilabel', the predict value match the label,
@ -82,36 +105,53 @@ class Accuracy(EvaluationBase):
ValueError: class numbers of last input predicted data and current predicted data not match.
"""
# 检查输入参数的数量是否为2
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError异常，表示需要2个输入参数，但实际提供了{}个
raise ValueError("For 'Accuracy.update', it needs 2 inputs (predicted value, true value), "
"but got {}".format(len(inputs)))
# 将data型的输入值inputs[0]转换为numpy数组
y_pred = self._convert_data(inputs[0])
# 将data型的输入值inputs[1]转换为numpy数组
y = self._convert_data(inputs[1])
# 根据self._type的值判断是二分类任务还是多标签任务
if self._type == 'classification' and y_pred.ndim == y.ndim and _check_onehot_data(y):
# 对于分类任务，如果y_pred和y的维度相同，并且y是onehot编码的，则将y的维度降低到一个维度
y = y.argmax(axis=1)
# 最后，检查y_pred和y的形状和值是否符合要求
self._check_shape(y_pred, y)
self._check_value(y_pred, y)
# 对输入的类型做出判断，并根据不同的结果做出不同的修改
if self._class_num == 0:
# 若输入的类别数为0，则将其赋值为预测的类别数
self._class_num = y_pred.shape[1]
elif y_pred.shape[1] != self._class_num:
elif y_pred.shape[1]!= self._class_num:
# 若预测的类别数不等于输入的类别数，则抛出异常ValueError需检查预测值出错的原因
raise ValueError("For 'Accuracy.update', class number not match, last input predicted data contain {} "
"classes, but current predicted data contain {} classes, please check your predicted "
"value(inputs[0]).".format(self._class_num, y_pred.shape[1]))
# 根据self._type的值判断是二分类任务还是多标签任务
if self._type == 'classification':
# 对于分类任务，将y_pred的维度降低到一个维度
indices = y_pred.argmax(axis=1)
# 比较y_pred和y是否相等
result = (np.equal(indices, y) * 1).reshape(-1)
elif self._type == 'multilabel':
elif self._type =='multilabel':
# 对于多标签任务，获取预测结果的维度并减一
dimension_index = y_pred.ndim - 1
# 将预测结果的维度改为-1
y_pred = y_pred.swapaxes(1, dimension_index).reshape(-1, self._class_num)
# 将标签的维度改为-1，将y_pred和y的维度降低到一个维度
y = y.swapaxes(1, dimension_index).reshape(-1, self._class_num)
# 将预测结果和标签做比较，并将结果改为1
result = np.equal(y_pred, y).all(axis=1) * 1
# 计算result中所有元素的和，并将其加到self._correct_num和self._total_num中
self._correct_num += result.sum()
self._total_num += result.shape[0]
def eval(self):
# 计算模型的准确性
"""
Computes the accuracy.
@ -121,8 +161,12 @@ class Accuracy(EvaluationBase):
Raises:
RuntimeError: If the sample size is 0.
"""
# 首先，检查self._total_num是否为0
if self._total_num == 0:
# 如果是，则抛出一个RuntimeError异常，表示无法计算准确性，因为样本数量为0
raise RuntimeError("The 'Accuracy' can not be calculated, because the number of samples is 0, "
"please check whether your inputs(predicted value, true value) are empty, "
"or has called update method before calling eval method.")
# 如果self._total_num不为0，则计算self._correct_num除以self._total_num的结果准确率，并返回该结果。
# 注：在计算准确性时，需要确保已经调用了update方法，以便能够更新self._correct_num和self._total_num的值
return self._correct_num / self._total_num

65
mindspore/python/mindspore/nn/metrics/auc.py
View File

@ -13,10 +13,16 @@
# limitations under the License.
# ============================================================================
"""auc"""
# 用Python编写的，主要用于计算曲线下面积（AUC），下属于metrics模块用于对机器学习的性能进行评估
# 首先，导入了两个常用的库：numpy。numpy是Python的科学计算库，用于处理大型数组和矩阵；
import numpy as np
def auc(x, y, reorder=False):
# 代码的主要功能是计算曲线下面积（AUC），这是一种评估二分类模型性能的指标。AUC表示了模型将正样本分片分配到比负样本更小的概率时，
# 能成功识别正样本的百分比。在二分类问题中，通常将正样本定义为1，负样本定义为0
# 需要注意的是，这段代码仅适用于二分类问题。对于多分类问题，需要使用其他方法来计算AUC
# 该函数接受两个列表作为输入：y_true和y_pred。y_true是一个表示真实标签的列表，y_pred是一个表示模型预测概率的列表
"""
Computes the AUC(Area Under the Curve) using the trapezoidal rule. This is a general function, given points on a
curve. For computing the area under the ROC-curve.
@ -49,68 +55,118 @@ def auc(x, y, reorder=False):
>>> print(output)
0.5357142857142857
"""
"""
以下这段代码主要用于检查参数并处理
"""
# 首先检查x和y是否为numpy数组
if not isinstance(x, np.ndarray) or not isinstance(y, np.ndarray):
# 如果不是，则抛出一个TypeError，提醒使用者
raise TypeError("For 'auc', the argument 'x' and 'y' must all be np.ndarray, but got {}, {}"
.format(type(x), type(y)))
# 然后，使用numpy的_check_consistent_length函数检查x和y的长度是否一致。如果长度不一致，则抛出一个ValueError
_check_consistent_length(x, y)
# 最后，将x和y转换为列向量
x = _column_or_1d(x)
y = _column_or_1d(y)
# 首先检查x的形状是否为2维数组
if x.shape[0] < 2:
# 如果形状为1维数组，则抛出一个ValueError
raise ValueError("For 'auc', the shape of the argument 'x' in axis 0 must be greater than 2, "
"but got {}.".format(x.shape[0]))
# 如果x的维度为1，定义一个名为direction的变量设置为1
# 注：direction用于表示计算AUC时所使用的方向。通常情况下，我们使用左端点(0)和右端点(1)来计算AUC，此时direction的值为1；
# 如果使用右端点(1)和左端点(0)来计算AUC，此时direction的值为-1
direction = 1
"""
以下这段代码主要用于处理x和y的顺序
"""
# 首先，检查reorder是否为True
if reorder:
# 如果为True，则使用np.lexsort对y和x进行排序
# 注：np.lexsort是一个快速排序算法，它首先对y进行排序，然后根据x中的值对已经排序的y和x进行排序。这样，x和y的顺序就得到了处理，返回一个排列顺序order
order = np.lexsort((y, x))
# 将x和y按照order中的顺序进行排列
x, y = x[order], y[order]
else:
# 计算x中相邻元素的差值，并将结果存储在dx中。dx是一个一维numpy数组，其长度比x少1
dx = np.diff(x)
# 如果reorder为False，则检查x中的相邻元素是否满足递增或递减的条件
if np.any(dx < 0):
if np.all(dx <= 0):
# 满足则将direction设置为-1
direction = -1
else:
# 如果不满足，则抛出一个ValueError
raise ValueError("For 'auc', if the argument is False, the argument 'x' array should be increasing "
"or decreasing, but got 'x': {}".format(x))
"""
这段代码计算曲线下面积
"""
# np.trapz(y, x)表示计算从x[0]到x[-1]的等差数列中各个点对应的梯形面积，然后将结果乘以direction
area = direction * np.trapz(y, x)
# 如果area是一个numpy内存映射对象
if isinstance(area, np.memmap):
# 最后，将area转换为原来的数据类型
area = area.dtype.type(area)
# 返回计算的面积
return area
def _column_or_1d(y):
# 函数用于将输入的y值转换为1D数组
"""
Ravel column or 1D numpy array, otherwise raise a ValueError.
"""
# 获取数组y的形状
shape = np.shape(y)
# 如果y是一个1D数组，或者是一个2D数组，其中第二维的长度为1
if len(shape) == 1 or (len(shape) == 2 and shape[1] == 1):
# 函数返回1D数组
return np.ravel(y)
# 否则，抛出一个ValueError
raise ValueError("For 'auc', the input must be a 1-dimensional array, or a 2-dimensional array with the second "
"dimension of 1, but got shape {}.".format(shape))
def _num_samples(x):
# 函数用于计算输入x的样本数量。x可以是列表、数组或其他类似序列的集合
"""Return the number of samples in array-like x."""
"""
该函数首先检查样本是否拥有可以直接使用的属性，然后根据该属性获取样本数量
"""
# 函数首先检查x是否有一个fit方法
if hasattr(x, 'fit') and callable(x.fit):
# 如果是，则抛出一个TypeError
raise TypeError('Expected sequence or array-like, got estimator {}.'.format(x))
# 然后，检查x是否有一个__len__方法或一个shape属性
if not hasattr(x, '__len__') and not hasattr(x, 'shape'):
# 如果没有这些属性，则判断是否有数组属性
if hasattr(x, '__array__'):
# 如果有，则尝试将其转换为numpy数组
x = np.asarray(x)
else:
# 否则，抛出一个TypeError
raise TypeError("Expected sequence or array-like, got {}." .format(type(x)))
# 首先检查x是否有一个shape属性
if hasattr(x, 'shape'):
if x.ndim == 0:
# 如果x是一个单元素数组，即x.shape == ()，则抛出一个TypeError
raise TypeError("Singleton array {} cannot be considered as a valid collection.".format(x))
# 如果存在，则计算x的第一个维度（即样本数量）
res = x.shape[0]
else:
# 如果x是一个多元素数组，即x.shape != ()，那么计算x的大小（即元素数量），并将其赋值给变量res
res = x.size
# 最后，返回样本数量
return res
def _check_consistent_length(*arrays):
# 函数用于检查输入的多个数组（arrays）是否具有一致的第一个维度。函数的主要目的是确保输入的对象（数组）具有相同的形状或长度
r"""
Check that all arrays have consistent first dimensions. Check whether all objects in arrays have the same shape
or length.
@ -119,9 +175,12 @@ def _check_consistent_length(*arrays):
- **(*arrays)** - (Union[tuple, list]): list or tuple of input objects. Objects that will be checked for
consistent length.
"""
# 首先遍历输入的数组（arrays），并使用_num_samples函数计算每个数组的大小（第一个维度）。然后，将计算得到的大小列表存储在变量lengths中
lengths = [_num_samples(array) for array in arrays if array is not None]
# 找到lengths列表中的所有唯一长度，并将结果存储在变量uniques中。np.unique函数会返回一个有序的unique元素列表，其中元素不重复
uniques = np.unique(lengths)
# 检查uniques列表的长度是否大于1
if len(uniques) > 1:
# 如果是，则抛出一个ValueError，其中包含所有不重复的长度（即输入数组的样本数量）
raise ValueError("Found input variables with inconsistent numbers of samples: {}."
.format([int(length) for length in lengths]))

92
mindspore/python/mindspore/nn/metrics/bleu_score.py
View File

@ -12,14 +12,23 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 该类继承自Metric。BleuScore类用于计算BLEU分数，是一种用于评估机器翻译文本质量的指标
"""BleuScore."""
# 首先，从collections模块导入Counter类，用于计数器
from collections import Counter
# 接下来，从numpy模块导入numpy库，用于处理数值计算
import numpy as np
# 然后，从mindspore._checkparam模块导入Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 最后，从mindspore.metric模块导入Metric类，用于计算其他指标
from .metric import Metric, rearrange_inputs
class BleuScore(Metric):
# 参数：n_gram和smooth。n_gram参数表示计算BLEU分数时使用的n-gram值，取值范围为1到4。smooth参数表示是否应用平滑算法，取值为True或False
# n-gram是一种用于表示文本中连续词项的方法，它将文本分成大小为n的子文本片段，然后计算每个子文本片段中不同词项的频率。
# ,BleuScore类中n-gram的范围决定了计算BLEU分数时使用的n-gram数量。通常情况下，n-gram值为4
# 在示例中，我们首先创建了一个BleuScore对象，然后更新其内部评估结果。最后，我们调用eval方法计算BLEU分数并输出结果
"""
Calculates the BLEU score. BLEU (bilingual evaluation understudy) is a metric for evaluating
the quality of text translated by machine.
@ -48,27 +57,42 @@ class BleuScore(Metric):
0.5946035575013605
"""
def __init__(self, n_gram=4, smooth=False):
# 调用父类的初始化函数，确保基本功能正常工作
super().__init__()
# 然后，对n_gram和smooth进行验证，确保它们的类型为int和bool
self.n_gram = validator.check_value_type("n_gram", n_gram, [int])
if self.n_gram > 4 or self.n_gram < 1:
# 如果n_gram的值大于4或小于1，则抛出一个ValueError异常，表示n_gram的值不合法
raise ValueError("For 'BleuScore', the argument 'n_gram' should range from 1 to 4, "
"but got {}.".format(n_gram))
self.smooth = validator.check_value_type("smooth", smooth, [bool])
# 清空残留的内部评估结果
self.clear()
def clear(self):
# 该方法用于清空内部评估结果
"""Clear the internal evaluation result."""
# 存储计算分子时所需的值
self._numerator = np.zeros(self.n_gram)
# 存储计算分母时所需的值
self._denominator = np.zeros(self.n_gram)
# 存储计算precision分数时所需的值
self._precision_scores = np.zeros(self.n_gram)
# 存储计算BLEU分数时所需的值
self._c = 0.0
# 存储计算BLEU分数时所需的值
self._r = 0.0
# 存储翻译文本的长度
self._trans_len = 0
# 存储参考文本的长度
self._ref_len = 0
# 标记是否已经更新了类变量
self._is_update = False
def _count_ngram(self, ngram_input_list, n_gram):
# 统计输入文本中每个单词出现的次数，同时使用ngram值
# ngram_input_list是一个列表，包含要统计的翻译文本或参考文本
"""
Counting how many times each word appears in a given text with ngram.
@ -82,15 +106,23 @@ class BleuScore(Metric):
ngram_counter = Counter()
# 遍历输入文本
for i in range(1, n_gram + 1):
# 对于每个ngram值，从当前索引开始向前查找相应长度的子列表
for j in range(len(ngram_input_list) - i + 1):
# 将子列表转换为元组
ngram_key = tuple(ngram_input_list[j:(i + j)])
# 作为ngram键，并将出现次数存储在Counter对象中
ngram_counter[ngram_key] += 1
# 返回值是一个collections.Counter对象，用于存储统计结果
return ngram_counter
@rearrange_inputs
def update(self, *inputs):
# 该方法用于更新内部评估结果
# 接受两个输入参数：candidate_corpus和reference_corpus。candidate_corpus是一个包含机器翻译文本的迭代器，
# reference_corpus是一个包含参考文本的迭代器。两个输入参数的长度应相同
"""
Updates the internal evaluation result with `candidate_corpus` and `reference_corpus`.
@ -103,40 +135,73 @@ class BleuScore(Metric):
ValueError: If the number of inputs is not 2.
ValueError: If the lengths of `candidate_corpus` and `reference_corpus` are not equal.
"""
"""
检查输入，获取参数并检查其长度是否相等
"""
# 如果输入数组长度不为2
if len(inputs) != 2:
# 抛出异常ValueError
raise ValueError("For 'BleuScore.update', it needs 2 inputs (candidate_corpus, reference_corpus), "
"but got {}.".format(len(inputs)))
# 从input中获取候选文本与参考文本
candidate_corpus = inputs[0]
reference_corpus = inputs[1]
if len(candidate_corpus) != len(reference_corpus):
raise ValueError("For 'BleuScore.update', 'translate_corpus' (inputs[0]) and 'reference_corpus' "
# 判断两者长度是否相等
if len(candidate_corpus)!= len(reference_corpus):
# 如果candidate_corpus和reference_corpus的长度不相等，抛出ValueError
raise ValueError("For 'BleuScore.update', 'translate_corpus' (inputs[0]) and'reference_corpus' "
"(inputs[1]) should be equal in length, but got {}, {}"
.format(len(candidate_corpus), len(reference_corpus)))
"""
计算候选文本和参考文本的n-gram计数
"""
# 首先对输入的candidate_corpus和reference_corpus进行zip操作，将它们一一对应地组合成一个新的元组列表，然后，遍历新的元组列表
for (candidate, references) in zip(candidate_corpus, reference_corpus):
# 对于每个元组，将其第一个元素（candidate）的长度加到self._c中
self._c += len(candidate)
# 同时将其第二个元素（references）的长度加到ref_len_list中
ref_len_list = [len(ref) for ref in references]
# 接下来，计算candidate和每个reference的长度差
ref_len_diff = [abs(len(candidate) - x) for x in ref_len_list]
# 并找到长度差的最小值对应的下标
self._r += ref_len_list[ref_len_diff.index(min(ref_len_diff))]
# 最后，使用self._count_ngram方法统计candidate中的n-gram
translation_counter = self._count_ngram(candidate, self.n_gram)
# 并将结果存储在translation_counter中
reference_counter = Counter()
"""
计算候选文本和参考文本的n-gram重叠部分
"""
# 遍历references中的每个元素ref
for ref in references:
# 使用self._count_ngram方法统计ref中的n-gram
reference_counter |= self._count_ngram(ref, self.n_gram)
# 并将结果与translation_counter进行按位与操作，最后，将结果存储在ngram_counter_clip中
ngram_counter_clip = translation_counter & reference_counter
"""
更新n-gram计数和 denominator
"""
# 首先遍历ngram_counter_clip中的每个键值对counter_clip
for counter_clip in ngram_counter_clip:
# 然后将键的长度减1，值加到self._numerator中
self._numerator[len(counter_clip) - 1] += ngram_counter_clip[counter_clip]
# 遍历translation_counter中的每个键值对counter
for counter in translation_counter:
# 将键的长度减1，值加到self._denominator中
self._denominator[len(counter) - 1] += translation_counter[counter]
# 首先将self._c转换为numpy数组，并将其存储在self._trans_len中
self._trans_len = np.array(self._c)
# 然后，将self._r转换为numpy数组，并将其存储在self._ref_len中
self._ref_len = np.array(self._r)
# 最后，将self._is_update的值设置为True
self._is_update = True
def eval(self):
# 该方法用于计算BLEU分数
"""
Computes the bleu score.
@ -146,21 +211,36 @@ class BleuScore(Metric):
Raises:
RuntimeError: If the update method is not called first, an error will be reported.
"""
"""
首先检查是否已经更新过内部评估结果
"""
if self._is_update is False:
# 如果没有更新，则抛出一个运行时错误
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 检查self._numerator中的最小值是否为0.0
if min(self._numerator) == 0.0:
# 如果最小值等于0.0，则返回一个numpy数组中的0.0
return np.array(0.0)
"""
计算BLEU分数
"""
# 计算平滑后的n-gram精度分数
if self.smooth:
# 如果smooth参数为True，则应用平滑算法
precision_scores = np.add(self._numerator, np.ones(self.n_gram)) / np.add(self._denominator,
np.ones(self.n_gram))
else:
# 否则，直接计算精度分数
precision_scores = self._numerator / self._denominator
# 计算与self.n_gram长度相同的列表log_precision_scores，其中所有元素的值都为1除以self.n_gram
log_precision_scores = np.array([1.0 / self.n_gram] * self.n_gram) * np.log(precision_scores)
# 计算长度几何平均值和布雷维尼系数
geometric_mean = np.exp(np.sum(log_precision_scores))
brevity_penalty = np.array(1.0) if self._c > self._r else np.exp(1 - (self._ref_len / self._trans_len))
# 最后计算BLEU分数
bleu = brevity_penalty * geometric_mean
# 返回BLEU分数
return bleu

280
mindspore/python/mindspore/nn/metrics/confusion_matrix.py
View File

@ -13,12 +13,23 @@
# limitations under the License.
# ============================================================================
"""ConfusionMatrixMetric & ConfusionMatrix."""
# ConfusionMatrixMetric类继承自Metric类，用于计算混淆矩阵。ConfusionMatrix函数用于根据真实标签和预测标签计算混淆矩阵
import numpy as np
# 导入所需的库：numpy库用于处理数值计算，mindspore._checkparam库中的Validator类用于参数验证
from mindspore._checkparam import Validator as validator
# 从metric模块中导入Metric类和rearrange_inputs函数。Metric类是用于计算指标的基类，而rearrange_inputs函数用于对输入数据进行重新排列，以适应某些计算需求
from .metric import Metric, rearrange_inputs
class ConfusionMatrix(Metric):
# 参数：num_classes和normalize。num_classes表示数据集中的类数，normalize表示混淆矩阵的normalization模式。有四种normalization模式：
# "no_norm" (None)：不进行normalization。默认模式。
# "target"：根据真实标签进行normalization。
# "prediction"：根据预测标签进行normalization。
# "all"：根据整个混淆矩阵进行normalization。
# threshold参数用于比较输入张量，默认值为0.5。
"""
Computes the confusion matrix, which is commonly used to evaluate the performance of classification models,
including binary classification and multiple classification.
@ -55,24 +66,34 @@ class ConfusionMatrix(Metric):
[1. 1.]]
"""
def __init__(self, num_classes, normalize="no_norm", threshold=0.5):
# 调用父类构造函数初始化
super(ConfusionMatrix, self).__init__()
# 检查输入参数num_classes是否为int型，否则抛出TypeError
self.num_classes = validator.check_value_type("num_classes", num_classes, [int])
# 如果normalize的模式不在预定的四个之内，则抛出异常ValueError
if normalize not in ["target", "prediction", "all", "no_norm"]:
raise ValueError("For 'ConfusionMatrix', the argument 'normalize' should be in "
"['all', 'prediction', 'label', 'no_norm'(None)], but got {}.".format(normalize))
# 获取normalize的值
self.normalize = normalize
# 检查输入参数threshold是否为float型，否则抛出TypeError
self.threshold = validator.check_value_type("threshold", threshold, [float])
# 清空类中残余数据
self.clear()
def clear(self):
# 用于清空类中的数据
"""Clears the internal evaluation result."""
# 定义self.confusion_matrix，用于存储混淆矩阵
self.confusion_matrix = np.zeros((self.num_classes, self.num_classes))
# 表示未更新混淆矩阵
self._is_update = False
@rearrange_inputs
def update(self, *inputs):
# 用于更新混淆矩阵的状态
# inputs参数接受两个张量y_pred和y作为输入。y_pred是预测值，y是真实值。
# y_pred的形状可以是(N, C, ...)或(N, ...)，其中N是样本数量，C是类别数量。y的形状是(N, ...)
"""
Update state with y_pred and y.
@ -86,62 +107,88 @@ class ConfusionMatrix(Metric):
ValueError: If the number of inputs is not 2.
ValueError: If the dim of y_pred and y are not equal.
"""
# 检查inputs参数的数量是否为2
if len(inputs) != 2:
# 如果不是，则抛出一个错误ValueError
raise ValueError("For 'ConfusionMatrix.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 然后，将inputs[0]和inputs[1]转换为适当的数据格式（例如，张量或numpy数组）
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 如果预测值和真实值的维度不匹配，或者预测值的维度与真实值的维度之差大于1
if not (y_pred.ndim == y.ndim or y_pred.ndim == y.ndim + 1):
# 抛出一个错误ValueError
raise ValueError(f"For 'ConfusionMatrix.update', predicted value (input[0]) and true value "
f"(input[1]) should have same dimensions, or the dimension of predicted value "
f"equals the dimension of true value add 1, but got predicted value ndim: "
f"{y_pred.ndim}, true value ndim: {y.ndim}.")
# 检查预测值的维度是否等于真实值的维度加1
if y_pred.ndim == y.ndim + 1:
# 如果是，则将预测值转换为概率分布，即使用np.argmax函数找到每个样本的预测类别，然后将预测类别转换为整数
y_pred = np.argmax(y_pred, axis=1)
# 检查预测值的维度是否等于真实值的维度，以及预测值的的数据类型是否为np.float16、np.float32或np.float64
if y_pred.ndim == y.ndim and y_pred.dtype in (np.float16, np.float32, np.float64):
# 如果是，则将预测值转换为整数，将大于等于self.threshold的预测值设置为1，其他值设置为0
y_pred = (y_pred >= self.threshold).astype(int)
# 接下来，将真实值和预测值按照num_classes进行拼接
trans = (y.reshape(-1) * self.num_classes + y_pred.reshape(-1)).astype(int)
# 然后使用np.bincount函数计算拼接后的结果的计数向量
bincount = np.bincount(trans, minlength=self.num_classes ** 2)
# 最后将计数向量转换为混淆矩阵
confusion_matrix = bincount.reshape(self.num_classes, self.num_classes)
# 将更新后的混淆矩阵赋值给self.confusion_matrix
self.confusion_matrix += confusion_matrix
# 并将self._is_update设置为True
self._is_update = True
def eval(self):
# 用于计算混淆矩阵，需先在调用update函数后调用
"""
Computes confusion matrix.
Returns:
numpy.ndarray, the computed result.
"""
# 首先检查self._is_update是否为True
if not self._is_update:
# 如果是，则抛出一个错误RuntimeError
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 然后，将混淆矩阵转换为浮点数类型
confusion_matrix = self.confusion_matrix.astype(float)
# 并计算目标、预测和所有类别下的混淆矩阵
matrix_target = confusion_matrix / confusion_matrix.sum(axis=1, keepdims=True)
matrix_pred = confusion_matrix / confusion_matrix.sum(axis=0, keepdims=True)
matrix_all = confusion_matrix / confusion_matrix.sum()
# 根据self.normalize的值，返回相应的混淆矩阵
normalize_dict = {"target": matrix_target,
"prediction": matrix_pred,
"all": matrix_all}
# 如果self.normalize设置为"no_norm"
if self.normalize == "no_norm":
# 则直接返回混淆矩阵
return confusion_matrix
# 从normalize_dict中获取self.normalize对应的混淆矩阵
matrix = normalize_dict.get(self.normalize)
# 检查矩阵中是否存在np.nan值
if matrix[np.isnan(matrix)].size != 0:
# 如果存在np.nan值，则将它们设置为0
matrix[np.isnan(matrix)] = 0
# 最后返回处理后的混淆矩阵
return matrix
class ConfusionMatrixMetric(Metric):
# 继承自Metric。该类主要用于计算与混淆矩阵相关的指标，基于全量张量，并收集批次、类通道和迭代过程中的指标值。
# 用户可以通过传入metric_name参数来选择支持 metrics 的名称，例如 "sensitivity"、"specificity"、"precision" 等。
# 同时，用户还可以通过传入calculation_method参数来选择是否计算每个样本的指标值，以及使用哪种 reduction 方法来减少数据批次
r"""
Computes metrics related to confusion matrix. The calculation based on full-scale tensor, average values of
batch, class channel and iteration are collected. All metrics supported by the interface are listed in comments
@ -188,29 +235,51 @@ class ConfusionMatrixMetric(Metric):
metric_name="sensitivity",
calculation_method=False,
decrease="mean"):
# 首先调用父类的__init__方法
super(ConfusionMatrixMetric, self).__init__()
"""
检查skip_channel、metric_name、calculation_method和decrease的类型
"""
# 创建一个名为_ConfusionMatrix的类对象，并传入相关参数，用于计算混淆矩阵
self.confusion_matrix = _ConfusionMatrix(skip_channel=skip_channel, metric_name=metric_name,
calculation_method=calculation_method, decrease=decrease)
# 检查skip_channel是否为bool值
self.skip_channel = validator.check_value_type("skip_channel", skip_channel, [bool])
# 检查calculation_method是否为bool值
self.calculation_method = validator.check_value_type("calculation_method", calculation_method, [bool])
# 检查metric_name是否为str类
self.metric_name = validator.check_value_type("metric_name", metric_name, [str])
# 将skip_channel、calculation_method、metric_name和decrease属性设置为类的属性
decrease_list = ["none", "mean", "sum", "mean_batch", "sum_batch", "mean_channel", "sum_channel"]
# 检查decrease是否为str类
decrease = validator.check_value_type("decrease", decrease, [str])
# 检查decrease的值是否在decrease_list中，如果在，则将其设置为类的属性decrease
self.decrease = validator.check_string(decrease, decrease_list, "decrease")
# 清除类中可能残余的数据
self.clear()
def clear(self):
# 用于清空类中的混淆矩阵计算结果数据
"""Clears the internal evaluation result."""
# 存储混淆矩阵的总量
self._total_num = 0
# 存储类通道数量
self._class_num = 0
# 存储真阳性
self._total_tp = 0.0
# 存储假阳性
self._total_fp = 0.0
# 存储真负性
self._total_tn = 0.0
# 存储假负性
self._total_fn = 0.0
@rearrange_inputs
def update(self, *inputs):
# 用于更新混淆矩阵的状态
# inputs参数接受两个张量y_pred和y，以及可能的其他参数。y_pred是一个张量，表示预测结果，可以是 onehot 格式或者 category index 格式。
# 对于二分类任务，y_pred的形状应该是 [N, C, ...] 或者 [N, ...]，其中 N 大于 1。
# 对于多分类任务，y_pred的形状应该是 [N, C, H, W] 或者 [N, H, W]。y是一个张量，表示真实标签，应该是 onehot 格式
"""
Update state with predictions and targets.
@ -224,47 +293,68 @@ class ConfusionMatrixMetric(Metric):
Raises:
ValueError: If the number of the inputs is not 2.
"""
# 检查inputs参数的数量是否为 2
if len(inputs) != 2:
# 如果不是，则抛出一个错误ValueError
raise ValueError("For 'ConfusionMatrixMetric.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 接着，将inputs[0]和inputs[1]转换为适当的数据格式
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 如果calculation_method为真
if self.calculation_method:
# 调用confusion_matrix方法计算混淆矩阵
score, not_nans = self.confusion_matrix(y_pred, y)
# 则计算平均值并累加到_total_num和_class_num中
# 注：not_nans是一个张量，表示非空值的数量
# 将结果与not_nans进行乘积
not_nans = int(not_nans.item())
# 并累加到_total_num和_class_num中
self._total_num += score.item() * not_nans
self._class_num += not_nans
else:
# 调用confusion_matrix方法计算混淆矩阵
confusion_matrix = self.confusion_matrix(y_pred, y)
# 否则，直接累加混淆矩阵的各个元素
# 最后，根据decrease方法对混淆矩阵进行处理
confusion_matrix, _ = _decrease_metric(confusion_matrix, "sum")
# 将混淆矩阵的各个元素累加到_total_tp、_total_fp、_total_tn和_total_fn中
self._total_tp += confusion_matrix[0].item()
self._total_fp += confusion_matrix[1].item()
self._total_tn += confusion_matrix[2].item()
self._total_fn += confusion_matrix[3].item()
def eval(self):
# 用于计算混淆矩阵，需先在调用update函数后调用
"""
Computes confusion matrix metric.
Returns:
ndarray, the computed result.
"""
# 检查calculation_method是否为真
if self.calculation_method:
# 如果是，则检查_class_num是否为 0
if self._class_num == 0:
# 如果为 0，则抛出一个错误RuntimeError
raise RuntimeError("The 'ConfusionMatrixMetric' can not be calculated, because the number of samples "
"is 0, please check whether your inputs(predicted value, true value) are empty, or "
"has called update method before calling eval method.")
# 如果非 0，则计算平均值并返回
return self._total_num / self._class_num
# 创建了一个 numpy 数组confusion_matrix，其中包含四个元素：self._total_tp、self._total_fp、self._total_tn和self._total_fn。这个数组表示混淆矩阵，用于计算分类指标
confusion_matrix = np.array([self._total_tp, self._total_fp, self._total_tn, self._total_fn])
# 否则，计算混淆矩阵并返回
return _compute_confusion_matrix_metric(self.metric_name, confusion_matrix)
class _ConfusionMatrix:
# 用于计算混淆矩阵相关指标。类有两个参数：skip_channel和metric_name。skip_channel用于判断是否跳过第一个通道的计算。
# metric_name用于定义指标的名称，可以设置为 industries 通用的指标名称或行业特定指标名称。
# calculation_method用于判断是否计算每个样本的测量值。如果为真，则计算每个样本的测量值并累加；否则，直接返回混淆矩阵。
# decrease参数用于定义 reduction 方法，用于减少计算结果的一个 batch
"""
Compute confusion matrix related metrics.
@ -286,13 +376,20 @@ class _ConfusionMatrix:
def __init__(self, skip_channel=True, metric_name="hit_rate", calculation_method=False,
decrease="mean"):
# 初始化其属性
super().__init__()
# skip_channel用于判断是否跳过第一个通道的计算，默认为 True
self.skip_channel = skip_channel
# metric_name用于定义指标的名称，默认为 "hit_rate"
self.metric_name = metric_name
# calculation_method用于判断是否计算每个样本的测量值，默认为 False
self.calculation_method = calculation_method
# decrease参数用于定义 reduction 方法，默认为 "mean"
self.decrease = decrease
def __call__(self, y_pred, y):
# 用于计算混淆矩阵相关指标
# 接收两个参数：y_pred和y。y_preds是输入数据，预期为二进制格式，第一维是批次。y是真实值，预期为二进制格式，第一维是批次。
"""
'y_preds' is expected to have binarized predictions and 'y' should be in one-hot format.
@ -304,41 +401,66 @@ class _ConfusionMatrix:
ValueError: If `metric_name` is empty.
ValueError: when `y_pred` has less than two dimensions.
"""
# 检查y是否为二进制格式
if not np.all(y.astype(np.uint8) == y):
# 如果不是，则抛出一个错误ValueError
raise ValueError("For 'ConfusionMatrix.update', the true value (input[1]) should be a binarized ndarray.")
# 计算y_pred的维度（dims），然后将其存储在变量dims中
dims = y_pred.ndim
# 接着，检查y_pred是否具有至少两个维度
if dims < 2:
# 如果不是，则抛出一个错误ValueError
raise ValueError(f"For 'ConfusionMatrix.update', the predicted value (input[0]) should have at least 2 "
f"dimensions, but got {dims}.")
# 如果dims小于2或（dims等于3且y_pred的最后一个维度为1）
if dims == 2 or (dims == 3 and y_pred.shape[-1] == 1):
# 如果计算每个样本的测量值
if self.calculation_method:
# 则将calculation_method设置为False
self.calculation_method = False
# 调用_get_confusion_matrix方法计算混淆矩阵
confusion_matrix = _get_confusion_matrix(y_pred=y_pred, y=y, skip_channel=self.skip_channel)
# 是否计算每个样本的测量值
if self.calculation_method:
# 如果是，则检查metric_name是否为字符串
if isinstance(self.metric_name, str):
# 如果是，则调用_compute_confusion_matrix_metric方法计算混淆矩阵指标
sub_confusion_matrix = _compute_confusion_matrix_metric(self.metric_name, confusion_matrix)
# 并调用_decrease_metric方法对结果进行处理
chart, not_nans = _decrease_metric(sub_confusion_matrix, self.decrease)
# 返回处理后的混淆矩阵指标chart和未处理的混淆矩阵指标not_nans
return chart, not_nans
# 如果metric_name为空，则抛出一个错误ValueError
if not self.metric_name:
raise ValueError("For 'ConfusionMatrix', the argument 'metric_name' cannot be None.")
# 创建results数组用于存储计算结果
results = []
# 遍历self.metric_name中的每个指标名称
for metric_name in self.metric_name:
# 调用_compute_confusion_matrix_metric方法计算混淆矩阵指标
sub_confusion_matrix = _compute_confusion_matrix_metric(metric_name, confusion_matrix)
# 调用_decrease_metric方法对结果进行处理
chart, not_nans = _decrease_metric(sub_confusion_matrix, self.decrease)
# 最后，将处理后的结果添加到results列表中
results.append(chart)
results.append(not_nans)
# 返回计算后的结果
return results
# 返回混淆矩阵
return confusion_matrix
def _get_confusion_matrix(y_pred, y, skip_channel=True):
# 接受两个参数：y_pred和y，以及一个可选参数skip_channel。y_pred是一个输入数据，应该是一个one-hot格式的数组，第一维是批量大小。
# y是一个目标标签，也应该是一个one-hot格式的数组，第一维是批量大小。skip_channel是一个布尔参数，用于指定是否跳过第一个通道的计算。
# 默认情况下，skip_channel为True
"""
The confusion matrix is calculated. An array of shape [BC4] is returned. The third dimension represents each channel
of each sample in the input batch.Where B is the batch size and C is the number of classes to be calculated.
@ -354,79 +476,121 @@ def _get_confusion_matrix(y_pred, y, skip_channel=True):
Raises:
ValueError: when `y_pred` and `y` have different shapes.
"""
# 根据skip_channel的值，跳过第一个通道的计算
if not skip_channel:
# 如果是false，则根据y.shape[1]的大小，如果大于1，则跳过第一个通道，否则不跳过
y = y[:, 1:] if y.shape[1] > 1 else y
# 然后，根据y_pred.shape[1]的大小，如果大于1，则跳过第一个通道，否则不跳过
y_pred = y_pred[:, 1:] if y_pred.shape[1] > 1 else y_pred
# 将y和y_pred转换为浮点数类型
y = y.astype(float)
y_pred = y_pred.astype(float)
# 检查y_pred和y是否有不同的形状，如果有不同的形状，则抛出一个ValueError异常
validator.check('y_shape', y.shape, 'y_pred_shape', y_pred.shape)
# 计算y_pred和y的批量大小batch_size和类别数量n_class
batch_size, n_class = y_pred.shape[:2]
# 然后，将y_pred和y调整为新的形状，其中第二维是类别数
y_pred = y_pred.reshape(batch_size, n_class, -1)
y = y.reshape(batch_size, n_class, -1)
# 接下来，计算真阳性（tp）、真负例（tn）、假阳性（fp）和假负例（fn）的数量
tp = ((y_pred + y) == 2).astype(float)
tn = ((y_pred + y) == 0).astype(float)
tp = tp.sum(axis=2)
tn = tn.sum(axis=2)
# 最后，计算准确率（accuracy）、精确度（precision）、召回率（recall）和F1分数（f1_score）
p = y.sum(axis=2)
n = y.shape[-1] - p
fn = p - tp
fp = n - tn
# 将计算出的真阳性、假阳性、真负例和假负例数量封装到一个形状为[batch_size, n_class, 4]的数组中，并返回
return np.stack([tp, fp, tn, fn], axis=-1)
def _decrease_mean(not_nans, chart):
# 计算不缺失值的数量和图表的平均值
# 函数接受两个参数：not_nans和chart。not_nans是一个形状为[batch_size, n_class]的数组，用于存储不缺失值的数量。
# chart是一个形状为[batch_size, n_class]的数组，用于存储图表的平均值
# 首先计算not_nans的行和列之和
not_nans = not_nans.sum(axis=1)
# 然后将结果除以不缺失值的数量，并将结果存储在chart中
chart = np.where(not_nans > 0, chart.sum(axis=1) / not_nans, np.zeros(1, dtype=float))
# 将not_nans转换为布尔数组，然后将布尔数组转换为浮点数数组
not_nans = (not_nans > 0).astype(float).sum(axis=0)
# 接着，计算沿着列方向的和，并将结果除以不缺失值的数量，并将结果存储在chart中
chart = np.where(not_nans > 0, chart.sum(axis=0) / not_nans, np.zeros(1, dtype=float))
# 最后，返回not_nans和chart
return not_nans, chart
def _decrease_sum(not_nans, chart):
# 计算不缺失值的数量和图表的总和
# 计算not_nans的行和列之和，并将结果存储在not_nans中
not_nans = not_nans.sum(axis=(0, 1))
# 接着，计算chart的行和列之和，并将结果存储在chart中
chart = np.sum(chart, axis=(0, 1))
# 最后，返回not_nans和chart
return not_nans, chart
def _decrease_mean_batch(not_nans, chart):
# 批计算不缺失值的数量和图表的平均值
# 首先计算not_nans的行之和，然后将结果除以不缺失值的数量，并将结果存储在chart中
not_nans = not_nans.sum(axis=0)
# 接着，计算不缺失值的总数，并将结果除以不缺失值的数量，并将结果存储在chart中
chart = np.where(not_nans > 0, chart.sum(axis=0) / not_nans, np.zeros(1, dtype=float))
# 最后，返回not_nans和chart
return not_nans, chart
def _decrease_sum_batch(not_nans, chart):
# 批计算不缺失值的数量和图表的总和
# 计算not_nans的行之和，并将结果存储在not_nans中
not_nans = not_nans.sum(axis=0)
# 接着，计算chart的行之和，并将结果存储在chart中
chart = chart.sum(axis=0)
# 最后，返回not_nans和chart
return not_nans, chart
def _decrease_mean_channel(not_nans, chart):
# 通道法计算不缺失值的数量和图表的平均值
# 首先计算not_nans的列之和，然后将结果除以不缺失值的数量，并将结果存储在chart中
not_nans = not_nans.sum(axis=1)
# 接着，计算不缺失值的总数，并将结果除以不缺失值的数量，并将结果存储在chart中
chart = np.where(not_nans > 0, chart.sum(axis=1) / not_nans, np.zeros(1, dtype=float))
# 最后，返回not_nans和chart
return not_nans, chart
def _decrease_sum_channel(not_nans, chart):
# 通道法计算不缺失值的数量和图表的总和
# 计算not_nans的列之和，并将结果存储在not_nans中
not_nans = not_nans.sum(axis=1)
# 接着，计算chart的列之和，并将结果存储在chart中
chart = chart.sum(axis=1)
# # 最后，返回not_nans和chart
return not_nans, chart
def _decrease_none(not_nans, chart):
# 啥也不干地计算不缺失值的数量和图表
# 函数直接返回not_nans和chart，不做任何处理
return not_nans, chart
def _decrease_metric(chart, decrease="mean"):
# 用于减少计算指标。函数接受两个参数：chart和decrease。
# chart是一个形状为[batch_size, n_class]的数组，用于存储计算出的测量评分。
# decrease是一个字符串，定义了减少计算结果的方法。默认值为"mean"
"""
This function is used to reduce the calculated metrics for each class of each example.
@ -437,10 +601,15 @@ def _decrease_metric(chart, decrease="mean"):
when 'calculation_method' is True. Default: "mean".
"""
# 首先计算chart中缺失值的位置，并将结果存储在nans中
nans = np.isnan(chart)
# 接着，将nans转换为布尔数组，，然后将布尔数组转换为浮点数数组，接着，计算not_nans的行和列之和，并将结果存储在not_nans中
not_nans = (~nans).astype(float)
# 将chart中缺失值的位置设置为0
chart[nans] = 0
# 然后，定义了一个名为decrease_dict的字典，用于存储不同的减少方法。decrease_dict的键是减少方法的字符串，值是对应的函数。
# _decrease_mean、_decrease_sum、_decrease_mean_batch、_decrease_sum_batch、
# _decrease_mean_channel和_decrease_sum_channel是这些函数在上方的具体实现
decrease_dict = {"mean": _decrease_mean(not_nans, chart),
"sum": _decrease_sum(not_nans, chart),
"mean_batch": _decrease_mean_batch,
@ -448,125 +617,177 @@ def _decrease_metric(chart, decrease="mean"):
"mean_channel": _decrease_mean_channel(not_nans, chart),
"sum_channel": _decrease_sum_channel(not_nans, chart),
"none": _decrease_none(not_nans, chart)}
# 最后，根据decrease的值，调用相应的减少方法，并将结果存储在not_nans和chart中
not_nans, chart = decrease_dict.get(decrease)
# 最后，返回chart和not_nans
return chart, not_nans
"""
以下这些函数都接受两个参数：tp和p（或tn和n），用于计算不同类型的性能指标
"""
def _calculate_tpr(tp, p):
# 计算True Positive Rate (TPR)性能指标
"""Calculate tpr."""
return tp, p
def _calculate_tnr(tn, n):
# 计算True Negative Rate (TNR)性能指标
"""Calculate tnr."""
return tn, n
def _calculate_ppv(tp, fp):
# 计算Positive Predictive Value (PPV)性能指标
"""Calculate ppv."""
return tp, (tp + fp)
def _calculate_npv(tn, fn):
# 计算Negative Predictive Value (NPV)性能指标
"""Calculate npv."""
return tn, (tn + fn)
def _calculate_fnr(fn, p):
# 计算False Negative Rate (FNR)性能指标
"""Calculate fnr."""
return fn, p
def _calculate_fpr(fp, n):
# 计算False Positive Rate (FPR）性能指标
"""Calculate fpr."""
return fp, n
def _calculate_fdr(tp, fp):
# 计算False Discovery Rate (FDR)性能指标
"""Calculate fdr."""
return fp, (fp + tp)
def _calculate_for(tn, fn):
# 计算False Omission Rate (FOR)性能指标
"""Calculate for."""
return fn, (fn + tn)
def _calculate_pt(tp, tn, p, n):
# 计算Precision (PT)性能指标
"""Calculate pt."""
# 首先使用np.where函数计算真阳率（TPR）和真阴率（TNR）
tpr = np.where(p > 0, tp / p, np.array(float("nan")))
tnr = np.where(n > 0, tn / n, np.array(float("nan")))
# 然后计算分子（numerator）和分母（denominator）
numerator = np.sqrt(tpr * (1.0 - tnr)) + tnr - 1.0
denominator = tpr + tnr - 1.0
# 最后，返回分子和分母的平方根，以计算PT值
return numerator, denominator
def _calculate_ts(tp, fp, fn):
# 计算Recall (TS)性能指标
"""Calculate ts."""
return tp, (tp + fn + fp)
def _calculate_acc(tp, tn, p, n):
# 计算Accuracy (ACC)性能指标
"""Calculate acc."""
return (tp + tn), (p + n)
def _calculate_ba(tp, tn, p, n):
# 计算Balanced Accuracy (BA)性能指标
"""Calculate ba."""
# 首先，计算True Positive Rate (TPR)
tpr = np.where(p > 0, tp / p, np.array(float("nan")))
# 接着，计算True Negative Rate (TNR)
tnr = np.where(n > 0, tn / n, np.array(float("nan")))
# 最后，计算numerator，denominator
numerator, denominator = (tpr + tnr), 2.0
# 返回numerator和denominator
return numerator, denominator
def _calculate_f1(tp, fp, fn):
# 计算F1-score (F1)性能指标
"""Calculate f1."""
return tp * 2.0, (tp * 2.0 + fn + fp)
def _calculate_mcc(tp, fp, tn, fn):
# 计算Matthews Correlation Coefficient (MCC)性能指标
"""Calculate mcc."""
# 首先，计算numerator
numerator = tp * tn - fp * fn
# 接着，计算denominator
denominator = np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
# 最后，返回numerator和denominator
return numerator, denominator
def _calculate_fm(tp, fp, p):
# 计算Fowlkes-Mallows Index (FM)性能指标
"""Calculate fm."""
# 首先，计算True Positive Rate (TPR)
tpr = np.where(p > 0, tp / p, np.array(float("nan")))
# 接着，计算准确率（Accuracy）
ppv = np.where((tp + fp) > 0, tp / (tp + fp), np.array(float("nan")))
# 最后，计算numerator
numerator = np.sqrt(ppv * tpr)
# 设置denominator = 1.0
denominator = 1.0
# 最后，返回numerator和denominator
return numerator, denominator
def _calculate_bm(tp, tn, p, n):
# 计算Brier Score (BM)性能指标
"""Calculate bm."""
# 首先，计算True Positive Rate (TPR)
tpr = np.where(p > 0, tp / p, np.array(float("nan")))
# 接着，计算True Negative Rate (TNR)
tnr = np.where(n > 0, tn / n, np.array(float("nan")))
# 最后，计算numerator
numerator = tpr + tnr - 1.0
# 设置denominator = 1.0
denominator = 1.0
# 最后，返回numerator和denominator
return numerator, denominator
def _calculate_mk(tp, fp, tn, fn):
# 计算Matthews Correlation Coefficient (MCC)性能指标
"""Calculate mk."""
# 首先，计算准确率（Accuracy）
ppv = np.where((tp + fp) > 0, tp / (tp + fp), np.array(float("nan")))
# 接着，计算假正例率（False Positive Rate）
npv = np.where((tn + fn) > 0, tn / (tn + fn), np.array(float("nan")))
# 计算假负例率（False Negative Rate）
npv = tn / (tn + fn)
# 接下来，计算numerator
numerator = ppv + npv - 1.0
# 最后，计算denominator
denominator = 1.0
# 最后，返回numerator和denominator
return numerator, denominator
def _compute_confusion_matrix_metric(metric_name, confusion_matrix):
# 该函数用于计算与混淆矩阵相关的度量
# metric_name (字符串)：表示要计算的混淆矩阵的指标名称。混淆矩阵是一种用于评估分类器性能的指标，它将实际的分类结果与预测的分类结果进行比较。
# 不同的指标可能有不同的别名，例如“准确率”（Accuracy）和“正确率”（Correctness）等。您可以输入这些名称之一，或者直接输入混淆矩阵的指标名称。
# confusion_matrix (数组)：表示要计算的混淆矩阵。这是一个numpy数组，其中包含实际分类结果和预测分类结果。
# _get_confusion_matrix是一个函数，用于从给定的实际分类结果和预测分类结果中计算混淆矩阵。
"""
This function is used to compute confusion matrix related metric.
@ -580,23 +801,31 @@ def _compute_confusion_matrix_metric(metric_name, confusion_matrix):
NotImplementedError: when specify a not implemented metric_name.
"""
# 首先检查metric_name是否有效
metric = _check_metric_name(metric_name)
# 然后计算混淆矩阵的维度
input_dim = confusion_matrix.ndim
# 如果维度为1
if input_dim == 1:
# 则扩展最后一个维度以使其为4
confusion_matrix = np.expand_dims(confusion_matrix, 0)
# 检查混淆矩阵的最后一个维度是否为4
if confusion_matrix.shape[-1] != 4:
# 如果不是，则抛出一个 ValueError 异常，提示用户混淆矩阵的维度不正确
raise ValueError(f"For 'ConfusionMatrix', the size of the last dimension of confusion_matrix should be 4, "
f"but got {confusion_matrix.shape[-1]}.")
# 接下来，从混淆矩阵中提取真阳例（tp）、假阳例（fp）、真阴例（tn）和假阴例（fn）
tp = confusion_matrix[..., 0]
fp = confusion_matrix[..., 1]
tn = confusion_matrix[..., 2]
fn = confusion_matrix[..., 3]
# 并计算正例（p）和负例（n）
p = tp + fn
n = fp + tn
# 定义了一个字典metric_name_dict，其中包含了各种指标的计算函数
metric_name_dict = {"tpr": _calculate_tpr(tp, p),
"tnr": _calculate_tnr(tn, n),
"ppv": _calculate_ppv(tp, fp),
@ -614,16 +843,23 @@ def _compute_confusion_matrix_metric(metric_name, confusion_matrix):
"fm": _calculate_fm(tp, fp, p),
"bm": _calculate_bm(tp, tn, p, n),
"mk": _calculate_mk(tp, fp, tn, fn)}
# 从metric_name_dict字典中获取输入的metric_name对应的计算结果，并将结果存储在numerator和denominator变量中
numerator, denominator = metric_name_dict.get(metric)
# 检查denominator是否是一个numpy数组
if isinstance(denominator, np.ndarray):
# 使用numpy的where函数对denominator进行处理。处理后的结果将赋值给result变量
result = np.where(denominator != 0, numerator / denominator, np.array(float("nan")))
else:
# 直接计算numerator除以denominator的结果，并将结果存储在result变量中
result = numerator / denominator
# 返回result变量
return result
def _check_metric_name(metric_name):
# 这个函数主要用于处理混淆矩阵中的指标名称，使它们更加简洁
"""
There are many metrics related to confusion matrix, and some of the metrics have more than one names. In addition,
some of the names are very long. Therefore, this function is used to check and simplify the name.
@ -634,8 +870,11 @@ def _check_metric_name(metric_name):
Raises:
NotImplementedError: when the metric is not implemented.
"""
# 首先将指标名称中的空格替换为下划线
metric_name = metric_name.replace(" ", "_")
# 并将名称转换为小写
metric_name = metric_name.lower()
# 然后，使用一个字典metric_name_dict来映射指标名称到简化后的名称
metric_name_dict = {"sensitivity": "tpr",
"recall": "tpr",
"hit_rate": "tpr",
@ -682,9 +921,10 @@ def _check_metric_name(metric_name):
"markedness": "mk",
"deltap": "mk",
"mk": "mk"}
# 最后，函数从字典中获取指标的简化的名称，并将其返回
metric_name_info = metric_name_dict.get(metric_name)
# 如果在字典中找不到匹配的指标名称，那么函数将抛出一个NotImplementedError异常，表示该指标尚未实现
if metric_name_info is None:
raise NotImplementedError("The metric is not implemented.")

47
mindspore/python/mindspore/nn/metrics/cosine_similarity.py
View File

@ -13,12 +13,27 @@
# limitations under the License.
# ============================================================================
"""CosineSimilarity."""
# CosineSimilarity模块实现了两个函数：cosine_similarity和_checkparam。
# cosine_similarity函数用于计算两个向量之间的余弦相似度，_checkparam函数用于验证参数是否符合要求
# 导入numpy库，用于处理矩阵和向量计算
import numpy as np
# 从mindspore库中导入Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 从当前模块中导入Metric类和rearrange_inputs函数
from .metric import Metric, rearrange_inputs
class CosineSimilarity(Metric):
# 用于计算两个向量之间的余弦相似度
"""
参数：
similarity (str)：可选为'dot'或'cosine'，表示计算余弦相似度时使用的公式。默认值为'cosine'。
reduction (str)：表示计算结果 reduction的方式。可选为'none'、'sum'或'mean'，分别表示不进行reduction、按列求和、按列求平均。默认值为'none'。
zero_diagonal (bool)：表示是否将矩阵对角线上的值设置为0。默认值为True。
"""
"""
Computes representation similarity.
@ -45,41 +60,63 @@ class CosineSimilarity(Metric):
[0.95162452 0.86146098 0.]]
"""
def __init__(self, similarity='cosine', reduction='none', zero_diagonal=True):
# 初始化类参数
# 调用父类的构造函数
super().__init__()
# 定义两个列表similarity_list和reduction_list，分别包含可选的余弦相似度和reduction方式，用于后续验证
similarity_list = ['dot', 'cosine']
reduction_list = ['none', 'sum', 'mean']
# 检查similarity参数是否为字符串，并在similarity_list中进行验证
similarity = validator.check_value_type("similarity", similarity, [str])
self.similarity = validator.check_string(similarity, similarity_list, "similarity")
# 检查reduction参数是否为字符串，并在reduction_list中进行验证
reduction = validator.check_value_type("reduction", reduction, [str])
self.reduction = validator.check_string(reduction, reduction_list, "reduction")
# 检查zero_diagonal参数是否为布尔值
self.zero_diagonal = validator.check_value_type("zero_diagonal", zero_diagonal, [bool])
# 初始化一个变量self.sqr_mtx_res，用于存储计算结果
self.sqr_mtx_res = 0
# 清空类中残余的数据
self.clear()
# 设置一个标志位self._is_update，用于判断是否需要更新计算结果，初始为否
self._is_update = None
def clear(self):
# 用于清空类中的数据
"""Clears the internal evaluation result."""
# 将计算结果设置为0
self.sqr_mtx_res = 0
# 表示未更新计算结果
self._is_update = False
@rearrange_inputs
def update(self, inputs):
# 用于更新内部评估结果
"""
Updates the internal evaluation result with 'inputs'.
Args:
inputs (Union[Tensor, list, numpy.ndarray]): The input matrix.
"""
# 使用_convert_data函数将输入数据转换为numpy数组
input_data = self._convert_data(inputs)
# 如果self.similarity为'cosine'，则计算每个输入向量的二范数，然后将输入向量除以相应的expand_dims
# 注：expand_dims函数为输入数组添加一个新维度
if self.similarity == 'cosine':
# 计算每个输入向量的二范数
data = np.linalg.norm(input_data, ord=2, axis=1)
# 将输入向量除以相应的expand_dims
input_data = input_data / np.expand_dims(data, 1)
# 更新self.sqr_mtx_res为输入向量与其转置的点积
self.sqr_mtx_res = np.dot(input_data, input_data.transpose(1, 0))
# 设置self._is_update为True，表示已经更新了计算结果
self._is_update = True
def eval(self):
# 用于计算相似度矩阵，通常在update函数后调用
"""
Computes the similarity matrix.
@ -89,16 +126,26 @@ class CosineSimilarity(Metric):
Raises:
RuntimeError: If the update method is not called first, an error will be reported.
"""
# 首先检查是否已经调用过update方法
if not self._is_update:
# 如果没有，则抛出一个RuntimeError
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 如果self.zero_diagonal为True
if self.zero_diagonal:
# 则将相似度矩阵的对角线上的值设置为0
np.fill_diagonal(self.sqr_mtx_res, 0)
# 如果self.reduction为'mean'（平均）
if self.reduction == 'mean':
# 则计算相似度矩阵的列平均值
self.sqr_mtx_res = np.mean(self.sqr_mtx_res, axis=-1)
# 如果self.reduction为'sum'（求和）
if self.reduction == 'sum':
# 则计算相似度矩阵的列和
self.sqr_mtx_res = np.sum(self.sqr_mtx_res, axis=-1)
# 返回计算得到的相似度矩阵
return self.sqr_mtx_res

37
mindspore/python/mindspore/nn/metrics/dice.py
View File

@ -13,12 +13,19 @@
# limitations under the License.
# ============================================================================
"""Dice"""
# 继承自Metric类。Dice类主要用于计算Dice系数，是一种常用的评估方法，用于衡量两个集合之间的相似度
# 导入numpy库，用于处理矩阵运算
import numpy as np
# 从mindspore库中导入_checkparam.Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 从当前模块中导入Metric类，用于继承
from .metric import Metric, rearrange_inputs
class Dice(Metric):
# 用于衡量两个集合之间的相似度。Dice系数计算公式如下：dice = \frac{2 * (pred \bigcap true)}{pred \bigcup true}
# 其中，pred表示预测结果，true表示真实结果；smooth是一个平滑参数，用于提高数值稳定性。
r"""
The Dice coefficient is a set similarity metric. It is used to calculate the similarity between two samples. The
value of the Dice coefficient is 1 when the segmentation result is the best and is 0 when the segmentation result
@ -50,20 +57,32 @@ class Dice(Metric):
"""
def __init__(self, smooth=1e-5):
# 初始化类参数
# 调用父类Metric的__init__方法，传入平滑参数smooth
super(Dice, self).__init__()
# 初始化self.smooth为正浮点数，用于计算Dice系数，并检查其是否为正浮点数
self.smooth = validator.check_positive_float(smooth, "smooth")
# 用于存储Dice系数的和
self._dice_coeff_sum = 0
# 用于存储样本数量
self._samples_num = 0
# 清空类中残余的数据
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 将self._dice_coeff_sum重置为0
self._dice_coeff_sum = 0
# 将self._samples_num的值重置为0
self._samples_num = 0
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果
# 接受两个输入：预测值y_pred和真实值y。预测值和真实值的形状都是(N, ...)
"""
Updates the internal evaluation result :math:`y_pred` and :math:`y`.
@ -75,26 +94,39 @@ class Dice(Metric):
ValueError: If the number of the inputs is not 2.
ValueError: If y_pred and y do not have the same shape.
"""
"""
首先获取输入参数并检查
"""
# 检查输入数量是否为2
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError
raise ValueError("For 'Dice.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 获取输入数组中的值并将输入的预测值和真实值转换为numpy.ndarray类型
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 更新self._samples_num，将输入的样本数量累加到self._samples_num中
self._samples_num += y.shape[0]
# 检查预测值和真实值的形状是否相同
if y_pred.shape != y.shape:
# 如果不相同，则抛出一个ValueError
raise ValueError(f"For 'Dice.update', predicted value (input[0]) and true value (input[1]) "
f"should have same shape, but got predicted value shape: {y_pred.shape}, "
f"true value shape: {y.shape}.")
# 首先，将预测值y_pred和真实值y转换为1D数组，然后使用numpy的dot函数计算交集和并集
intersection = np.dot(y_pred.flatten(), y.flatten())
unionset = np.dot(y_pred.flatten(), y_pred.flatten()) + np.dot(y.flatten(), y.flatten())
# 计算Dice系数
single_dice_coeff = 2 * float(intersection) / float(unionset + self.smooth)
# 将单个Dice系数累加到self._dice_coeff_sum中
self._dice_coeff_sum += single_dice_coeff
def eval(self):
# 用于计算Dice系数，通常在update函数后调用
r"""
Computes the Dice.
@ -104,9 +136,12 @@ class Dice(Metric):
Raises:
RuntimeError: If the total number of samples is 0.
"""
# 首先检查self._samples_num是否为0
if self._samples_num == 0:
# 如果是，则抛出一个RuntimeError
raise RuntimeError("The 'Dice coefficient' can not be calculated, because the number of samples is 0, "
"please check whether your inputs(predicted value, true value) are empty, or has "
"called update method before calling eval method.")
# 然后，计算Dice系数的和，并将其除以self._samples_num，得到最终的结果
return self._dice_coeff_sum / float(self._samples_num)

39
mindspore/python/mindspore/nn/metrics/error.py
View File

@ -13,11 +13,16 @@
# limitations under the License.
# ============================================================================
"""Error."""
# 定义了两个类，分别是MAE（均方误差）和MSE（均方误差）。这两个类都继承自Metric类，用于计算预测值和真实值之间的相关指标
# 导入了numpy库，，用于处理矩阵和向量计算
import numpy as np
# 导入Metric类，Metric类是用于计算评估指标的基类，rearrange_inputs是用于重新排列输入的辅助函数
from .metric import Metric, rearrange_inputs
class MAE(Metric):
# MAE类用于计算预测值和真实值之间的平均绝对误差（MAE），即每个元素预测值和真实值之间的绝对差值的平均值
# 平均绝对误差（MAE）是衡量模型预测结果与真实值之间差异程度的一种指标，数值越小，表示模型预测结果与真实值越接近
r"""
Calculates the mean absolute error(MAE).
@ -47,16 +52,22 @@ class MAE(Metric):
0.037499990314245224
"""
def __init__(self):
# 构造函数，接受一个平滑参数smooth，用于计算Dice系数
super(MAE, self).__init__()
# 清空类中残余的数据
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# _abs_error_sum用于存储累计的绝对误差和，重置其为0
self._abs_error_sum = 0
# _samples_num用于存储样本数量，重置其为0
self._samples_num = 0
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果。接受两个输入：预测值y_pred和真实值y
"""
Updates the internal evaluation result :math:`y_{pred}` and :math:`y`.
@ -67,16 +78,23 @@ class MAE(Metric):
Raises:
ValueError: If the number of the input is not 2.
"""
# 通过len(inputs) != 2检查输入是否只有两个
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError
raise ValueError("For 'MAE.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 然后，获取输入值并将输入转换为numpy数组
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 并计算预测值和真实值之间的绝对差值
abs_error_sum = np.abs(y.reshape(y_pred.shape) - y_pred)
# 将绝对差值的累加和self._abs_error_sum相加
self._abs_error_sum += abs_error_sum.sum()
# 并将样本数量累加到self._samples_num中
self._samples_num += y.shape[0]
def eval(self):
# 用于计算MAE
"""
Computes the mean absolute error(MAE).
@ -86,14 +104,19 @@ class MAE(Metric):
Raises:
RuntimeError: If the total number of samples is 0.
"""
# 首先检查self._samples_num是否为0
if self._samples_num == 0:
# 如果是，则抛出一个RuntimeError
raise RuntimeError("The 'MAE' can not be calculated, because the number of samples is 0, "
"please check whether your inputs(predicted value, true value) are empty, "
"or has called update method before calling eval method.")
# 然后计算Dice系数的和，并将其除以self._samples_num，得到最终的结果
return self._abs_error_sum / self._samples_num
class MSE(Metric):
# MSE类用于计算预测值和真实值之间的均方误差（MSE），即每个元素预测值和真实值之间的平方差之和的平均值。
# 均方误差（MSE）是衡量模型预测结果与真实值之间差异程度的一种指标，数值越小，表示模型预测结果与真实值越接近
r"""
Measures the mean squared error(MSE).
@ -123,16 +146,22 @@ class MSE(Metric):
0.0031250009778887033
"""
def __init__(self):
# 构造函数，与MAE类相同，接受一个平滑参数smooth，用于计算Dice系数。
super(MSE, self).__init__()
# 清空类中残余的数据
self.clear()
def clear(self):
# 与MAE类相同，用于清空内部评估结果
"""Clear the internal evaluation result."""
# _abs_error_sum用于存储累计的绝对误差和，重置其为0
self._squared_error_sum = 0
# _samples_num用于存储样本数量，重置其为0
self._samples_num = 0
@rearrange_inputs
def update(self, *inputs):
# 与MAE类相同，用于更新内部评估结果。接受两个输入：预测值y_pred和真实值y
"""
Updates the internal evaluation result :math:`y_{pred}` and :math:`y`.
@ -143,17 +172,24 @@ class MSE(Metric):
Raises:
ValueError: If the number of inputs is not 2.
"""
# 通过len(inputs) != 2检查输入是否只有两个
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError
raise ValueError("For 'MSE.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 然后，获取输入值并将输入转换为numpy数组
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 计算了预测值和真实值之间的平方差
squared_error_sum = np.power(y.reshape(y_pred.shape) - y_pred, 2)
# 并将平方差累加和self._squared_error_sum相加
self._squared_error_sum += squared_error_sum.sum()
# 并将样本数量累加到self._samples_num中
self._samples_num += y.shape[0]
def eval(self):
# 用于计算MSE，通常在update函数后调用
"""
Computes the mean squared error(MSE).
@ -163,8 +199,11 @@ class MSE(Metric):
Raises:
RuntimeError: If the number of samples is 0.
"""
# 首先检查self._samples_num是否为0
if self._samples_num == 0:
# 如果是，则抛出一个RuntimeError
raise RuntimeError("The 'MSE' can not be calculated, because the number of samples is 0, "
"please check whether your inputs(predicted value, true value) are empty, "
"or has called update method before calling eval method.")
# 然后计算平方误差的总和，并将其除以self._samples_num，得到最终的结果
return self._squared_error_sum / self._samples_num

56
mindspore/python/mindspore/nn/metrics/fbeta.py
View File

@ -13,13 +13,25 @@
# limitations under the License.
# ============================================================================
"""Fbeta."""
# 用于计算F分数。F分数是一种评估模型预测结果与真实值之间相关性的指标，可以用于衡量模型的精确率和召回率
# 导入了sys模块，用于处理Python运行时状态
import sys
# 导入了numpy库，用于处理数值计算
import numpy as np
# 导入了mindspore._checkparam.Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 导入了mindspore.metric.Metric类，用于计算评估指标
# 导入了_check_onehot_data函数，用于检查输入数据是否为one-hot编码。
# 导入了rearrange_inputs函数，用于处理输入数据，将其转换为numpy数组
from .metric import Metric, rearrange_inputs, _check_onehot_data
class Fbeta(Metric):
# 用于计算F分数。F分数是一种评估模型预测结果与真实值之间相关性的指标，可以用于衡量模型的精确率和召回率。
# Fbeta分数通过计算 precision 和 recall 的加权平均值来计算，其中beta是一个权重系数
r"""
Calculates the Fbeta score.
@ -49,22 +61,36 @@ class Fbeta(Metric):
[0.66666667 0.66666667]
"""
def __init__(self, beta):
# 在构造函数中，首先调用父类的构造函数
super(Fbeta, self).__init__()
# 然后初始化一个用于存储eps的变量self.eps
self.eps = sys.float_info.min
# 检查beta是否大于0
if not beta > 0:
# 如果不满足条件，则抛出一个ValueError异常
raise ValueError("For 'Fbeta', the argument 'beta' must be greater than 0, but got {}.".format(beta))
# 以及一个用于存储beta的变量self.beta
self.beta = beta
# 清空类中残余的数据
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 它将_true_positives的值重置为0
self._true_positives = 0
# 它将_actual_positives的值重置为0
self._actual_positives = 0
# 它将_positives的值重置为0
self._positives = 0
# 它将_class_num的值重置为0
self._class_num = 0
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果y_pred和y。inputs是一个元组，包含y_pred和y。
# y_pred通常是一个浮点数列表，范围在0到1之间，形状为(N, C)，
# 其中N是样本数量，C是类别数量。y是一个整数列表，形状为(N, C)或(N,)
"""
Updates the internal evaluation result `y_pred` and `y`.
@ -79,39 +105,60 @@ class Fbeta(Metric):
ValueError: class numbers of last input predicted data and current predicted data not match.
ValueError: If the predicted value and true value contain different classes.
"""
# 首先检查输入数据的数量是否为2
if len(inputs) != 2:
# 如果不满足条件，则抛出一个ValueError异常
raise ValueError("For 'Fbeta.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 然后获取输入值并将输入数据转换为numpy数组
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 并检查y_pred是否为one-hot编码
if y_pred.ndim == y.ndim and _check_onehot_data(y):
# 如果满足条件，将y_pred和y转换为二进制数组
y = y.argmax(axis=1)
# 检查_class_num是否为0
if self._class_num == 0:
# 如果是，则将其设置为y_pred的类别数量
self._class_num = y_pred.shape[1]
# 如果y_pred的类别数量与_class_num不相等
elif y_pred.shape[1] != self._class_num:
# 则抛出一个ValueError异常，提示用户检查输入的预测数据
raise ValueError("For 'Fbeta.update', class number not match, last input predicted data contain {} "
"classes, but current predicted data contain {} classes, please check your "
"predicted value(inputs[0]).".format(self._class_num, y_pred.shape[1]))
# 最后，将class_num设置为_class_num的值
class_num = self._class_num
# 首先检查y的最大值是否大于class_num
if y.max() + 1 > class_num:
# 如果是，则抛出一个ValueError异常，提示用户检查输入的预测数据和真实数据
raise ValueError("For 'Fbeta.update', predicted value(inputs[0]) and true value(inputs[1]) "
"should contain same classes, but got predicted value contains {} classes"
" and true value contains {} classes.".format(class_num, y.max() + 1))
# 接下来，将y转换为一个one-hot编码的数组
y = np.eye(class_num)[y.reshape(-1)]
# 并将y_pred转换为一个二进制数组
indices = y_pred.argmax(axis=1).reshape(-1)
# 最后，将y和y_pred调整为具有相同类别的形状
y_pred = np.eye(class_num)[indices]
# 计算y_pred中每个类别的真阳性数量
positives = y_pred.sum(axis=0)
# 计算y_pred中每个类别的正例数量
actual_positives = y.sum(axis=0)
# 计算y_pred中每个类别的实际正例数量
true_positives = (y * y_pred).sum(axis=0)
# 并将上述这些值添加到相应的_true_positives、_positives和_actual_positives属性中
# 这样，在计算F分数时，这些变量将用于存储新的评估结果
self._true_positives += true_positives
self._positives += positives
self._actual_positives += actual_positives
def eval(self, average=False):
# 用于计算F分数。函数接受一个布尔参数average，用于指定是否计算平均F分数，该函数需要在update函数后调用
"""
Computes the fbeta.
@ -121,21 +168,29 @@ class Fbeta(Metric):
Returns:
numpy.ndarray or numpy.float64, the computed result.
"""
# 首先，检查average参数的类型是否为布尔值，如果不满足条件，则抛出一个TypeError异常
validator.check_value_type("average", average, [bool], self.__class__.__name__)
# 然后，检查_class_num是否为0
if self._class_num == 0:
# 如果是，则抛出一个RuntimeError异常，提示用户检查输入的数据是否为空，或者是否已经调用了update方法
raise RuntimeError("The 'Fbeta' can not be calculated, because the number of samples is 0, "
"please check whether your inputs(predicted value, true value) are empty, "
"or has called update method before calling eval method.")
# 接下来，计算F分数
fbeta = (1.0 + self.beta ** 2) * self._true_positives / \
(self.beta ** 2 * self._actual_positives + self._positives + self.eps)
# 最后，如果average为True
if average:
# 则返回F分数的平均值
return fbeta.mean()
# 否则返回完整的F分数数组
return fbeta
class F1(Fbeta):
# 定义了一个名为F1的子类，继承自Fbeta类。F1类主要用于计算F1分数，它是Fbeta类的特例，当beta为1时成立
r"""
Calculates the F1 score. F1 is a special case of Fbeta when beta is 1.
Refer to class :class:`mindspore.nn.Fbeta` for more details.
@ -159,4 +214,5 @@ class F1(Fbeta):
[0.66666667 0.66666667]
"""
def __init__(self):
# F1分数的计算公式为：F_1=\frac{2\cdot true\_positive}{2\cdot true\_positive + false\
super(F1, self).__init__(1.0)

158
mindspore/python/mindspore/nn/metrics/hausdorff_distance.py
View File

@ -13,17 +13,29 @@
# limitations under the License.
# ============================================================================
"""HausdorffDistance."""
# 定义了一个名为HausdorffDistance的类，该类继承自collections.abc模块中的Sequence类，并实现了__call__方法。
# HausdorffDistance类的主要目的是计算两个图像之间的 Hausdorff 距离
# 首先，从abc模块导入Sequence类，这是 Python 的一个抽象基类，用于表示可迭代对象
from collections import abc
from abc import ABCMeta
# # 然后，从scipy.ndimage模块导入morphology函数，用于对图像进行形态学操作
from scipy.ndimage import morphology
# 接下来，导入numpy库，用于处理数值计算
import numpy as np
# Tensor是一个表示张量的类，用于在 MindSpore 系统中处理张量数据
from mindspore.common.tensor import Tensor
# Validator是一个用于验证参数的类，用于确保输入参数符合预期的格式和范围
from mindspore._checkparam import Validator as validator
# 从metric.py文件中导入Metric类和rearrange_inputs函数。Metric类是用于计算图像质量的类，而rearrange_inputs函数用于对输入参数进行重排，以适应 Metric 类的要求
from .metric import Metric, rearrange_inputs
class _ROISpatialData(metaclass=ABCMeta):
# 它继承自ABCMeta元类。这个类用于计算两个图像之间的 Hausdorff 距离。
# _ROISpatialData类的主要功能是支持对 ND 空间数据进行裁剪，即生成一个 ROI（区域）。ROI 是指一个包含特定区域的空间子集。
# ROISpatialData类通过使用metaclass=ABCMeta元类来声明其是一个抽象基类，这意味着不能直接实例化这个类。
# roi_center、roi_size、roi_start和roi_end是这个类的参数，用于指定 ROI 的中心、大小、开始和结束坐标。
"""
Produce Region Of Interest (ROI). Support to crop ND spatial data. The center and size of the space should be
provided, if not, the start and end coordinates of the ROI must be provided.
@ -37,19 +49,30 @@ class _ROISpatialData(metaclass=ABCMeta):
def __init__(self, roi_center=None, roi_size=None, roi_start=None, roi_end=None):
# 当提供roi_center和roi_size时
if roi_center is not None and roi_size is not None:
# roi_start和roi_end将由它们计算得到
# 首先将roi_center转换为np.int16类型的数组
roi_center = np.asarray(roi_center, dtype=np.int16)
# 首先将roi_size转换为np.int16类型的数组
roi_size = np.asarray(roi_size, dtype=np.int16)
# 然后np.maximum(roi_center - np.floor_divide(roi_size, 2), 0)计算的是roi_center减去roi_size的一半向下取整
self.roi_start = np.maximum(roi_center - np.floor_divide(roi_size, 2), 0)
# 最后np.maximum(self.roi_start + roi_size, self.roi_start)计算的是roi_start加上roi_size，如果结果大于self.roi_start，则取self.roi_start
self.roi_end = np.maximum(self.roi_start + roi_size, self.roi_start)
else:
# 当只提供roi_start和roi_end时，需要确保roi_center和roi_size为None
if roi_start is None or roi_end is None:
# 否则会抛出一个错误ValueError
raise ValueError("For 'HausdorffDistance.update', When either 'roi_center' or 'roi_size' is None,"
"neither 'roi_start' nor 'roi_end' can be None.")
# 如果roi_start大于self.roi_start，则取self.roi_start作为roi_start
self.roi_start = np.maximum(np.asarray(roi_start, dtype=np.int16), 0)
# 如果roi_end大于self.roi_end，则取self.roi_end作为roi_end
self.roi_end = np.maximum(np.asarray(roi_end, dtype=np.int16), self.roi_start)
def __call__(self, data):
# __call__方法接收两个图像张量作为输入，并返回它们的 Hausdorff 距离，用于根据 ROI 的范围从输入数据中裁剪出子集
"""
Transform the data, if the data is channel first, slicing is not applicable to channel dim.
@ -59,12 +82,28 @@ class _ROISpatialData(metaclass=ABCMeta):
Returns:
np.ndarray, transform result.
"""
# 首先，使用min()函数计算sd，即 ROI 的最短轴长度
sd = min(len(self.roi_start), len(self.roi_end), len(data.shape[1:]))
# 然后，使用列表推导式创建一个元组，其中每个元素都是一个切片，用于从输入数据中裁剪出子集
slices = [slice(None)] + [slice(s, e) for s, e in zip(self.roi_start[:sd], self.roi_end[:sd])]
# 最后，使用tuple()函数将切片列表转换为一个元组，并使用data[]操作符获取裁剪后的子集
return data[tuple(slices)]
class HausdorffDistance(Metric):
# 用于计算两个点集之间的 Hausdorff 距离。Hausdorff 距离是两个点集之间最远距离的最大值。
# 给定两个特征集A和B，Hausdorff距离定义为：max(h(A, B), h(B, A))
# 其中h(A, B)是set A到set B中最近点的最大距离，h(B, A)是set B到set A中最近点的最大距离。
# 计算距离是 oriented 的，这意味着在大多数情况下，h(A, B) 不等于 h(B, A)。
"""
参数：
distance_metric (string)：支持三种距离计算方法："euclidean"，"chessboard"或"taxicab"。默认："euclidean"。
percentile (float)：浮点数范围在0到100之间。指定百分比参数以获取Hausdorff距离的百分位数。默认：None。
directed (bool)：如果为True，则仅计算h(y_pred, y)距离，否则，返回max(h(y_pred, y), h(y, y_pred))。默认：False。
crop (bool)：如果为True，则裁剪输入图像并只保留前景。为了保持两个输入图像的形状，这里使用(y_pred | y)来表示它们的并集集合。默认：True。
"""
r"""
Calculates the Hausdorff distance. Hausdorff distance is the maximum and minimum distance between two point sets.
Given two feature sets A and B, the Hausdorff distance between two point sets A and B is defined as follows:
@ -108,17 +147,30 @@ class HausdorffDistance(Metric):
1.4142135623730951
"""
def __init__(self, distance_metric="euclidean", percentile=None, directed=False, crop=True):
# 用于初始化类的参数
super(HausdorffDistance, self).__init__()
# 定义了一个字符串列表string_list，用于存储支持的三种距离计算方法
string_list = ["euclidean", "chessboard", "taxicab"]
# 接下来，它使用validator函数验证distance_metric是否为str类
distance_metric = validator.check_value_type("distance_metric", distance_metric, [str])
# 使用validator函数验证distance_metric, string_list是否是字符串
self.distance_metric = validator.check_string(distance_metric, string_list, "distance_metric")
# 使用if percentile is None判断percentile是否为None。如果是None，则将percentile的值赋给self.percentile；
# 否则，使用validator函数验证percentile是否为float类型，如果是，则将percentile的值赋给self.percentile
self.percentile = percentile if percentile is None else validator.check_value_type("percentile",
percentile, [float])
# 首先，使用if directed is None判断directed是否为None。如果是None，则将directed的值赋给self.directed；
# 否则，使用validator函数验证directed是否为bool类型，如果是，则将directed的值赋给self.directed
self.directed = directed if directed is None else validator.check_value_type("directed", directed, [bool])
# 使用if crop is None判断crop是否为None。如果是None，则将crop的值赋给self.crop；
# 否则，使用validator函数验证crop是否为bool类型，如果是，则将crop的值赋给self.crop
self.crop = crop if crop is None else validator.check_value_type("crop", crop, [bool])
# 最后，它将这些参数赋值给类的属性
# 清空类中残余的数据
self.clear()
def _is_tuple_rep(self, tup, dim):
# 用于检查输入的元组tup是否满足给定的维度dim
"""
Returns the tup containing the dim value by shortening or repeating the input.
@ -126,27 +178,49 @@ class HausdorffDistance(Metric):
ValueError: When tup is a sequence and tup length is not dim.
"""
# 将结果设置为NONE
result = None
# 如果tup不是一个可迭代序列
if not _is_iterable_sequence(tup):
# 那么将tup重复dim次，作为结果
result = (tup,) * dim
# 如果tup是一个可迭代序列，且长度等于dim
elif len(tup) == dim:
# 那么将tup作为结果
result = tuple(tup)
# 如果tup的长度不等于dim
if result is None:
# 那么抛出一个ValueError异常
raise ValueError(f"The sequence length should be {dim}, but got {len(tup)}.")
# 返回结果
return result
def _is_tuple(self, inputs):
# 用于将输入的序列转换为元组
"""
Returns a tuple of inputs.
"""
# 如果inputs不是一个可迭代序列
if not _is_iterable_sequence(inputs):
# 那么将inputs重复1次，作为结果
inputs = (inputs,)
# 如果inputs是一个可迭代序列，那么将inputs作为结果
return tuple(inputs)
def _create_space_bounding_box(self, image, func=lambda x: x > 0, channel_indices=None, margin=0):
# 用于从给定的图像中创建一个空间边界框，该边界框包含前景。用户可以通过定义一个函数来选择期望的前景，也可以指定要选择的通道。此外，还可以为边界框添加 margins
"""
函数的参数如下：
image：源图像，用于生成边界框。
func：函数，用于选择期望的前景，默认值为选择值大于 0 的值。
channel_indices：如果定义，则仅选择指定通道的前景，否则选择整个图像上的前景。
margin：可以添加到空间维度上的 margins，如果只提供一个值，则使用该值for所有维度。
"""
"""
The position of the space bounding box that generates the foreground in an image with start end.
The user can define any function to select the desired foreground from the whole image or the specified channel.
@ -160,21 +234,37 @@ class HausdorffDistance(Metric):
margin: add margin value to spatial dims of the bounding box, if only a single value is provided,
use it for all dims.
"""
# 如果定义了 channel_indices，则从 image 中选择指定通道的前景
# 否则，选择整个图像上的前景
data = image[[*(self._is_tuple(channel_indices))]] if channel_indices is not None else image
# 将前景转换为布尔值，得到一个二值图像
data = np.any(func(data), axis=0)
# 使用 np.nonzero 函数找到前景中的非零索引
nonzero_idx = np.nonzero(data)
# 首先计算边缘值
# 注：_is_tuple_rep 函数用于将边缘值转换为具有相同长度的元组，如果提供的边缘值是一个数字，则在每个维度上使用相同的边缘值
margin = self._is_tuple_rep(margin, data.ndim)
# 然后创建两个空列表 box_start 和 box_end
box_start = list()
box_end = list()
# 使用一个 for 循环遍历数据的维度。range(data.ndim) 表示从 0 到数据维度减 1 的范围，这意味着将遍历数据的所有维度
for i in range(data.ndim):
# 如果找到的索引数量为 0
if nonzero_idx[i].size <= 0:
# 则抛出一个错误ValueError，表示在空间维度上没有找到非零索引
raise ValueError("Did not find nonzero index at the spatial dim {}".format(i))
# 计算边界框的起始和结束位置。对于每个空间维度，计算最大非零索引减去边缘值
box_start.append(max(0, np.min(nonzero_idx[i]) - margin[i]))
# 然后计算最小非零索引加上边缘值，最后将这两个值作为边界框的起始和结束位置
box_end.append(min(data.shape[i], np.max(nonzero_idx[i]) + margin[i] + 1))
# 返回边界框的起始和结束位置
return box_start, box_end
def _calculate_percent_hausdorff_distance(self, y_pred_edges, y_edges):
# 用于计算定向 Hausdorff 距离。函数的输入参数有两个：y_pred_edges 和 y_edges。
# y_pred_edges 是一个表示预测边缘的 numpy 数组，y_edges 是一个表示真实边缘的 numpy 数组
"""
Calculate the directed Hausdorff distance.
@ -182,20 +272,30 @@ class HausdorffDistance(Metric):
y_pred_edges (np.ndarray): the edge of the predictions.
y_edges (np.ndarray): the edge of the ground truth.
"""
# 使用 _get_surface_distance 函数计算表面距离
surface_distance = self._get_surface_distance(y_pred_edges, y_edges)
# 如果表面距离的形状为 (0,)
if surface_distance.shape == (0,):
# 则返回正无穷
return np.inf
# 如果 self.percentile 不存在或为 False
if not self.percentile:
# 则返回表面距离的最大值
return surface_distance.max()
# 如果 0 <= self.percentile <= 100
if 0 <= self.percentile <= 100:
# 则返回百分比对应的表面距离百分位数
return np.percentile(surface_distance, self.percentile)
# 否则，抛出一个 ValueError 异常，表示 percentile 参数的值应在 [0, 100] 范围内
raise ValueError(f"For 'HausdorffDistance', the value of the argument 'percentile' should be [0, 100], "
f"but got {self.percentile}.")
def _get_surface_distance(self, y_pred_edges, y_edges):
# 用于计算从 y_pred_edges 到 y_edges 的表面距离。函数的输入参数有两个：y_pred_edges 和 y_edges。
# y_pred_edges 是一个表示预测边缘的 numpy 数组，y_edges 是一个表示真实边缘的 numpy 数组
"""
Calculate the surface distances from `y_pred_edges` to `y_edges`.
@ -203,23 +303,34 @@ class HausdorffDistance(Metric):
y_pred_edges (np.ndarray): the edge of the predictions.
y_edges (np.ndarray): the edge of the ground truth.
"""
# 如果 y_pred_edges 中没有非零值
if not np.any(y_pred_edges):
# 则返回一个空数组
return np.array([])
# 如果 y_edges 中没有非零值
if not np.any(y_edges):
# 则返回一个全为正无穷的数组
dis = np.inf * np.ones_like(y_edges)
else:
# 如果 self.distance_metric 为 "euclidean"
if self.distance_metric == "euclidean":
# 则使用 morphology.distance_transform_edt 函数计算欧氏距离
dis = morphology.distance_transform_edt(~y_edges)
# 如果 self.distance_metric 为 "chessboard" 或 "taxicab"
elif self.distance_metric == "chessboard" or self.distance_metric == "taxicab":
# 则使用 morphology.distance_transform_cdt 函数计算切比雪夫距离
dis = morphology.distance_transform_cdt(~y_edges, metric=self.distance_metric)
# 计算表面距离，将结果乘以 y_pred_edges 中非零值的个数
surface_distance = dis[y_pred_edges]
# 返回计算的表面距离
return surface_distance
def _get_mask_edges_distance(self, y_pred, y):
# 用于计算从 y_pred 到 y 的边缘距离。函数的输入参数有两个：y_pred 和 y。
# y_pred 是一个表示预测边缘的 numpy 数组，y 是一个表示真实边缘的 numpy 数组
"""
Do binary erosion and use XOR for input to get the edges. This function is helpful to further
calculate metrics such as Average Surface Distance and Hausdorff Distance.
@ -228,30 +339,48 @@ class HausdorffDistance(Metric):
y_pred (np.ndarray): the edge of the predictions.
y (np.ndarray): the edge of the ground truth.
"""
# 如果 self.crop 为 True
if self.crop:
# 首先检查 y_pred 和 y 是否全为零
if not np.any(y_pred | y):
# 如果是，首先定义两个与 y_pred 和 y 具有相同形状的全为零数组 res1 和 res2
res1 = np.zeros_like(y_pred)
res2 = np.zeros_like(y)
# 返回两个全为零的数组
return res1, res2
# 对 y_pred 和 y 进行扩展，以便它们具有相同的形状
y_pred, y = np.expand_dims(y_pred, 0), np.expand_dims(y, 0)
# 使用 _create_space_bounding_box 函数创建一个空间边界框，用于对 y_pred 和 y 进行裁剪
box_start, box_end = self._create_space_bounding_box(y_pred | y)
# 使用 _ROISpatialData 类对 y_pred 和 y 进行裁剪
cropper = _ROISpatialData(roi_start=box_start, roi_end=box_end)
# 对裁剪后的 y_pred 和 y 进行二进制腐蚀，并使用 XOR 运算符获取边缘
y_pred, y = np.squeeze(cropper(y_pred)), np.squeeze(cropper(y))
# 首先对 y_pred 和 y 进行二进制腐蚀，然后使用 XOR 运算符获取边缘。XOR 运算符的性质是：
# 对于任意整数 x，x ^ x = 0。因此，对 y_pred 和 y 进行二进制腐蚀后，只有边缘处的值为 1，其他位置的值为 0
y_pred = morphology.binary_erosion(y_pred) ^ y_pred
# 然后将得到的边缘与原始边缘进行 XOR 运算，得到一个新的边缘数组，其中只有边缘处的值为 1，其他位置的值为 0
y = morphology.binary_erosion(y) ^ y
# 返回裁剪后的 y_pred 和 y
return y_pred, y
def clear(self):
# 用于清空类中的计算数据
"""Clears the internal evaluation result."""
# 将y_pred_edges的值重置为0
self.y_pred_edges = 0
# 将y_edges的值重置为0
self.y_edges = 0
# 设置_is_update为False，表示未更新参数
self._is_update = False
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果。函数的输入参数有三个：y_pred、y 和 label_idx。
# y_pred 和 y 是一个 Tensor、列表或 numpy.ndarray，y_pred 是预测的二值图像，y 是实际的二值图像。label_idx 的数据类型为 int 或 float
"""
Updates the internal evaluation result with the inputs: 'y_pred', 'y' and 'label_idx'.
@ -266,36 +395,52 @@ class HausdorffDistance(Metric):
ValueError: If the value of label_idx is not in y_pred or y.
ValueError: If y_pred and y have different shapes.
"""
# 将_is_update设置为True，表示已更新参数
self._is_update = True
# 首先检查输入参数的数量是否为 3
if len(inputs) != 3:
# 如果不是，则抛出一个 ValueError 异常
raise ValueError("For 'HausdorffDistance.update', it needs 3 inputs (predicted value, true value, "
"label index), but got {}.".format(len(inputs)))
# 从输入中获取参数y_pred并将其转换为适当的数据格式
y_pred = self._convert_data(inputs[0])
# 从输入中获取参数y并将其转换为适当的数据格式
y = self._convert_data(inputs[1])
# 从输入中获取参数 label_idx
label_idx = inputs[2]
# 检查 label_idx 的数据类型是否为 int 或 float
if not isinstance(label_idx, (int, float)):
# 如果不是，则抛出一个 TypeError 异常
raise ValueError(f"For 'HausdorffDistance.update', the label index (input[2]) must be int or float, "
f"but got {type(label_idx)}.")
# 检查 label_idx 是否在 y_pred 中，也不在 y 中
if label_idx not in y_pred and label_idx not in y:
# 如果不是，则抛出一个 ValueError 异常
raise ValueError("For 'HausdorffDistance.update', the label index (input[2]) should be in predicted "
"value (input[0]) or true value (input[1]), but {} is not.".format(label_idx))
# # 检查 y_pred 和 y 是否具有相同的形状
if y_pred.size == 0 or y_pred.shape != y.shape:
# 如果不是，则抛出一个 ValueError 异常
raise ValueError(f"For 'HausdorffDistance.update', the size of predicted value (input[0]) and true value "
f"(input[1]) should be greater than 0, in addition to that, predicted value and true "
f"value should have the same shape, but got predicted value size: {y_pred.size}, shape: "
f"{y_pred.shape}, true value size: {y.size}, shape: {y.shape}.")
# 首先将 y_pred 和 y 转换为布尔类型
y_pred = (y_pred == label_idx) if y_pred.dtype is not bool else y_pred
# 然后根据 label_idx 的值设置它们的值
y = (y == label_idx) if y.dtype is not bool else y
# 最后，调用 self._get_mask_edges_distance 函数计算 y_pred 和 y 的边缘距离，并将结果分别赋值给 self.y_pred_edges 和 self.y_edges
self.y_pred_edges, self.y_edges = self._get_mask_edges_distance(y_pred, y)
def eval(self):
# 用于计算无向或有向 Hausdorff 距离，需先在调用update函数后调用
"""
Calculate the no-directed or directed Hausdorff distance.
@ -305,21 +450,32 @@ class HausdorffDistance(Metric):
Raises:
RuntimeError: If the update method is not called first, an error will be reported.
"""
# 检查 self._is_update 是否为 False
if self._is_update is False:
# 如果是，则抛出一个 RuntimeError 异常
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 调用 _calculate_percent_hausdorff_distance 函数计算 y_pred_edges 和 y_edges 的边缘距离
hd = self._calculate_percent_hausdorff_distance(self.y_pred_edges, self.y_edges)
# 如果 self.directed 为 True
if self.directed:
# 则返回计算得到的 Hausdorff 距离
return hd
# 否则，计算 y_edges 和 y_pred_edges 的边缘距离
hd2 = self._calculate_percent_hausdorff_distance(self.y_edges, self.y_pred_edges)
# 并返回较大的值
return max(hd, hd2)
def _is_iterable_sequence(inputs):
# 用于判断输入是否是一个可迭代的序列且不是字符串
"""
Determine if the input is an iterable sequence and it is not a string.
"""
# 首先判断输入是否是一个 Tensor 类型的对象
if isinstance(inputs, Tensor):
# 如果是，则判断其维度是否大于0并返回
return int(inputs.dim()) > 0
# 如果不是，则判断输入是否是一个 abc.Iterable 类型的对象且不是字符串。如果满足这两个条件之一，则返回 True，否则返回 False
return isinstance(inputs, abc.Iterable) and not isinstance(inputs, str)

26
mindspore/python/mindspore/nn/metrics/loss.py
View File

@ -13,10 +13,14 @@
# limitations under the License.
# ============================================================================
"""Loss for evaluation"""
# 该类用于评估损失。这个类继承自 Metric 类，并实现了 __init__ 和 update 方法
from .metric import Metric, rearrange_inputs
# 从 metric 模块中导入 Metric 和 rearrange_inputs 函数。Metric 类是用于计算评估指标的基类，而 rearrange_inputs 函数用于对输入进行重新排列，以适应不同的计算模式
class Loss(Metric):
# 继承自 Metric 类。Loss 类用于计算评估损失，通过重写 update 方法来计算损失值，并将损失值除以调用 update 方法的次数来计算平均损失
r"""
Calculates the average of the loss. If method 'update' is called every :math:`n` iterations, the result of
evaluation will be:
@ -41,16 +45,23 @@ class Loss(Metric):
0.20000000298023224
"""
def __init__(self):
# 调用父类构造函数初始化
super(Loss, self).__init__()
# 清空类中残余数据
self.clear()
def clear(self):
# 用于清空类中的数据
"""Clears the internal evaluation result."""
# 将_sum_loss重置为0
self._sum_loss = 0
# 将_total_num重置为0
self._total_num = 0
@rearrange_inputs
# 使用 @rearrange_inputs 装饰器，将输入重新排列为 (loss,) 形式
def update(self, *inputs):
# 用于更新内部评估结果。接收一个或多个输入，但输入只能有一个元素，即损失
"""
Updates the internal evaluation result.
@ -62,23 +73,33 @@ class Loss(Metric):
ValueError: If the length of inputs is not 1.
ValueError: If the dimension of loss is not 1 or 0.
"""
# 检查输入的数量是否为1
if len(inputs) != 1:
# 如果不是，则抛出一个 ValueError 异常
raise ValueError("For 'Loss.update', it needs 1 input (loss), but got {}".format(len(inputs)))
# 将输入的损失值转换为张量
loss = self._convert_data(inputs[0])
# 如果损失的维度为0
if loss.ndim == 0:
# 则将其reshape为 (1,) 形式
loss = loss.reshape(1)
# 如果损失的维度不是1
if loss.ndim != 1:
# 则抛出一个 ValueError 异常
raise ValueError("For 'Loss.update', the dimension of your input (loss) must be 1, "
"but got {}.".format(loss.ndim))
# 计算平均损失
loss = loss.mean(-1)
# 并将损失值求和
self._sum_loss += loss
# 将平均损失添加到内部评估结果中，同时更新 total_num
self._total_num += 1
def eval(self):
# 用于计算评估损失的平均值，需先在调用update函数后调用
"""
Calculates the average of the loss.
@ -88,7 +109,10 @@ class Loss(Metric):
Raises:
RuntimeError: If the total number is 0.
"""
# 首先检查 total_num 是否为0
if self._total_num == 0:
# 如果是，则抛出一个 RuntimeError 异常
raise RuntimeError("The 'Loss' can not be calculated, because the number of samples is 0, please "
"check whether has called update method before calling eval method.")
# 然后计算平均损失，并将结果返回
return self._sum_loss / self._total_num

83
mindspore/python/mindspore/nn/metrics/mean_surface_distance.py
View File

@ -12,14 +12,33 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 用于计算平均表面距离。这个类继承自 Metric 类，并实现了 __init__ 和 update 方法
"""MeanSurfaceDistance."""
# 导入了 scipy.ndimage 模块中的 morphology 函数，用于计算图像形态学
from scipy.ndimage import morphology
# 导入了 numpy 模块，用于处理数值计算
import numpy as np
# 从 mindspore._checkparam 模块中导入 Validator 类，用于参数验证
from mindspore._checkparam import Validator as validator
# 从 .metric 模块中导入 Metric 和 rearrange_inputs 函数，用于处理评估指标和输入重排
from .metric import Metric, rearrange_inputs
class MeanSurfaceDistance(Metric):
# 通过重写 update 方法来计算损失值，并将损失值除以调用 update 方法的次数来计算平均损失
"""
初始化参数：
distance_metric (string)：支持三种距离度量方法："euclidean"（欧氏距离）、"chessboard"（棋盘距离）和 "taxicab"（曼哈顿距离）。默认："euclidean"。
symmetric (bool)：是否计算 y_pred 和 y 的平均表面距离。如果为 False，则只计算 AvgSurDis({y_pred} \rightarrow y)，否则计算 MeanSurDis(y_{pred} \leftrightarrow y)。默认：False。
计算方法：
导入所需的库。
定义一个名为 _convert_data 的辅助方法，用于将输入的数据转换为张量。
定义一个名为 update 的辅助方法，用于更新内部评估结果。
定义一个名为 eval 的辅助方法，用于计算评估损失的平均值。
"""
r"""
Computes the Average Surface Distance from `y_pred` to `y` under the default setting. It measures how much,
on average, the surface varies between the segmentation and the GT (ground truth).
@ -70,23 +89,37 @@ class MeanSurfaceDistance(Metric):
"""
def __init__(self, symmetric=False, distance_metric="euclidean"):
# 调用父类的 __init__ 方法，初始化 Metric 类
super(MeanSurfaceDistance, self).__init__()
# 定义一个名为 distance_metric_list 的列表，用于存储支持的三种距离度量方法
self.distance_metric_list = ["euclidean", "chessboard", "taxicab"]
# 检查 distance_metric 是否为str类并将 distance_metric 参数设置为有效的值
distance_metric = validator.check_value_type("distance_metric", distance_metric, [str])
# 检查 distance_metric 参数的值是否在 distance_metric_list 中，如果不是，则抛出一个 ValueError 异常
self.distance_metric = validator.check_string(distance_metric, self.distance_metric_list, "distance_metric")
# 检查 symmetric 参数的值是否为布尔类型，如果不是，则抛出一个 ValueError 异常
self.symmetric = validator.check_value_type("symmetric", symmetric, [bool])
# 清空类中残余的数据
self.clear()
# 初始化一个变量 _is_update，用于记录是否已经调用过 update 方法
self._is_update = None
# 初始化变量 _y_edges 用于记录输入的 y 数据
self._y_edges = None
# 初始化两个变量 _y_pred_edges 用于记录输入的 y_pred 数据
self._y_pred_edges = None
def clear(self):
# 用于清空类中的数据
"""Clears the internal evaluation result."""
# 将_y_pred_edges设置为0
self._y_pred_edges = 0
# 将_y_edges设置为0
self._y_edges = 0
# 将_is_update设置为False，表示结果未更新
self._is_update = False
def _get_surface_distance(self, y_pred_edges, y_edges):
# 用于计算从 y_pred_edges 到 y_edges 的表面距离
"""
Calculate the surface distances from `y_pred_edges` to `y_edges`.
@ -94,21 +127,32 @@ class MeanSurfaceDistance(Metric):
y_pred_edges (np.ndarray): the edge of the predictions.
y_edges (np.ndarray): the edge of the ground truth.
"""
# 检查 y_pred_edges 和 y_edges 是否为空数组
if not np.any(y_pred_edges):
# 如果是，则返回一个空数组
return np.array([])
# 如果 y_edges 不为空
if np.any(y_edges):
# 如果 distance_metric 参数为 "euclidean"
if self.distance_metric == "euclidean":
# 则使用 morphology.distance_transform_edt 函数计算欧氏距离
dis = morphology.distance_transform_edt(~y_edges)
# 如果 distance_metric 参数为 "chessboard" 或 "taxicab"
elif self.distance_metric in self.distance_metric_list[-2:]:
# 则使用 morphology.distance_transform_cdt 函数计算棋盘距离
dis = morphology.distance_transform_cdt(~y_edges, metric=self.distance_metric)
# 如果 y_edges 为空
else:
# 返回一个全为无穷大的数组
dis = np.full(y_edges.shape, np.inf)
# 返回 dis 数组中 y_pred_edges 对应的元素值
return dis[y_pred_edges]
@rearrange_inputs
# 使用 @rearrange_inputs 装饰器处理输入参数，确保输入参数的个数为 3
def update(self, *inputs):
# 用于更新内部评估结果 y_pred、y 和 label_idx
"""
Updates the internal evaluation result 'y_pred', 'y' and 'label_idx'.
@ -123,37 +167,57 @@ class MeanSurfaceDistance(Metric):
ValueError: If the value of label_idx is not in y_pred or y.
ValueError: If y_pred and y have different shapes.
"""
# 检查 label_idx 的数据类型是否为整数或浮点数
if len(inputs) != 3:
# 如果不是，则抛出一个 ValueError 异常
raise ValueError("For 'MeanSurfaceDistance.update', it needs 3 inputs (predicted value, true value, "
"label index), but got {}".format(len(inputs)))
# 将输入参数 inputs 中第 0 个元素转换为 y_pred
y_pred = self._convert_data(inputs[0])
# 第 1 个元素转换为 y
y = self._convert_data(inputs[1])
# 并将第 2 个元素赋值给 label_idx。
# _convert_data 方法用于将输入数据转换为所需的数据类型和形状
label_idx = inputs[2]
# 检查 label_idx 的值是否在 y_pred 或 y 的范围内
if not isinstance(label_idx, (int, float)):
# 如果不是，则抛出一个 ValueError 异常
raise ValueError(f"For 'MeanSurfaceDistance.update', the label index (input[2]) must be int or float, "
f"but got {type(label_idx)}.")
# 检查 y_pred 和 y 是否具有相同的形状
if label_idx not in y_pred and label_idx not in y:
# 如果不是，则抛出一个 ValueError 异常
raise ValueError("For 'MeanSurfaceDistance.update', the label index (input[2]) should be in predicted "
"value (input[0]) or true value (input[1]), but {} is not.".format(label_idx))
# 检查 y_pred 和 y 的尺寸是否为 0，以及它们是否具有相同的形状
if y_pred.size == 0 or y_pred.shape != y.shape:
# 如果满足这些条件，它将检查 y_pred 和 y 的数据类型是否为布尔值
raise ValueError(f"For 'MeanSurfaceDistance.update', the size of predicted value (input[0]) and true "
f"value (input[1]) should be greater than 0, in addition to that, predicted value and "
f"true value should have the same shape, but got predicted value size: {y_pred.size}, "
f"shape: {y_pred.shape}, true value size: {y.size}, shape: {y.shape}.")
# 检查 y_pred 的数据类型是否为布尔值
if y_pred.dtype != bool:
# 如果不是，则将它们转换为布尔值
y_pred = y_pred == label_idx
# 检查 y_pred 和 y 的数据类型是否为布尔值
if y.dtype != bool:
# 如果不是，则将它们转换为布尔值
y = y == label_idx
# 然后，使用 morphology.binary_erosion 函数对 y_pred 和 y 进行二值腐蚀操作
self._y_pred_edges = morphology.binary_erosion(y_pred) ^ y_pred
# 最后，将 self._y_pred_edges 和 self._y_edges 更新为 erosion 后的值
self._y_edges = morphology.binary_erosion(y) ^ y
# 并设置 self._is_update 为 True，表示已更新参数
self._is_update = True
def eval(self):
# 用于计算平均表面距离，通常在update函数后调用
"""
Calculate mean surface distance.
@ -163,22 +227,35 @@ class MeanSurfaceDistance(Metric):
Raises:
RuntimeError: If the update method is not called first, an error will be reported.
"""
# 首先，检查 self._is_update 是否为 True
if self._is_update is False:
# 如果不是，则抛出一个 RuntimeError 异常，要求在调用 eval 方法之前先调用 update 方法
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 如果是，则继续计算平均表面距离
mean_surface_distance = self._get_surface_distance(self._y_pred_edges, self._y_edges)
# 检查 mean_surface_distance 的形状是否为 (0,)
if mean_surface_distance.shape == (0,):
# 如果是，则返回正无穷
return np.inf
# 否则计算平均表面距离，并将其赋值给 avg_surface_distance
avg_surface_distance = mean_surface_distance.mean()
# 如果 self.symmetric 为 True
if not self.symmetric:
# 则返回 avg_surface_distance
return avg_surface_distance
# 接下来，使用 _get_surface_distance 方法计算 _y_pred_edges 和 _y_edges 的表面距离
contrary_mean_surface_distance = self._get_surface_distance(self._y_edges, self._y_pred_edges)
# 如果计算结果的形状为 (0,)
if contrary_mean_surface_distance.shape == (0,):
# 则返回正无穷
return np.inf
# 否则计算平均表面距离
contrary_avg_surface_distance = contrary_mean_surface_distance.mean()
# 上述两则判断均不符合则计算两个平均表面距离的平均值并返回
return np.mean((avg_surface_distance, contrary_avg_surface_distance))

132
mindspore/python/mindspore/nn/metrics/metric.py
View File

@ -12,16 +12,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 在机器学习中，性能指标(Metrics)是衡量一个模型好坏的关键，通过衡量模型输出y_predict 和 y_true之间的某种"距离"得出的
# 定义了一个名为MetricBase的基类，用于计算模型准确性
"""Metric base class."""
# 该类使用了ABCMeta元类，这意味着它是一个抽象基类，需要继承的子类需要实现其所有抽象方法
from abc import ABCMeta, abstractmethod
# 从functools导入了functools模块，用于实现一些高阶函数，例如partial
import functools
# 导入了numpy模块，用于处理数值计算
import numpy as np
# 导入了Tensor类，用于表示计算图中的张量
from mindspore.common.tensor import Tensor
# _eval_types是一个字典，用于存储支持的评估类型，包括'classification'和'multilabel'
_eval_types = {'classification', 'multilabel'}
def rearrange_inputs(func):
# 这段代码是一个装饰器，用于重排输入。装饰器通常用于修改函数的行为，而这里是用于修改mindspore.nn.Metric类中的update方法的行为。
# 这个装饰器的主要作用是根据indexes属性对输入进行重排
"""
This decorator is used to rearrange the inputs according to its `indexes` attribute of the class.
@ -61,13 +71,24 @@ def rearrange_inputs(func):
"""
@functools.wraps(func)
def wrapper(self, *inputs):
'''
调用函数，并将输入参数转换为索引
:param self:
:param inputs:
:return:
'''
# 获取indexes
indexes = self.indexes
# 如果indexes不存在，则调用func函数，并将inputs作为参数传入
inputs = inputs if not indexes else [inputs[i] for i in indexes]
# 返回func函数的执行结果
return func(self, *inputs)
# 返回重排后的参数
return wrapper
class Metric(metaclass=ABCMeta):
# 定义了一个名为Metric的基类，用于计算模型准确性
"""
Base class of metric. This class is used to evaluate metrics.
@ -81,7 +102,9 @@ class Metric(metaclass=ABCMeta):
Supported Platforms:
``Ascend`` ``GPU`` ``CPU``
"""
# 初始化Metric类的实例
def __init__(self):
# 首先将self._indexes设置为None
self._indexes = None
def _convert_data(self, data):
@ -94,24 +117,35 @@ class Metric(metaclass=ABCMeta):
Returns:
Ndarray, data with `np.ndarray` type.
"""
# 判断data是否为Tensor类型
if isinstance(data, Tensor):
# 将data转换为numpy数组
data = data.asnumpy()
# 判断data是否为list类型
elif isinstance(data, list):
# 将list转换为numpy数组
data = np.array(data)
# 判断data是否为numpy.ndarray类型
elif isinstance(data, np.ndarray):
pass
# 判断data是否为其他类型
else:
# 抛出类型错误TypeError
raise TypeError(f"For 'Metric' and its derived classes, the input data type must be tensor, list or "
f"numpy.ndarray, but got {type(data)}.")
# 返回转换后的数据
return data
@property
def indexes(self):
# 用于获取当前的indexes值。默认值是None，可以通过set_indexes方法修改
"""Get the current indexes value. The default value is None and can be changed by `set_indexes`.
"""
return getattr(self, '_indexes', None)
def set_indexes(self, indexes):
# 用于设置indexes属性
# 注：设置update方法的输入顺序。通过设置indexes属性，可以重新排列update方法接收的输入参数顺序，以便根据需要从输入参数中提取所需的参数
"""
This interface is to rearrange the inputs of `update`.
@ -146,13 +180,17 @@ class Metric(metaclass=ABCMeta):
>>> print(accuracy)
0.3333333333333333
"""
# 检查indexes参数是否为列表且所有元素为整数
if not isinstance(indexes, list) or not all(isinstance(i, int) for i in indexes):
raise ValueError("For 'set_indexes', the argument 'indexes' should be a list and all its elements should "
# 如果不是，则抛出一个ValueError异常，表示indexes参数无效
raise ValueError("For'set_indexes', the argument 'indexes' should be a list and all its elements should "
"be int, please check whether it is correct.")
# 将indexes值设置为self._indexes，并返回self
self._indexes = indexes
return self
def __call__(self, *inputs):
# 用于评估输入数据一次。当调用Metric类的实例时，会自动调用__call__方法
"""
Evaluate input data once.
@ -162,12 +200,16 @@ class Metric(metaclass=ABCMeta):
Returns:
Float, compute result.
"""
# 清空计算的中间结果
self.clear()
# 更新输入数据
self.update(*inputs)
# 返回计算结果
return self.eval()
@abstractmethod
def clear(self):
# 用于重置中间结果，在clear方法中，将self._indexes设置为None
"""
An interface describes the behavior of clearing the internal evaluation result.
@ -178,6 +220,9 @@ class Metric(metaclass=ABCMeta):
@abstractmethod
def eval(self):
# 用于计算模型的准确性。首先，它会检查self._total_num是否为0，如果是，则抛出一个RuntimeError异常，表示无法计算准确性，
# 因为样本数量为0。如果self._total_num不为0，则计算self._correct_num除以self._total_num的结果，并返回该结果。
# 在计算准确性时，需要确保已经调用了update方法，以便能够更新self._correct_num和self._total_num的值
"""
An interface describes the behavior of computing the evaluation result.
@ -188,6 +233,9 @@ class Metric(metaclass=ABCMeta):
@abstractmethod
def update(self, *inputs):
# 用于更新中间结果。在update方法中，首先检查indexes属性是否存在，如果不存在，则直接返回None。如果indexes属性存在，
# 则使用列表推导式根据indexes属性中的索引提取输入参数，并将提取到的输入参数作为新的输入参数列表传入原始的update方法。
# 最后，返回update方法的结果
"""
An interface describes the behavior of updating the internal evaluation result.
@ -201,6 +249,7 @@ class Metric(metaclass=ABCMeta):
class EvaluationBase(Metric):
# EvaluationBase类用于进行评估，包括分类和多标签两种类型
"""
Base class of evaluation.
@ -214,40 +263,59 @@ class EvaluationBase(Metric):
TypeError: If the input type is not classification or multilabel.
"""
def __init__(self, eval_type):
# eval_type参数用于指定评估类型，必须是'classification'或'multilabel'之一。如果输入类型不是分类或多标签，会抛出一个TypeError异常
super(EvaluationBase, self).__init__()
# 评估类型如果不在分类和多标签两种类型之一
if eval_type not in _eval_types:
# 抛出一个TypeError异常
raise TypeError("The argument 'eval_type' must be in {}, but got {}".format(_eval_types, eval_type))
# 设置评估类型
self._type = eval_type
def _check_shape(self, y_pred, y):
"""
Checks the shapes of y_pred and y.
Checks the shapes of y_pred and y.
检查y_pred和y的形状
Args:
y_pred (Tensor): Predict array.
y (Tensor): Target array.
y_pred (Tensor): Predict array.
y (Tensor): Target array.
y_pred (Tensor): 预测数组
y (Tensor): 真实数组
"""
# 如果是分类标签
if self._type == 'classification':
if y_pred.ndim != y.ndim + 1:
# 首先检查y_pred的维度是否等于y的维度加1
if y_pred.ndim!= y.ndim + 1:
# 如果不是，则抛出一个ValueError异常，表示在分类情况下，y_pred的维度应该等于y的维度加1，但实际维度为y_pred.ndim和y.ndim
raise ValueError("In classification case, the dimension of y_pred (predicted value) should equal to "
"the dimension of y (true value) add 1, but got y_pred dimension: {} and y "
"dimension: {}.".format(y_pred.ndim, y.ndim))
if y.shape != (y_pred.shape[0],) + y_pred.shape[2:]:
"the dimension of y (true value) add 1, but got y_pred dimension: {} and y "
"dimension: {}.".format(y_pred.ndim, y.ndim))
# 接下来，检查y的形状是否等于y_pred的形状减去第1个维度（因为第1个维度通常表示样本数量）
if y.shape!= (y_pred.shape[0],) + y_pred.shape[2:]:
# 如果不是，则抛出一个ValueError异常，表示在分类情况下，y的形状应该等于y_pred的形状减去第1个维度，但实际形状为y.shape和y_pred.shape
raise ValueError("In classification case, y_pred (predicted value) shape and y (true value) shape "
"can not match, y shape should be equal to y_pred shape that the value at index 1 "
"is deleted. Such as y_pred shape (1, 2, 3), then y shape should be (1, 3). "
"But got y_pred shape {} and y shape {}".format(y_pred.shape, y.shape))
"can not match, y shape should be equal to y_pred shape that the value at index 1 "
"is deleted. Such as y_pred shape (1, 2, 3), then y shape should be (1, 3). "
"But got y_pred shape {} and y shape {}".format(y_pred.shape, y.shape))
# 如果是多标签
else:
if y_pred.ndim != y.ndim:
# 首先检查y_pred的维度是否等于y的维度
if y_pred.ndim!= y.ndim:
# 如果不是，则抛出一个ValueError异常，表示在self._type类型的评估情况下，y_pred的维度应该等于y的维度
raise ValueError("In {} case, the dimension of y_pred (predicted value) should equal to the dimension"
" of y (true value), but got y_pred dimension: {} and y dimension: {}."
.format(self._type, y_pred.ndim, y.ndim))
if y_pred.shape != y.shape:
" of y (true value), but got y_pred dimension: {} and y dimension: {}."
.format(self._type, y_pred.ndim, y.ndim))
# 接下来，检查y的形状是否等于y_pred的形状
if y_pred.shape!= y.shape:
# 如果不是，则抛出一个ValueError异常，表示在self._type类型的评估情况下，y的形状应该等于y_pred的形状
raise ValueError("In {} case, the shape of y_pred (predicted value) should equal to the shape of y "
"(true value), but got y_pred shape: {} and y shape: {}."
.format(self._type, y_pred.shape, y.shape))
"(true value), but got y_pred shape: {} and y shape: {}."
.format(self._type, y_pred.shape, y.shape))
def _check_value(self, y_pred, y):
# 用于检查y_pred和y的值是否符合要求
"""
Checks the values of y_pred and y.
@ -255,11 +323,14 @@ class EvaluationBase(Metric):
y_pred (Tensor): Predict array.
y (Tensor): Target array.
"""
# 在多标签评估情况下，这个方法用于检查y_pred和y中的所有元素是否都是0或1
if self._type != 'classification' and not (np.equal(y_pred ** 2, y_pred).all() and np.equal(y ** 2, y).all()):
# 如果不是，则抛出一个ValueError异常，表示在多标签评估情况下，y_pred和y中的所有元素应该都是0或1，但实际值不为0或1
raise ValueError("In multilabel case, all elements in y_pred (predicted value) and y (true value) should "
"be 0 or 1.Please check whether your inputs y_pred and y are correct.")
def clear(self):
# 用于重置中间结果，在clear方法中，将self._indexes设置为None
"""
A interface describes the behavior of clearing the internal evaluation result.
@ -269,6 +340,9 @@ class EvaluationBase(Metric):
raise NotImplementedError
def update(self, *inputs):
# 用于更新中间结果。在update方法中，首先检查indexes属性是否存在，如果不存在，则直接返回None。如果indexes属性存在，
# 则使用列表推导式根据indexes属性中的索引提取输入参数，并将提取到的输入参数作为新的输入参数列表传入原始的update方法。
# 最后，返回update方法的结果
"""
A interface describes the behavior of updating the internal evaluation result.
@ -281,6 +355,9 @@ class EvaluationBase(Metric):
raise NotImplementedError
def eval(self):
# 用于计算模型的准确性。首先，它会检查self._total_num是否为0，如果是，则抛出一个RuntimeError异常，表示无法计算准确性，
# 因为样本数量为0。如果self._total_num不为0，则计算self._correct_num除以self._total_num的结果，并返回该结果。
# 在计算准确性时，需要确保已经调用了update方法，以便能够更新self._correct_num和self._total_num的值
"""
A interface describes the behavior of computing the evaluation result.
@ -291,6 +368,7 @@ class EvaluationBase(Metric):
def _check_onehot_data(data):
# 用于检查输入数据是否是独热编码
"""
Whether input data is one-hot encoding.
@ -300,38 +378,56 @@ def _check_onehot_data(data):
Returns:
bool, return true, if input data is one-hot encoding.
"""
# 判断data的维度是否大于1，且data是否平方等于data
if data.ndim > 1 and np.equal(data ** 2, data).all():
# 获取data的维度
shp = (data.shape[0],) + data.shape[2:]
# 判断data是否全部等于1
if np.equal(np.ones(shp), data.sum(axis=1)).all():
# 如果满足上述两个条件，则返回True，表示输入数据是独热编码
return True
# 否则返回False
return False
def _binary_clf_curve(preds, target, sample_weights=None, pos_label=1):
# 用于计算二分类分类曲线的True Positives（真正例）和False Positives（假正例）
"""Calculate True Positives and False Positives per binary classification threshold."""
# 如果sample_weights不是空且不是一个ndarray，则将其转换为ndarray
if sample_weights is not None and not isinstance(sample_weights, np.ndarray):
sample_weights = np.array(sample_weights)
# 如果preds的维度大于target的维度，则将preds的第一维放置为1
if preds.ndim > target.ndim:
preds = preds[:, 0]
# 对preds和target进行排序，使preds从大到小排列
desc_score_indices = np.argsort(-preds)
# 将preds和target按照desc_score_indices中的顺序进行排列
preds = preds[desc_score_indices]
target = target[desc_score_indices]
# 如果提供了sample_weights，则对preds和target进行加权
if sample_weights is not None:
weight = sample_weights[desc_score_indices]
else:
weight = 1.
# 首先找到preds中不相同的值的下标，这些下标表示分类曲线的分界点。np.where(preds[1:] - preds[:-1])[0]会返回一个元组，
# 其中第一个元素是分界点的下标，第二个元素是分界点对应的preds值
distinct_value_indices = np.where(preds[1:] - preds[:-1])[0]
# 使用np.pad函数对distinct_value_indices进行填充，使其长度与target的形状相等
threshold_idxs = np.pad(distinct_value_indices, (0, 1), constant_values=target.shape[0] - 1)
# 将target转换为int64
target = np.array(target == pos_label).astype(np.int64)
# 计算tps
tps = np.cumsum(target * weight, axis=0)[threshold_idxs]
# 如果sample_weights不为空，则计算fps
if sample_weights is not None:
fps = np.cumsum((1 - target) * weight, axis=0)[threshold_idxs]
else:
# 否则fps为1 + threshold_idxs - tps
fps = 1 + threshold_idxs - tps
# 返回fps、tps、preds[threshold_idxs]
return fps, tps, preds[threshold_idxs]

131
mindspore/python/mindspore/nn/metrics/occlusion_sensitivity.py
View File

@ -12,22 +12,50 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
"""OcclusionSensitivity."""
import numpy as np
from mindspore import nn
from mindspore.common.tensor import Tensor
from mindspore._checkparam import Validator as validator
from .metric import Metric, rearrange_inputs
# 用于计算遮挡敏感度（Occlusion Sensitivity）。类中包含一个名为 update 的方法，用于更新预测值和真实值；一个名为 eval 的方法，用于计算平均表面距离
"""OcclusionSensitivity."""
# numpy：这是一个用于数值计算的Python库，提供了高性能的多维数组对象和相关工具
import numpy as np
# mindspore：这是一个全场景深度学习框架，适用于训练和推理
from mindspore import nn
# mindspore.common.tensor：提供了一个用于处理张量的模块
from mindspore.common.tensor import Tensor
# Validator：一个用于验证参数的模块
from mindspore._checkparam import Validator as validator
# .metric：一个包含metric计算相关功能的模块
from .metric import Metric, rearrange_inputs
# rearrange_inputs：一个用于重排输入的函数
# 这段代码是用于导入tqdm库，用于在训练和推理过程中显示进度条
try:
# 首先尝试从tqdm库中导入trange函数
from tqdm import trange
# 如果导入失败，则使用Python的range函数替代
except (ImportError, AttributeError):
trange = range
finally:
# 最后，如果导入成功或失败，都执行一个pass语句
pass
class OcclusionSensitivity(Metric):
# 继承自Metric模块。Occlusion sensitivity是一种用于评估模型在遮挡部分上的敏感性的指标。
# 它表示当遮盖住图像中的一个部分时，预测概率的变化程度。值越高，表明遮盖部分在决策过程中越重要
"""
OcclusionSensitivity类的主要参数如下：
pad_val (float)：遮盖值，表示遮盖部分的值。默认值为0.0。
margin (Union[int, Sequence])：创建一个 cuboid / cube around the voxel you want to occlude。默认值为2。
n_batch (int)：批量中的图像数量。默认值为128。
b_box (Sequence)：用于分析的边界框。输出图像将匹配边界框的大小。应该有最小和最大值，除了批量维度：[min1, max1, min2, max2,...]。
如果没有提供边界框，那么输出图像将与输入图像具有相同的大小。如果提供了边界框，那么输出图像将根据边界框进行裁剪。默认值为None。
这个类继承了Metric模块，主要实现了以下方法：
update：用于更新指标的计算。接收输入图像、预测概率和标签作为参数。
eval：用于评估指标。接收数据列表作为参数，返回评估结果。
"""
"""
Calculates the occlusion sensitivity of the model for a given image. It illustrates which parts of an image are
most important for a network's classification.
@ -74,21 +102,33 @@ class OcclusionSensitivity(Metric):
[0.29999995 0.6 1. 0.9]
"""
def __init__(self, pad_val=0.0, margin=2, n_batch=128, b_box=None):
# 调用父类的构造函数，初始化类
super().__init__()
# 检查pad_val的值是否为float浮点数，如果不符合要求，将抛出一个TypeError异常
self.pad_val = validator.check_value_type("pad_val", pad_val, [float])
# 检查margin是否为整数或列表类型。如果不符合要求，将抛出一个TypeError异常
self.margin = validator.check_value_type("margin", margin, [int, list])
# 检查n_batch的值是否为int型，如果不符合要求，将抛出一个TypeError异常
self.n_batch = validator.check_value_type("n_batch", n_batch, [int])
# 检查b_box的值是否为列表类型，如果不符合要求，将抛出一个TypeError异常
self.b_box = b_box if b_box is None else validator.check_value_type("b_box", b_box, [list])
# 清除残留的计算结果
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 将基础概率_baseline重置为0
self._baseline = 0
# 将敏感性图像_sensitivity_im重置为0
self._sensitivity_im = 0
# 将_is_update重置为false，表示未更新参数
self._is_update = False
@rearrange_inputs
def update(self, *inputs):
# 用于更新输入。它接收任意数量的关键字参数，包括model、y_pred和label。y_pred是一个批量测试图像，可以是2D或3D。
# label是用于检查变化的分类标签，通常为TRUE，但也可以不是。model是神经网络模型
"""
Updates input, including `model`, `y_pred` and `label`.
@ -104,64 +144,114 @@ class OcclusionSensitivity(Metric):
RuntimeError: If y_pred.shape[0] is not 1.
RuntimeError: If the number of labels is different from the number of batches.
"""
"""
OcclusionSensitivity类中update方法的预处理部分
"""
# 首先，它检查输入是否为3个
if len(inputs) != 3:
# 如果不是，将抛出一个ValueError异常
raise ValueError("For 'OcclusionSensitivity.update', it needs 3 inputs (classification model, "
"predicted value, label), but got {}.".format(len(inputs)))
# 首先，它从输入列表中提取model
model = inputs[0]
# 然后，它将y_pred转换为nn.Tensor类型
y_pred = self._convert_data(inputs[1])
# 然后，它将label转换为nn.Tensor类型
label = self._convert_data(inputs[2])
# 然后，它将输入的model转换为nn.Cell类型
model = validator.check_value_type("model", model, [nn.Cell])
# 检查y_pred的第一个维度是否大于1
if y_pred.shape[0] > 1:
# 如果是，将抛出一个RuntimeError异常
raise RuntimeError(f"For 'OcclusionSensitivity.update', the shape at index 0 of the predicted value "
f"(input[1]) should be 1, but got {y_pred.shape[0]}.")
# 检查label是否为整数类型
if isinstance(label, int):
# 如果是，将将其转换为整数类型的numpy数组
label = np.array([[label]], dtype=int)
# If the label is a tensor, make sure there's only 1 element
# 否则，检查label的形状是否与y_pred的形状相同
elif np.prod(label.shape) != y_pred.shape[0]:
# 如果不相同，将抛出一个RuntimeError异常
raise RuntimeError(f"For 'OcclusionSensitivity.update', the number of the label (input[2]) should be "
f"same as the batches, but got the label number {np.prod(label.shape)}, "
f"and batches {y_pred.shape[0]}.")
"""
OcclusionSensitivity类中update方法的计算基础概率部分
"""
# 首先，它计算y_pred的形状
y_pred_shape = np.array(y_pred.shape[1:])
# 然后使用_check_input_bounding_box方法检查b_box是否有效
b_box_min, b_box_max = _check_input_bounding_box(self.b_box, y_pred_shape)
# 接下来，它使用model函数计算y_pred的值，并将结果转换为numpy数组
temp = model(Tensor(y_pred)).asnumpy()
# 将计算出的基础概率存储在self._baseline属性中
self._baseline = temp[0, label].item()
# 创建两个空列表batch_images和batch_ids，用于存储批次图像和批次ID
batch_images = []
batch_ids = []
# 首先，它创建一个空numpy数组sensitivity_im，用于存储敏感性图像
sensitivity_im = np.empty(0, dtype=float)
# 然后，它计算output_im_shape，即y_pred的形状（如果b_box为空）或者b_box的最大值减去最小值加1（否则）
output_im_shape = y_pred_shape if self.b_box is None else b_box_max - b_box_min + 1
# 最后，它计算所需的预测数量num_required_predictions，即output_im_shape的乘积
num_required_predictions = np.prod(output_im_shape)
"""
OcclusionSensitivity类中update方法的遮罩生成部分
"""
# 使用一个循环遍历所需的预测数量num_required_predictions
for i in trange(num_required_predictions):
# 并为每个预测计算一个遮罩
idx = np.unravel_index(i, output_im_shape)
# 遮罩的生成是通过计算min_idx和max_idx来确定遮罩的边界
# 首先，如果b_box_min不为空
if b_box_min is not None:
# 那么将idx加上b_box_min
idx += b_box_min
# 然后，计算min_idx和max_idx，分别表示遮罩的左上角和右下角坐标
min_idx = [max(0, i - self.margin) for i in idx]
max_idx = [min(j, i + self.margin) for i, j in zip(idx, y_pred_shape)]
# 将y_pred在遮罩范围内的部分复制到一个新变量occlu_im中
occlu_im = y_pred.copy()
# 然后将y_pred在遮罩范围内的部分替换为self.pad_val
occlu_im[(...,) + tuple(slice(i, j) for i, j in zip(min_idx, max_idx))] = self.pad_val
# 最后，将生成的遮罩和对应的标签添加到batch_images列表中
batch_images.append(occlu_im)
# 将生成的遮罩和对应的标签添加到batch_ids列表中
batch_ids.append(label)
"""
OcclusionSensitivity类中update方法的最终部分
"""
# 如果batch_images的长度达到self.n_batch
if len(batch_images) == self.n_batch or i == num_required_predictions - 1:
# 那么将sensitivity_im更新为当前batch_images和batch_ids的计算结果
sensitivity_im = _append_to_sensitivity_im(model, batch_images, batch_ids, sensitivity_im)
# 并将batch_images和batch_ids重置为空列表
batch_images = []
batch_ids = []
# 首先，它将计算得到的敏感性图像sensitivity_im重塑为与output_im_shape相同的形状，然后将其赋值给self._sensitivity_im属性
self._sensitivity_im = sensitivity_im.reshape(output_im_shape)
# 最后，将is_update属性设置为True，表示已经更新了敏感性图像
self._is_update = True
def eval(self):
# 它用于计算敏感性图像的差异，通常在update函数后调用
"""
Computes the occlusion_sensitivity.
@ -172,50 +262,77 @@ class OcclusionSensitivity(Metric):
RuntimeError: If the update method is not called first, an error will be reported.
"""
# 首先，检查is_update属性是否为True
if not self._is_update:
# 如果不为True，则抛出一个RuntimeError错误
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 然后，计算敏感性图像sensitivity_im与基本线baseline的差异，得到最终的敏感性值sensitivity
sensitivity = self._baseline - np.squeeze(self._sensitivity_im)
# 最后，返回sensitivity
return sensitivity
def _append_to_sensitivity_im(model, batch_images, batch_ids, sensitivity_im):
# OcclusionSensitivity类中的一个辅助函数，用于将一组图像和对应的标签添加到敏感性图像sensitivity_im中
"""
For a given number of images, the probability of predicting a given label is obtained. Attach to previous
assessment.
"""
# 首先将batch_images堆叠成一个二维数组
batch_images = np.vstack(batch_images)
# 然后将batch_ids扩展为一个二维数组
batch_ids = np.expand_dims(batch_ids, 1)
# 接下来，使用model模型对这些图像进行预测
model_numpy = model(Tensor(batch_images)).asnumpy()
# 通过将first_indices扩展为一个二维数组，可以方便地将其与batch_ids相匹配
first_indices = np.arange(batch_ids.shape[0])[:, None]
# 并将预测结果与first_indices相匹配
scores = model_numpy[first_indices, batch_ids]
# 如果sensitivity_im为空
if sensitivity_im.size == 0:
# 则直接返回scores
return np.vstack(scores)
# 否则，将sensitivity_im和scores合并并返回
return np.vstack((sensitivity_im, scores))
def _check_input_bounding_box(b_box, im_shape):
# 检查边界框（如果提供了）是否符合预期
"""Check that the bounding box (if supplied) is as expected."""
# 如果还没有提供边界框（b_box），那么就将边界框的最小值（b_box_min）和最大值（b_box_max）设置为None
# If no bounding box has been supplied, set min and max to None
# 首先判断边界框（b_box）是否为None
if b_box is None:
# 如果为None，则将边界框的最小值（b_box_min）和最大值（b_box_max）设置为None
b_box_min = b_box_max = None
# 否则，检查边界框的长度是否为2乘以预测值的长度（im_shape）
else:
if len(b_box) != 2 * len(im_shape):
# 如果不是，则抛出一个ValueError异常，表示边界框的格式不正确
raise ValueError(f"For 'OcclusionSensitivity', the bounding box should contain upper and lower for "
f"all dimensions (except batch number), and the length of 'b_box' should be twice "
f"as long as predicted value's (except batch number), but got 'b_box' length "
f"{len(b_box)}, predicted value length (except batch number) {len(im_shape)}.")
# 首先将边界框（b_box）的上下边界（b_box_min和b_box_max）转换为numpy数组
b_box_min = np.array(b_box[::2])
b_box_max = np.array(b_box[1::2])
# 然后，对于所有小于0的值，将其设置为0
b_box_min[b_box_min < 0] = 0
# 对于所有小于0的值，将它们设置为图像尺寸减1
b_box_max[b_box_max < 0] = im_shape[b_box_max < 0] - 1
# 最后，检查最大边界框是否大于等于图像尺寸
if np.any(b_box_max >= im_shape):
# 如果是，则抛出一个ValueError异常，表示最大边界框尺寸不正确
raise ValueError("For 'OcclusionSensitivity', maximum bounding box should be smaller than image size "
"for all values.")
# 检查最小边界框是否小于最大边界框
if np.any(b_box_min > b_box_max):
# 如果是，则抛出一个ValueError异常，表示最小边界框尺寸不正确
raise ValueError("For 'OcclusionSensitivity', minimum bounding box should be smaller than maximum "
"bounding box for all values.")
# 返回边界框的最小值（b_box_min）和最大值（b_box_max）
return b_box_min, b_box_max

59
mindspore/python/mindspore/nn/metrics/perplexity.py
View File

@ -12,14 +12,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# Perplexity模块主要用于计算困惑度（Perplexity），是一种用于评估语言模型性能的指标。
# 在深度学习中，困惑度是一种衡量模型预测结果好坏的标准，数值越小，模型预测越准确
"""Perplexity"""
# 导入数学库，包含了一些基本的数学函数，如sqrt、log等
import math
# 导入NumPy库，是一个用于数值计算和数据处理的库
import numpy as np
# 从mindspore._checkparam模块中导入Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 从当前模块的metric文件中导入Metric类和rearrange_inputs函数
from .metric import Metric, rearrange_inputs
# Metric类是一个通用的评估指标类，可以用于计算多种指标，如准确率、精确度、召回率等。rearrange_inputs函数用于对输入的数据进行重排，以便适合于计算指标
class Perplexity(Metric):
# 类的主要作用是计算困惑度（Perplexity），是一种用于评估语言模型性能的指标。
# Perplexity类重写了Metric类中的update方法，用于更新混淆矩阵和计算困惑度。
# 在计算困惑度时，首先对输入的数据进行重排，然后计算概率分布的对数，最后对数求平方根。
# 参数ignore_label用于指定一个无效标签，在计算混淆矩阵和计算困惑度时，忽略该标签。如果设置为None，则包含所有标签。默认值为None
r"""
Computes perplexity. Perplexity is a measurement about how well a probability distribution or a model predicts a
sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows:
@ -50,21 +61,33 @@ class Perplexity(Metric):
"""
def __init__(self, ignore_label=None):
# 首先调用父类的__init__方法
super(Perplexity, self).__init__()
# 然后检查ignore_label参数的类型
if ignore_label is None:
# 如果为None，则将ignore_label设置为None
self.ignore_label = ignore_label
else:
# 否则，检查ignore_label参数是否为整数类型
self.ignore_label = validator.check_value_type("ignore_label", ignore_label, [int])
# 如果不是，则抛出异常
self.clear()
# 最后，清空混淆矩阵和计数器
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 将_sum_metric的值重置为0.0
self._sum_metric = 0.0
# 将_num_inst的值设置为0
self._num_inst = 0
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果。update方法接受任意数量的关键字参数，将这些参数作为输入，并将其重新排列以适应计算指标。
# rearrange_inputs函数用于对输入的数据进行重排，以便适合于计算指标。重排后的数据格式为(N, C)，其中N是样本数量，C是类别数量
"""
Updates the internal evaluation result `preds` and `labels`.
@ -78,40 +101,73 @@ class Perplexity(Metric):
RuntimeError: If preds and labels have different lengths.
RuntimeError: If label shape is not equal to pred shape.
"""
"""
检查输入的参数是否符合要求
"""
# 如果输入参数的数量不为2
if len(inputs) != 2:
# 则抛出一个ValueError异常，表示需要传入2个输入参数，但实际传入的数量为len(inputs)
raise ValueError("For 'Perplexity.update', it needs 2 inputs (predicted value, label), but got {}."
.format(len(inputs)))
# 从inputs元组中获取第一项为preds
preds = [self._convert_data(inputs[0])]
# 从inputs元组中获取第二项为labels
labels = [self._convert_data(inputs[1])]
# 检查输入的preds和labels列表的长度是否相同
if len(preds) != len(labels):
# 如果不相同，则抛出一个RuntimeError异常，表示预测值（input[0]）和标签（input[1]）应该具有相同的长度，但实际得到的是preds长度为len(preds)，标签长度为len(labels)
raise RuntimeError("For 'Perplexity.update', predicted value (input[0]) and label (input[1]) should have "
"the same length, but got predicted value length {}, label length {}."
.format(len(preds), len(labels)))
"""
update方法中，用于计算损失（loss）和数量（num）的部分
"""
loss = 0.
num = 0
# 首先，将labels和preds列表中的元素一一对应地组合成一个新的元组列表，然后遍历这个列表
for label, pred in zip(labels, preds):
# 检查每个元组中的标签（label）和预测值（pred）的形状是否相同
if label.size != pred.size / pred.shape[-1]:
# 如果不同，则抛出一个RuntimeError异常，表示预测值（input[0]）和标签（input[1]）应该具有相同的长度，但实际得到的是preds长度为len(preds)，标签长度为len(labels)
raise RuntimeError("For 'Perplexity.update', predicted value (input[0]) and label (input[1]) should "
"have the same shape, but got predicted value shape {}, label shape {}."
.format(pred.shape, label.shape))
"""
这段代码是用于处理标签（label）数据的
"""
# 首先，将标签的形状调整为一个一维数组
label = label.reshape((label.size,))
# 然后将其转换为整数类型
label_expand = label.astype(int)
# 接下来，使用np.expand_dims方法在标签的形状上添加一个维度
label_expand = np.expand_dims(label_expand, axis=1)
# 最后使用np.arange方法生成一个从0开始的一维数组，并将其与标签数组相乘
first_indices = np.arange(label_expand.shape[0])[:, None]
# 最后，使用np.squeeze方法删除不必要的维度
pred = np.squeeze(pred[first_indices, label_expand])
# 如果self.ignore_label不为None
if self.ignore_label is not None:
# 那么对于标签（label）中的值为self.ignore_label的元素，将其转换为1，其他元素转换为0
ignore = (label == self.ignore_label).astype(pred.dtype)
# 并累加得到新的预测值
num -= np.sum(ignore)
# 然后，将转换后的标签数组与预测值（pred）相乘
pred = pred * (1 - ignore) + ignore
# 最后，将新的预测值求对数并累加得到损失值
loss -= np.sum(np.log(np.maximum(1e-10, pred)))
# 在计算损失（loss）和数量（num）之后，将预测值（pred）的尺寸累加到num中
num += pred.size
# 将损失值（loss）累加到self._sum_metric中
self._sum_metric += loss
# 并将数量（num）累加到self._num_inst中
self._num_inst += num
def eval(self):
# 用于计算困惑度（Perplexity），需要在update函数后调用，否则会报错
r"""
Returns the current evaluation result.
@ -121,8 +177,11 @@ class Perplexity(Metric):
Raises:
RuntimeError: If the sample size is 0.
"""
# 首先，检查self._num_inst是否为0
if self._num_inst == 0:
# 如果为0，则抛出一个RuntimeError
raise RuntimeError("The 'Perplexity' can not be calculated, because the number of samples is 0, please "
"check whether has called update method before calling eval method.")
# 然后，计算困惑度，即math.exp(self._sum_metric / self._num_inst)。最后，返回计算得到的困惑度
return math.exp(self._sum_metric / self._num_inst)

70
mindspore/python/mindspore/nn/metrics/precision.py
View File

@ -12,16 +12,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# Precision的类，用于计算准确率
"""Precision."""
# 首先，导入了所需的库和模块，如sys、numpy等
import sys
import numpy as np
# 从mindspore库中导入了一个名为Validator的类，并将其重命名为validator。这个类用于验证参数是否符合预期的范围和类型
from mindspore._checkparam import Validator as validator
# 然后，定义了一个名为Precision的类，继承自EvaluationBase
from .metric import EvaluationBase, rearrange_inputs, _check_onehot_data
# 从metric模块中导入了三个函数：EvaluationBase、rearrange_inputs和_check_onehot_data。这些函数用于计算准确率，并处理输入数据格式
class Precision(EvaluationBase):
# 用于计算准确率。这个类继承自EvaluationBase。在计算准确率时，会使用两个局部变量true_positive和false_positive来记录正确的正例数和错误的正例数。计算公式为：
# math::\text{precision} = \frac{\text{true_positive}}{\text{true_positive} + \text{false_positive}}
# 注意：在多标签 cases 中，y和y_pred中的元素必须为0或1
r"""
Calculates precision for classification and multilabel data.
@ -54,25 +63,40 @@ class Precision(EvaluationBase):
[0.5 1. ]
"""
# 在初始化时，需要传入一个参数eval_type，默认为'classification'
def __init__(self, eval_type='classification'):
# 接着，调用父类的__init__方法，将eval_type传递给父类
super(Precision, self).__init__(eval_type)
# 然后，初始化一个局部变量eps，其值为sys.float_info.min，用于避免除以0的情况
self.eps = sys.float_info.min
# 最后，调用clear方法清空记录的准确率数据
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 首先将_class_num重置为0
self._class_num = 0
# 然后根据_type的值清空不同的记录数据
if self._type == "multilabel":
# 在multilabel cases中，清空_true_positives和_positives两个数组
self._true_positives = np.empty(0)
self._positives = np.empty(0)
# 并将_true_positives_average和_positives_average重置为0
self._true_positives_average = 0
self._positives_average = 0
else:
# 在classification cases中，只清空_true_positives和_positives两个变量
self._true_positives = 0
# 并将_true_positives_average和_positives_average重置为0
self._positives = 0
@rearrange_inputs
def update(self, *inputs):
# 在update方法中，接收两个输入参数y_pred和y，并对它们进行处理
# 输入的y_pred和y可以是Tensor、列表或numpy.ndarray类型。在classification cases中，y_pred通常是一个浮点数列表，
# 其范围在0到1之间，且形状为(N, C)，其中N是样本数，C是类别数。对于multilabel cases，输入的y_pred和y必须是one-hot编码的，
# 其中1表示正类。y的形状通常是(N, C)，而y_pred的形状通常是(N, C)。
"""
Updates the internal evaluation result with `y_pred` and `y`.
@ -90,50 +114,85 @@ class Precision(EvaluationBase):
Raises:
ValueError: If the number of inputs is not 2.
"""
"""
检查输入的y_pred和y是否符合准确率计算的要求
"""
# 如果输入参数的数量不是2
if len(inputs) != 2:
# 将抛出一个ValueError异常
raise ValueError("For 'Precision.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 首先，使用_convert_data方法将输入的数据转换为正确的格式
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 首先，检查_type是否为'classification'，且y_pred和y的维度是否相同，然后，使用_check_onehot_data函数检查y是否为one-hot编码
if self._type == 'classification' and y_pred.ndim == y.ndim and _check_onehot_data(y):
# 。如果满足这些条件，将y转换为argmax索引，即实际类别标签
y = y.argmax(axis=1)
# 最后，使用_check_shape和_check_value函数检查y_pred和y的形状和值是否符合准确率计算的要求
self._check_shape(y_pred, y)
self._check_value(y_pred, y)
# 检查Precision类的_class_num是否为0
if self._class_num == 0:
# 如果匹配，将_class_num更新为y_pred的类别数
self._class_num = y_pred.shape[1]
# 以及y_pred的形状是否与_class_num匹配
elif y_pred.shape[1] != self._class_num:
# 如果不匹配，将抛出一个ValueError异常
raise ValueError("For 'Precision.update', class number not match, last input predicted data contain {} "
"classes, but current predicted data contain {} classes, please check your predicted "
"value(inputs[0])".format(self._class_num, y_pred.shape[1]))
# 最后，从y_pred中提取class_num，以便在后续计算中使用
class_num = self._class_num
# 首先，检查_type是否为'classification'（二分类）
if self._type == "classification":
# 然后，检查y的最大值是否大于class_num
if y.max() + 1 > class_num:
# 如果是，则抛出一个ValueError异常
raise ValueError("For 'Precision.update', predicted value (input[0]) should have the same classes "
"number as true value (input[1]), but got predicted value classes {}, true value "
"classes {}.".format(class_num, y.max() + 1))
# 然后，根据_type的值，将y转换为one-hot编码
y = np.eye(class_num)[y.reshape(-1)]
indices = y_pred.argmax(axis=1).reshape(-1)
# 并将y_pred转换为相应的格式
y_pred = np.eye(class_num)[indices]
# 对于multilabel（多分类） cases
elif self._type == "multilabel":
# 将y_pred和y交换轴
y_pred = y_pred.swapaxes(1, 0).reshape(class_num, -1)
# 并将其reshape为(class_num, -1)的形状
y = y.swapaxes(1, 0).reshape(class_num, -1)
"""
以下代码计算准确率
"""
# 首先，计算y_pred中每个类别的正例数量，并将它们累加到一个变量positives中
positives = y_pred.sum(axis=0)
# 然后，计算y和y_pred中每个类别的真实正例数量，并将它们累加到一个变量true_positives中
true_positives = (y * y_pred).sum(axis=0)
# 接下来，根据_type的值进行处理。对于multilabel cases
if self._type == "multilabel":
# 将true_positives除以positives，然后对结果进行求和，并将和累加到self._true_positives_average中
self._true_positives_average += np.sum(true_positives / (positives + self.eps))
# 同时，将positives累加到self._positives_average中
self._positives_average += len(positives)
# 最后，将true_positives和positives concatenate到相应的self._true_positives和self._positives中
self._true_positives = np.concatenate((self._true_positives, true_positives), axis=0)
self._positives = np.concatenate((self._positives, positives), axis=0)
# 对于classification cases
else:
# 直接将true_positives和positives累加到相应的变量中
self._true_positives += true_positives
self._positives += positives
def eval(self, average=False):
# eval方法，用于计算准确率,该方法需要在update方法后调用，否则会报错
"""
Computes the precision.
@ -143,16 +202,25 @@ class Precision(EvaluationBase):
Returns:
numpy.float64, the computed result.
"""
# 首先，检查self._class_num是否为0
if self._class_num == 0:
# 如果是，则抛出一个RuntimeError异常
raise RuntimeError("The 'Precision' can not be calculated, because the number of samples is 0, "
"please check whether your inputs (predicted value, true value) are empty, or "
"has called update method before calling eval method.")
# 然后，检查average参数的类型是否为bool，如果不是，则抛出一个TypeError异常
validator.check_value_type("average", average, [bool], self.__class__.__name__)
# 接下来，计算准确率，即true_positives除以positives，并将结果累加到一个变量result中
result = self._true_positives / (self._positives + self.eps)
# 如果average为True，对结果进行求和，并返回平均值
if average:
# 最后，根据self._type的值进行处理
if self._type == "multilabel":
# 对于multilabel cases，将self._true_positives_average和self._positives_average除以self.eps，然后对结果进行求和，并将和累加到result中
result = self._true_positives_average / (self._positives_average + self.eps)
# 对于classification cases，直接将true_positives和positives累加到result中
return result.mean()
# 否则直接返回结果
return result

60
mindspore/python/mindspore/nn/metrics/recall.py
View File

@ -12,16 +12,22 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# Recall类，用于计算召回率（Recall）
"""Recall."""
# 用于访问与Python解释器相关的变量和函数
import sys
# 导入了所需的库和模块，如numpy和mindspore
import numpy as np
# 从mindspore._checkparam模块中导入了一个名为validator的类，用于验证参数
from mindspore._checkparam import Validator as validator
# Recall类继承自EvaluationBase类
from .metric import EvaluationBase, rearrange_inputs, _check_onehot_data
# 从mindspore.metric模块中导入了一个名为EvaluationBase的类和三个函数，分别是rearrange_inputs、_check_onehot_data
class Recall(EvaluationBase):
# 用于计算召回率（Recall）。Recall类继承自EvaluationBase类。在计算召回率时，会使用两个本地变量true_positive和false_negative来存储真实阳性和假负例的数量。
# 需要注意的是，在多分类 cases 中，元素的y和y_pred必须为0或1
r"""
Calculates recall for classification and multilabel data.
@ -54,25 +60,38 @@ class Recall(EvaluationBase):
>>> print(recall)
[1. 0.5]
"""
# 在定义Recall类时，需要初始化一个eval_type参数，默认为'classification'
def __init__(self, eval_type='classification'):
# 然后，调用父类的构造函数，将self传递给父类
super(Recall, self).__init__(eval_type)
# 接着，初始化一个eps属性，用于存储一个极小值
self.eps = sys.float_info.min
# 最后，调用clear方法来清空内部变量
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 首先，将_class_num属性设置为0
self._class_num = 0
# 如果_type属性为"multilabel"
if self._type == "multilabel":
# 则将_true_positives和_actual_positives属性设置为空数组
self._true_positives = np.empty(0)
self._actual_positives = np.empty(0)
# 并将_true_positives_average和_actual_positives_average属性设置为0
self._true_positives_average = 0
self._actual_positives_average = 0
else:
# 否则，将_true_positives和_actual_positives属性设置为0
self._true_positives = 0
self._actual_positives = 0
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果。接收一个或多个输入参数，分别是y_pred和y。对于'classification'评估类型，y_pred通常是一个浮点数列表，范围在0到1之间，
# 形状为(N, C)，其中N是样本数量，C是类别数量。对于'multilabel'评估类型，y_pred和y必须是one-hot编码的数组，值全为0或1。
# indices中值为1的索引表示正类别。y_pred和y的形状都是(N, C)
"""
Updates the internal evaluation result with `y_pred` and `y`.
@ -91,49 +110,78 @@ class Recall(EvaluationBase):
Raises:
ValueError: If the number of inputs is not 2.
"""
# 首先检查输入参数的数量是否为2
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError异常
raise ValueError("For 'Recall.update', it needs 2 inputs (predicted value, true value), "
"but got {}.".format(len(inputs)))
# 然后，将输入的y_pred和true value转换为适当的数据格式
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 如果评估类型为'classification'，并且y_pred的形状与y相同，并且是one-hot编码
if self._type == 'classification' and y_pred.ndim == y.ndim and _check_onehot_data(y):
# 那么将y转换为argmax轴1的值
y = y.argmax(axis=1)
# 最后，检查y_pred和y的形状和值是否符合要求
self._check_shape(y_pred, y)
self._check_value(y_pred, y)
# 检查_class_num属性是否为0
if self._class_num == 0:
# 如果是，则将其设置为y_pred的形状[1]
self._class_num = y_pred.shape[1]
# 如果y_pred的形状[1]与_class_num不同
elif y_pred.shape[1] != self._class_num:
# 则抛出一个ValueError异常
raise ValueError("For 'Recall.update', class number not match, last input predicted data contain {} "
"classes, but current predicted data contain {} classes, please check your predicted "
"value(inputs[0]).".format(self._class_num, y_pred.shape[1]))
# 首先获取_class_num属性，然后根据评估类型进行相应的处理
class_num = self._class_num
# 如果评估类型为'classification'
if self._type == "classification":
# 首先检查y的最大值是否大于_class_num
if y.max() + 1 > class_num:
# 如果是，则抛出一个ValueError异常
raise ValueError("For 'Recall.update', predicted value (input[0]) should have the same classes number "
"as true value (input[1]), but got predicted value classes {}, true value classes {}."
.format(class_num, y.max() + 1))
# 接着，将y转换为one-hot编码
y = np.eye(class_num)[y.reshape(-1)]
# 并获取y_pred中每个类别概率最大的索引
indices = y_pred.argmax(axis=1).reshape(-1)
# 最后，将y_pred转换回one-hot编码
y_pred = np.eye(class_num)[indices]
# 如果评估类型为'multilabel'
elif self._type == "multilabel":
# 则将y_pred和y交换轴
y_pred = y_pred.swapaxes(1, 0).reshape(class_num, -1)
# 并reshape为(class_num, -1)
y = y.swapaxes(1, 0).reshape(class_num, -1)
# 首先，使用sum函数将y沿axis=0轴求和，得到actual_positives
actual_positives = y.sum(axis=0)
# 然后使用点乘运算将y_pred和y相乘，再沿axis=0轴求和，得到true_positives
true_positives = (y * y_pred).sum(axis=0)
# 如果评估类型为'multilabel'
if self._type == "multilabel":
# 那么将true_positives除以(actual_positives + self.eps)后求和，然后累加到self._true_positives_average中
self._true_positives_average += np.sum(true_positives / (actual_positives + self.eps))
# 接着，将actual_positives累加到self._actual_positives_average中
self._actual_positives_average += len(actual_positives)
# 最后，将true_positives和actual_positives拼接在一起，存储在self._true_positives和self._actual_positives中
self._true_positives = np.concatenate((self._true_positives, true_positives), axis=0)
self._actual_positives = np.concatenate((self._actual_positives, actual_positives), axis=0)
else:
# 如果评估类型为'classification'，那么直接将true_positives和actual_positives累加在一起
self._true_positives += true_positives
# 存储在self._true_positives和self._actual_positives中
self._actual_positives += actual_positives
def eval(self, average=False):
# 用于计算召回率。方法接收一个名为average的布尔参数，用于指定是否计算平均召回率
"""
Computes the recall.
@ -143,16 +191,24 @@ class Recall(EvaluationBase):
Returns:
numpy.float64, the computed result.
"""
# 检查输入参数是否为空
if self._class_num == 0:
# 如果是，则抛出一个RuntimeError异常
raise RuntimeError("The 'Recall' can not be calculated, because the number of samples is 0, please check "
"whether your inputs (predicted value, true value) are empty, or has called update "
"method before calling eval method.")
# 检查输入参数average是否为布尔值，如果不是，则抛出一个TypeError异常
validator.check_value_type("average", average, [bool], self.__class__.__name__)
# 然后，计算召回率
result = self._true_positives / (self._actual_positives + self.eps)
# 如果average为True，则计算平均结果
if average:
# 如果type为多标签，则计算多标签评估下的召回率
if self._type == "multilabel":
result = self._true_positives_average / (self._actual_positives_average + self.eps)
# 否则，计算正确结果的平均值
return result.mean()
# 否则，返回计算得到的召回率
return result

77
mindspore/python/mindspore/nn/metrics/roc.py
View File

@ -12,13 +12,22 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 主要用于计算接收者操作特征（ROC）曲线。它使用了NumPy库来处理数据，以及MindSpore库中的Metric和_binary_clf_curve函数。
# 这段代码的主要功能是计算ROC曲线，用于二分类问题的评估
"""ROC"""
# 首先，导入了所需的库，包括NumPy库和MindSpore库
import numpy as np
# 导入了一个名为"validator"的类，用于验证参数
from mindspore._checkparam import Validator as validator
# 从"metric"模块中导入了一个名为"Metric"的类，用于计算评估指标
from .metric import Metric, rearrange_inputs, _binary_clf_curve
# 从"metric"模块中导入了一个名为"_binary_clf_curve"的函数，用于计算二分类问题的ROC曲线
class ROC(Metric):
# 该类的主要功能是计算接收者操作特征（ROC）曲线，用于二分类问题的评估。在多分类问题中，它会根据一个vs所有（one-vs-all）的方式计算ROC曲线
# 参数说明：class_num (int)：类别的数量。在二分类问题中，不需要提供此参数。默认值：None。
# pos_label (int)：确定正类的整数。对于二分类问题，默认值为1。对于多分类问题，此参数不应设置，因为它会根据range [0,num_classes-1]的整数迭代变化。默认值：None。
"""
Calculates the ROC curve. It is suitable for solving binary classification and multi classification problems.
In the case of multiclass, the values will be calculated based on a one-vs-the-rest approach.
@ -69,20 +78,28 @@ class ROC(Metric):
array([1.75, 0.75, 0.05])]
"""
def __init__(self, class_num=None, pos_label=None):
# 首先，使用super()调用父类的构造函数，完成类的初始化
super().__init__()
# 然后，将传入的"class_num"和"pos_label"参数赋值给类的成员变量。这里使用validator库中的check_value_type函数来验证参数的类型
self.class_num = class_num if class_num is None else validator.check_value_type("class_num", class_num, [int])
self.pos_label = pos_label if pos_label is None else validator.check_value_type("pos_label", pos_label, [int])
# 最后，调用self.clear()函数清空类的其他成员变量
self.clear()
def clear(self):
# 用于清空类内部评估的结果
"""Clear the internal evaluation result."""
# 将类的成员变量"y_pred"、"y"和"sample_weights"的值重置为0
self.y_pred = 0
self.y = 0
self.sample_weights = None
# 并将"_is_update"设置为False
self._is_update = False
# 这样，在计算ROC曲线时，可以重新开始计算
@rearrange_inputs
def update(self, *inputs):
# 用于更新状态，以便在计算ROC曲线时使用
"""
Update state with predictions and targets.
@ -93,51 +110,81 @@ class ROC(Metric):
is the number of categories. y contains values of integers. The shape is :math:`(N,C)` if one-hot
encoding is used. Shape can also be :math:`(N,)` if category index is used.
"""
# 首先，检查传入的输入参数数量是否为2
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError异常
raise ValueError("For 'ROC.update', it needs 2 inputs (predicted value, true value), but got {}"
.format(len(inputs)))
# 然后，将传入的"y_pred"和"y"参数转换为NumPy数组
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 调用_precision_recall_curve_update函数，计算ROC曲线的参数。这个函数会根据传入的"y_pred"、"y"、"class_num"和"pos_label"参数计算ROC曲线的相关参数
y_pred, y, class_num, pos_label = _precision_recall_curve_update(y_pred, y, self.class_num, self.pos_label)
# 将计算得到的参数赋值给类的成员变量"y_pred"、"y"、"class_num"和"pos_label"
self.y_pred = y_pred
self.y = y
self.class_num = class_num
self.pos_label = pos_label
# 最后，将"_is_update"设置为True，表示已经更新了状态
self._is_update = True
def _roc_eval(self, y_pred, y, class_num, pos_label, sample_weights=None):
# 一个辅助函数，用于计算ROC曲线。它根据传入的"y_pred"、"y"、"class_num"和"pos_label"参数计算ROC曲线的真阳率（tpr）和假阳率（fpr）
"""Computes the ROC curve."""
# 首先，检查"class_num"是否为1
if class_num == 1:
# 如果是，则调用_binary_clf_curve函数计算二分类问题的ROC曲线
fps, tps, thresholds = _binary_clf_curve(y_pred, y, sample_weights=sample_weights, pos_label=pos_label)
# 将"tps"和"fps"数组分别压缩和拼接成一个新的数组。
tps = np.squeeze(np.hstack([np.zeros(1, dtype=tps.dtype), tps]))
fps = np.squeeze(np.hstack([np.zeros(1, dtype=fps.dtype), fps]))
# 同时，将"thresholds"数组拼接成一个新数组，并在开头插入一个0
thresholds = np.hstack([thresholds[0][None] + 1, thresholds])
# 这样，就可以得到一个新的数组，其中包含了所有可能的阈值，以及对应的真阳率（tpr）和假阳率（fpr）
# 检查假阳率（fpr）和真阳率（tpr）是否有效
# 如果假阳率（fpr）小于等于0
if fps[-1] <= 0:
# 表示没有负样本在真实值中，假阳率是无意义的
raise ValueError("For 'ROC.eval', there is no negative samples in true value, "
"false positive value is meaningless.")
fpr = fps / fps[-1]
# 如果真阳率（tpr）为0
if tps[-1] <= 0:
# 表示没有正样本在真实值中，真阳率是无意义的
raise ValueError("For 'ROC.eval', there is no positive samples in true value, "
"true positive value is meaningless.")
tpr = tps / tps[-1]
# 返回假阳率（fpr）、真阳率（tpr）和阈值数组。通过计算假阳率（fpr）和真阳率（tpr）曲线，可以直观地看出模型的性能
return fpr, tpr, thresholds
# 如果class_num不等于1，首先初始化假阳率（fpr）、真阳率（tpr）和阈值数组
fpr, tpr, thresholds = [], [], []
# 然后遍历每个类别
for c in range(class_num):
# 首先，从"y_pred"数组中提取当前类别的预测结果（preds_c）
preds_c = y_pred[:, c]
# 然后调用_roc方法计算ROC曲线
res = self._roc(preds_c, y, class_num=1, pos_label=c, sample_weights=sample_weights)
# 最后，将计算得到的假阳率（fpr）、真阳率（tpr）和阈值数组分别添加到相应的数组中
fpr.append(res[0])
tpr.append(res[1])
thresholds.append(res[2])
# 返回假阳率（fpr）、真阳率（tpr）和阈值数组
return fpr, tpr, thresholds
def _roc(self, y_pred, y, class_num=None, pos_label=None, sample_weights=None):
# 一个私有方法_roc，用于计算ROC曲线。它接收以下参数：
# y_pred：预测结果，通常是一个浮点数列表，范围在0到1之间，形状为(N, C)，其中N是样本数量，C是类别数量。
# y：真实值，通常是一个整数列表。
# class_num：类别数量。对于二分类问题，不需要提供这个参数。默认值为None。
# pos_label：正类标签。对于二分类问题，可以设置为1。对于多分类问题，不要设置这个参数，因为它是迭代更新的。默认值为None。
# sample_weights：样本权重。如果为None，则权重值为1；如果为ndarray，则权重值为ndarray值。
"""
Update curve and return the result of the ROC curve.
@ -154,11 +201,14 @@ class ROC(Metric):
sample_weights (Union[None, np.ndarray]): If sample_weights is None, the weight value is 1.
If sample_weights is ndarray, the weight value is the ndarray value.
"""
# 首先对y_pred和y进行预处理
y_pred, y, class_num, pos_label = _precision_recall_curve_update(y_pred, y, class_num, pos_label)
# 然后调用_roc_eval方法计算ROC曲线
return self._roc_eval(y_pred, y, class_num, pos_label, sample_weights)
def eval(self):
# 用于计算ROC曲线
"""
Computes the ROC curve.
@ -177,44 +227,71 @@ class ROC(Metric):
RuntimeError: If the update method is not called first, an error will be reported.
"""
# 如果_is_update is False，表示参数未更新
if self._is_update is False:
# 则抛出错误RuntimeError
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 将"y_pred"和"y"数组分别压缩和拼接成一个新的数组。
y_pred = np.squeeze(np.vstack(self.y_pred))
y = np.squeeze(np.vstack(self.y))
# 然后调用_roc_eval方法计算ROC曲线
return self._roc_eval(y_pred, y, self.class_num, self.pos_label)
def _precision_recall_curve_update(y_pred, y, class_num, pos_label):
"""update curve"""
# 定义了一个名为_precision_recall_curve_update的私有方法，用于更新曲线。它接收以下参数：
# y_pred：预测结果，通常是一个浮点数列表，范围在0到1之间，形状为(N, C)，其中N是样本数量，C是类别数量。
# y：真实值，通常是一个整数列表。
# class_num：类别数量。对于二分类问题，不需要提供这个参数。默认值为None。
# pos_label：正类标签。对于二分类问题，可以设置为1。对于多分类问题，不要设置这个参数，因为它是迭代更新的。默认值为None
# 检查预测结果（y_pred）和真实值（y）的形状是否相同，或者预测结果的形状是否等于真实值加1
if not (len(y_pred.shape) == len(y.shape) or len(y_pred.shape) == len(y.shape) + 1):
# 如果不满足这些条件，则抛出一个错误
raise ValueError(f"For 'ROC', predicted value (input[0]) and true value (input[1]) should have same "
f"dimensions, or the dimension of predicted value equal the dimension of true value add "
f"1, but got predicted value ndim: {len(y_pred.shape)}, true value ndim: {len(y.shape)}.")
# single class evaluation
# 检查预测结果（y_pred）和真实值（y）的形状是否相同
if len(y_pred.shape) == len(y.shape):
# 以及是否提供了正确的类别数量（class_num）和正类标签（pos_label）
if class_num is not None and class_num != 1:
# 如果没有提供正确的类别数量和正类标签，或者预测结果和真实值的形状相同，则抛出一个错误
raise ValueError(f"For 'ROC', when predicted value (input[0]) and true value (input[1]) have the same "
f"shape, the 'class_num' should be 1, but got {class_num}.")
# 将类别数量（class_num）设置为1
class_num = 1
# 然后检查正类标签（pos_label）是否为None
if pos_label is None:
# 如果为None，则将其设置为1
pos_label = 1
# 最后，将预测结果（y_pred）和真实值（y）展平为一个一维数组
y_pred = y_pred.flatten()
y = y.flatten()
# multi class evaluation
# 当预测结果的形状等于真实值加1时
elif len(y_pred.shape) == len(y.shape) + 1:
# 如果提供了正类标签（pos_label）
if pos_label is not None:
# 则抛出一个错误ValueError
raise ValueError(f"For 'ROC', when the dimension of predicted value (input[0]) equals the dimension "
f"of true value (input[1]) add 1, the 'pos_label' should be None, "
f"but got {pos_label}.")
# 如果类别数量（class_num）与预测结果的形状[1]不同
if class_num != y_pred.shape[1]:
# 则抛出一个错误ValueError
raise ValueError("For 'ROC', the 'class_num' should equal the number of classes from predicted value "
"(input[0]), but got 'class_num' {}, the number of classes from predicted value {}."
.format(class_num, y_pred.shape[1]))
# 最后，将预测结果的形状转换为(class_num, -1)
y_pred = y_pred.transpose(0, 1).reshape(class_num, -1).transpose(0, 1)
# 然后展平为一个一维数组
y = y.flatten()
# 返回假阳率（fpr）、真阳率（tpr）和阈值数组
return y_pred, y, class_num, pos_label

73
mindspore/python/mindspore/nn/metrics/root_mean_square_surface_distance.py
View File

@ -12,14 +12,27 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 用于计算根均方表面距离（RMSD）
"""RootMeanSquareSurfaceDistance."""
# 首先，从scipy.ndimage模块导入morphology函数，用于对图像进行形态学操作
from scipy.ndimage import morphology
# 然后，从numpy模块导入np，用于处理数值计算
import numpy as np
# 从mindspore._checkparam模块导入Validator类，用于参数验证
from mindspore._checkparam import Validator as validator
# 从mindspore.metric模块导入Metric类，用于计算评估指标
from .metric import Metric, rearrange_inputs
# rearrange_inputs函数接收一个输入数据inputs和一个整数batch_size，然后将输入数据按照batch_size进行分组，并将每组数据转换为一个一维数组
class RootMeanSquareDistance(Metric):
# 它是Metric类的子类。RootMeanSquareDistance类用于计算两个集合A和B之间的Root Mean Square Surface Distance。
# Root Mean Square Surface Distance (RMSSD)是一种衡量两个集合A和B之间的距离度量。
# 参数distance_metric用于指定距离度量方法，支持"euclidean"（欧氏距离）、"chessboard"（棋盘距离）和"taxicab"（出租车距离）。默认方法是"euclidean"。
# 参数symmetric用于指定是否计算对称的Root Mean Square Surface Distance。如果为False，只计算RmsSurDis(y_{pred}, y)，
# 即从y_pred到y的表面距离；如果为True，计算从y_pred到y和从y到y_pred的均值，即RmsSurDis({y_pred} \leftrightarrow y)。默认值为False。
r"""
Computes the Root Mean Square Surface Distance from `y_pred` to `y` under the default setting.
@ -69,46 +82,70 @@ class RootMeanSquareDistance(Metric):
1.0000000000000002
"""
# 在初始化RootMeanSquareDistance类时，需要传入两个参数：symmetric和distance_metric。
def __init__(self, symmetric=False, distance_metric="euclidean"):
# 首先，代码调用父类的__init__方法，初始化Metric类
super(RootMeanSquareDistance, self).__init__()
# 后，定义了一个名为distance_metric_list的列表，用于存储支持的所有距离度量方法
self.distance_metric_list = ["euclidean", "chessboard", "taxicab"]
# 接下来，将distance_metric参数转换为小写
distance_metric = validator.check_value_type("distance_metric", distance_metric, [str])
# 并检查其是否在distance_metric_list中
self.distance_metric = validator.check_string(distance_metric, self.distance_metric_list, "distance_metric")
# 最后，检查symmetric是否为bool值，并设置symmetric和distance_metric为类的属性
self.symmetric = validator.check_value_type("symmetric", symmetric, [bool])
# 初始化clear方法，清空残余数据
self.clear()
# 用于存储y_pred顶点的边，初始为None
self._y_pred_edges = None
# 用于存储是否已经更新过_y_pred_edges的值，初始为None
self._is_update = None
# 用于存储y顶点的边，初始为None
self._y_edges = None
def _get_surface_distance(self, y_pred_edges, y_edges):
# 用于计算从y_pred_edges到y_edges的表面距离。y_pred_edges是y_pred顶点的边，y_edges是y顶点的边
"""
Calculate the surface distances from `y_pred_edges` to `y_edges`.
Args:
Args:
y_pred_edges (np.ndarray): the edge of the predictions.
y_edges (np.ndarray): the edge of the ground truth.
"""
# 如果y_pred_edges为空，则返回空数组
if not np.any(y_pred_edges):
# 如果为空，则返回一个空数组
return np.array([])
# 如果y_edges为空
if not np.any(y_edges):
# 则返回一个全部为inf的数组
dis = np.full(y_edges.shape, np.inf)
# 否则，如果distance_metric为euclidean，则计算距离
else:
# 如果distance_metric为"euclidean"
if self.distance_metric == "euclidean":
# 则使用morphology.distance_transform_edt函数计算欧氏距离
dis = morphology.distance_transform_edt(~y_edges)
# 如果distance_metric在distance_metric_list中
elif self.distance_metric in self.distance_metric_list[-2:]:
# 则使用morphology.distance_transform_cdt函数计算棋盘距离
dis = morphology.distance_transform_cdt(~y_edges, metric=self.distance_metric)
# 最后，返回计算得到的表面距离数组
return dis[y_pred_edges]
def clear(self):
# 用于清空内部评估结果
"""Clears the internal evaluation result."""
# 它将_y_pred_edges、_y_edges和_is_update属性的值设置为0，表示没有计算过任何结果
self._y_pred_edges = 0
self._y_edges = 0
self._is_update = False
@rearrange_inputs
def update(self, *inputs):
# 用于更新内部评估结果。它接受三个输入：y_pred（预测的二值图像）、y（实际的二值图像）和label_idx（整数或浮点数）。
# update方法的主要目的是更新_y_pred_edges、_y_edges和_is_update属性，以便在计算过程中使用
"""
Updates the internal evaluation result 'y_pred', 'y' and 'label_idx'.
@ -123,36 +160,52 @@ class RootMeanSquareDistance(Metric):
ValueError: If the value of label_idx is not in y_pred or y.
ValueError: If y_pred and y have different shapes.
"""
if len(inputs) != 3:
# 首先，代码检查输入的数量是否为3
if len(inputs)!= 3:
# 如果不是，则抛出一个ValueError异常
raise ValueError("For 'RootMeanSquareDistance.update', it needs 3 inputs"
"(predicted value, true value, label index), but got {}.".format(len(inputs)))
# 然后，将y_pred和y转换为Tensor、列表或numpy.ndarray类型
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 将输入列表中的第三个元素（即label_idx）转换为整数类型
label_idx = inputs[2]
# 接下来，代码检查label_idx的数据类型是否为整数或浮点数
if not isinstance(label_idx, (int, float)):
# 如果不是，则抛出一个TypeError异常
raise TypeError("For 'RootMeanSquareDistance.update', the label index (input[2]) must be int or float, "
"but got label index type: {}.".format(type(label_idx)))
# 检查label_idx是否在y_pred和y中
if label_idx not in y_pred and label_idx not in y:
# 如果不在其中，则抛出一个ValueError异常
raise ValueError("For 'RootMeanSquareDistance.update', the label index (input[2]) "
"should be in predicted value (input[0]) or true value (input[1]), "
"but {} is not.".format(label_idx))
# 最后，代码检查y_pred和y是否具有相同的形状
if y_pred.size == 0 or y_pred.shape != y.shape:
# 如果不是，则抛出一个ValueError异常
raise ValueError("For 'RootMeanSquareDistance.update', the size of predicted value (input[0]) "
"and true value (input[1]) should be greater than 0, in addition to that, "
"predicted value and true value should have the same shape, "
"but got predicted value size: {}, shape: {}, true value size: {}, shape: {}. "
.format(y_pred.size, y_pred.shape, y.size, y.shape))
# 首先检查y_pred和y的数据类型是否为布尔类型
if y_pred.dtype != bool:
# 如果不是，则将它们转换为布尔类型
y_pred = y_pred == label_idx
if y.dtype != bool:
# 如果不是，则将它们转换为布尔类型
y = y == label_idx
# 接下来，使用morphology.binary_erosion函数对y_pred_edges和y_edges进行二值腐蚀
self._y_pred_edges = morphology.binary_erosion(y_pred) ^ y_pred
self._y_edges = morphology.binary_erosion(y) ^ y
# 最后，将_y_pred_edges、_y_edges和_is_update属性设置为True，表示已经更新了内部评估结果
self._is_update = True
def eval(self):
# 用于计算Root Mean Square Distance（RMSD），需在update函数后调用
"""
Calculate Root Mean Square Distance.
@ -163,24 +216,38 @@ class RootMeanSquareDistance(Metric):
RuntimeError: If the update method is not called first, an error will be reported.
"""
# 首先，它检查_is_update属性是否为True
if self._is_update is False:
# 如果是，则表示已经更新了内部评估结果，如果不是，则抛出一个RuntimeError异常，要求先调用update方法
raise RuntimeError("Please call the 'update' method before calling 'eval' method.")
# 接下来，代码计算_y_pred_edges和_y_edges之间的表面距离，并将其存储在residual_mean_square_distance中
residual_mean_square_distance = self._get_surface_distance(self._y_pred_edges, self._y_edges)
# 如果residual_mean_square_distance的形状为(0,)
if residual_mean_square_distance.shape == (0,):
# 则表示没有计算过任何结果，返回np.inf
return np.inf
# 然后，代码计算RMSD，即均方根误差
rms_surface_distance = (residual_mean_square_distance**2).mean()
# 如果symmetric属性为False
if not self.symmetric:
# 则直接返回RMSD
return rms_surface_distance
# 否则，计算与_y_edges相反的_y_pred_edges之间的表面距离，并将其存储在contrary_residual_mean_square_distance中
contrary_residual_mean_square_distance = self._get_surface_distance(self._y_edges, self._y_pred_edges)
# 如果contrary_residual_mean_square_distance的形状为(0,)
if contrary_residual_mean_square_distance.shape == (0,):
# 则表示没有计算过任何结果，返回np.inf
return np.inf
# 最后，代码计算相反的RMSD，即contrary_residual_mean_square_distance的平方根
contrary_rms_surface_distance = (contrary_residual_mean_square_distance**2).mean()
# 并将其与原始的RMSD进行平均
rms_distance = np.sqrt(np.mean((rms_surface_distance, contrary_rms_surface_distance)))
# 返回计算得到的RMSD
return rms_distance

40
mindspore/python/mindspore/nn/metrics/topk.py
View File

@ -12,12 +12,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# Topk类，继承自Metric类。Topk类主要用于计算top-k准确率
"""Topk."""
# 导入所需的库和模块，包括numpy库和Metric类
import numpy as np
# rearrange_inputs函数，用于处理输入数据，使其符合Topk类的计算要求。
from .metric import Metric, rearrange_inputs, _check_onehot_data
# _check_onehot_data函数，用于检查输入数据是否为one-hot编码
class TopKCategoricalAccuracy(Metric):
# TopKCategoricalAccuracy类用于计算top-k分类准确率
# 参数：k的整数变量，用于指定计算top-k分类准确率
"""
Calculates the top-k categorical accuracy.
@ -46,24 +52,35 @@ class TopKCategoricalAccuracy(Metric):
>>> print(output)
0.6666666666666666
"""
# 定义Topk类的初始化方法，用于初始化类的属性
def __init__(self, k):
# 调用父类的构造函数
super(TopKCategoricalAccuracy, self).__init__()
# 首先检查k的类型是否为整数
if not isinstance(k, int):
# 如果不是，则抛出一个TypeError异常
raise TypeError("For 'TopKCategoricalAccuracy', the type of "
"the argument 'k' should be int, but got 'k' type: {}.".format(type(k)))
# 然后检查k的值是否大于等于1
if k < 1:
# 如果不是，则抛出一个ValueError异常
raise ValueError("For 'TopKCategoricalAccuracy', "
"the argument 'k' must be at least 1, but got 'k' value: {}.".format(k))
# 最后，将k的值赋给self.k
self.k = k
# 并调用self.clear()方法清除之前的记录
self.clear()
def clear(self):
# 用于清空内部评估结果
"""Clear the internal evaluation result."""
# 将self._correct_num和self._samples_num的值重置为0
self._correct_num = 0
self._samples_num = 0
@rearrange_inputs
def update(self, *inputs):
# 定义一个名为update的方法，用于更新模型的预测结果和真实结果
"""
Updates the internal evaluation result `y_pred` and `y`.
@ -78,36 +95,54 @@ class TopKCategoricalAccuracy(Metric):
The method `update` must receive input of the form :math:`(y_{pred}, y)`. If some samples have
the same accuracy, the first sample will be chosen.
"""
# 首先，检查输入数据的数量是否为2
if len(inputs) != 2:
# 如果不是，则抛出一个ValueError异常
raise ValueError("For 'TopKCategoricalAccuracy.update', "
"it needs 2 inputs (predicted value, true value), "
"but got 'inputs' size: {}.".format(len(inputs)))
# 然后，将输入数据转换为适当的数据类型
y_pred = self._convert_data(inputs[0])
y = self._convert_data(inputs[1])
# 并检查y预测值和y维度是否相同，输入数据是否为one-hot编码
if y_pred.ndim == y.ndim and _check_onehot_data(y):
# 如果满足这些条件，则将输入数据转换为argmax索引
y = y.argmax(axis=1)
# 首先，使用np.argsort函数对y_pred进行降序排序，并获取前self.k个元素的索引
indices = np.argsort(-y_pred, axis=1)[:, :self.k]
# 然后，将yreshape为(-1, 1)，并使用repeat函数将每个元素重复self.k次
repeated_y = y.reshape(-1, 1).repeat(self.k, axis=1)
# 最后，使用np.equal函数比较indices和repeated_y，并计算每个元素是否相等。将所有相等元素的计数累加，得到每个样本的top-k分类准确率
correct = np.equal(indices, repeated_y).sum(axis=1)
# 将correct的值累加到self._correct_num中
self._correct_num += correct.sum()
# 将repeated_y.shape[0]累加到self._samples_num中
self._samples_num += repeated_y.shape[0]
def eval(self):
# 定义一个名为eval的方法，用于计算top-k准确率，该函数需要在update函数后调用
"""
Computes the top-k categorical accuracy.
Returns:
numpy.float64, computed result.
"""
# 首先，检查self._samples_num是否为0
if self._samples_num == 0:
# 如果是，则抛出一个RuntimeError异常
raise RuntimeError("The 'TopKCategoricalAccuracy' "
"can not be calculated, because the number of samples is 0, "
"please check whether your inputs (predicted value, true value) are empty, "
"or has called update method before calling eval method.")
# 如果self._samples_num不为0
return self._correct_num / self._samples_num
# 则计算self._correct_num除以self._samples_num，得到top-k分类准确率，并返回该结果
class Top1CategoricalAccuracy(TopKCategoricalAccuracy):
# 继承自TopKCategoricalAccuracy类。这个类用于计算top-1分类准确率。
# Top1CategoricalAccuracy类重写了eval方法，使其返回top-1准确率，而不是top-k准确率。
# 这可以通过将self._correct_num除以self._samples_num来实现
"""
Calculates the top-1 categorical accuracy. This class is a specialized class for TopKCategoricalAccuracy.
Refer to :class:`TopKCategoricalAccuracy` for more details.
@ -129,11 +164,15 @@ class Top1CategoricalAccuracy(TopKCategoricalAccuracy):
>>> print(output)
0.0
"""
# 调用父类的__init__方法，并将k设置为1。这样，我们创建了一个只计算top-1准确率的TopKCategoricalAccuracy对象
def __init__(self):
super(Top1CategoricalAccuracy, self).__init__(1)
class Top5CategoricalAccuracy(TopKCategoricalAccuracy):
# 继承自TopKCategoricalAccuracy类。这个类用于计算top-5分类准确率
# Top5CategoricalAccuracy类重写了eval方法，使其返回top-5准确率，而不是top-k准确率。
# 这可以通过将self._correct_num除以self._samples_num来实现
"""
Calculates the top-5 categorical accuracy. This class is a specialized class for TopKCategoricalAccuracy.
Refer to :class:`TopKCategoricalAccuracy` for more details.
@ -155,5 +194,6 @@ class Top5CategoricalAccuracy(TopKCategoricalAccuracy):
>>> print(output)
1.0
"""
# 调用父类的__init__方法，并将k设置为5。这样，我们创建了一个只计算top-5准确率的TopKCategoricalAccuracy对象
def __init__(self):
super(Top5CategoricalAccuracy, self).__init__(5)

7
mindspore/python/mindspore/nn/optim/__init__.py
View File

@ -12,14 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 定义了名为Optimizer的Python模块
# 提供了许多常用的优化器，如SGD（随机梯度下降）、ADAM（适应性矩估计）、Momentum（动量）、LARS（线性权重归一化）等。这些优化器用于训练过程中计算和更新梯度
"""
Optimizer.
Provide common optimizers for training, such as SGD, ADAM, Momentum.
The optimizer is used to calculate and update the gradients.
"""
# 从optimizer模块中导入Optimizer基类
from .optimizer import Optimizer
# 从momentum模块中导入Momentum类
from .momentum import Momentum
# 从adam模块中导入Adam、AdamWeightDecay和AdamOffload类
from .adam import Adam, AdamWeightDecay, AdamOffload
from .lamb import Lamb
from .sgd import SGD
@ -33,8 +38,10 @@ from .lazyadam import LazyAdam
from .ada_grad import Adagrad
from .thor import thor
from .adafactor import AdaFactor
# 从adasum模块中导入AdaSumByDeltaWeightWrapCell和AdaSumByGradWrapCell类
from .adasum import AdaSumByDeltaWeightWrapCell, AdaSumByGradWrapCell
# 在导入optimizer模块时，就可以直接导入这个列表中的所有类，而无需单独导入每个类。这有助于减少代码的重复和提高代码的可读性
__all__ = ['Optimizer', 'Momentum', 'LARS', 'Adam', 'AdamWeightDecay', 'LazyAdam', 'AdamOffload',
'Lamb', 'SGD', 'ASGD', 'Rprop', 'FTRL', 'RMSProp', 'ProximalAdagrad', 'Adagrad', 'thor', 'AdaFactor',
'AdaSumByDeltaWeightWrapCell', 'AdaSumByGradWrapCell']

158
mindspore/python/mindspore/nn/optim/ada_grad.py
View File

@ -1,4 +1,4 @@
# Copyright 2020-2021 Huawei Technologies Co., Ltd
# Copyright 2020 Huawei Technologies Co., Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -13,125 +13,106 @@
# limitations under the License.
# ============================================================================
"""ADA_GRAD"""
# Adagrad算法实现的优化器
# 导入ops算子模块
from mindspore.ops import functional as F, composite as C, operations as P
# 导入检查模块
from mindspore._checkparam import Validator as validator
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
# 导入optimizer模块
from.optimizer import Optimizer
_ada_grad_opt = C.MultitypeFuncGraph("ada_grad_opt")
# 为_ada_grad_opt添加一个名为_tensor_run_opt的函数，参数为opt，learning_rate，weight，accum，gradient
@_ada_grad_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor")
def _tensor_run_opt(opt, learning_rate, weight, accum, gradient):
"""Apply ada_grad optimizer to the weight parameter."""
success = True
# 检查weight、accum、learning_rate、gradient是否有效
success = F.depend(success, opt(weight, accum, learning_rate, gradient))
# 返回成功状态
return success
def _check_param_value(accum, update_slots, prim_name=None):
"""Check inputs param."""
# 检查accum的类型是否为float
validator.check_value_type("accum", accum, [float], prim_name)
# 检查update_slots的类型是否为bool
validator.check_value_type("update_slots", update_slots, [bool], prim_name)
# 检查accum的值是否大于0
validator.check_non_negative_float(accum, "accum", prim_name)
class Adagrad(Optimizer):
r"""
Implements the Adagrad algorithm.
Implements the Adagrad algorithm with ApplyAdagrad Operator.
Adagrad is an online Learning and Stochastic Optimization.
Refer to paper `Efficient Learning using Forward-Backward Splitting
<https://proceedings.neurips.cc/paper/2009/file/621bf66ddb7c962aa0d22ac97d69b793-Paper.pdf>`_.
Adagrad can adaptively assign different learning rates to each parameter in response to the uneven number of
samples for different parameters.
The updating Pseudo codes are as follows,
The updating formulas are as follows,
.. math::
\begin{aligned} \\
&\newline
&\hline \\
&\textbf{Parameters}: \text{learning rate } \gamma, \: \text{ params } w_0, \:
\: \text{ weight decay } \lambda, \\
&\hspace{12mm} \text{ initial accumulator value } state\_sum\\
&\textbf{Init}: state\_sum_0 \leftarrow 0 \\[-1.ex]
&\newline
&\hline \\
&\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\
&\hspace{5mm}g_t \leftarrow \nabla_{w} f_t (w_{t-1}) \\
&\hspace{5mm} \textbf{if} \: \lambda \neq 0 \\
&\hspace{10mm} g_t \leftarrow g_t + \lambda w_{t-1} \\
&\hspace{5mm}state\_sum_t \leftarrow state\_sum_{t-1} + g^2_t \\
&\hspace{5mm}w_t \leftarrow w_{t-1}- \gamma*\frac{g_t}{\sqrt{state\_sum_t} + \epsilon} \\
&\newline
&\hline \\
&\bf{return} \: w_t \\[-1.ex]
&\newline
&\hline \\
\end{aligned}
\begin{array}{ll} \\
h_{t+1} = h_{t} + g\\
w_{t+1} = w_{t} - lr*\frac{1}{\sqrt{h_{t+1}}}*g
\end{array}
:math:`h` represents the cumulative sum of gradient squared, :math:`g` represents `gradients`.
:math:`lr` represents `learning_rate`, :math:`w` represents `params`.
Note:
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
When separating parameter groups, the weight decay in each group will be applied on the parameters if the
weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied
on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive.
When separating parameter groups, if you want to centralize the gradient, set grad_centralization to True,
but the gradient centralization can only be applied to the parameters of the convolution layer.
If the parameters of the non convolution layer are set to True, an error will be reported.
To improve parameter groups performance, the customized order of parameters can be supported.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", "grad_centralization" and
"order_params" are the keys can be parsed.
params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated,
the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params",
"lr", "weight_decay" and "order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- params: Required. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
If not, the `learning_rate` in the API will be used.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
will be used. If not, the `weight_decay` in the API will be used.
- grad_centralization: Optional. Must be Boolean. If "grad_centralization" is in the keys, the set value
will be used. If not, the `grad_centralization` is False by default. This configuration only works on the
convolution layer.
- order_params: Optional. If "order_params" in the keys, the value must be the order of parameters and
the order will be followed in optimizer. There are no other keys in the `dict` and the parameters which
in the value of 'order_params' must be in one of group parameters.
- order_params: Optional. When parameters is grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
- grad_centralization: Optional. The data type of "grad_centralization" is Bool. If "grad_centralization"
is in the keys, the set value will be used. If not, the `grad_centralization` is False by default.
This parameter only works on the convolution layer.
accum (float): The starting value for `h`, must be zero or positive values. Default: 0.1.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 0.001.
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of current step.
update_slots (bool): Whether the `h` will be updated. Default: True.
accum (float): The starting value for accumulators, must be zero or positive values. Default: 0.1.
learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate.
When the learning_rate is an Iterable or a Tensor in a 1D dimension, use dynamic learning rate, then
the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule,
use dynamic learning rate, the i-th learning rate will be calculated during the process of training
according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero
dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be
equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float.
Default: 0.001.
update_slots (bool): If true, update accumulation. Default: True.
loss_scale (float): Value for the loss scale. It must be greater than 0.0. In general, use the default value.
Only when `FixedLossScaleManager` is used for training and the `drop_overflow_update` in
`FixedLossScaleManager` is set to False, then this value needs to be the same as the `loss_scale` in
`FixedLossScaleManager`. Refer to class :class:`mindspore.FixedLossScaleManager` for more details.
Default: 1.0.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
weight_decay (Union[float, int]): Weight decay value to multiply weight, must be zero or positive value.
Default: 0.0.
Inputs:
- **grads** (tuple[Tensor]) - The gradients of `params` in the optimizer, the shape is the same as the `params`
@ -153,8 +134,6 @@ class Adagrad(Optimizer):
``Ascend`` ``CPU`` ``GPU``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.Adagrad(params=net.trainable_params())
@ -176,25 +155,40 @@ class Adagrad(Optimizer):
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
@opt_init_args_register
def __init__(self, params, accum=0.1, learning_rate=0.001,
update_slots=True, loss_scale=1.0, weight_decay=0.0):
# 初始化Adagrad类，参数分别为：参数params，梯度accum，学习率learning_rate，更新槽update_slots，损失缩放因子loss_scale，权重衰减weight_decay
super(Adagrad, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 检查参数值，设置参数accum
_check_param_value(accum, update_slots, self.cls_name)
# 创建参数clone函数，用于拷贝参数
self.accum = self.parameters.clone(prefix="accum", init=accum)
# 创建HyperMap函数，用于计算梯度
self.hyper_map = C.HyperMap()
# 创建ApplyAdagrad函数，用于计算梯度
self.update_slots = update_slots
self.opt = P.ApplyAdagrad(update_slots=update_slots)
def construct(self, grads):
# 获取参数
params = self.parameters
# 获取梯度accum
accum = self.accum
# 调用decay_weight函数，计算梯度
grads = self.decay_weight(grads)
# 调用gradients_centralization函数，计算梯度
grads = self.gradients_centralization(grads)
# 调用scale_grad函数，计算梯度
grads = self.scale_grad(grads)
# 获取学习率
lr = self.get_lr()
# 如果是分组学习率，则调用map_函数
if self.is_group_lr:
success = self.map_reverse(F.partial(_ada_grad_opt, self.opt), lr, params, accum,
grads)
success = self.map_(F.partial(_ada_grad_opt, self.opt), lr, params, accum,
grads)
# 否则调用map_函数
else:
success = self.map_reverse(F.partial(_ada_grad_opt, self.opt, lr), params, accum,
grads)
return success
success = self.map_(F.partial(_ada_grad_opt, self.opt, lr), params, accum,
grads)
# 返回更新成功标志
return success

202
mindspore/python/mindspore/nn/optim/adafactor.py
View File

@ -13,6 +13,7 @@
# limitations under the License.
# ============================================================================
"""adafactor"""
# AdaFactor算法是减少显存占用的Adam优化器的一种变体
from mindspore import context
from mindspore.common import dtype as mstype
from mindspore.log import logging
@ -25,22 +26,29 @@ from mindspore.common.tensor import Tensor
from mindspore._checkparam import Validator as validator
from mindspore._checkparam import Rel
from mindspore.nn.optim.optimizer import opt_init_args_register
from .optimizer import Optimizer
from.optimizer import Optimizer
def _rms(update_tensor):
"""calculate rms"""
# 计算梯度的平方根
return F.sqrt(P.ReduceMean(False)(F.square(update_tensor)))
def _approx_sq_grad(exp_avg_sq_row, exp_avg_sq_col):
"""Approximation of exponential moving average of square of gradient"""
# 计算梯度的平方根的平均值
reduce_mean = P.ReduceMean(keep_dims=True)(exp_avg_sq_row, -1)
# 计算梯度的平均值的平方根
div_val = 1.0 / P.Sqrt()(P.Div()(exp_avg_sq_row, reduce_mean))
# 将平均值的平方根拉平
r_factor = (P.ExpandDims()(div_val, -1))
# 计算梯度的平方根的平均值
exp_avg_sq_col = P.ExpandDims()(exp_avg_sq_col, -2)
# 计算梯度的平均值的平方根
c_factor = 1.0 / P.Sqrt()(exp_avg_sq_col)
# 返回拉平后的梯度
return P.Mul()(r_factor, c_factor)
@ -48,6 +56,7 @@ reduce_mean_keep_alive = P.ReduceMean().add_prim_attr("keep_alive", True)
_adafactor_opt = C.MultitypeFuncGraph("adafactor_opt")
# 定义一个名为_run_opt_with_one_number的函数，用于接收任意类型的参数，并返回一个布尔值
@_adafactor_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool", "Bool", "Bool", "Tensor",
"Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor")
def _run_opt_with_one_number(eps, clip_threshold, beta1, beta2t, weight_decay, scale_parameter,
@ -55,64 +64,103 @@ def _run_opt_with_one_number(eps, clip_threshold, beta1, beta2t, weight_decay, s
grad, param, exp_avg, exp_avg_sq_row, exp_avg_sq_col, exp_avg_sq):
"""Apply ada factor optimizer to the weight parameter using Tensor."""
success = True
# 获取梯度的数据类型
grad_dtype = F.dtype(grad)
# 获取梯度的形状
grad_shape = F.shape(grad)
# 将参数转换为float32类型
if grad_dtype == mstype.float16:
grad = F.cast(grad, mstype.float32)
p_data_fp32 = param
# 如果参数的数据类型为float16，则将参数转换为float32类型
if F.dtype(p_data_fp32) == mstype.float16:
p_data_fp32 = F.cast(p_data_fp32, mstype.float32)
# 判断是否需要梯度矩阵乘法
factored = len(grad_shape) >= 2
# 如果需要，则求平方梯度和，并将结果赋值给update
if scale_parameter:
# 求平方梯度和
rms = _rms(p_data_fp32)
# 将eps[1]和rms转换为float32类型
param_scale = P.Maximum()(eps[1], rms)
# 将learning_rate乘以param_scale和rms转换为float32类型
learning_rate_update = learning_rate * param_scale * F.ones_like(rms)
else:
# 否则，learning_rate乘以1
learning_rate_update = learning_rate
# 定义梯度更新函数
update = (grad ** 2) + eps[0]
if factored:
# 将exp_avg_sq_row_update（行矩阵）转换为grad_dtype
exp_avg_sq_row_update = F.cast(exp_avg_sq_row, grad_dtype)
# 将exp_avg_sq_row_update乘以beta2t
exp_avg_sq_row_update = P.Mul()(exp_avg_sq_row_update, beta2t)
# 更新mean
update_mean = reduce_mean_keep_alive(update, -1) * (1.0 - beta2t)
# 将exp_avg_sq_row_update更新
exp_avg_sq_row_update = P.Add()(exp_avg_sq_row_update, update_mean)
# 将exp_avg_sq_row_update赋值给exp_avg_sq_row
exp_avg_sq_row_update = F.assign(exp_avg_sq_row, F.cast(exp_avg_sq_row_update, F.dtype(exp_avg_sq_row)))
# 将exp_avg_sq_col_update（列矩阵）转换为grad_dtype
exp_avg_sq_col_update = F.cast(exp_avg_sq_col, grad_dtype)
# 将exp_avg_sq_col_update乘以beta2t
exp_avg_sq_col_update = P.Mul()(exp_avg_sq_col_update, beta2t)
# 更新mean
update_mean = reduce_mean_keep_alive(update, -2) * (1.0 - beta2t)
# 将exp_avg_sq_col_update更新
exp_avg_sq_col_update = P.Add()(exp_avg_sq_col_update, update_mean)
# 将exp_avg_sq_col_update赋值给exp_avg_sq_col
exp_avg_sq_col_update = F.assign(exp_avg_sq_col, F.cast(exp_avg_sq_col_update, F.dtype(exp_avg_sq_col)))
# 更新update
update = _approx_sq_grad(exp_avg_sq_row_update, exp_avg_sq_col_update)
# 将update乘以grad
update = P.Mul()(update, grad)
else:
# 将exp_avg_sq_update（一般矩阵）转换为grad_dtype
exp_avg_sq_update = F.cast(exp_avg_sq, grad_dtype)
# 更新update
update = update * (1.0 - beta2t)
# 将exp_avg_sq_update更新
exp_avg_sq_update = P.Add()(P.Mul()(exp_avg_sq_update, beta2t), update)
# 将exp_avg_sq_update赋值给exp_avg_sq
exp_avg_sq_update = F.assign(exp_avg_sq, F.cast(exp_avg_sq_update, F.dtype(exp_avg_sq)))
# 将exp_avg_sq_update除以1/Sqrt()
exp_avg_sq_update = 1.0 / P.Sqrt()(exp_avg_sq_update)
# 将exp_avg_sq_update乘以grad
update = P.Mul()(exp_avg_sq_update, grad)
# 计算更新的RMS阈值
update_rms_thres = _rms(update) / clip_threshold
# 计算更新的折叠因子
update_coff = P.Maximum()(update_rms_thres, P.OnesLike()(update_rms_thres))
# 更新参数
update = P.Mul()(P.Div()(update, update_coff), learning_rate_update)
# 如果使用第一个梯度的平均更新
if use_first_moment:
# 计算更新的平均更新
exp_avg_update = exp_avg
# 如果压缩，将更新的数据类型转换为更新的数据类型
if compression:
exp_avg_update = F.cast(exp_avg, grad_dtype)
# 计算更新的平均更新
exp_avg_update = P.Add()(P.Mul()(exp_avg_update, beta1), update * (1 - beta1))
# 将更新的平均更新赋值给更新的参数
update = F.assign(exp_avg, F.cast(exp_avg_update, F.dtype(exp_avg)))
# 如果使用权重衰减
if weight_decay_flag:
# 计算更新的参数
p_data_fp32_coff = p_data_fp32 * -weight_decay * learning_rate_update
p_data_fp32 = P.Add()(p_data_fp32, p_data_fp32_coff)
p_data_fp32 = P.Sub()(p_data_fp32, update)
p_data_fp32 = P.Sub()(p_data_fp32, update)
# 返回更新的参数
return F.depend(success, P.Assign()(param, F.cast(p_data_fp32, F.dtype(param))))
@ -120,6 +168,23 @@ def _run_opt_with_one_number(eps, clip_threshold, beta1, beta2t, weight_decay, s
"Tensor", "Tensor", "Tensor", "Tensor")
def _run_fused_ada_factor(fused_ada_factor, eps, clip_threshold, beta1, beta2t, weight_decay, learning_rate,
grad, param, exp_avg, exp_avg_sq_row, exp_avg_sq_col, exp_avg_sq):
'''
计算fused_ada_factor梯度
:param fused_ada_factor: fused_ada_factor函数
:param eps: epsilon
:param clip_threshold: 当前梯度的阈值
:param beta1: beta1
:param beta2t: beta2t
:param weight_decay: weight_decay
:param learning_rate: 学习率
:param grad: 梯度
:param param: 参数
:param exp_avg: 平均值
:param exp_avg_sq_row: 平均值的列矩阵
:param exp_avg_sq_col: 平均值的行矩阵
:param exp_avg_sq: 平均值的平方
:return: 梯度
'''
success = True
ret = fused_ada_factor(eps, clip_threshold, beta1, beta2t, weight_decay, learning_rate,
grad, param, exp_avg, exp_avg_sq_row, exp_avg_sq_col, exp_avg_sq)
@ -127,19 +192,24 @@ def _run_fused_ada_factor(fused_ada_factor, eps, clip_threshold, beta1, beta2t,
def trans_to_tensor(param, is_tuple=False, fp32=True):
# 将参数转化为张量
"""
Transform params to tensor.
"""
if param is None or isinstance(param, bool):
# 如果参数为None或者参数为布尔值，则返回参数
return param
# 如果参数为元组，则将元组转换为Tensor
data_type = mstype.float32 if fp32 else mstype.float16
if is_tuple:
new_param = [Tensor(ele, data_type) for ele in param]
return tuple(new_param)
# 否则，将参数转换为Tensor
return Tensor(param, data_type)
class AdaFactor(Optimizer):
# AdaFactor算法是减少显存占用的Adam优化器的一种变体
r"""
Updates gradients by the Adaptive Learning Rates with Sublinear Memory Cost (Adafactor) algorithm.
@ -266,56 +336,95 @@ class AdaFactor(Optimizer):
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
# 支持并行模式下的优化器
_support_parallel_optimizer = True
@opt_init_args_register
def __init__(self,
params,
learning_rate=None,
eps=(1e-30, 1e-3),
clip_threshold=1.0,
decay_rate=0.8,
beta1=0.9,
weight_decay=0.0,
scale_parameter=True,
relative_step=True,
warmup_init=False,
compression=False,
loss_scale=1.0):
params,
learning_rate=None,
eps=(1e-30, 1e-3),
clip_threshold=1.0,
decay_rate=0.8,
beta1=0.9,
weight_decay=0.0,
scale_parameter=True,
relative_step=True,
warmup_init=False,
compression=False,
loss_scale=1.0):
'''
初始化模型参数
:param params: 参数列表
:param learning_rate: 学习率
:param eps: 极小值
:param clip_threshold: 允许的最大梯度
:param decay_rate: 动量因子
:param beta1: Beta1
:param weight_decay: 权重衰减
:param scale_parameter: 标量参数
:param relative_step: 是否使用相对步长
:param warmup_init: 是否使用热身初始化
:param compression: 是否使用压缩
:param loss_scale: 使用损失缩放
:return:
'''
if learning_rate is not None and relative_step:
# 如果learning_rate和relative_step都存在，则抛出异常
raise ValueError("Cannot combine manual lr and relative_step options", learning_rate)
if warmup_init and not relative_step:
# 如果warmup_init存在，但不是relative_step，则抛出异常
raise ValueError("warmup_init requires relative_step=True")
# 如果learning_rate为None，且不是relative_step，则抛出异常
if learning_rate is None and not relative_step:
raise ValueError("Cannot learning_rate is None and relative_step=False")
# 如果learning_rate为None，则learning_rate设置为0.0
if learning_rate is None:
learning_rate = 0.0
if beta1 is None:
# 如果beta1为空，则将beta1设置为0.0
beta1 = 0.0
# 如果learning_rate不是数字，则将scale_lr设置为False
self.scale_lr = True
if not isinstance(learning_rate, (float, int)) and learning_rate is not None:
# 如果learning_rate不是数字，则将scale_lr设置为False
self.scale_lr = False
# 如果learning_rate支持update learning rate，则报错
if relative_step or scale_parameter:
logging.warning("When learning_rate is learning scheduler, it not support update learning rate!")
# 调用父类构造函数
super(AdaFactor, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 检查eps的类型
validator.check_value_type("eps", eps, [list, tuple], self.cls_name)
if len(eps) != 2:
# 检查eps的长度
if len(eps)!= 2:
raise ValueError("eps must have 2 value: (eps1, eps2).")
# 检查eps1和eps2的类型
for i, ele in enumerate(eps):
validator.check_value_type("eps{}".format(i), ele, [float], self.cls_name)
validator.check_non_negative_float(ele, "eps{}".format(i), self.cls_name)
# 检查clip_threshold的类型
validator.check_value_type("clip_threshold", clip_threshold, [float], self.cls_name)
# 检查clip_threshold的范围
validator.check_non_negative_float(clip_threshold, "clip_threshold", self.cls_name)
# 检查decay_rate的类型
validator.check_value_type("decay_rate", decay_rate, [float], self.cls_name)
# 检查decay_rate的范围
validator.check_float_range(decay_rate, 0, 1, Rel.INC_NEITHER, "decay_rate", self.cls_name)
# 检查weight_decay的类型
validator.check_float_range(weight_decay, 0, 1, Rel.INC_LEFT, "weight_decay", self.cls_name)
# 检查scale_parameter的类型
validator.check_value_type("scale_parameter", scale_parameter, [bool], self.cls_name)
# 检查relative_step的类型
validator.check_value_type("relative_step", relative_step, [bool], self.cls_name)
# 检查compression的类型
validator.check_value_type("compression", compression, [bool], self.cls_name)
# 检查beta1的类型
validator.check_value_type("beta1", beta1, [int, float], self.cls_name)
# 检查beta1的范围
validator.check_non_negative_float(float(beta1), "beta1", self.cls_name)
# 定义各个属性
self.eps = trans_to_tensor(eps)
self.clip_threshold = trans_to_tensor(clip_threshold)
self.decay_rate = trans_to_tensor(-decay_rate)
@ -323,68 +432,95 @@ class AdaFactor(Optimizer):
self.weight_decay = trans_to_tensor(weight_decay)
self.weight_decay_flag = bool(weight_decay)
# 将参数转换为Tensor类型
self.step = Parameter(Tensor(0, dtype=mstype.float32), name="train_step")
# 将scale_parameter设置为False
self.scale_parameter = scale_parameter
# 将relative_step设置为False
self.relative_step = relative_step
# 将warmup_init设置为False
self.warmup_init = warmup_init
# 将compression设置为False
self.compression = compression
# 如果scale_lr为False，将scale_parameter设置为False
if not self.scale_lr:
self.scale_parameter = False
# 将init_ada_factor_state设置为beta1
self.init_ada_factor_state(beta1)
# 将step设置为0
self.step = Parameter(initializer(0, [1], mstype.float32), name='afactor_step')
# 将fused_ada_factor设置为FusedAdaFactor
self.fused_ada_factor = P.FusedAdaFactor(enable_scale_parameter=self.scale_parameter,
enable_first_moment=self.use_first_moment,
enable_weight_decay=self.weight_decay_flag)
# 如果当前设备目标为CPU，将use_fused_ada_factor设置为True
if context.get_context("device_target") == "CPU":
self.use_fused_ada_factor = True
else:
self.use_fused_ada_factor = False
# 打印AdaFactor初始化完成
print("AdaFactor init completed", self.learning_rate)
def init_ada_factor_state(self, beta1):
"""init adafactor variables"""
if beta1 > 0:
# 如果beta1大于0，则使用第一个梯度下降
self.use_first_moment = True
# 初始化exp_avg参数为克隆的参数
self.exp_avg = self.parameters.clone(prefix="exp_avg", init='zeros')
else:
# 如果beta1小于等于0，则使用第一个梯度下降
self.use_first_moment = False
# 初始化exp_avg参数为多个参数合成的参数组
self.exp_avg = ParameterTuple([Parameter(Tensor(0.0))] * len(self.parameters))
# 初始化矩阵，列矩阵，行矩阵，用于后续计算
self.exp_avg_sq = []
self.exp_avg_sq_col = []
self.exp_avg_sq_row = []
for param in self.parameters:
# 获取参数的数据类型
param_dtype = param.dtype
# 获取参数的形状
param_shape = param.shape
# 获取参数的名称
param_name = param.name
# 如果参数形状大于1
if len(param_shape) > 1:
# 将指定行矩阵和指定列矩阵添加到self.exp_avg_sq_row和self.exp_avg_sq_col中
self.exp_avg_sq_row.append(Parameter(initializer(0, shape=param_shape[:-1], dtype=param_dtype),
name="exp_avg_sq_row_{}".format(param_name)))
self.exp_avg_sq_col.append(Parameter(initializer(0, shape=param_shape[:-2] + param_shape[-1:],
dtype=param_dtype),
name="exp_avg_sq_col_{}".format(param_name)))
# 将矩阵添加到self.exp_avg_sq中
self.exp_avg_sq.append(Parameter(initializer(0, shape=(1,), dtype=param_dtype),
name="exp_avg_sq_{}".format(param_name)))
# 如果参数形状小于1
else:
# 将形状相反的指定行矩阵和列矩阵添加到self.exp_avg_sq_row和self.exp_avg_sq_col中
self.exp_avg_sq_row.append(Parameter(initializer(0, shape=(1,), dtype=param_dtype),
name="exp_avg_sq_row_{}".format(param_name)))
self.exp_avg_sq_col.append(Parameter(initializer(0, shape=(1,), dtype=param_dtype),
name="exp_avg_sq_col_{}".format(param_name)))
if self.compression:
# 如果compression为True，则将指定参数为mstype16的矩阵添加到self.exp_avg_sq中
self.exp_avg_sq.append(Parameter(initializer(0, shape=param_shape, dtype=mstype.float16),
name="exp_avg_sq_{}".format(param_name)))
else:
# 否则，将指定参数为param_dtype的矩阵添加到self.exp_avg_sq中
self.exp_avg_sq.append(Parameter(initializer(0, shape=param_shape, dtype=param_dtype),
name="exp_avg_sq_{}".format(param_name)))
# 整理各类矩阵
self.exp_avg_sq_row = ParameterTuple(self.exp_avg_sq_row)
self.exp_avg_sq_col = ParameterTuple(self.exp_avg_sq_col)
self.exp_avg_sq = ParameterTuple(self.exp_avg_sq)
@property
def supports_memory_efficient_fp16(self):
# 对fp16型提供内存便利
"""
Support memory efficient for fp16
"""
@ -392,34 +528,46 @@ class AdaFactor(Optimizer):
@property
def supports_flat_params(self):
# 支持扁平化参数
"""
Support flatten params
"""
return False
def construct(self, gradients):
'''
构建优化器
:param gradients: 梯度
:return:
'''
lr = self.get_lr()
# 计算步长
step = F.assign_add(self.step, 1)
# 如果scale_lr为True，且relative_step为True，则lr = min(min_step, 1/sqrt(step))
if self.scale_lr and self.relative_step:
if self.warmup_init:
# 如果warmup_init为True，则min_step为1e-6 * step
min_step = 1e-6 * step
else:
# 否则，min_step为1e-2
min_step = 1e-2
lr = P.Minimum()(min_step, 1.0 / P.Sqrt()(step * 1.0))
# beta2t = 1.0 - pow(step, decay_rate)
beta2t = 1.0 - P.Pow()(step, self.decay_rate)
# 如果use_fused_ada_factor为True，则success = _adafactor_opt(fused_ada_factor, eps, clip_threshold, beta1, beta2t, weight_decay, lr)
if self.use_fused_ada_factor:
success = self.hyper_map(F.partial(_adafactor_opt, self.fused_ada_factor, self.eps, self.clip_threshold,
self.beta1, beta2t, self.weight_decay, lr),
gradients, self.parameters, self.exp_avg, self.exp_avg_sq_row,
self.exp_avg_sq_col, self.exp_avg_sq)
self.beta1, beta2t, self.weight_decay, lr),
gradients, self.parameters, self.exp_avg, self.exp_avg_sq_row,
self.exp_avg_sq_col, self.exp_avg_sq)
# 否则success = _adafactor_opt(eps, clip_threshold, beta1, beta2t, weight_decay, scale_parameter, compression, use_first_moment, weight_decay_flag, lr)
else:
success = self.hyper_map(F.partial(_adafactor_opt, self.eps, self.clip_threshold, self.beta1, beta2t,
self.weight_decay, self.scale_parameter, self.compression,
self.use_first_moment, self.weight_decay_flag, lr),
gradients, self.parameters, self.exp_avg, self.exp_avg_sq_row,
self.exp_avg_sq_col, self.exp_avg_sq)
self.weight_decay, self.scale_parameter, self.compression,
self.use_first_moment, self.weight_decay_flag, lr),
gradients, self.parameters, self.exp_avg, self.exp_avg_sq_row,
self.exp_avg_sq_col, self.exp_avg_sq)
return success
@Optimizer.target.setter
@ -428,7 +576,9 @@ class AdaFactor(Optimizer):
If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation.
"""
# 设置基target
self._set_base_target(value)
# 如果输入值为CPU，则更新host参数，将use_fused_ada_factor设置为True
if value == 'CPU':
self.fused_ada_factor.add_prim_attr("primitive_target", "CPU")
self.use_fused_ada_factor = True

598
mindspore/python/mindspore/nn/optim/adam.py
View File

@ -1,4 +1,4 @@
# Copyright 2020-2021 Huawei Technologies Co., Ltd
# Copyright 2020 Huawei Technologies Co., Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -13,6 +13,8 @@
# limitations under the License.
# ============================================================================
"""adam"""
# 优化器 Adaptive Moment Estimation (Adam)算法的实现。
import numpy as np
from mindspore.common import dtype as mstype
@ -24,17 +26,21 @@ from mindspore.common.parameter import Parameter
from mindspore.common.tensor import Tensor
from mindspore._checkparam import Validator as validator
from mindspore._checkparam import Rel
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
from.optimizer import Optimizer
# 定义adam求解器
# 定义求解器的名称
_adam_opt = C.MultitypeFuncGraph("adam_opt")
# 定义求解器的操作
_scaler_one = Tensor(1, mstype.int32)
_scaler_ten = Tensor(10, mstype.float32)
@_adam_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor",
# 定义求解器的注册值
@_adam_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Number", "Tensor", "Tensor", "Tensor",
"Tensor", "Bool", "Bool")
def _update_run_op(beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flag, optim_filter):
# 用于更新参数
"""
Update parameters.
@ -43,7 +49,7 @@ def _update_run_op(beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, d
beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0).
eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0.
lr (Tensor): Learning rate.
weight_decay (numbers.Number): Weight decay. Should be equal to or greater than 0.
weight_decay (Number): Weight decay. Should be equal to or greater than 0.
param (Tensor): Parameters.
m (Tensor): m value of parameters.
v (Tensor): v value of parameters.
@ -54,96 +60,150 @@ def _update_run_op(beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, d
Returns:
Tensor, the new value of v after updating.
"""
op_cast = P.Cast()
if optim_filter:
# 定义乘法运算
op_mul = P.Mul()
# 定义平方运算
op_square = P.Square()
# 定义平方根运算
op_sqrt = P.Sqrt()
# 定义转换运算
op_cast = P.Cast()
# 定义reshape运算
op_reshape = P.Reshape()
# 定义shape运算
op_shape = P.Shape()
# 将参数转换为float32类型
param_fp32 = op_cast(param, mstype.float32)
# 将模转换为float32类型
m_fp32 = op_cast(m, mstype.float32)
# 将方差转换为float32类型
v_fp32 = op_cast(v, mstype.float32)
# 将梯度转换为float32类型
gradient_fp32 = op_cast(gradient, mstype.float32)
# 将beta1的类型设置为float32
next_m = op_mul(beta1, m_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32)
- beta1, gradient_fp32)
- beta1, gradient_fp32)
# 将beta2的类型设置为float32
next_v = op_mul(beta2, v_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32)
- beta2, op_square(gradient_fp32))
- beta2, op_square(gradient_fp32))
# 将梯度求平均
update = next_m / (eps + op_sqrt(next_v))
# 如果没有指定weight_decay，则设置为0
if decay_flag:
update = op_mul(weight_decay, param_fp32) + update
# 将梯度求平均，并将梯度转换为float32类型
update_with_lr = op_mul(lr, update)
# 将参数更新
next_param = param_fp32 - op_reshape(update_with_lr, op_shape(param_fp32))
# 将参数更新，并将参数转换为float32类型
next_param = F.depend(next_param, F.assign(param, op_cast(next_param, F.dtype(param))))
# 将模更新，并将模转换为float32类型
next_param = F.depend(next_param, F.assign(m, op_cast(next_m, F.dtype(m))))
# 将方差更新，并将方差转换为float32类型
next_param = F.depend(next_param, F.assign(v, op_cast(next_v, F.dtype(v))))
# 返回参数
return op_cast(next_param, F.dtype(param))
return op_cast(gradient, F.dtype(param))
return gradient
# 定义一个名为_adam_opt的函数，接收参数：Function，Function，Function，Bool，Bool，Bool，Tensor，Tensor，Tensor，Tensor，Tensor，RowTensor，Tensor，Tensor，Tensor，Tensor，Bool，Bool
@_adam_opt.register("Function", "Function", "Function", "Function", "Bool", "Bool", "Bool", "Tensor", "Tensor",
"Tensor", "Tensor", "Tensor", "Tensor", "RowTensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool")
def _run_opt_with_sparse(opt, sparse_opt, push, pull, use_locking, use_nesterov, target, beta1_power,
beta2_power, beta1, beta2, eps, lr, gradient, param, m, v, ps_parameter, cache_enable):
# 启动适配稀疏矩阵梯度的adam优化器
"""Apply sparse adam optimizer to the weight parameter when the gradient is sparse."""
# 定义一个布尔值，用来表示是否成功执行opt
success = True
# 获取梯度的索引
indices = gradient.indices
# 获取梯度的值
values = gradient.values
if ps_parameter and not cache_enable:
# 如果ps_parameter为True，且cache_enable为False，则使用P.Shape()函数计算op_shape
op_shape = P.Shape()
# 将op_shape(param), op_shape(m), op_shape(v),
# op_shape(beta1_power), op_shape(beta2_power), op_shape(lr), op_shape(beta1),
# op_shape(beta2), op_shape(eps), op_shape(values), op_shape(indices)计算出来
shapes = (op_shape(param), op_shape(m), op_shape(v),
op_shape(beta1_power), op_shape(beta2_power), op_shape(lr), op_shape(beta1),
op_shape(beta2), op_shape(eps), op_shape(values), op_shape(indices))
op_shape(beta1_power), op_shape(beta2_power), op_shape(lr), op_shape(beta1),
op_shape(beta2), op_shape(eps), op_shape(values), op_shape(indices))
# 将success的值与pull函数的值进行比较，如果比较结果为True，则使用push函数推送
success = F.depend(success, pull(push((beta1_power, beta2_power, lr, beta1, beta2,
eps, values, indices), shapes), param))
eps, values, indices), shapes), param))
# 返回success的值
return success
# 如果指定了target，则使用sparse_opt
if not target:
success = F.depend(success, sparse_opt(param, m, v, beta1_power, beta2_power, lr, beta1, beta2,
eps, values, indices))
else:
# 创建一个乘积运算器
op_mul = P.Mul()
# 创建平方运算器
op_square = P.Square()
# 创建平方根运算器
op_sqrt = P.Sqrt()
# 创建ScatterAdd运算器
scatter_add = P.ScatterAdd(use_locking)
# 更新参数
success = F.depend(success, F.assign(m, op_mul(beta1, m)))
success = F.depend(success, F.assign(v, op_mul(beta2, v)))
# 获取梯度索引
grad_indices = gradient.indices
# 获取梯度值
grad_value = gradient.values
# 更新下一个梯度
next_m = scatter_add(m,
grad_indices,
op_mul(F.tuple_to_array((1.0,)) - beta1, grad_value))
# 更新下一个方差
next_v = scatter_add(v,
grad_indices,
op_mul(F.tuple_to_array((1.0,)) - beta2, op_square(grad_value)))
# 如果使用Nesterov梯度，则更新下一个梯度
if use_nesterov:
# 将下一个梯度乘以beta1
m_temp = next_m * _scaler_ten
# 更新m
F.assign(m, op_mul(beta1, next_m))
# 将梯度累加到m上
div_value = scatter_add(m,
op_mul(grad_indices, _scaler_one),
op_mul(F.tuple_to_array((1.0,)) - beta1, grad_value))
# 更新param_update
param_update = div_value / (op_sqrt(next_v) + eps)
# 更新m
F.assign(m, m_temp / _scaler_ten)
else:
# 更新param_update
param_update = next_m / (op_sqrt(next_v) + eps)
# 更新学习率
lr_t = lr * op_sqrt(1 - beta2_power) / (1 - beta1_power)
# 更新下一个参数
next_param = param - lr_t * param_update
# 更新参数
success = F.depend(success, F.assign(param, next_param))
# 更新梯度
success = F.depend(success, F.assign(m, next_m))
# 更新方差
success = F.depend(success, F.assign(v, next_v))
return success
@ -154,15 +214,22 @@ def _run_opt_with_sparse(opt, sparse_opt, push, pull, use_locking, use_nesterov,
def _run_opt_with_one_number(opt, sparse_opt, push, pull, use_locking, use_nesterov, target,
beta1_power, beta2_power, beta1, beta2, eps, lr, gradient, param,
moment1, moment2, ps_parameter, cache_enable):
# 启动适配矩阵梯度的adam优化器
"""Apply adam optimizer to the weight parameter using Tensor."""
# 定义一个布尔值，用来表示是否成功执行opt
success = True
# 如果ps_parameter为True，且cache_enable为False，则执行pull，否则执行opt
if ps_parameter and not cache_enable:
# 获取op_shape
op_shape = P.Shape()
# 执行pull
success = F.depend(success, pull(push((beta1_power, beta2_power, lr, beta1, beta2, eps, gradient),
(op_shape(param), op_shape(moment1), op_shape(moment2))), param))
else:
# 执行opt
success = F.depend(success, opt(param, moment1, moment2, beta1_power, beta2_power, lr, beta1, beta2,
eps, gradient))
# 返回success
return success
@ -170,144 +237,116 @@ def _run_opt_with_one_number(opt, sparse_opt, push, pull, use_locking, use_neste
"Tensor", "Tensor")
def _run_off_load_opt(opt, beta1_power, beta2_power, beta1, beta2, eps, lr, gradient, param, moment1, moment2):
"""Apply AdamOffload optimizer to the weight parameter using Tensor."""
# 启动适配矩阵梯度的adamoffload优化器
# 初始化一个布尔值，用来表示是否执行完成
success = True
# 将AdamOffload优化器的参数更新到参数中
delat_param = opt(moment1, moment2, beta1_power, beta2_power, lr, beta1, beta2, eps, gradient)
# 将参数和参数之间的差值更新到参数中
success = F.depend(success, F.assign_add(param, delat_param))
# 返回是否执行完成的布尔值
return success
def _check_param_value(beta1, beta2, eps, prim_name):
"""Check the type of inputs."""
"""
检查输入参数
"""
# 检查beta1的类型是否为float
validator.check_value_type("beta1", beta1, [float], prim_name)
# 检查beta2的类型是否为float
validator.check_value_type("beta2", beta2, [float], prim_name)
# 检查eps的类型是否为float
validator.check_value_type("eps", eps, [float], prim_name)
# 检查beta1的取值范围是否在0.0和1.0之间
validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name)
# 检查beta2的取值范围是否在0.0和1.0之间
validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name)
# 检查eps的取值是否为正数
validator.check_positive_float(eps, "eps", prim_name)
class Adam(Optimizer):
# Adam优化器
r"""
Implements the Adaptive Moment Estimation (Adam) algorithm.
Updates gradients by the Adaptive Moment Estimation (Adam) algorithm.
The Adam optimizer can dynamically adjust the learning rate of each parameter using the first-order
moment estimation and the second-order moment estimation of the gradient.
The Adam algorithm is proposed in `Adam: A Method for Stochastic Optimization <https://arxiv.org/abs/1412.6980>`_.
The updating formulas are as follows,
.. math::
\begin{array}{l}
&\newline
&\hline \\
&\textbf{Parameters}: \: 1^{\text {st }}\text {moment vector} \: m , \: 2^{\text {nd}} \:
\text{moment vector} \: v , \\
&\:\text{gradients } g, \: \text{learning rate} \: \gamma, \text
{ exponential decay rates for the moment estimates} \: \beta_{1} \: \beta_{2} , \\
&\:\text {parameter vector} \: w_{0}, \:\text{timestep} \: t , \text{ weight decay } \lambda \\
&\textbf{Init}: m_{0} \leftarrow 0, \: v_{0} \leftarrow 0, \: t \leftarrow 0, \:
\text{init parameter vector} \: w_{0} \\[-1.ex]
&\newline
&\hline \\
&\textbf{while} \: w_{t} \: \text{not converged} \: \textbf{do} \\
&\hspace{5mm}\boldsymbol{g}_{t} \leftarrow \nabla_{w} \boldsymbol{f}_{t}\left(\boldsymbol{w}_{t-1}\right) \\
&\hspace{5mm}\textbf {if } \lambda \neq 0 \\
&\hspace{10mm}\boldsymbol{g}_{t} \leftarrow \boldsymbol{g}_{t}+\lambda \boldsymbol{w}_{t-1} \\
&\hspace{5mm}\boldsymbol{m}_{t} \leftarrow \beta_{1} \boldsymbol{m}_{t-1}+\left(1-\beta_{1}\right)
\boldsymbol{g}_{t} \\
&\hspace{5mm}\boldsymbol{v}_{t} \leftarrow \beta_{2} \boldsymbol{v}_{t-1}+\left(1-\beta_{2}\right)
\boldsymbol{g}_{t}^{2} \\
&\hspace{5mm}\hat{\boldsymbol{m}}_{t} \leftarrow \boldsymbol{m}_{t} /\left(1-\beta_{1}^{t}\right) \\
&\hspace{5mm}\hat{\boldsymbol{v}}_{t} \leftarrow \boldsymbol{v}_{t} /\left(1-\beta_{2}^{t}\right) \\
&\hspace{5mm}\boldsymbol{w}_{t} \leftarrow \boldsymbol{w}_{t-1}-\gamma \hat{\boldsymbol{m}}_{t}
/(\sqrt{\hat{\boldsymbol{v}}_{t}}+\epsilon) \\
&\textbf{end while} \\[-1.ex]
&\newline
&\hline \\[-1.ex]
&\textbf{return} \: \boldsymbol{w}_{t} \\[-1.ex]
&\newline
&\hline \\[-1.ex]
\begin{array}{ll} \\
m = \beta_1 * m + (1 - \beta_1) * g \\
v = \beta_2 * v + (1 - \beta_2) * g * g \\
l = \alpha * \frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t} \\
w = w - l * \frac{m}{\sqrt{v} + \epsilon}
\end{array}
:math:`m` represents the 1st moment vector, :math:`v` represents the 2nd moment vector,
:math:`g` represents `gradients`, :math:`\beta_1, \beta_2` represent `beta1` and `beta2`,
:math:`t` represents the current step while :math:`beta_1^t` and :math:`beta_2^t` represent
`beta1_power` and `beta2_power`, :math:`\gamma` represents `learning_rate`, :math:`w` represents `params`,
:math:`m` represents the 1st moment vector `moment1`, :math:`v` represents the 2nd moment vector `moment2`,
:math:`g` represents `gradients`, :math:`l` represents scaling factor `lr`, :math:`\beta_1, \beta_2` represent
`beta1` and `beta2`, :math:`t` represents updating step while :math:`beta_1^t` and :math:`beta_2^t` represent
`beta1_power` and `beta2_power`, :math:`\alpha` represents `learning_rate`, :math:`w` represents `params`,
:math:`\epsilon` represents `eps`.
Note:
The sparse strategy is applied while the SparseGatherV2 operator is used for forward network. If the sparse
strategy wants to be executed on the host, set the target to the CPU.
The sparse feature is under continuous development.
When separating parameter groups, the weight decay in each group will be applied on the parameters if the
weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied
on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
When separating parameter groups, if you want to centralize the gradient, set grad_centralization to True,
but the gradient centralization can only be applied to the parameters of the convolution layer.
If the parameters of the non convolution layer are set to True, an error will be reported.
To improve parameter groups performance, the customized order of parameters is supported.
The sparse strategy is applied while the SparseGatherV2 operator is used for forward network.
The sparse feature is under continuous development. If the sparse strategy wants to be executed on the host,
set the target to the CPU.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", "grad_centralization" and
"order_params" are the keys can be parsed.
params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated,
the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params",
"lr", "weight_decay" and "order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- params: Required. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
- lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used.
If not, the `learning_rate` in the API will be used.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
- weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay
will be used. If not, the `weight_decay` in the API will be used.
- grad_centralization: Optional. Must be Boolean. If "grad_centralization" is in the keys, the set value
will be used. If not, the `grad_centralization` is False by default. This configuration only works on the
convolution layer.
- order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and
the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters
which in the 'order_params' must be in one of group parameters.
- order_params: Optional. When parameters is grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3.
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of current step.
- grad_centralization: Optional. The data type of "grad_centralization" is Bool. If "grad_centralization"
is in the keys, the set value will be used. If not, the `grad_centralization` is False by default.
This parameter only works on the convolution layer.
learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate.
When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then
the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule,
use dynamic learning rate, the i-th learning rate will be calculated during the process of training
according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero
dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be
equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float.
Default: 1e-3.
beta1 (float): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0).
Default: 0.9.
beta2 (float): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0).
Default: 0.999.
eps (float): Term added to the denominator to improve numerical stability. Should be greater than 0. Default:
1e-8.
use_locking (bool): Whether to enable a lock to protect the updating process of variable tensors.
If true, updates of the `w`, `m`, and `v` tensors will be protected by a lock.
use_locking (bool): Whether to enable a lock to protect variable tensors from being updated.
If true, updates of the var, m, and v tensors will be protected by a lock.
If false, the result is unpredictable. Default: False.
use_nesterov (bool): Whether to use Nesterov Accelerated Gradient (NAG) algorithm to update the gradients.
If true, update the gradients using NAG.
If false, update the gradients without using NAG. Default: False.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0.
loss_scale (float): A floating point value for the loss scale. Should be greater than 0. In general, use the
default value. Only when `FixedLossScaleManager` is used for training and the `drop_overflow_update` in
`FixedLossScaleManager` is set to False, then this value needs to be the same as the `loss_scale` in
@ -331,11 +370,9 @@ class Adam(Optimizer):
ValueError: If `weight_decay` is less than 0.
Supported Platforms:
``Ascend`` ``GPU`` ``CPU``
``Ascend`` ``GPU``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.Adam(params=net.trainable_params())
@ -357,14 +394,17 @@ class Adam(Optimizer):
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
@opt_init_args_register
def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-8, use_locking=False,
use_nesterov=False, weight_decay=0.0, loss_scale=1.0):
# 初始化Adam类
super(Adam, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 检查参数值
_check_param_value(beta1, beta2, eps, self.cls_name)
# 检查参数类型
validator.check_value_type("use_locking", use_locking, [bool], self.cls_name)
validator.check_value_type("use_nesterov", use_nesterov, [bool], self.cls_name)
# 初始化参数
self.beta1 = Tensor(beta1, mstype.float32)
self.beta2 = Tensor(beta2, mstype.float32)
self.beta1_power = Parameter(initializer(1, [1], mstype.float32), name="beta1_power")
@ -375,154 +415,138 @@ class Adam(Optimizer):
self.moment1 = self.parameters.clone(prefix="moment1", init='zeros')
self.moment2 = self.parameters.clone(prefix="moment2", init='zeros')
# 是否为设备
self._is_device = True
# HyperMap算法
self.hyper_map = C.HyperMap()
# Adam算法
self.opt = P.Adam(use_locking, use_nesterov)
# FusedSparseAdam
self.sparse_opt = P.FusedSparseAdam(use_locking, use_nesterov)
# FusedSparseAdam的primitive_target属性
self.sparse_opt.add_prim_attr("primitive_target", "CPU")
# ops算子Pull
self._ps_pull = P.Pull()
# ops算子Push
self._ps_push = P.Push("Adam", [0, 1, 2])
# Push的use_nesterov属性
self._ps_push.add_prim_attr("use_nesterov", use_nesterov)
def construct(self, gradients):
'''
构建Adam优化器
:param gradients: 梯度
:return:
'''
params = self.parameters
moment1 = self.moment1
moment2 = self.moment2
gradients = self.decay_weight(gradients)
# 将梯度按照学习率调整
gradients = self.gradients_centralization(gradients)
# 将梯度归一化
gradients = self.scale_grad(gradients)
# 将梯度转换为稀疏矩阵
gradients = self._grad_sparse_indices_deduplicate(gradients)
# 获取学习率
lr = self.get_lr()
# 计算beta1_power
beta1_power = self.beta1_power * self.beta1
# 更新beta1_power
self.beta1_power = beta1_power
# 计算beta2_power
beta2_power = self.beta2_power * self.beta2
# 更新beta2_power
self.beta2_power = beta2_power
# 如果是分组学习率
if self.is_group_lr:
# 用分组方法将梯度传入adam_opt函数
success = self.map_(F.partial(_adam_opt, self.opt, self.sparse_opt, self._ps_push, self._ps_pull,
self.use_locking, self.use_nesterov, self._is_device,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps),
self.use_locking, self.use_nesterov, self._is_device,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps),
lr, gradients, params, moment1, moment2, self.ps_parameters, self.cache_enable)
# 否则
else:
# 将梯度传入adam_opt函数
success = self.map_(F.partial(_adam_opt, self.opt, self.sparse_opt, self._ps_push, self._ps_pull,
self.use_locking, self.use_nesterov, self._is_device,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps, lr),
self.use_locking, self.use_nesterov, self._is_device,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps, lr),
gradients, params, moment1, moment2, self.ps_parameters, self.cache_enable)
# 返回成功状态
return success
@Optimizer.target.setter
# 设置optimizer基类中的target属性
def target(self, value):
"""
If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation.
"""
self._set_base_target(value)
"""If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation."""
# 如果输入的值不是字符串类型，抛出类型错误
if not isinstance(value, str):
raise TypeError("The value must be str type, but got value type is {}".format(type(value)))
# 如果输入的值不在CPU、Ascend、GPU中，抛出值错误
if value not in ('CPU', 'Ascend', 'GPU'):
raise ValueError("The value must be 'CPU', 'Ascend' or 'GPU', but got value {}".format(value))
# 如果设置的target值为CPU，且输入的值为Ascend或GPU，抛出值错误
if self._target == "CPU" and value in('Ascend', 'GPU'):
raise ValueError("In the CPU environment, target cannot be set to 'GPU' and 'Ascend'.")
# 如果设置的target值为Ascend，且输入的值为GPU，抛出值错误
if self._target == "Ascend" and value == 'GPU':
raise ValueError("In the Ascend environment, target cannot be set to 'GPU'.")
# 设置target值为不是CPU
self._is_device = (value!= 'CPU')
# 设置target值为输入的值
self._target = value
class AdamWeightDecay(Optimizer):
r"""
Implements the Adam algorithm with weight decay.
.. math::
\begin{array}{l}
&\newline
&\hline \\
&\textbf{Parameters}: \: 1^{\text {st }}\text {moment vector} \: m , \: 2^{\text {nd}} \:
\text{moment vector} \: v , \\
&\: gradients \: g, \: \text{learning rate} \: \gamma,
\text {exponential decay rates for the moment estimates} \: \beta_{1} \: \beta_{2} , \\
&\:\text {parameter vector} \: w_{0}, \:\text{timestep} \: t, \: \text{weight decay} \: \lambda \\
&\textbf{Init}: m_{0} \leftarrow 0, \: v_{0} \leftarrow 0, \: t \leftarrow 0, \:
\text{init parameter vector} \: w_{0} \\[-1.ex]
&\newline
&\hline \\
&\textbf{repeat} \\
&\hspace{5mm} t \leftarrow t+1 \\
&\hspace{5mm}\boldsymbol{g}_{t} \leftarrow \nabla f_{t}\left(\boldsymbol{w}_{t-1}\right) \\
&\hspace{5mm}\boldsymbol{m}_{t} \leftarrow \beta_{1} \boldsymbol{m}_{t-1}+\left(1-\beta_{1}\right)
\boldsymbol{g}_{t} \\
&\hspace{5mm}\boldsymbol{v}_{t} \leftarrow \beta_{2} \boldsymbol{v}_{t-1}+\left(1-\beta_{2}\right)
\boldsymbol{g}_{t}^{2} \\
&\hspace{5mm}\hat{\boldsymbol{m}}_{t} \leftarrow \boldsymbol{m}_{t} /\left(1-\beta_{1}^{t}\right) \\
&\hspace{5mm}\hat{\boldsymbol{v}}_{t} \leftarrow \boldsymbol{v}_{t} /\left(1-\beta_{2}^{t}\right) \\
&\hspace{5mm}\boldsymbol{w}_{t} \leftarrow \boldsymbol{w}_{t-1}-\left(\gamma \hat{\boldsymbol{m}}_{t}
/\left(\sqrt{\hat{\boldsymbol{v}}_{t}}+\epsilon\right)+\lambda \boldsymbol{w}_{t-1}\right) \\
&\textbf{until}\text { stopping criterion is met } \\[-1.ex]
&\newline
&\hline \\[-1.ex]
&\textbf{return} \: \boldsymbol{w}_{t} \\[-1.ex]
&\newline
&\hline \\[-1.ex]
\end{array}
:math:`m` represents the 1st moment vector `moment1`, :math:`v` represents the 2nd moment vector `moment2`,
:math:`g` represents `gradients`, :math:`\gamma` represents `learning_rate`,
:math:`\beta_1, \beta_2` represent `beta1` and `beta2`, :math:`t` represents the current step,
:math:`w` represents `params`, :math:`\gamma` represents `weight_decay`.
# Adam优化器梯度衰减
"""
Implements the Adam algorithm to fix the weight decay.
Note:
There is usually no connection between a optimizer and mixed precision. But when `FixedLossScaleManager` is used
and `drop_overflow_update` in `FixedLossScaleManager` is set to False, optimizer needs to set the 'loss_scale'.
As this optimizer has no argument of `loss_scale`, so `loss_scale` needs to be processed by other means, refer
document `LossScale <https://www.mindspore.cn/docs/programming_guide/zh-CN/master/lossscale.html>`_ to process
`loss_scale` correctly.
When separating parameter groups, the weight decay in each group will be applied on the parameters if the
weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied
on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
To improve parameter groups performance, the customized order of parameters can be supported.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params"
are the keys can be parsed.
params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated,
the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params",
"lr", "weight_decay" and "order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- params: Required. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
- lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used.
If not, the `learning_rate` in the API will be used.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
- weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay
will be used. If not, the `weight_decay` in the API will be used.
- order_params: Optional. When parameters is grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3.
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of current step.
- order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and
the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters
which in the 'order_params' must be in one of group parameters.
learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate.
When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then
the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule,
use dynamic learning rate, the i-th learning rate will be calculated during the process of training
according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero
dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be
equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float.
Default: 1e-3.
beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9.
Should be in range (0.0, 1.0).
beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999.
Should be in range (0.0, 1.0).
eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6.
Should be greater than 0.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0.
Inputs:
- **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`.
@ -540,11 +564,9 @@ class AdamWeightDecay(Optimizer):
ValueError: If `weight_decay` is less than 0.
Supported Platforms:
``Ascend`` ``GPU`` ``CPU``
``Ascend`` ``GPU``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.AdamWeightDecay(params=net.trainable_params())
@ -563,40 +585,50 @@ class AdamWeightDecay(Optimizer):
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
_support_parallel_optimizer = True
def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0):
# 初始化AdamWeightDecay类
super(AdamWeightDecay, self).__init__(learning_rate, params, weight_decay)
# 检查参数值
_check_param_value(beta1, beta2, eps, self.cls_name)
# 初始化beta1, beta2, eps
self.beta1 = Tensor(np.array([beta1]).astype(np.float32))
self.beta2 = Tensor(np.array([beta2]).astype(np.float32))
self.eps = Tensor(np.array([eps]).astype(np.float32))
# 初始化moments1, moments2
self.moments1 = self.parameters.clone(prefix="adam_m", init='zeros')
self.moments2 = self.parameters.clone(prefix="adam_v", init='zeros')
# 初始化HyperMap
self.hyper_map = C.HyperMap()
def construct(self, gradients):
weight_decay = self.get_weight_decay()
# 获取学习率
lr = self.get_lr()
# 如果是分组，则使用分组学习率
if self.is_group:
if self.is_group_lr:
# 如果是分组学习率，则按分组方式将学习率按分组方式传入使用HyperMap
optim_result = self.hyper_map(F.partial(_adam_opt, self.beta1, self.beta2, self.eps),
lr, weight_decay, self.parameters, self.moments1,
self.moments2, gradients, self.decay_flags, self.optim_filter)
lr, self.weight_decay, self.parameters, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
else:
# 否则按分组方式使用HyperMap
optim_result = self.hyper_map(F.partial(_adam_opt, self.beta1, self.beta2, self.eps, lr),
weight_decay, self.parameters, self.moments1, self.moments2,
self.weight_decay, self.parameters, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
else:
optim_result = self.hyper_map(F.partial(_adam_opt, self.beta1, self.beta2, self.eps, lr, weight_decay),
# 否则使用HyperMap
optim_result = self.hyper_map(F.partial(_adam_opt, self.beta1, self.beta2, self.eps, lr, self.weight_decay),
self.parameters, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
# 如果使用并行，则广播参数
if self.use_parallel:
self.broadcast_params(optim_result)
# 返回优化结果
return optim_result
class AdamOffload(Optimizer):
#AdamOffload优化器
r"""
This optimizer will offload Adam optimizer to host CPU and keep parameters being updated on the device,
to minimize the memory cost. Although that would bring about an increase of performance overhead,
@ -608,85 +640,65 @@ class AdamOffload(Optimizer):
.. math::
\begin{array}{ll} \\
m_{t+1} = \beta_1 * m_{t} + (1 - \beta_1) * g \\
v_{t+1} = \beta_2 * v_{t} + (1 - \beta_2) * g * g \\
m = \beta_1 * m + (1 - \beta_1) * g \\
v = \beta_2 * v + (1 - \beta_2) * g * g \\
l = \alpha * \frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t} \\
w_{t+1} = w_{t} - l * \frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon}
w = w - l * \frac{m}{\sqrt{v} + \epsilon}
\end{array}
:math:`m` represents the 1st moment vector `moment1`, :math:`v` represents the 2nd moment vector `moment2`,
:math:`g` represents `gradients`, :math:`l` represents scaling factor, :math:`\beta_1, \beta_2` represent
`beta1` and `beta2`, :math:`t` represents the current step while :math:`beta_1^t` and :math:`beta_2^t` represent
:math:`g` represents `gradients`, :math:`l` represents scaling factor `lr`, :math:`\beta_1, \beta_2` represent
`beta1` and `beta2`, :math:`t` represents updating step while :math:`beta_1^t` and :math:`beta_2^t` represent
`beta1_power` and `beta2_power`, :math:`\alpha` represents `learning_rate`, :math:`w` represents `params`,
:math:`\epsilon` represents `eps`.
Note:
This optimizer only supports `GRAPH_MODE` currently.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
When separating parameter groups, the weight decay in each group will be applied on the parameters if the
weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied
on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive.
To improve parameter groups performance, the customized order of parameters is supported.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params"
are the keys can be parsed.
params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated,
the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params",
"lr", "weight_decay" and "order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- params: Required. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
- lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used.
If not, the `learning_rate` in the API will be used.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
- weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay
will be used. If not, the `weight_decay` in the API will be used.
- order_params: Optional. When parameters is grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3.
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of current step.
- order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and
the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters
which in the 'order_params' must be in one of group parameters.
learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate.
When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then
the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule,
use dynamic learning rate, the i-th learning rate will be calculated during the process of training
according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero
dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be
equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float.
Default: 1e-3.
beta1 (float): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0).
Default: 0.9.
beta2 (float): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0).
Default: 0.999.
eps (float): Term added to the denominator to improve numerical stability. Should be greater than 0. Default:
1e-8.
use_locking (bool): Whether to enable a lock to protect the updating process of variable tensors.
If true, updates of the `w`, `m`, and `v` tensors will be protected by a lock.
use_locking (bool): Whether to enable a lock to protect variable tensors from being updated.
If true, updates of the var, m, and v tensors will be protected by a lock.
If false, the result is unpredictable. Default: False.
use_nesterov (bool): Whether to use Nesterov Accelerated Gradient (NAG) algorithm to update the gradients.
If true, update the gradients using NAG.
If false, update the gradients without using NAG. Default: False.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0.
loss_scale (float): A floating point value for the loss scale. Should be greater than 0. In general, use the
default value. Only when `FixedLossScaleManager` is used for training and the `drop_overflow_update` in
`FixedLossScaleManager` is set to False, then this value needs to be the same as the `loss_scale` in
@ -713,8 +725,6 @@ class AdamOffload(Optimizer):
``Ascend`` ``GPU`` ``CPU``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.AdamOffload(params=net.trainable_params())
@ -736,39 +746,77 @@ class AdamOffload(Optimizer):
def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-8, use_locking=False,
use_nesterov=False, weight_decay=0.0, loss_scale=1.0):
'''
参数：
params：参数列表
learning_rate：学习率
beta1：beta1的值
beta2：beta2的值
eps：epsilon的值
use_locking：是否使用锁定
use_nesterov：是否使用Nesterov梯度
weight_decay：权重衰减
loss_scale：损失缩放因子
'''
super(AdamOffload, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 调用_check_param_value方法检测输入参数值是否符合要求
_check_param_value(beta1, beta2, eps, self.cls_name)
# 检查beta1、beta2、eps参数的值是否符合要求
validator.check_value_type("use_locking", use_locking, [bool], self.cls_name)
validator.check_value_type("use_nesterov", use_nesterov, [bool], self.cls_name)
# 将参数赋值给变量
self.beta1 = Tensor(beta1, mstype.float32)
self.beta2 = Tensor(beta2, mstype.float32)
self.beta1_power = Parameter(initializer(1, [1], mstype.float32), name="beta1_power")
self.beta2_power = Parameter(initializer(1, [1], mstype.float32), name="beta2_power")
self.eps = Tensor(eps, mstype.float32)
self.use_nesterov = use_nesterov
self.use_locking = use_locking
# 创建参数
self.moment1 = self.parameters.clone(prefix="moment1", init='zeros')
self.moment2 = self.parameters.clone(prefix="moment2", init='zeros')
# 创建HyperMap
self.hyper_map = C.HyperMap()
# 创建AdamNoUpdateParam
self.opt = P.AdamNoUpdateParam(use_locking, use_nesterov)
# 将primitive_target设置为CPU
self.opt.add_prim_attr("primitive_target", "CPU")
def construct(self, gradients):
'''
参数：
gradients：梯度
'''
params = self.parameters
moment1 = self.moment1
moment2 = self.moment2
# 将梯度衰减
gradients = self.decay_weight(gradients)
# 将梯度缩放
gradients = self.scale_grad(gradients)
# 获取学习率
lr = self.get_lr()
# 计算beta1的平方
beta1_power = self.beta1_power * self.beta1
# 更新beta1的平方
self.beta1_power = beta1_power
# 计算beta2的平方
beta2_power = self.beta2_power * self.beta2
# 更新beta2的平方
self.beta2_power = beta2_power
# 如果是分组学习率
if self.is_group_lr:
success = self.map_reverse(F.partial(_adam_opt, self.opt,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps),
lr, gradients, params, moment1, moment2)
# 调用_adam_opt函数，传入参数
success = self.map_(F.partial(_adam_opt, self.opt,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps),
lr, gradients, params, moment1, moment2)
# 否则
else:
success = self.map_reverse(F.partial(_adam_opt, self.opt,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps, lr),
gradients, params, moment1, moment2)
return success
# 调用_adam_opt函数，传入参数
success = self.map_(F.partial(_adam_opt, self.opt,
beta1_power, beta2_power, self.beta1, self.beta2, self.eps, lr),
gradients, params, moment1, moment2)
return success

263
mindspore/python/mindspore/nn/optim/adasum.py
View File

@ -13,6 +13,7 @@
# limitations under the License.
# ============================================================================
"""adasum"""
# Adaptive Summation (AdaSum)算法的实现
import copy
import hashlib
import math
@ -30,72 +31,114 @@ from mindspore.ops.operations._inner_ops import Send, Receive
from mindspore.common.tensor import Tensor
from mindspore.common import dtype as mstype
from mindspore.communication.management import create_group
# 包含两种功能相近的优化器
__all__ = ["AdaSumByDeltaWeightWrapCell", "AdaSumByGradWrapCell"]
MAX_NUM_HASH = 2 ** 31
# 创建一个MultitypeFuncGraph对象，用于更新参数
_update_parameters = C.MultitypeFuncGraph("update_parameters")
# 创建一个MultitypeFuncGraph对象，用于reshape梯度
_reshape_grads = C.MultitypeFuncGraph("reshape_grads")
@_update_parameters.register("Tensor", "Tensor", "Tensor", "Tensor", "Function")
def _update_parameters_adasum(delta_weight, update_delta_weight, parameter, old_parameter, reshape):
'''
更新参数
:param delta_weight: 参数梯度
:param update_delta_weight: 更新参数梯度
:param parameter: 参数
:param old_parameter: 原始参数
:param reshape: 将参数转换为形状
:return: 更新后的参数
'''
shape = F.shape(delta_weight)
update_delta_weight = reshape(update_delta_weight, shape)
new_parameter = old_parameter - update_delta_weight
return P.Assign()(parameter, new_parameter)
@_reshape_grads.register("Tensor", "Tensor", "Function")
def reshape_grads_adasum(grads, update_grads, reshape):
"""
Reshape gradient.
"""
# 获取梯度的形状
shape = F.shape(grads)
# 将梯度reshape到指定形状
update_grads = reshape(update_grads, shape)
# 返回reshaped梯度
return update_grads
def _send_before_receive(send_part, send, recv):
'''
接收前发送
:param send_part: 发送的部分
:param send: 发送函数
:param recv: 接收函数
:return: 接收结果
'''
send_ok = send(send_part)
# 发送send_part到服务器
return recv(send_ok)
def _receive_before_send(send_part, send, recv):
'''
接收前发送
'''
# 接收发送部分
receive_ok = recv(send_part)
# 将发送部分依赖
send_part = F.depend(send_part, receive_ok)
# 将依赖发送
return F.depend(receive_ok, send(send_part))
def _send_recv_res(left_send, recv_part, local_part, allreduce, parameter_divisibility, allreduce_node_num):
"""send result and receive result."""
if parameter_divisibility:
# 将recv_part拆分为两部分
recv_part = P.Squeeze()(recv_part)
# 如果recv_part的形状为None，则将recv_part设置为一个元素
if F.shape(recv_part) is None:
recv_part = Tensor([recv_part])
# 将local_part和recv_part合并
local_part = F.depend(local_part, recv_part)
# 将eps设置为1e-12
eps = 1e-12
# 计算local_part的最大值，并将其值除以scale_value
scale_value = P.ReduceMax()(local_part) + eps
# 将local_part除以scale_value
local_part_scale = local_part / scale_value
# 将recv_part除以scale_value
recv_part_scale = recv_part / scale_value
# 将recv_part_scale和local_part_scale合并
recv_part_scale = F.depend(recv_part_scale, local_part_scale)
# 计算value_0
value_0 = P.ReduceSum()(local_part_scale * recv_part_scale) + eps
# 如果left_send为True，计算value_1
if left_send:
value_1 = P.ReduceSum()(local_part_scale * local_part_scale) + eps
value_2 = P.ReduceSum()(recv_part_scale * recv_part_scale) + eps
# 如果left_send为False，计算value_1
else:
value_1 = P.ReduceSum()(recv_part_scale * recv_part_scale) + eps
value_2 = P.ReduceSum()(local_part_scale * local_part_scale) + eps
# 将value_0和value_1合并
value_0 = allreduce(value_0)
value_1 = F.depend(allreduce(value_1), value_0)
value_2 = F.depend(allreduce(value_2), value_1)
if left_send:
# 如果左边发送，则res等于1-value_0/(2*value_1)的和，否则res等于1-value_0/(2*value_1)的和
res = (1 - (value_0 / (2 * value_1))) * local_part + (1 - (value_0 / (2 * value_2))) * recv_part
else:
# 如果右边发送，则res等于1-value_0/(2*value_1)的和，否则res等于1-value_0/(2*value_1)的和
res = (1 - (value_0 / (2 * value_1))) * recv_part + (1 - (value_0 / (2 * value_2))) * local_part
else:
# 如果所有节点都是自身，则直接使用allreduce
res = allreduce(local_part)
# 计算并行计算的节点数
res = res / allreduce_node_num
return res
@ -107,60 +150,92 @@ _adasum_opt_rollback = C.MultitypeFuncGraph("adasum_opt_rollback")
@_adasum_opt_forward.register("Bool", "Function", "Bool", "Int64", "Function", "Function", "Tensor")
def _adasum_opt_forward_process(left_send, allreduce, parameter_divisibility, allreduce_node_num, send, recv, delta_w):
"""adasum optimizer process."""
# 如果parameter_divisibility为True，则delta_w的维度为1
if parameter_divisibility:
delta_w = P.Squeeze()(delta_w)
# 将delta_w的维度除以2
ori_len = F.shape(delta_w)[0]
divide_len = ori_len / 2
# 将delta_w的前divide_len个元素取平均
left_part = delta_w[:divide_len]
right_part = delta_w[divide_len:]
# 如果parameter_divisibility为False，则delta_w的维度为2
else:
left_part = delta_w
right_part = delta_w
# 如果left_send为True，则发送delta_w到发送端
if left_send:
if parameter_divisibility:
# 如果参数除法，则将左边部分发送到右边部分，并将右边部分接收到左边部分
recv_part = _send_before_receive(left_part, send, recv)
else:
# 否则，将右边部分发送到左边部分，并将左边部分接收到右边部分
recv_part = right_part
# 发送delta_w到发送端，并计算update_delta_w
update_delta_w = _send_recv_res(left_send, recv_part, right_part, allreduce, parameter_divisibility,
allreduce_node_num)
# 如果left_send为False，则接收delta_w从发送端
else:
# 如果参数除法，则将右边部分发送到左边部分，并将左边部分接收到右边部分
if parameter_divisibility:
recv_part = _receive_before_send(right_part, send, recv)
# 否则，将左边部分发送到右边部分，并将右边部分接收到左边部分
else:
recv_part = left_part
# 接收delta_w从发送端，并计算update_delta_w
update_delta_w = _send_recv_res(left_send, recv_part, left_part, allreduce, parameter_divisibility,
allreduce_node_num)
# 返回update_delta_w
return update_delta_w
@_adasum_opt_rollback.register("Bool", "Bool", "Tensor", "Function", "Function")
def _adasum_opt_rollback_process(left_send, parameter_divisibility, delta_w, send, recv):
"""adasum optimizer rollback process."""
# 如果参数除法
if parameter_divisibility:
# 如果左发送
if left_send:
# 发送前接收
recv_part = _send_before_receive(delta_w, send, recv)
# 如果右发送
else:
# 接收前发送
recv_part = _receive_before_send(delta_w, send, recv)
# 将发送和接收部分拉平
recv_part = P.Squeeze()(recv_part)
# 如果接收部分为空
if F.shape(recv_part) is None:
# 将接收部分设置为一个tensor
recv_part = Tensor([recv_part])
# 如果发送部分为空
if F.shape(delta_w) is None:
# 将发送部分设置为一个tensor
delta_w = Tensor([delta_w])
# 将接收部分和发送部分压缩
recv_part = P.Reshape()(recv_part, (-1,))
delta_w = P.Reshape()(delta_w, (-1,))
# 如果左发送
if left_send:
# 将接收部分和发送部分拼接
res = P.Concat()((recv_part, delta_w))
# 如果右发送
else:
# 将发送部分和接收部分拼接
res = P.Concat()((delta_w, recv_part))
# 如果不参数除法
else:
# 将发送部分设置为发送部分
res = delta_w
# 返回拼接后的结果
return res
class _AdaSum(Cell):
# 自适应求和算法是一种改进深度学习模型的分布式数据并行训练的新算法。
r"""
The Adaptive Summation, or AdaSum, is a novel algorithm for improving distributed data
parallel training of Deep Learning models.
@ -174,44 +249,68 @@ class _AdaSum(Cell):
- **adasum_parameters** (Tuple(Tensor)) - Tuple of parameters after adasum process.
"""
def __init__(self, rank, device_number, group_number, parameter_tuple):
# 初始化_AdaSum类
super(_AdaSum, self).__init__()
# 设置rank
self.rank = rank
# 设置device_number
self.device_number = device_number
# 设置group_number
self.group_number = group_number
# 设置参数tuple
self.parameter_tuple = parameter_tuple
# 生成通信操作
self._generate_communication_op()
# 初始化HyperMap
self.hyper_map = C.HyperMap()
# 初始化update_reshape_list
self.update_reshape_list = []
# 遍历参数tuple
for parameter in self.parameter_tuple:
# 初始化reshape
reshape = P.Reshape().add_prim_attr("target_param", "adasum_delta_weight." + parameter.name)
# 将reshape添加到update_reshape_list中
self.update_reshape_list.append(reshape)
@staticmethod
def _hash(step, target, weights_index):
# 将step、target、weights_index拼接成字符串
target = "tag" + str(step) + str(target) + str(weights_index)
# 将字符串转换为sha1格式的字节码
target_hash = hashlib.sha1(target.encode()).hexdigest()
# 将字节码转换为整数
hash_res = int(int(target_hash, 16) % MAX_NUM_HASH)
# 返回hash_res
return hash_res
def construct(self, delta_weights, parameters, old_parameters):
# 初始化反向传播权重
forward_weights = [delta_weights]
# 循环反向传播
for i in range(self.calc_times):
# 计算反向传播权重
process_weights = self.hyper_map(F.partial(_adasum_opt_forward, self.send_node[i]), self.allreduce_list[i],
self.parameter_divisibility_list[i], self.allreduce_node_num_list[i],
self.send_list_forward[i], self.recv_list_forward[i], forward_weights[-1])
# 将反向传播权重添加到反向传播权重列表中
forward_weights.append(process_weights)
# 循环反向传播
for i in range(self.calc_times):
# 计算反向传播权重
j = self.calc_times - i - 1
process_weights = self.hyper_map(F.partial(_adasum_opt_rollback, self.send_node[j]),
self.parameter_divisibility_list[j], forward_weights[j + 1],
self.send_list_rollback[j], self.recv_list_rollback[j])
# 将反向传播权重添加到反向传播权重列表中
forward_weights[j] = process_weights
# 计算反向传播参数
adasum_parameters = self.hyper_map(F.partial(_update_parameters), delta_weights, forward_weights[0],
parameters, old_parameters, self.update_reshape_list)
# 返回反向传播参数
return adasum_parameters
def _generate_communication_op(self):
"""generate communication op."""
# 初始化参数
self.calc_times = int(math.log(self.group_number, 2))
self.send_node = []
self.send_list_forward = []
@ -222,14 +321,19 @@ class _AdaSum(Cell):
self.parameter_divisibility_list = []
self.allreduce_node_num_list = []
last_delta_weights = []
# 如果当前设置的并行模式为data_parallel或hybrid_parallel，则将fusion_attr设置为fusion
fusion_attr = "fusion" if context.get_auto_parallel_context("parallel_mode") \
in ["data_parallel", "hybrid_parallel"] else "origin_fusion"
for step in range(self.calc_times):
# 计算步骤
current_group = self.device_number * (2 ** step)
# 设置源节点
sr_target = self.rank
# 如果源节点的组数为奇数，则将源节点的组数加上当前步骤的组数
if (sr_target // current_group) % 2 == 0:
dest_target = sr_target + current_group
self.send_node.append(True)
# 如果源节点的组数为偶数，则将源节点的组数减去当前步骤的组数
else:
dest_target = sr_target - current_group
self.send_node.append(False)
@ -239,26 +343,43 @@ class _AdaSum(Cell):
recv_left = []
recv_right = []
allreduce_node_num = ()
# 获取上一次的delta_weights信息
left_delta_weights, right_delta_weights, delta_weights_divisibility = \
self._get_delta_weights_info(last_delta_weights)
# 将delta_weights_divisibility添加到参数分配列表中
self.parameter_divisibility_list.append(delta_weights_divisibility)
# 初始化变量weights_index
weights_index = 0
# 计算fusion_id
fusion_id = (step + 1) * 3
for shape, dtype, name in left_delta_weights:
# 创建发送标签
send_tag = self._hash(step, sr_target, weights_index)
# 创建发送对象
send = Send(sr_tag=send_tag, dest_rank=dest_target, group="hccl_world_group")
# 添加fusion_attr属性
send.add_prim_attr(fusion_attr, fusion_id)
# 添加opposite_rank属性
send.add_prim_attr("opposite_rank", dest_target)
# 添加target_param属性
send.add_prim_attr("target_param", name)
# 创建接收标签
recv_tag = self._hash(step, dest_target, weights_index)
# 创建接收对象
recv = Receive(sr_tag=recv_tag, src_rank=dest_target, shape=shape, dtype=dtype,
group="hccl_world_group")
# 添加fusion_attr属性
recv.add_prim_attr(fusion_attr, fusion_id)
# 添加opposite_rank属性
recv.add_prim_attr("opposite_rank", dest_target)
# 添加target_param属性
recv.add_prim_attr("target_param", name)
# 将发送对象和接收对象添加到send_left列表中
send_left.append(send)
recv_left.append(recv)
# 更新weights_index
weights_index += 1
# 重复上述操作
for shape, dtype, name in right_delta_weights:
send_tag = self._hash(step, sr_target, weights_index)
send = Send(sr_tag=send_tag, dest_rank=dest_target, group="hccl_world_group")
@ -275,131 +396,206 @@ class _AdaSum(Cell):
recv_right.append(recv)
weights_index += 1
if self.send_node and self.send_node[-1]:
# 如果发送节点有最后一个节点，则将发送节点添加到发送列表中
self.send_list_forward.append(send_left)
self.send_list_rollback.append(send_right)
# 将接收节点添加到接收列表中
self.recv_list_forward.append(recv_right)
self.recv_list_rollback.append(recv_left)
# 将右边的delta_weights添加到最后一个delta_weights中
last_delta_weights = right_delta_weights
else:
# 如果发送节点没有最后一个节点，则将右边的节点添加到发送列表中
self.send_list_forward.append(send_right)
self.send_list_rollback.append(send_left)
# 将左边的节点添加到接收列表中
self.recv_list_forward.append(recv_left)
self.recv_list_rollback.append(recv_right)
# 将左边的delta_weights添加到最后一个delta_weights中
last_delta_weights = left_delta_weights
param_allreduce_list = []
neighbor_ids = []
rank_ids = []
for index in range(2 ** (step + 1)):
# 计算节点的编号
node_rank = self.rank // self.device_number
# 计算节点的双边
double_d = 2 ** (step + 1)
# 计算邻节点的编号
neighbor_id = (node_rank // double_d * double_d + index) * self.device_number + \
self.rank % self.device_number
# 将节点的编号添加到neighbor_ids列表中
neighbor_ids.append(str(neighbor_id))
# 将节点的编号添加到rank_ids列表中
rank_ids.append(neighbor_id)
# 将neighbor_ids列表转换为字符串
group_name = "-".join(neighbor_ids)
# 如果parallel_mode为data_parallel或hybrid_parallel，则创建组
if context.get_auto_parallel_context("parallel_mode") in ["data_parallel", "hybrid_parallel"]:
create_group(group_name, rank_ids)
for parameter in self.parameter_tuple:
# 创建一个所有计算的参数
allreduce = P.AllReduce("sum", group_name)
# 添加target_param属性，值为adasum_delta_weight.parameter.name
allreduce.add_prim_attr("target_param", "adasum_delta_weight." + parameter.name)
# 添加fusion_attr属性，值为fusion_id + 2
allreduce.add_prim_attr(fusion_attr, fusion_id + 2)
# 添加step属性，值为step
allreduce.add_prim_attr("step", step)
# 添加param_allreduce_list列表
param_allreduce_list.append(allreduce)
# 添加param_allreduce_list列表
self.allreduce_list.append(param_allreduce_list)
# 创建一个每个参数的除数
for param_divisibility in delta_weights_divisibility:
# 如果param_divisibility为真，则计算节点数为0
if param_divisibility:
allreduce_node_num += (0,)
# 否则计算节点数为2的幂次方
else:
allreduce_node_num += (2 ** (step + 1),)
# 添加allreduce_node_num_list列表
self.allreduce_node_num_list.append(allreduce_node_num)
def _get_delta_weights_info(self, last_delta_weights):
# 获取delta_weights的信息
"""get delta weights info."""
half_delta_weights = []
if last_delta_weights:
# 如果有上一次的delta_weights
half_delta_weights = last_delta_weights
else:
# 否则
for parameter in self.parameter_tuple:
# 遍历参数
new_shape = [int(x) for x in parameter.shape]
# 获取参数形状
half_delta_weights.append((new_shape, parameter.dtype, "adasum_delta_weight." + parameter.name))
left_delta_weights = []
right_delta_weights = []
delta_weights_divisibility = ()
for shape, dtype, name in half_delta_weights:
# 创建一个新的shape，用来存储左右边界
left_shape = copy.deepcopy(shape)
right_shape = copy.deepcopy(shape)
# 初始化divisibility_flag为False
divisibility_flag = False
# 遍历shape，如果当前值大于1，则将当前值除以2，并将除以2后的值存入left_shape，并将当前值减去除以2后的值存入right_shape
for i, value in enumerate(shape):
if value > 1:
left_shape[i] = int(value // 2)
right_shape[i] = value - int(value // 2)
divisibility_flag = True
break
# 将左右边界shape和dtype和name添加到left_delta_weights和right_delta_weights中
left_delta_weights.append((left_shape, dtype, name))
right_delta_weights.append((right_shape, dtype, name))
# 将divisibility_flag赋值给delta_weights_divisibility
delta_weights_divisibility += (divisibility_flag,)
# 返回left_delta_weights和right_delta_weights和delta_weights_divisibility
return left_delta_weights, right_delta_weights, delta_weights_divisibility
class _AdaSumByGrad(_AdaSum):
# Adaptive Summation (AdaSum)算法的实现，根据梯度计算。
"""Apply adasum by gradients"""
def construct(self, grads):
# 初始化前向传播的梯度
forward_grads = [grads]
# 循环计算梯度
for i in range(self.calc_times):
# 计算前向传播的梯度
process_weights = self.hyper_map(F.partial(_adasum_opt_forward, self.send_node[i]), self.allreduce_list[i],
self.parameter_divisibility_list[i], self.allreduce_node_num_list[i],
self.send_list_forward[i], self.recv_list_forward[i], forward_grads[-1])
self.parameter_divisibility_list[i], self.allreduce_node_num_list[i],
self.send_list_forward[i], self.recv_list_forward[i], forward_grads[-1])
# 将计算的梯度添加到前向传播的梯度列表中
forward_grads.append(process_weights)
# 循环计算回滚梯度
for i in range(self.calc_times):
# 计算回滚梯度的梯度
j = self.calc_times - i - 1
process_weights = self.hyper_map(F.partial(_adasum_opt_rollback, self.send_node[j]),
self.parameter_divisibility_list[j], forward_grads[j + 1],
self.send_list_rollback[j], self.recv_list_rollback[j])
self.parameter_divisibility_list[j], forward_grads[j + 1],
self.send_list_rollback[j], self.recv_list_rollback[j])
# 将计算的梯度添加到回滚梯度的梯度列表中
forward_grads[j] = process_weights
# 计算更新梯度
update_grads = self.hyper_map(F.partial(_reshape_grads), grads, forward_grads[0],
self.update_reshape_list)
self.update_reshape_list)
# 返回更新梯度
return update_grads
# 定义_get_delta_weight函数，用于获取delta的权重
_get_delta_weight = C.MultitypeFuncGraph("_get_delta_weight")
# 定义_save_weight函数，用于保存权重
_save_weight = C.MultitypeFuncGraph("_save_weight")
# 定义scale_mul函数，用于乘以delta的权重
scale_mul = P.Mul().add_prim_attr("keep_alive", True)
# 定义_clone_weight函数，用于克隆权重
_clone_weight = C.MultitypeFuncGraph("_clone_weight")
@_get_delta_weight.register("Tensor", "Tensor")
def _get_delta_weight_process(new_parameter, old_parameter):
'''
计算参数更新的梯度
:param new_parameter: 新的参数
:param old_parameter: 旧的参数
:return: 梯度
'''
delta_w = old_parameter - new_parameter
return delta_w
@_save_weight.register("Tensor", "Tensor")
def _save_weight_process(new_parameter, old_parameter):
'''
保存参数的处理函数
:param new_parameter: 新参数
:param old_parameter: 之前的参数
:return:
'''
return P.Assign()(new_parameter, old_parameter)
@_clone_weight.register("Tensor", "Tensor")
def _clone_weight_process(scale, weight):
'''
给定一个缩放因子和权重，返回缩放后的权重
:param scale: 缩放因子
:param weight: 权重
:return: 缩放后的权重
'''
return scale_mul(weight, scale)
def _parallel_check():
# 检查自动并行信息
"""Parallel infos checking"""
# 如果自动并行模式为stand_alone，抛出异常
if context.get_auto_parallel_context("parallel_mode") == "stand_alone":
raise RuntimeError("Stand alone mode is not supported to apply adasum.")
# 如果自动并行模式为data_parallel或hybrid_parallel，警告
if context.get_auto_parallel_context("parallel_mode") in ["data_parallel", "hybrid_parallel"]:
logger.warning("For data parallel mode or hybrid parallel mode, "
"it is recommended to using mindspore.boost to enable adasum.")
# 如果自动并行模式为enable_parallel_optimizer，抛出异常
if context.get_auto_parallel_context("enable_parallel_optimizer"):
raise RuntimeError("Currently, the optimizer shard is not supported with applying adasum.")
# 如果自动并行模式为pipeline_stages大于1，抛出异常
if context.get_auto_parallel_context("pipeline_stages") > 1:
raise RuntimeError("Currently, the pipeline parallel is not supported with applying adasum.")
# 获取每个stage的设备数量
stage_device_num = _get_stage_device_num()
if stage_device_num < 16 or (stage_device_num & (stage_device_num - 1) != 0):
# 如果stage的设备数量小于16或者stage的设备数量与stage的设备数量的最低位不同，抛出异常
if stage_device_num < 16 or (stage_device_num & (stage_device_num - 1)!= 0):
raise RuntimeError("The device_num should be at least 16 and should be the power of 2 when applying adasum.")
class AdaSumByGradWrapCell(Cell):
# Adaptive Summation (AdaSum)算法的实现，根据梯度计算，且用于自动并行模式。
r"""
Enable the adasum in "auto_parallel/semi_auto_parallel" mode.
The implementation of the Adaptive Summation (AdaSum) algorithm is calculated by gradients.
@ -446,26 +642,51 @@ class AdaSumByGradWrapCell(Cell):
>>> model = Model(net, loss_fn=loss, optimizer=optim, metrics=None)
"""
def __init__(self, optimizer):
'''
初始化AdaSumByGradWrapCell类
:param optimizer: 优化器
:return: None
'''
super(AdaSumByGradWrapCell, self).__init__(auto_prefix=False)
# 定义设备数量
_device_number = 8
# 检查设备数量
_parallel_check()
# 将优化器赋值给变量
self.optimizer = optimizer
# 检查优化器类型
validator.check_value_type('optimizer', optimizer, (nn.Optimizer,))
# 将参数赋值给变量
self.parameters = optimizer.parameters
# 定义网络映射
self.hyper_map = C.HyperMap()
# 获取网络设备数量
group_number = _get_stage_device_num() // _device_number
# 定义参数克隆
self.grad_clone = ParameterTuple(self.parameters)
# 定义AdaSum
self.adasum = _AdaSumByGrad(_get_global_rank(), _device_number, group_number, self.grad_clone)
# 定义同步Tensor
self.sync_tensor = Parameter(Tensor(0, dtype=mstype.int32))
def construct(self, grads):
'''
构建优化器
:param grads: 优化器梯度
:return:
'''
# 调用adasum函数求和
adasum_res = self.adasum(grads)
# 调用sync_tensor函数同步
sync_tensor = F.depend(self.sync_tensor, adasum_res)
# 调用AllReduce函数并发
sync_flag = P.AllReduce()(sync_tensor)
# 调用optimizer函数优化
return F.depend(self.optimizer(adasum_res), sync_flag)
class AdaSumByDeltaWeightWrapCell(Cell):
# Adaptive Summation (AdaSum)算法的实现，根据更新前后的参数差计算。
r"""
Enable the adasum in "auto_parallel/semi_auto_parallel" mode.
The implementation of the Adaptive Summation (AdaSum) algorithm is calculated based on the difference of weights
@ -513,27 +734,49 @@ class AdaSumByDeltaWeightWrapCell(Cell):
>>> model = Model(net, loss_fn=loss, optimizer=optim, metrics=None)
"""
def __init__(self, optimizer):
# 初始化类
super(AdaSumByDeltaWeightWrapCell, self).__init__(auto_prefix=False)
# 检查优化器是否为nn.Optimizer类型
_parallel_check()
# 将优化器赋值给变量
self.optimizer = optimizer
# 检查优化器是否为nn.Optimizer类型
validator.check_value_type('optimizer', optimizer, (nn.Optimizer,))
# 将参数赋值给变量
self.parameters = optimizer.parameters
# 初始化网络
self.hyper_map = C.HyperMap()
# 获取设备数量
_device_number = 8
# 获取网络分块数量
group_number = _get_stage_device_num() // _device_number
# 初始化参数克隆
self.grad_clone = ParameterTuple(self.parameters)
# 初始化AdaSum
self.adasum = _AdaSum(_get_global_rank(), _device_number, group_number, self.grad_clone)
# 初始化同步变量
self.sync_tensor = Parameter(Tensor(0, dtype=mstype.int32))
# 初始化缩放因子
self.scale = Tensor(1.0, dtype=mstype.float32)
def construct(self, grads):
# 将梯度复制到参数上
grad_clone = self.hyper_map(F.partial(_clone_weight, self.scale), self.parameters)
# 将复制的梯度添加到优化器中
grads = F.depend(grads, grad_clone)
# 使用优化器更新参数
opt_result = self.optimizer(grads)
# 使用参数更新后的梯度更新参数
parameters = F.depend(self.parameters, opt_result)
# 计算参数的梯度
delta_w = self.hyper_map(F.partial(_get_delta_weight), parameters, grad_clone)
# 使用AdaSum更新参数
adasum_res = self.adasum(delta_w, parameters, grad_clone)
# 使用同步计算
sync_tensor = F.depend(self.sync_tensor, adasum_res)
# 使用AllReduce更新参数
sync_flag = P.AllReduce()(sync_tensor)
# 使用参数更新后的参数
updated_weights = F.depend(parameters, sync_flag)
# 返回更新后的参数
return updated_weights

50
mindspore/python/mindspore/nn/optim/asgd.py
View File

@ -13,6 +13,10 @@
# limitations under the License.
# ============================================================================
"""asgd"""
# 随机平均梯度下降（ASGD）算法的实现。ASGD也成为SAG，均表示随机平均梯度下降(Averaged Stochastic Gradient Descent)，简单地说ASGD就是用空间换时间的一种SGD
# 首先从mindspore.ops中导入了一些操作类，如operations as P用于创建一些操作，Parameter用于表示参数变量，Tensor用于表示张量变量。
# 接着，导入了一些其他模块，如mindspore.common.dtype用于表示数据类型，mindspore.common.tensor用于表示张量。
# 最后，从mindspore._checkparam模块中导入了一个Validator类，用于验证参数是否符合要求。
from mindspore.ops import operations as P
from mindspore.common.parameter import Parameter
from mindspore.common.tensor import Tensor
@ -147,22 +151,38 @@ class ASGD(Optimizer):
@opt_init_args_register
def __init__(self, params, learning_rate=0.1, lambd=1e-4, alpha=0.75, t0=1e6, weight_decay=0.):
'''
初始化ASGD算法，包括设置学习率、权重衰减等参数，以及初始化移动平均和指数加权移动平均等变量。
:param params: 参数
:param learning_rate: 学习率
:param lambd: 损失函数
:param alpha: 平滑系数
:param t0: 开始时间的点
:param weight_decay: 权重衰减
'''
super(ASGD, self).__init__(learning_rate, params, weight_decay)
# 检查lambd是否为浮点型
validator.check_value_type("lambd", lambd, [float], self.cls_name)
# 检查alpha是否为浮点型
validator.check_value_type("alpha", alpha, [float], self.cls_name)
# 检查t0是否为浮点型
validator.check_value_type("t0", t0, [float], self.cls_name)
# 初始化参数
self.lambd = lambd
self.alpha = alpha
self.t0 = Tensor([t0], dtype=mstype.float32)
mu, eta = [], []
for param in self.parameters:
# 创建mu和eta变量并指定其矩阵类型
mu.append(Parameter(Tensor(1., dtype=mstype.float32), name='%s%s' % ("mu_", param.name)))
eta.append(Parameter(Tensor(0., dtype=mstype.float32), name='%s%s' % ("eta_", param.name)))
self.lens = len(self.parameters)
# 将一个包含多个参数的列表mu转换为一个ParameterTuple对象。
self.mu = mindspore.ParameterTuple(mu)
# 将一个包含多个参数的列表eta转换为一个ParameterTuple对象。
self.eta = mindspore.ParameterTuple(eta)
# ax是一个张量变量，用于存储梯度的指数移动平均参数值。
self.ax = self.parameters.clone(prefix="ax_", init='zeros')
self.pow = P.Pow()
self.maximum = P.Maximum()
@ -173,33 +193,59 @@ class ASGD(Optimizer):
self.squeeze = P.Squeeze()
def construct(self, gradients):
# 实现ASGD算法的更新规则。首先对梯度进行权重衰减、梯度 centralization 处理，然后将梯度缩放以适应不同的学习率，最后更新参数、移动平均和指数加权移动平均等变量。
# 对梯度进行权重衰减
gradients = self.decay_weight(gradients)
# 将梯度中心化处理
gradients = self.gradients_centralization(gradients)
# 将梯度缩放
gradients = self.scale_grad(gradients)
# 获取学习率
lrs = self.get_lr()
# 是否成功
success = True
# 遍历梯度
for index, (grad, param, mu, eta, ax) in enumerate(zip(gradients, self.parameters, self.mu, self.eta, self.ax)):
# 获取学习率
lr = lrs[index] if self.is_group_lr else lrs
# 将学习率平均
lr = self.squeeze(lr)
# 如果当前步数为1，则将学习率设置为原来的学习率
if self.squeeze(self.global_step) == 1:
self.assign(eta, lr)
# 获取参数
param_fp32 = self.cast(param, mstype.float32)
# 获取梯度
gradient_fp32 = self.cast(grad, mstype.float32)
# 获取平均参数值
ax_fp32 = self.cast(ax, mstype.float32)
# 将参数乘以1-lambd*eta，并与梯度相加（函数算法）
param_fp32 = param_fp32 * (1. - self.lambd * eta) - eta * gradient_fp32
# 将参数赋值给参数
self.assign(param, self.cast(param_fp32, param.dtype))
if mu != 1:
# 如果mu不等于1，则将参数乘以mu，并与参数相加
if mu!= 1:
self.assignadd(ax, self.cast((param_fp32 - ax_fp32) * mu, ax.dtype))
else:
# 否则，将参数赋值给参数
self.assign(ax, param)
# 1.将学习率除以(1.0 + (self.lambd * lr * self.cast(self.squeeze(self.global_step), mstype.float32))^self.alpha)，得到一个计算公式。
# 2.将计算公式赋值给变量eta。
self.assign(eta, lr / (self.pow((1. + (self.lambd * lr * self.cast(
self.squeeze(self.global_step), mstype.float32))), self.alpha)))
# 1.将全局步数self.squeeze(self.global_step)转换为浮点数类型，并命名为step_float。
# 2.将step_float与1.0进行比较，得到一个布尔值。
# 3.将布尔值转换为浮点数类型，并命名为compare_float。
# 4.使用self.maximum()函数计算compare_float与1.0的最大值，并命名为max_float。
# 5.将max_float除以step_float，得到一个计算公式。
# 6.将计算公式赋值给变量mu。
self.assign(mu, 1. / self.squeeze(self.maximum(1., self.cast(
self.squeeze(self.global_step), mstype.float32) - self.t0)))
# 返回成功消息
return success

166
mindspore/python/mindspore/nn/optim/ftrl.py
View File

@ -1,4 +1,4 @@
# Copyright 2020-2021 Huawei Technologies Co., Ltd
# Copyright 2020 Huawei Technologies Co., Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -13,14 +13,15 @@
# limitations under the License.
# ============================================================================
"""FTRL"""
# 本文件为FTRL优化器的构建
from mindspore.ops import functional as F, composite as C, operations as P
from mindspore.common import Tensor
import mindspore.common.dtype as mstype
from mindspore._checkparam import Validator as validator
from mindspore._checkparam import Rel
from .optimizer import Optimizer, _apply_decay, _grad_scale
from .optimizer import opt_init_args_register
from.optimizer import Optimizer, _apply_decay, _grad_scale
# 定义_ftrl_opt函数，用于接收四个参数：opt（优化器），spars_opt（模式优化器），push（推送），pull（拉取），l1（L1正则化），l2（L2正则化），lr_power（学习率指数），learning_rate（学习率），linear（线性），gradient（梯度），weight（权重），moment（动量），ps_parameter（参数），cache_enable（是否使用缓存）
_ftrl_opt = C.MultitypeFuncGraph("ftrl_opt")
@ -29,14 +30,23 @@ _ftrl_opt = C.MultitypeFuncGraph("ftrl_opt")
def _tensor_run_opt_with_sparse(opt, spars_opt, push, pull, l1, l2, lr_power, learning_rate, linear,
gradient, weight, moment, ps_parameter, cache_enable):
"""Apply sparse ftrl optimizer to the weight parameter when the gradient is sparse."""
# 对为稀疏矩阵的参数权重使用FTRL优化器
# 判断是否满足条件
success = True
# 获取梯度的索引和值
indices = gradient.indices
values = gradient.values
# 如果没有指定ps参数，且不满足cache_enable条件
if ps_parameter and not cache_enable:
# 获取操作的形状
op_shape = P.Shape()
# 获取形状
shapes = (op_shape(weight), op_shape(moment), op_shape(linear), op_shape(values), op_shape(indices))
# 执行pull操作
success = F.depend(success, pull(push((values, indices), shapes), weight))
# 如果指定了ps参数，且满足cache_enable条件
else:
# 执行spars_opt操作
success = F.depend(success, spars_opt(weight, moment, linear, values, indices))
return success
@ -45,37 +55,54 @@ def _tensor_run_opt_with_sparse(opt, spars_opt, push, pull, l1, l2, lr_power, le
"Tensor", "Tensor", "Tensor", "Bool", "Bool")
def _tensor_run_opt(opt, spars_opt, push, pull, l1, l2, lr_power, learning_rate, linear,
gradient, weight, moment, ps_parameter, cache_enable):
# 对权重参数应用FTRL优化器
"""Apply ftrl optimizer to the weight parameter."""
success = True
# 如果ps_parameter为True，且不支持缓存，则使用pull函数拉取push函数的输出
if ps_parameter and not cache_enable:
op_shape = P.Shape()
success = F.depend(success, pull(push((gradient, learning_rate, l1, l2, lr_power),
(op_shape(weight), op_shape(moment), op_shape(linear))), weight))
# 否则，使用opt函数计算weight的梯度，并使用push函数推送梯度
else:
success = F.depend(success, opt(weight, moment, linear, gradient, learning_rate, l1, l2, lr_power))
# 返回更新成功标志
return success
def _check_param(initial_accum, lr_power, l1, l2, use_locking, prim_name=None):
"""Check param."""
"""
检查参数
"""
# 检查initial_accum的类型是否为float
validator.check_value_type("initial_accum", initial_accum, [float], prim_name)
# 检查initial_accum的值是否在0.0以上
validator.check_number("initial_accum", initial_accum, 0.0, Rel.GE, prim_name)
# 检查lr_power的类型是否为float
validator.check_value_type("lr_power", lr_power, [float], prim_name)
# 检查lr_power的值是否在0.0和Rel.LE之间
validator.check_number("lr_power", lr_power, 0.0, Rel.LE, prim_name)
# 检查l1的类型是否为float
validator.check_value_type("l1", l1, [float], prim_name)
# 检查l1的值是否在0.0和Rel.GE之间
validator.check_number("l1", l1, 0.0, Rel.GE, prim_name)
# 检查l2的类型是否为float
validator.check_value_type("l2", l2, [float], prim_name)
# 检查l2的值是否在0.0和Rel.GE之间
validator.check_number("l2", l2, 0.0, Rel.GE, prim_name)
# 检查use_locking的类型是否为bool
validator.check_value_type("use_locking", use_locking, [bool], prim_name)
class FTRL(Optimizer):
# FTRL优化器
r"""
Implements the FTRL algorithm.
Implements the FTRL algorithm with ApplyFtrl Operator.
FTRL is an online convex optimization algorithm that adaptively chooses its regularization function
based on the loss functions. Refer to paper `Adaptive Bound Optimization for Online Convex Optimization
@ -98,47 +125,46 @@ class FTRL(Optimizer):
\end{cases}\\
\end{array}
:math:`m` represents accumulators, :math:`g` represents `grads`, :math:`t` represents the current step,
:math:`u` represents the linear coefficient to be updated, :math:`p` represents `lr_power`, :math:`\alpha`
represents `learning_rate`, :math:`\omega` represents `params`.
:math:`m` represents `accum`, :math:`g` represents `grads`, :math:`t` represents updating step,
:math:`u` represents `linear`, :math:`p` represents `lr_power`, :math:`\alpha` represents `learning_rate`,
:math:`\omega` represents `params`.
Note:
The sparse strategy is applied while the SparseGatherV2 operator is used for forward network. If the sparse
strategy wants to be executed on the host, set the target to the CPU.
The sparse feature is under continuous development.
When separating parameter groups, the weight decay in each group will be applied on the parameters if the
weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied
on all of the parameters.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
When separating parameter groups, if you want to centralize the gradient, set grad_centralization to True,
but the gradient centralization can only be applied to the parameters of the convolution layer.
If the parameters of the non convolution layer are set to True, an error will be reported.
To improve parameter groups performance, the customized order of parameters can be supported.
The sparse strategy is applied while the SparseGatherV2 operator being used for forward network.
The sparse feature is under continuous development. If the sparse strategy wants to be executed on the host,
set the target to the CPU.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "weight_decay", "grad_centralization" and "order_params"
are the keys can be parsed.
params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated,
the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params",
"lr", "weight_decay" and "order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- params: Required. The value must be a list of `Parameter`.
- lr: Using different learning rate by grouping parameters is currently not supported.
- lr: Using different learning rate by separating parameters is currently not supported.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
will be used. If not, the `weight_decay` in the API will be used.
- grad_centralization: Optional. Must be Boolean. If "grad_centralization" is in the keys, the set value
will be used. If not, the `grad_centralization` is False by default. This configuration only works on the
convolution layer.
- order_params: Optional. If "order_params" in the keys, the value must be the order of parameters and
the order will be followed in optimizer. There are no other keys in the `dict` and the parameters which
in the value of 'order_params' must be in one of group parameters.
- order_params: Optional. When parameters is grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
- grad_centralization: Optional. The data type of "grad_centralization" is Bool. If "grad_centralization"
is in the keys, the set value will be used. If not, the `grad_centralization` is False by default.
This parameter only works on the convolution layer.
initial_accum (float): The starting value for accumulators `m`, must be zero or positive values. Default: 0.1.
initial_accum (float): The starting value for accumulators, must be zero or positive values. Default: 0.1.
learning_rate (float): The learning rate value, must be zero or positive, dynamic learning rate is currently
not supported. Default: 0.001.
lr_power (float): Learning rate power controls how the learning rate decreases during training, must be less
@ -151,21 +177,15 @@ class FTRL(Optimizer):
`FixedLossScaleManager` is set to False, then this value needs to be the same as the `loss_scale` in
`FixedLossScaleManager`. Refer to class :class:`mindspore.FixedLossScaleManager` for more details.
Default: 1.0.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
weight_decay (Union[float, int]): Weight decay value to multiply weight, must be zero or positive value.
Default: 0.0.
Inputs:
- **grads** (tuple[Tensor]) - The gradients of `params` in the optimizer, the shape is the same as the `params`
in optimizer.
Outputs:
Tuple[Parameter], the updated parameters, the shape is the same as `params`.
tuple[Parameter], the updated parameters, the shape is the same as `params`.
Raises:
TypeError: If `initial_accum`, `learning_rate`, `lr_power`, `l1`, `l2` or `loss_scale` is not a float.
@ -180,8 +200,6 @@ class FTRL(Optimizer):
``Ascend`` ``GPU``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.FTRL(params=net.trainable_params())
@ -195,75 +213,97 @@ class FTRL(Optimizer):
>>> optim = nn.FTRL(group_params, learning_rate=0.1, weight_decay=0.0)
>>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01 and grad
>>> # centralization of True.
>>> # The no_conv_params's parameters will use default learning rate of 0.1 will use default weight decay
>>> # of 0.0 and grad centralization of False.
>>> # The no_conv_params's parameters will use default weight decay of 0.0 and grad centralization of False.
>>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'.
>>>
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
@opt_init_args_register
def __init__(self, params, initial_accum=0.1, learning_rate=0.001, lr_power=-0.5, l1=0.0, l2=0.0,
use_locking=False, loss_scale=1.0, weight_decay=0.0):
# 初始化FTRL类
super(FTRL, self).__init__(learning_rate, params, weight_decay, loss_scale=loss_scale)
# 检查参数
if self.dynamic_lr or self.is_group_lr:
raise ValueError(f"For 'FTRL', dynamic learning rate and group learning rate are currently not supported "
f"in FTRL, they should all be false, but got dynamic learning rate {self.dynamic_lr} and"
f" group learning rate {self.is_group_lr}.")
raise ValueError('Dynamic learning rate or group learning rate is currently not supported.')
_check_param(initial_accum, lr_power, l1, l2, use_locking, self.cls_name)
# 初始化参数
self.moments = self.parameters.clone(prefix="moments", init=initial_accum)
self.linear = self.parameters.clone(prefix="linear", init='zeros')
self.l1 = l1
self.l2 = l2
self.lr = learning_rate
self.lr_power = lr_power
# 判断是否是分组
if not self.is_group:
# 如果不是组，则将decay_flags设置为True
self.decay_flags = tuple((lambda: True)() for x in self.parameters)
# 初始化优化器
self.hyper_map = C.HyperMap()
self.opt = P.ApplyFtrl(use_locking=use_locking)
self.use_locking = use_locking
# 初始化sparse优化器
self.sparse_opt = P.SparseApplyFtrl(learning_rate, l1, l2, lr_power, use_locking=use_locking)
# 初始化sparse_opt
self._ps_pull = P.Pull()
# 初始化_ps_pull
self._ps_push = P.Push("Ftrl", [0, 1, 2])
# 初始化_ps_push
self._ps_push.add_prim_attr("init_accum", initial_accum)
# 向_ps_push中添加init_accum属性
self._ps_push.add_prim_attr("lr", learning_rate)
# 向_ps_push中添加lr属性
self._ps_push.add_prim_attr("l1", l1)
# 向_ps_push中添加l1属性
self._ps_push.add_prim_attr("l2", l2)
# 向_ps_push中添加l2属性
self._ps_push.add_prim_attr("lr_power", lr_power)
def construct(self, grads):
'''
构建FTRL优化器
:param grads: 梯度
:return: 更新后的参数
'''
params = self.parameters
moments = self.moments
linear = self.linear
# 对梯度权重衰减
grads = self.decay_weight(grads)
# 对梯度中心化
grads = self.gradients_centralization(grads)
# 对梯度进行缩放
grads = self.scale_grad(grads)
# 对梯度进行去重
grads = self._grad_sparse_indices_deduplicate(grads)
# 更新参数
lr = self.get_lr()
# 将_ftrl_opt函数放入参数
success = self.map_(F.partial(_ftrl_opt, self.opt, self.sparse_opt, self._ps_push, self._ps_pull,
self.l1, self.l2, self.lr_power, lr),
linear, grads, params, moments, self.ps_parameters, self.cache_enable)
self.l1, self.l2, self.lr_power, lr),
linear, grads, params, moments, self.ps_parameters, self.cache_enable)
return success
@Optimizer.target.setter
# 优化器target设置
def target(self, value):
"""
If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation.
"""
"""If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation."""
# 如果value不为str类，则抛出TypeError异常
if not isinstance(value, str):
raise TypeError("For 'FTRL', the property 'target' must be string type, "
"but got type {}.".format(type(value)))
raise TypeError("The value must be str type, but got value type is {}".format(type(value)))
# 如果value值不在三者之中，则抛出ValueError异常
if value not in ('CPU', 'Ascend', 'GPU'):
raise ValueError("For 'FTRL', the property 'target' must be 'CPU', 'Ascend' or 'GPU', "
"but got {}".format(value))
raise ValueError("The value must be 'CPU', 'Ascend' or 'GPU', but got value {}".format(value))
if value == 'CPU':
# 如果输入值为CPU，则使用FusedSparseFtrl优化器更新参数
self.sparse_opt = P.FusedSparseFtrl(self.lr, self.l1, self.l2, self.lr_power, self.use_locking)
self.sparse_opt.add_prim_attr("primitive_target", "CPU")
else:
# 如果输入值为GPU，则使用SparseApplyFtrl优化器更新参数
self.sparse_opt = P.SparseApplyFtrl(self.lr, self.l1, self.l2, self.lr_power, self.use_locking)
self._target = value
# 设置目标值
self._target = value

273
mindspore/python/mindspore/nn/optim/lamb.py
View File

@ -13,26 +13,29 @@
# limitations under the License.
# ============================================================================
"""lamb"""
# 优化器LAMB（Layer-wise Adaptive Moments optimizer for Batching training，用于批训练的分层自适应矩优化器）算法的实现。
import numpy as np
from mindspore import context
from mindspore.common import dtype as mstype
from mindspore.common.initializer import initializer
from mindspore.ops import operations as P
from mindspore.ops import composite as C
from mindspore.ops import functional as F
from mindspore.common.parameter import Parameter
from mindspore.common.tensor import Tensor
from mindspore._checkparam import Validator as validator
from mindspore._checkparam import Rel
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
from .. import layer
from.. import layer
num_one = Tensor(np.ones([1]), mstype.float32)
# 定义lamb_opt操作
_lamb_opt = C.MultitypeFuncGraph("lamb_opt")
@_lamb_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor",
@_lamb_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Number", "Tensor", "Tensor", "Tensor",
"Tensor", "Bool", "Bool")
def _update_run_op(beta1, beta2, eps, global_step, lr, weight_decay, param, m, v, gradient, decay_flag, optim_filter):
"""
@ -43,7 +46,7 @@ def _update_run_op(beta1, beta2, eps, global_step, lr, weight_decay, param, m, v
beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0).
eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0.
lr (Tensor): Learning rate.
weight_decay (numbers.Number): Weight decay. Should be equal to or greater than 0.
weight_decay (Number): Weight decay. Should be equal to or greater than 0.
global_step (Tensor): Global step.
param (Tensor): Parameters.
m (Tensor): m value of parameters.
@ -56,65 +59,99 @@ def _update_run_op(beta1, beta2, eps, global_step, lr, weight_decay, param, m, v
Tensor, the new value of v after updating.
"""
if optim_filter:
# 创建一个Mul操作
op_mul = P.Mul()
# 创建一个Sqrt操作
op_sqrt = P.Sqrt()
# 创建一个Rsqrt操作
op_rsqrt = P.Rsqrt()
# 创建一个Square操作
op_square = P.Square()
# 创建一个Cast操作
op_cast = P.Cast()
# 创建一个Reshape操作
op_reshape = P.Reshape()
# 创建一个Shape操作
op_shape = P.Shape()
# 创建一个Pow操作
op_pow = P.Pow()
# 创建一个Norm操作
op_norm = layer.Norm()
# 创建一个Select操作
op_select = P.Select()
# 创建一个Greater操作
op_greater = P.Greater()
# 创建一个Fill操作
op_fill = P.Fill()
# 创建一个DType操作
op_dtype = P.DType()
# 将param转换为float32类型
param_fp32 = op_cast(param, mstype.float32)
# 将m转换为float32类型
m_fp32 = op_cast(m, mstype.float32)
# 将v转换为float32类型
v_fp32 = op_cast(v, mstype.float32)
# 将gradient转换为float32类型
gradient_fp32 = op_cast(gradient, mstype.float32)
# 计算next_m
next_m = op_mul(beta1, m_fp32) + op_mul(op_cast(num_one, mstype.float32) - beta1, gradient_fp32)
# 计算next_v
next_v = op_mul(beta2, v_fp32) + op_mul(op_cast(num_one, mstype.float32) - beta2, op_square(gradient_fp32))
# 计算next_mm
next_mm = next_m / (op_cast(num_one, mstype.float32)
- op_pow(beta1, op_cast(global_step, mstype.float32)))
- op_pow(beta1, op_cast(global_step + num_one, mstype.float32)))
# 计算next_vv
next_vv = next_v / (op_cast(num_one, mstype.float32) -
op_pow(beta2, op_cast(global_step, mstype.float32)))
op_pow(beta2, op_cast(global_step + num_one, mstype.float32)))
# 计算w_norm
w_norm = op_norm(param_fp32)
# 计算g_norm
g_norm = op_norm(gradient_fp32)
# 计算g_norm_hat
g_norm_hat = op_norm(op_mul(next_mm, op_rsqrt(next_vv + eps)) + weight_decay * param_fp32)
# 创建一个zeros变量
zeros = F.zeros_like(w_norm)
# 创建一个ones变量
ones = op_fill(op_dtype(w_norm), op_shape(w_norm), 1.0)
# 计算trust_ratio
trust_ratio = op_select(
op_greater(w_norm, zeros),
op_select(op_greater(g_norm, zeros), w_norm / g_norm_hat, ones),
ones)
# 将trust_ratio转换为float32类型
tens = op_fill(op_dtype(trust_ratio), op_shape(trust_ratio), 10.0)
# 将trust_ratio限制在zeros和tens之间
trust_ratio = C.clip_by_value(trust_ratio, zeros, tens)
# 计算update
update = next_mm / (op_sqrt(next_vv) + eps)
if decay_flag:
# 将梯度乘以权重衰减
update = update + op_mul(weight_decay, param_fp32)
# 将梯度乘以学习率
update_with_lr = op_mul(op_mul(trust_ratio, lr), update)
# 将更新后的参数更新到参数列表中
next_param = param_fp32 - op_reshape(update_with_lr, op_shape(param_fp32))
# 将参数列表中的参数更新到参数列表中
next_param = F.depend(next_param, F.assign(param, op_cast(next_param, F.dtype(param))))
next_param = F.depend(next_param, F.assign(m, op_cast(next_m, F.dtype(m))))
next_param = F.depend(next_param, F.assign(v, op_cast(next_v, F.dtype(v))))
# 将参数列表中的参数更新到参数列表中
return op_cast(next_param, F.dtype(param))
return gradient
_lamb_opt_ascend = C.MultitypeFuncGraph("lamb_opt_ascend")
@_lamb_opt_ascend.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor",
@_lamb_opt_ascend.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Number", "Tensor", "Tensor", "Tensor",
"Tensor", "Bool", "Bool")
def _update_run_op_ascend(beta1, beta2, eps, global_step, lr, weight_decay, param, m, v, gradient, decay_flag,
optim_filter):
@ -126,7 +163,7 @@ def _update_run_op_ascend(beta1, beta2, eps, global_step, lr, weight_decay, para
beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0).
eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0.
lr (Tensor): Learning rate.
weight_decay (numbers.Number): Weight decay. Should be equal to or greater than 0.
weight_decay (Number): Weight decay. Should be equal to or greater than 0.
global_step (Tensor): Global step.
param (Tensor): Parameters.
m (Tensor): m value of parameters.
@ -139,158 +176,109 @@ def _update_run_op_ascend(beta1, beta2, eps, global_step, lr, weight_decay, para
Tensor, the new value of v after updating.
"""
if optim_filter:
# 定义一个Cast操作符
op_cast = P.Cast()
# 定义一个Norm操作符
op_norm = layer.Norm()
# 定义一个LambApplyOptimizerAssign操作符
op_lamb_apply_optimizer_assign = P.LambApplyOptimizerAssign()
# 定义一个LambApplyWeightAssign操作符
op_lamb_apply_weight_assign = P.LambApplyWeightAssign()
# 将参数转换为float32类型
param_fp32 = op_cast(param, mstype.float32)
# 将梯度转换为float32类型
gradient_fp32 = op_cast(gradient, mstype.float32)
new_global_step = op_cast(global_step, mstype.float32)
# 将全局步数转换为float32类型
new_global_step = op_cast(global_step + num_one, mstype.float32)
# 将权重衰减标志转换为float32类型
weight_decay_flag = op_cast(decay_flag, mstype.float32)
# 将梯度更新，使用LambApplyOptimizerAssign操作符
update, _, _ = op_lamb_apply_optimizer_assign(gradient_fp32, v, m, param_fp32,
beta1, 1.0 - beta1, beta2, 1.0 - beta2, eps,
new_global_step, weight_decay_flag, weight_decay)
beta1, 1.0 - beta1, beta2, 1.0 - beta2, eps,
new_global_step, weight_decay_flag, weight_decay)
# 将参数梯度转换为float32类型
w_norm = op_norm(param_fp32)
# 将梯度更新转换为float32类型
g_norm = op_norm(update)
# 将梯度更新，使用LambApplyWeightAssign操作符
update = F.depend(update, op_lamb_apply_weight_assign(w_norm, g_norm, lr, update, param))
# 返回梯度更新
return update
return gradient
def _check_param_value(beta1, beta2, eps, prim_name):
'''检查参数值的类型和范围'''
validator.check_value_type("beta1", beta1, [float], prim_name)
# 检查beta1的类型是否为float，是否在0.0和1.0之间，是否为INC_NEITHER，检查beta1是否在0.0和1.0之间
validator.check_value_type("beta2", beta2, [float], prim_name)
# 检查beta2的类型是否为float，是否在0.0和1.0之间，是否为INC_NEITHER，检查beta2是否在0.0和1.0之间
validator.check_value_type("eps", eps, [float], prim_name)
# 检查eps的类型是否为float，是否在0.0和1.0之间，是否为INC_NEITHER，检查eps是否在0.0和1.0之间
validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name)
# 检查beta1是否在0.0和1.0之间
validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name)
# 检查beta2是否在0.0和1.0之间
validator.check_positive_float(eps, "eps", prim_name)
class Lamb(Optimizer):
r"""
Implements the Lamb(Layer-wise Adaptive Moments optimizer for Batching training) algorithm.
# 优化器LAMB（Layer-wise Adaptive Moments optimizer for Batching training，用于批训练的分层自适应矩优化器）算法的实现。
"""
Lamb Dynamic Learning Rate.
LAMB is an optimization algorithm employing a layerwise adaptive large batch optimization technique.
Refer to the paper `LARGE BATCH OPTIMIZATION FOR DEEP LEARNING: TRAINING BERT IN 76
LAMB is an optimization algorithm employing a layerwise adaptive large batch
optimization technique. Refer to the paper `LARGE BATCH OPTIMIZATION FOR DEEP LEARNING: TRAINING BERT IN 76
MINUTES <https://arxiv.org/abs/1904.00962>`_.
The LAMB optimizer aims to increase the training batch size without reducing the accuracy,
and it supports adaptive element-by-element update and accurate layered correction.
The updating of parameters follows:
.. math::
\begin{array}{l}
&\newline
&\hline \\
&\textbf{Parameters}: \: 1^{\text {st }}\text {moment vector} \: m , \: 2^{\text {nd}} \:
\text{moment vector} \: v , \\
&\hspace{5mm}\text{learning rate } \left\{ \gamma_{t}\right\}_{t=1}^{T} , \: \text
{exponential decay rates for the moment estimates} \: \beta_{1} \: \beta_{2} , \\
&\hspace{5mm}\text{scaling function } \phi \\
&\textbf{Init}: \boldsymbol{m}_{0} \leftarrow 0, \: \boldsymbol{v}_{0} \leftarrow 0 \\[-1.ex]
&\newline
&\hline \\
&\textbf{for} \text { t=1 to T } \textbf{do} \\
&\hspace{5mm}\text{Draw b samples } S_{t} \text{ from } \mathbb{P} \text{ . } \\
&\hspace{5mm}\text{Compute } g_{t}=\frac{1}{\left|\mathcal{S}_{t}\right|} \sum_{s_{t} \in \mathcal{S}_{t}}
\nabla \ell\left(x_{t}, s_{t}\right) . \\
&\hspace{5mm}\boldsymbol{m}_{t} \leftarrow \beta_{1} \boldsymbol{m}_{t-1}+\left(1-\beta_{1}\right)
\boldsymbol{g}_{t} \\
&\hspace{5mm}\boldsymbol{v}_{t} \leftarrow \beta_{2} \boldsymbol{v}_{t-1}+\left(1-\beta_{2}\right)
\boldsymbol{g}_{t}^{2} \\
&\hspace{5mm}\hat{\boldsymbol{m}}_{t} \leftarrow \boldsymbol{m}_{t} /\left(1-\beta_{1}^{t}\right) \\
&\hspace{5mm}\hat{\boldsymbol{v}}_{t} \leftarrow \boldsymbol{v}_{t} /\left(1-\beta_{2}^{t}\right) \\
&\hspace{5mm}\text{Compute ratio } \boldsymbol{r}_{t}=\hat{\boldsymbol{m}}_{t}
/(\sqrt{\hat{\boldsymbol{v}}_{t}}+\epsilon) \\
&\hspace{5mm}\boldsymbol{w}_{t+1}^{(i)}=\boldsymbol{w}_{t}^{(i)}- \gamma_{t}
\frac{\boldsymbol{\phi}\left(\left\|\boldsymbol{w}_{t}^{(i)}\right\|\right)}
{\left\|\boldsymbol{w}_{t}^{(i)}+\lambda \boldsymbol{w}_{t}^{(i)}\right\|}\left(\boldsymbol{r}_{t}^{(i)}+
\lambda \boldsymbol{w}_{t}^{(i)}\right) \\
&\textbf{end for} \\[-1.ex]
&\newline
&\hline \\[-1.ex]
&\textbf{return} \: \boldsymbol{w}_{t+1}\\[-1.ex]
&\newline
&\hline \\[-1.ex]
\end{array}
:math:`m` represents the 1st moment vector `moment1`, :math:`v` represents the 2nd moment vector `moment2`,
:math:`g` represents `gradients`, :math:`\beta_1, \beta_2` represent `beta1` and `beta2`,
:math:`t` represents the current step while :math:`beta_1^t` and :math:`beta_2^t` represent
`beta1_power` and `beta2_power`, :math:`\gamma` represents `learning_rate`, :math:`w` represents `params`,
:math:`\epsilon` represents `eps`, :math:`\lambda` represents `weight_decay`.
Note:
There is usually no connection between a optimizer and mixed precision. But when `FixedLossScaleManager` is used
and `drop_overflow_update` in `FixedLossScaleManager` is set to False, optimizer needs to set the 'loss_scale'.
As this optimizer has no argument of `loss_scale`, so `loss_scale` needs to be processed by other means, refer
document `LossScale <https://www.mindspore.cn/docs/programming_guide/zh-CN/master/lossscale.html>`_ to process
`loss_scale` correctly.
When separating parameter groups, the weight decay in each group will be applied on the parameters if the
weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied
on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
When separating parameter groups, if you want to centralize the gradient, set grad_centralization to True,
but the gradient centralization can only be applied to the parameters of the convolution layer.
If the parameters of the non convolution layer are set to True, an error will be reported.
To improve parameter groups performance, the customized order of parameters can be supported.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", "grad_centralization" and
"order_params" are the keys can be parsed.
params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated,
the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params",
"lr", "weight_decay" and "order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- params: Required. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
If not, the `learning_rate` in the API will be used.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
will be used. If not, the `weight_decay` in the API will be used.
- grad_centralization: Optional. Must be Boolean. If "grad_centralization" is in the keys, the set value
will be used. If not, the `grad_centralization` is False by default. This configuration only works on the
convolution layer.
- order_params: Optional. If "order_params" in the keys, the value must be the order of parameters and
the order will be followed in optimizer. There are no other keys in the `dict` and the parameters which
in the value of 'order_params' must be in one of group parameters.
- order_params: Optional. When parameters is grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]):
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of current step.
- grad_centralization: Optional. The data type of "grad_centralization" is Bool. If "grad_centralization"
is in the keys, the set value will be used. If not, the `grad_centralization` is False by default.
This parameter only works on the convolution layer.
learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate.
When the learning_rate is an Iterable or a Tensor in a 1D dimension, use dynamic learning rate, then
the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule,
use dynamic learning rate, the i-th learning rate will be calculated during the process of training
according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero
dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be
equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float.
beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9.
Should be in range (0.0, 1.0).
beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999.
Should be in range (0.0, 1.0).
eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6.
Should be greater than 0.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
weight_decay (float): Weight decay (L2 penalty). Default: 0.0. Should be equal to or greater than 0.
Inputs:
- **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`.
@ -311,9 +299,6 @@ class Lamb(Optimizer):
``Ascend`` ``GPU``
Examples:
>>> from mindspore import nn, Model
>>> from mindspore.nn import learning_rate_schedule
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.Lamb(params=net.trainable_params(), learning_rate=0.1)
@ -336,45 +321,75 @@ class Lamb(Optimizer):
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
_support_parallel_optimizer = True
@opt_init_args_register
def __init__(self, params, learning_rate, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0):
# 初始化Lamb类
super(Lamb, self).__init__(learning_rate, params, weight_decay)
# 检查参数值
_check_param_value(beta1, beta2, eps, self.cls_name)
# turn them to scalar when me support scalar/tensor mix operations
# 将参数转换为tensor
self.beta1 = Tensor(np.array([beta1]).astype(np.float32))
self.beta2 = Tensor(np.array([beta2]).astype(np.float32))
self.eps = Tensor(np.array([eps]).astype(np.float32))
self.params = self.parameters
# 初始化moments1和moments2参数
self.moments1 = self.params.clone(prefix="lamb_m", init='zeros')
self.moments2 = self.params.clone(prefix="lamb_v", init='zeros')
# 如果不支持动态学习率，则将全局步骤设置为0
if not self.dynamic_lr:
self.global_step = Parameter(initializer(0, [1]), name='global_step')
self.assignadd = P.AssignAdd()
# 获取当前设备的排序
self.hyper_map = C.HyperMap()
# 获取当前上下文的device_target属性，并判断是否为Ascend
self.device_ascend = context.get_context("device_target") == "Ascend"
def construct(self, gradients):
weight_decay = self.get_weight_decay()
'''
构建优化器
:param gradients: 梯度
:return: 优化器
'''
lr = self.get_lr()
# 判断是否使用深度学习
lamb_opt = _lamb_opt_ascend if self.device_ascend else _lamb_opt
# 判断是否使用分组
gradients = self.gradients_centralization(gradients)
# 判断是否使用分组学习
if self.is_group:
# 判断是否使用分组学习率
if self.is_group_lr:
# 使用分组学习并使用分组学习率
optim_result = self.hyper_map(F.partial(lamb_opt, self.beta1, self.beta2, self.eps,
self.global_step),
lr, weight_decay, self.params, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
self.global_step),
lr, self.weight_decay, self.params, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
else:
# 使用分组学习普通学习率
optim_result = self.hyper_map(F.partial(lamb_opt, self.beta1, self.beta2, self.eps,
self.global_step, lr),
weight_decay, self.params, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
self.global_step, lr),
self.weight_decay, self.params, self.moments1, self.moments2,
gradients, self.decay_flags, self.optim_filter)
# 判断是否使用分组学习
else:
# 使用普通学习
optim_result = self.hyper_map(F.partial(lamb_opt, self.beta1, self.beta2, self.eps,
self.global_step, lr, weight_decay),
self.params, self.moments1, self.moments2, gradients,
self.decay_flags, self.optim_filter)
self.global_step, lr, self.weight_decay),
self.params, self.moments1, self.moments2, gradients,
self.decay_flags, self.optim_filter)
# 判断是否使用并行模式
if self.use_parallel:
# 输出结果
optim_result = F.depend(optim_result, self.broadcast_params(optim_result))
return optim_result
# 判断是否使用动态学习率
if not self.dynamic_lr:
# 输出结果
optim_result = F.depend(optim_result, self.assignadd(self.global_step, 1))
# 返回优化结果
return optim_result

67
mindspore/python/mindspore/nn/optim/lars.py
View File

@ -13,37 +13,59 @@
# limitations under the License.
# ============================================================================
"""lars optimizer"""
# LARS算法的实现。采用大量的优化技术
# 用于操作定义的模块
from mindspore.ops import operations as P
# 用于组合操作的模块
from mindspore.ops import composite as C
# 用于定义一些辅助函数的模块
from mindspore.ops import functional as F
# 用于验证参数的模块
from mindspore._checkparam import Validator as validator
# 用于表示张量，参数，数据类型
from mindspore.common import Tensor, Parameter, dtype as mstype
# 用于对梯度进行缩放，优化器的函数
from .optimizer import _grad_scale, Optimizer
# 用于注册优化器初始化时的参数的函数
from .optimizer import opt_init_args_register
# 创建一个名为lars_opt的多类型函数图。多类型函数图是一种用于表示多类型函数的图结构。
_lars_opt = C.MultitypeFuncGraph("lars_opt")
@_lars_opt.register("Function", "Number", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool")
def _tensor_run_opt(lars, loss_scale, learning_rate, weight_decay, gradient, weight, decay_flag, lars_flag):
# 用于对张量weight进行梯度下降更新。
"""Apply lars optimizer to the weight parameter."""
# 如果lars_flag为True，则使用LARS优化器
if lars_flag:
# 计算权重张量的平方和与梯度的平方和，并将结果传入P.SquareSumAll()对象op_reduce_sum。
op_reduce_sum = P.SquareSumAll()
w_square_sum, grad_square_sum = op_reduce_sum(weight, gradient)
# 计算学习率与权重衰减系数的乘积，并传入lars()函数。如果应用了权重衰减(decay_flag)，则将结果除以loss_scale。
if decay_flag:
# 应用权重衰减
grad_t = lars(weight, gradient, w_square_sum, grad_square_sum, weight_decay / loss_scale, learning_rate)
else:
# 初始化num_zero
num_zero = 0.0
# 不应用权重衰减
grad_t = lars(weight, gradient, w_square_sum, grad_square_sum, num_zero, learning_rate)
# 返回LARS后的gradient
return grad_t
# 否则，直接返回gradient
return gradient
def _check_param_value(optimizer, epsilon, coefficient, use_clip, prim_name):
# 用于检查优化器、epsilon、coefficient和use_clip的参数值是否符合要求。
# 检查optimizer的类型是否符合要求
validator.check_value_type("optimizer", optimizer, Optimizer, prim_name)
# 检查epsilon的类型是否符合要求
validator.check_value_type("epsilon", epsilon, [float], prim_name)
# 检查coefficient的类型是否符合要求
validator.check_value_type("coefficient", coefficient, [float], prim_name)
# 检查use_clip的类型是否符合要求
validator.check_value_type("use_clip", use_clip, [bool], prim_name)
@ -114,77 +136,120 @@ class LARS(Optimizer):
@opt_init_args_register
def __init__(self, optimizer, epsilon=1e-05, coefficient=0.001, use_clip=False,
lars_filter=lambda x: 'LayerNorm' not in x.name and 'bias' not in x.name):
# 调用父类Optimizer的初始化方法，传入0.0作为学习率，并创建一个虚拟参数变量fake_param。
super(LARS, self).__init__(0.0, [Parameter(Tensor(0.0), name="fake_param")])
# 检查参数值是否合法，包括optimizer、epsilon、coefficient、use_clip和lars_filter。
_check_param_value(optimizer, epsilon, coefficient, use_clip, self.cls_name)
# 将optimizer赋值给变量self.opt。
self.opt = optimizer
# 获取optimizer的动态衰减标志dynamic_decay_flags、动态权重衰减dynamic_weight_decay、权重衰减weight_decay、全局步数global_step和参数变量parameters。
self.dynamic_decay_flags = optimizer.dynamic_decay_flags
self.dynamic_weight_decay = optimizer.dynamic_weight_decay
self.weight_decay = optimizer.weight_decay
self.global_step = optimizer.global_step
self.parameters = optimizer.parameters
# 将parameters添加到self._user_parameters中。
self._user_parameters += [param.name for param in self.parameters]
# 判断use_clip是否为True，如果是，则将lars_filter函数应用到每个参数上，并将结果作为self.lars_flag。
self.use_clip = use_clip
self.lars_flag = tuple(lars_filter(x) for x in self.parameters)
# 判断optimizer是否为分组优化器，如果是，则将is_group设置为True。
self.is_group = optimizer.is_group
# 创建一个虚拟参数变量fake_lr，并将其赋值给self.learning_rate。
self.learning_rate = Parameter(Tensor(0.0, dtype=mstype.float32), name="fake_lr")
# 获取optimizer的衰减标志decay_flags、递归缩放reciprocal_scale和需要缩放need_scale。
self.decay_flags = optimizer.decay_flags
self.reciprocal_scale = optimizer.reciprocal_scale
# 如果need_scale为True，则将reciprocal_scale乘以loss_scale，并将结果赋值给self.scale。
self.need_scale = optimizer.need_scale
# 创建一个P.LARSUpdate对象self.lars，传入epsilon、coefficient和use_clip。
self.lars = P.LARSUpdate(epsilon, coefficient, use_clip)
# 创建一个P.Cast对象self.cast。
self.cast = P.Cast()
# 获取optimizer的损失缩放loss_scale。
self.loss_scale = optimizer.loss_scale
# 根据use_clip参数判断是否使用裁剪。
if use_clip:
# 使用裁剪，将优化器的is_group_lr、dynamic_lr和learning_rate属性赋值给self。
self.is_group_lr = optimizer.is_group_lr
self.dynamic_lr = optimizer.dynamic_lr
self.origin_learning_rate = optimizer.learning_rate
# 如果同时设置use_clip和is_group为True，则抛出异常。
if self.is_group_lr and self.dynamic_lr:
raise ValueError("For 'LARS', if the argument 'use_clip' is set to True, then the dynamic "
"learning rate and group learning rate cannot both be true.")
# 根据is_group参数判断是否使用分组学习率。
if self.is_group:
# 如果使用分组学习率，将优化器的dynamic_decay_flags设置为元组，其中所有元素都为False。
optimizer.dynamic_decay_flags = tuple(map(lambda x: False, self.dynamic_decay_flags))
else:
# 否则，将优化器的dynamic_decay_flags设置为False。
optimizer.dynamic_decay_flags = False
# 将优化器的decay_flags、dynamic_weight_decay和reciprocal_scale属性设置为False。
optimizer.decay_flags = tuple(map(lambda x: False, self.decay_flags))
optimizer.dynamic_weight_decay = False
optimizer.reciprocal_scale = 1.0
optimizer.exec_weight_decay = False
def _get_lr(self):
# 用于获取当前步骤的学习率
"""Get the learning rate of current step."""
# 从self中获取origin_learning_rate属性
lr = self.origin_learning_rate
# 判断是否使用动态学习率
if self.dynamic_lr:
# 如果使用动态学习率，根据is_group_lr参数判断是否为分组学习率
if self.is_group_lr:
# 遍历origin_learning_rate元组，计算每个分组的动态学习率，并将结果作为元组添加到lr中
lr = ()
for learning_rate in self.origin_learning_rate:
# 获取当前步数的动态学习率
current_dynamic_lr = learning_rate(self.global_step)
# 将动态学习率添加到列表中
lr += (current_dynamic_lr,)
# 否则，直接使用origin_learning_rate作为lr。最后返回lr
else:
lr = self.origin_learning_rate(self.global_step)
return lr
def construct(self, gradients):
# 构建LARS优化器的计算图
# 从self中获取参数属性
params = self.parameters
# 如果使用裁剪
if self.use_clip:
# 获取学习率并计算裁剪后的梯度
lr = self._get_lr()
else:
# 否则，直接使用learning_rate作为学习率。
lr = self.learning_rate
# 获取权重衰减
weight_decay = self.get_weight_decay()
# 根据need_scale属性判断是否需要缩放梯度
if self.need_scale:
# 使用hyper_map函数缩放梯度
gradients = self.hyper_map(F.partial(_grad_scale, self.reciprocal_scale), gradients)
# 是否使用分组
if self.is_group:
# 如果是分组学习率
if self.is_group_lr:
# 使用hyper_map函数结合_lars_opt函数进行分组学习率优化
gradients = self.hyper_map(F.partial(_lars_opt, self.lars, self.loss_scale), lr, weight_decay,
gradients, params, self.decay_flags, self.lars_flag)
else:
# 使用hyper_map函数结合_lars_opt函数进行分组优化
gradients = self.hyper_map(F.partial(_lars_opt, self.lars, self.loss_scale, lr), weight_decay,
gradients, params, self.decay_flags, self.lars_flag)
else:
# 否则，使用hyper_map函数结合_lars_opt函数进行普通学习率优化
gradients = self.hyper_map(F.partial(_lars_opt, self.lars, self.loss_scale, lr, weight_decay),
gradients, params, self.decay_flags, self.lars_flag)
# 运行优化
success = self.opt(gradients)
# 返回结果
return success

72
mindspore/python/mindspore/nn/optim/lazyadam.py
View File

@ -13,6 +13,11 @@
# limitations under the License.
# ============================================================================
"""lazy adam"""
# 首先，从MindSpore的common模块中导入了一些常用的类，如dtype（数据类型）、initializer（初始化器）、operations（操作）、composite（组合操作）等。
# 接下来，从MindSpore的common模块中导入了一些验证器（Validator）类，用于验证参数的合法性。
# 接着，定义了一个名为"opt_init_args_register"的函数，用于注册优化器的初始化参数。这个函数的主要作用是让优化器在初始化时传入一些参数，以便在训练过程中使用。
# 最后，定义了一些Adam优化器的内部方法，如更新参数、计算梯度等。这些方法在实现Adam优化器时会使用到。
# 本文件包含了一个名为Adam的优化器类，用于实现自适应学习率优化。这个库的主要目的是为了让用户在训练深度学习模型时方便地使用Adam优化器。
from mindspore.common import dtype as mstype
from mindspore.common.initializer import initializer
from mindspore.ops import operations as P
@ -25,6 +30,7 @@ from mindspore._checkparam import Rel
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
# 定义了一个名为_lazy_adam_opt的多类型函数图（MultitypeFuncGraph），用于实现Adam优化器。
_lazy_adam_opt = C.MultitypeFuncGraph("lazy_adam_opt")
@ -33,42 +39,61 @@ _lazy_adam_opt = C.MultitypeFuncGraph("lazy_adam_opt")
"Bool")
def _run_opt_with_sparse(opt, sparse_opt, push, pull, use_locking, use_nesterov, target, beta1_power, beta2_power,
beta1, beta2, eps, lr, gradient, params, m, v, ps_parameter, cache_enable):
# 用于对权重参数应用稀疏Adam优化器。
"""Apply sparse lazy adam optimizer to the weight parameter when the gradient is sparse."""
success = True
indices = gradient.indices
values = gradient.values
# 如果没有ps_parameter，且cache_enable为False
if ps_parameter and not cache_enable:
# 创建op_shape函数，用于获取参数的形状。
op_shape = P.Shape()
# 获取参数的形状，并将它们存储在一个元组（tuple）中。
shapes = (op_shape(params), op_shape(m), op_shape(v),
op_shape(beta1_power), op_shape(beta2_power), op_shape(lr), op_shape(beta1),
op_shape(beta2), op_shape(eps), op_shape(values), op_shape(indices))
# 将参数拉取到push函数中
success = F.depend(success, pull(push((beta1_power, beta2_power, lr, beta1, beta2,
eps, values, indices), shapes), params))
# 检查拉取操作是否成功，返回success
return success
# 检查目标参数是否为空（target为空表示没有要更新的参数）
if not target:
# 将success作为依赖项，应用稀疏Adam优化器（sparse_opt）到参数上。
success = F.depend(success, sparse_opt(params, m, v, beta1_power, beta2_power, lr, beta1, beta2,
eps, values, indices))
else:
# 定义一个用于从张量中获取指定索引的值的操作。
op_gather = P.Gather()
# 定义一个用于计算平方根的操作。
op_sqrt = P.Sqrt()
# 定义一个用于对张量中的指定索引的值进行加法操作的函数，并支持同步锁。
scatter_add = P.ScatterAdd(use_locking)
# 定义一个用于对张量中的指定索引的值进行更新操作的函数，并支持同步锁。
scatter_update = P.ScatterUpdate(use_locking)
# 使用op_gather()函数从m和v中获取指定索引的值，并将它们存储在m_slice和v_slice中。
m_slice = op_gather(m, indices, 0)
v_slice = op_gather(v, indices, 0)
# 1.计算下一个动量（next_m）和下一个方差（next_v），公式为：next_m = m_slice * beta1 + values * (1 - beta1)
# 2.和next_v = v_slice * beta2 + values * values * (1 - beta2)。
next_m = m_slice * beta1 + values * (1 - beta1)
next_v = v_slice * beta2 + values * values * (1 - beta2)
# 3.计算学习率（lr_t），公式为：lr_t = lr * op_sqrt(1 - beta2_power) / (1 - beta1_power)。
lr_t = lr * op_sqrt(1 - beta2_power) / (1 - beta1_power)
# 如果使用了Nesterov动量，计算动量平均值，并更新参数。
if use_nesterov:
m_temp = beta1 * next_m + values * (1 - beta1)
param_update = m_temp / (op_sqrt(next_v) + eps)
# 否则，直接更新参数。
else:
param_update = next_m / (op_sqrt(next_v) + eps)
# 使用P.ScatterAdd()、P.ScatterUpdate()等函数更新模型参数。
success = F.depend(success, scatter_add(params, indices, - lr_t * param_update))
success = F.depend(success, scatter_update(m, indices, next_m))
success = F.depend(success, scatter_update(v, indices, next_v))
@ -80,31 +105,50 @@ def _run_opt_with_sparse(opt, sparse_opt, push, pull, use_locking, use_nesterov,
"Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool")
def _run_opt_with_one_number(opt, sparse_opt, push, pull, use_locking, use_nesterov, target, beta1_power, beta2_power,
beta1, beta2, eps, lr, gradient, params, moment1, moment2, ps_parameter, cache_enable):
# 该函数用于对权重参数应用 lazy adam 优化器
# opt: 优化器，用于更新参数。sparse_opt: 稀疏优化器，用于更新稀疏参数。push: 用于将参数和梯度对推入队列的函数。pull: 用于从队列中取出参数和梯度的函数。
# use_locking: 是否使用锁标志。use_nesterov: 是否使用 Nesterov 动量。target: 目标值。beta1_power: 一阶矩估计的指数移动平均系数。beta2_power: 二阶矩估计的指数移动平均系数。
# beta1: 一阶矩估计的系数。beta2: 二阶矩估计的系数。eps: 动态学习率调整系数。lr: 学习率。gradient: 梯度。params: 参数。moment1: 一阶矩估计。moment2: 二阶矩估计。
# ps_parameter: 是否为稀疏参数。cache_enable: 是否启用缓存。
"""Apply lazy adam optimizer to the weight parameter using Tensor."""
success = True
# 检查ps_parameter是否为真且cache_enable是否为假
if ps_parameter and not cache_enable:
# 如果是，则使用P.Shape()函数获取参数的形状
op_shape = P.Shape()
# 并使用F.depend()函数将参数和梯度对推入队列。接下来，使用pull()函数从队列中取出参数和梯度
success = F.depend(success, pull(push((beta1_power, beta2_power, lr, beta1, beta2, eps, gradient),
(op_shape(params), op_shape(moment1), op_shape(moment2))), params))
else:
# 否则使用opt()函数直接更新参数
success = F.depend(success, opt(params, moment1, moment2, beta1_power, beta2_power, lr, beta1, beta2,
eps, gradient))
# 最后，返回success标志
return success
def _check_param_value(beta1, beta2, eps, weight_decay, prim_name):
# 检查输入参数的类型和范围
"""Check the type of inputs."""
# 检查beta1、beta2、eps和weight_decay的类型是否为float
validator.check_value_type("beta1", beta1, [float], prim_name)
validator.check_value_type("beta2", beta2, [float], prim_name)
validator.check_value_type("eps", eps, [float], prim_name)
validator.check_value_type("weight_dacay", weight_decay, [float], prim_name)
# 检查beta1的类型是否为float，是否在0.0和1.0之间，是否为Rel.INC_NEITHER，名称为beta1
validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name)
# 检查beta2的类型是否为float，是否在0.0和1.0之间，是否为Rel.INC_NEITHER，名称为beta2
validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name)
# 检查eps的类型是否为float，是否在0.0和1.0之间，是否为Rel.INC_NEITHER，名称为eps
validator.check_positive_float(eps, "eps", prim_name)
# 检查weight_decay的类型是否为float，是否在0.0和1.0之间，是否为Rel.INC_NEITHER，名称为weight_decay
validator.check_non_negative_float(weight_decay, "weight_decay", prim_name)
class LazyAdam(Optimizer):
# 定义了一个名为LazyAdam的类，继承自Optimizer。LazyAdam类实现了 Adaptive Moment Estimation (Adam) 算法，
# 该算法由 Adam: A Method for Stochastic Optimization <https://arxiv.org/abs/1412.6980>_ 论文提出。
# 当梯度为稀疏时，LazyAdam优化器将应用懒汉式 adam 算法进行更新。
r"""
Implements the Adaptive Moment Estimation (Adam) algorithm. The Adam algorithm is proposed
in `Adam: A Method for Stochastic Optimization <https://arxiv.org/abs/1412.6980>`_.
@ -254,57 +298,79 @@ class LazyAdam(Optimizer):
@opt_init_args_register
def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-8, use_locking=False,
use_nesterov=False, weight_decay=0.0, loss_scale=1.0):
# 初始化一些参数，并设置一些相关属性，如学习率、一阶矩估计和二阶矩估计的系数等。同时，代码还检查了输入参数的类型和范围，以确保它们在后续操作中正确使用。
super(LazyAdam, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 检查输入参数的类型和范围
_check_param_value(beta1, beta2, eps, weight_decay, self.cls_name)
validator.check_value_type("use_locking", use_locking, [bool], self.cls_name)
validator.check_value_type("use_nesterov", use_nesterov, [bool], self.cls_name)
# 初始化beta1、beta2（衰减速率）、eps（添加到分母中，以提高数值稳定性）和use_locking（是否对参数更新加锁保护）等参数，并将它们转换为Tensor类型
self.beta1 = Tensor(beta1, mstype.float32)
self.beta2 = Tensor(beta2, mstype.float32)
# 初始化beta1_power和beta2_power为Parameter类型，并设置初始值为1
self.beta1_power = Parameter(initializer(1, [1], mstype.float32), name="beta1_power")
self.beta2_power = Parameter(initializer(1, [1], mstype.float32), name="beta2_power")
self.eps = Tensor(eps, mstype.float32)
self.use_nesterov = use_nesterov
self.use_locking = use_locking
# 表示当前优化器正在使用设备（如GPU或CPU）
self._is_device = True
# 初始化moment1和moment2为Parameter类型，并设置初始值为0
self.moment1 = self.parameters.clone(prefix="moment1", init='zeros')
self.moment2 = self.parameters.clone(prefix="moment2", init='zeros')
# 定义opt和sparse_opt为P.Adam和P.FusedSparseLazyAdam类型，并设置相关属性，如use_locking和use_nesterov（是否使用NAG算法更新梯度）
self.opt = P.Adam(use_locking, use_nesterov)
self.sparse_opt = P.FusedSparseLazyAdam(use_locking, use_nesterov)
self.sparse_opt.add_prim_attr("primitive_target", "CPU")
# 将primitive_target添加到sparse_opt中
self._ps_pull = P.Pull()
# 创建一个Pull操作，拉取操作
self._ps_push = P.Push("Adam", [0, 1, 2])
self._ps_push.add_prim_attr("use_nesterov", use_nesterov)
def construct(self, gradients):
# 构建LazyAdam优化器的更新逻辑
# 对梯度进行权重衰减
gradients = self.decay_weight(gradients)
# 对梯度进行中心化
gradients = self.gradients_centralization(gradients)
# 对梯度进行缩放
gradients = self.scale_grad(gradients)
# 对梯度进行去重
gradients = self._grad_sparse_indices_deduplicate(gradients)
# 获取学习率
lr = self.get_lr()
# 更新一阶矩估计和二阶矩估计的指数移动平均系数。
# 将beta1_power乘以beta1
self.beta1_power = self.beta1_power * self.beta1
# 将beta2_power乘以beta2
self.beta2_power = self.beta2_power * self.beta2
# 如果使用分组学习率
if self.is_group_lr:
# 调用map_reverse方法，传入_lazy_adam_opt函数、梯度、参数、一阶矩估计、二阶矩估计、稀疏梯度索引去重工具、缓存启用标志等参数，进行分组学习率优化更新。
success = self.map_reverse(F.partial(_lazy_adam_opt, self.opt, self.sparse_opt, self._ps_push,
self._ps_pull, self.use_locking, self.use_nesterov, self._is_device,
self.beta1_power, self.beta2_power, self.beta1, self.beta2, self.eps),
lr, gradients, self.parameters, self.moment1, self.moment2, self.ps_parameters,
self.cache_enable)
else:
# 调用map_reverse方法，传入_lazy_adam_opt函数、梯度、参数、一阶矩估计、二阶矩估计、稀疏梯度索引去重工具、缓存启用标志等参数，进行普通更新。
success = self.map_reverse(F.partial(_lazy_adam_opt, self.opt, self.sparse_opt, self._ps_push,
self._ps_pull, self.use_locking, self.use_nesterov, self._is_device,
self.beta1_power, self.beta2_power, self.beta1, self.beta2, self.eps,
lr),
gradients, self.parameters, self.moment1, self.moment2, self.ps_parameters,
self.cache_enable)
# 返回更新成功与否的布尔值
return success
@Optimizer.target.setter
def target(self, value):
# 根据输入的value字符串来更新模型的训练目标。
"""
If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation.
"""
# 调用_set_base_target方法来设置模型的基本目标。
self._set_base_target(value)

36
mindspore/python/mindspore/nn/optim/momentum.py
View File

@ -13,32 +13,47 @@
# limitations under the License.
# ============================================================================
"""momentum"""
# 该文件的主要目的是为了实现动量优化方法，即使用动量（momentum）来加速梯度下降过程。动量优化方法是一种常用的优化方法，可以提高训练效率。
# 首先，从mindspore.ops模块中导入了一些常用的功能，如F（功能函数）、C（组合函数）、P（操作）等。
# 然后，从mindspore.common.parameter模块中导入了一个Parameter类，用于创建参数。接着，从mindspore.common.tensor模块中导入了一个Tensor类，用于表示张量。
from mindspore.ops import functional as F, composite as C, operations as P
from mindspore.common.parameter import Parameter
from mindspore.common.tensor import Tensor
# 接下来，定义了一个名为mstype的常量，用于表示MindSpore中的数据类型。然后，从mindspore._checkparam模块中导入了一个Validator类，用于验证参数。
import mindspore.common.dtype as mstype
from mindspore._checkparam import Validator
# 最后，从momentum.optimizer模块中导入了一个Optimizer类，用于实现动量优化方法。从momentum.optimizer模块中导入了一个名为opt_init_args_register的函数，用于注册优化器的关键参数。
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
# 定义了一个名为_momentum_opt的组合函数图（Composite Function Graph），用于实现动量优化方法
_momentum_opt = C.MultitypeFuncGraph("momentum_opt")
@_momentum_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool")
def _tensor_run_opt_ext(opt, momentum, learning_rate, gradient, weight, moment, ps_parameter, cache_enable):
# 应用动量优化方法到张量权重参数。momentum：动量系数，用于平滑梯度更新。moment：动量张量，用于存储 previous step 的梯度。
"""Apply momentum optimizer to the weight parameter using Tensor."""
# 首先判断ps_parameter和cache_enable是否为True
if ps_parameter and not cache_enable:
# 使用P.Shape()获取learning_rate、gradient和momentum张量的形状。
op_shape = P.Shape()
# 使用P.Pull()和P.Push()创建一个名为ApplyMomentum的操作，并将learning_rate、gradient和momentum作为输入,可以在后续的计算中使用，以确保输入参数的形状符合预期。
_ps_pull = P.Pull()
_ps_push = P.Push("ApplyMomentum", [])
shapes = (op_shape(learning_rate), op_shape(gradient), op_shape(momentum))
# 使用F.depend()函数将True作为依赖项，执行完动量优化方法后，返回优化后的权重张量作为计算结果。
success = F.depend(True, _ps_pull(_ps_push((learning_rate, gradient, momentum), shapes), weight))
else:
# 使用opt函数应用动量优化方法，并传入weight、moment、learning_rate、gradient和momentum作为参数
success = F.depend(True, opt(weight, moment, learning_rate, gradient, momentum))
# 返回优化后的权重张量
return success
class Momentum(Optimizer):
# 动量优化方法是一种在训练神经网络时常用的优化方法，它通过在每次迭代时存储上一迭代的梯度，并使用动量系数u来平滑梯度更新，从而提高训练速度。
r"""
Implements the Momentum algorithm.
@ -169,28 +184,45 @@ class Momentum(Optimizer):
"""
@opt_init_args_register
def __init__(self, params, learning_rate, momentum, weight_decay=0.0, loss_scale=1.0, use_nesterov=False):
# 初始化Momentum类，参数分别为：参数，学习率，梯度，权重衰减，损失函数缩放因子，是否使用Nesterov动量加速梯度下降
super(Momentum, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 检查momentum是否为float类型，且小于0
Validator.check_value_type("momentum", momentum, [float], self.cls_name)
if isinstance(momentum, float) and momentum < 0.0:
raise ValueError("For 'Momentum', the argument 'momentum' should be at least 0.0, "
"but got {}".format(momentum))
# 如果momentum为float类型且小于0，抛出异常
raise ValueError("For 'Momentum', the argument'momentum' should be at least 0.0, "
"but got {}".format(momentum))
# 创建参数momentum，用于存储上一迭代的梯度
self.momentum = Parameter(Tensor(momentum, mstype.float32), name="momentum")
# 将params和moments克隆出来，分别命名为self.params和self.moments
self.params = self.parameters
# 检查use_nesterov类型是否为bool，并克隆
self.use_nesterov = Validator.check_bool(use_nesterov)
self.moments = self.params.clone(prefix="moments", init='zeros')
# 代码创建一个P.ApplyMomentum操作实例，使用Nesterov动量
self.opt = P.ApplyMomentum(use_nesterov=self.use_nesterov)
def construct(self, gradients):
# 实现了动量优化方法。
# 首先对gradients进行以下操作
params = self.params
moments = self.moments
# 计算权重衰减后的梯度。
gradients = self.decay_weight(gradients)
# 对梯度进行 centralization 处理
gradients = self.gradients_centralization(gradients)
# 对梯度进行 scale 处理
gradients = self.scale_grad(gradients)
# 获取学习率
lr = self.get_lr()
# 如果使用了分组学习率
if self.is_group_lr:
# 用分组方法对lr、gradients、params、moments和ps_parameters进行处理，并调用_momentum_opt函数进行动量优化。
success = self.hyper_map_reverse(F.partial(_momentum_opt, self.opt, self.momentum),
lr, gradients, params, moments, self.ps_parameters, self.cache_enable)
else:
# 用普通方法对lr、gradients、params、moments和ps_parameters进行处理，并调用_momentum_opt函数进行动量优化。
success = self.hyper_map_reverse(F.partial(_momentum_opt, self.opt, self.momentum, lr),
gradients, params, moments, self.ps_parameters, self.cache_enable)
# 返回优化后的权重张量。
return success

404
mindspore/python/mindspore/nn/optim/optimizer.py
View File

File diff suppressed because it is too large Load Diff

523
mindspore/python/mindspore/nn/optim/proximal_ada_grad.py
View File

@ -1,234 +1,289 @@
# Copyright 2020 Huawei Technologies Co., Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
"""PROXIMAL_ADA_GRAD"""
from mindspore.ops import functional as F, composite as C, operations as P
from mindspore.common import Tensor
import mindspore.common.dtype as mstype
from mindspore._checkparam import Validator as validator
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
_proximal_ada_grad_opt = C.MultitypeFuncGraph("proximal_ada_grad_opt")
@_proximal_ada_grad_opt.register("Function", "Function", "Tensor", "Tensor", "Tensor", "RowTensor", "Tensor",
"Tensor")
def _tensor_run_opt_with_sparse(opt, sparse_opt, l1, l2, learning_rate, gradient, weight, accum):
"""Apply sparse proximal_ada_grad optimizer to the weight parameter."""
success = True
success = F.depend(success, sparse_opt(weight, accum, learning_rate, l1, l2, gradient.values, gradient.indices))
return success
@_proximal_ada_grad_opt.register("Function", "Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor")
def _tensor_run_opt(opt, sparse_opt, l1, l2, learning_rate, gradient, weight, accum):
"""Apply proximal_ada_grad optimizer to the weight parameter."""
success = True
success = F.depend(success, opt(weight, accum, learning_rate, l1, l2, gradient))
return success
def _check_param_value(accum, l1, l2, use_locking, prim_name=None):
"""Check inputs param."""
validator.check_value_type("accum", accum, [float], prim_name)
validator.check_value_type("l1", l1, [float], prim_name)
validator.check_value_type("l2", l2, [float], prim_name)
validator.check_value_type("use_locking", use_locking, [bool], prim_name)
validator.check_non_negative_float(accum, "accum", prim_name)
validator.check_non_negative_float(l1, "l1", prim_name)
validator.check_non_negative_float(l2, "l2", prim_name)
class ProximalAdagrad(Optimizer):
r"""
Implements the ProximalAdagrad algorithm.
ProximalAdagrad is an online Learning and Stochastic Optimization.
Refer to paper `Efficient Learning using Forward-Backward Splitting
<http://papers.nips.cc//paper/3793-efficient-learning-using-forward-backward-splitting.pdf>`_.
.. math::
accum_{t+1} = accum_{t} + g * g
.. math::
\text{prox_v} = w_{t} - \gamma * g * \frac{1}{\sqrt{accum_{t+1}}}
.. math::
w_{t+1} = \frac{sign(\text{prox_v})}{1 + \gamma * l2} * \max(\left| \text{prox_v} \right| - \gamma * l1, 0)
Here : where :math:`g` , :math:`\gamma`, :math:`w` , :math:`accum` and :math:`t` denote the `grads`,
`learning_rate`, `params`, accumulation and current step respectively.
Note:
The sparse strategy is applied while the SparseGatherV2 operator is used for forward network. If the sparse
strategy wants to be executed on the host, set the target to the CPU.
The sparse feature is under continuous development.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", "grad_centralization" and
"order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
- grad_centralization: Optional. Must be Boolean. If "grad_centralization" is in the keys, the set value
will be used. If not, the `grad_centralization` is False by default. This configuration only works on the
convolution layer.
- order_params: Optional. When parameters are grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
accum (float): The starting value for accumulators `accum`, must be zero or positive values. Default: 0.1.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 0.001.
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of the current step.
l1 (float): l1 regularization strength, must be greater than or equal to zero. Default: 0.0.
l2 (float): l2 regularization strength, must be greater than or equal to zero. Default: 0.0.
use_locking (bool): If true, use locks for updating operation. Default: False.
loss_scale (float): Value for the loss scale. It must be greater than 0.0. In general, use the default value.
Only when `FixedLossScaleManager` is used for training and the `drop_overflow_update` in
`FixedLossScaleManager` is set to False, then this value needs to be the same as the `loss_scale` in
`FixedLossScaleManager`. Refer to class :class:`mindspore.FixedLossScaleManager` for more details.
Default: 1.0.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
Inputs:
- **grads** (tuple[Tensor]) - The gradients of `params` in the optimizer, the shape is the same as the `params`
in optimizer.
Outputs:
Tensor[bool], the value is True.
Raises:
TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule.
TypeError: If element of `parameters` is neither Parameter nor dict.
TypeError: If `accum`, `l1`, `l2` or `loss_scale` is not a float.
TypeError: If `weight_decay` is neither float nor int.
ValueError: If `loss_scale` is less than or equal to 0.
ValueError: If `accum`, `l1`, `l2` or `weight_decay` is less than 0.
Supported Platforms:
``Ascend``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.ProximalAdagrad(params=net.trainable_params())
>>>
>>> #2) Use parameter groups and set different values
>>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params()))
>>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params()))
>>> group_params = [{'params': conv_params, 'weight_decay': 0.01, 'grad_centralization':True},
... {'params': no_conv_params, 'lr': 0.01},
... {'order_params': net.trainable_params()}]
>>> optim = nn.ProximalAdagrad(group_params, learning_rate=0.1, weight_decay=0.0)
>>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01 and grad
>>> # centralization of True.
>>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0 and grad
>>> # centralization of False.
>>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'.
>>>
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
@opt_init_args_register
def __init__(self, params, accum=0.1, learning_rate=0.001, l1=0.0, l2=0.0,
use_locking=False, loss_scale=1.0, weight_decay=0.0):
super(ProximalAdagrad, self).__init__(learning_rate, params, weight_decay, loss_scale)
_check_param_value(accum, l1, l2, use_locking, self.cls_name)
self.accum = self.parameters.clone(prefix="accum", init=accum)
self.l1 = Tensor(l1, mstype.float32)
self.l2 = Tensor(l2, mstype.float32)
self.use_locking = use_locking
self.opt = P.ApplyProximalAdagrad(use_locking=use_locking)
self.sparse_opt = P.SparseApplyProximalAdagrad(use_locking=use_locking)
def construct(self, grads):
params = self.parameters
accum = self.accum
grads = self.decay_weight(grads)
grads = self.gradients_centralization(grads)
grads = self.scale_grad(grads)
grads = self._grad_sparse_indices_deduplicate(grads)
lr = self.get_lr()
if self.is_group_lr:
success = self.map_reverse(F.partial(_proximal_ada_grad_opt, self.opt, self.sparse_opt, self.l1, self.l2),
lr, grads, params, accum)
else:
success = self.map_reverse(F.partial(_proximal_ada_grad_opt, self.opt, self.sparse_opt, self.l1, self.l2,
lr),
grads, params, accum)
return success
@Optimizer.target.setter
def target(self, value):
"""
If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation.
"""
if not isinstance(value, str):
raise TypeError("For 'ProximalAdagrad', the property 'target' must be string type, "
"but got {}".format(type(value)))
if value not in ('CPU', 'Ascend', 'GPU'):
raise ValueError("For 'ProximalAdagrad', the property 'target' must be 'CPU', 'Ascend' or 'GPU', "
"but got {}.".format(value))
if value == 'CPU':
self.sparse_opt = P.FusedSparseProximalAdagrad(self.use_locking).add_prim_attr("primitive_target", "CPU")
else:
self.sparse_opt = P.SparseApplyProximalAdagrad(self.use_locking)
self._target = value
# Copyright 2020 Huawei Technologies Co., Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
"""PROXIMAL_ADA_GRAD"""
# 从mindspore.ops模块中导入了functional、composite和operations等类，这些类主要用于实现各种计算图操作。
from mindspore.ops import functional as F, composite as C, operations as P
# 从mindspore.common模块中导入了Tensor和mstype类，这些类主要用于处理张量数据和数据类型。
from mindspore.common import Tensor
import mindspore.common.dtype as mstype
# 从mindspore._checkparam模块中导入了Validator类，这个类主要用于验证参数是否符合预期的值。
from mindspore._checkparam import Validator as validator
# 从当前文件夹optimizer.py中导入了Optimizer类和opt_init_args_register函数，这些类和函数主要用于实现优化器的基本功能。
from .optimizer import Optimizer
from .optimizer import opt_init_args_register
# 定义了一个名为proximal_ada_grad_opt的多类型计算图函数
_proximal_ada_grad_opt = C.MultitypeFuncGraph("proximal_ada_grad_opt")
@_proximal_ada_grad_opt.register("Function", "Function", "Tensor", "Tensor", "Tensor", "RowTensor", "Tensor",
"Tensor")
def _tensor_run_opt_with_sparse(opt, sparse_opt, l1, l2, learning_rate, gradient, weight, accum):
"""Apply sparse proximal_ada_grad optimizer to the weight parameter."""
# 使用sparse_opt对weight进行更新，并将更新后的结果存储在accum中。同时，返回一个布尔值，表示更新是否成功。
"""
opt: 优化器实例。
sparse_opt: 稀疏优化器实例。
l1: L1 正则化参数。
l2: L2 正则化参数。
learning_rate: 学习率。
gradient: 损失函数的梯度。
weight: 需要更新的权重参数。
accum: 累积变量，用于存储上一个迭代的权重更新。
"""
success = True
# 使用sparse_opt对weight进行更新，并将更新后的结果存储在accum中
success = F.depend(success, sparse_opt(weight, accum, learning_rate, l1, l2, gradient.values, gradient.indices))
# 返回一个布尔值，表示更新是否成功
return success
@_proximal_ada_grad_opt.register("Function", "Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor")
def _tensor_run_opt(opt, sparse_opt, l1, l2, learning_rate, gradient, weight, accum):
"""Apply proximal_ada_grad optimizer to the weight parameter."""
# 使用opt对weight进行更新，并将更新后的结果存储在accum中。同时，返回一个布尔值，表示更新是否成功。
success = True
success = F.depend(success, opt(weight, accum, learning_rate, l1, l2, gradient))
return success
def _check_param_value(accum, l1, l2, use_locking, prim_name=None):
# 检查accum、l1、l2和use_locking的值是否符合预期
"""Check inputs param."""
# 检查accum参数的类型是否为float，如果不是，抛出异常
validator.check_value_type("accum", accum, [float], prim_name)
# 检查l1参数的类型是否为float，如果不是，抛出异常
validator.check_value_type("l1", l1, [float], prim_name)
# 检查l2参数的类型是否为float，如果不是，抛出异常
validator.check_value_type("l2", l2, [float], prim_name)
# 检查use_locking参数的类型是否为bool，如果不是，抛出异常
validator.check_value_type("use_locking", use_locking, [bool], prim_name)
# 检查accum参数的值是否小于0，如果不小于0，抛出异常
validator.check_non_negative_float(accum, "accum", prim_name)
# 检查l1参数的值是否小于0，如果不小于0，抛出异常
validator.check_non_negative_float(l1, "l1", prim_name)
# 检查l2参数的值是否小于0，如果不小于0，抛出异常
validator.check_non_negative_float(l2, "l2", prim_name)
class ProximalAdagrad(Optimizer):
# 继承自Optimizer类。ProximalAdagrad是一种在线学习率和优化算法，用于解决大规模数据集的学习问题。
# 首先，计算累积梯度的更新公式。然后，计算 proximal_v 公式，最后，计算更新后的权重向量 w_{t+1} 的公式
# 其中，:math:g , :math:\gamma, :math:w , :math:accum and :math:t 分别表示梯度、学习率、参数、累积梯度和当前步数。
r"""
Implements the ProximalAdagrad algorithm.
ProximalAdagrad is an online Learning and Stochastic Optimization.
Refer to paper `Efficient Learning using Forward-Backward Splitting
<http://papers.nips.cc//paper/3793-efficient-learning-using-forward-backward-splitting.pdf>`_.
.. math::
accum_{t+1} = accum_{t} + g * g
.. math::
\text{prox_v} = w_{t} - \gamma * g * \frac{1}{\sqrt{accum_{t+1}}}
.. math::
w_{t+1} = \frac{sign(\text{prox_v})}{1 + \gamma * l2} * \max(\left| \text{prox_v} \right| - \gamma * l1, 0)
Here : where :math:`g` , :math:`\gamma`, :math:`w` , :math:`accum` and :math:`t` denote the `grads`,
`learning_rate`, `params`, accumulation and current step respectively.
Note:
The sparse strategy is applied while the SparseGatherV2 operator is used for forward network. If the sparse
strategy wants to be executed on the host, set the target to the CPU.
The sparse feature is under continuous development.
If parameters are not grouped, the `weight_decay` in optimizer will be applied on the network parameters without
'beta' or 'gamma' in their names. Users can group parameters to change the strategy of decaying weight. When
parameters are grouped, each group can set `weight_decay`, if not, the `weight_decay` in optimizer will be
applied.
Args:
params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the
`params` is a list of `dict`, the string "params", "lr", "weight_decay", "grad_centralization" and
"order_params" are the keys can be parsed.
- params: Required. Parameters in current group. The value must be a list of `Parameter`.
- lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used.
If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported.
- weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay
will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight
decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic
weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only
with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule
to get the weight decay value of current step.
- grad_centralization: Optional. Must be Boolean. If "grad_centralization" is in the keys, the set value
will be used. If not, the `grad_centralization` is False by default. This configuration only works on the
convolution layer.
- order_params: Optional. When parameters are grouped, this usually is used to maintain the order of
parameters that appeared in the network to improve performance. The value should be parameters whose
order will be followed in optimizer.
If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in
one group of `params`.
accum (float): The starting value for accumulators `accum`, must be zero or positive values. Default: 0.1.
learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 0.001.
- float: The fixed learning rate value. Must be equal to or greater than 0.
- int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float.
- Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied.
For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate.
- Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate.
- LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of
LearningRateSchedule with step as the input to get the learning rate of the current step.
l1 (float): l1 regularization strength, must be greater than or equal to zero. Default: 0.0.
l2 (float): l2 regularization strength, must be greater than or equal to zero. Default: 0.0.
use_locking (bool): If true, use locks for updating operation. Default: False.
loss_scale (float): Value for the loss scale. It must be greater than 0.0. In general, use the default value.
Only when `FixedLossScaleManager` is used for training and the `drop_overflow_update` in
`FixedLossScaleManager` is set to False, then this value needs to be the same as the `loss_scale` in
`FixedLossScaleManager`. Refer to class :class:`mindspore.FixedLossScaleManager` for more details.
Default: 1.0.
weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0.
- float: The fixed weight decay value. Must be equal to or greater than 0.
- int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float.
- Cell: Weight decay is dynamic. During training, the optimizer calls the instance of
the Cell with step as the input to get the weight decay value of current step.
Inputs:
- **grads** (tuple[Tensor]) - The gradients of `params` in the optimizer, the shape is the same as the `params`
in optimizer.
Outputs:
Tensor[bool], the value is True.
Raises:
TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule.
TypeError: If element of `parameters` is neither Parameter nor dict.
TypeError: If `accum`, `l1`, `l2` or `loss_scale` is not a float.
TypeError: If `weight_decay` is neither float nor int.
ValueError: If `loss_scale` is less than or equal to 0.
ValueError: If `accum`, `l1`, `l2` or `weight_decay` is less than 0.
Supported Platforms:
``Ascend``
Examples:
>>> from mindspore import nn, Model
>>>
>>> net = Net()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.ProximalAdagrad(params=net.trainable_params())
>>>
>>> #2) Use parameter groups and set different values
>>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params()))
>>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params()))
>>> group_params = [{'params': conv_params, 'weight_decay': 0.01, 'grad_centralization':True},
... {'params': no_conv_params, 'lr': 0.01},
... {'order_params': net.trainable_params()}]
>>> optim = nn.ProximalAdagrad(group_params, learning_rate=0.1, weight_decay=0.0)
>>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01 and grad
>>> # centralization of True.
>>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0 and grad
>>> # centralization of False.
>>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'.
>>>
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = Model(net, loss_fn=loss, optimizer=optim)
"""
@opt_init_args_register
def __init__(self, params, accum=0.1, learning_rate=0.001, l1=0.0, l2=0.0,
use_locking=False, loss_scale=1.0, weight_decay=0.0):
# 调用父类__init__方法，传入learning_rate、params、weight_decay和loss_scale
super(ProximalAdagrad, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 使用_check_param_value函数检查accum、l1和l2的值是否符合预期
_check_param_value(accum, l1, l2, use_locking, self.cls_name)
# 创建一个名为accum的变量，并将其克隆赋值给self.accum。
self.accum = self.parameters.clone(prefix="accum", init=accum)
# 创建一个名为l1的张量，并将其赋值给self.l1
self.l1 = Tensor(l1, mstype.float32)
# 创建一个名为l2的张量，并将其赋值给self.l2
self.l2 = Tensor(l2, mstype.float32)
# 传入更新操作使用锁保护，默认False
self.use_locking = use_locking
# 创建一个名为opt的计算图函数，用于应用 proximal_ada_grad 优化器
self.opt = P.ApplyProximalAdagrad(use_locking=use_locking)
# 创建一个名为sparse_opt的计算图函数，用于应用稀疏 proximal_ada_grad 优化器
self.sparse_opt = P.SparseApplyProximalAdagrad(use_locking=use_locking)
def construct(self, grads):
# 用于计算更新权重参数
# 定义参数，获取params和accum变量
params = self.parameters
# 定义求导累加器
accum = self.accum
# 对grads进行去重处理，并计算权重衰减后的梯度。
grads = self.decay_weight(grads)
# 对grads进行 centralization 处理
grads = self.gradients_centralization(grads)
# 对grads进行 scale 处理。
grads = self.scale_grad(grads)
# 对grads进行稀疏索引处理
grads = self._grad_sparse_indices_deduplicate(grads)
# 获取学习率
lr = self.get_lr()
# 如果是分组学习率
if self.is_group_lr:
# 调用map_reverse函数，用分组方法计算梯度
success = self.map_reverse(F.partial(_proximal_ada_grad_opt, self.opt, self.sparse_opt, self.l1, self.l2),
lr, grads, params, accum)
else:
# 调用map_reverse函数，用普通方法计算梯度
success = self.map_reverse(F.partial(_proximal_ada_grad_opt, self.opt, self.sparse_opt, self.l1, self.l2,
lr),
grads, params, accum)
# 返回成功结果
return success
@Optimizer.target.setter
def target(self, value):
# 用于设置ProximalAdagrad类的target属性
"""
If the input value is set to "CPU", the parameters will be updated on the host using the Fused
optimizer operation.
"""
# 检查target属性的类型是否为字符串。如果不是字符串类型，将引发TypeError异常。
if not isinstance(value, str):
raise TypeError("For 'ProximalAdagrad', the property 'target' must be string type, "
"but got {}".format(type(value)))
# 如果target属性设置为非"CPU"、"Ascend"或"GPU"的值，将引发错误。
if value not in ('CPU', 'Ascend', 'GPU'):
raise ValueError("For 'ProximalAdagrad', the property 'target' must be 'CPU', 'Ascend' or 'GPU', "
"but got {}.".format(value))
if value == 'CPU':
# 如果target属性设置为CPU，则使用FusedSparseProximalAdagrad优化器进行更新
self.sparse_opt = P.FusedSparseProximalAdagrad(self.use_locking).add_prim_attr("primitive_target", "CPU")
else:
# 如果target属性设置为GPU，则使用SparseApplyProximalAdagrad稀疏优化器进行更新
self.sparse_opt = P.SparseApplyProximalAdagrad(self.use_locking)
# 将输入值设置为target
self._target = value

88
mindspore/python/mindspore/nn/optim/rmsprop.py
View File

@ -13,33 +13,73 @@
# limitations under the License.
# ============================================================================
"""rmsprop"""
# 实现RMSProp优化器，均方根传播（RMSProp）算法的实现。
# 1.从mindspore.ops模块中导入所需的函数，例如functional、composite和operations。
from mindspore.ops import functional as F, composite as C, operations as P
# 2.从mindspore._checkparam模块中导入Validator类。
from mindspore._checkparam import Validator as validator
# 3.定义一个名为Optimizer的类，用于封装优化器的实现。
from .optimizer import Optimizer
# 4.定义一个名为opt_init_args_register的函数，用于注册优化器的关键参数。
from .optimizer import opt_init_args_register
# 定义了两个名为_rmsprop_opt和_centered_rmsprop_opt的MultitypeFuncGraph对象。MultitypeFuncGraph是MindSpore中的一个用于封装多类型函数图的类。
_rmsprop_opt = C.MultitypeFuncGraph("rmsprop_opt")
_centered_rmsprop_opt = C.MultitypeFuncGraph("rmsprop_opt")
# 这两个函数图可以用于实现RMSProp和centered RMSProp优化器。
@_rmsprop_opt.register("Function", "Number", "Number", "Number", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor")
def _rmsprop_opt_(opt, decay, epsilon, momentum, learning_rate, weight, ms, mom, grad):
# 用于应用RMSProp优化器到权重参数上
"""Apply rmsprop optimizer to the weight parameter using dynamic learning rate."""
"""
opt：优化器实例，例如_rmsprop_opt或_centered_rmsprop_opt。
decay：一阶矩衰减系数。
epsilon：用于防止除以零的常量。
momentum：一阶矩系数。
learning_rate：学习率。
weight：权重参数。
ms：一阶矩参数。
mom：二阶矩参数。
grad：梯度参数。
"""
# 将success设置为True
success = True
# 调用opt函数，将weight、ms、mom、learning_rate、grad、decay、momentum和epsilon作为参数传入。将优化后的权重参数返回，并将success设置为优化成功。
success = F.depend(success, opt(weight, ms, mom, learning_rate, grad, decay, momentum, epsilon))
# 返回success
return success
@_centered_rmsprop_opt.register("Function", "Number", "Number", "Number", "Tensor", "Tensor", "Tensor", "Tensor",
"Tensor", "Tensor")
def _centered_rmsprop_opt_(opt, decay, epsilon, momentum, learning_rate, weight, mg, ms, mom, grad):
# 函数用于应用centered RMSProp优化器到权重参数上
"""Apply centered rmsprop optimizer to the weight parameter using dynamic learning rate."""
"""
opt：优化器实例，例如_rmsprop_opt或_centered_rmsprop_opt。
decay：一阶矩衰减系数。
epsilon：用于防止除以零的常量。
momentum：一阶矩系数。
learning_rate：学习率。
weight：权重参数。
mg：一阶矩参数。
ms：二阶矩参数。
mom：二阶矩参数。
grad：梯度参数。
"""
success = True
# # 调用opt函数，将weight、ms、mom、learning_rate、grad、decay、momentum和epsilon作为参数传入。将优化后的权重参数返回，并将success设置为优化成功。
success = F.depend(success, opt(weight, mg, ms, mom, grad, learning_rate, decay, momentum, epsilon))
# 如果成功，则将success的值传入opt函数，并将success的值设置为opt函数的返回值
return success
class RMSProp(Optimizer):
# 定义了一个名为RMSProp的类，继承自Optimizer类。主要实现了RMSProp优化器的基本算法
# 1.计算一阶矩ms和二阶矩mg的更新公式
# 2.计算权重更新w的更新公式
# 3.更新权重参数
"""
Implements Root Mean Squared Propagation (RMSProp) algorithm.
@ -198,51 +238,97 @@ class RMSProp(Optimizer):
@opt_init_args_register
def __init__(self, params, learning_rate=0.1, decay=0.9, momentum=0.0, epsilon=1e-10,
use_locking=False, centered=False, loss_scale=1.0, weight_decay=0.0):
'''
初始化RMSProp算法
:param params: 权重参数列表
:param learning_rate: 学习率
:param decay: 一阶矩衰减系数
:param momentum: 一阶矩系数
:param epsilon: 用于防止除以零的常量
:param use_locking: 是否使用锁定
:param centered: 如果为True，则梯度将通过梯度的估计方差进行归一
:param loss_scale: 损失缩放因子
:param weight_decay: 权重衰减系数
:return:
'''
super(RMSProp, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 验证参数类型和范围
# 检查decay的类型
validator.check_value_type("decay", decay, [float], self.cls_name)
# 检查decay的最小值
validator.check_non_negative_float(decay, "decay", self.cls_name)
# 检查momentum的类型
validator.check_value_type("momentum", momentum, [float], self.cls_name)
# 检查momentum的最小值
validator.check_non_negative_float(momentum, "momentum", self.cls_name)
# 检查epsilon的类型
validator.check_value_type("epsilon", epsilon, [float], self.cls_name)
# 检查epsilon的最小值
validator.check_positive_float(epsilon, "epsilon", self.cls_name)
# 检查use_locking的类型
validator.check_value_type("use_locking", use_locking, [bool], self.cls_name)
# 检查centered的类型
validator.check_value_type("centered", centered, [bool], self.cls_name)
self.centered = centered
# 如果centered为真，则使用ApplyCenteredRMSProp算法
if centered:
self.opt = P.ApplyCenteredRMSProp(use_locking)
# 创建偏差参数
self.mg = self.parameters.clone(prefix="mean_grad", init='zeros')
else:
# 否则使用ApplyRMSProp算法
self.opt = P.ApplyRMSProp(use_locking)
# 创建优化系数
self.momentum = momentum
# 创建平方梯度参数
self.ms = self.parameters.clone(prefix="mean_square", init='ones')
# 创建优化系数
self.moment = self.parameters.clone(prefix="moment", init='zeros')
# 创建正则化系数
self.epsilon = epsilon
# 创建抑制系数
self.decay = decayilon
self.decay = decay
def construct(self, gradients):
'''
构建优化器
:param gradients: 梯度
:return: 是否成功
'''
params = self.parameters
# 首先对梯度进行去噪处理（decay_weight权重衰减）
gradients = self.decay_weight(gradients)
# 对梯度进行中心化
gradients = self.gradients_centralization(gradients)
# 对梯度进行缩放
gradients = self.scale_grad(gradients)
# 获取学习率
lr = self.get_lr()
# 判断应用算法，若为TRUE，则梯度将通过梯度的估计方差进行归一。否则，应用一般的RMSProp算法
if self.centered:
# 如果是分组学习率，则用分组方法使用_centered_rmsprop_opt函数
if self.is_group_lr:
success = self.hyper_map_reverse(F.partial(_centered_rmsprop_opt, self.opt, self.decay, self.epsilon,
self.momentum),
lr, params, self.mg, self.ms, self.moment, gradients)
else:
# 否则，用普通方法使用_centered_rmsprop_opt函数
success = self.hyper_map_reverse(F.partial(_centered_rmsprop_opt, self.opt, self.decay, self.epsilon,
self.momentum, lr),
params, self.mg, self.ms, self.moment, gradients)
else:
# 不为TRUE，则用分组方法使用_rmsprop_opt函数
if self.is_group_lr:
success = self.hyper_map_reverse(F.partial(_rmsprop_opt, self.opt, self.decay, self.epsilon,
self.momentum),
lr, params, self.ms, self.moment, gradients)
else:
# 否则，用普通方法使用_rmsprop_opt函数
success = self.hyper_map_reverse(F.partial(_rmsprop_opt, self.opt, self.decay, self.epsilon,
self.momentum, lr),
params, self.ms, self.moment, gradients)
# 返回成功消息
return success

66
mindspore/python/mindspore/nn/optim/rprop.py
View File

@ -13,6 +13,10 @@
# limitations under the License.
# ============================================================================
"""rprop"""
# 定义了一个名为rprop的模块。rprop是反向传播优化算法的一种，主要用于处理非线性问题
# 1.从mindspore库中导入所需的库和模块，例如ops、P、mstype等
# 2.定义一个名为rprop的类，继承自Optimizer类
# 3.注册一个名为opt_init_args_register的函数，用于初始化rprop优化器的参数
from mindspore import ops
from mindspore.ops import operations as P
import mindspore.common.dtype as mstype
@ -23,6 +27,17 @@ from .optimizer import opt_init_args_register
class Rprop(Optimizer):
# Rprop的类，继承自Optimizer类。Rprop优化器是一种反向传播优化算法，用于处理非线性问题。
# 在更新参数时，它会根据梯度的正负性来调整步长，从而提高在非线性问题中的性能。
# 更新公式如下：
"""
如果g_{t-1} g_t > 0，则Delta_t = min(Delta_{t-1} * eta_{+}, Delta_{max})。
如果g_{t-1} g_t < 0，则Delta_t = max(Delta_{t-1} * eta_{-}, Delta_{min})。
如果g_{t-1} g_t = 0，则Delta_t = Delta_{t-1}。
更新参数w_{t} = w_{t-1} - Delta_t * sign(g_t)。
这里的\eta_{+/-}表示正负梯度时的步长因子，\Delta_{min/max}表示最小/最大的步长。
"""
r"""
Implements Resilient backpropagation.
@ -150,35 +165,55 @@ class Rprop(Optimizer):
@opt_init_args_register
def __init__(self, params, learning_rate=0.1, etas=(0.5, 1.2), step_sizes=(1e-6, 50.), weight_decay=0.):
'''
初始化Rprop算法
:param params: 参数
:param learning_rate: 学习率
:param etas: 缩放因子
:param step_sizes: 步长
:param weight_decay: 权重衰减
:return:
'''
# 首先调用父类的__init__方法，传入learning_rate、params和weight_decay参数，然后将结果赋值给当前类的实例
super(Rprop, self).__init__(learning_rate, params, weight_decay)
# 检查etas和step_sizes的类型和大小是否符合Rprop优化器的要求
# 首先，检查etas是否为tuple类型，如果不是，则抛出TypeError
if not isinstance(etas, tuple):
raise TypeError("For Rprop, etas should be a tuple, but got {}.".format(type(etas)))
if len(etas) != 2:
# 然后，检查etas的长度是否为2，如果不是，则抛出ValueError
if len(etas)!= 2:
raise ValueError("For Rprop, etas should be a tuple with the size of 2, but got {}.".format(len(etas)))
# 接着，检查step_sizes是否为tuple类型，如果不是，则抛出TypeError
if not isinstance(step_sizes, tuple):
raise TypeError("For Rprop, step_sizes should be a tuple, but got {}.".format(type(etas)))
if len(step_sizes) != 2:
# 然后，检查step_sizes的长度是否为2，如果不是，则抛出ValueError
if len(step_sizes)!= 2:
raise ValueError("For Rprop, step_sizes should be a tuple with the size of 2, "
"but got {}.".format(len(step_sizes)))
"but got {}.".format(len(step_sizes)))
# 最后，检查step_sizes的第一个元素是否大于第二个元素，如果不是，则抛出ValueError
if step_sizes[0] > step_sizes[1]:
raise ValueError("For Rprop, maximal step size should not be less than minimal step size, "
"but got {} > {}.".format(step_sizes[0], step_sizes[1]))
# 首先，使用validator.check_float_range函数检查etas[0]是否在0.0到1.0之间，如果不是，则抛出ValueError
validator.check_float_range(etas[0], 0.0, 1.0, Rel.INC_NEITHER, "etaminus", self.cls_name)
# 接着，使用validator.check_value_type函数检查etas[1]是否为float类型，如果不是，则抛出TypeError。
validator.check_value_type("etaplus", etas[1], [float], self.cls_name)
# 最后，检查etas[1]是否大于1.0，如果是，则抛出ValueError
if etas[1] <= 1.0:
raise ValueError("For Rprop, etaplus should be greater than 1.0, but got etaplus {}.".format(etas[1]))
# 检查step_sizes的第一个元素和第二个元素是否为float类型，如果不是，则抛出TypeError。
validator.check_value_type("min_step_sizes", step_sizes[0], [float], self.cls_name)
validator.check_value_type("max_step_sizes", step_sizes[1], [float], self.cls_name)
# 定义了Rprop优化器的属性，包括etaminus、etaplus、step_size_min和step_size_max。
self.etaminus, self.etaplus = etas
self.step_size_min, self.step_size_max = step_sizes
self.prev = self.parameters.clone(prefix="prev", init='zeros')
self.step_size = self.parameters.clone(prefix="step_size", init='zeros')
# 同时，它还定义了一些用于操作张量的P函数，如Fill、Sign、Assign、AssignAdd、Cast、Select和OnesLike。这些P函数通常用于构建和操作Tensor。
self.fill = P.Fill()
self.sign = P.Sign()
self.assign = P.Assign()
@ -188,37 +223,56 @@ class Rprop(Optimizer):
self.ones_like = P.OnesLike()
def construct(self, gradients):
# 对梯度进行权重衰减
gradients = self.decay_weight(gradients)
# 对梯度进行求和，并且添加中心化操作
gradients = self.gradients_centralization(gradients)
# 对梯度进行缩放操作
gradients = self.scale_grad(gradients)
# 获取学习率
lrs = self.get_lr()
# 判断是否成功
success = True
# 用于遍历gradients、self.parameters、self.prev和self.step_size的元素
for index, (grad, param, prev, step_size) in enumerate(zip(gradients, self.parameters,
self.prev, self.step_size)):
# 在循环内部，首先计算学习率lr
lr = lrs[index] if self.is_group_lr else lrs
# 并根据self.global_step是否为1来选择初始化step_size_fp32
if self.global_step == 1:
# 如果全局步数为1，则使用self.ones_like函数创建一个与step_size形状相同的张量，并将其乘以学习率lr
step_size_fp32 = self.ones_like(step_size) * lr
else:
# 否则，将step_size转换为fp32类型，并将其赋值给step_size_fp32
step_size_fp32 = self.cast(step_size, mstype.float32)
# 然后，将grad转换为float32类型，param也转换为float32类型
gradient_fp32 = self.cast(grad, mstype.float32)
param_fp32 = self.cast(param, mstype.float32)
# 接下来，计算sign，它是gradient_fp32与prev的符号函数
sign = self.sign(gradient_fp32 * prev)
# sign被裁剪到[self.etaplus, self.etaminus]之间，然后被转换为float32类型
sign = self.select(sign > 0, self.fill(mstype.float32, sign.shape, self.etaplus), sign)
sign = self.select(sign < 0, self.fill(mstype.float32, sign.shape, self.etaminus), sign)
sign = self.select(sign == 0, self.fill(mstype.float32, sign.shape, 1.), sign)
# 使用函数裁剪step_size_fp32，使其在[self.step_size_min, self.step_size_max]之间
step_size_fp32 = ops.clip_by_value(step_size_fp32 * sign, self.step_size_min, self.step_size_max)
# 最后，根据sign的值计算gradient_update，当sign为负时，gradient_update为0；当sign为正时，gradient_update为-gradient_fp32
gradient_update = self.select(sign == self.etaminus, self.fill(mstype.float32, sign.shape, 0.),
gradient_fp32)
# 然后计算next_param，它是param_fp32减去sign乘以step_size_fp32
next_param = param_fp32 - self.sign(gradient_update) * step_size_fp32
# 将next_param转换为与param相同的数据类型
self.assign(param, self.cast(next_param, param.dtype))
# 将gradient_update转换为与prev相同的数据类型
self.assign(prev, self.cast(gradient_update, prev.dtype))
# 将step_size_fp32转换为与step_size相同的数据类型，并赋值给param、prev和step_size
self.assign(step_size, self.cast(step_size_fp32, step_size.dtype))
# 返回成功信息
return success

78
mindspore/python/mindspore/nn/optim/sgd.py
View File

@ -13,26 +13,58 @@
# limitations under the License.
# ============================================================================
"""sgd"""
# 定义了SGD类，继承自Optimizer。SGD类实现了 stochastic gradient descent（SGD）优化算法，支持动态学习率和权重衰减。
# SGD类的主要目的是为了处理多层神经网络中的参数更新。
# 1.从mindspore.ops模块中导入所需的函数，例如functional、composite和operations。
from mindspore.ops import functional as F, composite as C, operations as P
# 2.从mindspore.common.parameter模块中导入Parameter类，用于创建和操作参数。
from mindspore.common.parameter import Parameter
# 3.从mindspore.common.tensor模块中导入Tensor类，用于创建和操作张量。
from mindspore.common.tensor import Tensor
# 4.从mindspore.common.dtype模块中导入mstype类，用于表示数据类型。
import mindspore.common.dtype as mstype
# 5.从mindspore._checkparam模块中导入Validator类。
from mindspore._checkparam import Validator as validator
# 6.定义一个名为Optimizer的类，用于封装优化器的实现。
from .optimizer import Optimizer
# 7.定义一个名为opt_init_args_register的函数，用于注册优化器的关键参数。
from .optimizer import opt_init_args_register
# 创建一个名为"sgd_opt"的多类型函数图。多类型函数图是一种用于表示多类型函数的图形结构，可以用于实现各种优化算法。
_sgd_opt = C.MultitypeFuncGraph("sgd_opt")
@_sgd_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor")
def _tensor_run_opt_ext(opt, momentum, learning_rate, gradient, weight, accum, stat):
"""Apply sgd optimizer to the weight parameter using Tensor."""
# 使用优化器对权重参数进行更新，并返回更新是否成功的标志。
success = True
# 调用opt函数，传入weight，gradient，learning_rate，accum，momentum，stat参数
success = F.depend(success, opt(weight, gradient, learning_rate, accum, momentum, stat))
return success
class SGD(Optimizer):
# SGD（随机梯度下降）是一种常用的优化算法，用于解决深度学习中的梯度下降问题。
# SGD的主要思想是每次更新权重时，使用随机梯度（gradient）来计算更新方向，然后沿着这个方向更新权重。
# 在更新过程中，可以使用动量（momentum）来平滑梯度更新，以提高训练速度。
# SGD类实现了两种动量算法：标准动量（standard momentum）和Nesterov动量（Nesterov momentum）。标准动量计算公式如下：
"""
math::
v_{t+1} = u \ast v_{t} + gradient \ast (1-dampening)
其中，u是一个超参数，用于控制动量系数。
如果使用Nesterov动量，计算公式如下：
math::
p_{t+1} = p_{t} - lr \ast (gradient + u \ast v_{t+1})
这里，p表示权重参数，v表示累积梯度，u表示动量系数。与标准动量不同，Nesterov动量在计算更新方向时，将当前梯度与上一次更新方向（v）相加。这样可以提高在梯度下降过程中，权重参数的更新速度。
为了实现Nesterov动量，需要在SGD类中添加一个名为"nesterov"的布尔参数，并在更新权重时根据该参数选择使用哪种动量算法。
"""
r"""
Implements stochastic gradient descent. Momentum is optional.
@ -144,56 +176,88 @@ class SGD(Optimizer):
def __init__(self, params, learning_rate=0.1, momentum=0.0, dampening=0.0, weight_decay=0.0, nesterov=False,
loss_scale=1.0):
# 初始化SGD类，并设置SGD类的参数
super(SGD, self).__init__(learning_rate, params, weight_decay, loss_scale)
# 判断momentum的类型
if isinstance(momentum, int):
# 如果是int类型，则将其转换为float类型
momentum = float(momentum)
# 如果momentum的类型不是int类型，则抛出异常TypeError
if not isinstance(momentum, float):
raise TypeError("For 'SGD', the argument 'momentum' should be float type, "
raise TypeError("For 'SGD', the argument'momentum' should be float type, "
"but got {}.".format(type(momentum)))
# 判断momentum的值
if isinstance(momentum, float) and momentum < 0.0:
raise ValueError("For 'SGD', the argument 'momentum' should be at least 0.0, "
"but got {}".format(momentum))
# 如果momentum的值小于0，则抛出异常ValueError
raise ValueError("For 'SGD', the argument'momentum' should be at least 0.0, "
"but got {}".format(momentum))
# 判断dampening的类型
if isinstance(dampening, int):
# 如果是int类型，则将其转换为float类型
dampening = float(dampening)
# 如果转换后dampening的类型不是float类型，则抛出异常TypeError
if not isinstance(dampening, float):
raise TypeError("For 'SGD', the argument 'dampening' should be float type, "
"but got {}.".format(type(dampening)))
# 定义SGD梯度解析器，参数dampening为梯度衰减系数，weight_decay为权重衰减系数，nesterov为是否使用Nesterov梯度解析器，momentum为动量，dampening为权重衰减系数
# 如果dampening的值小于0
if dampening < 0.0:
# 则抛出异常ValueError
raise ValueError("For 'SGD', the argument 'dampening' should be at least 0.0, "
"but got 'dampening' {}".format(dampening))
# 传入dampening的值
self.dampening = dampening
# 判断weight_decay的类型
if isinstance(weight_decay, int):
# 如果是int类型，则将其转换为float类型
weight_decay = float(weight_decay)
# 使用"validator.check_value_type"函数检查"nesterov"参数的类型是否为布尔类型，如果不是，则抛出异常。
validator.check_value_type("nesterov", nesterov, [bool], self.cls_name)
if nesterov and (momentum <= 0.0 or dampening != 0.0):
raise ValueError("For 'SGD', if 'nesterov' is true, 'momentum' must be > 0.0 and 'dampening' must "
"equal to 0.0, but got 'momentum' {}, 'dampening' {}".format(momentum, dampening))
# 如果nesterov为True，且momentum小于等于0.0或者dampening不等于0.0，抛出异常ValueError
if nesterov and (momentum <= 0.0 or dampening!= 0.0):
raise ValueError("For 'SGD', if 'nesterov' is true,'momentum' must be > 0.0 and 'dampening' must "
"equal to 0.0, but got'momentum' {}, 'dampening' {}".format(momentum, dampening))
# 传入nesterov的值
self.nesterov = nesterov
# 使用P.SGD函数创建一个SGD优化器
self.opt = P.SGD(dampening, weight_decay, nesterov)
# 创建一个Parameter类型的变量，并赋值为momentum
self.momentum = Parameter(Tensor(momentum, mstype.float32), name="momentum")
# 创建一个Parameter类型的变量，并赋值为accum
self.accum = self.parameters.clone(prefix="accum", init='zeros')
# 创建一个Parameter类型的变量，并赋值为stat
self.stat = self.parameters.clone(prefix="stat", init='ones')
def construct(self, gradients):
# 处理输入的梯度并更新参数
params = self.parameters
accum = self.accum
stat = self.stat
# 首先对梯度进行 centralization（去均值）处理
gradients = self.gradients_centralization(gradients)
# 然后对梯度进行 scale_grad（缩放）处理
gradients = self.scale_grad(gradients)
# 获取学习率
lr = self.get_lr()
# 最后根据学习率和参数更新规则进行参数更新。
# 实现了hyper_map_reverse方法，用于对输入的参数进行批量操作。这个方法可以用于实现学习率动态更新、参数组分组等功能。
# 如果是分组学习率，则用分组方法使用hyper_map_reverse方法
if self.is_group_lr:
success = self.hyper_map_reverse(F.partial(_sgd_opt, self.opt, self.momentum),
lr, gradients, params, accum, stat)
else:
# 否则，用普通方法使用hyper_map_reverse方法
success = self.hyper_map_reverse(F.partial(_sgd_opt, self.opt, self.momentum, lr),
gradients, params, accum, stat)
# 返回成功消息
return success

984
mindspore/python/mindspore/nn/optim/thor.py
View File

File diff suppressed because it is too large Load Diff

3
mindspore/python/mindspore/nn/reinforcement/__init__.py
View File

@ -15,8 +15,11 @@
"""
TensorArray.
"""
# TensorArray类用于实现一个类似于Python列表的Tensor数组，用于存储和操作Tensor
from .tensor_array import (TensorArray)
# 从tensor_array模块中导入TensorArray类
__all__ = [
"TensorArray",
]
# 将TensorArray添加到__all__列表中，这样在导入时，只需要导入__all__列表中的内容即可

16
mindspore/python/mindspore/nn/reinforcement/_batch_read_write.py
View File

@ -12,12 +12,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 这段代码定义了两个类，BatchWrite和BatchRead，它们都是Cell类的子类。
# BatchWrite类用于将源参数列表写入目标参数列表，而BatchRead类用于从源参数列表读取目标参数列表。
"""
BatchReadWrite
"""
# 从mindspore.nn.cell模块中导入Cell类和BatchAssign操作
from mindspore.nn.cell import Cell
# BatchAssign操作是mindspore.ops模块中的一个内部操作，用于将源参数列表写入目标参数列表
from mindspore.ops.operations._rl_inner_ops import BatchAssign
# 在示例中，我们创建了两个网络模型SNet和DNet，分别用于源模型和目标模型。
# 然后，我们创建了一个Write类，它使用BatchWrite操作将源模型的参数写入目标模型，并创建了一个Read类，它使用BatchRead操作从源模型的参数读取目标模型。
# 最后，我们使用这两个类分别对源模型和目标模型进行参数的写入和读取操作
class BatchWrite(Cell):
r"""BatchWrite: write a list of parameters to assign the target.
@ -62,9 +68,11 @@ class BatchWrite(Cell):
def __init__(self):
"""Initialize BatchWrite"""
super(BatchWrite, self).__init__()
# 初始化BatchAssign
self.write = BatchAssign(lock=True)
def construct(self, dst, src):
# BatchWrite类的construct方法接受两个参数dst和src，分别表示目标参数列表和源参数列表。它首先使用BatchAssign操作将源参数列表写入目标参数列表，然后返回True
"""
Write the source parameter list to assign the dst.
@ -75,6 +83,9 @@ class BatchWrite(Cell):
Returns:
Bool, true.
"""
"""
将src指向的内容复制到dst指向的内容中
"""
self.write(dst, src)
return True
@ -123,9 +134,11 @@ class BatchRead(Cell):
def __init__(self):
"""Initialize BatchRead"""
super(BatchRead, self).__init__()
# 初始化读取
self.read = BatchAssign(lock=False)
def construct(self, dst, src):
# BatchRead类的construct方法也接受两个参数dst和src，分别表示目标参数列表和源参数列表。它首先使用BatchAssign操作从源参数列表读取目标参数列表，然后返回True
"""
Read the source parameter list to assign the dst.
@ -136,5 +149,6 @@ class BatchRead(Cell):
Returns:
Bool, true.
"""
# 读取源参数列表
self.read(dst, src)
return True

31
mindspore/python/mindspore/nn/reinforcement/_tensors_queue.py
View File

@ -12,16 +12,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# TensorsQueue类，继承自Cell。TensorsQueue是一个队列，用于存储张量列表。它是mindspore.nn模块的一部分，用于实现深度学习任务中的张量队列
"""
TensorsQueue, each element in the queue is a list of tensors.
"""
# 导入所需的库
# 模块中包含了Cell类，用于构建神经网络
from mindspore.nn.cell import Cell
# 模块中包含了_rl_inner_ops，用于实现一些与Reinforcement Learning相关的操作
from mindspore.ops.operations import _rl_inner_ops as rl_ops
# 模块中包含了Rel和Validator类，用于验证参数的合法性
from mindspore._checkparam import Rel, Validator
from mindspore.common import dtype as mstype
# 在示例代码中，我们创建了一个TensorsQueue对象，并将其用于存储两个张量变量data1和data2。然后，我们将这两个张量元组放入队列中，并从队列中获取一个张量元组并打印出来
class TensorsQueue(Cell):
# TensorsQueue类的构造函数接受四个参数：数据类型dtype、形状shapes、大小size和名称name。
# 其中，dtype必须为mindspore.dtype类型的数字类型，size必须大于等于0，shapes的大小必须大于等于1
r'''
TensorsQueue: a queue which stores tensors lists.
@ -56,17 +65,29 @@ class TensorsQueue(Cell):
def __init__(self, dtype, shapes, size=0, name="TQ"):
"""Initialize TensorsQueue"""
super(TensorsQueue, self).__init__()
# 检查dtype是否为数字类型或布尔型，Rel.GE表示大于等于（greater than or equal to）的关系
Validator.check_subclass("dtype", dtype, mstype.number_type + (mstype.bool_,), self.cls_name)
# 检查size是否为0或大于0
Validator.check_int(size, 0, Rel.GE, "size", self.cls_name)
# 获取shapes的长度
elements_num = len(shapes)
# 检查elements_num是否为1或大于1
Validator.check_int(elements_num, 1, Rel.GE, "len(shapes)", self.cls_name)
# 创建TensorsQueue，传入dtype、shapes、size和name作为参数
self.handle_ = rl_ops.TensorsQueueCreate(dtype, shapes, size, name)()
# 创建TensorsQueuePut，分别表示TensorsQueue的put、get、pop、clear、close和size方法。这些属性都是rl_ops模块中的函数，用于执行相应的操作
self.tensors_q_put = rl_ops.TensorsQueuePut(dtype, shapes)
# 创建TensorsQueueGet
self.tensors_q_get = rl_ops.TensorsQueueGet(dtype, shapes)
# 创建TensorsQueueGet
self.tensors_q_pop = rl_ops.TensorsQueueGet(dtype, shapes, pop_after_get=True)
# 创建TensorsQueueClear
self.tensors_q_clear = rl_ops.TensorsQueueClear()
# 创建TensorsQueueClose
self.tensors_q_close = rl_ops.TensorsQueueClose()
# 获取TensorsQueueSize
self.tensors_q_size = rl_ops.TensorsQueueSize()
# 总之，这段代码定义了一个名为TensorsQueue的类，继承自nn.cell，并实现了__init__方法来初始化类的属性。同时，定义了一些方法来执行TensorsQueue的相关操作
def put(self, element):
"""
@ -78,6 +99,7 @@ class TensorsQueue(Cell):
Returns:
Bool, true.
"""
# 将元组(Tensors)转换为TensorsQueue的元素
self.tensors_q_put(self.handle_, element)
return True
@ -88,6 +110,7 @@ class TensorsQueue(Cell):
Returns:
tuple(Tensors), the element in TensorsQueue.
"""
# 从TensorsQueue中取出一个元素
element = self.tensors_q_get(self.handle_)
return element
@ -98,7 +121,9 @@ class TensorsQueue(Cell):
Returns:
tuple(Tensors), the element in TensorsQueue.
"""
# 获取第一个元素
element = self.tensors_q_pop(self.handle_)
# 返回元素
return element
def size(self):
@ -108,9 +133,10 @@ class TensorsQueue(Cell):
Returns:
Tensor(mindspore.int64), the used size of TensorsQueue.
"""
# 获取当前可用/可用大小
size = self.tensors_q_size(self.handle_)
# 返回当前可用/可用大小
return size
def close(self):
"""
Close the created TensorsQueue.
@ -123,6 +149,7 @@ class TensorsQueue(Cell):
Returns:
Bool, true.
"""
# 关闭TensorsQueue
self.tensors_q_close(self.handle_)
return True
@ -134,5 +161,7 @@ class TensorsQueue(Cell):
Returns:
Bool, true.
"""
# 清空tensors_q_clear函数，并将handle_参数赋值给self.handle_
self.tensors_q_clear(self.handle_)
# 返回True
return True

22
mindspore/python/mindspore/nn/reinforcement/tensor_array.py
View File

@ -12,16 +12,23 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# 继承自Cell类。TensorArray类用于实现一个类似于Python列表的Tensor数组，用于存储和操作Tensor
"""
TensorArray
"""
# Cell类，用于构建神经网络
from mindspore.nn.cell import Cell
# _tensor_array，用于实现与TensorArray相关的操作
from mindspore.ops.operations import _tensor_array as ta
# Rel和Validator类，用于验证参数的合法性
from mindspore._checkparam import Rel, Validator
# dtype类，用于表示数据类型
from mindspore.common import dtype as mstype
class TensorArray(Cell):
# TensorArray类用于实现一个类似于Python列表的Tensor数组，用于存储和操作Tensor
r"""TensorArray: a dynamic array to store tensors.
.. warning::
@ -59,14 +66,23 @@ class TensorArray(Cell):
def __init__(self, dtype, element_shape, dynamic_size=True, size=0, name="TA"):
"""Initialize TensorArray"""
super(TensorArray, self).__init__()
# 检查dtype是否是数字类型或布尔类型
Validator.check_subclass("dtype", dtype, mstype.number_type + (mstype.bool_,), self.cls_name)
# 检查size是否大于等于0
Validator.check_int(size, 0, Rel.GE, "size", self.cls_name)
# 初始化TensorArray
self.handle_ = ta.TensorArray(dtype, element_shape, dynamic_size, size, name)()
# 初始化TensorArrayWrite
self.tensor_array_write = ta.TensorArrayWrite()
# 初始化TensorArrayRead
self.tensor_array_read = ta.TensorArrayRead(dtype, element_shape)
# 初始化TensorArrayClose
self.tensor_array_close = ta.TensorArrayClose()
# 初始化TensorArrayClear
self.tensor_array_clear = ta.TensorArrayClear()
# 初始化TensorArrayStack
self.tensor_array_stack = ta.TensorArrayStack(dtype, element_shape, dynamic_size, size)
# 初始化TensorArraySize
self.tensor_array_size = ta.TensorArraySize()
def write(self, index, value):
@ -80,6 +96,7 @@ class TensorArray(Cell):
Returns:
Bool, true.
"""
# 将value写入TensorArray，并返回True
self.tensor_array_write(self.handle_, index, value)
return True
@ -93,6 +110,7 @@ class TensorArray(Cell):
Returns:
Tensor, the value in position index.
"""
# 读取TensorArray中指定位置的值，并返回
value = self.tensor_array_read(self.handle_, index)
return value
@ -108,6 +126,7 @@ class TensorArray(Cell):
Returns:
Bool, true.
"""
# 关闭TensorArray
self.tensor_array_close(self.handle_)
return True
@ -119,6 +138,7 @@ class TensorArray(Cell):
Returns:
Bool, true.
"""
# 清空TensorArray
self.tensor_array_clear(self.handle_)
return True
@ -129,6 +149,7 @@ class TensorArray(Cell):
Returns:
Tensor, all the values will be stacked into one tensor.
"""
# 使用tensor_array_stack函数将TensorArray中的值堆叠到一个新的Tensor中
ans = self.tensor_array_stack(self.handle_)
return ans
@ -139,5 +160,6 @@ class TensorArray(Cell):
Returns:
Tensor, the size of TensorArray.
"""
# 使用tensor_array_size函数获取TensorArray中的大小
size = self.tensor_array_size(self.handle_)
return size

Some files were not shown because too many files have changed in this diff Show More