diff --git a/docs/api/api_python/dataset/mindspore.dataset.BatchInfo.rst b/docs/api/api_python/dataset/mindspore.dataset.BatchInfo.rst new file mode 100644 index 00000000000..70903e5e45a --- /dev/null +++ b/docs/api/api_python/dataset/mindspore.dataset.BatchInfo.rst @@ -0,0 +1,15 @@ +mindspore.dataset.BatchInfo +=========================== + +.. py:class:: mindspore.dataset.BatchInfo + + 此类提供了两种方法获取数据集的批处理数量（batch size）和迭代数（epoch）， + 用于 `batch` 操作中的输入参数 `batch_size` 和 `per_batch_map`。 + + .. py:method:: get_batch_num() + + 返回数据集的批处理数量（batch size）。 + + .. py:method:: get_epoch_num() + + 返回数据集的迭代数（epoch）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.Caltech101Dataset.rst b/docs/api/api_python/dataset/mindspore.dataset.Caltech101Dataset.rst index a4a2a0d2f52..69c71dd66c5 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.Caltech101Dataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.Caltech101Dataset.rst @@ -34,7 +34,7 @@ mindspore.dataset.Caltech101Dataset - **ValueError** - `target_type` 参数取值不为'category'、'annotation'或'all'。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数值错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.Caltech256Dataset.rst b/docs/api/api_python/dataset/mindspore.dataset.Caltech256Dataset.rst index bdcfca4db68..0ea9c0464bd 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.Caltech256Dataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.Caltech256Dataset.rst @@ -25,7 +25,7 @@ mindspore.dataset.Caltech256Dataset - **ValueError** - `target_type` 参数取值不为'category'、'annotation'或'all'。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数值错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.CelebADataset.rst b/docs/api/api_python/dataset/mindspore.dataset.CelebADataset.rst index f1b0d689dcd..20057a7b62e 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.CelebADataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.CelebADataset.rst @@ -27,7 +27,7 @@ mindspore.dataset.CelebADataset - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **ValueError** - `usage` 参数取值不为'train'、'valid'、'test'或'all'。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数值错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.Cifar100Dataset.rst b/docs/api/api_python/dataset/mindspore.dataset.Cifar100Dataset.rst index ce8c544434c..b0ba339a141 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.Cifar100Dataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.Cifar100Dataset.rst @@ -26,7 +26,7 @@ mindspore.dataset.Cifar100Dataset - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **ValueError** - `usage` 参数取值不为'train'、'test'或'all'。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数错误（小于0或者大于等于 `num_shards`）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.Cifar10Dataset.rst b/docs/api/api_python/dataset/mindspore.dataset.Cifar10Dataset.rst index db93a595450..fb9615544c1 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.Cifar10Dataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.Cifar10Dataset.rst @@ -26,7 +26,7 @@ mindspore.dataset.Cifar10Dataset - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **ValueError** - `usage` 参数取值不为'train'、'test'或'all'。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.CityscapesDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.CityscapesDataset.rst index 1ab95e93234..00531e5aad9 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.CityscapesDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.CityscapesDataset.rst @@ -29,7 +29,7 @@ mindspore.dataset.CityscapesDataset - **RuntimeError** - `dataset_dir` 路径下不包含任何数据文件。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `dataset_dir` 路径非法或不存在。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.CocoDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.CocoDataset.rst index 791c72e579e..3897c942338 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.CocoDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.CocoDataset.rst @@ -60,7 +60,7 @@ - **RuntimeError** - `dataset_dir` 路径下不包含任何数据文件。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **RuntimeError** - 解析 `annotation_file` 指定的JSON文件失败。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.DIV2KDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.DIV2KDataset.rst index 8631fbd1f86..855deaecec8 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.DIV2KDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.DIV2KDataset.rst @@ -5,7 +5,7 @@ mindspore.dataset.DIV2KDataset 读取和解析DIV2K数据集的源文件构建数据集。 - 生成的数据集有两列 `[hr_image, lr_image]` 。'hr_image` 列和 `lr_image` 列的数据类型都为uint8。 + 生成的数据集有两列 `[hr_image, lr_image]` 。 `hr_image` 列和 `lr_image` 列的数据类型都为uint8。 **参数：** @@ -28,13 +28,15 @@ mindspore.dataset.DIV2KDataset - **RuntimeError** - `dataset_dir` 路径下不包含任何数据文件。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `dataset_dir` 路径非法或不存在。 - **ValueError** - `usage` 参数取值不为'train'、'valid'或'all'。 - **ValueError** - `downgrade` 参数取值不为'bicubic'、'unknown'、'mild'、'difficult'或'wild'。 - **ValueError** - `scale` 参数取值不在给定的字段中，或与 `downgrade` 参数的值不匹配。 + - **ValueError** - `scale` 参数取值为8，但 `downgrade` 参数的值不为 'bicubic'。 + - **ValueError** - `downgrade` 参数取值为'mild'、'difficult'或'wild'，但 `scale` 参数的值不为4。 - **ValueError** - `shard_id` 参数值错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.GeneratorDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.GeneratorDataset.rst index 723304906ac..fb009323956 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.GeneratorDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.GeneratorDataset.rst @@ -33,7 +33,7 @@ - **RuntimeError** - `column_names` 参数指定的列名数量与 `source` 参数输出的数据数量不匹配。 - **ValueError** - `num_parallel_workers` 参数超过最大线程数。 - **ValueError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **ValueError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **ValueError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **ValueError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **ValueError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数值错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.ImageFolderDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.ImageFolderDataset.rst index b2b0db4c0d8..8447eb7534a 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.ImageFolderDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.ImageFolderDataset.rst @@ -26,7 +26,7 @@ mindspore.dataset.ImageFolderDataset - **RuntimeError** - `dataset_dir` 不包含任何数据文件。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **RuntimeError** - `class_indexing` 参数的类型不是dict。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.ManifestDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.ManifestDataset.rst index f4b95dcfa62..596cdcaefe1 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.ManifestDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.ManifestDataset.rst @@ -26,7 +26,7 @@ - **RuntimeError** - `dataset_files` 路径下不包含任何数据文件。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **RuntimeError** - `class_indexing` 参数的类型不是dict。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.MnistDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.MnistDataset.rst index 447d30331a0..59753c5c9c4 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.MnistDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.MnistDataset.rst @@ -26,7 +26,7 @@ mindspore.dataset.MnistDataset - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **ValueError** - `usage` 参数取值不为'train'、'test'或'all'。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.NumpySlicesDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.NumpySlicesDataset.rst index b332c70b740..d846ef03961 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.NumpySlicesDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.NumpySlicesDataset.rst @@ -53,7 +53,7 @@ mindspore.dataset.NumpySlicesDataset - **RuntimeError** - `column_names` 列表的长度与数据的输出列表长度不匹配。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 - **ValueError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **ValueError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **ValueError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **ValueError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **ValueError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `shard_id` 参数值错误（小于0或者大于等于 `num_shards` ）。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.OutputFormat.rst b/docs/api/api_python/dataset/mindspore.dataset.OutputFormat.rst new file mode 100644 index 00000000000..02ed252787b --- /dev/null +++ b/docs/api/api_python/dataset/mindspore.dataset.OutputFormat.rst @@ -0,0 +1,11 @@ +mindspore.dataset.OutputFormat +============================== + +.. py:class:: mindspore.dataset.OutputFormat + + 通过API `get_all_neighbors` 获取所有相邻节点时，指定节点的存储格式。 + + - **OutputFormat.NORMAL** - 普通格式，不进行压缩。 + - **OutputFormat.COO** - 采用COO（Coordinate）格式进行存储。 + - **OutputFormat.CSR** - 采用CSR（Compressed Sparse Row）格式进行存储。 + \ No newline at end of file diff --git a/docs/api/api_python/dataset/mindspore.dataset.SamplingStrategy.rst b/docs/api/api_python/dataset/mindspore.dataset.SamplingStrategy.rst new file mode 100644 index 00000000000..cb229acf998 --- /dev/null +++ b/docs/api/api_python/dataset/mindspore.dataset.SamplingStrategy.rst @@ -0,0 +1,9 @@ +mindspore.dataset.SamplingStrategy +================================== + +.. py:class:: mindspore.dataset.SamplingStrategy + + 指定图数据采样策略的枚举类。 + + - **SamplingStrategy.RANDOM** - 随机抽样，带放回采样。 + - **SamplingStrategy.EDGE_WEIGHT** - 以边缘的权重为概率进行采样。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.Shuffle.rst b/docs/api/api_python/dataset/mindspore.dataset.Shuffle.rst new file mode 100644 index 00000000000..91b6bddddbb --- /dev/null +++ b/docs/api/api_python/dataset/mindspore.dataset.Shuffle.rst @@ -0,0 +1,10 @@ +mindspore.dataset.Shuffle +========================= + +.. py:class:: mindspore.dataset.Shuffle + + 指定混洗模式的枚举类。 + + - **Shuffle.GLOBAL** - 混洗文件和文件中的数据。 + - **Shuffle.FILES** - 仅混洗文件。 + - **Shuffle.INFILE** - 保持读入文件的序列，仅混洗每个文件中的数据。 diff --git a/docs/api/api_python/dataset/mindspore.dataset.VOCDataset.rst b/docs/api/api_python/dataset/mindspore.dataset.VOCDataset.rst index d2b19153200..43632ad17ab 100644 --- a/docs/api/api_python/dataset/mindspore.dataset.VOCDataset.rst +++ b/docs/api/api_python/dataset/mindspore.dataset.VOCDataset.rst @@ -37,7 +37,7 @@ mindspore.dataset.VOCDataset - **RuntimeError** - 读取的xml文件缺失 `object` 属性。 - **RuntimeError** - 读取的xml文件缺失 `bndbox` 属性。 - **RuntimeError** - 同时指定了 `sampler` 和 `shuffle` 参数。 - - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数。 + - **RuntimeError** - 同时指定了 `sampler` 和 `num_shards` 参数或同时指定了 `sampler` 和 `shard_id` 参数。 - **RuntimeError** - 指定了 `num_shards` 参数，但是未指定 `shard_id` 参数。 - **RuntimeError** - 指定了 `shard_id` 参数，但是未指定 `num_shards` 参数。 - **ValueError** - `num_parallel_workers` 参数超过系统最大线程数。 diff --git a/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.JiebaTokenizer.rst b/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.JiebaTokenizer.rst index 661defdde55..3b71431dc9e 100644 --- a/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.JiebaTokenizer.rst +++ b/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.JiebaTokenizer.rst @@ -28,29 +28,29 @@ mindspore.dataset.text.transforms.JiebaTokenizer - **TypeError** - 参数 `hmm_path` 和 `mp_path` 类型不为string。 - **TypeError** - 参数 `with_offsets` 类型不为bool。 -.. py:method:: add_word(self, word, freq=None) + .. py:method:: add_word(self, word, freq=None) - 将用户定义的词添加到 JiebaTokenizer 的字典中。 + 将用户定义的词添加到 JiebaTokenizer 的字典中。 - **参数：** + **参数：** - - **word** (str) - 要添加到 JiebaTokenizer 词典中的单词，注意通过此接口添加的单词不会被写入本地的模型文件中。 - - **freq** (int，可选) - 要添加的单词的频率。频率越高，单词被分词的机会越大。默认值：None，使用默认频率。 + - **word** (str) - 要添加到 JiebaTokenizer 词典中的单词，注意通过此接口添加的单词不会被写入本地的模型文件中。 + - **freq** (int，可选) - 要添加的单词的频率。频率越高，单词被分词的机会越大。默认值：None，使用默认频率。 -.. py:method:: add_dict(self, user_dict) + .. py:method:: add_dict(self, user_dict) - 将用户定义的词添加到 JiebaTokenizer 的字典中。 + 将用户定义的词添加到 JiebaTokenizer 的字典中。 - **参数：** + **参数：** - - **user_dict** (Union[str, dict]) - 有两种输入方式。可以通过指定jieba字典格式的文件路径加载。 - 要求的jieba字典格式为：[word，freq]，如： + - **user_dict** (Union[str, dict]) - 有两种输入方式。可以通过指定jieba字典格式的文件路径加载。 + 要求的jieba字典格式为：[word，freq]，如： - .. code-block:: + .. code-block:: - word1 freq1 - word2 None - word3 freq3 + word1 freq1 + word2 None + word3 freq3 - 也可以通过Python dict加载，要求的 Python 字典格式为：{word1:freq1, word2:freq2,...}。 - 只有用户提供的文件中有效的词对才会被添加到字典中，无的效输入行将被忽略，且不返回错误或警告状态。 + 也可以通过Python dict加载，要求的 Python 字典格式为：{word1:freq1, word2:freq2,...}。 + 只有用户提供的文件中有效的词对才会被添加到字典中，无的效输入行将被忽略，且不返回错误或警告状态。 diff --git a/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.SentencePieceTokenizer.rst b/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.SentencePieceTokenizer.rst index 416e48b6378..1916e02e1a8 100644 --- a/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.SentencePieceTokenizer.rst +++ b/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.SentencePieceTokenizer.rst @@ -16,5 +16,5 @@ mindspore.dataset.text.transforms.SentencePieceTokenizer **异常：** - - **TypeError** - 参数 `mode` 的类型不是string或SentencePieceVocab。 - - **TypeError** - 参数 `out_type` 的类型不是SPieceTokenizerOutType。 + - **TypeError** - 参数 `mode` 的类型不是string或 :class:`mindspore.dataset.text.SentencePieceVocab`。 + - **TypeError** - 参数 `out_type` 的类型不是 :class::`mindspore.dataset.text.SPieceTokenizerOutType`。 diff --git a/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.WordpieceTokenizer.rst b/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.WordpieceTokenizer.rst index a0114b2be9c..f32c31ee11c 100644 --- a/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.WordpieceTokenizer.rst +++ b/docs/api/api_python/dataset_text/mindspore.dataset.text.transforms.WordpieceTokenizer.rst @@ -21,4 +21,4 @@ mindspore.dataset.text.transforms.WordpieceTokenizer - **TypeError** - 参数 `max_bytes_per_token` 的类型不为int。 - **TypeError** - 参数 `unknown_token` 的类型不为string。 - **TypeError** - 参数 `with_offsets` 的类型不为bool。 - - **TypeError** - 参数 `max_bytes_per_token` 的值为非负数。 + - **ValueError** - 参数 `max_bytes_per_token` 的值为负数。 diff --git a/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.c_transforms.Mask.rst b/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.c_transforms.Mask.rst index cbb99a2fb2a..dfd89caa8dd 100644 --- a/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.c_transforms.Mask.rst +++ b/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.c_transforms.Mask.rst @@ -7,8 +7,9 @@ mindspore.dataset.transforms.c_transforms.Mask **参数：** - - **operator** (Relational) - 关系操作符，可以取值为 Relational.EQ、Relational.NE、Relational.LT、 - Relational.GT、Relational.LE、Relational.GE。以Relational.EQ为例，将找出Tensor中与 `constant` 相等的元素。 + - **operator** (:class:`mindspore.dataset.transforms.c_transforms.Relational`) - 关系操作符，可以取值为 + Relational.EQ、Relational.NE、Relational.LT、 Relational.GT、Relational.LE、Relational.GE。 + 以Relational.EQ为例，将找出Tensor中与 `constant` 相等的元素。 - **constant** (Union[str, int, float, bool]) - 要进行比较的基准值。 - **dtype** (mindspore.dtype, 可选) - 生成的掩码Tensor的数据类型，默认值：mstype.bool。 diff --git a/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.Compose.rst b/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.Compose.rst index b0730dd6c95..08c713b8ae4 100644 --- a/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.Compose.rst +++ b/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.Compose.rst @@ -5,6 +5,10 @@ mindspore.dataset.transforms.py_transforms.Compose 将多个数据增强算子组合使用。 + .. note:: Compose可以将 `mindspore.dataset.transforms.py_transforms` 模块中的数据增强算子以及用户自定义的Python可调用对象 + 合并成单个数据增强。对于用户定义的Python可调用对象，要求其返回值是numpy.ndarray类型。有关如何使用，请参阅Compose的示例，或阅读 + :class:`mindspore.dataset.vision.py_transforms.FiveCrop` 的示例，学习如何与用户自定义Python可调用对象配合使用。 + **参数：** - **transforms** (list) - 一个数据增强的列表。 @@ -14,3 +18,11 @@ mindspore.dataset.transforms.py_transforms.Compose - **TypeError** - 参数 `transforms` 类型不为 list。 - **ValueError** - 参数 `transforms` 的长度为空。 - **TypeError** - 参数 `transforms` 的元素不是Python的callable对象。 + + .. py:method:: reduce(operations) + + 静态方法，可以调用这个函数将多个数据增强算子组合起来。 + + **参数：** + + - **operations** (list) - 数据增强的列表。 \ No newline at end of file diff --git a/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.OneHotOp.rst b/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.OneHotOp.rst index 4f9aef60a7f..dec6a4b3a8f 100644 --- a/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.OneHotOp.rst +++ b/docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.py_transforms.OneHotOp.rst @@ -8,7 +8,7 @@ mindspore.dataset.transforms.py_transforms.OneHotOp **参数：** - **num_classes** (int) - 数据集的类别数，它应该大于数据集中最大的label编号。 - - **num_classes** (float，可选) - 标签平滑的系数，默认值：0.0。 + - **smoothing_rate** (float，可选) - 标签平滑的系数，默认值：0.0。 **异常：** diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.c_transforms.RandomPosterize.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.c_transforms.RandomPosterize.rst index d014c515e3e..53e77ed4036 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.c_transforms.RandomPosterize.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.c_transforms.RandomPosterize.rst @@ -13,5 +13,5 @@ mindspore.dataset.vision.c_transforms.RandomPosterize **异常：** - **TypeError** - 如果 `bits` 不是int或整数序列类型。 - - **ValueErro：** - 如果 `bits` 不在 [1, 8] 范围内。 + - **ValueError** - 如果 `bits` 不在 [1, 8] 范围内。 - **RuntimeError** - 如果输入图像的shape不是 或 。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Pad.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Pad.rst index 3167f4f92e6..0c20e8fee4d 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Pad.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Pad.rst @@ -1,7 +1,7 @@ mindspore.dataset.vision.py_transforms.Pad ========================================== -.. py:class:: mindspore.dataset.vision.py_transforms.Pad(padding, fill_value=0, padding_mode=) +.. py:class:: mindspore.dataset.vision.py_transforms.Pad(padding, fill_value=0, padding_mode=Border.CONSTANT) 对输入PIL图像的各边进行填充。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomAffine.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomAffine.rst index aa66de2b4a3..b1e922260d4 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomAffine.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomAffine.rst @@ -1,7 +1,7 @@ mindspore.dataset.vision.py_transforms.RandomAffine =================================================== -.. py:class:: mindspore.dataset.vision.py_transforms.RandomAffine(degrees, translate=None, scale=None, shear=None, resample=, fill_value=0) +.. py:class:: mindspore.dataset.vision.py_transforms.RandomAffine(degrees, translate=None, scale=None, shear=None, resample=Inter.NEAREST, fill_value=0) 对输入PIL图像进行随机仿射变换。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomCrop.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomCrop.rst index a95b5191451..19efe0c7267 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomCrop.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomCrop.rst @@ -1,7 +1,7 @@ mindspore.dataset.vision.py_transforms.RandomCrop ================================================= -.. py:class:: mindspore.dataset.vision.py_transforms.RandomCrop(size, padding=None, pad_if_needed=False, fill_value=0, padding_mode=) +.. py:class:: mindspore.dataset.vision.py_transforms.RandomCrop(size, padding=None, pad_if_needed=False, fill_value=0, padding_mode=Border.CONSTANT) 在输入PIL图像上的随机位置，裁剪指定大小的子图。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomPerspective.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomPerspective.rst index 6b95a2cf98b..461cb82a1e2 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomPerspective.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomPerspective.rst @@ -1,7 +1,7 @@ mindspore.dataset.vision.py_transforms.RandomPerspective ======================================================== -.. py:class:: mindspore.dataset.vision.py_transforms.RandomPerspective(distortion_scale=0.5, prob=0.5, interpolation=) +.. py:class:: mindspore.dataset.vision.py_transforms.RandomPerspective(distortion_scale=0.5, prob=0.5, interpolation=Inter.BICUBIC) 按照指定的概率对输入PIL图像进行透视变换。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomResizedCrop.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomResizedCrop.rst index 4eebc220bf4..f5fec9b1e46 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomResizedCrop.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomResizedCrop.rst @@ -1,7 +1,7 @@ mindspore.dataset.vision.py_transforms.RandomResizedCrop ======================================================== -.. py:class:: mindspore.dataset.vision.py_transforms.RandomResizedCrop(size, scale=(0.08, 1.0), ratio=(0.75, 1.3333333333333333), interpolation=, max_attempts=10) +.. py:class:: mindspore.dataset.vision.py_transforms.RandomResizedCrop(size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation=Inter.BILINEAR, max_attempts=10) 在输入PIL图像上的随机位置裁剪子图，并放缩到指定大小。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomRotation.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomRotation.rst index ba0be46346a..efe21cc5038 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomRotation.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.RandomRotation.rst @@ -1,10 +1,12 @@ mindspore.dataset.vision.py_transforms.RandomRotation ===================================================== -.. py:class:: mindspore.dataset.vision.py_transforms.RandomRotation(degrees, resample=, expand=False, center=None, fill_value=0) +.. py:class:: mindspore.dataset.vision.py_transforms.RandomRotation(degrees, resample=Inter.NEAREST, expand=False, center=None, fill_value=0) 将输入PIL图像旋转随机角度。 + .. note:: 参阅Pillow的 `rotate `_ 功能以了解更多用法。 + **参数：** - **degrees** (Union[int, float, sequence]) - 旋转角度的随机选取范围。若输入单个数字，则从(-degrees, degrees)中随机生成旋转角度；若输入2元素序列，需按(min, max)顺序排列。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Resize.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Resize.rst index f840f73b8bf..6a6a9263245 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Resize.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.Resize.rst @@ -1,7 +1,7 @@ mindspore.dataset.vision.py_transforms.Resize ============================================= -.. py:class:: mindspore.dataset.vision.py_transforms.Resize(size, interpolation=) +.. py:class:: mindspore.dataset.vision.py_transforms.Resize(size, interpolation=Inter.BILINEAR) 将输入PIL图像放缩为指定大小。 diff --git a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.ToTensor.rst b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.ToTensor.rst index 7f66b2033b0..ab1190ad46f 100644 --- a/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.ToTensor.rst +++ b/docs/api/api_python/dataset_vision/mindspore.dataset.vision.py_transforms.ToTensor.rst @@ -1,10 +1,12 @@ mindspore.dataset.vision.py_transforms.ToTensor =============================================== -.. py:class:: mindspore.dataset.vision.py_transforms.ToTensor(output_type=) +.. py:class:: mindspore.dataset.vision.py_transforms.ToTensor(output_type=numpy.float32) 将输入的PIL或numpy.ndarray图像转换为指定数据类型的numpy.ndarray图像，此时像素值取值将由[0, 255]变为[0.0, 1.0]，图像的shape将由(H, W, C)变为(C, H, W)。 + .. note:: 输入图像中的像素值将从 [0, 255] 缩放为 [0.0, 1.0]。图像的数据类型将被转换为由 `output_type` 参数指定的类型，图像通道数保持不变。 + **参数：** - **output_type** (numpy.dtype，可选) - 输出图像的数据类型，默认值：numpy.float32。 diff --git a/mindspore/python/mindspore/dataset/engine/datasets.py b/mindspore/python/mindspore/dataset/engine/datasets.py index 30eb093fecb..8b7fa62a3b5 100644 --- a/mindspore/python/mindspore/dataset/engine/datasets.py +++ b/mindspore/python/mindspore/dataset/engine/datasets.py @@ -557,9 +557,9 @@ class Dataset: >>> # and drops the last incomplete batch if there is one. >>> dataset = dataset.batch(100, True) >>> # resize image according to its batch number, if it's 5-th batch, resize to (5^2, 5^2) = (25, 25) - >>> def np_resize(col, batchInfo): + >>> def np_resize(col, BatchInfo): ... output = col.copy() - ... s = (batchInfo.get_batch_num() + 1) ** 2 + ... s = (BatchInfo.get_batch_num() + 1) ** 2 ... index = 0 ... for c in col: ... img = Image.fromarray(c.astype('uint8')).convert('RGB') diff --git a/mindspore/python/mindspore/dataset/engine/datasets_audio.py b/mindspore/python/mindspore/dataset/engine/datasets_audio.py index dfad8df35a0..26a7496e2d3 100644 --- a/mindspore/python/mindspore/dataset/engine/datasets_audio.py +++ b/mindspore/python/mindspore/dataset/engine/datasets_audio.py @@ -64,13 +64,13 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset): Raises: RuntimeError: If source raises an exception during execution. - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - CMUArctic dataset doesn't support PKSampler. @@ -199,13 +199,13 @@ class GTZANDataset(MappableDataset, AudioBaseDataset): Raises: RuntimeError: If source raises an exception during execution. - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - GTZAN doesn't support PKSampler. @@ -334,13 +334,13 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset): Raises: RuntimeError: If source raises an exception during execution. - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - LibriTTS dataset doesn't support PKSampler. @@ -472,19 +472,19 @@ class LJSpeechDataset(MappableDataset, AudioBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -617,13 +617,13 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -748,19 +748,19 @@ class TedliumDataset(MappableDataset, AudioBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain stm files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain stm files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -949,13 +949,13 @@ class YesNoDataset(MappableDataset, AudioBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. diff --git a/mindspore/python/mindspore/dataset/engine/datasets_standard_format.py b/mindspore/python/mindspore/dataset/engine/datasets_standard_format.py index 7351fea98e6..88643e2f533 100644 --- a/mindspore/python/mindspore/dataset/engine/datasets_standard_format.py +++ b/mindspore/python/mindspore/dataset/engine/datasets_standard_format.py @@ -68,18 +68,18 @@ class CSVDataset(SourceDataset, UnionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: RuntimeError: If dataset_files are not valid or do not exist. ValueError: If field_delim is invalid. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> csv_dataset_dir = ["/path/to/csv_dataset_file"] # contains 1 or multiple csv files @@ -129,8 +129,8 @@ class MindDataset(MappableDataset, UnionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. sampler (Sampler, optional): Object used to choose samples from the dataset (default=None, sampler is exclusive with shuffle and block_reader). Support list: SubsetRandomSampler, @@ -146,10 +146,10 @@ class MindDataset(MappableDataset, UnionBaseDataset): Raises: ValueError: If dataset_files are not valid or do not exist. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -270,21 +270,21 @@ class TFRecordDataset(SourceDataset, UnionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. shard_equal_rows (bool, optional): Get equal rows for all shards(default=False). If shard_equal_rows is false, number of rows of each shard may be not equal, and may lead to a failure in distributed training. When the number of samples of per TFRecord file are not equal, it is suggested to set to true. - This argument should only be specified when num_shards is also specified. + This argument should only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: ValueError: If dataset_files are not valid or do not exist. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> from mindspore import dtype as mstype diff --git a/mindspore/python/mindspore/dataset/engine/datasets_text.py b/mindspore/python/mindspore/dataset/engine/datasets_text.py index 2f6c8d2755e..f6a516c93f2 100644 --- a/mindspore/python/mindspore/dataset/engine/datasets_text.py +++ b/mindspore/python/mindspore/dataset/engine/datasets_text.py @@ -62,8 +62,8 @@ class AGNewsDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). @@ -154,18 +154,18 @@ class AmazonReviewDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the mindspore.dataset.config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> amazon_review_dataset_dir = "/path/to/amazon_review_dataset_dir" @@ -245,8 +245,8 @@ class CLUEDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). @@ -396,10 +396,10 @@ class CLUEDataset(SourceDataset, TextBaseDataset): ValueError: If dataset_files are not valid or do not exist. ValueError: task is not in 'AFQMC', 'TNEWS', 'IFLYTEK', 'CMNLI', 'WSC' or 'CSL'. ValueError: usage is not in 'train', 'test' or 'eval'. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> clue_dataset_dir = ["/path/to/clue_dataset_file"] # contains 1 or multiple clue files @@ -479,18 +479,18 @@ class CoNLL2000Dataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> conll2000_dataset_dir = "/path/to/conll2000_dataset_dir" @@ -541,17 +541,17 @@ class DBpediaDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> dbpedia_dataset_dir = "/path/to/dbpedia_dataset_directory" @@ -630,8 +630,8 @@ class EnWik9Dataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). @@ -705,19 +705,19 @@ class IMDBDataset(MappableDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - The shape of the test column. @@ -855,18 +855,18 @@ class IWSLT2016Dataset(SourceDataset, TextBaseDataset): - Shuffle.FILES: Shuffle files only. num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> iwslt2016_dataset_dir = "/path/to/iwslt2016_dataset_dir" @@ -976,18 +976,18 @@ class IWSLT2017Dataset(SourceDataset, TextBaseDataset): - Shuffle.FILES: Shuffle files only. num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> iwslt2017_dataset_dir = "/path/to/iwslt207_dataset_dir" @@ -1075,18 +1075,18 @@ class Multi30kDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If usage is not 'train', 'test', 'valid' or 'all'. + RuntimeError: If `dataset_dir` does not contain data files. + RuntimeError: If `usage` is not 'train', 'test', 'valid' or 'all'. RuntimeError: If the length of language_pair is not equal to 2. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. RuntimeError: If num_samples is less than 0. Examples: @@ -1174,8 +1174,8 @@ class PennTreebankDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). @@ -1261,18 +1261,18 @@ class SogouNewsDataset(SourceDataset, TextBaseDataset): - Shuffle.FILES: Shuffle files only. num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> sogou_news_dataset_dir = "/path/to/sogou_news_dataset_dir" @@ -1353,16 +1353,16 @@ class SQuADDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> squad_dataset_dir = "/path/to/squad_dataset_file" @@ -1457,17 +1457,17 @@ class TextFileDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: ValueError: If dataset_files are not valid or do not exist. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> text_file_dataset_dir = ["/path/to/text_file_dataset_file"] # contains 1 or multiple text files @@ -1514,18 +1514,18 @@ class UDPOSDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> udpos_dataset_dir = "/path/to/udpos_dataset_dir" @@ -1570,8 +1570,8 @@ class WikiTextDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). @@ -1654,17 +1654,17 @@ class YahooAnswersDataset(SourceDataset, TextBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> yahoo_answers_dataset_dir = "/path/to/yahoo_answers_dataset_directory" @@ -1745,18 +1745,18 @@ class YelpReviewDataset(SourceDataset, TextBaseDataset): - Shuffle.FILES: Shuffle files only. num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. num_parallel_workers (int, optional): Number of workers to read the data (default=None, number set in the config). cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. Examples: >>> yelp_review_dataset_dir = "/path/to/yelp_review_dataset_dir" diff --git a/mindspore/python/mindspore/dataset/engine/datasets_user_defined.py b/mindspore/python/mindspore/dataset/engine/datasets_user_defined.py index 3b5fd5dca0b..c40923e63c4 100644 --- a/mindspore/python/mindspore/dataset/engine/datasets_user_defined.py +++ b/mindspore/python/mindspore/dataset/engine/datasets_user_defined.py @@ -476,7 +476,7 @@ class GeneratorDataset(MappableDataset, UnionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). Random accessible input is required. When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This argument must be specified only + shard_id (int, optional): The shard ID within `num_shards` (default=None). This argument must be specified only when num_shards is also specified. Random accessible input is required. python_multiprocessing (bool, optional): Parallelize Python operations with multiple worker process. This option could be beneficial if the Python operation is computational heavy (default=True). @@ -486,12 +486,12 @@ class GeneratorDataset(MappableDataset, UnionBaseDataset): Raises: RuntimeError: If source raises an exception during execution. RuntimeError: If len of column_names does not match output len of source. - ValueError: If num_parallel_workers exceeds the max thread numbers. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. ValueError: If sampler and shuffle are specified at the same time. ValueError: If sampler and sharding are specified at the same time. ValueError: If num_shards is specified but shard_id is None. - ValueError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If shard_id is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - Input `source` accept user defined Python function(PyFuncs), Do not add network computing operators from @@ -801,7 +801,7 @@ class NumpySlicesDataset(GeneratorDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). Random accessible input is required. When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This argument must be specified only + shard_id (int, optional): The shard ID within `num_shards` (default=None). This argument must be specified only when num_shards is also specified. Random accessible input is required. Note: @@ -836,12 +836,12 @@ class NumpySlicesDataset(GeneratorDataset): Raises: RuntimeError: If len of column_names does not match output len of data. - ValueError: If num_parallel_workers exceeds the max thread numbers. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. ValueError: If sampler and shuffle are specified at the same time. ValueError: If sampler and sharding are specified at the same time. ValueError: If num_shards is specified but shard_id is None. - ValueError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If shard_id is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> # 1) Input data can be a list diff --git a/mindspore/python/mindspore/dataset/engine/datasets_vision.py b/mindspore/python/mindspore/dataset/engine/datasets_vision.py index 3f94983d624..54f027e1edc 100644 --- a/mindspore/python/mindspore/dataset/engine/datasets_vision.py +++ b/mindspore/python/mindspore/dataset/engine/datasets_vision.py @@ -122,9 +122,9 @@ class Caltech101Dataset(GeneratorDataset): dataset_dir (str): Path to the root directory that contains the dataset. This root directory contains two subdirectories, one is called 101_ObjectCategories, which stores images, and the other is called Annotations, which stores annotations. - target_type (str, optional): Target of the image. If target_type is 'category', return category represents - the target class. If target_type is 'annotation', return annotation. - If target_type is 'all', return category and annotation (default=None, means 'category'). + target_type (str, optional): Target of the image. If `target_type` is 'category', return category represents + the target class. If `target_type` is 'annotation', return annotation. + If `target_type` is 'all', return category and annotation (default=None, means 'category'). num_samples (int, optional): The number of images to be included in the dataset (default=None, all images). num_parallel_workers (int, optional): Number of workers to read the data (default=1). @@ -136,18 +136,18 @@ class Caltech101Dataset(GeneratorDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. Raises: - RuntimeError: If dataset_dir does not contain data files. - ValueError: If target_type is not set correctly. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `target_type` is not set correctly. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -295,19 +295,20 @@ class Caltech256Dataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `target_type` is not 'category', 'annotation' or 'all'. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -429,14 +430,14 @@ class CelebADataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - ValueError: If usage is not 'train', 'valid', 'test' or 'all'. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + ValueError: If `usage` is not 'train', 'valid', 'test' or 'all'. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -589,20 +590,20 @@ class Cifar10Dataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - ValueError: If usage is not 'train', 'test' or 'all'. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + ValueError: If `usage` is not 'train', 'test' or 'all'. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -720,20 +721,20 @@ class Cifar100Dataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - ValueError: If usage is not 'train', 'test' or 'all'. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + ValueError: If `usage` is not 'train', 'test' or 'all'. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -848,23 +849,23 @@ class CityscapesDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir is invalid or does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If dataset_dir is not exist. - ValueError: If task is invalid. - ValueError: If quality_mode is invalid. - ValueError: If usage is invalid. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is invalid or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `dataset_dir` is not exist. + ValueError: If `task` is invalid. + ValueError: If `quality_mode` is invalid. + ValueError: If `usage` is invalid. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -1015,8 +1016,8 @@ class CocoDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). extra_metadata(bool, optional): Flag to add extra meta-data to row. If True, an additional column will be @@ -1059,17 +1060,17 @@ class CocoDataset(MappableDataset, VisionBaseDataset): +-------------------------+----------------------------------------------+ Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. RuntimeError: If parse JSON file failed. - ValueError: If num_parallel_workers exceeds the max thread numbers. - ValueError: If task is not in ['Detection', 'Stuff', 'Panoptic', 'Keypoint', 'Captioning']. - ValueError: If annotation_file is not exist. - ValueError: If dataset_dir is not exist. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + ValueError: If `task` is not in ['Detection', 'Stuff', 'Panoptic', 'Keypoint', 'Captioning']. + ValueError: If `annotation_file` is not exist. + ValueError: If `dataset_dir` is not exist. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - Column '[_meta-filename, dtype=string]' won't be output unless an explicit rename dataset op is added @@ -1261,25 +1262,25 @@ class DIV2KDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir is invalid or does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If dataset_dir is not exist. - ValueError: If usage is invalid. - ValueError: If downgrade is invalid. - ValueError: If scale is invalid. - ValueError: If scale equal to 8 and downgrade not equal to 'bicubic'. - ValueError: If downgrade in ['mild', 'difficult', 'wild'] and scale not equal to 4. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is invalid or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `dataset_dir` is not exist. + ValueError: If `usage` is invalid. + ValueError: If `downgrade` is invalid. + ValueError: If `scale` is invalid. + ValueError: If `scale` equal to 8 and downgrade not equal to 'bicubic'. + ValueError: If `downgrade` in ['mild', 'difficult', 'wild'] and `scale` not equal to 4. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -1450,11 +1451,11 @@ class EMnistDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -1581,12 +1582,12 @@ class FakeImageDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. 'sampler' and 'shuffle' are mutually exclusive. @@ -1670,13 +1671,13 @@ class FashionMnistDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -1786,21 +1787,21 @@ class FlickrDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir is not valid or does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If dataset_dir is not exist. - ValueError: If annotation_file is not exist. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is not valid or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `dataset_dir` is not exist. + ValueError: If `annotation_file` is not exist. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -2029,17 +2030,17 @@ class Flowers102Dataset(GeneratorDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). Random accessible input is required. When this argument is specified, 'num_samples' reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This argument must be specified only + shard_id (int, optional): The shard ID within `num_shards` (default=None). This argument must be specified only when num_shards is also specified. Random accessible input is required. Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. 'sampler' and 'shuffle' are mutually exclusive. @@ -2197,20 +2198,20 @@ class ImageFolderDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. RuntimeError: If class_indexing is not a dictionary. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - The shape of the image column is [image_size] if decode flag is False, or [H,W,C] otherwise. @@ -2582,13 +2583,13 @@ class ManifestDataset(MappableDataset, VisionBaseDataset): Raises: RuntimeError: If dataset_files are not valid or do not exist. - ValueError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. RuntimeError: If class_indexing is not a dictionary. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - The shape of the image column is [image_size] if decode flag is False, or [H,W,C] otherwise. @@ -2697,13 +2698,14 @@ class MnistDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + ValueError: If `usage` is not 'train'、'test' or 'all'. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but shard_id is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -2824,17 +2826,17 @@ class PhotoTourDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If dataset_dir is not exist. - ValueError: If usage is not in ["train", "test"]. + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `dataset_dir` is not exist. + ValueError: If `usage` is not in ["train", "test"]. ValueError: If name is not in ["notredame", "yosemite", "liberty", "notredame_harris", "yosemite_harris", "liberty_harris"]. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. `sampler` and `shuffle` are mutually exclusive. The table @@ -2972,15 +2974,15 @@ class Places365Dataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). - ValueError: If usage is not in ["train-standard", "train-challenge", "val"]. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). + ValueError: If `usage` is not in ["train-standard", "train-challenge", "val"]. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. 'sampler' and 'shuffle' are mutually exclusive. @@ -3114,13 +3116,13 @@ class QMnistDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -3230,8 +3232,8 @@ class RandomDataset(SourceDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. """ @check_random_dataset @@ -3335,20 +3337,20 @@ class SBDataset(GeneratorDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. Raises: - RuntimeError: If dataset_dir is not valid or does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If dataset_dir is not exist. - ValueError: If task is not in ['Boundaries', 'Segmentation']. - ValueError: If usage is not in ['train', 'val', 'train_noval', 'all']. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is not valid or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `dataset_dir` is not exist. + ValueError: If `task` is not in ['Boundaries', 'Segmentation']. + ValueError: If `usage` is not in ['train', 'val', 'train_noval', 'all']. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. `sampler` and `shuffle` are mutually exclusive. @@ -3469,13 +3471,13 @@ class SBUDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. 'sampler' and 'shuffle' are mutually exclusive. @@ -3577,18 +3579,18 @@ class SemeionDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing (default=None, which means no cache is used). Raises: - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. @@ -3703,20 +3705,20 @@ class STL10Dataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, 'num_samples' reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir is not valid or does not exist or does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If usage is invalid. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is not valid or does not exist or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `usage` is invalid. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. 'sampler' and 'shuffle' are mutually exclusive. @@ -3869,18 +3871,18 @@ class SVHNDataset(GeneratorDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). Random accessible input is required. When this argument is specified, 'num_samples' reflects the max sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This argument must be specified only + shard_id (int, optional): The shard ID within `num_shards` (default=None). This argument must be specified only when num_shards is also specified. Random accessible input is required. Raises: - RuntimeError: If dataset_dir is not valid or does not exist or does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If usage is invalid. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is not valid or does not exist or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `usage` is invalid. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a sampler. 'sampler' and 'shuffle' are mutually exclusive. @@ -3995,14 +3997,14 @@ class USPSDataset(SourceDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir is not valid or does not exist or does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If usage is invalid. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` is not valid or does not exist or does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `usage` is invalid. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Examples: >>> usps_dataset_dir = "/path/to/usps_dataset_directory" @@ -4091,27 +4093,27 @@ class VOCDataset(MappableDataset, VisionBaseDataset): num_shards (int, optional): Number of shards that the dataset will be divided into (default=None). When this argument is specified, `num_samples` reflects the maximum sample number of per shard. - shard_id (int, optional): The shard ID within num_shards (default=None). This - argument can only be specified when num_shards is also specified. + shard_id (int, optional): The shard ID within `num_shards` (default=None). This + argument can only be specified when `num_shards` is also specified. cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. (default=None, which means no cache is used). extra_metadata(bool, optional): Flag to add extra meta-data to row. If True, an additional column named :py:obj:`[_meta-filename, dtype=string]` will be output at the end (default=False). Raises: - RuntimeError: If dataset_dir does not contain data files. + RuntimeError: If `dataset_dir` does not contain data files. RuntimeError: If xml of Annotations is an invalid format. RuntimeError: If xml of Annotations loss attribution of `object`. RuntimeError: If xml of Annotations loss attribution of `bndbox`. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If num_parallel_workers exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. ValueError: If task is not equal 'Segmentation' or 'Detection'. ValueError: If task equal 'Segmentation' but class_indexing is not None. ValueError: If txt related to mode is not exist. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - Column '[_meta-filename, dtype=string]' won't be output unless an explicit rename dataset op @@ -4298,17 +4300,17 @@ class WIDERFaceDataset(MappableDataset, VisionBaseDataset): (default=None, which means no cache is used). Raises: - RuntimeError: If dataset_dir does not contain data files. - RuntimeError: If num_parallel_workers exceeds the max thread numbers. - RuntimeError: If sampler and shuffle are specified at the same time. - RuntimeError: If sampler and sharding are specified at the same time. - RuntimeError: If num_shards is specified but shard_id is None. - RuntimeError: If shard_id is specified but num_shards is None. - ValueError: If shard_id is invalid (< 0 or >= num_shards). - ValueError: If usage is not in ['train', 'test', 'valid', 'all']. - ValueError: If annotation_file is not exist. - ValueError: If dataset_dir is not exist. - ValueError: If shard_id is invalid (< 0 or >= num_shards). + RuntimeError: If `dataset_dir` does not contain data files. + ValueError: If `num_parallel_workers` exceeds the max thread numbers. + RuntimeError: If `sampler` and `shuffle` are specified at the same time. + RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time. + RuntimeError: If `num_shards` is specified but `shard_id` is None. + RuntimeError: If `shard_id` is specified but `num_shards` is None. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). + ValueError: If `usage` is not in ['train', 'test', 'valid', 'all']. + ValueError: If `annotation_file` is not exist. + ValueError: If `dataset_dir` is not exist. + ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`). Note: - This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive. diff --git a/mindspore/python/mindspore/dataset/text/transforms.py b/mindspore/python/mindspore/dataset/text/transforms.py index d34d05f6380..75b17f3102a 100644 --- a/mindspore/python/mindspore/dataset/text/transforms.py +++ b/mindspore/python/mindspore/dataset/text/transforms.py @@ -366,7 +366,7 @@ class SentencePieceTokenizer(TextTensorOperation): Raises: TypeError: If `mode` is not of type string or SentencePieceVocab. - TypError: If `out_type` is not of type SPieceTokenizerOutType. + TypeError: If `out_type` is not of type SPieceTokenizerOutType. Supported Platforms: ``CPU`` @@ -401,7 +401,7 @@ class SlidingWindow(TextTensorOperation): Raises: TypeError: If `width` is not of type int. ValueError: If value of `width` is not positive. - TypError: If `axis` is not of type int. + TypeError: If `axis` is not of type int. Supported Platforms: ``CPU``