278 lines
10 KiB
Plaintext
278 lines
10 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 文本变换样例库\n",
|
||
"\n",
|
||
"[](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/docs/api_python/samples/dataset/text_gallery.ipynb) \n",
|
||
"[](https://gitee.com/mindspore/mindspore/blob/master/docs/api/api_python/samples/dataset/text_gallery.ipynb)\n",
|
||
"\n",
|
||
"此指南展示了[mindspore.dataset.text](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#%E6%96%87%E6%9C%AC)模块中各种变换的用法。"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 环境准备"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Downloading data from https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/bert-base-uncased-vocab.txt (226 kB)\n",
|
||
"\n",
|
||
"file_sizes: 100%|████████████████████████████| 232k/232k [00:00<00:00, 2.21MB/s]\n",
|
||
"Successfully downloaded file to ./bert-base-uncased-vocab.txt\n",
|
||
"Downloading data from https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/article.txt (9 kB)\n",
|
||
"\n",
|
||
"file_sizes: 100%|██████████████████████████| 9.06k/9.06k [00:00<00:00, 1.83MB/s]\n",
|
||
"Successfully downloaded file to ./article.txt\n",
|
||
"['text_gallery.ipynb', 'article.txt', 'bert-base-uncased-vocab.txt']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import os\n",
|
||
"from download import download\n",
|
||
"\n",
|
||
"import mindspore.dataset as ds\n",
|
||
"import mindspore.dataset.text as text\n",
|
||
"\n",
|
||
"# Download opensource datasets\n",
|
||
"# citation: https://www.kaggle.com/datasets/drknope/bertbaseuncasedvocab\n",
|
||
"url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/bert-base-uncased-vocab.txt\"\n",
|
||
"download(url, './bert-base-uncased-vocab.txt', replace=True)\n",
|
||
"\n",
|
||
"url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/article.txt\"\n",
|
||
"download(url, './article.txt', replace=True)\n",
|
||
"\n",
|
||
"# Show the directory\n",
|
||
"print(os.listdir())\n",
|
||
"\n",
|
||
"def call_op(op, input):\n",
|
||
" print(op(input), flush=True)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Vocab\n",
|
||
"\n",
|
||
"[mindspore.dataset.text.Vocab](https://mindspore.cn/docs/zh-CN/master/api_python/dataset_text/mindspore.dataset.text.Vocab.html#mindspore.dataset.text.Vocab) 用于存储多对字符与ID。其包含一个映射，可以将每个单词（str）映射到一个ID（int）。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"ids [18863, 18279]\n",
|
||
"tokens ['##nology', 'crystalline']\n",
|
||
"lookup: ids [18863 18279]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Load bert vocab\n",
|
||
"vocab_file = open(\"bert-base-uncased-vocab.txt\")\n",
|
||
"vocab_content = list(set(vocab_file.read().splitlines()))\n",
|
||
"vocab = text.Vocab.from_list(vocab_content)\n",
|
||
"\n",
|
||
"# lookup tokens to ids\n",
|
||
"ids = vocab.tokens_to_ids([\"good\", \"morning\"])\n",
|
||
"print(\"ids\", ids)\n",
|
||
"\n",
|
||
"# lookup ids to tokens\n",
|
||
"tokens = vocab.ids_to_tokens([128, 256])\n",
|
||
"print(\"tokens\", tokens)\n",
|
||
"\n",
|
||
"# Use Lookup op to lookup index\n",
|
||
"op = text.Lookup(vocab)\n",
|
||
"ids = op([\"good\", \"morning\"])\n",
|
||
"print(\"lookup: ids\", ids)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## AddToken\n",
|
||
"\n",
|
||
"[mindspore.dataset.text.AddToken](https://mindspore.cn/docs/zh-CN/master/api_python/dataset_text/mindspore.dataset.text.AddToken.html#mindspore.dataset.text.AddToken) 将分词(token)添加到序列的开头或结尾处。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"['TOKEN' 'a' 'b' 'c' 'd' 'e']\n",
|
||
"['a' 'b' 'c' 'd' 'e' 'END']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"txt = [\"a\", \"b\", \"c\", \"d\", \"e\"]\n",
|
||
"add_token_op = text.AddToken(token='TOKEN', begin=True)\n",
|
||
"call_op(add_token_op, txt)\n",
|
||
"\n",
|
||
"add_token_op = text.AddToken(token='END', begin=False)\n",
|
||
"call_op(add_token_op, txt)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## SentencePieceTokenizer\n",
|
||
"\n",
|
||
"[mindspore.dataset.text.SentencePieceTokenizer](https://mindspore.cn/docs/zh-CN/master/api_python/dataset_text/mindspore.dataset.text.SentencePieceTokenizer.html#mindspore.dataset.text.SentencePieceTokenizer) 使用SentencePiece分词器对字符串进行分词。\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Downloading data from https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/sentencepiece.bpe.model (4.8 MB)\n",
|
||
"\n",
|
||
"file_sizes: 100%|██████████████████████████| 5.07M/5.07M [00:01<00:00, 2.93MB/s]\n",
|
||
"Successfully downloaded file to ./sentencepiece.bpe.model\n",
|
||
"['▁Today' '▁is' '▁Tuesday' '.']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Construct a SentencePieceVocab model\n",
|
||
"url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/sentencepiece.bpe.model\"\n",
|
||
"download(url, './sentencepiece.bpe.model', replace=True)\n",
|
||
"sentence_piece_vocab_file = './sentencepiece.bpe.model'\n",
|
||
"\n",
|
||
"# Use the model to tokenize text\n",
|
||
"tokenizer = text.SentencePieceTokenizer(sentence_piece_vocab_file, out_type=text.SPieceTokenizerOutType.STRING)\n",
|
||
"txt = \"Today is Tuesday.\"\n",
|
||
"call_op(tokenizer, txt)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## WordpieceTokenizer\n",
|
||
"\n",
|
||
"[mindspore.dataset.text.WordpieceTokenizer](https://mindspore.cn/docs/zh-CN/master/api_python/dataset_text/mindspore.dataset.text.WordpieceTokenizer.html#mindspore.dataset.text.WordpieceTokenizer) 将输入的字符串切分为子词。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"['token' '##izer' 'will' 'outputs' 'sub' '##words']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Reuse the vocab defined above as input vocab\n",
|
||
"tokenizer = text.WordpieceTokenizer(vocab=vocab, unknown_token='[UNK]')\n",
|
||
"txt = [\"tokenizer\", \"will\", \"outputs\", \"subwords\"]\n",
|
||
"call_op(tokenizer, txt)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 在数据Pipeline中加载和处理TXT文件\n",
|
||
"\n",
|
||
"使用 [mindspore.dataset.TextFileDataset](https://mindspore.cn/docs/zh-CN/master/api_python/dataset/mindspore.dataset.TextFileDataset.html#mindspore.dataset.TextFileDataset) 将磁盘中的文本文件内容加载到数据Pipeline中，并应用分词器对其中的内容进行分词。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Load text content into dataset pipeline\n",
|
||
"text_file = \"article.txt\"\n",
|
||
"dataset = ds.TextFileDataset(dataset_files=text_file, shuffle=False)\n",
|
||
"\n",
|
||
"# check the column names inside the dataset\n",
|
||
"print(\"column names:\", dataset.get_col_names())\n",
|
||
"\n",
|
||
"# tokenize all text content into tokens with bert vocab\n",
|
||
"dataset = dataset.map(text.BertTokenizer(vocab=vocab), input_columns=[\"text\"])\n",
|
||
"\n",
|
||
"for data in dataset:\n",
|
||
" print(data)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "ly37",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.7.5"
|
||
},
|
||
"vscode": {
|
||
"interpreter": {
|
||
"hash": "9f0efe8a0d8ccef1406a56130f5ab5480567fb275f7fbf51bbc40aede97503df"
|
||
}
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
}
|