vilc
/
mindspore2022
forked from huawei/mindspore2022
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,576 Commits 18 Branches 22 Tags 3.6 GiB
Commit Graph

55 Commits

Author SHA1 Message Date
Peilin Wang 848e07d022 initial commit, start of BucketBatchByLengthOp 
c implementation done, just need to call batch/pad

added python api and validator

added pybind/de_pipeline stuff, fixed some compile errors, figure out how null py::function works

added tiny bit of doc

integrated with static batch methods

fixed some bugs

some more bug fixes and cleanup

ci fix

fix ci

ci fix

fix ci

added test_cases and debugged

addressed code review comments

addressed code review comments

ci fix

ci fix

addressed code review comments

addressed code review comments
 2020-06-15 10:20:01 -04:00
mindspore-ci-bot a8cf83acf2 !1932 Add CLUE dataset 
Merge pull request !1932 from jiangzhiwen/dataset/clue
 2020-06-15 11:48:35 +08:00
jiangzhiwen e0e167a000 add CLUE dataset 2020-06-15 10:40:46 +08:00
mindspore-ci-bot c661087b99 !2029 Create TensorRow class that supports a row id 
Merge pull request !2029 from lixiachen/lixia/tensorrow
 2020-06-13 09:43:58 +08:00
Lixia Chen 6ae88c3987 Create TensorRow class that supports a row id. 2020-06-12 15:06:56 -04:00
mindspore-ci-bot 1b5d9a1194 !1981 Implemented from_dataset, a way to build a vocab from existing dataset 
Merge pull request !1981 from ZiruiWu/from_dataset_dev
 2020-06-12 08:08:19 +08:00
Zirui Wu 880ce5ea26 implemented from_dataset 
fix complie error

more tests

address CI complains

fix ci

adress review comments

address review cmts
 2020-06-11 16:20:19 -04:00
Alexey Shevlyakov 4e333a2f22 MindData profiler infrastructure 2020-06-11 14:52:36 -04:00
mindspore-ci-bot b3da41bd7a !1913 CocoDataset implementation 
Merge pull request !1913 from xiefangqi/xfq_support_coco
 2020-06-11 14:37:44 +08:00
xiefangqi 097cbcc0cb CocoDataset implementation 2020-06-11 11:52:17 +08:00
heleiwang 3ece8dd090 1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API 
2. mod cora and citeseer conversion script
 2020-06-10 15:52:26 +08:00
liyong feff8899ac support padding samples 2020-06-05 09:14:42 +08:00
mindspore-ci-bot 5c21616293 !1807 Implemented Ngram TensorOp for dataset 
Merge pull request !1807 from ZiruiWu/ngram_dev
 2020-06-05 01:04:27 +08:00
Zirui Wu dbf9936ec4 Implemented n-gram for dataset TensorOp 2020-06-04 09:57:44 -04:00
Jamie Nisbet 51bc0c0460 consistent design for num_samples 
updates

more update

more work

more fixin

post rebase updates

clang formatting

code review recovery

ci fixes

updates

update

update

update
 2020-06-03 15:01:18 -04:00
xiefangqi 5e4728c50f add voc support split 2020-05-28 11:18:32 +08:00
anthony ba2eaea7e1 Changed uniform_aug_op py::list to std::vector with some cleanup and enhancement. 2020-05-27 16:17:48 -04:00
mindspore-ci-bot 458436186c !1365 Clean up work for text python sub-package 
Merge pull request !1365 from h.farahat/text_namespace
 2020-05-22 23:16:31 +08:00
hesham 6c21e556c4 Clean up work for text python package 2020-05-22 04:35:58 -04:00
heleiwang 599a449e0b Support processing GNN data 2020-05-22 14:15:25 +08:00
mindspore-ci-bot 58e6d7d950 !1341 Added lookup and vocab to mindspore.dataset.text 
Merge pull request !1341 from ZiruiWu/vocab_and_lookup
 2020-05-22 10:19:42 +08:00
mindspore-ci-bot 2e3d55ed87 !1281 Implementation of SplitOp 
Merge pull request !1281 from Peilin/splitOp
 2020-05-22 09:29:03 +08:00
Peilin Wang 71e8bb1960 general split case done, chaining sampler (basic case) is working 
implementation 99% complete

everything and tested except for repeatable shuffling

tested most basic/typical split usecases

cleanup

some more cleanup

fix CI

more ci fix

more ci fixes

more ci fix

more ci fix

more ci fix

added more tests, fixed some bugs

some more clean up and test cases

added shard/shuffle before split warning/error

addressed code review comments and ci

fixed ci
 2020-05-21 20:42:51 -04:00
Zirui Wu 25ab2ef303 Implemented lookup and vocab 2020-05-21 17:17:24 -04:00
qianlong 451c20a6f5 Add UnicodeCharTokenizer for nlp 2020-05-21 09:22:45 +08:00
xulei2020 163b6b7ea7 add jieba c++ code 2020-05-20 15:55:12 +08:00
hesham e8ca243364 -Add DE_STRING 
-replace switch'case by indexing

- Add test case
- Add constructors
- Add getItem string

- Fix bugs
- Add more tests

- Tensor iterator
- asNumpy
- TextFileDataset

- Tensor(Numpy)

- Super > 2D
- Add more test cases for GeneratorDataset

- Change StartAddr to GetBuffer and GetMutableNuffer

- Raise an error if batch is used with strings

Clean-up work
 2020-05-15 20:33:28 -04:00
mindspore-ci-bot c680cfbf27 !1157 dataset: add concat operation for dataset 
Merge pull request !1157 from ms_yan/concat_dataset
 2020-05-15 16:07:19 +08:00
mindspore-ci-bot ab031ee9ea !1126 VOCDataset support object detection function 
Merge pull request !1126 from xiefangqi/voc_support_detection
 2020-05-15 15:56:39 +08:00
xiefangqi c937bad53f minddata support voc 2020-05-15 13:24:03 +08:00
ms_yan c0fa7b4b19 init commit of concat dataset 
change to use __add__ operation instead ds.concat
 2020-05-15 13:14:13 +08:00
liyong aa3f89e74f mindrecord support read file list 2020-05-13 14:11:59 +08:00
xiefangqi 9a2daae84c del unused change_mode_op 2020-05-06 11:54:05 +08:00
Jesse Lee 270bf831a9 Random Data Op 2020-04-29 10:26:00 -04:00
liyong bfba630aa2 update pK_sampler 2020-04-27 21:40:49 +08:00
Alexey Shevlyakov 6acae622dc fix random seed behaviour 2020-04-22 14:25:57 -04:00
mindspore-ci-bot dc0491caf9 !508 [Dataset] Adding sync_wait operator for dataset 
Merge pull request !508 from EricZ/master
 2020-04-22 06:15:24 +08:00
mindspore-ci-bot b0f4b36f48 !511 Uniform Augmentation C++ OP 
Merge pull request !511 from AdelShafiei/uniform_augmentation
 2020-04-22 06:06:04 +08:00
eric cd94518769 X# This is a combination of 2 commits. 
Initial commit for dataset op python

Added signature to barrier

Adde compiling barrier code

Rebasing, fixed new compile errors

Final fix for make_unique

Added pybind API for barrier

Fixed pyfunc invocation

python interface - sync_wait

!1 sync_wait python interface
* python interface - sync_wait

fix test

update test

update test

Added new test case

add test case

test for shuffle + batch

Added two-sync test case

Restrited that no shuffle after sync

Added sync to pipeline info

block first databuffer as well

Intelligently get batch size

Fix default case

Lock Pair shares among all iterators

Added fix for empty character

Fixed up test case formatting

Fix end of epoch in sync_wait

Fixing CI
 2020-04-21 17:42:13 -04:00
Adel Shafiei 5fcd3f01a6 Added C++ UniformAugOp support 2020-04-21 17:04:05 -04:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
mindspore-ci-bot 822a3160e4 !404 [Dataset] Add Python Sampler support for CPP dataset 
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
 2020-04-20 22:45:51 +08:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
Junhan Hu 43a2e99833 Add python sampler support for CPP dataset 2020-04-19 20:36:37 -04:00
jzw 3f7054dccb add skip dataset op 2020-04-17 19:48:25 +08:00
mindspore-ci-bot c0c0b0985e !378 Multiple Iterators can cause performance problem 
Merge pull request !378 from h.farahat/multi_itr
 2020-04-17 04:18:44 +08:00
hesham fb6c7ba2e1 Fix two problem when we create multiple instances of the same dataset (2 for-loops) 
-- Iterator list is keeping  all created iterators wihtout cleaning them up
-- alter tree modifies the original.
 2020-04-16 15:04:29 -04:00
mindspore-ci-bot cf026096a6 !183 Mindspore.dataset CPP sampler for GeneratorDataset 
Merge pull request !183 from JunhanHu/cpp_sampler
 2020-04-16 22:30:59 +08:00
chenjianping 1286767d0e support building on windows 2020-04-15 09:36:48 +00:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00