vilc
/
mindspore2022
forked from huawei/mindspore2022
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,217 Commits 18 Branches 22 Tags 3.6 GiB
Commit Graph

44 Commits

Author SHA1 Message Date
mindspore-ci-bot f82e63fecc !671 Added testcase for sync_wait 
Merge pull request !671 from EricZ/master
 2020-04-27 22:14:10 +08:00
mindspore-ci-bot c47ef8ee4e !711 Address first epoch always shuffle with same seed init of random sampler 
Merge pull request !711 from ZiruiWu/mindspore_master_only
 2020-04-27 21:39:12 +08:00
mindspore-ci-bot e6473f0765 !736 Skip blank line for TextFileDataset 
Merge pull request !736 from yanghaitao/yht_tftextfile_black_line
 2020-04-27 19:02:08 +08:00
yanghaitao 1cb3bead32 skip black line for TextFileDataset 2020-04-27 14:54:26 +08:00
Zirui Wu 8c3931cf1d fix first epoch always shuffle with default seed in random sampler 
fix ut
 2020-04-26 10:58:59 -04:00
zhoufeng b681cec8f2 cmake refactor 2020-04-26 20:51:18 +08:00
eric 2d115cd04e Added example for multiple iterator 
Added new testcase for multi iterator

Addressing review

Fixed typo
 2020-04-24 17:28:40 -04:00
xiefangqi abd4239da9 fix random stuck problem 2020-04-22 21:13:33 +08:00
xulei2020 b5b506c4c3 add code 2020-04-22 11:10:15 +08:00
eric cd94518769 X# This is a combination of 2 commits. 
Initial commit for dataset op python

Added signature to barrier

Adde compiling barrier code

Rebasing, fixed new compile errors

Final fix for make_unique

Added pybind API for barrier

Fixed pyfunc invocation

python interface - sync_wait

!1 sync_wait python interface
* python interface - sync_wait

fix test

update test

update test

Added new test case

add test case

test for shuffle + batch

Added two-sync test case

Restrited that no shuffle after sync

Added sync to pipeline info

block first databuffer as well

Intelligently get batch size

Fix default case

Lock Pair shares among all iterators

Added fix for empty character

Fixed up test case formatting

Fix end of epoch in sync_wait

Fixing CI
 2020-04-21 17:42:13 -04:00
qianlong db80f4ff92 The num_samples and numRows in schema for TFRecordDataset are conflict 2020-04-21 17:08:59 +08:00
mindspore-ci-bot d9e4dcc33b !483 Optimize skip dataset op 
Merge pull request !483 from jiangzhiwen/dataset/skip_opt
 2020-04-21 15:10:23 +08:00
mindspore-ci-bot d307c264b9 !517 [Data]Delete sort logic for filter operation 
Merge pull request !517 from xulei/filter_master
 2020-04-21 15:04:32 +08:00
xulei2020 eb053a6233 add filterOp code 2020-04-21 14:06:23 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
jiangzhiwen e1b109e8b8 optimize skip dataset op 2020-04-21 09:16:37 +08:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
mindspore-ci-bot 98fbd30a5b !460 [Data]Add filter operation 
Merge pull request !460 from xulei/filter_master
 2020-04-20 23:33:44 +08:00
mindspore-ci-bot 822a3160e4 !404 [Dataset] Add Python Sampler support for CPP dataset 
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
 2020-04-20 22:45:51 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
mindspore-ci-bot d8176a77f4 !425 New api "TextFileDataset" 
Merge pull request !425 from yanghaitao/yht_textfiledatasetv2
 2020-04-20 22:14:26 +08:00
ms_yan c5cfb09e66 Repair some MS_LOG problem 2020-04-20 19:45:46 +08:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
Junhan Hu d0ae610832 Review 2020-04-19 20:36:37 -04:00
Junhan Hu 43a2e99833 Add python sampler support for CPP dataset 2020-04-19 20:36:37 -04:00
ms_yan f0c07c3fa6 Realize take op and add ut 2020-04-18 18:33:57 +08:00
xiefangqi d4822137f3 fix segment bug on windows 2020-04-18 15:52:38 +08:00
jzw 3f7054dccb add skip dataset op 2020-04-17 19:48:25 +08:00
mindspore-ci-bot cf026096a6 !183 Mindspore.dataset CPP sampler for GeneratorDataset 
Merge pull request !183 from JunhanHu/cpp_sampler
 2020-04-16 22:30:59 +08:00
mindspore-ci-bot b2b3e24a8e !329 [MS]support building on windows 10 
Merge pull request !329 from chenjianping/building-on-windows-2
 2020-04-15 19:10:50 +08:00
chenjianping 1286767d0e support building on windows 2020-04-15 09:36:48 +00:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00
mindspore-ci-bot 30de261c3c !243 Support nested repeat 
Merge pull request !243 from h.farahat/nested_repeat
 2020-04-14 23:07:32 +08:00
hesham 0fc23eee0f Support nested repeat 2020-04-14 10:13:18 -04:00
mindspore-ci-bot d9cd681c30 !207 Fix potential overflow in batch by changing int32 to int64 
Merge pull request !207 from ZiruiWu/batch_info_overflow
 2020-04-14 01:55:35 +08:00
dengwentao b5ca2a3e30 :modify protobuf cmake 2020-04-11 11:46:23 +08:00
Zirui Wu 19f2ea6d41 change int32 to int64 to avoid overflow in batch 
fix clang

pybind fix
 2020-04-09 11:23:43 -04:00
mindspore-ci-bot cc75cb357c !168 remove mindspore::make_unique and make_unique.h 
Merge pull request !168 from Alexey_Shevlyakov/remove_mindspore_make_unique
 2020-04-09 05:32:52 +08:00
jonyguo 20d1b64443 fix: error info is not exactly when column list invalid 2020-04-08 16:11:13 +08:00
Jonathan Yan 9d0fde29f4 remove ENABLE_MINDRECORD flag 2020-04-07 14:55:45 -04:00
Alexey Shevlyakov 84d780c1a4 remove make_unique.h 2020-04-07 14:00:09 -04:00
Peilin Wang 0ae77bb0db TFReaderOp fix, threads will exit after reading necessary amount of rows 
changes from yanpanhui 524009: added set_dataset_size and changed get_dataest_size according to ME requirements

CI fixes
 2020-03-31 13:29:54 -04:00
zhunaipan 930a1fb0a8 initial version 
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
 2020-03-27 22:54:54 +08:00