Bessie
/
mindspore2022
forked from huawei/mindspore2022
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,104 Commits 18 Branches 0 Tags 564 MiB
Commit Graph

59 Commits

Author SHA1 Message Date
Adel Shafiei f29eacbb34 fixed an input validation error for uniform augment op 2020-04-23 17:37:51 -04:00
mindspore-ci-bot fe67db75a3 !580 fix dataset & mindrecord log print 
Merge pull request !580 from panfengfeng/panff/fix_log_print_in_ge
 2020-04-23 14:26:29 +08:00
Alexey Shevlyakov 6acae622dc fix random seed behaviour 2020-04-22 14:25:57 -04:00
xiefangqi abd4239da9 fix random stuck problem 2020-04-22 21:13:33 +08:00
panfengfeng 6328edf455 modify dataset & mindrecord log compile 2020-04-22 17:10:40 +08:00
xulei2020 b5b506c4c3 add code 2020-04-22 11:10:15 +08:00
mindspore-ci-bot dc0491caf9 !508 [Dataset] Adding sync_wait operator for dataset 
Merge pull request !508 from EricZ/master
 2020-04-22 06:15:24 +08:00
mindspore-ci-bot b0f4b36f48 !511 Uniform Augmentation C++ OP 
Merge pull request !511 from AdelShafiei/uniform_augmentation
 2020-04-22 06:06:04 +08:00
eric cd94518769 X# This is a combination of 2 commits. 
Initial commit for dataset op python

Added signature to barrier

Adde compiling barrier code

Rebasing, fixed new compile errors

Final fix for make_unique

Added pybind API for barrier

Fixed pyfunc invocation

python interface - sync_wait

!1 sync_wait python interface
* python interface - sync_wait

fix test

update test

update test

Added new test case

add test case

test for shuffle + batch

Added two-sync test case

Restrited that no shuffle after sync

Added sync to pipeline info

block first databuffer as well

Intelligently get batch size

Fix default case

Lock Pair shares among all iterators

Added fix for empty character

Fixed up test case formatting

Fix end of epoch in sync_wait

Fixing CI
 2020-04-21 17:42:13 -04:00
Adel Shafiei 5fcd3f01a6 Added C++ UniformAugOp support 2020-04-21 17:04:05 -04:00
mindspore-ci-bot 3c307cf486 !552 Update clang format rule 
Merge pull request !552 from ZhouFeng/clang-rule-update
 2020-04-21 22:53:08 +08:00
zhoufeng c2b3360d69 update clang format rule 2020-04-21 21:21:19 +08:00
mindspore-ci-bot 6e6b3eeab0 !547 delete unsafe option 
Merge pull request !547 from xiefangqi/del_unsafe_option
 2020-04-21 21:07:25 +08:00
xiefangqi a931fbbae2 delete unsafe compile option 2020-04-21 19:46:10 +08:00
qianlong db80f4ff92 The num_samples and numRows in schema for TFRecordDataset are conflict 2020-04-21 17:08:59 +08:00
mindspore-ci-bot d9e4dcc33b !483 Optimize skip dataset op 
Merge pull request !483 from jiangzhiwen/dataset/skip_opt
 2020-04-21 15:10:23 +08:00
mindspore-ci-bot d307c264b9 !517 [Data]Delete sort logic for filter operation 
Merge pull request !517 from xulei/filter_master
 2020-04-21 15:04:32 +08:00
xulei2020 eb053a6233 add filterOp code 2020-04-21 14:06:23 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
jiangzhiwen e1b109e8b8 optimize skip dataset op 2020-04-21 09:16:37 +08:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
mindspore-ci-bot 98fbd30a5b !460 [Data]Add filter operation 
Merge pull request !460 from xulei/filter_master
 2020-04-20 23:33:44 +08:00
mindspore-ci-bot 822a3160e4 !404 [Dataset] Add Python Sampler support for CPP dataset 
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
 2020-04-20 22:45:51 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
mindspore-ci-bot d8176a77f4 !425 New api "TextFileDataset" 
Merge pull request !425 from yanghaitao/yht_textfiledatasetv2
 2020-04-20 22:14:26 +08:00
ms_yan c5cfb09e66 Repair some MS_LOG problem 2020-04-20 19:45:46 +08:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
Junhan Hu d0ae610832 Review 2020-04-19 20:36:37 -04:00
Junhan Hu 43a2e99833 Add python sampler support for CPP dataset 2020-04-19 20:36:37 -04:00
ms_yan f0c07c3fa6 Realize take op and add ut 2020-04-18 18:33:57 +08:00
xiefangqi d4822137f3 fix segment bug on windows 2020-04-18 15:52:38 +08:00
jzw 3f7054dccb add skip dataset op 2020-04-17 19:48:25 +08:00
mindspore-ci-bot c0c0b0985e !378 Multiple Iterators can cause performance problem 
Merge pull request !378 from h.farahat/multi_itr
 2020-04-17 04:18:44 +08:00
hesham fb6c7ba2e1 Fix two problem when we create multiple instances of the same dataset (2 for-loops) 
-- Iterator list is keeping  all created iterators wihtout cleaning them up
-- alter tree modifies the original.
 2020-04-16 15:04:29 -04:00
mindspore-ci-bot 7874e96b9b !377 Tensor end() iterator performance problem 
Merge pull request !377 from h.farahat/end_iterator
 2020-04-17 01:51:33 +08:00
hesham c91a58dc55 end() iterator is causing performance problem 2020-04-16 10:37:05 -04:00
mindspore-ci-bot cf026096a6 !183 Mindspore.dataset CPP sampler for GeneratorDataset 
Merge pull request !183 from JunhanHu/cpp_sampler
 2020-04-16 22:30:59 +08:00
mindspore-ci-bot b2b3e24a8e !329 [MS]support building on windows 10 
Merge pull request !329 from chenjianping/building-on-windows-2
 2020-04-15 19:10:50 +08:00
chenjianping 1286767d0e support building on windows 2020-04-15 09:36:48 +00:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00
mindspore-ci-bot 30de261c3c !243 Support nested repeat 
Merge pull request !243 from h.farahat/nested_repeat
 2020-04-14 23:07:32 +08:00
hesham 0fc23eee0f Support nested repeat 2020-04-14 10:13:18 -04:00
mindspore-ci-bot f1fa2a9941 !273 [MD] update subset random sampler in minddataset 
Merge pull request !273 from liyong126/mindrecord_subset_sampler_python
 2020-04-14 11:18:36 +08:00
mindspore-ci-bot e95945969b !270 fix complite error while using .so with clang on macos 
Merge pull request !270 from SanjayChan/mac_os
 2020-04-14 09:16:25 +08:00
mindspore-ci-bot d9cd681c30 !207 Fix potential overflow in batch by changing int32 to int64 
Merge pull request !207 from ZiruiWu/batch_info_overflow
 2020-04-14 01:55:35 +08:00
liyong 0ce83e39e1 fix TestShardSampleWrongNumber 
adjust internal api
 2020-04-13 21:31:45 +08:00
chenzomi 65a49a21e5 fix complite error while using .so with clang on macos 2020-04-13 21:23:12 +08:00
liyong 11403492ae add mindrecord subset random sampler 2020-04-11 18:48:23 +08:00
dengwentao b5ca2a3e30 :modify protobuf cmake 2020-04-11 11:46:23 +08:00