Commit Graph

22 Commits

Author SHA1 Message Date
liyong bfba630aa2 update pK_sampler 2020-04-27 21:40:49 +08:00
liyong 1f222ddb9e fix mindrecord c ut 2020-04-23 21:48:58 +08:00
mindspore-ci-bot fe67db75a3 !580 fix dataset & mindrecord log print
Merge pull request !580 from panfengfeng/panff/fix_log_print_in_ge
2020-04-23 14:26:29 +08:00
jonwe 353b3b7944 optimize mindrecord writer performance 2020-04-22 11:09:28 -04:00
panfengfeng b31946750b 回退 'Pull Request !182 : Tuning mindrecord writer performance' 2020-04-22 19:19:57 +08:00
mindspore-ci-bot c73301bad9 !182 Tuning mindrecord writer performance
Merge pull request !182 from jonathan_yan/write_a
2020-04-22 17:43:46 +08:00
panfengfeng 6328edf455 modify dataset & mindrecord log compile 2020-04-22 17:10:40 +08:00
jonwe f4bae5f364 optimize mindrecord writer performance 2020-04-21 09:12:13 -04:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
xiefangqi d4822137f3 fix segment bug on windows 2020-04-18 15:52:38 +08:00
chenjianping 1286767d0e support building on windows 2020-04-15 09:36:48 +00:00
jonyguo ea5da25d10 fix: use exactly read option 2020-04-14 16:42:00 +08:00
liyong 0ce83e39e1 fix TestShardSampleWrongNumber
adjust internal api
2020-04-13 21:31:45 +08:00
liyong 11403492ae add mindrecord subset random sampler 2020-04-11 18:48:23 +08:00
mindspore-ci-bot 7cec28526a !95 Modify cmake of protobuf
Merge pull request !95 from dengwentao/protobuf
2020-04-11 16:11:03 +08:00
dengwentao b5ca2a3e30 :modify protobuf cmake 2020-04-11 11:46:23 +08:00
jonyguo a9443635b7 fix: mindpage enhance parameter check and search by filename failed 2020-04-11 10:19:14 +08:00
mindspore-ci-bot bd7ac259af !147 Parallelize mindrecord index writer via std::thread
Merge pull request !147 from ZiruiWu/master
2020-04-09 05:31:44 +08:00
Zirui Wu 5637f80692 implemented multi-thread index writer for mindrecord
num threads cannot be more than num shards

minor fix

clang style fix

address review comments
2020-04-08 10:03:19 -04:00
jonyguo 20d1b64443 fix: error info is not exactly when column list invalid 2020-04-08 16:11:13 +08:00
jonyguo c688265671 fix: when use MindDataset block_reade=True hung 2020-04-07 17:53:23 +08:00
zhunaipan 930a1fb0a8 initial version
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
2020-03-27 22:54:54 +08:00