pgjmblvch
/
mindspore2022
forked from huawei/mindspore2022
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,300 Commits 18 Branches 22 Tags 3.6 GiB
Commit Graph

46 Commits

Author SHA1 Message Date
luoyang cd86788cf3 fix bugs of execute & update doc description in minddata 2021-02-07 22:52:52 +08:00
mindspore-ci-bot 620b2f4256 !10821 #I28T16[improve examples in dataset API document to make it okay to run(part1)] 
From: @ming__blue
Reviewed-by: 
Signed-off-by:
 2021-02-03 09:06:58 +08:00
hesham 1185218335 Support list of IDs as a sampler 2021-02-02 11:44:38 -05:00
13465716071 3594bd581c Mindcon Shandong bug fix part1 
fix rebase confict
 2021-02-02 20:13:35 +08:00
Mahdi 8f34faeb7a Changed bindings to SamplerObj 2021-01-29 14:20:05 -05:00
hesham 7120c66998 Add SubsetSampler 2021-01-27 15:00:18 -05:00
luoyang 415c8b08a5 fix issues of MD: 
1. description of num_shard in datasets.py
2. add param validation for sampler
3. add description for add_child in sampler.py
 2021-01-27 09:30:42 +08:00
Xiao Tianci f7093efe6a modify return description in comments 2021-01-18 20:52:04 +08:00
Nat Sutyanyong 5e1bb0b697 boilerplate code for future IR optimizer 
add 2 test cases to IRNode Deepcopy()

address review cmts

fix ut

samplerObj copy

ci

fix ci

fix ci round III

address further review cmts

add a missing macro

fix merge conflict

fix complie err

fix lite compile err

fix compile err

fix lite compile round III

address an issue

fix minor comments
 2020-11-26 09:42:01 -05:00
liyong 5b035408f6 fix deepycopy bug 2020-11-26 16:19:03 +08:00
ms_yan 0cb5c47856 add err modify 2020-11-18 16:30:24 +08:00
YangLuo 82cc7c3f63 [MD] Revert changes to weighted_random_sampler in PR7866 2020-10-31 12:56:17 +08:00
mindspore-ci-bot 8f44074796 !7866 [MD] C++ API support build_sentence_piece_vocab_node & sentence_piece_tokenizer 
Merge pull request !7866 from luoyang/c-api-pyfunc
 2020-10-29 14:26:13 +08:00
luoyang 52d278e858 [MD] C++ API support build_sentence_piece_vocab_node & sentence_piece_tokenizer 2020-10-29 11:05:28 +08:00
Xiao Tianci 823e52f1dc add params checking of offset in distributed_sampler 
add related comment to explain
add related UT
 2020-10-28 16:46:12 +08:00
luoyang f0e976dbda [MD] Fix bugs in WeightedRandomSampler & SubsetRandomSampler 2020-10-25 17:58:51 +08:00
Cathy Wong e971a3aed9 dataset API docstring: Update datasets.py and config.py 2020-09-17 20:03:03 -04:00
Cathy Wong f7adf648e9 dataset API docstring: Update datasets, samplers, graphdata and text 2020-09-14 18:10:44 -04:00
nhussain 3bac9d3713 switch input columns and operation 
change ImagefolderDV2 name

change ds.transforms.vision to ds.vision

change batch api to match map api more closely

compose op changes

test_pylint

remove compose op from vision, move to transform module, refactor map and batch to use column_order
 2020-09-08 10:26:20 -04:00
nhussain eb9a611041 remove old defaults 2020-08-19 10:11:19 -04:00
genglishuai 4d4df68d1e add DistributedSampler for Concat op 2020-08-13 22:52:02 +08:00
liyong 7341421d3b fix num samples in pk sampler 2020-08-10 19:05:47 +08:00
liyong e2ea1fa0df activate num_samples in distributed samplers 2020-07-17 16:55:57 +08:00
YangLuo 4e3bfcf4c9 !2306 [Dataset] Code review & improve quality 2020-06-22 14:01:22 +08:00
mindspore-ci-bot d7d094e1c5 !2231 fix report error in get_dataset_size 
Merge pull request !2231 from yanghaitao/yht_get_num_samples
 2020-06-20 19:58:16 +08:00
yanghaitao1 94cebe4d53 fix sampler get_num_samples 2020-06-17 03:54:54 -04:00
liyong d4f8f57c7e add split in minddataset 2020-06-16 19:50:41 +08:00
mindspore-ci-bot 9282b9473d !2059 fix SubsetSampler error 
Merge pull request !2059 from yanghaitao/yht_fix_subset_samplers
 2020-06-12 20:05:06 +08:00
yanghaitao 5b0fb71343 fix SubsetSampler 2020-06-12 16:36:04 +08:00
mindspore-ci-bot a3ab142f1c !2042 fix document api error 
Merge pull request !2042 from yanghaitao/yht_fix_api_document
 2020-06-12 15:53:39 +08:00
yanghaitao c12ffc1e90 fix api document 2020-06-12 11:16:51 +08:00
yanghaitao 8f16cff12e add para check for sampler 2020-06-11 22:26:06 +08:00
yanghaitao 9c2a72c55e check if num_samples is 0 in sampler 2020-06-06 11:40:09 +08:00
Jamie Nisbet 51bc0c0460 consistent design for num_samples 
updates

more update

more work

more fixin

post rebase updates

clang formatting

code review recovery

ci fixes

updates

update

update

update
 2020-06-03 15:01:18 -04:00
yanghaitao 2e3e31de90 fix subsetrandomsampler 2020-05-28 19:33:29 +08:00
Peilin Wang d4c93575e7 fixed bug for split, RandomSampler and some other cleanup 
add another test case

typo

merge conflict

another PR changed testing behavior, updated test cases in this commit

added input check for use_sampler

addressed code review comments

fixed pylint, not related to my changes
 2020-05-26 16:29:52 -04:00
Peilin Wang 71e8bb1960 general split case done, chaining sampler (basic case) is working 
implementation 99% complete

everything and tested except for repeatable shuffling

tested most basic/typical split usecases

cleanup

some more cleanup

fix CI

more ci fix

more ci fixes

more ci fix

more ci fix

more ci fix

added more tests, fixed some bugs

some more clean up and test cases

added shard/shuffle before split warning/error

addressed code review comments and ci

fixed ci
 2020-05-21 20:42:51 -04:00
Junhan Hu 1e904ddcee Fix CI warning of samplers.py 2020-04-27 13:10:23 -04:00
liyong bfba630aa2 update pK_sampler 2020-04-27 21:40:49 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
Junhan Hu 43a2e99833 Add python sampler support for CPP dataset 2020-04-19 20:36:37 -04:00
Yanjun Peng c69f3be758 fix dataset api description 2020-04-18 09:59:43 +08:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00
liyong 0ce83e39e1 fix TestShardSampleWrongNumber 
adjust internal api
 2020-04-13 21:31:45 +08:00
Yanjun Peng 363632ca9d fix dataset para validator check 2020-04-09 20:15:22 +08:00
zhunaipan 930a1fb0a8 initial version 
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
 2020-03-27 22:54:54 +08:00