1st draft of python iterator
Added Cifar10 and Cifar100 pybind port
Change pybind to use IR for Skip and Manifest
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
DatasetNode as a base for all IR nodes
namespace change
Fix the namespace issue and make ut tests work
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
Add VOCDataset
!63 Added RandomDataset
* Added RandomDataset
add imagefolder ir
Pybind switch: CelebA and UT
!61 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working
add ManifestDataset IR
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding
!69 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset
!67 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1
!71 Fix indent error
* Fix indentation error
!72 Fix c_api tests cases
* Fix c_api tests cases
!73 Added CSV Dataset
* Added CSVDataset
pybind switch: Take and CelebA fixes
!75 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation
!77 Add Python API tree
* Python API tree
add minddataset
TextFileDataset pybind
Rename to skip test_concat.py and test_minddataset_exception.py
!80 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind
Enable more c_api take and CelebA tests; delete util_c_api
!84 Schema changes in datasets.py
* Schema changes
!85 Remove input_indexes from sub-classes
* remove input_index from each subclass
!83 Remove C datasets
* Removed c_dataset package
* Remove c_datasets
!82 pybind switch: shuffle
* pybind switch: shuffle
!86 Add build_vocab
* Add build_vocab
Rebase with upstream/master
_shuffle conflict
BatchNode error
!88 Fix rebase problem
* fix rebase problem
Enable more unit tests; code typo/nit fixes
!91 Fix python vocag hang
* Fix python vocab hang
!89 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength
Update and enable more tet_c_api_*.py tests
!95 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab
!96 Fix more tests
* - Fix some tests
- Enable more test_c_api_*
- Add syncwait
!99 pybind switch for device op
* pybind switch for device op
!93 Add getters to python API
* Add getters to python API
!101 Validate tree, error if graph
* - Add sync wait
!103 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem
!102 Added filter pybind switch
* Added Filter pybind switch
!104 Fix num_samples
* - TfRecord/Random schem aproblem
!105 Fix to_device hang
* Fix to_device hang
!94 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion
Add save pybind
fix compile err
init modify concat_node
!107 Fix some tests cases
* Fix tests cases
Enable and fix more tests
!109 pybind switch for get dataset size
* pybind_get_dataset_size
some check-code fixes for pylint, cpplint and clang-format
!113 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work
!114 Make Android compile clean
* Make Android Compile Clean
Fix build issues due to rebase
!115 Fix more tests
* Fix tests cases
* !93 Add getters to python API
fix test_profiling.py
!116 fix get dataset size
* fix get dataset size
!117 GetColumnNames pybind switch
* Added GetColumnNames pybind switch
code-check fixes: clangformat, cppcheck, cpplint, pylint
Delete duplicate test_c_api_*.py files; more lint fixes
!121 Fix cpp tests
* Remove extra call to getNext in cpp tests
!122 Fix Schema with Generator
* Fix Schema with Generator
fix some cases of csv & mindrecord
!124 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size
!125 getter separation
* Getter separation
!126 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler
!127 Assign runtime getter to each get function
* Assign runtime getter to each get function
Fix compile issues
!128 Match master code
* Match master code
!129 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code
!130 Add cache fix
* Added cache fix for map and image folder
!132 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json
!131 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests
!134 Cleanup datasets.py part1
* Cleanup dataset.py part1
!133 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset
Rebase on master 181120 10:20
fix profiling
temporary solution of catching stauts from Node.Build()
!141 ToDevice Termination
* ToDevice termination
pylint fixes
!137 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests
!138 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)
Rebase fix
!143 Fix cpplint issue
* Fix cpplint issue
pylint fixes in updated testcases
!145 Reset exceptions testcase
* reset exception test to master
!146 Fix Check_Pylint Error
* Fix Check_Pylint Error
!147 fix android
* fix android
!148 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit
!149 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit
!150 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit
!152 ExecutionTree error
* ET destructor error
!153 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map
!156 early __del__ of iterator/to_device
* early __del__ of iterator
!155 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1
!158 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1
!154 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…
!157 add replace none
* Add replace_none to datasets.py, address comments in tests
Trying to resolve copy
Override the deepcopy method of deviceop
Create_ir_tree method
Create_ir_tree method 2
Create_ir_tree method 2
del to_device if already exists
del to_device if already exists
cache getters shapes and types
Added yolov3 relaxation, to be rolled back
Get shapes and types together
bypass yolo
NumWorkers for MapOp
revert Yolo
revert Thor
Print more info
Debug code: Update LOG INFO to LOG ERROR
do not remove epochctrl for getter pass
Remove repeat(1)
pritn batch size
add log to tree_consumer and device_queue op
Revert PR 8744
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
__del__ toDEvice
__del__ toDevice2
!165 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print
revert some changes
!166 getter: get_data_info
* getter: get_data_info
!168 add back tree print
* revert info to warnning in one log
* add back the missed print tree log
Release GIL in GetDataInfo
add string implementation, can handle partial string slices
finish core implementation, added Index object
fix up the logic
add in new SliceOption object, cleaning up tests
decided to use project after batch as well
per_batch_map 1->x colunm works
add more test case
add test cases, project after map
col order hahaah
fix ci
add test cases
addr review cmts
ci
fix build err
fix test case failure
ci
ci fix 1
fix ci 2
fix ci 3
fix ci 4
fix ci 5
fix ci 6
fix ci 7
change var name in CelebA from dataset_type to usage
address review cmts
fix cpp ut fail
change mode to usage in VOCDataset
fix cmts
fix ci test case fail
change ImagefolderDV2 name
change ds.transforms.vision to ds.vision
change batch api to match map api more closely
compose op changes
test_pylint
remove compose op from vision, move to transform module, refactor map and batch to use column_order
implment pause in MapOp, added more to callback
add ds_callback
- Initial drop of Python DSCallback
- Pybind DSCallback
- Pybind DSCallback
added callback to mapOp
- de_pipeline DSCallback
- de_pipeline DSCallback
add test case, segfault for now
fix seg fault
- de_pipeline DSCallback
remove 1 line
update callback test case, now works
use builder class for mapOp callback
- de_pipeline DSCallback
- de_pipeline DSCallback
- de_pipeline DSCallback
better test case
minor fix
add comments and minor clean ups
get rid of nullptr in MapOp, use other flag instead
fix a bug ParseMapOp only takes 1 callback
- Added WaitedDSCalabck
refactor callback param
fix text case incorrect number
- added testing
fix cpp test case
- added testing
- revert back lenet changes
- cleanup test_callbacks.py
- cleanup test_callbacks.py
fix CI stage I
fix CI stage II
fix CI and update epoch counter
- add validation
- add more testing test_callbacks.py
use random data op to do tests
adjust when to call EpochBegin/End
- add repeat with callback
- addressing reviewers' comments
- docstring and CI fixes
- docstring and CI fixes
- docstring and CI fixes
- rebase with upstream/master
fix cpp test case
fix review comments
addr review cmts, add test case
Eric Zhang
hesham
Zirui Wu
Xiao Tianci
mindspore-ci-bot
Lixia Chen
Mahdi
Jesse Lee
genglishuai
qianlong