Add Last batch policy support for remaining image readers #217
Conversation
fiona-gladwin
commented
Oct 1, 2024
fiona-gladwin
commented
- Add Last batch policy, pad last batch, stick to shard and shard size support for remaining image readers
- Add changes to COCO, caffe, caffe2, mxnet, tf and cifar10 readers
…nto swbs/audio/pr10
…into swbs/audio/pr10
Remove changes to update the filenames vector incase of padding Fix the pipeline when pad is off and use idx in a continuous manner
To skip and batch and start with the previously padded idx in last batch
…into lbp_fix_pr10
Fix shuffle in caffe Init variable in caffe2
… into swbs/lbp_readers_pr11_merged
|
@fiona-gladwin can you fix the merge conflicts |
fiona-gladwin
dismissed
LakshmiKumar23’s stale review
The merge-base changed after approval.
Fixed merge conflicts |
| @@ -68,6 +68,8 @@ class CaffeLMDBRecordReader : public Reader { | |||
|
|
|||
| CaffeLMDBRecordReader(); | |||
|
|
|||
| size_t last_batch_padded_size() override; // The size of the number of samples padded in the last batch | |||
There was a problem hiding this comment.
I think we should consider merging the caffe and caffe2 classes. Or derive caffe2 from caffe. I see lots of repeated member variables in both classes
| @@ -68,6 +68,8 @@ class Caffe2LMDBRecordReader : public Reader { | |||
|
|
|||
| Caffe2LMDBRecordReader(); | |||
|
|
|||
| size_t last_batch_padded_size() override; // The size of the number of samples padded in the last batch | |||
There was a problem hiding this comment.
I see the same variables and functions are repeated in all the image reader classes. That is why we needs to make so many changes for accomodating something small like last_batch_policy.
We need to consolidate all common functions and variables in a common base class and derive all different readers from that.
| return _file_names.size(); | ||
|
|
||
| int ret = ((int)_file_names.size() - _read_counter); | ||
| int ret = 0; |
There was a problem hiding this comment.
please add new base class and move all common stuff to that class
| @@ -223,7 +223,11 @@ def video(sequence_length, file_list_frame_num=False, file_root="", image_type=t | |||
| "file_list_frame_num": file_list_frame_num} # VideoMetaDataReader | |||
| b.videoMetaDataReader(Pipeline._current_pipeline._handle, | |||
| *(kwargs_pybind_reader.values())) | |||
|
|
|||
| RocalShardingInfo = b.RocalShardingInfo() | |||
There was a problem hiding this comment.
please add these defaults into constructor of RocalShardingInfo()
