* [126] J. Wang, L. Song, Z. Li, H. Sun, J. Sun, and N. Zheng, “End-to-end object detection with fully convolutional network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 15 849–15 858.
  * [127] M. Pandey and S. Lazebnik, “Scene recognition and weakly supervised object localization with deformable part-based models,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2011, pp. 1307–1314.
  * [128] W. Wang, Y. Wang, F. Chen, and A. Sowmya, “A weakly supervised approach for object detection based on soft-label boosting,” in _Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV)_ , 2013, pp. 331–338.
  * [129] P. Tang, X. Wang, X. Bai, and W. Liu, “Multiple instance detection network with online instance classifier refinement,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2843–2851.
  * [130] X. Zhang, Y. Wei, J. Feng, Y. Yang, and T. S. Huang, “Adversarial complementary learning for weakly supervised object localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1325–1334.
  * [131] K. Yu, B. Leng, Z. Zhang, D. Li, and K. Huang, “Weakly-supervised learning of mid-level features for pedestrian attribute recognition and localization,” _arXiv preprint arXiv:1611.05603_ , 2016.
  * [132] K. K. Htike and D. Hogg, “Weakly supervised pedestrian detector training by unsupervised prior learning and cue fusion in videos,” in _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , 2014, pp. 2338–2342.
  * [133] Y. Cai, Y. He, H. Wang, X. Sun, L. Chen, and H. Jiang, “Pedestrian detection algorithm in traffic scene based on weakly supervised hierarchical deep model,” _International Journal of Advanced Robotic Systems_ , vol. 14, no. 1, p. 1729881417692311, 2016.
  * [134] C. Jiang and B. Zhang, “Weakly-supervised vehicle detection and classification by convolutional neural network,” in _Proceedings of the IEEE International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI)_ , 2016, pp. 570–575.
  * [135] L. Cao, F. Luo, L. Chen, Y. Sheng, H. Wang, C. Wang, and R. Ji, “Weakly supervised vehicle detection in satellite images via multi-instance discriminative learning,” _Pattern Recognition (PR)_ , vol. 64, pp. 417–424, 2017.
  * [136] S. Chadwick and P. Newman, “Radar as a teacher: Weakly supervised vehicle detection using radar labels,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 222–228.
  * [137] Q. Huang, C. K. Jia, X. Zhang, and Y. Ye, “Learning discriminative subspace models for weakly supervised face detection,” _IEEE Transactions on Industrial Informatics (TII)_ , vol. 13, no. 6, pp. 2956–2964, 2017.
  * [138] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [139] S. S. A. Zaidi, M. S. Ansari, A. Aslam, N. Kanwal, M. Asghar, and B. Lee, “A survey of modern deep learning based object detection models,” _arXiv preprint arXiv:2104.11892_ , 2021.
  * [140] L. Jiao, R. Zhang, F. Liu, S. Yang, B. Hou, L. Li, and X. Tang, “New generation deep learning for video object detection: A survey,” _IEEE Transactions on Neural Networks and Learning Systems (TNNLS)_ , 2021.
  * [141] X. Yan, Z. Chen, A. Xu, X. Wang, X. Liang, and L. Lin, “Meta r-cnn: Towards general solver for instance-level low-shot learning,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 9577–9586.
  * [142] T.-I. Hsieh, Y.-C. Lo, H.-T. Chen, and T.-L. Liu, “One-shot object detection with co-attention and co-excitation,” _arXiv preprint arXiv:1911.12529_ , 2019.
  * [143] Z. Yang, Y. Wang, X. Chen, J. Liu, and Y. Qiao, “Context-transformer: tackling object confusion for few-shot detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 34, no. 07, 2020, pp. 12 653–12 660.
  * [144] W. Zhang and Y.-X. Wang, “Hallucination improves few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 13 008–13 017.
  * [145] G. Han, S. Huang, J. Ma, Y. He, and S.-F. Chang, “Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment,” _arXiv preprint arXiv:2104.07719_ , 2021.
  * [146] A. Li and Z. Li, “Transformation invariant few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 3094–3102.
  * [147] T.-I. Chen, Y.-C. Liu, H.-T. Su, Y.-C. Chang, Y.-H. Lin, J.-F. Yeh, and W. H. Hsu, “Should i look at the head or the tail? dual-awareness attention for few-shot object detection,” _arXiv preprint arXiv:2102.12152_ , 2021.
  * [148] G. Zhang, Z. Luo, K. Cui, and S. Lu, “Meta-detr: Few-shot object detection via unified image-level meta-learning,” _arXiv preprint arXiv:2103.11731_ , 2021\.
  * [149] X. Li, L. Zhang, Y. P. Chen, Y.-W. Tai, and C.-K. Tang, “One-shot object detection without fine-tuning,” _arXiv preprint arXiv:2005.03819_ , 2020\.
  * [150] B. Singh, M. Najibi, and L. S. Davis, “Sniper: Efficient multi-scale training,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , vol. 31, 2018, pp. 9310–9320.
  * [151] S. Zhang, D. Luo, L. Wang, and P. Koniusz, “Few-shot object detection by second-order pooling,” in _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , 2020.
  * [152] Y. Li, W. Feng, S. Lyu, Q. Zhao, and X. Li, “Mm-fsod: Meta and metric integrated few-shot object detection,” _arXiv preprint arXiv:2012.15159_ , 2020.
  * [153] C. Zhu, F. Chen, U. Ahmed, Z. Shen, and M. Savvides, “Semantic relation reasoning for shot-stable few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8782–8791.
  * [154] L. Zhang, S. Zhou, J. Guan, and J. Zhang, “Accurate few-shot object detection with support-query mutual guidance and hybrid loss,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14 424–14 432.
  * [155] W. Zhang, Y.-X. Wang, and D. A. Forsyth, “Cooperating rpn’s improve few-shot object detection,” _arXiv preprint arXiv:2011.10142_ , 2020.
  * [156] H. Hu, S. Bai, A. Li, J. Cui, and L. Wang, “Dense relation distillation with context-aware aggregation for few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 10 185–10 194.
  * [157] A. Wu, Y. Han, L. Zhu, Y. Yang, and C. Deng, “Universal-prototype augmentation for few-shot object detection,” _arXiv preprint arXiv:2103.01077_ , 2021\.
  * [158] D.-J. Chen, H.-Y. Hsieh, and T.-L. Liu, “Adaptive image transformer for one-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 12 247–12 256.
  * [159] J. Deng, X. Li, and Y. Fang, “Few-shot object detection on remote sensing images,” _arXiv preprint arXiv:2006.07826_ , 2020.
  * [160] L. Karlinsky, J. Shtok, S. Harary, E. Schwartz, A. Aides, R. Feris, R. Giryes, and A. M. Bronstein, “Repmet: Representative-based metric learning for classification and few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 5197–5206.
  * [161] C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2014, pp. 391–405.
  * [162] S. Li, W. Song, S. Li, A. Hao, and H. Qin, “Meta-retinanet for few-shot object detection.” in _Proceedings of the British Machine Vision Conference (BMVC)_ , 2020.
  * [163] W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network regularization,” _arXiv preprint arXiv:1409.2329_ , 2014.
  * [164] Q. Cai, Y. Pan, T. Yao, C. Yan, and T. Mei, “Memory matching networks for one-shot image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 4080–4088.
  * [165] Y.-X. Wang, D. Ramanan, and M. Hebert, “Meta-learning to detect rare objects,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 9925–9934.
  * [166] P. Li, Y. Li, and D. Wang, “Class-incremental few-shot object detection,” _arXiv preprint arXiv:2105.07637_ , 2021.
  * [167] B. Sun, B. Li, S. Cai, Y. Yuan, and C. Zhang, “Fsce: Few-shot object detection via contrastive proposal encoding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 7352–7362.
  * [168] X. Chen, M. Jiang, and Q. Zhao, “Leveraging bottom-up and top-down attention for few-shot object detection,” _arXiv preprint arXiv:2007.12104_ , 2020\.
  * [169] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2117–2125.
  * [170] L. Zhang, C. Zhang, S. Quan, H. Xiao, G. Kuang, and L. Liu, “A class imbalance loss for imbalanced object recognition,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 2778–2792, 2020.
  * [171] D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 1301–1310.
  * [172] N. Dvornik, J. Mairal, and C. Schmid, “Modeling visual context is key to augmenting object detection datasets,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 364–380.
  * [173] S. Tripathi, S. Chandra, A. Agrawal, A. Tyagi, J. M. Rehg, and V. Chari, “Learning to generate synthetic data via compositing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 461–470.
  * [174] J. Wu, L. Song, T. Wang, Q. Zhang, and J. Yuan, “Forest r-cnn: Large-vocabulary long-tailed object detection and instance segmentation,” in _Proceedings of the ACM International Conference on Multimedia (MM)_ , 2020, pp. 1570–1578.
  * [175] X. Dong, L. Zheng, F. Ma, Y. Yang, and D. Meng, “Few-shot object detection,” _arXiv preprint Arxiv:1706.08249_ , pp. 1–11, 2017.
  * [176] A. Bar, X. Wang, V. Kantorov, C. J. Reed, R. Herzig, G. Chechik, A. Rohrbach, T. Darrell, and A. Globerson, “Detreg: Unsupervised pretraining with region priors for object detection,” _arXiv preprint arXiv:2106.04550_ , 2021.
  * [177] J. Yoon, S. Hong, and M.-K. Choi, “Semi-supervised object detection with sparsely annotated dataset,” in _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , 2021, pp. 719–723.
  * [178] J. Zhao, M. Mathieu, R. Goroshin, and Y. Lecun, “Stacked what-where auto-encoders,” _arXiv preprint arXiv:1506.02351_ , 2015.
  * [179] L. Maaløe, C. K. Sønderby, S. K. Sønderby, and O. Winther, “Auxiliary deep generative models,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2016, pp. 1445–1453.
  * [180] Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. Salakhutdinov, “Good semi-supervised learning that requires a bad gan,” _arXiv preprint arXiv:1705.09783_ , 2017.
  * [181] J.-C. Su, Y.-H. Tsai, K. Sohn, B. Liu, S. Maji, and M. Chandraker, “Active adversarial domain adaptation,” in _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 739–748.
  * [182] A. Shaban, A. Rahimi, T. Ajanthan, B. Boots, and R. Hartley, “Few-shot weakly-supervised object detection via directional statistics,” _arXiv preprint arXiv:2103.14162_ , 2021.
  * [183] L. Karlinsky, J. Shtok, A. Alfassy, M. Lichtenstein, S. Harary, E. Schwartz, S. Doveh, P. Sattigeri, R. Feris, A. Bronstein _et al._ , “Starnet: towards weakly supervised few-shot object detection,” _arXiv preprint arXiv:2003.06798_ , 2020.
  * [184] Z. Yang, M. Shi, C. Xu, V. Ferrari, and Y. Avrithis, “Training object detectors from few weakly-labeled and many unlabeled images,” _Pattern Recognition (PR)_ , vol. 120, p. 108164, 2021.
  * [185] G. Quellec, M. Lamard, P.-H. Conze, P. Massin, and B. Cochener, “Automatic detection of rare pathologies in fundus photographs using few-shot learning,” _Medical image analysis_ , vol. 61, p. 101660, 2020.
  * [186] M. Wei, “Few shot learning for small lesion detection,” _Meta_ , vol. 85, no. 22, pp. 0–54, 2021.
  * [187] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa, “Generate to adapt: Aligning domains using generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 8503–8512.
  * [188] C. Chen, Q. Dou, H. Chen, and P.-A. Heng, “Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation,” in _Proceedings of the International Workshop on Machine Learning in Medical Imaging (MLMI)_ , 2018, pp. 143–151.
  * [189] C. Chen, W. Xie, Y. Wen, Y. Huang, and X. Ding, “Multiple-source domain adaptation with generative adversarial nets,” _Knowledge-Based Systems (KBS)_ , vol. 199, p. 105962, 2020.
  * [190] H. Chen, Y. Wang, G. Wang, X. Bai, and Y. Qiao, “Progressive object transfer detection,” _IEEE Transactions on Image Processing (TIP)_ , vol. 29, pp. 986–1000, 2019.
  * [191] L. Zhu, Y. Chen, and A. Yuille, “Unsupervised learning of a probabilistic grammar for object detection and parsing,” in _Proceedings of the Neural Information Processing Systems (NIPS)_ , 2006.
  * [192] C. Michaelis, M. Bethge, and A. S. Ecker, “Closing the generalization gap in one-shot object detection,” _arXiv preprint arXiv:2011.04267_ , 2020.
  * [193] J.-M. Perez-Rua, X. Zhu, T. M. Hospedales, and T. Xiang, “Incremental few-shot object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 13 846–13 855.
  * [194] Z. Fan, Y. Ma, Z. Li, and J. Sun, “Generalized few-shot object detection without forgetting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 4527–4536.

TABLE X: Summarization of metric-based methods for the $N$-Way $K$-Shot limited-supervised problem. Abbreviations: Region Proposal Network (RPN), EdgeBoxes (EB) [161], Concatenate (CAT), Embedding Network of Support/Query Branch (S/Q). See Tab. III & Fig. 12 for more details about fusion nodes. Method |  Pipeline Used |  Data Preprocessing |  Embedding Network |  Fusion Node |  Published In | Full Name of Method and Highlights
---|---|---|---|---|---|---
AFD-Net [29] |  Faster R-CNN | Mask+CAT |  ResNet-101 ResNet-50 |  a-5 a-6 | arXiv2021 |  (Adaptive Fully-Dual Network, AFD); Propose AFD to generate task-specific vectors for support & query; Combine multiple aggregators for better feature fusion.
FSOD [80] |  Faster R-CNN | Crop+Context | ResNet-50 |  a-2 a-5 | CVPR2020 |  Include 16-pixel image context for better class prototypes; Reweight query features by class-specific vectors per class to make RPN generate high-quality proposals (meta RPN); Use global, local & patch matching to measure similarity more accurately.
CoAE-Net [142] |  Faster R-CNN | Crop | ResNet-50 |  a-1 a-3 a-5 | NIPS2019 |  (Co-Attention and Co-Excitation Network, CoAE); Use a non-local variant to provide aligned class specific features for RPN to make high-quality proposals; Take a shared SE block to learn for further refine query & support features.
META RCNN [81] |  Faster R-CNN |  Crop from Features |  VGG-16 ResNet-50 | a-1 |  ACM MM2020 |  Implicitly enhance category prototypes with contextual information outside instances.
AIT-Net [158] |  Faster R-CNN | Crop | ResNet-50 |  a-1 a-4 | CVPR2021 |  Introduce multi-head attention to explore cooccurrences between query & support features; Employ a transformer variant to explore proposal-support attention.
ZSD [77] | R-CNN | N/A |  Q: Inception S: MLP | a-5 | ECCV2018 |  (Zero-Shot Object Detection, ZSD); Take a MLP to transform word embedding from semantic space to visual space (i.e., cross-domain alignment); Assign a list of background images with a set of categories randomly sampled from corpus, which is disjoint with seen classes, to force ZSD to learn a latent background concept.
NP-RepMet [78] |  Faster R-CNN | N/A |  ResNet-101 (DCN, FPN) | a-5 | NIPS2020 |  (Representative-based metric learning with negative and positive information, NP-RepMet); Employ negatives ($0.2<IoU(P,G)<0.3$) to learn negative prototypes; Take the spectral clustering to get K negative prototypes for inference.
Meta DETR [148] |  Deformable DETR | Crop | ResNet-101 | a-1 | arXiv2021 |  Adopt a novel BERT to get rid of heuristic design in Faster R-CNN; Leverage skip connection to regularize low-/high-level features.
S-FRCNN [192] |  Faster R-CNN | Crop |  ResNet-50 ResNetXt-101 | a-1 | arXiv2020 |  (Siamese Faster R-CNN, S-FRCNN); Explore SUB for feature fusion.
DAnA [147] |  Faster R-CNN / RetinaNet | Crop | ResNet-50 |  a-2 a-4 | TMM2021 |  (Dual-Awareness Attention, DAnA); BA block first generate a foreground heatmap, then sum up features reweighted by such a heatmap along spatial axes to generate attention vectors, and finally add it to original features; CISA is an affinity matrix for semantic alignment.
DCNet [156] |  Faster R-CNN | Crop | ResNet-101 | a-1 | CVPR2021 |  Design a non-local variant which captures global similarity maps between query & support features of each category, then perform softmax on such similarity maps among all support categories, and finally use normalized maps to refine query features; Take serval RoI-Align layers to capture features at various scales.
OSOD [149] |  Faster R-CNN | Crop | ResNet-50 |  a-2 a-4 | arXiv2020 |  (One-Shot Object Detection, OSOD); Take FPN to get a global prototypes per class which is used to compute a cosine similarity map for RPN at each scale; Take a 1x1 conv to correlate query & support features at each pixel (spatial semantics); Pair a query image with the most similar support instance to alleviate intra-class variance in the training stage; Sample negative boxes with certain IoU with gt boxes to force a classifier to distinguish low-quality proposals.
PNSD [151] |  Faster R-CNN | Crop | ResNet-50 |  a-2 a-4 | ACCV2020 |  Propose a multi-scale feature aggregator to enhance features of both query & support; Adopt SOP/PN to capture feature cooccurrences; Design a SN module to combine global/local/patch matching to evaluate similarity more accurately.
FSODM [159] | YOLOv3 | Crop |  Q: DarkNet-53 S: N/S | b-1 | TGRS2020 |  Take a FPN variant to learn features of different scales to alleviate large-scale variations.
CME [79] | YOLOv2 | Mask+CAT |  Q: DarkNet-19 S: N/S | a-4 | CVPR2021 |  (Class Margin Equilibrium, CME); Design a max-margin loss to guarantee that intra-class distance is less than inter-class distance as much as possible in meta-training stage; Erase discriminative pixels in a support mask by its gradient map for data augmentation.
FSOD [151] |  Faster R-CNN | Crop |  ResNet-50 ResNet-101 |  a-1 a-5 | CVPR2021 |  Average similarity (scores) at all levels by a RN variant to get a reliable score for each RoI; Dynamically assign weights to determine class prototypes by similarity between query & support instead of simple average to learn a robust detector; Design a kernel generator to dynamically generate support-specific kernels to explore feature cooccurrences; Take a adaptive margin to improve a contrastive loss to distinguish a correct category from background and other foreground categories; Adopt focal loss to tackle the foreground-background imbalance problem.
Meta Faster R-CNN [145] |  Faster R-CNN | Crop+Context |  ResNet-50 ResNet-101 |  a-2 a-5 | arXiv2021 |  Combine MUL, SUB & CAT to generate more robust features; Calculate an affinity matrix of RoIs and all category prototypes for spatial alignment, and then sum up such an affinity matrix by row for background suppression.
TIP [146] |  Faster R-CNN | Mask+CAT | N/S | a-4 | CVPR2021 |  (Transformation Invariant Principle, TIP); Naive data augmentation may have negative effect on FSOD; Design a consistency regularization to learn a robust backbone to narrow intra-class variations (e.g., rotation, occlusion), while a CE loss is adopted to expand inter-class distance for category prototypes; Design a proposal consistent regularization which takes transformed images to predict RoIs, and then get such RoIs features from features of original images to make final results.
MM-FSOD [152] |  Faster R-CNN | Crop | ResNet-34 | a-5 | arXiv2020 |  (Meta and Metric integrated Few-Shot Object Detection, MM-FSOD); Take Pearson distance to distinguish hard negative proposals; View background patches as instances of the background category.
ONCE [193] | CenterNet | N/S | ResNet-50 | b-1 | CVPR2020 |  (OpeNended Centre nEt, ONCE); Fix all category-agnostic parameters in the finetuning stage; Learn a generator to generate a category representation for each novel category.
FSOD | YOLOv2 | Mask+CAT |  Q: DarkNet-19 S: N/S | b-1 | ICCV2019 |  Design a feature reweighting module to generate category-specific features by a given category to detect all instances of such a category.
RepMet [160] |  Faster R-CNN | N/A | Inception | a-5 | CVPR2019 |  (Representative-based metric learning, RepMet); Take K category prototypes for each category to match each RoI; Design an embedding loss (a kind of margin loss) to minimize intra-category variations and maximize inter-category distance.
Meta R-CNN [141] |  Faster R-CNN | Mask+CAT | ResNet-101 | a-4 | ICCV2019 |  Propose a meta loss to make category prototypes distinguish each other.
SRR-FSD [153] |  Faster R-CNN | N/A |  Q:ResNet-101 S:A knowledge graph | a-5 | CVPR2021 |  (Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection, SRR-FSD); Propose a knowledge graph to model inter-category relations; Add fc layers respectively to provide more suitable features for the classifier and locator; Design a MLP for cross-domain alignment.

TABLE XI: Summarization of transfer-learning methods for the N-Way K-Shot limited-supervised problem. Method |  Pipeline Used | Backbone |  Published In | Full Name of Method and Highlights
---|---|---|---|---
CT-Net [143] | SSD | VGG-16 | AAAI2020 |  (Context-Transformer Network, CT-Net); Adopt max-pooling to capture more contextual information outside a prior anchor; Generate an affinity matrix between features of prior anchor w/ & w/o contextual fields to refine features of prior anchor for more semantic context to classify each proposal.
LEAST [166] |  Faster R-CNN | ResNet-101 | arXiv2021 |  (Less forgetting, fEwer training resources, And Stronger Transfer capability, LEAST); Make a trade-off between two kinds of updating rules (i.e., FIX-ALL & FIT-ALL) to fine-tune parameters of the Faster R-CNN head; Employ KL divergence to make consistent probability distribution of base categories in the fine-tuning phase; Take a cluster-based exemplar selection algorithm to choose the most representative K exemplars of each base category instead of randomly sampling.
Retentive R-CNN [194] |  Faster R-CNN | ResNet-101 | CVPR2021 |  Adopt a cosine classifier to filter drastic variations of feature norms between base & novel categories; Take KL divergence to maintain probability distribution of base categories in the fine-tuning stage; Add an extra category-agnostic classifier for novel categories to capture more high-quality proposals of novel categories as possible; Assign more weight on a base learner to predict a proposal of base categories to make more reliable results in inference.
$FSOD^{up}$ [157] |  Faster R-CNN | ResNet-101 | arXiv2021 |  Learn a set of category-agnostic prototypes (e.g., shared features) to enhance features of a given image to make more high-quality proposals; Apply a learnable affine transformation on universal prototypes to generate conditional prototypes which are employed to provide discriminative features for the further locator and classifier; Employ a consistent constraint (KL divergence) between conditional & augmented probability distributions produced by a shared classifier which takes conditional & augmented prototypes as inputs to learn a more robust detector.
AttFDNet [168] | RFB Net | VGG-16 | arXiv2020 |  (Attentive Few-Shot Object Detection Network, AttFDNet); Employ a saliency sub-network to force AttFDNet to pay more attention to foreground regions; Combine a pixel-wise convolution layer with a softmax layer to generate a heatmap, and sum up such a heatmap along spatial axes to generate global representations for semantic alignment; Leverage RFB block to capture more context information without excessive parameters; Design a object-concentration loss to maximize cosine similarity between all positive proposals & associated category-specific weights; Leverage OHEM to enforce AttFDNet to distinguish low-quality positive proposals.
FAFRCNN [37] |  Faster R-CNN | VGG-16 | CVPR2019 |  (Few-shot Adaptive Faster-RCNN, FAFRCNN); Design split pooling to randomly crop feature patches by a set of predefined anchors from source & target domain to form source-source & source-target pairs for domain-adversarial learning; Randomly sample RoIs from source & target domain to semantically align paired object features; Take L2 penalty to make consistent features between the pre-trained network and the fine-tuned network to alleviate overfitting.
LSTD [39] |  Faster R-CNN | VGG-16 | AAAI2018 |  (Low-Shot Transfer Detector, LSTD); Design a BD regularization which take ground-truth boxes of a given image to generate a background mask to depress background regions; Propose a KL divergence to utilize cross-domain similarity to transfer general knowledge.
TFA [40] |  Faster R-CNN | ResNet-101 | arXiv2020 |  (Two-stage Fine-tuning Approach, TFA); Design a two-stage training approach which first trains a base learner on a large-scale base benchmark, and then only fine-tunes Faster R-CNN head on a small-scale novel benchmark; Take a cosine classifier to reduce intra-category variance; Revise evaluation protocols that use more repeated runs to reduce sample variance for fair comparison.
MPSR [38] |  Faster R-CNN | ResNet-101 | ECCV2020 |  (Multi-Scale Positive Sample Refinement, MPSR); Randomly sample a batch of instances in an image, and resize such instances to various scales/aspect ratios (object pyramid) for more positive training data to refine FPN.
FSCE [167] |  Faster R-CNN | ResNet-101 | CVPR2021 |  (Few-Shot Object Detection via Contrastive Proposals Encoding, FSCE); Reuse positive proposals with low confidence suppressed by NMS & remove negative proposals by half to rebalance gradient; Design a contrastive proposal encoding (CPE) loss to learn a discriminative feature extractor.


# Energy Efficient Foot-Shape Design for Bipedal Walkers on Granular Terrain

Xunjie Chen  Jingang Yi  Hao Wang  Department of Mechanical and Aerospace
Engineering
Rutgers University, Piscataway, NJ 08854 USA
(e-mail: {xc337<EMAIL_ADDRESS> Department of Civil and Environment
Engineering
Rutgers University, Piscataway, NJ 08854 USA
(e-mail<EMAIL_ADDRESS>
###### Abstract

It is important to understand how bipedal walkers balance and walk effectively
on granular materials, such as sand and loose dirt, etc. This paper first
presents a computational approach to obtain the motion and energy analysis of
bipedal walkers on granular terrains and then discusses an optimization method
for the robot foot-shape contour design for energy efficiently walking. We
first present the foot-terrain interaction characteristics of the intrusion
process using the resistive force theory that provides comprehensive force
laws. Using human gait profiles, we compute and compare the ground reaction
forces and the external work for walking gaits with various foot shapes on
granular terrains. A multi-objective optimization problem is finally
formulated for the foot contour design considering energy saving and walking
efficiency. It is interesting to find out a non-convex foot shape gives the
best performance in term of energy and locomotion efficiency on hard granular
terrains. The presented work provides an enabling tool to further understand
and design efficient and effective bipedal walkers on granular terrains.

###### keywords:

Bipedal walkers, granular terrains, foot-ground interactions, gait design

††thanks: Address all correspondence to J. Yi.

##  1 Introduction

Bipedal robots appear advantageous for locomotion in complex uneven and
deformable, granular terrain (Xiong et al., 2017). However, the physical
modeling of the granular intrusion interactions complicates the walking
dynamics because unlike the rigid flat ground, there is lack of the
comprehensive force laws for foot-granular terrain contact. Although there are
applications of wheel-based robot locomotion to investigate terradynamic
performance on granular materials such as sand, few studies focus on humanoid
bipedal foot-granular terrain interaction and corresponding influence on
walking dynamics and performance.

Bipedal walking gaits has been investigated from the energy expenditure view.
Inverted pendulum model with rocker feet has proved that rolling feet with a
certain curvature leads to an advantage for reducing step-to-step transition
work and saving energy (Adamczyk et al., 2006; Trkov et al., 2019). Complex
curved foot shape strongly affects performance and efficiency in bipedal
walking. Properties (e.g., radius) of rocker feet are investigated to show the
effect on the energetic efficiency of a gait and the hip kinematics (Martin
and Schmiedeler, 2011, 2012). However, all of the above-mentioned work focus
on the robot walking on the rigid flat ground and locomotion on the granular
materials are much complicated.

Although conventional terramechanics reveals good wheel sinkage estimation of
off-road vehicles on soil, legged robots face a more complicated scenario.
Large-scale numerical computation algorithms such as discrete element method
and material point method can predict forces accurately for diverse movements
(Dunatunga and Kamrin, 2017; Wang et al., 2021). However, it is impractical to
use these methods for real-time estimation and robot control applications. An
alternative computation method, namely resistive force theory (RFT), was used
to capture the interaction of arbitrary shapes with dry granular materials for
2D and 3D trajectories (Li et al., 2013; Treers et al., 2021). RFT has already
been successfully applied to calculate reaction forces for wheels rotating on
sand (Agarwal et al., 2019, 2021; Huang et al., 2022) and generalized for
similar dry materials by introducing a scale factor (Zhang and Goldman, 2014).
One static intrusion reaction force model was used to rationalize a static
stability region criterion for the inverted pendulum model on granular
materials (Xiong et al., 2017). Nevertheless, few works focus on complex foot
shape impact on granular materials following resistive force theory
principles.

In this paper, we follow the general rules of RFT to quantify the reaction
forces during the interaction of robot feet with dry granular terrains. We
present an optimization approach for robot foot shapes by considering the
compensation energy and walking efficiency. The intrusion orientation and
motion direction of any arbitrary point on the foot contour are calculated. A
computational method is proposed to calculate the ground resistive forces on
walking dynamics by RFT. Various commonly used foot shapes are used as
examples to analyze and compare the robot walking performance. Finally, we
optimize the locations of finite waypoints to identify and estimate the foot
shape that generates energy-saving and efficient walking performance.
Different types of granular materials are also tested for the optimized foot
shape. It is found that the optimized foot shape is non-convex for energy-
efficient walking gaits on hard granular terrains. The main contribution of
this work lies in that the integrated bipedal walking gait dynamics with the
resistive force theory for studying foot-granular terrain interactions. The
proposed integration foot-ground modeling framework is not restricted for only
minimizing energy efficiency for bipedal walking on granular materials, and it
indeed lays down the foundation for future work such as balance and gait
control for bipedal robotic walkers on deformable, granular terrains.

The rest of this paper is organized as follows. Section 2 presents a
computational method that integrates the RFT for the robot walking on the
granular terrain. Section 3 discusses the walking energy efficiency with
various foot shapes on granular terrain. We present the foot shape
optimization in Section 4. Finally, results and conclusions are presented in
Sections 5 and 6, respectively.

##  2 Bipedal Dynamics on Granular Terrains

###  2.1 Robot Foot Intrusion Forces through RFT

Fig. 1 illustrates the systems configuration of the bipedal walker on granular
terrains. A double-link and one-lumped-mass model is used to present the
motion of the robotic walker which is in the sagittal plane. Besides the
inertial frame ($G:\,xoz$), a local body-fixed frame ($B:x_{B}o_{B}z_{B}$) at
the center of mass (COM) and a local ankle frame ($A:x_{A}o_{A}z_{A}$) are
established to describe relative positions and velocities. The double-link
legs are symmetric and the lengths of the thing and shank links are denoted as
$l_{1}$ and $l_{2}$, respectively. The hip (with the vertical line) and knee
angles are denoted as $\theta_{1}$ and $\theta_{2}$, respectively.

Figure 1: The schematic of a humanoid robot walking and the arbitrary-shaped
foot intrusion with granular materials. RFT is illustrated with discrete
elementary plates.

The robot foot intrusion problem is also illustrated in Fig. 1. To simplify
the problem, the foot is considered as rigid and the RFT rules are applied to
intrusion of vertical plates allowing asymmetric stress distributions. The
resistive forces are also considered to be uniformly distributed in the foot
width direction so that the intrusion forces and motion of interest are only
considered in the $x$-$z$ plane. It is further assumed that the intrusion
forces are not sensitive to foot gait velocity under a slow interaction
motion.

We consider the robotic foot with an arbitrary out-shape contour
$\mathcal{C}$. The entire contour $\mathcal{C}$ is partitioned into an $N$
elementary plates with the same length $d\mathcal{C}$. The width of each plate
is negligible. For the $i$th plate, according to RFT, the drag and lift forces
of intrusion parts, denoted respectively as $f_{x}^{i}$ and $f_{z}^{i}$, are
approximately proportional to the intrusion depth, $|z|$ (free level of
granular materials is $z=0$) as

$f_{j}^{i}=\alpha_{j}(\beta_{i},\gamma_{i})|z_{i}|H(-z_{i}),\;j=x,z,$ (1)

where $i=1,\ldots,N$. $\alpha_{x}$ and $\alpha_{z}$ reflect local stresses in
the horizonal and vertical direction per unit depth, respectively. $H(x)$ is
Heaviside function. As shown in Fig. 1, for the $i$th plate, the orientation
angle, $\beta_{i}$ is defined clockwise with respect to $x$-axis while the
plate moves towards a certain direction described by a velocity vector angle
$\gamma_{i}$. The total horizontal and vertical forces applied on feet are

$F_{j}=\sum_{i=1}^{N}\alpha_{j}(\beta_{i},\gamma_{i})|z_{i}|H(-z_{i}),\;j=x,z.$
(2)

The local horizontal and vertical stresses $\alpha_{x}$ and $\alpha_{z}$ are
obtained by small plate intrusion experiments and gathered as two 2-D force
lookup maps with respect to intrusion orientations ($\beta_{i}$) and motion
directions ($\gamma_{i}$). Given a foot contour and a walking gait, the plate
orientation and motion direction are calculated as

$\beta_{i}=\tan^{-1}{\left(\frac{z_{i2}-z_{i1}}{x_{i2}-x_{i1}}\right)},\gamma_{i}=\tan^{-1}{\left(\frac{v_{iz}}{v_{ix}}\right)},$
(3)

where $(x_{i1},z_{i1})$ and $(x_{i2},z_{i2})$ are the two vertices coordinates
of the $i$th plate and $\textbf{\em v}_{i}=[v_{ix}\;v_{iz}]^{T}$ is the
velocity at the center of each plate in the inertial frame ($xoz$). For
humanoid walking gaits, the stance foot rolls on the terrain and swing foot
touches down with a low relative velocity with respect to the COM. Hence, the
intrusion velocity is in a small value region where the local stresses are not
sensitive to the velocity (Ding et al., 2011; Li et al., 2013). We do not
consider cases with high intrusion velocity such as in running gaits in this
paper.

###  2.2 Kinematics and Kinetics of Bipedal Walker

#### Kinematics

For an arbitrary point $p_{i}$ of the foot, the position and corresponding
velocity in the global frame are calculated as

$^{G}\textbf{\em r}_{p_{i}}=~{}^{G}\mathbf{R}^{A}~{}^{A}\textbf{\em
r}_{p_{i}}+~{}^{G}\textbf{\em r}_{A},~{}^{G}\textbf{\em
v}_{p_{i}}=\dot{{}^{G}\mathbf{R}^{A}}~{}^{A}\textbf{\em
r}_{p_{i}}+~{}^{G}\textbf{\em v}_{A},$ (4)

where ${}^{A}\textbf{\em r}_{p_{i}}$ is the relative position in the local
ankle frame, ${}^{G}\mathbf{R}^{A}$ is the transform matrix from the local
ankle frame $A$ to the inertial frame $G$, namely,

${}^{G}\mathbf{R}^{A}=\begin{bmatrix}\mathop{\rm
c}\nolimits_{\theta_{A}}&-\mathop{\rm s}\nolimits_{\theta_{A}}\\\ \mathop{\rm
s}\nolimits_{\theta_{A}}&\mathop{\rm
c}\nolimits_{\theta_{A}}\end{bmatrix},\;^{G}\dot{\mathbf{R}}^{A}=~{}^{G}\omega_{A}\begin{bmatrix}-\mathop{\rm
s}\nolimits_{\theta_{A}}&-\mathop{\rm c}\nolimits_{\theta_{A}}\\\ \mathop{\rm
c}\nolimits_{\theta_{A}}&-\mathop{\rm s}\nolimits_{\theta_{A}}\end{bmatrix},$

where ${}^{G}\omega_{A}$ is the local ankle frame rotation angular velocity,
$\theta_{A}$ is the ankle joint angle, and notations $\mathop{\rm
c}\nolimits_{\theta_{A}}=\cos\theta_{A}$ and $\mathop{\rm
s}\nolimits_{\theta_{A}}=\sin\theta_{A}$ for $\theta_{A}$. To obtain the
position and corresponding velocity of point $p_{i}$, we can calculate the
inverse kinematics of the robot leg in the local body frame $B$ such that
$\theta_{A}=\pi/2-(\theta_{1}+\theta_{2})$ and
${}^{G}\omega_{A}=~{}^{B}\omega_{A}=\omega_{1}+\omega_{2}$ for 2D problem.
Furthermore, in this paper, we treat the robot walking gait as a given profile
and the relative ankle position and velocity in the body-fixed frame, $B$, can
be calculated in advance.

#### Kinetics

The ground reaction forces (GRFs) applied on foot feet are directly
transmitted to the bipedal walker body with a mass, $M$. Denoting
${\bm{r}}_{C}=[x_{COM}\;z_{COM}]^{T}$ is the COM position, the kinetics of the
COM is written as

$\ddot{{\bm{r}}}_{C}=\dot{{\bm{v}}}_{C}=\frac{1}{M}\textbf{\em F},$ (5)

where ${\bm{a}}_{C}=\dot{\textbf{\em v}}_{C}=[a_{x}\;a_{z}]^{T}$ is the
acceleration of the COM in frame $G$ and $\textbf{\em
F}=[F_{x}\;F_{z}-Mg]^{T}$. $F_{x}$ and $F_{z}$ are the GRFs given by (2).
Given a small time step $\Delta t$, the velocity and position states of COM
are updated by the explicit forward integration method,

${\bm{v}}_{C}(t+\Delta t)={\bm{v}}_{C}(t)+{\bm{a}}_{C}(t)\Delta
t,{\bm{r}}_{C}(t+\Delta t)={\bm{r}}_{C}(t)+{\bm{v}}_{C}(t)\Delta t.$ (6)

##  3 Walking Energy Compensation

###  3.1 Instantaneous Power and Cumulative Work

During the foot intrusion partially through granular terrain, the distributed
resistive forces $f_{x}^{i}$ and $f_{z}^{i}$ on the foot perform negative work
because the forces result in the opposite motion direction of the intruder.
The bipedal walker needs to compensate for the negative work performed by
reaction forces to maintain the normal gait. Therefore, the energy expense by
the walker is a significant factor for the walking efficiency with given
terminal states (e.g., the forward walking distance and average walking
velocity). The power and work performed by resistive forces are calculated

$P_{i}=\textbf{\em f}_{i}\cdot\textbf{\em
v}_{i},\;W=\int_{0}^{t}{\left(\sum_{i=1}^{N}P_{i}\right)dt}.$ (7)

where $\textbf{\em f}_{i}=[f_{x}^{i}\;f_{z}^{i}]^{T}$, $\textbf{\em
v}_{i}=[v_{ix}\;v_{iz}]^{T}$ and $P_{i}$ are the force and velocity vectors
and the instantaneous power of the $i$th plate, and $W$ is the cumulative
work, respectively.

Figure 2: (a) Human walking with optical markers on hip, knees, and ankles.
(b) The motion profiles of the left and right ankle positions relative to the
hip. Figure 3: Five types of different foot shapes. 1: elliptical curve; 2:
rectangle; 3: circular curve; 4: reversed-L; 5: reversed triangle.

Figure 4: Comparison results under five different foot shapes. (a) The body
COM position ${\bm{r}}_{C}$ and velocity ${\bm{v}}_{C}$ profiles. The first
row is the $(x_{COM},z_{COM})$ and the second and third rows are horizontal
$V_{x}$ and vertical velocity $V_{z}$, respectively. (b) Power ($P_{l}$ and
$P_{r}$) and cumulative work ($W_{l}$ and $W_{r}$) by resistive forces on the
left and right feet.

###  3.2 Walking Analysis of Different Foot Shapes

Foot-granular terrain intrusion depends on walking gaits. In this work, we use
human walking gaits as the reference input to the intrusion forces calculation
and then focus on the foot-shape impact on intrusion forces and resultant
walker dynamics. Fig. 2 shows the experimental setup for human subject walking
on the flat ground with motion capture systems to obtain the hip, knee, and
ankle joint angles. The temporal ankle position profile in the global frame is
then transferred to a spatial profile relative to the hip shown in Fig. 2.

Five different foot shapes are selected in this study and Fig. 3 illustrates
these shapes, including elliptical (e.g., C-shape Xu et al. (2015)), rectangle
(e.g., flat foot Xiong et al. (2017)), circular (e.g., Martin and Schmiedeler
(2011)), reversed-L shape (Wang et al. (2021)), and triangle shape. These
shapes are selected since they are previously reported in literature. To make
a fair comparison, the effective foot-terrain contact lengths of all shapes,
that is, the red lines in Fig. 3, are the same. Similarly, to minimize the
possible geometry effect, the aspect ratio (long and short axes) of the
ellipse is set as $2:1$ and for other shapes, the ratio $a:b$ of side lengths
$a$ and $b$ is also set as $2:1$.

We conducted simulation studies for a bipedal walker with mass $M=60$ kg and
with the above five-mentioned foot shapes. Walker body position ${\bm{r}}_{C}$
and velocity ${\bm{v}}_{C}$ and the energy expenses to compensate for the
intrusion forces are used as comparison metrics. Fig. 4 shows the COM position
${\bm{r}}_{C}$ and velocity ${\bm{v}}_{C}$ under five foot shapes. Fig. 4
shows the corresponding power and work for the left and right legs. From these
results, we obtain the following observations.

First, among the five foot shapes, the walker with the triangular foot walks
slightly fastest for the given ankle motion profile, followed by the flat foot
as the next one. The walker with the circular foot however maintains the
highest COM height and ends at the shortest forward walking distance. Second,
the walker’s horizontal velocity with the triangle feet is highest, while the
circle feet give a significant velocity reduction at each stance switch. No
obvious vertical velocity difference is observed among these five foot shapes.
Finally, the walker with circular feet needs the highest power during the
single stance phase while elliptical and rectangular feet have the least.
Therefore, by given walker’s gait profile, the gait stability and energy
efficiency depend on the foot shapes and their intrusion on granular terrains.

##  4 Optimal Foot Contour Design

###  4.1 Waypoints of Foot Contour

Fig. 5 illustrates the schematic of the walker’s foot contour under a set of
waypoints, which allows flexibility to adapt the shape corresponding to
intrusion interactions. Considering the real foot-terrain interaction,
granular materials intrusion mainly happens at the foot bottom and a set of
$n$ waypoints, denoted by $p_{i}$, $i=1,\ldots,n$, are set up at the bottom
contour. The foot contour is restricted in a rectangular domain ($2L\times
2H$, gray shadowed area) with a local foot frame $x_{p}o_{p}z_{p}$ at the
center. For simplicity, waypoints $p_{i}$, $i=1,\ldots,n$, are located
uniformly in the $x_{p}$ direction while the $z_{p}$ coordinates of points are
free bounded in the domain. A cubic spline is then used to connect any two
adjacent points and the coordinates of $p_{i}$ are

${\bm{p}}_{i}=\left(\frac{(i-1)(3-n)}{n-1}L,~{}-k_{i}\frac{H}{K}\right),\;i=1,\ldots,n,$
(8)

where $K\in\mathbb{N}$ is the total number of discretization of the height $H$
and $k_{i}\in\mathbb{Z_{+}},k_{i}\leq K$ is the index location of point
$p_{i}$ in the $z_{p}$ direction. Therefore, for each $p_{i}$, we just need to
determine value of $k_{i}$s and this can be formulated as an integer
optimization problem. We define design variable
${\bm{k}}=[k_{1}\;k_{2}\;\cdots\,k_{n}]^{T}$ for the optimization problem that
will be introduced later in this section.

Figure 5: The schematic of the walk foot with finite waypoints to control the
contour. The foot shape is controlled by a set of waypoint location
${\bm{p}}_{i}$.

###  4.2 Optimization of Foot Contour

We formulate an optimization problem with the following considerations. First,
for given a certain period $t_{f}$, the bipedal walker should walk for a as
long distance as possible, that is, $x_{COM}(t_{f})$ is large as possible.
Second, the foot sinkage should be as small as possible because drag force and
compensation work arise as the intrusion depth increases. We evaluate the
sinkage depth through the (average) height of $z_{COM}(t)$. Third, during a
single-stance walking phase, the instantaneous power peak
$p_{\max}=\max{\left(\sum_{i=1}^{N}P_{i}\right)}$ should be small for reducing
impact effect. The work $|W(t_{f})|$ by the drag and lift forces should be
small to save energy. Finally, the walker should not lose too much forward
velocity change $\Delta V_{x}$ when the walker switches the stance legs.

Figure 7: One step on hard sand from touching down to pushing off. The first
row is the optimal foot shape of hard sand (red dashed line in Fig. 6) and the
second row is the optimal shape of regular sand (green solid line in Fig. 6).
Red arrows reflect the intrusion force distribution showing the directions and
magnitudes.

The cost function, denoted by $J_{W}$, that reflects aforementioned objectives
is chosen as a form of positive and negative rewards as

$J_{W}=-\frac{R_{p}}{R_{n}},$ (9)

where $R_{p}$ and $R_{n}$ indicates the positive and negative rewards,
respectively,

$R_{p}=x_{COM}(t_{f})\bar{z}_{COM}(t),\;R_{n}=|W(t_{f})|p_{\max}\Delta V_{x},$
(10)

and $\bar{z}_{COM}(t)$ is the mean value of ${z}_{COM}(t)$ over period
$t_{f}$. Therefore, the optimization problem for the foot-shape design is
formulated as

$\displaystyle\min_{{\bm{k}}}~{}J_{W}$ (11) subj
$\displaystyle\text{to:}~{}~{}k_{i}\in\mathbb{Z_{+}},~{}k_{i}\leq
K,\,i=1,\ldots,n.$

The cost function $J_{W}$ can be viewed as an index of walking efficiency
since the term $\frac{x_{COM}(t_{f})}{|W(t_{f})|}$ actually reflects the
walking distance per unit work. The larger is this value, the more efficiently
the robot walks on the granular terrain. The optimization problem (11) is
solved by genetic algorithm for the mixed integer optimization. We will
present the optimal results in the next section.

##  5 Results and Discussions

###  5.1 Simulation Setup

We first determine the simulation model parameters such as foot shape domain
size, the domain grid, and the number of intrusion plates. We consider a human
walker in the simulation study. The bipedal walker first stands with the left
leg steadily and steps forwards following the given human walking gait profile
on granular terrains. The simulation ends at a terminal time $t_{f}$. Table 1
lists the values of the simulation parameters used in this study.

Table 1: The parameter values for the bipedal walkers and foot shape.

$L$ | $H$ | $n$ | $K$ | $N$ | $M$ | $l_{1}$ | $l_{2}$
---|---|---|---|---|---|---|---
$0.13$ m |  $0.03$ m | $11$ | $10$ | $100$ |  $60$ kg |  $0.47$ m |  $0.45$ m

Figure 6: Three optimal foot contours for walking on three different types of
sand.

Figure 8: (a) The COM position and velocities on three different types of
sand. (b) The ground reaction forces during walking on the sand. The left
column: drag force; the right column: lift (normal) force. (c) Instantaneous
power and cumulative work compensation caused by sand resistive forces on
right legs. The left column: instantaneous power; the right column: work.

Three different types of sand are tested for analysis of the impact on the
foot contour and walking dynamics. We label “regular”, “soft”, and “hard” to
describe the intrusion interaction type of the foot with sand. The intrusion
interaction is characterized by the local stress maps that are described by
$\alpha_{x}(\beta,\gamma)$ and $\alpha_{z}(\beta,\gamma)$. One simple scale
factor $\zeta$ is used to distinguish different types of granular materials.
The specific local stress $\alpha_{j}=\zeta\alpha_{j}^{\mathrm{generic}}$,
$j=x,z$, where $\alpha_{j}^{\mathrm{generic}}$ is the local stress of generic
materials discussed in Zhang and Goldman (2014). In this paper, $\zeta_{r}=1$,
$\zeta_{s}=0.2$, and $\zeta_{h}=5$ are for regular, soft, and hard sand,
respectively. We chose these three numbers primarily to significantly
distinguish the local stress difference in the results.

###  5.2 Optimal Foot Contours on Different Types of Sand

Fig. 6 shows the optimal foot contour results for different types of sand. It
is noted that for walking on the hard sand the optimal foot shape appears
unnecessarily convex, while the contour tends to be the curve with a small
curvature for walking on regular and soft sand. This is partially because the
small-curvature shape provides large effective contact zone to avoid
significant sinkage. Fig. 7 illustrates the process of the foot stepping on
the hard sand with foot shapes optimized for hard and soft sands. From (heel)
touch-down to (toe) push-off, the concave shape (red dashed line in Fig. 6)
reduces unnecessary contact zone in the middle area of the foot. This leads to
less numbers of intrusion plates and saves the work. From the force
distribution results, we can see that the step pattern of the single stance
with the concave foot shape is similar to the human walker: the force moving
the body forwards is mostly provided in the (toe) push-off phase (e.g.,
$t>0.27$ s). However, for the “overall convex” curved foot shape like the
green solid line in Fig. 6, a larger contact zone is involved during the
interaction and shifts to the leading part of the foot, which reveals that the
duration of the push-off phase becomes longer (good for increasing the forward
velocity) while larger instantaneous power and work are generated to be
compensated by the joint actuators.

Fig. 8 shows the walker’s COM position and velocity profiles. The walker can
walk furthest over the same time period on the hard sand that is closed to
normal firm ground, while the sinkage level is the smallest among three types
of sand terrain. The soft sand leads to the largest sinkage and shortest
walking distance. From velocity profiles, the walker experiences an obvious
loss of the horizontal velocity at the time when changing the stance leg
(i.e., at $t=1.1$, $1.7$, and $2.3$ s on the soft sand. Meanwhile, the
walker’s acceleration on the soft sand is also smallest, resulting more time
to reach the normal walking velocity around $1$ m/s. Fig. 8 shows the GRF
profiles (intrusion drag and lift forces). The larger sinkage on the soft sand
results in larger drag forces in both the fore and aft directions (blue dashed
line) and longer contact/intrusion phase compared with the results on regular
and hard sand. Fig. 8 shows the corresponding instantaneous power and
cumulative work of both legs. After three steps, the cumulative work on soft
sand is largest. All results demonstrate that walking on soft granular
materials needs energy to compensate for resistive forces as well as a short
walking distance with a slow velocity, which implies that soft terrain reduces
the walking efficiency.

##  6 Conclusions and Future Work

We presented a computational approach to calculate the ground reaction forces
during the interaction of bipedal walker feet with dry granular terrains. The
method first determined the foot/terrain intrusion interaction motion. We then
quantified five different foot shapes by the walker’s COM motion profile, the
ground resistive forces and the energy efficacy. A multi-objective
optimization problem for foot shape design was formulated to minimize the work
compensation for intrusion forces. Different types of sand were used for
testing the walking gait profiles with the optimized foot shapes. The
simulation results showed that the non-convex curved foot shape generated the
best energy efficiency for the human walking gait on hard granular terrain.
One ongoing research task is to conduct experimental validation of the
proposed interaction model. Furthermore, human walking locomotion with
assistance of knee exoskeletons (Chen et al., 2021) and gait trajectory
(including velocity) effect will be investigated for advanced bipedal
locomotion control on granular terrains.

## References

  * Adamczyk et al. (2006) Adamczyk, P.G., Collins, S.H., and Kuo, A.D. (2006).  The advantages of a rolling foot in human walking.  _J. Exp. Biol._ , 209(20), 3953–3963.
  * Agarwal et al. (2021) Agarwal, S., Karsai, A., Goldman, D.I., and Kamrin, K. (2021).  Surprising simplicity in the modeling of dynamic granular intrusion.  _Sci. Adv._ , 7(17), eabe0631.
  * Agarwal et al. (2019) Agarwal, S., Senatore, C., Zhang, T., Kingsbury, M., Iagnemma, K., Goldman, D.I., and Kamrin, K. (2019).  Modeling of the interaction of rigid wheels with dry granular media.  _J. Terramech_ , 85, 1–14.
  * Chen et al. (2021) Chen, S., Stevenson, D.T., Yu, S., Mioskowska, M., Yi, J., Su, H., and Trkov, M. (2021).  Wearable knee assistive devices for kneeling tasks in construction.  _IEEE/ASME Trans. Mechatronics_ , 26(4), 1989–1996.
  * Ding et al. (2011) Ding, Y., Gravish, N., and Goldman, D.I. (2011).  Drag induced lift in granular media.  _Phy. Rev. Lett._ , 106(2).  article 028001.
  * Dunatunga and Kamrin (2017) Dunatunga, S. and Kamrin, K. (2017).  Continuum modeling of projectile impact and penetration in dry granular media.  _J. Mech. Phys. Solids_ , 100, 45–60.
  * Huang et al. (2022) Huang, L., Zhu, J., Yuan, Y., and Yin, Y. (2022).  A dynamic resistive force model for designing mobile robot in granular media.  _IEEE Robot. Autom. Lett._ , 7(2), 5357–5364.
  * Li et al. (2013) Li, C., Zhang, T., and Goldman, D.I. (2013).  A terradynamics of legged locomotion on granular media.  _Science_ , 339(6126), 1408–1412.
  * Martin and Schmiedeler (2011) Martin, A.E. and Schmiedeler, J.P. (2011).  Experimental validation of a walking model for planar bipeds with curved feet.  In _Proc. ASME Design Eng. Tech. Conf. Comp. Inform. Eng. Conf._ , volume 54839, 777–785.
  * Martin and Schmiedeler (2012) Martin, A.E. and Schmiedeler, J.P. (2012).  The effects of curved foot design parameters on planar biped walking.  In _Proc. ASME Design Eng. Tech. Conf. Comp. Inform. Eng. Conf._ , volume 45035, 815–824.
  * Treers et al. (2021) Treers, L.K., Cao, C., and Stuart, H.S. (2021).  Granular resistive force theory implementation for three-dimensional trajectories.  _IEEE Robot. Autom. Lett._ , 6(2), 1887–1894.
  * Trkov et al. (2019) Trkov, M., Chen, K., and Yi, J. (2019).  Bipedal model and extended hybrid zero dynamics of human walking with foot slips.  _ASME J. Computat. Nonlinear Dyn._ , 14(10).  article 101002.
  * Wang et al. (2021) Wang, G., Riaz, A., and Balachandran, B. (2021).  Continuum modeling and simulation of robotic appendage interaction with granular material.  _ASME J. Appl. Mech._ , 88(2).  article 021013.
  * Xiong et al. (2017) Xiong, X., Ames, A.D., and Goldman, D.I. (2017).  A stability region criterion for flat-footed bipedal walking on deformable granular terrain.  In _Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst._ , 4552–4559. Vancouver, Canada.
  * Xu et al. (2015) Xu, L., Zhang, S., Jiang, N., and Xu, R. (2015).  A hybrid force model to estimate the dynamics of curved legs in granular material.  _J. Terramech_ , 59, 59–70.
  * Zhang and Goldman (2014) Zhang, T. and Goldman, D.I. (2014).  The effectiveness of resistive force theory in granular locomotion.  _Phy. Fluids_ , 26(10).  article 101308.