| 摘要: | 這些年來隨著深度學習對人們生活的影響越來越大，人們越來越重視這個領域的發展。其中從單鏡頭rgb圖像預測人的手部pose和shape的任務一直是計算機視覺領域長期存在的問題。不同於常見的手部姿態預測那樣只預測手部骨架點的坐標，這項任務要還原出手部原本的外形。很多地方都會應用到這個任務例如增強現實（augmented reality）和虛擬現實（virtual reality），但是由於手部佔圖像面積較小的部分，手部動作靈活多樣且容易遮擋，所以這項任務任然非常具有挑戰性。
我們在本文中提出來一種完整的端到端網路架構，可以從rgb手部圖像得到3D mesh的手部形狀。具體地來說，在編碼器的部分，我們使用的是ResNet-50來提取圖像特征，為了後面更好的回歸模型參數，我們通過一些卷積層得到一些2D的特征圖，例如2D heatmap和mask圖像。在模型參數回歸的部分我們使用了全連接層用迭代回歸的方式進行模型參數的回歸。因為model-base的方法生成的手部模型都會有一些缺陷，例如不夠自然。所以最後我們添加了手部mesh坐標修正的部分，我們把模型生成的手部模型（MANO）當做粗糙的初始手部模型，接著添加進前面網路的一些特征，進入圖卷積網路層回歸出每個坐標點的偏移量，最後加到初始手部模型上得到最終的手部模型。
;In recent years, as the impact of deep learning on people′s lives has grown, more and more attention has been paid to the development of this field. The task of human hand pose and shape estimation from a rgb image has been a long-standing problem in the field of computer vision. Unlike common hand posture prediction, which only predicts the coordinates of the skeletal points of the hand, this task restores the original shape of the hand. Many places will apply this task such as augmented reality and virtual reality, but the task is still very challenging because the hand occupies a relatively small part of the image area, and the hand movements are flexible and easy to block.
In this paper we propose a complete end-to-end network architecture to obtain 3D mesh hand shapes from rgb hand images. Specifically, in the encoder part, we use ResNet-50 to extract the image features, and for better regression of model parameters later, we obtain some 2D feature maps, such as 2D heatmap and mask images, through some convolutional layers. In the model parameter regression part, we use the fully connected layer for iterative regression of the model parameters. Because the hand models generated by the model-base method have some defects, such as not natural enough. So finally we add the hand mesh coordinate correction part, we treat the hand model (MANO) generated by the model as the rough initial hand model, then add some features from the previous network, enter the graphical convolutional network layer to regress the offset of each coordinate point, and finally add it to the initial hand model to get the final hand model. |
