JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 1

Pixel-wise Regression: 3D Hand Pose Estimationvia Spatial-form Representation and Differentiable

DecoderXingyuan Zhang, Fuhai Zhang

Abstract—3D Hand pose estimation from a single depth imageis an essential topic in computer vision and human-computerinteraction. Although the rising of deep learning method booststhe accuracy a lot, the problem is still hard to solve due to thecomplex structure of the human hand. Existing methods withdeep learning either lose spatial information of hand structureor lack a direct supervision of joint coordinates. In this paper, wepropose a novel Pixel-wise Regression method, which use spatial-form representation (SFR) and differentiable decoder (DD) tosolve the two problems. To use our method, we build a model,in which we design a particular SFR and its correlative DDwhich divided the 3D joint coordinates into two parts, planecoordinates and depth coordinates and use two modules namedPlane Regression (PR) and Depth Regression (DR) to dealwith them respectively. We conduct an ablation experiment toshow the method we proposed achieve better results than theformer methods. We also make an exploration on how differenttraining strategies influence the learned SFRs and results. Theexperiment on three public datasets demonstrates that our modelis comparable with the existing state-of-the-art models and in oneof them our model can reduce mean 3D joint error by 25%.

I. INTRODUCTION

Hand pose estimation is always an essential topic in com-puter vision and human-computer interaction [1] since thehuman hand is one of the most important things to understandthe intention of human, and how humans interact with anotherobject. This field is greatly boosted by the arising of deeplearning and availability of depth cameras such as MicrosoftKinect and Intel Realsense [2]. However, the problem is stillvery hard to solve, because the complex structure of humanhands, along with variety of view-point, causes severe self-occlusion in the image.

3D Hand pose estimation based on single-frame depthimages mostly uses convolution neural networks (CNNs)based methods [3]. The most popular method is regression-based method, which treat hand pose estimation as an end-to-end learning problem. That is, they build a gigantic neuralnetwork to regress the 3D coordinates of each joint point inan end-to-end fashion and let the neural network find outwhat features it should learn. The obvious defect of thisregression-based method is that the output of CNNs must beflattened to go through full-connection (FC) layers to get thejoint coordinates. This flatten operation completely destroy thespatial information in the original image, so it is unfavorable

The authors are with the State Key Laboratory of Robotics and Sys-tem, Harbin Institute of Technology, Harbin 150001, China (e-mail: zfh-hit@hit.edu.cn)

for recognizing such a complex structure as the hand and resultin a highly nonlinear regression problem.

In order to preserve the spatial information of humanhands, the most commonly used method is the detection-basedmethod with heatmaps, which also has a lot of applicationsin other similar fields such as human pose estimation. Thedetection-based method uses fully convolution network (FCN)to maintain the spatial information, and instead of joint co-ordinates they design an encoded spatial-form representation(SFR) as the learning target. Then a decoder should beused to convert the encoded SFR back to joint coordinates.Heatmaps (or sometimes called believe maps [4]) is the mostcommonly used and the most native choice for this encodedSFR. Research [3] shows that detection-based method has ahigher accuracy than the regression-based method.

However, heatmap is only an approximate representationfor 2D coordinates in plane space, so it is not capable to dealwith 3D coordinates even when the depth image is providedin 3D hand pose estimation problem. Because the depth imageis only a projection of the real 3D data, there exists anoffset between actual joint depth value and the value on thedepth image. This offset can vary a lot among different jointsand different gestures for the severe self-occlusion and heavynoise in depth image. Therefore, in order to use detection-based method in 3D hand pose estimation, researchers eithercome up with additional representations along with heatmapor transform the depth image back to the 3D space then use3D heatmap or other representations.

Although the detection-based method maintains the spatialinformation with SFR, it also has defects. First, as we talkabove, the depth image is only a projection of the real 3D data,so it is nontrivial to define such a transformation that perfectlyreconstructs the original 3D data from a single depth image.Second, the detection-based method lacks a direct supervisionfrom the 3D coordinates. Unlike the regression-based method,in detection-based method, we supervise SFR instead of 3Dcoordinates. Since the way we build SFR is essentially en-coding a low-dimensional data into high-dimension, the SFRmust contain some redundancy. The decoder that deals withthese redundancies are always non-linear and nondifferentiablein the most of time, thus has to be put outside the neuralnetwork. This nondifferentiable structure disable the directsupervision form the 3D coordinates and build a gap betweenwhat the neural network learns and what we really want. Likeany complex non-linear system, there is no guarantee that aslight error in the SFR wont result in a dramatic change in 3D

arX

iv:1

905.

0208

5v2

[cs

.CV

] 2

7 M

ay 2

019

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 2

Fig. 1. Three types of 3D hand pose estimation method. (a) shows the regression-based method which uses CNNs to extract features and then uses FCsto regress the joint coordinates. The flatten operation between CNNs and FCs destroy the spatial information. (b) shows the detection-based method whichuses CNNs to extract a designed spatial-form representation then uses a nondifferentiable decoder outside the neural netwoek to convert the SFR to the jointcoordinates. Some models may perform a data transformation to convert the input depth image to another form like voxel map. (c) shows our proposedPixel-wise Regression method which is a combination of the former two methods. We use CNNs to extract a designed spatial-form representation like thedetection-based method, and enables the direct supervision of 3D coordinates like regression-based method by putting a differentiable decoder inside theneural network.

coordinates. The 3D coordinates just make the situation worsebecause, in plane space, we need more than one form of SFRto encode the 3D coordinates. Since the decoder is outsidethe neural network, we cannot directly use the 3D coordinatesto supervise the learning. Without such supervision, the SFRsmay not corporate with each other.

In this paper, in order to overcome the defects of existingmethod we state above, we integrate existing methods andpropose a Pixel-wise Regression method, which uses spatial-form representation to maintain the spatial information andenables the direct supervision of 3D coordinates by putting adifferentiable decoder (DD) inside the neural network. To useour method, we build a particular model, in which we designa particular SFR and its correlative DD which divided the 3Dcoordinates into two parts, plane coordinates and depth coordi-nates and use two modules named Plane Regression (PR) andDepth Regression (DR) to deal with them respectively. Therelationship among our method and regression-based methodand detection-based method is shown in Fig. 1.

The contributions of this work are summarized as follows:• We propose a novel method for 3D pose estimation

called Pixel-wise regression, which is a combination ofexisting two popular methods. Our method uses SFR of3D coordinates in plane space to maintain the spatialinformation, and enables the direct supervision of 3Dcoordinates by putting a DD inside the neural network.

• We design a particular SFR and its correlative DD tobuild a model for 3D hand pose estimation problem fromsingle depth image. We divided the 3D coordinates intotwo parts, plane coordinates and depth coordinates anduse two modules named PR and DR to deal with themrespectively.

• We implement several baseline models and conduct anablation experiment to show that our method can achieve

better results than former methods. We also study howdifferent training strategies influence the learned SFRsand provide meaningful insights. The experiment onthree public datasets shows that our model can achievecomparable result with the state-of-the-art models.

The remainder of the paper is organized as follows. InSection II, we give a brief review of the regression-based anddetection-based methods and point out the main differencesbetween our models and their models. In Section III, weformalize the problem and describe our proposed Pixel-wiseRegression method. We show how we design the SFR andits correlative DD to fulfill the requirement of our method.We also demonstrate other modules we use in our modeland the training details. In Section IV, we designed ourablation experiments to analyze our method and performedexperiments on three challenging public datasets, MSRA,ICVL and HAND17. Experiments show that our model canachieve state-of-the-art levels on the test set. Finally, we makea conclusion and discuss future potential of our method inSection V.

II. RELATED WORK

In this section, we will review some related works ofour proposed model. Firstly, we will review the regression-based method, which is the most popular method in handpose estimation. Secondly, we talk about the detection-basedmethod that, in general, make a more accurate prediction thanthe regression-based method but lack the direct supervisionfrom 3D coordinates. Our Pixel-wise regression method is acombination of these two methods.

A. Regression-based MethodRegression-based method [5]–[14] is widely used in hand

pose estimation field. It treats hand pose estimation as an

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 3

end-to-end problem and regresses the 3D coordinate of eachjoint directly. Some of these methods have learned the low-dimensional space or latent space representation of the handlocation. For example, Oberweger, M et al. [11] believethat joint vectors can be regarded as some low-dimensionalspatial representations. They use a special bottleneck structureto force the network to learn such pattern, but since thislow-dimensional space is originally a kind of approximateexpression, its effect is not good. X. Zhou et al. [12] usedthe joint angle as a hidden variable to learn angle informationin the network and convert the angle into 3D coordinatesthrough a predefined hand model. This model requires a pre-defined hand model, lacking generalization ability for newsamples. Both of these models have destroyed the spatialinformation with a flattened representation. There are alsosome attempts to include spatial information in the network.Chen, Xinghao et al. [13] used local feature informationto correct 3D coordinates. However, because it uses localsegmentation to obtain local information, the network canobtain limited information and lack the information about theoverall structure of the hand. Wu, X, etc. [14] adopts a schemecombined with the detection-based method, which is similarto ours, but it needs to first convert the input depth map intoa three-dimensional voxel map, and our method is to obtainthe 3D coordinates directly in the plane space.

B. Detection-based MethodComparing to the regression-based approach, detection-

based methods [15]–[18] hopes use an intermediate SFR tomaintain the spatial inforamtions, but often requires artificiallydesign of SFR and lack direct supervision from the 3Dcoordinates. Since the method of directly applying heatmapcan solve the problem of 2D coordinate recognition well, thedetection-based method based on heatmap is very commonlyused in human pose estimation. However, the detection-basedmethod is not easy to implement in the hand pose estimationproblem, because in addition to the need to identify the 2D co-ordinates in the image, it is also necessary to identify the depthcoordinates, which is difficult to represent with a heatmap.However, due to the high precision and intuitiveness of thedetection-based method, many scholars still try to use theDetection-based method in hand pose estimation. Tompson, Jet al. [16] used a heatmap for 2D coordinate recognition, andthen used a hand model-based iterative method PSO for post-processing to obtain depth coordinates. Gyeongsik Moon et al.[17] converted the input depth image into a three-dimensionalvoxel map, and established a three-dimensional heatmap inthe voxel map to convert the two-dimensional problem into athree-dimensional problem, and achieved good results. But thismodel relies on artificially defined pre-processing to completethe transformation from depth map to 3D voxel map. Wan, Cet al. [18] proposed a method based on an offset vector field,which allows the neural network to learn the bias vector field,and then uses the mean-shift algorithm as decoder to convertthe obtained vector field into 3D coordinates. Our design ofSFRs is inspired by this work, but instead of bias vector fieldwe use local offset map. We will compare with these modelsin the experiment part.

III. METHODOLOGY

To make the problem formally and prevent confusion in therest of the paper, the input of our model is a depth imageID ∈ Rn×n, which only contains one single complete hand.That is our basic assumption. Some filter method and resizeneed to apply to the raw data to make such an image. We willdiscuss this in Section IV. And the output of our model is thenormalized UVD coordinates in the image plane of each jointP

j∈{1...J}j ∈ R3. Specifically, we separate the coordinates into

two parts, Pj(uv) and Pj(d). Pj(uv) ∈ R2 denote the planecoordinates in camera plane space. And Pj(d) ∈ R denotedepth coordinates which is the distance between the cameraand joint j. Notice that, we do not use XYZ coordinates asoutput like some model did, because the UVD coordinatesare more direct information from a depth image without atransformation influenced by the internal parameters of thecamera. In this section, we will first propose the Pixel-wise method, clarify the problem and why our method canhelp. Then we will design a particular model use Pixel-wiseRegression. Specifically, we design the SFR and its correlativeDD which consists of two module CR and DR. Finally, weshow other techniques we use when building our model andsome implement details about the network.

A. Pixel-wise Regression method

The 3D hand pose estimation problem from single depthimage can be formalized as a function mapping from depthimage to 3D joint coordinates, i.e. φ : ID → P . Theregression-based method just wants approximate this functionwith a single neural network. On the other hand, as we discussabove, the key motivation of detection-based method is thatwe can make a detour. That is instead of directly mappingto the joint coordinates we can first mapping to an encodedrepresentation of it, i.e. ϕ : ID → L, then use a decoderto recover the joint coordinates, i.e. f : L → P . In which Ldenotes the encoded representation. To use the detection-basedmethod, one must first define the encoder g : P → L. Ideally,the encoder and the decoder are reciprocal, i.e. P = f(g(P )).

The intuition here is quite simple, since the SFR and thedirect supervision from 3D coordinates both benefit the resultof the neural network model, we can just put the decoder insidethe network and make use of both, just like what we shown inthe Fig. 1. We call this type of method Pixel-wise Regression.

The essential requirement for this method is that the decoderg must be differentiable. However, the decoder of the com-monly used heatmap is argmax, which is a nondifferentiableoperation and also not reciprocal of the complex encoder.The truth is, due to the complex nature of the encoder,which maps low-dimensional data to high dimension, therepresentation must contain certain redundancy which make ithard to perfectly recover the original data with a differentiabledecoder. To use the Pixel-wise Regression method, we needto build a particular model. That is we need to design an SFRwith a DD which is nontrivial. The overview of our model isshown in Fig. 2.

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 4

Fig. 2. Overview of our designed model. The input depth image goes through a low-level CNN to extract some low-level feature and down sampling to therepresentation scale. The hourglass module is used to extract the overall features of the input image. The Plane Regression module goes first after the featureextraction, predicting the plane coordinates of each joint and also passing the predicted heat map to Depth Regression module. On the other hand, the DepthRegression module, getting the heat map from Plane Regression module, predicting the depth coordinates of each joint and also output the predicted depthmap. We concatenate the heatmap, depth map and representation scale image as the input of the refine stage whose structure is the same as the initial stage.

Fig. 3. Detail structure of Plane Regression. Input features go through aCNN network and predict the heatmap for each joint. Then the heatmap goesthrough a COM(center of mass) based convolution to convert the heatmap to2D coordinates.

B. Design of SFR and DD

As we discuss in Section I, since we are dealing with 3Dcoordinates in 2D plane, we need more than one form ofSFR to encode the 3D coordinates. In our design, we use twoSFRs, heat map and local offset depth map to encode planecoordinates and depth coordinates receptively. We assembleeach SFR and its DD to module, thus we have two modulesnamed Plane Regression and Depth Regression. The followingof this part is to describe these two modules in details.

1) Plane Regression: We start from the easy part; the SFRof plane coordinates. The detailed structure of PR is shownin Fig. 3. Generally, PR take a feature map F ∈ Rm×m×f

as input and predict two things, a heatmap H̃j ∈ Rm×m andP̃j(uv) for each joint.

What we need here is an SFR for plane coordinates with aDD. And because heatmap matches our human intuition andalready achieve good result in 2D problem, we want to modifyit to meet our requirement. The problem is that the normalheatmap use argmax as decoder which is nondifferentiable. Asimple idea is to use the center of mass (COM) of the heatmapto represent the desire plane coordinates, i.e.

Pj(u) =∑i,k

Hj(i, k) · u(i, k) (1)

Pj(v) =∑i,k

Hj(i, k) · v(i, k) (2)

where the u(i, j) and v(i, j) denote the normalized u, vcoordinates for the pixel (i, j) .

This COM-based decoder has two clear advantages. First,it meets our requirement of being differentiable. Moreover, itis easy to make this decoder in a convolutional form, which

is the most common building block in deep learning thus canaccelerate computation. In the convolutional form, the aboveequation can be rewritten as:

Pj(uv) = Hj ∗ C (3)

where C ∈ Rm×m×2 is the COM convolutional kernel,defined as:

C(i, j, k) =

{u(i, j) k = 1v(i, j) k = 2

(4)

To complete this module, we need to define an encoder tobuild our desired heatmap. Although the heatmap we want isunlike the commonly used heatmap, we still want to retain itsunique intuitive nature, that is, the closer the pixel to the givenjoint is, the larger the value it will be. We divided the encoderinto two steps. In the first step, we only take the four nearbypixels into account. Notice that it is still an indeterminatesystem because the COM in plane space can only providetwo condition and we have four variables to solve. But sinceit is still a probability map, we can form some constraint asHj(i, k) ≥ 0 and

∑i,kHj(i, k) = 1. And we choose the

middle value of the efficient solution set as the four cornersvalue for our basic heatmap. Then we apply a gauss kernelwith size k to the basic heatmap we get from the first step toform our desired heatmap. It is obvious that the convolutionhere does not change the COM for the given heatmap. Thekernel size here measures how much redundancy we wantto put into the heatmap. And in practice, the valid region,where the heatmap have a non-zero value, in the heatmapalso measures the uncertainty that the network feels about theprediction of a certain joint, we will discuss more about thisin Section IV.

We use the mean-square-error between predicted value andground truth as loss of this module which contains twopart Luv and LH which denote the coordinates loss andrepresentation loss respectively:

Luv =∑j

∥∥∥Pj(uv) − P̃j(uv)

∥∥∥22

(5)

LH =∑i,j,k

[Hj(i, k)− H̃j(i, k)

]2(6)

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 5

Fig. 4. Detail structure of Depth Regression. Input features go through aCNN network and predict the depth map for each joint. Then combining withlabel scale image, we recover the depth information in local joint. The maskfilter values that do not on the hand. Finally, a weighted sum is conducted toconvert the heatmap and depth map to the depth coordinates.

2) Depth Regression: Then we deal with the depth coordi-nates. The detailed structure of DR is shown in Fig. 4. Gen-erally, DR take a feature map F ∈ Rm×m×f , a representationscale image I ∈ Rm×m , a mask matrix M ∈ Rm×m and thepredicted heatmap H̃

j={1...J}j from PR as input and predict

two things, a local offset depth map D̃j ∈ Rm×m and P̃j(d)

for each joint.As we discuss in the PR part, we need an SFR of depth

coordinates with a DD. We have shown in Section I that thereal depth coordinates of the joints have varied offsets for thevalues on the depth image. Therefore, the idea here is that wecan use the neural network to predict these offsets in a spatial-form. Intuitively, like the heatmap, only the pixels that areclose to the joints can give the information about this offset.So, we build a local offset depth map (depth map for short)to encode the depth information into plane space. The depthmap is defined as:

Dj(i, k) =

{ (Pj(d) − I(i, k)

)·M(i, k) if Hj(i, k) > 0

0 otherwise(7)

We use the heatmap we build in the PR to define thelocal region of the given joint to make sure that the heatmapand the depth map have same level of redundancy. And therepresentation scale image I is just a resized image from theinput depth image LD. The reason we need this image is thatsince we have some down sampling process in the network, thesize of the depth map is different from the input depth image.Since we use the depth offset to encode the depth coordinates,it is meaningless if there are some values not on the hand. Theoffset value for the background is just the exact value for thedepth coordinates of the joints, which makes the problem noeasier than the direct regression. Hence, we use mask matrixM which denotes which pixel is on the hand to filter the depthmap. The mask matrix M is defined as:

M(i, k) =

{1 if I(i, k) > 00 otherwise (8)

After we design the SFR of depth coordinates, we also needa DD to fuse the redundant depth map to a scalar. Ideally, wecan fully recover the depth coordinates by adding any twovalue of D and I in the local region of the given joint. So,it seems that the DD can be simply defined as averaging therecover values. However, since the depth map D̃j is predictedby the neural network, the result will not be perfect. We wantthe depth map has some same property as heatmaps, whichhas more accuracy when closer to the given joint. Therefore,our decoder uses the heatmap to weight the recover valuesgiven by the neural network:

P̃j(d) =

∑i,kM(i, k)H̃j(i, k)

[I(i, k) + D̃j(i, k)

]∑

i,kM(i, k)H̃j(i, k)(9)

Due to the background problem we state above, we also usethe mask matrix M here to ignore the pixel off the hand.

Like the PR, we use the mean-square-error as well. The lossof this module also contains two part Ld and LD which denotethe coordinates loss and representation loss respectively:

Ld =∑j

(Pj(d) − P̃j(d)

)2(10)

LD =∑i,j,k

[Dj(i, k)− D̃j(i, k)

]2(11)

C. Network architecture

As shown in previous works [4], [13], [18], [19], the multi-stage network can refine the result and deal with occlusions byinferring from results from the previous stage. In our model,making a tradeoff between speed and accuracy, we use atypically two-stage network, which uses the same structure.The loss function of the multi-stage network can be definedas the sum loss of each stage:

L =∑s

L(s) (12)

In our model, the loss function for a single stage is the sumof loss of PR and DR:

L(s) = L(s)iw + L

(s)d + λHL

(s)H + λDL

(s)D (13)

The λH and λD are two scale factors to make sure thelosses are on a similar scale. In our model, we empirically setλH = λD = 1.

We also use the hourglass module [20] as our main featureextractor. The hourglass module, due to its unique recursivestructure, can provide features extracted in multi-scales andwidely used in pose estimation problems.

We have two sizes of the image in our model. Only theinput depth image is in a bigger size of Rm×m, and all theother inputs and representations are in the size of Rn×n. Weuse a bigger image to contain more details in the low-levelfeatures as we have a low-level CNN block before the initialstage which extracts low-level features and down sampling itto Rn×n.

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 6

D. Implement Detial

We use TensorFlow framework to build and train ournetwork using Adam optimizer with a learning rate of 2e− 4.Since our model uses massive channel-wise operation, we useinstance normalization [21] to accelerate training. We use asimilar data normalization method to [11], which extracts afixed-size cube centered on the center of mass of this objectfrom the depth image. We set the input image size m to 128,and representation size n to 64. We also implement a [-30,30] random rotation as data augmentation. Since our model isan FCN which is translation invariant, we do not perform anyrandom translation for data augmentation. Empirically, we setthe kernel size k to 7 to get certain level of redundancy. Thebatch size is set to 32, and we train our model with 10 epochs.In the test time, our model can achieve about 100 FPS on asingle TITAN XP GPU.

IV. EXPERIMENT

In this section, we conduct experiments on three challengepublic datasets, i.e. MSRA, ICVL and HAND17. We choosethe MSRA dataset to conduct the ablation experiment becausethe dataset contains only full hand image that fit our assump-tion for the input image best. We used two metrics in allexperiments. The first is the mean 3D distance error for alljoint points. Another metric, which is more challenging, is thatthe percentage of frames in which all joint errors are below acertain threshold.

A. Datasets

1) MSRA dataset: The MSRA hand pose dataset [22] con-tains 76500 frames from 9 different subjects captured by IntelsCreative Interactive Camera. The dataset provides annotationsfor 21 joint points, four joints per finger, and one joint on thepalm. Since the dataset itself does not divide the training setand the test set, we randomly divide the training set and thetest set by a ratio of 8:1. Since the data itself is a segmentedhand image, we do not conduct any new segmentation.

2) ICVL dataset: The ICVL dataset [23] contains 330kframes from 10 different subjects also using Intels CreativeInteractive Gesture Camera. This dataset contains annotationsfor 16 joint points, three for each finger and one for thepalm. The dataset contains random rotations, and since weused online data augmentation, we only used 22k of originalimages. Since the image contains the background, we first usea boundary box built by the ground truth of joints coordinatesto roughly remove the background. Then we use an empiricalthreshold to remove the rest.

3) HAND17 dataset: The HAND17 dataset [3] is thelargest scale dataset in 3D hand pose estimation field whichcontains 957k frames for training and 295k frames for testing.The dataset captures the depth images by the latest IntelRealSense SR300 camera and automatic annotates 21 jointsusing six 6D magnetic sensors and inverse kinematics. Sincethe images provided by the dataset has background, we usethe same procedure we describe in ICVL dataset to removethem. The test set provides the boundary box for us, thus weonly use the empirical threshold.

Fig. 5. Setup of the ablation experiment.

B. Ablation Experiment

To demonstrate the validity of our method, we design andperform an ablation experiment. As shown in Fig. 5 we modifysome structures in our model to form four different baselinemodels. To set up a fare comparison, we retain the featureextraction part of the model and take the extracted features asthe input of the ablation experiment. Generally, we conductthis ablation experiment for three purposes.

First, we want to prove that the Pixel-wise Regressionmethod we proposed is better than the former two methods. Inorder to compare with the regression-based method, we replacePR and DR with three fully connected layers and directlyregress 3D coordinates from features. We call this modelDirect Rregression. On the other hand, in order to comparewith the detection-based method, we remove the supervisionof the joint coordinates and put the decoder outside the neuralnetwork. We call this model Coordinate Unsupervised.

Second, we want to explore how different training strategiesinfluence the learned SFRs and results. As we discuss inSection III, although we design reasonable SFRs, we controlthe redundancy in such representation by an empirical kernelsize k. In the original design we use, we supervised thecoordinates along with the representation, which can guide therepresentation predicted by the network converge to the rep-resentation we design. However, even if we do not supervisethe representation, the model can still learn a representationof 3D coordinates due to the gradient backpropagated bythe decoder. The decoder here serves not only a regressionmodule but also a constraint that limits the search spaceof the representation. Because there are infinite such SFRs,the learning outcomes of the network may be different fromsupervised learning model. To check out the outcome of such amodel and get deeper understanding of our model, we removethe supervision for the representation and call this networkRepresentation Unsupervised.

Finally, we want to verify that the multi-stage network isable to improve the accuracy of the predictions. So, in thelast comparison experiment we only keep the first stage, sowe call this model One Stage. We compare these four modelswith our designed two-stage model which we call Two Stagehere on the MSRA dataset.

The results of the ablation experiment are shown in Fig. 6.We can see that as we estimated, Direct Regression, which

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 7

Fig. 6. The result of the ablation experiment using the maximum alloweddistance metric.

Fig. 7. Learned equivalent representation. The top is the result of supervisedmodel, and the bottom is the result of unsupervised method. For each part,the first row is learned depth map, and the second row is learned heatmap.From left to right is MCP, PIP, DIP, TIP joint of the index finger receptively.Best view in color.

does not preserve spatial information, yields the worst results.Coordinate Unsupervised, a detection-based method, yieldsa better result than the regression-based method. However,since there is no direct supervision from coordinates, the twomodules are learned separately and cannot perfectly cooperate,so the result is not good either. Moreover, compared with theOne Stage model, it can be found that the Two Stage methoddoes have a significant improvement in the accuracy.

For the Representation Unsupervised method, we can findthat its performance is better than other methods, and yieldcomparable result with our original design. Since we wantto study the learned representation effect by supervision, wedraw the learned representation upon the input depth image,as shown in Fig. 7. The result is quite interesting. We can seethat the supervised representation tends to have a low-levelredundancy as we design and have same level of redundancyfor different joints. The valid region of depth map is slightlylarger than the heatmap, but they converge to the same joint.That is because the recovered depth values given by depthmap are weighted by the heatmap which cannot be preciselypredicted. Therefore, the depth map must make the predictionin a larger local region to compensate the uncertainty of theheatmap. This is also what make our Pixel-wise Regressionmethod better than the detection-based method.

On the other hand, the unsupervised model has more

TABLE IMEAN 3D ERROR ON MSRA DATASET

Model 3D error (mm)

REN-9x6x6 [24] 9.792Pose-REN [13] 8.649DenseReg [18] 7.2343DCNN [26] 9.584SHPR-Net [7] 7.756

HandPointNet [27] 8.505Point-to-Point [28] 7.707

Ours 5.186

redundancy for the larger joints, i.e. the MCP joint. That isbecause the larger the joint the more similar the local regionlooks like. For instance, the MCP joint is closer to the palmand other root joint of finger, so it is always surround by thedense and similar information provided by the depth image.Thus, without the supervision, these similarities can causeconfusion and result in uncertainty. On the other hand, the TIPjoint, which is usually locates at the boundary of the hand hasa clearer context and has lower uncertainty. The depth mappredicted by the unsupervised model is quite strange with acircular form and do not focus on a particular region. That isbecause the valid region of depth map is defined by heatmapand mask matrix, when we do not supervise the depth map,there are zero gradients for those invalid pixels which gives thenetwork more freedoms to choose the value for them. Thesefreedoms loosen the local constrain when we design the depthmap and may cause the model to overfit to particular patterns.Actually, the circular form of the depth map is an evidencethat the network is overfit to the random rotation we performduring data processing.

C. Comparing with State-of-the-Art

In this part, we compare our model with the state-of-the-artmodels. We have chosen models that are representative andhave the state-of-the-art accuracy. In addition to the modelswe mentioned in Section II, we also include Region Ensemblenetwork (include REN-9x6x6 [24] and REN-4x6x6 [25]), 3DConvolutional Neural Networks (3DCNN [26]), SHPR-Net[7], HandPointNet [27], Point-to-Point [28]. We use toolsprovide by Chen, X [13] to evaluate the result.

As shown in Fig. 8 and TABLE I. in MSRA dataset, ourmodel greatly outperforms current state-of-the-art models inall joints. For the mean error of all joints, we only have 75%of the best results present [18], which is a big improvement. Inaddition, as can be seen in Fig. 8. our model greatly exceedsother models with a small allowable threshold, which meansthat we propose that the model is very effective in high-precision recognition.

The results of ICVL dataset are shown in Fig. 9 and TABLEII. As can be seen from the results, we have achieved similarresults with the best model [17] on the average 3D error. Ascan be seen from Figure 9, when the error threshold is less than10mm, we exceed all other models, but when the threshold isgreater than 30mm, our model is not as effective as other

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 8

Fig. 8. The results of MSRA dataset. Left is mean error (mm) for each joint. Right is the proportion of frame that all joints error are under the giventhreshold.

Fig. 9. The results of ICVL dataset. Left is mean error (mm) for each joint. Right is the proportion of frame that all joints error are under the given threshold.

TABLE IIMEAN 3D ERROR ON ICVL DATASET

Model 3D error (mm)

DeepModel [12] 11.561REN-4x6x6 [25] 7.628REN-9x6x6 [24] 7.305Pose-REN [13] 6.791DenseReg [18] 7.239SHPR-Net [7] 7.219

V2V-PoseNet [17] 6.284

Ours 6.177

models. This is because we manually segmented the imageand the background of the image was not completely removed,which affects the performance of our model.

The result of HAND17 dataset is shown in TABLE III.Although it is two to three millimeters worse than the best

TABLE IIIMEAN 3D ERROR ON HAND17 DATASET

Team Name Average (mm) Seen (mm) Unseen (mm)

BUPT 8.39 6.06 10.33SNU CVLAB 9.95 6.97 12.43

NTU 11.30 8.86 13.33THU VCLab 11.70 9.15 13.83NAIST RV 12.74 9.73 15.24

HuPBA 14.74 11.87 17.14NAIST RVLab G2 16.61 13.53 19.18

Baseline 19.71 14.58 23.98

Ours 12.22 8.73 15.13

model, we still achieve comparable result with other competi-tors. Specially, we find that our model gets better performanceon seen joints than the unseen ones. We attribute this to thecause that our design of SFR is not efficient for the self-

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 9

occlusion joints which have larger variation range than theseen ones.

We also show some qualitative results in these two datasetsin Fig. 10. Some of these results are even better than theoriginal annotations, which demonstrate that our model learnthe essential information of the data and not just remember allthe patterns.

V. CONCLUSION

In this paper, we propose a Pixel-wise Regression methodfor 3D hand pose estimation, which use spatial-form rep-resentation and differentiable decoder to solve the losingspatial information and lacking direct supervision problemsfaced by existing methods. We design a particular modelthat use our proposed method. Specifically, we design thespatial-form representation and its correlative differentiabledecoder which consists of two modules Plane regression andDepth Regression that deal with plane coordinates and depthcoordinates respectively. The ablation experiment shows thatour proposed method is better than the former methods. Andit also shows that the supervision on the representation isvital to the performance for our model. Experiments on publicdatasets show that our model reach the state-of-the-art levelperformance.

In the future, we intend to explore more design of repre-sentation used for Pixel-wise Regression method or directlyuse our model in the field of human-computer interaction tocontrol real robotics hand.

REFERENCES

[1] J. Vogel, S. Haddadin, B. Jarosiewicz, J. Simeral, D. Bacher,L. Hochberg, J. Donoghue, and P. van der Smagt, “An assistive decision-and-control architecture for force-sensitive hand-arm systems drivenby human-machine interfaces,” The International Journal of RoboticsResearch, vol. 34, no. 6, pp. 763–780, 2015.

[2] J. S. S. Iii, G. Rogez, Y. Yang, J. Shotton, and D. Ramanan, “Depth-based hand pose estimation: Methods, data, and challenges,” Interna-tional Journal of Computer Vision, vol. 126, no. 11, pp. 1180–1198,2018.

[3] S. Yuan, G. Garcia-Hernando, B. Stenger, G. Moon, J. Y. Chang, K. M.Lee, P. Molchanov, J. Kautz, S. Honari, L. Ge, J. Yuan, X. Chen,G. Wang, F. Yang, K. Akiyama, Y. Wu, Q. Wan, M. Madadi, S. Escalera,S. Li, D. Lee, I. Oikonomidis, A. A. Argyros, and T.-K. Kim, “Depth-based 3d hand pose estimation: From current achievements to futuregoals,” in 2018 IEEE/CVF Conference on Computer Vision and PatternRecognition, 2018, pp. 2636–2645.

[4] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutionalpose machines,” in 2016 IEEE Conference on Computer Vision andPattern Recognition (CVPR), 2016, pp. 4724–4732.

[5] D. Tang, H. J. Chang, A. Tejani, and T.-K. Kim, “Latent regressionforest: Structured estimation of 3d hand poses,” IEEE Transactions onPattern Analysis and Machine Intelligence, vol. 39, no. 7, pp. 1374–1387, 2017.

[6] L. Ge, H. Liang, J. Yuan, and D. Thalmann, “Real-time 3d hand poseestimation with 3d convolutional neural networks,” IEEE Transactionson Pattern Analysis and Machine Intelligence, vol. 41, no. 4, pp. 956–970, 2019.

[7] X. Chen, G. Wang, C. Zhang, T.-K. Kim, and X. Ji, “Shpr-net: Deepsemantic hand pose regression from point clouds,” IEEE Access, vol. 6,pp. 43 425–43 439, 2018.

[8] Z. Hu, Y. Hu, J. Liu, B. Wu, D. Han, and T. Kurfess, “A crnn module forhand pose estimation,” Neurocomputing, vol. 333, pp. 157–168, 2019.

[9] Y. Wu, W. Ji, X. Li, G. Wang, J. Yin, and F. Wu, “Context-aware deepspatio-temporal network for hand pose estimation from depth images.”IEEE Transactions on Systems, Man, and Cybernetics, pp. 1–11, 2018.

[10] T.-Y. Chen, P.-W. Ting, M.-Y. Wu, and L.-C. Fu, “Learning a deepnetwork with spherical part model for 3d hand pose estimation,” PatternRecognition, vol. 80, pp. 1–20, 2018.

[11] M. Oberweger and V. Lepetit, “Deepprior++: Improving fast and accu-rate 3d hand pose estimation,” in 2017 IEEE International Conferenceon Computer Vision Workshops (ICCVW), 2017, pp. 585–594.

[12] X. Zhou, Q. Wan, W. Zhang, X. Xue, and Y. Wei, “Model-baseddeep hand pose estimation,” international joint conference on artificialintelligence, pp. 2421–2427, 2016.

[13] X. Chen, G. Wang, H. Guo, and C. Zhang, “Pose guided structuredregion ensemble network for cascaded hand pose estimation.” arXivpreprint arXiv:1708.03416, 2017.

[14] X. Wu, D. J. Finnegan, E. O’Neill, and Y.-L. Yang, “Handmap: Robusthand pose estimation via intermediate dense guidance map supervi-sion,” in Proceedings of the European Conference on Computer Vision(ECCV), 2018, pp. 246–262.

[15] L. Ge, H. Liang, J. Yuan, and D. Thalmann, “Robust 3d hand poseestimation from single depth images using multi-view cnns,” IEEETransactions on Image Processing, vol. 27, no. 9, pp. 4422–4436, 2018.

[16] J. Tompson, M. Stein, Y. LeCun, and K. Perlin, “Real-time continuouspose recovery of human hands using convolutional networks,” ACMTransactions on Graphics, vol. 33, no. 5, p. 169, 2014.

[17] J. Y. Chang, G. Moon, and K. M. Lee, “V2v-posenet: Voxel-to-voxelprediction network for accurate 3d hand and human pose estimationfrom a single depth map,” in 2018 IEEE/CVF Conference on ComputerVision and Pattern Recognition, 2018, pp. 5079–5088.

[18] C. Wan, T. Probst, L. V. Gool, and A. Yao, “Dense 3d regression forhand pose estimation,” in 2018 IEEE/CVF Conference on ComputerVision and Pattern Recognition, 2018, pp. 5147–5156.

[19] A. Bulat and G. Tzimiropoulos, “Human pose estimation via convo-lutional part heatmap regression,” european conference on computervision, pp. 717–732, 2016.

[20] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks forhuman pose estimation,” european conference on computer vision, pp.483–499, 2016.

[21] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky, “Instance normal-ization: The missing ingredient for fast stylization.” arXiv preprintarXiv:1607.08022, 2016.

[22] X. Sun, Y. Wei, S. Liang, X. Tang, and J. Sun, “Cascaded hand poseregression,” in 2015 IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2015, pp. 824–832.

[23] D. Tang, H. J. Chang, A. Tejani, and T.-K. Kim, “Latent regressionforest: Structured estimation of 3d articulated hand posture,” in CVPR’14 Proceedings of the 2014 IEEE Conference on Computer Vision andPattern Recognition, 2014, pp. 3786–3793.

[24] G. Wang, X. Chen, H. Guo, and C. Zhang, “Region ensemble network:Towards good practices for deep 3d hand pose estimation,” Journal ofVisual Communication and Image Representation, vol. 55, pp. 404–414,2018.

[25] H. Guo, G. Wang, X. Chen, C. Zhang, F. Qiao, and H. Yang, “Regionensemble network: Improving convolutional network for hand pose es-timation,” in 2017 IEEE International Conference on Image Processing(ICIP), 2017, pp. 4512–4516.

[26] L. Ge, H. Liang, J. Yuan, and D. Thalmann, “3d convolutional neuralnetworks for efficient and robust hand pose estimation from single depthimages,” in 2017 IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2017, pp. 5679–5688.

[27] L. Ge, Y. Cai, J. Weng, and J. Yuan, “Hand pointnet: 3d hand pose es-timation using point sets,” in 2018 IEEE/CVF Conference on ComputerVision and Pattern Recognition, 2018, pp. 8417–8426.

[28] L. Ge, Z. Ren, and J. Yuan, “Point-to-point regression pointnet for 3dhand pose estimation,” in Proceedings of the European Conference onComputer Vision (ECCV), 2018, pp. 489–505.

JOURNAL OF IEEE TRANS ON IMAGE PROCESSING, VOL. XX, NO. XX, XXXXX 2019 10

Fig. 10. Qualitative results of our model. (a), (b), (c) shows the result form MSRA, ICVL, HAND17 dataset respectively. Specially, in (a) and (b) we provideground truth in the second row for comparison.