Palmprint-based biometric recognition has received tremendous attention due to its several advantages such as high security, noninvasive and good hygiene propensities. Recent deep convolutional neural network (CNN) has been successfully applied for palmprint recognition and achieved promising performance due to its breakthroughs in image classification, which however usually requires a massive amount of labeled samples to finetune the network. In this paper, we propose a compact CNN with limited layers for palmprint recognition by embedding double attention mechanisms into the convolutional layers. Specifically, we first design a channel attention module to learn and select the discriminative channel maps by adaptively optimizing the attention weights among all channels. Then, we engineer a location attention module to learn the position-specific features of the palmprints. Both the channel and location attention modules are subsequently embedded into each convolutional layer, such that more discriminative features can be efficiently exploited during feature learning. Lastly, we train a fully convolutional network as the classifier for feature identification. Extensive experimental results on three widely used databases demonstrate the effectiveness of the proposed method in comparison with the state-ofthe-arts.