Human action recognition has always been a hot topic in computer vision research and widely applied in virtual reality, short video, etc.Meanwhile, the fast development of deep learning in recent years has also inspired the action recognition algorithms.Compared with traditional methods, the action recognition algorithms based on deep learning have advantages of strong robustness and high accuracy.Here, we make a survey on the action recognition algorithms based on deep learning proposed in recent years, and focus on those developed from two-stream network and 3D convolutional network, then summarize their performances and positive results, and finally make prospects in this field.