Video coding techniques effectively address the data volume problem of raw video, however, the achieved compression efficiency comes at the cost of video quality degradation. To improve the visual quality of compressed video, a Detail Recovery Convolutional Neural Network (DRCNN)-based video quality enhancement method is proposed in this paper, which consists of a main denoising branch and a detail compensation branch. To effectively extract and eliminate the compression distortions, a main denoising branch is proposed, in which a Multi-Scale Distortion Feature Extraction Block (MDFEBs) is proposed to pay more attention to the distortion areas in the compressed video, and improve the distortion feature learning ability of the proposed DRCNN. Furthermore, to enrich the details in the compressed video, a detail compensation branch is proposed, which firstly adopts a content feature extractor composed of a pre-trained ResNet-50 to provide abundant content features, such as salient objects, shapes, details and so on, and then a Detail Response Block (DRB) is designed to efficiently extract the detailed features from the content features. Extensive experimental results show that the proposed DRCNN achieves the best compressed video quality enhancement performance as compared with the state-of-the-art methods.