Abstract: When traditional convolutional neural networks are used to classify the spectrum images of impact echo signals, they are faced with the problems of insufficient feature extraction ability, unbalanced data sets and small scale. This paper presents the Multi-scale Hybrid Attention and Spatial Channel Reconstruction Convolutional Neural Network (MHA-SCConvNet). The model initiates with a multi-scale hybrid attention module that extracts spectral features at diverse scales, significantly sharpening the focus on essential waveform information. This is followed by the integration of a Spatial and Channel Reconstruction Convolution (SCConv) module, designed to optimize the representation of image features and reduce redundancy effectively. Additionally, the paper introduces the Gradient and Distribution Harmonized Margin Loss (GDHM Loss), a dynamic hybrid loss function tailored to address the challenges of classifying both hard-to-classify and minority class samples. The MHA-SCConvNet was rigorously evaluated on a proprietary dataset, where it demonstrated a notable accuracy of 94.58%, outperforming established models such as AlexNet, VGGNet, and GoogLeNet. Experimental results validate the MHA-SCConvNet's superior capability in enhancing the accuracy and efficiency of impact echo spectral image classification.