In view of the existing image inpainting algorithms, the texture may be blurred and the instability in the training process of the repaired image, a generative adversarial network image inpainting algorithm based on the diffusion process was proposed. The diffusion model is introduced into the dual discriminator generation adversarial network, and the image generated by the generator and the real image go through the forward diffusion process to obtain the inverted image with Gaussian noise and the real image, which is used as the input of the discriminator, which improves the repair quality and increases the stability of model training. Style loss and perceptual loss are introduced into the loss function to learn semantic feature differences, eliminate motion blur, and retain more details and edge information in the repair results. Qualitative and quantitative analysis and ablation experiments were done on the datasets CelebA and Places2, respectively, and according to the evaluation results and restoration effects show that the proposed algorithms have better performance. Compared with the current repair method, peak signal-to-noise ratio and structural similarity were improved by an average of 1.26db and 1.84%, respectively, and the L1 error was decreased by an average of 25.7%, and according to the change of the loss function it is seen that after the The training of image restoration algorithm with diffusion process is more stable.