In this paper, we propose a pruning and optimization algorithm (GWP) based on gradient weight Pursuit to solve the overfitting problem in the unsupervised domain, that is, the accuracy of downstream tasks is much lower than that of training sets. The dense-sparse-dense strategy is applied to solve the overfitting problem for the difference and adversarial adaptive methods in the unsupervised domain. The network is trained intensively and which connections are important are learned. The second is the pruning stage. Different from the pruning process in the original dense-sparse-dense strategy, the optimization algorithm in this paper considers both weight and gradient. On the one hand, weight information (i.e. zero-order information) is used, and on the other hand, the influence of gradient information (i.e. first-order information) on the network pruning process is also considered. In the final intensive phase, the pruned connections are restored and the intensive network is retrained with a smaller learning rate. Finally, the obtained network achieves ideal results in downstream tasks. The experimental results show that the proposed GWTA can effectively improve the accuracy of downstream tasks and has a plug-and-play effect compared with the original difference-based and adversarial domain adaptive methods.