Mode division multiplexed (MDM) in few mode fibers has gained attraction due to its potential to achieve ultra-high capacity beyond the nonlinear Shannon limit of the single mode fiber.Meanwhile,polarization division multiplexing (PDM) can make the system transmission capacity doubled without changing the fiber link.The combination of MDM and PDM can provide much more capacity up to several hundreds Gbit/s.However,it cannot be ignored that the effect of mode coupling and differential mode group delay on optical signals exists in transmission and mitigating these impairments is required in this multiplexing.This paper focuses on the frequency domain least mean square (LMS) digital equalization for MDM with PDM in few mode fiber system.A modified frequency domain LMS algorithm is proposed,which is operated with a butterfly finite impulse response (FIR) filter by means of modifying equations according to average power in each frequency bins.Simulation results show that the modified algorithm can improve the convergence speed and the system performance compared with traditional algorithm.