We consider axion cloud decay due to the axion-photon conversion with multi-pole background magnetic fields. We focus on the ℓ=m=1 and n=2 mode for the axion field configuration since it has the largest growth rate associated with superradiant instability. Under the existence of a background multi-pole magnetic field, the axion field can be converted into the electromagnetic field through the axion-photon coupling. Then the decay rate due to the dissipation of the converted photons is calculated in a successive approximation. We found that the decay rate is significantly dependent on the azimuthal quantum number characterizing the background magnetic field, and can be comparable to or larger than the growth rate of the superradiant instability.