Rare earth elements (REEs) are critical for many advanced technologies and are faced with potential supply disruptions. Recycling of permanent magnets (PMs) can be good sources for REEs which can help minimize global dependence on freshly mined REEs, but PMs are rarely recycled. Recycling of PMs has been discussed with respect to improving REEs resource sustainability. Some challenges to be addressed in order to establish industrially deployable technologies for PMs recycling have also been discussed, including profitability, energy efficiency and environmental impacts. Key considerations for promoting circular economy via PMs recycling is proposed with the focus on deciding the target points in the supply chain at which the recycled products will be inserted. Important technical considerations for recycling different forms of waste PMs, including swarfs, slags, shredded and intact hard disk drives magnets, have been presented. The aspects of circular economy considered include reusing magnets, remanufacturing magnets and recovering of REEs from waste PMs.