What should I do in this nightmare,horrible situation?

Today I just found out today that my so called partner that was well we was suppose to be getting back on the straight and narrow but basically he went to where I used to live and knocked on my next door neighbour and asked the male their how was I what did I used to get up too the guy said he herd "certain noises" coming from my room and that he used to see a few guys come to the house and said I slept with all of them which was a lie. But the fact that this was discussed being my back feels horrible. I've lived away from there for nearly 5 years and this person decided to talk being my back no my so called partner is calling me a **** and and every word under the sun because of this information that he has on me. Im an adult I didn't even do things with all of these people but that can I do from this point onwards. Can I take legal action because its distressing me what do I do? I hoped I explained this properly.