Landlord gave my neighbors a key to my basement?

Location: KY
Hello, I’m not sure exactly how this works and I am in need of advice. I live in a duplex and it has been brought to my attention that my neighbors have a key to my back door. A few months ago we had a storm and I went down to my basement due to tornado warnings and shortly after I settled in my neighbors unlocked the back door and 12 people came down into the basement (my neighbors and extended family). The basement is attached to only our side of the house and the only access to it is through our side of the house. We were never made aware that anyone else had a key. While it was alarming that so many people came in, I tried to be empathetic because they had kids and wanted to be safe. Since then I have noticed that our basement is being used by them frequently. Once a week I notice that our backdoor top lock isn’t locked and it obvious someone has come in and gone into the basement. Our backdoor key also is the same key that unlocks our front door. So our neighbors have access to our house completely, though I don’t think they have entered our direct living area. Is this legal? I would feel that it’s a complete invasion of privacy with our neighbors being able to access our part of the house and be in our basement at any given time without warning. I don’t know if there is anything I’m able to do by law. Please help.