How much mass can you take from the Moon to change the gravitational pull of the Earth/the Moon?

r/

Hypothetical question:

Let’s say you start a mining operation on the moon, where took material from the moon and brought it back to Earth, at what point would there be a gravitational catastrophe to happen? Like, if you took enough mass out of the moon and brought it to the Earth, would the moon start to fall to the Earth, or would the moon just have less a pull over the ocean? Would it be an instant reaction or would it be effect over time? This is a question I’ve been wondering for years and I figure I put it out there after asking a friend.