have you tried tightening the rivnut? you can easily make your own rivnut tool (check youtube).
to remove. first check theres a big enough exit/ breather hole in your frames bottom bracket or headtube, to get the rivnut out. then using thick tape around the area of the rivnut. then very carefully cut a cross (2 cuts) into the face of the rivnut, obviously being careful not to cut into the downtube. if you have very steady hands you can use a dremel cutting disc. you dont have to cut all the way, just thin enough that with a flat blade screw driver you can fold the cut edges of the rivnut in on itself, until the rivnut falls into the downtube. which is why its important to check you can get the rivnut out before you start! its easier than it sounds, just take your time