My point is not that memory is accurate, but only that it represents a kind of continuity. When I think of who I am, a huge component of that construct is the current version of the story of my life as I remember it. Of course that story is a fiction, built in large part on my beliefs about the world, along with my erroneous perceptions of my family and my favorite spins on pivotal events in my history.
The fact that very little of it may be true in any objective sense does not change my attachment to it or the way it affects my choices going forward.
As you say, my memory is as changeable as my body. My body is a colony of trillions of cells, of which only about 43% are human, and nearly all of which are regularly replaced. Why should my consciousness be any different?
I am a song, a story, possibly "a tale told by an idiot, full of sound and fury, signifying nothing," and sure to be forgotten in a few years. I can live with that.