Why is that? Just because you can't see me, how does it follow that I can't see you?

The problem comes in the nature of seeing. Presumably when we say "invisible" we are not talking about someone who can hide so well that she's never spotted. If you're really going to be invisible, you'll have to be transparent. In other words, light will have to be able to pass right through your body without being deflected or absorbed. That way, the light coming from the tree behind you will pass through your body unchanged and reach my eye where it forms an image of the tree exactly as it would if you weren't there.

But think about what we just said. The way I see the tree is by having light from it hit my eye. That interaction of light with my retina sends signals to my brain, which combine to form a visual image. See the problem? If you're invisible, light never hits your retina, or anything else. It passes right through. Unfortunately, it's not half as much fun to be invisible if all you can do is stumble around.