I have strongly considered getting rid of my side mirrors too. However, I was thinking about having a wide angle rear view mirror. I believe I've seen them in race cars before.
Regardless, that sounds like a cool idea, and I hope it works for you. I like the look without side mirrors. I'm planning to remove my side mirrors, but keep and modify the door handles (slim fit).