The problem is that the wheels will no longer be hubcentric, only lugcentric. This means that the lugs will bear the full weight and centering force for the wheel. You wouldn't be the first person to try it. Some people have done so and have experienced vibrations (and vibrations, left uncorrected, have a tendency to get worse and cause other problems like loosening parts, snapping studs, etc), while others haven't had problems doing so.
Personally, I think it's a bad idea, and I've learned not to do things that seem like a bad idea, especially when there's very little to be gained. But it's your car, and you're welcome to try it if you want.