It sounds like you're making the mistake of trying to match the specified width of the tire to the specified width of the wheel. You need to remember that the section width of the tire - in this case, 255 mm - is measured at the widest point of the tire, typically in the middle of the sidewall where it bulges out. OTOH the width of the wheel is measured at the bead, the point where the inside edge of the tire mounts to the wheel. The bead almost always is (or should be) narrower than the section width. This is why tire manufacturers publish specs including the range of approved wheel widths for each model and size of each tire they sell.
When you look at Hankook's specs for the R-S3, you'll find that there's no reason to widen those wheels. The 255/40-17 R-S3 is approved for mounting on wheels 8.5-10.0" wide, so you can mount them on the 17x8.5 wheels as is without widening them. And you won't have to worry about rubbing on the suspension that way, either.
However, there's another problem: the R-S3 is not made in a size that works on the NSX front. If you want to get supersticky extreme performance tires similar to the R-S3 in sizes that work on your stock wheels, I recommend one of these combinations:
Toyo Proxes R1R 205/45-16 and 245/40-17
Kumho Ecsta XS 215/45-16 and either 245/40-17 or 255/40-17
Falken Azenis RT-615K 215/45-16 and 255/40-17
All of these should work fine on the stock wheels as is, without any need to widen them.
