Yep that's what its supposed to look like. I believe its soft and pliable when new and then as it degrades it gets brittle (along with the plastic moisture barrier) and loses its stickiness. One of my doors had a more recently been updated with a new regulator and that side was far softer than the driver side and could be reused. I tried using a silicone caulking from home depot this time around to reseal mine because it was readily available and seemed similar. About 10 years later its still pliable and seemed to peel off a lot cleaner than Honda's stuff.
A credit card is decent at scraping off the majority without scratching up your door, then you can follow that with more of a solvent to get the rest off. I don't remember what I ended up using but I think a goo gone type product would probably work. Getting it off the plastic is probably lot harder. I think the replacement parts for those were around $70 each.