I removed all the remaining rotten rubber/plastic crap and the brackets/bolts that hold it on. Then I made end gap fillers from coroplast. I still need to figure out what to do with the top gaps. I was going to do the plexiglas thing but I'm terrible at cutting it so I think I'm just going to use wood (maybe with rubber laminated on).
Looking at those end gaps I realized my rear bumper was acting as a parachute and covering the rear bumper's end gaps would have to be a good thing. It bolts on to the car and overlaps the bumper without attaching. I thought the coroplast wouldn't be stiff enough but it's stiffer in this application than I thought. Still, I used 3 dabs of silicone to glue it to the inside of the bumper.
I bought some paint at the parts store that looks similar. In the photo it looks dead on but in person you can tell it's different. It's good enough for this stuff.