So I cut all the serging off, unpicked any extra seaming, cut the hems off, cut the shoulder seams off, recut the back, and resewed and rehemmed the whole thing inside out. I left the panel seams as they were and just topstitched them to keep them in place.
I am pleased with the result. The fit is much better and the contrast level in the garment is also much better. I can imagine wearing this one. The inside is still VERY striped!
As an aside I thought ages ago about making this up with the darker softer side as the outside, but somehow allowed that gut feeling to be overcome by doing it the 'proper way'. Maybe I should've listened!
(Also wondering if I need to add a little something, button or tab or
flower or .... at that point on the neckline where its pulled slightly
by the panels meeting. Thoughts?)