So I went ahead and enabled http2 (by setting http20enabled to true in the config) for one of my sites deployment slots. This after a couple minutes seemed to do the trick and I could see in chrome it working.
But then looking at other deployment slots and the production slot. And even a totally different site that happens to sit under the same domain all also ended up with it enabled. While they still had each of their own http20enabled set to false.
This seems very odd behaviour and a bit worrying as you cannot test the site before pushing to production. Do we know why this is?