Peter Mandelson opines that the leader of the opposition could do with losing a few pounds. Odd one, this, for at least two reasons. First, because I doubt anyone else has given a second thought to Keir Starmer’s body shape; it’s just plain normal. No, it’s better: it’s on the more attractive side of normal, if he doesn’t mind me saying so. Never mind fat‑shaming, I’ll objectify him instead. He’s a fine-lookin’ fella and no mistake.

Second, Mandelson’s appraisal of the prospective prime minister came as part of a discourse on how leaders’ appearances matter to voters. On the one hand, this is a fair point. It may also be tosh, given the extent to which Boris Johnson’s electoral appeal survived his considerable bulk and general dishevelment.

Have I just fat-shamed him? If so, I apologise. I doubt Johnson feels shame about his size any more than Starmer has considered Mandelson’s remarks – or the way that Jeremy Hunt used his budget speech to invite him to join him in his marathon training. But you never know. It’s not necessarily much to do with how overweight you are. In a society that sees fat as bad, you don’t have to be much above the average weight of your schoolmates for one of them to sling an insult at you. And these things stick. I can’t find any evidence of Johnson or Starmer saying this applies in their case, but, again, who knows? After all, if you are hung up about something, you are probably disinclined to mention it.

I’m probably somewhere between Starmer and Johnson on the body fat spectrum. If you want to fat-shame me, you will find it very easy; just call me overweight and I will be very hurt indeed. I’ve been heavier (tellingly, I can barely bring myself to write the word “fatter”) and I’ve been a bit lighter. But, honestly, I don’t think a day goes by when I don’t think about it. I look at a photo of a school-age Starmer and try to divine if he might have had any weight worries. I see nothing to suggest as much, but neither do I see anything when I look at photos of myself at that age. And yet specific references to my weight back then are seared – and I mean seared – on my memory. They still play out in HD in my mind.

A couple of examples. When I was about 12 and out of breath during school football practice, the teacher grabbed a pinch of me and said exactly this: “You’re carrying passengers, son. Too many potatoes.” I was appalled. One Christmas Day, my Auntie Marj told me to stand up so she could have a good look at me, because her sister had told her I was getting fat. “But you’re not too bad,” she said. This didn’t help.

Call me anything, but don’t call me fat. Time after time, I’ve been told I have a “face for radio” and not cared a jot. Well, a bit, maybe. My Midlands accent has been remarked upon a million times and it really hasn’t bothered me. And, like anyone who has worked in football, I’ve had dogs’ abuse – some awful stuff. Covering a match at Stoke City once, I popped out of the studio to use the toilet. One of the home fans yelled exactly this: “Fuck off, Chiles, and lose some fucking weight, you useless fat cunt.” The only words that touched me were “weight” and “fat”. I was mortified, even tearful.

I’m obviously not putting Mandelson in the same bracket as that potty-mouthed Stokie. I’m not even that annoyed with the Stokie – after all, he can’t have known about my PE teacher or my Auntie Marj. But, as a general rule, it’s well worth steering clear of the body stuff – body shape, body colour, body whatever. There’s too much potential for nasty things to be lurking beneath upturned stones.

• Adrian Chiles is a broadcaster, writer and Guardian columnist

• Do you have an opinion on the issues raised in this article? If you would like to submit a response of up to 300 words by email to be considered for publication in our letters section, please click here.