There’s literally nothing. I guess they’re just trying to start drama. Back in 2021 she said that they had offered him to stay on as the duke when there was a huge backlash from fans (who likely simply didn’t understand his character played no further part in the story because they had not read the books) after it was announced that he won’t be reprising his role in subsequent seasons. He declined. That’s all there was to it.
It probably wouldn’t have been a fully fledged out role anyway because his character isn’t a part of the story after the first book. It would have just been fan service and bad writing trying to fit him in. He was right to turn it down. Even Shonda acknowledged as much.
Yeah, he would have popped up for the croquet sequence and got a one-liner in, probably had a nice scene with one of Anthony's love interests about the challenges of joing their weird-ass family. Maybe they'd have written in a spicy scene for him and Daphne to air midseason since the structure of season two didn't allow for as much sex/nudity as season one.
In other words, he would have been demoted from male lead to minor supporting character. It's no wonder he didn't want to do that.
Now, I think in a perfect world, he leaves the show as a series regular but they get him back as a guest star for one episode, having him accompany Daphne to the big Bridgerton ball, letting him show up to the croquest scene, etc. As a one-episode thing, I think they could throw everything from my first paragraph into one episode and given him a big enough role to make it worth a week of his time.
Ah well!