I don’t think that’s possible. Usually, you’d have a field to store the value of the CustomStyleProperty, and that’s the field you modify from code. You’d probably also want to store a bool to indicate that the field has been set from code to avoid overwriting it when there’s a new CustomStyleResolvedEvent.
There’s a good example of this in Unity’s own Image element: check how they use m_TintColorIsInline and the tintColor property.
It is like ref in this context, the only difference between an out parameter and a ref parameter is that the out parameter is guaranteed to be assigned by the method that uses it. You would have the same problem if it was a ref parameter. The parameter would still a variable created by you, so it couldn’t be a reference to the original style property.
Maybe you are thinking of ref returns; that does do something similar to what you suggest, but it’s used very differently, and I believe Unity doesn’t use it in any of its public API.