I totally get your feedback, but it’s a tough balance. The BaseField<> class hierarchy is exactly the type of place where too much stuff was made available and almost completely blocks us from refactoring/simplifying the design.
That being said, here is a sample of a CustomField subclass made without access to visualInput: