Basically, color representation is a multiplication of light and base color, so I understand that this is correct if there is no specular.
However, this may not be the desired result when light baking.
Pure black materials will absorb all direct and indirect light. This is physically correct behavior. In real life, no naturally occurring material is completely black. One of the darkest natural materials, coal, measures approximately at 50,50,50 on an RGB luminosity scale.
Adjust your material color values to comply with physically based shading standards. In the built-in render pipeline, you can use Validate Albedo scene view draw mode to determine whether albedo values are PBR-compliant. Use the Rendering Debugger in URP and HDRP to do the same.
What I want to point out here is that the baked lightmap is not handling specular lights.
And, that’s the part where real-time lights and baked lights have different results.
Or, do you mean that the specular handling of realtime lights is not correct?
When it comes to the lack of specular response; that is an inherent limitation of baked lights. If you require proper specular response in your materials, you have these options:
Use mixed lights
Use emissives + relfection probes to simulate specular highlights