Access to unity_LightmapST in ShadowCaster pass

Ok this might be a bit of an obscure question, but I am using the per-object lightmap offset on renderers to store values used for vertex deformation in a custom shader.

Basically along the lines of this:

_renderer.lightmapIndex = 0;
_renderer.lightmapScaleOffset = offsetValues;

(setting the lightmapIndex to 0 or higher seems required to have access to those scaleOffset values)

in my regular shading pass this works fine, doing something like this:
input.positionOS.xyz += unity_LightmapST.xyz * input.color.a;
and vertices will displace nicely on a per-object basis.

In the shadowCaster pass however, this goes a bit crazy and I end up with jittering shadows when the camera moves, as if objects are receiving the wrong unity_LightmapST values.

My guess is that the render pipeline sets these values per object only for regular passes, and skips these for depth and shadow passes (after all, these ordinarily don’t need lightmap info), so these passes end up using whatever the last set value was. If so, would there be a not-too expensive workaround? Or am I simply missing something else, perhaps I directive/keyword I should be enabling for this pass?

(And I would go the per-object-material route, and just have a per-material offset value, but that would imply an unmanageable amount of materials)