2D automatic layer sorting extremely performance heavy

I have a scene where I spawn 50 agents. all of them follow the player. there were no performance issues. however 2D sprites need sorting so I turned on custom axis transparency sort mode in project settings. the sorting itself works great. when agents move ones higher up on y axis are sorted so they appear behind the agents which are below. however it’s a major performance hit. it ran smoothly on 60fps but when I use custom axis sort mode (unity’s automatic layer sorting) it goes down to 20-15 fps. this is my issue thank you in advance.