Why is my parallax effect glitching out with my perspective camera?

I have a 2d game with a perspective camera. In the gif above, you can see that around the edges, objects are not rendering/glitching past others. I have seen answers for this, but the only solution seems to be switching to an orthographic camera and faking the parallax effect. I was just wondering if there was any way around this?

Try this code Its perfectly works for me

using UnityEngine;
using System.Collections;

public class BackgroundScroll : MonoBehaviour {

    public float moveSpeed;
    private float offset;
    private Material currentMat;

	// Use this for initialization
	void Start () {
        currentMat = GetComponent<Renderer>().material;
	}
	
	// Update is called once per frame
	void Update () {
        offset += moveSpeed * Time.deltaTime;
        offset = offset % 1.0f;
        currentMat.mainTextureOffset = new Vector2(0f, offset);
	}
}