Is there a more efficient way to check nearby tile locations for colliders?

So am building a turn based game and something I need to be able to track is what nearby tiles are open so I currently have this code:

var selfPosition = gameObject.transform.position;
var availableNearbyPositions = new HashSet<Vector3>();
var hits = new RaycastHit2D[1];
var layerMask = LayerMask.GetMask(LayerName.CHARACTER) + LayerMask.GetMask(LayerName.GROUND);
var scanRadius = 3;

for (int x = -scanRadius; x <= scanRadius; x++) {
  for (int y = -scanRadius; y <= scanRadius; y++) {
   if (x == 0 & y == 0) {
     continue;
   }
    
   var checkPosition = VectorUtility.Round(
     new Vector3(selfPosition.x + x, selfPosition.y + y, 0), 
     1
   );
   Physics2D.RaycastNonAlloc(
     checkPosition,
     Vector2.zero,
     hits,
     float.PositiveInfinity,
     layerMask
   );

   if (hits[0].collider != null) {
     hits[0] = new RaycastHit2D();
     continue;
   }
    
   availableNearbyPositions.Add(checkPosition);
  }
}

So this code does give the desired effect however I am wondering if there is a more efficient way to do this. based on my profiling, this piece of code executed for 100 entity that are being tracked takes about 5ms.

While I might be able to figure out a way to scope down the amount of entities that I do this check for on each turn (and I am open to suggestions on this part too), I liked to know if there is a way I can trim down that 5ms down since that is close to half of the time I want to use for a frame, any suggestions?

Raycast is way to heavy for such a simple thing.
Store the tiles in an array. I personally like to use a multidimensional array for this; Tiles[x,y].
And then use data that links with Tiles[x,y] to determine what is on the tile; if(TilesOpen[x,y]){//this tile is open}.