WOW! Huge props for mentioning Xonix… I played that game a lot! The version I played was in CGA and it was awesome, a great throwback to the old Qix game.
So what you want is broadly called “floodfill” and there are many well-understood libraries out there to help you floodfill areas.
In Qix and Xonix, when you floodfill, you have to first do a “test” floodfill to see which side of the new line the enemies are on, and if you perhaps split the enemies.
Then you need to do an actual floodfill on the side they are not on, and during the floodfill you can count up how many squares.
Here is a discussion and pseudocode for making your own floodfill:
For everybody else, here is a screenshot of Xonix: don’t be fooled by the simple looks: it was a great game play-wise!