The Sub Mesh Mask setting allows you to specify which sub-mesh of your geometry to display/render.
This can be useful to only display a certain part of a Mesh. Or in more complex setup, in can be a way to pack multiple SubMeshes in on Mesh and one FBX and vary which submesh to display/render on a per-instance vfx level.
In Unity, Submeshes are defined by the Material ID specified in the DCC.
In this, this example, the “fractured” sphere has been assigned different materials for each of its 5 fragments.
This results with a mesh composed of 5 submeshes.

By default, a Mesh output in VFX Graph is set to display/render all existing Sub Meshes of a Mesh.
But you can Right-Click to “Check All/ Check None” or manually choose which submeshes you want to render:

Now, sadly, this setting cannot vary per-particles because, as stated by @Duckocide you cannot mix operations evaluated on the CPU and operations evaluated on the GPU.
If you want Per-particles mesh variation, the best would be to take a look at the Mesh Count feature in the Inspector (Will make a post later on this one).
Now, can this Sub Mesh mask still be useful? Here are some basic examples.
As we saw earlier, we need to use operators that are evaluated on the CPU.
Example A:
One mesh is composed of 4 submeshes, each Instanced VFX has one particle that can easily choose between SubMesh of the Mesh:

Example B:
While random per-Particle isn’t allowed, random Per-Component is fine, so you can randomize for each instance of your VFX which Submeshes are displayed.
Example C
The Time is also that can be used to animate the change of submesh and could be used for basic submeshes Flipbook packed in one Mesh.
I hope that you understand now more what can be done with this feature, and its current limitation. I will make a following post on the Mesh Count feature that allows per-particles variation of Meshes with the same Output Mesh Context.