Accessibility and screen reader: how to mask elements, deactivate groups and more?

Unity Engine
, , ,
1

I’m currently working on a mini-game app whose primary audience is people with visual impairments.

I’ve reviewed the available posts about the official accessibility module, as well as the LetterSpell GitHub project, which I’ve used as a basis to solve most of the issues I’ve encountered.

Even so, there are certain cases I haven’t been able to address for the design requirements I’ve been given.

Below is a list of problems, cases, and solutions I’ve come across.

Problem 1: Masking

Case 1.1: In the main menu, there are two main sections: the top bar where navigation is located and the bottom section. In the bottom section, there’s a scrollable list of games. The problem is that even when a button is completely covered by a mask, it can still be selected when TalkBack is enabled, even over the navigation buttons.
Here is a representation of how the Talkback would behave. Even pushing the tabs buttons selects the ones under it

image
image768×461 46.4 KB

image
image743×444 35.8 KB

Failed Solution 1.1.1:
I tried adding canvas elements and graphic raycasters to force a layer order change. I also tried adding an image as a blocker, but it seems the screen reader hierarchy is independent.

Semi-Failed Solution 1.1.2:
By changing the order of the hierarchy (placing the navigation bar lower), the AccessibleElements within it were prioritized. However, this creates a new problem: the screen reader detects the navigation bar as the last elements on the screen, which could cause confusion.

Case 1.2: In the mini-games, a pop-up appears before starting to allow the player to prepare. The pause button is located in a corner and can still be accessed using TalkBack.

Solution 1.2.1:
Deactivate all GameObjects that might interfere. This isn’t the optimal solution I’d prefer, but at least it works. In these types of cases, having an element similar to a Canvas Group where all child AccessibleElements can be deactivated would be very helpful.

Problem 2: AccessibleNode Elements and Reading Order

Case 2.1:
I’d like to add more variables to an AccessibleElement and control which are read and in what order. For example, if there’s a list of buttons, selecting one should read:
“Level 1, button, item 1 of 12, double-tap to interact.”

A more advanced case that’s been requested is to read the parent element the first time. For instance:

  • If the focus is on navigation and shifts to a level button, it would read:
    “Level selection, level 1, button [etc.].”
  • However, if navigation moves to a sibling, it would skip the parent and read:
    “Level 2, button [etc.].”

Solution 2:
I haven’t found a starting point for this. The best I’ve managed is creating a custom element that replaces certain parts of the label with variables.

Case 2.2: In the options menu, toggles currently read their active state before reading the label.
Current result: “Activated, music, toggle.”
Expected result: “Music, toggle, activated.”

Right now, there’s very little documentation on this topic, so I’m quite lost. I’m also referencing apps like WhatsApp to see how I can make the design as standardized as possible.

In apps with accessibility features, I’ve noticed some important small details missing from the accessibility package (or at least I haven’t been able to find them):

  • While scrolling, subtle clicks become sharper as you approach the end of the scroll.
  • Switching between sections (e.g., navbar to main content) plays a “boop” sound.
  • When opening a pop-up element (like a three-dot menu), the rest of the elements become inaccessible until the pop-up is closed.
2

Hi pixelf0x300,

Thank you for reaching out!

Problem 1

Cases 1.1 & 1.2
You are right that the accessibility hierarchy is independent of the game object hierarchy. That is why the visibility or the layering order of game objects will not affect the visibility of accessibility nodes. Neither will the active state of game objects (unless you create accessibility nodes only for the active game objects, as in the LetterSpell sample project).

By default, all accessibility nodes are visible to the screen reader. In cases where you need to hide certain accessibility nodes (such as when they move under the navigation bar or when a popup is opened), you will have to set AccessibilityNode.isActive to false for the nodes you need to hide.

Problem 2

Case 2.1
In your example, you could make the screen reader read “Level 1. Item 1 of 12. Button. Double-tap to interact” by setting the following properties on the accessibility node:

Note that if you add a callback to AccessibilityNode.selected, then TalkBack will automatically say, “Double-tap to activate” after a short pause since it will know that the node is interactable.

If you want it to also read the parent element when focused on a child element, then you should prepend the label of the parent node to the label of the child node.

Case 2.2
This is normal TalkBack behaviour. Unfortunately, there is nothing we can do about it.

Other notes
In some cases, the screen reader behaviour achieved with Unity’s Accessibility API will be close but not identical to the native behaviour. This is because, at this point, we chose to implement the most valuable subset of native accessibility APIs that are relevant on both iOS and Android.

Your feedback is invaluable to our progress. We have taken note of your experience and honestly thank you for sharing it!

I hope this helps! Let me know if there is anything else I could clarify for you.

We’d also love to know more about this project. :slightly_smiling_face: When it becomes available, would you kindly share a link to the app?

1 Like
3

Thank you so much for the detailed reply! This information will be incredibly helpful.

To make sure I understand correctly [problem 1]: Do I need to disable the AccessibleNode to simulate occlusion? There isn’t any raycast blocker or anything similar at the moment, right?

Understanding these limitations allows me to manage expectations with my bosses and explain that achieving 100% similarity to a native app may not be feasible. However, I’ll do my best to replicate as much as possible.

I have an idea for the masking, but is too complex for me to code it:
I’ve seen that the AccessibleScroll updates the position of the child elements on scroll, but the height and width are always the same.
Maybe there’s a way to know when an element is partially or fully out of the viewport and do some math magic to calculate the dimensions of the showing part.

So far, this package has covered most of the requirements, and thanks to the LeterSpell example, it has been a ready-to-use solution. It’s fantastic!

Currently, we are in the early stages of development and cannot share specific details due to the NDA. Once the project is completed later this year, I’ll be happy to share my findings and a link.

4

I am glad it helped. :slightly_smiling_face:

Yes, you need to disable the accessibility node to simulate occlusion. Accessibility nodes will not be affected by any raycast blocker because they are not visual elements – they are simply data structures shaped based on what the operating system requires to enable screen reader accessibility.

I am looking forward to any updates or feedback!

5

I’ve encountered a significant issue affecting the app’s development. Until now, I hadn’t tested TalkBack alongside the gameplay, and it turns out to be more problematic than I anticipated.

Context

The input is quite simple during gameplay: touching the screen corresponds to a specific action. In future minigames, it will be necessary to distinguish whether the touch occurs on the left or right side of the screen, but for now a simple tap anywhere (that isnt the pause button counts as “interact”)

My Implementation

I’ve set up an element with IPointerClickHandler that covers the entire screen. For minigames requiring specific areas, I could use multiple elements. This approach works perfectly without a screen reader.

The Problem

During gameplay, there will always be an accessible button to pause the game. Even if there wasn’t, I believe this issue would still occur. TalkBack intercepts single taps, which prevents my input system from working as intended. I’ve seen that iOS offers a “direct interaction” mode, but I don’t have access to an iPhone for testing.

I’ve implemented a gesture (pinch in/out) using the InputSystem to pause and unpause the game, and this works even with TalkBack enabled.

public class PinchScrollDetection : MonoBehaviour
{
	public float speed = 0.01f;
	private float prevMagnitude = 0;
	private int touchCount = 0;
	private void Start()
	{

		// pinch gesture
		var touch0contact = new InputAction
		(
			type: InputActionType.Button,
			binding: "<Touchscreen>/touch0/press"
		);
		touch0contact.Enable();
		var touch1contact = new InputAction
		(
			type: InputActionType.Button,
			binding: "<Touchscreen>/touch1/press"
		);
		touch1contact.Enable();

		touch0contact.performed += _ => touchCount++;
		touch1contact.performed += _ => touchCount++;
		touch0contact.canceled += _ =>
		{
			touchCount--;
			prevMagnitude = 0;
		};
		touch1contact.canceled += _ =>
		{
			touchCount--;
			prevMagnitude = 0;
		};

		var touch0pos = new InputAction
		(
			type: InputActionType.Value,
			binding: "<Touchscreen>/touch0/position"
		);
		touch0pos.Enable();
		var touch1pos = new InputAction
		(
			type: InputActionType.Value,
			binding: "<Touchscreen>/touch1/position"
		);
		touch1pos.Enable();
		touch1pos.performed += _ =>
		{
			if (touchCount < 2)
				return;
			var magnitude = (touch0pos.ReadValue<Vector2>() - touch1pos.ReadValue<Vector2>()).magnitude;
			if (prevMagnitude == 0)
				prevMagnitude = magnitude;
			var difference = magnitude - prevMagnitude;
			prevMagnitude = magnitude;
			if (-difference * speed > 0.1)
			{
				GetComponent<GameStateManager>().PauseGame();
			}
			else if (-difference * speed < -0.1)
			{
				GetComponent<GameStateManager>().ResumeGame();
			}
		};
	}
}

Before switching to IPointerClickHandler, I used to handle single tap events similarly to the pinch gesture script. However, this caused unintended behavior: tapping the pause button would trigger gameplay actions, and trying to perform the pause gesture could also activate gameplay events.

I reviewed the approach used in LetterSpell, but it won’t work for some of my minigames, as they require precise timing or quick reactions to sound cues. I need a method that allows direct interaction while ensuring gestures or interacting with onscreen buttons aren’t misinterpreted as gameplay input.

I’ve considered setting the IPointerClickHandler zone to be an Accessible node but it would still require a double tap instead of a single one. this isnt ideal for a single zone, much less for those minigames with more (having to change the focus first would add time to the interaction too).

This is my first time developing a smartphone game, and maybe the solution is simple, but i can’t find how to solve it adding the screen reader complexity.
Is there any way to achieve this? Any ideas or suggestions would be greatly appreciated :smile:.