US 12,169,723 B2
System for simultaneous recording of the pixels of a screen and of accessibility data
Graham Nigel Dean, Cheshire (GB)
Assigned to REALITYMINE LIMITED, Manchester (GB)
Filed by RealityMine Limited, Manchester (GB)
Filed on Dec. 12, 2022, as Appl. No. 18/079,413.
Claims priority of provisional application 63/288,792, filed on Dec. 13, 2021.
Prior Publication US 2023/0185588 A1, Jun. 15, 2023
Int. Cl. G06F 3/048 (2013.01); G06F 9/451 (2018.01); G06F 21/55 (2013.01); G06F 21/64 (2013.01); G11B 27/34 (2006.01); H04N 5/76 (2006.01)
CPC G06F 9/451 (2018.02) [G06F 21/554 (2013.01); G06F 21/645 (2013.01); G11B 27/34 (2013.01); H04N 5/76 (2013.01); G06F 2221/034 (2013.01)] 16 Claims
OG exemplary drawing
 
1. A method for simultaneously recording video and capturing structural data to provide a view of user interface elements exposed on a user's screen of a user device, the method comprising:
recording a video of pixels of the user's screen on a user device and entering a recording state;
marking the video with a timestamp with the video start time;
simultaneously capturing at least one snapshot of hierarchical data structure of user interface elements via an accessibility application programming interface during the recording of the video pixels in the recording state;
marking the at least one snapshot with a timestamp;
continuously polling the accessibility application programming interface, wherein the at least one snapshot is queried recursively starting at an exposed user interface element, wherein the exposed user interface element has zero or more child user interface elements, providing a hierarchy of user interface elements and a data structure of the exposed user interface at a moment in time;
comparing the data structure to a previously stored data structure and storing the data structure when there is a change from the previously stored data structure;
simultaneously recording the video and the at least one snapshot of hierarchical data structure of user interface elements until the user selects to stop recording;
exiting the recording state;
storing the video and the at least one snapshot of hierarchical data structure of user interface elements as a session; and
synchronizing the recorded video and the at least one snapshot of hierarchical data structure of user interface elements for playback.