Virtual Keyboard

Overview

This project aims to improve accessibility to computers for individuals with severe motor impairment, specifically advanced ALS patients. The proposed Camera mouse HCI-based virtual assistant includes a virtual keyboard with an embedded Internet browser and allows users to interact with the computer by registering any part of their body as the pointing device. The system was tested with ALS patients and showed promising results in improving their quality of life and providing uninterrupted access to the computer. This innovative system has the potential to revolutionize the way individuals with motor impairments interact with computers, ultimately providing greater accessibility and independence.

Click here to view the research paper

Collaborators

The contents of the page above and the research demonstrated on this page follows contributions as below:

  1. Spandita Sarmah – Design and Development of the Web Page, UX Prototype, User Research and Study of Previous Research Papers

  2. Rahulraj Singh – Implementation of the Algorithm

  3. Dr. M. Uma – Guidance during ideation, implementation and final paper authoring

Features

  • Individual Usage - The camera mouse operates as a single entity input. The user first registers an input point (eyes, forehead, fingers or any other body part) and the algorithm stores this information. It cannot be replaced

  • Near Real Time Response - The response time for the action performed in front of the camera and it reaching the CPU is 0.57 seconds and feels exactly like, well, directly operating a mouse

  • Voice Based Prompts - To make life even easier, the virtual keyboard uses Microsoft’s Cortana on the windows computer to enable voice based inputs from users into the text field they are using the keyboard

System Architecture and Design

Keyboard Layout: The layout of the keyboard, or the first form of the software shows the outline from where all the other elements of the keyboard can be accessed. The design is not the traditional one, with a qwerty model with only one reason being the user is expected to be using it for the first time, and would use it very less often. Apart from the alphabets and numbers there are eight functional keys on the top.

Text-To-Speech: This is an important feature for those who have difficulty in speaking and can use the camera mouse interface to type their needs, and then use this feature to speak it out, so that, they can allow people around them to realize that they are need of something.

Help: The self-help option is a button that is a one-stop help option for the user, and can be used to convey messages or demands to other people around the user asking for help, or simply displaying the needs of the user.

Music Player: The music player can be used for both audio and video files, and also provides a feature of creating the user’s personal favorites’ playlist.

Social Networking (Gmail / Facebook): Social Networking has become an important and significant part of our lives, and therefore this keyboard allows the user to access their social networking accounts and actively participate in them

Predictive Drop Down: Predictive drop down is a feature useful for both, accessing the Helper toolbox and also for the regular typing on the main screen. The drop down functions based on a database that contains words that the user has been using frequently and actions that need to be performed at a regular basis. This would help, save time for the user and he would not have to type the entire word again and again.

What Users Say

I tried working with the application and smoke testing it. The performance is great. It has great accuracy with clicking and typing - Mrs. Uma, Assistant Professor

I participated in the testing of the application and used features of logging in to Facebook and listening to music. The application is easy to use and has sufficient features of regular use - Anonymous Test User, SRM Institute of Medical Sciences

Diary Study

Method and why did we choose to do this:

A diary study was conducted to evaluate the usability of the Camera Mouse HCI-based virtual assistant developed for individuals with severe motor impairment. This method was chosen because it allowed participants to use the software in their own environment over an extended period of time, providing more naturalistic data and allowing us to observe how participants interacted with the software in their daily lives.

Participant Recruitment:

Five participants with advanced ALS were recruited for the study. Participants were recruited through the local ALS support group and were required to have severe motor impairment, making it difficult for them to use a standard mouse or keyboard.

Procedure:

Participants were given a brief tutorial on how to use the software and were asked to use the Camera Mouse HCI-based virtual assistant in their daily lives for a period of two weeks. Participants were instructed to keep a diary of their experiences using the software, including any issues they encountered, and to complete a brief survey at the end of the two-week period.

Data Collected:

Data collected included the participant diaries and survey responses. The diaries were analyzed for common themes related to the usability of the software, such as ease of use, satisfaction with the software, and any issues encountered. The survey responses were analyzed for participant satisfaction with the software and perceived ease of use.

Results:

Overall, participants found the Camera Mouse HCI-based virtual assistant to be highly usable and effective in improving their quality of life. Participants reported high levels of satisfaction with the software and found it easy to use. The virtual keyboard was particularly praised for its ease of use and the text-to-speech feature was found to be particularly helpful for those with difficulty in speaking. The predictive drop-down feature was also appreciated by participants, as it saved them time and effort when typing.

However, participants did encounter some issues with the software, including difficulties in registering certain body parts as the pointing device and occasional glitches in the music player. Some participants also reported difficulty accessing their social networking accounts, although this was not a major issue.

Overall, the results of this diary study suggest that the Camera Mouse HCI-based virtual assistant has the potential to revolutionize the way individuals with motor impairments interact with computers, providing greater accessibility and independence.