Summary of the impact

The specific impacts of this innovative research are two fold. Firstly, the research has resulted in the development of a novel intelligent framework called ALMIL (Adaptive Language Modelling Intermediate Layer) which acts as a communication layer between human and computer, to analyse noisy streams and provide disabled users with a better computer accessibility through prediction and corrections, speech and voice recognition. This has led to the development of an intelligent keyboard which provides a comprehensive solution for disabled people to help them to communicate more effectively with a computer. Secondly this research has produced 10 conference and journal papers so far and 1 significant paper is being reviewed in a highly ranked journal paper; also this research paves the way for the practical application development by providing theoretical basis of neural network approaches for noisy language modelling and reveals few directions in solving problems in specific areas such as biometrics.

Underpinning research

It is inevitable that users will make typing mistakes, which is particularly the case for many disabled users. These are different kinds of mistakes such as spelling errors; prolong key press and adjacent key press errors etc. Also, a series of research projects based on words vocabulary which apply both neural network and language modelling have been carried out. A model using neural network probabilistic language modelling has been suggested, to learn distributed representation for words that allow each training sentence to inform the model about the exponential number of semantically neighbouring sentences.

There are many types of errors caused by users, such as spelling errors, hitting adjacent key and cognitive difficulties. Prediction technology can foresee users' typing intention, but can't directly correct typing mistakes. Some efforts have been made to reduce these mistakes, although only a few tools can intelligently identify new type of mistakes. Intelligent models such as neural network models have been implemented in various directions, however, they are hardly seen to apply to noisy text entry processing such as user typing stream and its extracted sub-dataset, which implies all users' self-rectification actions, user's vocabulary, typing habits and typing performance. Moreover, although efforts have been made in multiple directions such as language modelling, Natural Language Processing and user interface design, those technologies, if used alone, will fail to meet the user's particular needs. Current models are short of self-adaptive ability (i.e. learning ability) and fail to fully recognize the right patterns from user's distinct performance. The shortfall in current research is that it has neglected the significance of negative influence incurred by the text entry noises, which have badly affected the accessibility and usability in human computer interaction, and a systematic solution as a bridge between user and computer to filter noises and make text entry more effective has never been on the agenda. In order to find a solution which is of scientific relevance and commercial significance, London Metropolitan University in partnership with Disability Essex embarked in a research under the KTP scheme and received a grant of £ 110,000 in June 2007 to develop an Adaptive Language Modelling Intermediate Layer (ALMIL) novel framework using a hybrid solution which is based on the combination of technologies (i.e. neural network and language modelling) and therefore to put all merits of those distinct solutions together; to develop a solution that is evolutionary and adjustable — a self learning model that can learn from users' past mistakes and can predict and/or correct these mistakes. A research associate (Academic Lead). A pan disability organisation of over 120 member clubs, serving over 10,000 individuals took part in the investigation. As a result a ground-breaking intelligent keyboard software has been developed which adapts to the challenges in a user's patterns of keyboard usage. It has helped individuals currently in work to continue to use standard IT hardware and software, notwithstanding any degradation of physical or cognitive abilities, for longer than is currently possible and also helped to assist disabled people in skills development. It has provided a solution to assist people with disabilities and learning difficulties to have a better learning and work experience, gain independence and improve quality of life.

References to the research

Details of the impact

What has evolved is a software that recognises, and tracks, variations in the user's condition, as evidenced by their use/misuse of keyboard strikes. This is a `learning' software which can learn from users' history and subsequently adapt. It appears to meet a currently unfilled gap in IT development. The research associate Dr. Jun Li was also awarded a PhD degree in June 2009 under the supervision of a team from the Intelligent System Research Centre, namely Prof. Ouazzane as director of studies and Prof. Kazemian and Dr. Yanguo Jing as second and third supervisors respectively. A novel fundamental concept in the area of neural network and language modelling has been developed and disseminated through journal papers and conferences. This research work brings forth an original concept, an ALMIL framework for noisy language processing, which acts as a noisy language filter and subsequently fills the gap between an input device such as a keyboard and a user applications. In this regard an Intelligent Keyboard framework, derived from ALMIL is developed as a hybrid solution based on modified neural network concepts and n-gram technologies. The user's typing data stream can be checked, rectified and predicted in sequence. With regards to scientific relevance, this research produced 10 research papers with one still being reviewed in a highly ranked journal and one good quality PhD completion in July 2009. More importantly, and at the commercial level, ALMIL-based Intelligent Keyboard had a great impact on disabled community as 15% of research was timely. Disability Essex was rewarded £ 2.2 million by the European Union in recognition of state of the art Intelligent Keyboard product. This has been used to build an advance energy efficient building, which opened officially in November 2010, as a centre for disability studies to assist disabled in skills development. Funded by the Technology Strategy Board, the Intelligent Keyboard was shortlisted for a Royal Association for Disability Rights (RADAR) award alongside the BBC and ITV under the category of `Doing IT differently' in November 2009, with Baroness Jane Campbell commenting: "Doing work differently enables disabled people to have maximum choice over their working lives". The product had also received Royal Acknowledgement. The Queen's Birthday Honours saw Richard Boyd, CEO of Disability Essex, being presented with an OBE for this work in June 2011. Prof. Karim Ouazzane, of the University's Faculty of Computing, was also mentioned in the Queen's official citation, which stated: "The University contribution, under the leadership of Dr Karim Ouazzane, was unique in the innovative partnership between a charity and a university to create new concepts based on user need." Dr. Debbie Buckley-Golder, head of Technology Strategy Board, who interviewed Richard Boyd, was also delighted of these KTP outcomes and subsequently issued a press release for publicity. TSB has classified the achievement amongst the best ever under the KTP scheme (see TSB web site).

The outcomes of this research has led to other research directions within ISRC. For example, the development of ALMIL machine learning novel approach has led to other industrial applications in the field of cyber security and biometrics. Lloyds TSB bank has recently reached an agreement with Prof. Ouazzane and Dr. Vassil Vassilev (also member of ISRC) to sponsor a research project based on the use of an approach derived from ALMIL framework. There has been the development of a concept for biometric information to analyse customer online transaction and customers' identities verification using typing stream and speech pattern recognition. The bank of England has recently made some queries with regards to the feasibility of using ALMIL for online banking security (this could be applied to both mobile and desktop applications).

LMU and Disability Essex are currently seeking more funding to enhance the product further by adding more features to the strategic product. Further research is needed to process input streams such as speech, scanning (OCR — style, size, font, grammar, flagging, numerical errors) and clicking stream. The framework will not only incorporate writing but also speech and voice recognition, and dictation (i.e. `hear it — type it, see it — type it, type it — check it''). The key aim will be to ensure that the human aspects of computing are taken into account in order to make the user experience a pleasant one to encourage and stimulate learning.

Sources to corroborate the impact

1. Technology Strategy Board Knowledge Transfer Partnership (TSB KTP) grant: London Metropolitan University in partnership with Disability Essex (2007-2009) (see KTP portal).

This claim corresponds to the following:

Reports, reviews, web links or other documented sources of information in the public domain. (e.g. www.ktponline.org.uk/news-2012-03-keyboard/)

The reports on the KTP portal corroborate the evidence, firstly, on helping individuals currently in work being able to continue to use standard IT hardware and software for longer than is currently possible and, secondly, helping to assist disabled people in skills development.

2. The following papers have been published:

K. Ouazzane, J. Li, H. Kazemian, Y. Jing and R. Boyd (2012) ` An artificial intelligence language modelling framework' International Journal of expert systems with applications; DOI: 10.1016/j.eswa.2011.11.121

J. Li, K. Ouazzane, H. Kazemian, Y. Jing, R. Boyd (2011) ` A neural Network Based Solution for Automatic Typing Errors Correction', Journal of Neural Computing Applications; DOI: 10.1007/s00521-010-0492-3

K. Ouazzane, Jun Li and H.B. Kazemian (2011) An Intelligent Keyboard Framework for Improving Disabled People Computer Accessibility, 12th Engineering Applications of Neural Networks and 7th Artificial Intelligence Applications and Innovations Joint Conferences, Corfu, Greece, Springer, Part I, International Federation for Information Processing AICT 363, pp. 382-391, 15th - 18th Sept 2011.

J. Li, K. Ouazzane, S. Afzal and H. Kazemian (2011) `Patterns identification for hitting adjacent key errors correction using neural network models' ICEIS 2011-13^th International Conference on Enterprise Information systems; Vol. 3 pp. 5, 2011.

K. Ouazzane, J. Li and H. Kazemian (2011) ` An Intelligent Keyboard Framework for Improving Disabled People Computer Accessibility' EANN/AIAI 2011, Part 1, IFIP AICT, PP. 382-391.

The above publications correspond to the following:

• Reports, reviews, web links or other documented sources of information in the public domain.
• Individual users/beneficiaries who could be contacted by the REF team to corroborate claims.