Technology Innovation Institute Announces Launch of NOOR, the Worlds Largest Arabic NLP Model
Technology Innovation Institute (TII), a global research center and applied research pillar of Abu Dhabis Advanced Technology Research Council, today announced the launch of NOOR, the worlds largest Arabic natural language processing (NLP) model to date.
TIIs team of advanced researchers and Artificial Intelligence (AI) specialists, has joined forces with LightOn, a technology company that unlocks extreme-scale machine intelligence for businesses, to transform the Arabic NLP model. The NOOR model has the capability to carry out tasks beyond the domain of language - offering end-to-end pipeline high quality data, including crawling, filtering, and curation at scale. The model facilitates extreme-scale distributed training and serving to deliver applications with efficient inference and model specialization.
Dr. Ray O. Johnson, CEO, TII and ASPIRE, said: With this development, we are well on track to enhance our research capabilities and credentials as well as elevate the status of Abu Dhabi and the UAE as a serious research ecosystem. Our expert teams have demonstrated yet again that this region can achieve breakthrough R&D outcomes to impact the world.
Dr. Ebtesam Almazrouei, Director, AI Cross-Center Unit, TII, said: Large language models have taken the world of natural language processing by storm, and we are proud to introduce this cutting-edge model with 10 billion parameters - the worlds largest Arabic NLP model. The uniquely large Arabic dataset collected to train the model is the result of months of work that included curating, scrapping, and filtering of varied sources. A special thank you to the entire team that worked on this project to make NOOR the go-to exploration model in Arabic for academicians and businesses everywhere.
Dr. Ebtesam Almazrouei said the NOOR model is based on the popular Transformer architecture. As a decoder-only model, similar in structure to GPT-3, it is programmed to tackle generative tasks with architecture upgraded to reflect the latest developments in the world of machine learning, including improvements such as better positional embeddings. To help ensure quality at scale in the NOOR dataset, the TII team designed an automated filtering pipeline based on machine learning techniques. These tools identify text like quality references and safeguard the model from exposure to spam content.
Leveraging state-of-the-art 3D parallelism, NOOR was trained on a High-Performance Computing resource with 128 A100 GPUs, allowing for the distribution of computations and ensuring efficient use of the available hardware resources. The Director of the AI Cross-Center Unit noted that this was only the first step in the Units efforts to contribute to the wider UAE Strategy for Artificial Intelligence.
