During the last decade or so, artificial intelligence (AI) has experienced a renaissance, with substantial technological advancements also arising in natural language processing (NLP). In addition to spawning more digital scenario applications, such as chatbots and intelligent writing, advances in NLP have resulted in dramatic improvements in machine translation quality, more accurate search and recommendations, and a rise in digital scenario applications such as conversational bots and intelligent writing. So, as the crown jewel of AI, how do diverse factors drive research advancement in NLP, which has attracted countless domestic and international businesses, skills, and capital? How do companies foster and utilize research findings, and how do practitioners evaluate the roadblocks and debates in the development of artificial intelligence?
In this article, we interview Mr. Shi Shuming, Director of NLP Research at Tencent AI Lab, to get his point of view on NLP and AI development in text understanding, text generation, and machine translation.
Far beyond a Lab: focus on the implementation of results and open source
Q: What are Tencent AI Lab's research objectives in natural language processing?
A: The Tencent AI Lab's natural language processing team performs research in four areas: text understanding, text production, intelligent dialogue, and machine translation. Over the past three years, the team has published more than fifty academic papers annually in top international conferences and journals, ranking us among the top research institutions in China. Two of our papers were selected as best papers at NAACL 2021 and outstanding papers at ACL 2021. We have received multiple prestigious academic contests, including first place in five tasks at the 2021 Sixth Conference on Machine Translation (WMT 21).
In addition to publications and academic contests, we have actively translated our research discoveries into systems and open-source data accessible to both internal and external users. These technologies and data include TexSmart, a text understanding system; TranSmart, an interactive translation system; and Effidit, an intelligent, creative assistant providing vector data for 8 million Chinese words.
Known as "Tencent Word Vector", the Chinese word vector data was published in 2018, which is at the top of the scale, accuracy, and freshness charts. It has already been extensively discussed, used, and integrated into many applications, and its performance has been continually improved.
Compared to similar systems, the TexSmart text understanding system offers fine-grained named entity recognition (NER), semantic association, deep semantic expression, and text graph. At the 19th China National Conference on Computational Linguistics (CCL 2020), it received the award for Best System Presentation.
The interactive translation system TranSmart is the first online interactive translation product offered to the general public in China, with standout features including translation input method, constraint decoding, and translation memory fusion. It supports multiple clients, enterprises, and scenarios inside and outside the organization, such as UN Docs, Memsource, Huatai Securities, Tencent Music, Yuewen Webnovel, Tencent Game abroad marketing, and Tencent Portfolio document translation, etc.
With its superior capabilities, the newly born intelligent creation assistant "Effidit" permits multidimensional text completion and various text editing options, utilizing artificial intelligence to assist authors in dispersing ideas, enhancing expressions, and enhancing the efficiency of editing and writing.
Q: Can you use "Effidit" as an example of intelligent cooperation and describe the inception and current status of the project?
A: The Effidit project, an intelligent writing assistant, was launched before October 2020. We did this project for two primary reasons: first, the pain point issues were addressed in writing, and second, our team has acquired the necessary skills to provide the required NLP technology for this situation.
Let us discuss the problematic aspects of writing. We often read news, novels, official-account articles, papers, and technical reports in our daily lives. We also need to produce technical documents, meeting minutes, reporting materials, etc. Reading is often a simple and uncomplicated activity, but writing is a different story. We frequently do not know how to use the correct words to describe our thoughts, and it can take a tremendous deal of work to compose phrases and paragraphs that still appear bland and contain several errors. Perhaps the majority of people are better readers than writers. Therefore, we pondered whether or not we might employ technology to alleviate these issues and enhance writing productivity.
The second reason is to consider how NLP technology may enhance human efficiency and quality of life. We have conducted extensive research in the NLP subdisciplines of text understanding, text generation, and machine translation in the past few years, among others. Most previous research focuses on a single point, and the point-like outcomes are complex for users to implement directly.
Therefore, we connect many points to form a line, i.e., a system. We are always looking for implementation options for text-generating research outcomes. We deliberated and decided to launch the intelligent writing helper Effidit project based on our previously identified writing challenges.
After one and a half years of research and development, the first version was launched. We will continue to iterate and refine, consider user input, enhance the efficacy of various features, and seek to provide users with a valuable and popular tool.
The interpretability and robustness of artificial intelligence are ignored
Q: In recent years, the community has paid considerable attention to trustworthy AI. Can you discuss the understanding and development of trustworthy AI in the NLP field?
A: I know very little about trustworthy AI; therefore, I can only offer broad concepts. Trustworthy AI is a vague idea for which there is yet no specific definition. However, it comprises several components from a technological standpoint, including model interpretability, robustness, fairness, and privacy protection.
In recent years, pre-trained language models based on Transformer have demonstrated astounding performance on various natural language processing tasks, garnering significant attention. However, such AI models are effectively data-driven black boxes with poor interpretability of prediction outcomes and low model resilience. They tend to acquire underlying biases in the data (e.g., gender prejudice), resulting in issues with model fairness. Word vectors, which appeared before pre-trained language models, also exhibit gender bias.
On the one hand, constructing trustworthy AI models is a current area of study interest in machine learning and NLP, and there have been several research initiatives and some progress. On the other hand, these advancements are still a long way from the objective; for instance, the development in the interpretability of deep models remains inadequate.
My workplace, Tencent AI Lab, is also researching trustworthy AI. Since 2018, Tencent AI Lab has invested in trustworthy AI and has made progress in three key areas: adversarial resilience, distributed migration learning, and interpretability. Tencent AI Lab will focus on AI fairness and interpretability in the future and continue to investigate its applications in the medical, pharmaceutical, and life science industries.
Fundamental challenge: statistical methods cannot comprehend semantics
Q: What is the current bottleneck in NLP research, in your opinion? What are the plans for the future?
A: Since the inception of natural language processing as an area of study, the most significant challenge it faces is how to comprehend the semantics of a natural language document. This backlog has not yet been eliminated.
Humans can comprehend the semantics of natural languages. When we encounter the line "She likes blue," we understand what it means since we know what "like" and "blue" implies. However, for NLP algorithms, there is no distinction between the above text and the unknown foreign language sentence "abc def xyz." Assume that "abc" means "she" in this unknown foreign language, whereas "def" means "likes" and "xyz" means "blue" When we know nothing about this language, we are unable to comprehend any statement in it. Suppose we are fortunate enough to witness a significant number of sentences written in this language. In that case, we may perform statistical analysis on them to establish the relationship between the words in this language and our language, hoping to one day comprehend it. This is a challenging procedure, and there is no assurance that it will finally succeed.
Deciphering a foreign language is more difficult for artificial intelligence than for humans. AI lacks the common sense of life, the mapping between the words of our native language and the thoughts in our heads. The symbolic approach in NLP research attempts to imbue AI with human-like abilities through the symbolic representation of text and knowledge mapping, thereby solving the fundamental problem of understanding. In contrast, the statistical approach temporarily disregards common sense and internal concepts and focuses on enhancing statistical methods and making the most of the information contained in the data itself. The second strategy has been the dominant mode of research in the industry thus far and has been more fruitful.
In terms of statistical NLP bottleneck breakthroughs and progress in the last decade, the word vector technique (i.e., representing a word with a medium-dimensional dense vector) broke the word computability bottleneck and, in conjunction with deep learning algorithms and GPU computing power, launched a series of NLP breakthroughs. In turn, the advent of novel network architectures (e.g., Transformer) and paradigms (e.g., pre-training) has dramatically enhanced the computability of text and the efficacy of text representation. However, statistical NLP can not mimic the common sense and underlying notions as well as humans do. It cannot comprehend natural language on a fundamental level. It is challenging to prevent common-sense mistakes.
The academic community has never given up on symbolization and deep semantic representation, with Wolfram Alpha and AMR being the most critical initiatives in the last decade or two (Abstract Meaning Representation). The primary obstacles are modeling many abstract notions and making them scalable. This is a lengthy and winding route (i.e., extending from understanding highly formalized sentences to understanding natural language text).
In terms of the underlying technology, potential solutions include next-generation language models, controlled text production, enhanced cross-domain migration of models, statistical models that incorporate knowledge efficiently, deep semantic representations, etc. These directions correspond to local research bottlenecks in NLP. In terms of applications, the path that must be investigated is how NLP technology can be used to improve human productivity and quality of life.
How should research and implementation be balanced?
Q: What are AI Lab NLP's future goals in fundamental research, cutting-edge technology, and industry implementation? What about the next steps?
Shi Shuming: In terms of fundamental research, our objective is to pursue breakthroughs, eliminate some research bottlenecks, and develop novel, functional, and high-impact products such as Word2vec, Transformer, and Bert. Fundamental researchers are given more freedom and encouraged to develop methodologies with a long-term impact. Meanwhile, we select a handful of challenges through brainstorming and other methods, and then we work together to solve them.
In terms of industrial application and technological transformation for the company's existing goods, we focus on developing one or two Tencent-led products to integrate research results to enhance work productivity or quality of life. TranSmart is an interactive translation system for translators, while Effidit is an intelligent creativity aid for text editing and scenario authoring. These two solutions will be continually enhanced.
Researchers need more liberty to express their views
Q: What are the varied priorities of researchers and algorithm developers at Tencent AI Lab?
A: An algorithm engineer's tasks on our team include implementing a new or optimizing an existing algorithm (such as the method in a published article) and implementing and enhancing the technical product. In addition to the two responsibilities of an algorithm engineer, a researcher is responsible for the proposal and publishing of novel research results. This divide is also not absolute; the limits are somewhat fuzzy and greatly dependent on the employee's interests and project requirements.
Q: As a manager, what are the distinctions between Tencent AILab's management practices and philosophies and those of conventional technical engineers?
For business teams, technical engineers must collaborate closely to create goods that have been designed using a specific project management approach, according to Shi Shuming. The majority of lab teams consist of fundamental researchers and engineers (and possibly a few product and operations personnel). Researchers must be allowed greater flexibility, less "direction," but more assistance for fundamental research. At the same time, their interests are respected, their potential is stimulated, and they are encouraged to conduct work with potential long-term effects. Most fundamental research breakthroughs are not planned or managed using the project management procedure. On the other hand, lab teams necessitate more communication between researchers and technical engineers while developing technical products and a lightweight and agile project management procedure.
AI positions in Tencent AI Lab: focus on three factors and be strong enough internally
Q: If a candidate has excellent research skills and has published several papers at high-level conferences but has terrible engineering skills, would you hire them?
A: This is an excellent issue, and we frequently encounter it while recruiting. In an ideal world, academia and industry would teach or attract individuals with outstanding research and engineering talents. However, such individuals are uncommon in practice and are frequently the subject of rivalry between businesses and research organizations. For individuals with excellent research skills, we will relax the standards for their engineering skills throughout the interview process, but they must still exceed a minimum level. Similarly, we reduce the criteria for their research skills for individuals with great engineering skills. If properly organized, personnel with solid research skills and those with strong technical skills will play to their respective strengths by collaborating to complete the project.
Q: Which applicant abilities do you appreciate the most?
A: Dr. Shen Xiangyang has identified three essential criteria for recruiting candidates: mathematical proficiency, programming proficiency, and a positive attitude. Good at math refers to research potential, good at programming corresponds to engineering skill, and a good mood includes "enthusiastic about one's job," "win-win collaboration with colleagues," "reliable in carrying out tasks," etc. Numerous research institutions esteem these three characteristics.
The actual interview process frequently assesses candidates' research ability and potential by reviewing their papers and discussing projects, evaluates their engineering ability through programming tests and project outcomes, and speculates whether candidates genuinely have a "good attitude" throughout the interview process. This type of conjecture and evaluation can occasionally be inaccurate but is often accurate.
Other qualities are impossible to assess in an hour or two interview. Therefore there is an element of luck involved. As a first step, you must be able to select an essential research topic. You also need to be capable of carrying out a task. It is possible for people or teams without this capacity to initiate a variety of topics or projects. However, they cannot complete them with good quality or cannot complete them at all. This may be related to execution, persistence, concentration, technological standards, etc.
The third characteristic is the capacity to withstand alone and criticism. Many people do not understand essential and influential things before they emerge. If they lack the inner strength to endure solitude and criticism, it may be difficult to persist; instead, they may jump into hot spots to partake in short-term interests.
Q: What is your advice for college grads and technical professionals who have changed occupations to pursue the AI field?
Q: Because each graduate's education, schooling, and project experience are unique, and technical individuals who transition into AI have even more professional and personal diversity, it is challenging to provide overly general guidance.
Momentarily, I can think of only a few points: First, do not overlook the accumulation of knowledge and intelligence by burying yourself in tasks. Please find a few more people to ask for advice, listen to them on their present employment, and appraise various work and organizations to comprehend the road they have traveled and the obstacles they have encountered. Simultaneously, gather information through forums, official accounts, short movies, and other means to assist you in making decisions at this crucial juncture in your life.
Second, choose a reputable internship program if it has been more than a year since you graduated and you have no job experience. Through internships, on the one hand, you may gain practical experience, expand your skills, and gain a sense of the work environment; on the other hand, internship experience will make your CV more appealing and increase your competitiveness.
Third, it is inescapable that things will go awry in the job, so manage your expectations, modify your perspective, and find strategies to process your negative feelings.
Fourth, keep your vision in mind, work hard, and do something commensurate with your abilities.
It is my sincerest wish that every college graduate finds the position of their dreams and succeeds in their current position and that every techie who makes the transition to AI is rewarded for his or her hard work.
About Guest
Graduated from Tsinghua University in Computer Science, Shi Shuming is the Director at NLP Research, Tencent AI Lab. His research interests include knowledge mining, natural language understanding, and intelligent conversation. With an H-index of 35, he has published more than 100 articles in ACL, EMNLP, AAAI, IJCAI, WWW, SIGIR, TACL, and other academic conferences and publications. He was a demonstration co-chair for KMNLP 2021 and CIKM 2013, a senior program committee member for KDD2022, and a program committee member for ACL and EMNLP.
