Disney's AI model creates animated sequences from scripts

In fact, the fact that AI can create original clips from one or more paragraphs is not a seismic information in the technology world. Last year, researchers described in detail how a neural network mining system - classes of mathematical functions simulated in the form of biological neuron networks (neurons) - to create passages. The video is 32 frames long and measures 64 × 64 pixels from many descriptive pieces of data that are suggestive, such as 'playing on the grass'. However, according to a new article published on Arxiv.org, scientists at Disney Research and Rutgers have succeeded in bringing this idea one step further to new heights from the end to end, can create a rough storyline as well as text description video from movie scripts. Specifically, the scientists' text-to-animation transfer model helps create animation without annotated data - the preliminary step which is used to produce descriptive input text for activities certain.

Disney's AI model creates animated sequences from scripts Picture 1

1. OpenAI artificial intelligence defeated the current world champion Dota 2

'Automatic creation of natural language text animation is a very useful technology that can be applied in a number of areas such as film scripting or video tutorial creation. In particular, these AI systems will be particularly valuable when applied to script writing by enabling iterations, prototyping and proof of concept more quickly. In this study, we have successfully developed an animated text transfer system that can handle complex words. The purpose of this AI system is not to completely replace writers or writers, but to create an AI assistant who can effectively support and help writers work more interesting ', the team shared.

As the researchers explain, translating text into animation is not a simple task. In fact, neither the sentence (input data) nor the animation (output data) have a fixed structure. This is also the reason why most text-to-video conversion tools cannot handle complex sentence patterns. To address the limitations of existing systems, the team has built a modular neural network that includes a number of components such as the new script, help script dynamically isolate relevant text from scene descriptions in the script; a natural language processing module that simplifies complex sentence patterns by using a set of language rules and extracting information from simplified sentences into predefined action representations ; and an animation creation module that converts the aforementioned representations into multiple animation sequences.

Disney's AI model creates animated sequences from scripts Picture 2

1. EU: AI systems should be developed and implemented in a transparent and responsible manner

According to the researchers, this simplified approach makes it easier to extract key script information, their system will be able to automatically determine when a piece of code gave the use of a specific syntax structure, and then parsed and assembled it into simpler sentences, then continued recursive processing until it was no longer possible to simplify further. The next 'step of coordination' will be applied to sentences that have the same syntactic relation and at the same time serve the same functional role. And finally, a vocabulary simulator that fits the descriptive actions in the sentences will be simplified with 52 different animations (expandable into 92 animations using the word dictionary. definition) in a predefined library.

Later, a system called Cardinal will use these animations as input for actions and create previsualization (the process of converting a scene and script into 3D images) in Unreal - A popular video game tool developed by Epic Games. Based on a predefined animation library, objects and also models that it can use to create pre-loaded characters, thereby creating 3D animated videos that describe near true to the script handled.

Disney's AI model creates animated sequences from scripts Picture 3

1. Chatbot AI supports finding information about frauds and frauds

To train this preeminent system, researchers had to embark on compiling a scene description database made up of 996 scripts, drawing from more than 1,000 scripts taken from free sources. including IMSDb, SimplyScripts and ScriptORama5. In total, this data warehouse includes 525,708 descriptions containing 1,402,864 sentences, 920,817 (more than 40%) including at least one verb describing the action.

In a qualitative test, scientists assigned 22 participants to rate 20 system-generated animations on a 5-point scale (for example, whether the displayed video is a logical animation with the text , or how much text information is described in the video and how much information in the video has been mentioned in the text), 68% of the participants said the system created the animation there fair value from input scenarios - a rate not particularly high but very commendable.

That shows that this is not a really perfect system. In fact, its list of actions and objects is incomplete, and sometimes, the process of simplifying vocabulary cannot successfully map complex verbs into similar animations, or only can create a few simple sentences for a verb with many subjects in the original sentence. However, this is still a fledgling study and such limitations are fully understandable. Researchers intend to address these shortcomings in the near future.

Disney's AI model creates animated sequences from scripts Picture 4

1. MIT AI model can capture the relationship between objects with the minimum amount of training data

'Internal and external evaluations both show the reasonable performance of this system, and we want to leverage the discourse information by examining the sequence of actions described in the paragraphs. This will also help resolve ambiguity in action-related text. Moreover, our system can completely be used to create the necessary data source used in training similar end-to-end nervous systems, research groups. save sharing.