Translation software must be able to distinguish between male and female characters in order to insert the correct pronouns and ensure that the dialogue is sequential. It must also be able to utilise the context of previous and subsequent speech bubbles and tackle ambiguities such as words with multiple meanings.
All sentences in any scene must be analysed before producing a suitable translation.
Comics and manga feature very few words and so the choice of words is crucial. Those words must fit into the speech bubbles without the necessity to resize or reposition them. Even if it were possible to resize the bubbles, changing them could negatively impact the graphic content of the publication.
The Mantra Inc. Solution
The team at Mantra have evolved a system which recognises and processes the scene, reading order and visual cues of comic book stories. They believe that they have developed the first multimodal, contextually aware translation system. To achieve this, they have been forced to overcome the usual limitations of NMT translations which tackle one sentence at a time and do not consider context. Text recognition, machine translation and image processing have been assimilated into a single system by the Mantra developers.
This significant advancement was delivered via automatic corpus construction. The AI software was trained using original manga stories and their translations. An incredible two million sentences were used to train the system.
The Mantra Inc. system recognizes Japanese manga texts and translates them into the target language using the trained neural machine translation model. The original Japanese texts are removed from the page. Translated texts are then added to the cleaned page. Font sizes are optimised to perfect the presentation.
Cloud-based system released
The Mantra Engine has been now been released. It is a cloud-based service that enables the swift rendering of manga into English and Chinese. Further languages will be added in the future. Aimed at both publishers and distributors, the system can reduce the time taken to translate manga by up to 50%. It is user-friendly and enables sharing for all parties involved in a project. Multiple stakeholders can participate in the translation process simultaneously.
The implications of the Mantra Engine
Manga has proved to be prohibitively expensive to translate, confining the distribution of many titles to Japan. English and Chinese speaking manga fans could soon be able to access more publications than ever before. There is no doubt about the appeal of manga, as many titles feature in the top twenty best-selling comics of all time. Indeed, One Piece, created by Eiichiro Oda, has sold more copies than any other comic on the planet.
But the implications of the Mantra Engine extend beyond improving the accessibility of manga. The system presents the possibility that all comics could reach larger audiences and reach them quickly.
Naturally this AI system represents a significant advance for machine translation. As such, it could be the beginning of a new era. One in which neural machine translation is far more effective. One day, NMT could provide accurate and appropriate translations when images are integral to the meaning and context of the source material.