Simplest phrases parsing instead of C# scripting

Scripting C# & .NET
1

Dear unity3d beta & experimental scripting team.
One of my interests is computational linguistics, more precisely - conversion of simple natural-language (english) phrases into unity3d game engine commands…

There are various programs (including open-source / c#) which can analyze English phrases or even sentences - for natural language translation or for coversion of analized text into audio-visual sequences.

What I 'ld really to see one day in Unite3d is a feature to include scripting in natural language. Of course that is not possible for most sentences. But what is possible is: to add certain functions into Unity3d which should have syntax similar to natural language.

What do I mean? Say, I have a Unity3d project, which I use as a basis for my 3d games. In this project I have input fields: I just type a name of subject (say, some Lego-style humanoid-biped - that’s universal), and its action (say, there are around 2500+ free animations for such bipeds - from carnegie-mellon university). These are two fields.

But what I want is to write such script not in input fileds (thus unity3d knows, what is subject, and what is action of this subject-actor), but as a script inside c# file (or a file of other filetype - some higher level language).

What I want to achieve is that functions should look not like this:
Public void Action (Transform actor, animationName anim)
But such function should ideally look like this:
Actor acts.

I suggest omitting keywords like public, void, omit brackets and “,” (comma), and omit even types of data/variables - they should be guessed automatically in a way, just like it is done programs which translate text from one language to another (like Google Translate) or it should be done in a way as it is done in programs which convert text to audio-visual sequences (with text-to-speech for audio).

What i’ld like to achieve is that it could be great if simple scenarios which are human-readable (look like real English sentences or more probabaly - as simple English NounVerb (NV) phrases - or other sentences of such kind - combinations of parts of speech) would be understandable/parsable by Unity3d. If such texts are included inside C# files, then fields of such sentences are to be marked : before such code block and after that should some tags like in html, say

Actor acts3
Actor2 acts6

Or like that.

So thus certain parts of c# script-files should include such fields, should be able to read / parse such fields…
In this way, programming would become easier and faster.
Of course, each 3d-object has its unique id/number, but if each such 3d model, including one which can play animations, if such 3d-meshes can be applied Named by “tags” or names (in fact, UNITY3D already has names for each object in scene editor) - in unique way, this would allow to precisely tell the engine which actor should act. Similarly, if each animation would have a unique identifier, then it can be used in such … code block for easy programming…

I think, this would be a good feature to consider adding into Unity3d script editor (3d scene view should also check that each 3d model/mesh should have unique name)…

if you agreethat this is good idea, then additional phrases support should be added to unity3d.

  • NVN (say, “a samurai”, “goes to”, “a troll”) - Noun+Verb+Noun2…
  • AdjN (“green”, “tree”) - Adjective+Noun. adjective should act as material/Texture
  • NPN (“an apple”, “is on”, “a table”) - Noun+Preposition+Noun2
    Etc etc

That’s a good idea, yes?

2

There is a reason why computer programs are written in programming languages and not in natural language. The programmer has to define the behavior of his program precisely, as he wants to exactly control what is happening at what moment. This is simply not possible in English or any other normal lanugage.

Keywords like ‘public’ or ‘void’ have a concrete meaning and cannot simply be removed from the language. How should the compiler know if you want your function to return a string, an integer or nothing at all? How should it be able to guess that? Again, a computer program has to be precise.
Natural language might seem to be easier to program in, but in the end it would make programming much more complicated.

How would you even try to make this work? You cannot just say, e.g. ‘Draw a circle’. You have to specify its radius, its position, its color. In the end, everything you specify in natural language will have to be the same as what you write in code. Additionally, high-level code is a lot more expressive in describing programs than natural language ever can be once you understand programming.

There are, however, tools for Unity such as visual scripting utilities etc. that can ease the process of programming if you do not want to code in the traditional way.

1 Like
3

These are pretty interesting ideas, but they are not something we can implement in Unity any time soon. It sounds like an interesting academic research idea though!

4

Some time around 2009-10, I was talking to students of Animation department in Kiev Institute of Cinema and Theatre of Karpenko-Karyi. I told about my idea that it could be great to make a computer program, which could automatically convert scenarios (of movies, plays or games) into audio-visual sequence. I just said that program should automatically analyze text just like Google Translate does. Then result of analysis should become input for 3d graphics engine and for text-to-speech (like Microsoft Agent).
Of course, it is pretty hard to make it, so their reaction was that even few thousands chinese would be working on it, they wouldn’t get make a good program…
However, after a short time I found out that a Russian movie company Bazelevs Innovations received a funding from Skolkovo fund - around 150 000 000 russian rubles for this technology. They called this Unity3d scripting extention “Film-language”. Timur Bekmambetov, founder of Bazelevs said that this is a great idealistic idea. That after writing a scenario - commands for actors, they act according to their characters. So it is kind of program which creates reality, converts a language into audio-visual sequence.
Bazelevs used Unity3d game engine to play animations for actors (I think that they used Carnegie-Mellon University’s motion-capture free database of 2500+ animations, though I’m not sure). Bazelevs also used ABBYY Compreno linguistic engine, which could match words with part-of-speech tags, analyze phrases and even sentences. But of course it could have been better to use something like Google Translate for linguistic analysis instead of just Abbyy compreno (abbyy lingvo is their dictionary, they don’t make programs which translates complicated sentences, unlike google translate)…
Bazelevs innovations had some agreement with some comedy company, and thus used their Filmlanguage to make a simple program Chromosapiens: a child could control two given robots, and give them commands: what actions to do, and what to say (using unity3d text-to-speech plugin).
My dream is to have something which Bazelevs innovations did - inside Unity3d, so thus anyone can control any actor (I think LEGO/minecraft-like style should simplify the program and will make it more usable for professional programmers)…

I think that there should be a special programming language which could convert faw texts/books/scenarios into unity3d c#/javascript code. It can be done if there is access to c#/javascript linguistic analysis functions inside Unity3d (I wish Unity3d cooperate/collaborate with some linguistic company to analize phrases, so they can serve as input for 3d/animation/text-to-speech inside Unity3d)…

There are several programs with such functionality:
*spinnakers story machine (1980-es). In this program/game a child enters simple scenario using around 50 predefined words. Very nice program, i like it very much

  • wordseye - another online cool program - it can convert descriptive sentences to 3d world. Words act as “bricks”, and this wordseye program is like LEGO programming program…
  • widgit online - another nice program, which convert many English words and phrases into 2d pictures. That’s kind of pictorial dictionary, applied, which acts as visualization program for phrases and sentences. In unity3d words can become 3d meshes (words may use some additional numbers, identifiers etc - to choose correct matching 3d object/actor for each word)…
  • scribblenauts - is a game, in which player/child types a word, and mesh/character which matches it appears… then added words-actors interact: fight, etc etc…
    *bazelevs innovations filmlanguage - useful extension (asset, i hope it would be an asset one day in assets store) of unity3d scripting. Here is text-to-speech (though, russian, can be added English etc), and conversion of commands for actors into actions/animations…
    *there are other programs/games of this kind. Problem is that there is no standard for such input scenarios. If there is a standard, and all functions of such program are documented - it may become easier to and faster making games, or even just 2d/3d/animated content in unity3d… if there are thousands and thousans people using unity3d, similarly programming everyday, they will be given boost if they can convert automatically phrases to audio-visual sequences…

Scenarios (scripts of movies/plays/games) have a pre-defined form/style. If some program is made to analyze them automatically, to convert analized phrases/simple sentences into Unity3d commands - that would be great…

Also, I like “widgit online” for a reason: even prepositions are visualized there. That (prepositions) can probably be relatively easily converted to Unity3d-analized “phrases”…

Of course, development of such scripting (text-to-audiovisual sequences) extensions should be driven by necessities of majority of programmers.

It is true that even “draw a circle” requires additional information, like colors, redius, location (x,y,z, and probably time coordinate: t - that is defined by verbs & other time-related parts-of-speech )… but there are also rules of composition, rules of cinema/theatre, so some colors are best, some sizes are best, some positions/locations are also best… so they should just become default values.

When in Spinnakers Story-Machine similar sentence is entered: “an apple” - it just appears randomly on the screen.
If one enters there “a boy eats an apple”, then a boy (its mesh should be loaded automatically from a file with name “a boy”.fbx etc) randomly appears on the screen, and walks to an apple, and eats it…

Of course, using mouse in some cases is better than input using keyboard, typing whole words. Well, but noone discards natural, human languages, so we have books, though programming languages like c#/javascript may look more “precise”. Natural languages are in some sense more precise because they use less words, but produce a lot of visual images (for artists, scenario-writers - for games, movies, cartoons, plays etc)…

And I don’t really understand why each phrase like “actor1 walks3-to actor3” should return any value… of course, there are sequences (in time) of different events… say, in my game, when I click on a troll (and samurai is a player-actor), it shoukd first approach a troll, and only then - when in good distance - only then he should attack a troll etc.
Yes, “samurai attacks a troll” should “return” a distance between samurai and troll, and if it’s not enough for attack, samurai should come closer… Then when distance check returns that distance is OK for attack, and actual verb was “attack (a troll)” only then he attacks…

But well, all “2-legged” (bi-pedal) creatures in unity3d games are similar - they are like LEGO/minecraft creatures.
I mean to say, thus any text/scenario/phrase about humans csn be easily converted to unity3d functions.

if in some sentence some human is mentioned (“human walks-to point-1” etc) then some LEGO-style minifigure can be loaded automatically from resources folder (some random skins/colors/material/textures can be applied to make that look more random and different), placed on some reasonable location (not flying in the air, or undergroud: they all usually should appear on a ground/Earth)

Most human books (scenarios/scripts) are about humans. So if one day unity3d analizer plug-in can analyze phrases, to find in them nouns which refer to humans (bi-pedals), then they can be loaded automatically to the scene, apllied any of 2500+ animations (verbs) from carnegie-mellon-university mo-cap library etc…