Berkelium plugin: rendering interactive websites

Community Showcases
1

Hi all,

Berkelium is a wrapper around Google Chromium, and provides a way to render and interact with browser windows. I am creating a Unity plugin around this, so we can have also have interactive websites inside a Unity environment, in Windows (Mac users can use the htmlTexture plugin. I may sync my script API to match with that plugin once it is finished).

Note that I had to use native code for this plugin, so this is Unity Pro only for now, and it can’t be used inside a browser.

You can find my first attempt at this plugin at http://research.edm.uhasselt.be/~nve/OSMA/UnityBerkeliumPlugin.zip

The plugin still only has basic functionality, but I am actively working on it as we speak. This release is just proof that I am actually getting some results Wink

The zip-file contains Berkelium, the source code and vs2008 solution of the plugin, a Unity test project and a test release so people without Unity Pro can see it working.

Next things on my list is implementing support for scrolling, and caching the dirty rectangles to support partial updates using SetPixels (see reissgrant’s comment above). After that, I will have a look at binding keyboard input.

All contributions and comments are more than welcome of course!

And a screenshot:

Greetz,
JeDi

2

sweet,keep it up!

3

So basically you can not only load the websites, but interact with them as well?
e.g. I load website on a plane, run the app and it will allow me to navigate through the website which is on a plane, click links etc?

if its that then sweet :slight_smile:

4

Yep, that’s exactly what it does. You can render and interact with basically everything that Google Chrome can render. I have to investigate this a bit (I have no experience with neither Unity nor Berkelium), but I guess this can also be used as a GUI texture (using transparent websites). And it is just a texture, so you can use it on other objects than planes too (in-game terminals for example).

Mind that the current implementation is not efficient yet, and scolling doesn’t work properly. But those features are almost finished, and then it will be pretty efficient.

5

I updated the plugin (same download link). These are the changes:

  • Fly around with the right mouse button pressed (very simple fly script)
  • Interaction can be enabled/disabled per object
6

Awesome job. Think using a shader to do the BGRA to RGBA swap instead of doing it in the loop would save many cycles? Probably not many as you have to loop through the whole thing to convert it to a float anyways, but might help a little bit.

7

Use a float lookup table instead of calculating the conversion to float, it will save some time:

static const float lookup[]=
{
         0.0f,  0.00392157f,  0.00784314f,   0.0117647f,   0.0156863f,   0.0196078f,   0.0235294f,    0.027451f,
   0.0313726f,   0.0352941f,   0.0392157f,   0.0431373f,   0.0470588f,   0.0509804f,    0.054902f,   0.0588235f,
   0.0627451f,   0.0666667f,   0.0705882f,   0.0745098f,   0.0784314f,   0.0823529f,   0.0862745f,   0.0901961f,
   0.0941177f,   0.0980392f,    0.101961f,    0.105882f,    0.109804f,    0.113725f,    0.117647f,    0.121569f,
     0.12549f,    0.129412f,    0.133333f,    0.137255f,    0.141176f,    0.145098f,     0.14902f,    0.152941f,
    0.156863f,    0.160784f,    0.164706f,    0.168627f,    0.172549f,    0.176471f,    0.180392f,    0.184314f,
    0.188235f,    0.192157f,    0.196078f,         0.2f,    0.203922f,    0.207843f,    0.211765f,    0.215686f,
    0.219608f,    0.223529f,    0.227451f,    0.231373f,    0.235294f,    0.239216f,    0.243137f,    0.247059f,
     0.25098f,    0.254902f,    0.258824f,    0.262745f,    0.266667f,    0.270588f,     0.27451f,    0.278431f,
    0.282353f,    0.286275f,    0.290196f,    0.294118f,    0.298039f,    0.301961f,    0.305882f,    0.309804f,
    0.313726f,    0.317647f,    0.321569f,     0.32549f,    0.329412f,    0.333333f,    0.337255f,    0.341176f,
    0.345098f,     0.34902f,    0.352941f,    0.356863f,    0.360784f,    0.364706f,    0.368627f,    0.372549f,
    0.376471f,    0.380392f,    0.384314f,    0.388235f,    0.392157f,    0.396078f,         0.4f,    0.403922f,
    0.407843f,    0.411765f,    0.415686f,    0.419608f,    0.423529f,    0.427451f,    0.431373f,    0.435294f,
    0.439216f,    0.443137f,    0.447059f,     0.45098f,    0.454902f,    0.458824f,    0.462745f,    0.466667f,
    0.470588f,     0.47451f,    0.478431f,    0.482353f,    0.486275f,    0.490196f,    0.494118f,    0.498039f,
    0.501961f,    0.505882f,    0.509804f,    0.513726f,    0.517647f,    0.521569f,     0.52549f,    0.529412f,
    0.533333f,    0.537255f,    0.541176f,    0.545098f,     0.54902f,    0.552941f,    0.556863f,    0.560784f,
    0.564706f,    0.568627f,    0.572549f,    0.576471f,    0.580392f,    0.584314f,    0.588235f,    0.592157f,
    0.596078f,         0.6f,    0.603922f,    0.607843f,    0.611765f,    0.615686f,    0.619608f,    0.623529f,
    0.627451f,    0.631373f,    0.635294f,    0.639216f,    0.643137f,    0.647059f,     0.65098f,    0.654902f,
    0.658824f,    0.662745f,    0.666667f,    0.670588f,     0.67451f,    0.678431f,    0.682353f,    0.686275f,
    0.690196f,    0.694118f,    0.698039f,    0.701961f,    0.705882f,    0.709804f,    0.713726f,    0.717647f,
    0.721569f,     0.72549f,    0.729412f,    0.733333f,    0.737255f,    0.741176f,    0.745098f,     0.74902f,
    0.752941f,    0.756863f,    0.760784f,    0.764706f,    0.768628f,    0.772549f,    0.776471f,    0.780392f,
    0.784314f,    0.788235f,    0.792157f,    0.796079f,         0.8f,    0.803922f,    0.807843f,    0.811765f,
    0.815686f,    0.819608f,    0.823529f,    0.827451f,    0.831373f,    0.835294f,    0.839216f,    0.843137f,
    0.847059f,     0.85098f,    0.854902f,    0.858824f,    0.862745f,    0.866667f,    0.870588f,     0.87451f,
    0.878431f,    0.882353f,    0.886275f,    0.890196f,    0.894118f,    0.898039f,    0.901961f,    0.905882f,
    0.909804f,    0.913726f,    0.917647f,    0.921569f,     0.92549f,    0.929412f,    0.933333f,    0.937255f,
    0.941177f,    0.945098f,     0.94902f,    0.952941f,    0.956863f,    0.960784f,    0.964706f,    0.968628f,
    0.972549f,    0.976471f,    0.980392f,    0.984314f,    0.988235f,    0.992157f,    0.996078f,         1.0f,
};

To use it, call:

				 for (int x = 0; x < *pRectW; x++)
				{
					floatpixel[0] = lookup[bytepixel[2]];
					floatpixel[1] = lookup[bytepixel[1]];
					floatpixel[2] = lookup[bytepixel[0]];
					floatpixel[3] = 1.0f; // A
					floatpixel+=4;
					bytepixel+=4;
				}
8

Ok, in case you need it, here is my entire update rect function for updating the dirty rect and converting to RGBA using the lookup table (for Awesomium, but think it is similar enough to help):

void UpdateTextureInPluginRect(float* colors_rect)
{
	if (webCore!=NULL)
	{
		webCore->update();
		 if(webView->isDirty())
		 {
			webView->render(buffer_rect, W4, 4, &rect);
			
			*pDirty = true;
			*pRectX = rect.x;
			*pRectY = rect.y;
			*pRectW = rect.width;
			*pRectH = rect.height;
			*pRectBool = rect.isEmpty();

			floatpixel = colors_rect;
				 
			for (int y = 0; y < *pRectH; y++)
			{
				 if(*pRectY < 2 )
				 {
					 *pRectOffset = (((( *pRectY + y ) * WIDTH) + *pRectX)*4);
					 
				 }
				 else
				 {
					 *pRectOffset = (((((*pRectY - 1) + y + 1) * WIDTH) + *pRectX)*4);
				 }
				bytepixel = buffer_rect + *pRectOffset;

				
				 for (int x = 0; x < *pRectW; x++)
				{
					floatpixel[0] = lookup[bytepixel[2]];
					floatpixel[1] = lookup[bytepixel[1]];
					floatpixel[2] = lookup[bytepixel[0]];
					floatpixel[3] = 1.0f; // A
					floatpixel+=4;
					bytepixel+=4;
				}
			}
		} else {*pDirty = false;}
	}
}
9

Here is my C# keyboard injection code (modified code from solmyr) This mimics a native windows key event:

void keyHandler()
{
	if (flashHit ==true) // does Flash window have focus?
	{
		if(Input.inputString!="")
		{
			c = Input.inputString[0];
			bs = Input.inputString;
			injectKeyboard(0x0102, c);
			if (bs == "\b")
			{ 
				injectKeyboard(0x0100, (char)0x08);  // backspace
			}
		}
	}
}
10

very cool stuff. On a similar note, I am still wanting to see the ability to do desktop sharing or application sharing. Like teleplace. Basically it would be nice to have an open office plugin for unity.

I love the concept of using interactive textures. This goes so far beyond just websites. I wonder if I can make something in silverlight and render it via this plugin on a object.

11

Probably not. While plugins should be able to be supported in the future, right now only Flash is supported by Berkelium.

12

Is there a plan to hand in-website events back to unity handling code?
Making it possible to use html to build UI sounds interesting (at least for me and in theory).

Cool work!

13

I think I found a new ad system…

MWU HA HA HA HAA.

14

Yes, that is the intent, communication through javascript to Flash and back to Unity. I have a working prototype using Awesomium that does this, using Flash as my main menu and in game HUD system with transparency; but unfortunately Awesomium requires a license. This however, does not!

15

@reissgrant Thanks for the info. I will have a look at using the tables, but first I have to implement using dirty rectangles in Unity, which will boost performance a whole lot more. Because I don’t have any experience with .Net though, I am having trouble returning an array of structs from my C code back to .NET. I was hoping to do something like this in C#:

struct Rect
{
  int top, left;
  int width, height;
}

[DllImport ("UnityBerkeliumPlugin")]
private static extern Rect[] Berkelium_Window_getUpdates(int windowID);

but I have no idea how to bind that to C code. I am reading up on C# and marshalling, but if anyone can give me some hints, that would be greatly apprectiated.
Note that I have a struct in C++ which has the same members (4 ints), and I hold a std::vector of those in my class, so I just have to get that vector into C# somehow…

@TobiasS As far as I know, callbacks aren’t implemented in Berkelium yet, but they are definitely on their TODO list. Awesomium does have them, but as reissgrant said, Awesomium isn’t free and/or open. For GUIs this is a necessary part of the plugin, but it could be useful for a whole bunch of other things too.
You can however call javascript functions (also with result) from Berkelium, so you can poll values for now.

16

Unfortunately my knowledge in C# is limited too, so I resigned to returning a simple int for each one of my rect values (returnRectX(), returnRectY(), returnRectHeight() etc.) It’s not pretty, but it works!

17

Yes, I will do it like that for now too.

On another note, I am trying to figure out keyboard injection. Could you elaborate on your code a couple of posts back please? Thanks!

18 
injectKeyboard(0x0102, c);

In the above code, 0x0102 is the WM_CHAR event (which is passed as an int) and c is the keycode (which is passed as a char).

We’ll need to pass the event and the vk_code to Berkelium for the keyEvent Function:

virtual void Berkelium::Window::keyEvent(
	bool	pressed,
	int     mods, // not sure yet
	int     vk_code, <--- put our "c" here (cast it to int)
	int     scancode <--- put our "int" event here   
   )
-------------------------------------
Parameters:
       pressed  <---  if true indicates the key was pressed, if false indicates it was released
       mods  <--- a modifier code created by a logical or of KeyModifiers
       vk_code   <--- the virtual key code received from the OS
       scancode   <--- the original scancode that generated the event

Let me know if you need more :smile:

19

Don’t think I’ve mentioned this yet, but great work getting this working! :smile:

I downloaded your code to help and noticed a few things so far (All in the C# script):

        m_Texture = new Texture2D (width, height, TextureFormat.ARGB32, false);

Should be TextureFormat.RGB24 instead, slight performance boost :wink:

Add this to counteract the reverse orientation of the texture (had the same issue in Awesomium):

renderer.material.mainTextureScale = new Vector2 (1,-1);

Also, you might want to add at the top, to be safe:

using System.Collections;

*EDIT TextureScale modified to display correctly(1,-1)

20

I was aiming to get the keyboard portion done tonight but am stuck now, because I’m getting a crash related to the mousemove() function. JeDi, are you getting the same crash or is it just me?