Why? Because I can I guess ... The good (skip to 2:15 for the actual demo, first part is small overview of code changes). The sound actually does work, it's just that I did not record any sound in the video.
https://www.youtube.com/watch?v=kx3RPqmhJrE
- vlcsnap-2017-03-16-14h16m26s498.jpg (204.22 KiB) Viewed 44634 times
Bit of background: The proof of concept basically uses the Chrome code base (CEF3) underneath. As per the Chrome processing model, the main application (in this case, Minetest) is separate from the browser process so browser crashes will not crash the main application. Awesomium is a wrapper around the Chromium Embedded Framework (CEF3) and takes care of managing the browser sub-processes and the complexities of passing messages, handling browser events in the main program and providing access to screen buffers for off-screen rendering.
I managed to get the off-screen rendering to work with formspecs by providing a new browser formspec element that, when parsed, will open a so called Awesomium web view. The web view provides it's buffer for copying into other buffers. All that remained was to create a GUI image and copy the web view buffer into the image's texture. This will of course work for arbitrary textures, so the next logical step is to copy the web view buffers to node textures, allowing players to place YouTube video blocks, HTML documentation blocks, etc.
Awesome, right? Well, now for the bad .....
CEF3 is an absolute beast. When I started I found that the Chromium source consisted of 27000 files, it took me hours and hours to compile. There are binary packages for download of 200 - 300 MB. The core libcef.so library is 60+ MB (uncompressed). It is under continuous development so there are no real stable branches. One version works, the next one crashes for no apparent reason. There are samples online showcasing off-screen rendering e.g. the Chrome browser in 3D applications but these don't take into consideration on how to embed CEF3 in an existing large code base such as Minetest. The CEF samples are no real help either. It looks like I would need to manually write code to share the browser buffers with the main Minetest program. Fun! -_-
TL;DR, CEF3 is a pain to use. I've been hacking around for about 2 weeks now, and would probably need another 2 weeks to get any off-screen rendering to work.
In comes Awesomium. It takes care of all the nasty things one does not want to take care of. I was actually able to put the above proof of concept together in 2 days. It's awesome! But: *Awesomium is closed source*... It is free to use in Open Source not for profit projects, but still, closed source. With 65MB, the libawesomium shared library is still large as it contains libcef of course.
Since the browser runs in a separate process, all sound handling (e.g. YouTube videos, audio played in the browser) is done by that process. That is great for formspecs, but if you'd have a node you would expect the volume to lower the further away you are from the node. To redirect the audio processing somehow, no clue yet how that could be done.
There are alternatives such as embedding Gecko. That will introduce a dependency on the Gecko code base and the Gecko engine would then have to run inside the Minetest process unless I would custom roll a sub-process model for Gecko. Also, the Gecko API doesn't look that simple so lot's of coding would probably be required to get that to work.
So that's the bad... But then of course, what do think about this kind of functionality? Besides this looking cool (and even cooler when I implement casting pages to nodes), is it worth pursuing. I can come up with a few use cases, and there is the possibility of abuse (e.g. a web browser node with configurable URL could link to inappropriate content) beyond what is already possible (e.g. insulting signs).
Also, what about the approach I took? Are these kinds of code base changes worth it? It seems to be minor so far as Awesomium provides a singleton, but still this adds a dependency on (yet) another (closed source) library. Would it be better to keep this outside of the main Minetest code base, package this as a mod (with Lua -> native bridge)? But in that case, is there currently a way to access to the texture buffers? As in, how to go from Awesomium/ CEF3 -> Lua -> formspec/ node ITexture buffers?
I'm really interested in what you think: Any feedback, whether it's good or bad, much appreciated!! :)
And then there are a bunch of TODO's I will still need to look into:
- More robust code, e.g. it crashes when I exit a game and I need to put some safety mechanisms in place when the browser sub-process crashes
- Passing events (e.g. mouse clicks) to the embedded browser
- Implement copy to node texture buffer for web browser nodes
- See if I can modify the Lua browser mod to allow for configurable URLs
- The IGUIImage is not properly created, it is not registered in the UI element children
- Make a fork of Minetest and upload my changes to the fork
