I’m trying to create a Unity data visualization that takes a KML/KMZ file, parses to an XmlDocument (or some other DOM that can be searched / manipulated), and then uses the XML to procedurally generate meshes, place objects, etc.
The actual parsing of the KML file I can figure out, but I’m running into an issue trying to extract KML from KMZ. The issue is that KMZ often contains multiple files in addition to the KML, and the only solutions I can find to extract a single file from a zipped archive of multiple files require things like starting a Java VM or shell instance. Obviously this is very brittle; it certainly wouldn’t work on mobile, and I’m doubtful such a solution would even work in a webplayer.
Does anyone know of a creative way to do this that would be platform independent? Maybe extract into a data stream and then regex (<xml.*/xml>) or look for file end markers? Are there any c# packages that might help?
Thanks much!
( BTW, the file I’m using to test this is at http://adn.agi.com/SatelliteDatabase/KmlNetworkLink.aspx - it returns a kmz)
Unless I’m mistaken, KMZ looks like it’s just a zip file of KML (a form of XML). Currently on iOS I am using the SharpZipLib libraries and they provide a ton of functionality as well as modifiable source. They work on a very minimal profile (.NET Subset + Byte Code Stripping) on iOS, and can be tuned to remove uneeded formats, such as tarballs and GZip to further shrink the size. You can either track down individual known entries with a downloaded file, or stream the data from a network and unzip directly from the network connection without ever needing the entire zip file (good for fast response, low memory from a network).
For XML we use the SmallXMLParser suggested here: http://unity3d.com/support/documentation/Manual/Reducing%20File%20size.html
And it’s also very lightweight and customizable. It gives you a parser with an empty structure for Handling the XMLs, this means you can process the XMLs and build your own customized reader without having to worry about handling all the different tags and formats for a much lighter weight than System.XML. The only thing we changed in the SmallXMLParser was to have it use the Flyweight Pattern, as it’s prone to producing a lot of Garbage for the GC. But that’s only really an issue reading MBs of XMLs on a low end iOS device.
Edit: Opening up your Sample KML, it can be opened by SharpZipLib, and the only thing unreadable by Unity is the .gif, the KML file is just a simple XML, though you’d need to have something to handle the Tags to build a proper Object type in memory from the KML. And the JPG and PNG can be loaded by Unity at Runtime with Texture2D.LoadImage()
Ntero,
Thanks much for your suggestions.
I’ve heard people mention SharpZipLib before, but I need a solution that is platform independent - something that works in webplayers and mobile. As I understand it, SharpZipLib extracts the files to disk, which you can’t do in the webplayer, so unless you know of some way to stream the output to memory and then extract a single file it won’t work for me. If I have to piece together separate solutions for different platforms I will, but for now I would greatly prefer a single solution.
As for the actual parsing, our data sets are massive - our largest (unzipped) KML is 277M. I’m using the very beefy System.Xml library with XmlDocuments and xPath to search the tree. It works well for now, though as we go on I’ll probably get fancier in the quest to save memory.
It doesn’t have to extract the files to disk. You can hook it up to a NetworkStream, and just store the results in the MemoryStream (byte[ ]. Have a look at the ZipInputStream class. Also, if you strip it down or compile with the .NET compact framework defines active, you can not have any Cryptography and therefore compile it using Byte code stripping and .NET subset on iOS.