Best Image Format?

So what I have learned

Break up the rgba into individual arrays
Look for matching values
Tag coordinates with a cheap costing byte to represent a loop of that value, and that applies uniformly to all arrays across all colour channels on rebuild. So you reduce colour repetition loop, and reference of location repeat values per r g b and a , so I don’t need to write coordinate 1 2 3 4 = colour white. I just have to say coordinate 1 repeats 3 times. Oh and the value of that white on the Red was used here in this Blue channel Etc. etc.

But I have 10 mins left on the vid. Just want to write this one down while it’s hot in my head ^

yeah I see. I see. So I see indeed. I may be able to achieve something pretty good: but I am not trying to break a world record here. But maybe if I can get the code in a state where I can progress. I guess I could drop the MS compressor and persist on the :

binary conversion, check for loop assign negative binary value for loopers, convert to byte and see the results. - if at this point the file is comfortably small, I will go ahead and try to rebuild the correct binary value before converting back into usable values to rebuild that image.

How exactly did you arrive at that conclusion ^^. It’s actually more the other way round. TIFF is an extremly flexible format which can even work as a container format to include jpeg data. Also TIFF supports tons of “tags” and specifications and different compression algorithms. You mentioned that TIFF also supports ZIP compression. You do realise that the ZIP format is the deflate algorithm that is used in PNG. Deflate wasn’t invented for PNG as it was invented by Phil Katz long before.

To quote wikipedia:

[quote]
TIFF is a complex format, defining many tags of which typically only a few are used in each file. This led to implementations supporting many varying subsets of the format, a situation that gave rise to the joke that TIFF stands for Thousands of Incompatible File Formats
[/quote]. TIFF supports a hell lot of different pixel bit formats or color spaces. PNG on the other hand is restricted to RGB images.

PNG actually is quite similar to TIFF as it also consists of named chunks. Though the standard is much simpler and doesn’t have that many variations

Note that PNG wasn’t meant to replace GIF. Since GIF was one of the most popular formats back then, they wanted a format that doesn’t have patent issues like GIF. So the intention wasn’t to replace GIF but to provide an alternative. GIF can do things PNG can’t (animations for example) and PNG can do things GIF can’t (proper color images with more than 256 colors).

Note that PNG is actually simple enough that they explained almost all details on the wikipedia page. TIFF is that complicated that the wiki page is only contains rough descriptions of some of the various features and a list of supported compression algorithms.

Deflate is one of the go-to standards when we talk about compression nowadays. It’s used literally everywhere. That’s why PNG support can be seen almost everywhere. Gzip and deflate are the most widely supported transfer compression algorithms for http (though brotli will probably take over at some point, though it’s also just a variant of LZ77).

Don’t get me wrong, I really like older formats. I’ve written this GIF loader, just for fun. I like to dive into file formats and what neat solutions they came up with in the past. (I remember loading dune2 paks some 25 years ago). Though I stay away from TIFF if possible :). Yes, Tiff allows higher quality images, that’s why it was and is widely used in the professional industry. Also a lot of custom, industry specific extensions has been developed for TIFF, so they continue using it. Though even formats like GIF or PNG allow extensions pretty much the same way, but almost nobody does it because they usually want maximum compatibility. When I was writing my gif loader I was thinking about using gif as a save game format. So we could store a screenshot at the save point as the actual image and use extension blocks to store the actual save data :slight_smile: That way the image could still be opened in any image program but still works as a savegame file ^^. Pretty pointless feature, but hey,

3 Likes
for (int m = 0; m < COMPACT_LIST.Count; m++)
                {
                    for (int F = 0; F < FRAME_ID[m].Length; F++)
                    {
                        BYTE.Add(Convert.ToByte(FRAME_ID[m][F]));
                    }
                    BYTE.Add(Convert.ToByte(COMPACT_LIST[m].width));
                    BYTE.Add(Convert.ToByte(COMPACT_LIST[m].height));
                    for (int a = 0; a < COL[m].Count; a++)
                    {
                        BYTE.Add(Convert.ToByte(COL[m][a].r));
                        BYTE.Add(Convert.ToByte(COL[m][a].g));
                        BYTE.Add(Convert.ToByte(COL[m][a].b));
                        BYTE.Add(Convert.ToByte(COL[m][a].a));
                        var C = COL_REFERENCE[m][a];
                        BYTE.Add(Convert.ToByte('i'));
                        bool SkipValue = false;
                        float NEXTVALUE = -1;
                        for (int v = 0; v < COL_COORDINATES[m][C].Count; v++)
                        {
                            var B = COL_COORDINATES[m][C][v].ToString().ToCharArray();

                            if (v != 0)
                            {
                                if (v + 1 < COL_COORDINATES.Count)
                                {
                                    if (COL_COORDINATES[m][C][v + 1] == COL_COORDINATES[m][C][v])
                                        BYTE.Add(Convert.ToByte('-'));
                                    else
                                    {
                                        if (COL_COORDINATES[m][C][v] > 99)
                                        {
                                            for (int BA = 0; BA < B.Length; BA++)
                                            {
                                                float VALUE = -1;
                                                float.TryParse(B[BA].ToString(), out VALUE);
                                                BYTE.Add(Convert.ToByte(B[BA]));
                                                BYTE.Add(Convert.ToByte("+"));
                                            }
                                        }
                                        else
                                            BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v]));
                                    }
                                }
                            }
                            else
                            {
                                for (int BA = 0; BA < B.Length; BA++)
                                {
                                    float VALUE = -1;
                                    float.TryParse(B[BA].ToString(), out VALUE);
                                    BYTE.Add(Convert.ToByte(B[BA]));
                                }
                            }
                        }
                    }
                }

Not had time to read your reply just

I misunderstood everything i previously wrote, please check algorithm for me and scrutinize.

Important bit

                        for (int v = 0; v < COL_COORDINATES[m][C].Count; v++)
                        {
                            var B = COL_COORDINATES[m][C][v].ToString().ToCharArray();
                            if (v != 0)
                            {
                                if (v + 1 < COL_COORDINATES.Count)
                                {
                                    if (COL_COORDINATES[m][C][v + 1] == COL_COORDINATES[m][C][v])
                                        BYTE.Add(Convert.ToByte('-'));
                                    else
                                    {
                                        if (COL_COORDINATES[m][C][v] > 99)
                                        {
                                            for (int BA = 0; BA < B.Length; BA++)
                                            {
                                                float VALUE = -1;
                                                float.TryParse(B[BA].ToString(), out VALUE);
                                                BYTE.Add(Convert.ToByte(B[BA]));
                                                BYTE.Add(Convert.ToByte("+"));
                                            }
                                        }
                                        else
                                            BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v]));
                                    }
                                }
                            }
                            else
                            {
                                for (int BA = 0; BA < B.Length; BA++)
                                {
                                    float VALUE = -1;
                                    float.TryParse(B[BA].ToString(), out VALUE);
                                    BYTE.Add(Convert.ToByte(B[BA]));
                                }
                            }

i note a greater than 100 value. and if our next byte is is our byte, then we basically put a ‘-’ single byte. It maybe checks out with image content, number of background space, number of colours etc. Its midnight. maybe i am crazy right now.

Maybe some junk code in there aswell. Sorry

What does that have to do with image size?

Anyway, for development it’s not just about how many bytes the finished file takes. If you’re using version control (and any software developer worth their salt should be) then you’re not just storing the finsl image, you’re storing every version of it.

So if you’ve got 55 frames and you tweak one, now you’re storing 56 frames. But if you bundle all of those together in one file and then twrak one, instead of 56 you’ve got 110. And for your stated use case of changing colours, every time you change a colour in the files, it’s another 55 images’ worth of space eaten by your version history.

For a 2D game that’ll get huge, fast.

Compare that to a shader based approach, as you mentioned earlier. Only one colour value changes. Basically no load time impact. If you want to use the same images with different colours you can with just a Material - no new copy of the files needed. And the shader itself is probsbly trivially cheap.

Playing with reading / writing data is fun, and it’s great to see people willing to dig into it. Most people treat data as black magic and assume that if there isn’t a library it can’t be done. So carry on with that for fun if you’d like. But when you get back to your keyed colour thing… give the shader a go. :wink:

1 Like

Sorry here is a cleaner one Junk Removed ( i think )

Edit even cleaner ;

                for (int m = 0; m < COMPACT_LIST.Count; m++)
                {
                    for (int F = 0; F < FRAME_ID[m].Length; F++)
                    {
                        BYTE.Add(Convert.ToByte(FRAME_ID[m][F]));
                    }
                    BYTE.Add(Convert.ToByte(COMPACT_LIST[m].width));
                    BYTE.Add(Convert.ToByte(COMPACT_LIST[m].height));
                    for (int a = 0; a < COL[m].Count; a++)
                    {
                        BYTE.Add(Convert.ToByte(COL[m][a].r));
                        BYTE.Add(Convert.ToByte(COL[m][a].g));
                        BYTE.Add(Convert.ToByte(COL[m][a].b));
                        BYTE.Add(Convert.ToByte(COL[m][a].a));
                        var C = COL_REFERENCE[m][a];
                        BYTE.Add(Convert.ToByte('i'));
                        for (int v = 0; v < COL_COORDINATES[m][C].Count; v++)
                        {
                            var B = COL_COORDINATES[m][C][v].ToString().ToCharArray();
                            if (v != 0)
                            {
                                if (v + 1 < COL_COORDINATES.Count)
                                {
                                    if (COL_COORDINATES[m][C][v + 1] == COL_COORDINATES[m][C][v])
                                        BYTE.Add(Convert.ToByte('-'));
                                    else
                                    {
                                        if (COL_COORDINATES[m][C][v] > 99)
                                        {
                                            for (int BA = 0; BA < B.Length; BA++)
                                            {
                                                BYTE.Add(Convert.ToByte(B[BA]));
                                                BYTE.Add(Convert.ToByte("+"));
                                            }
                                        }
                                        else
                                            BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v]));
                                    }
                                }
                            }
                            else
                            {
                                if (COL_COORDINATES[m][C][v] > 99)
                                {
                                    for (int BA = 0; BA < B.Length; BA++)
                                    {
                                        BYTE.Add(Convert.ToByte(B[BA]));
                                        BYTE.Add(Convert.ToByte("+"));
                                    }
                                }
                                else
                                    BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v]));
                            }
                        }
                    }
                }

Looks like 0 + 1 won’t be able to turn back into a value it actually should be if it does not = Length.

Clean algorithm checks out byte increase, Is cool. If it can be read and produce the image il write it tomorrow. Maybe i have made ultra-noob mistake that i am unable to see and i have lost alot of byte data that is irreversibly unobtainable. Maybe there are too many + in the hundred beat, and not enough + (or even existent) for a tens beat.

I hope i am progressing. Anyway i am self centered and only focussing on myself. I’l read ya’ll stuff but i probably sleep first :slight_smile:

If you want to test with my image here it is
8494982--1130627--A01.png
But i think it looks all spick and span bar a few minor individual byte handlings for the tens on values > 1-9 and then values > 99 for a different byte symbol ‘-’ & ‘+’ so we know the next three are 100, or the next two are 10’s

this is what i did with compact list

                for (int m = 0; m < COMPACT_LIST.Count; m++)
                {
                    COL_COORDINATES.Add(new List<List<int>>());
                    COL.Add(new List<Color>());
                    COL_REFERENCE.Add(new List<int>());

                    var TA = COMPACT_LIST[m].GetPixels();
                    for (int t = 0; t < TA.Length; t++)
                    {
                        if (!COL[COL.Count - 1].Contains(TA[t]))
                        {
                            COL_COORDINATES[COL_COORDINATES.Count - 1].Add(new List<int>());
                            if (!COL_COORDINATES[COL_COORDINATES.Count - 1][COL_COORDINATES[COL_COORDINATES.Count - 1].Count - 1].Contains(t))
                            {
                                COL_COORDINATES[COL_COORDINATES.Count - 1][COL_COORDINATES[COL_COORDINATES.Count - 1].Count - 1].Add(t);
                                COL_REFERENCE[COL_REFERENCE.Count - 1].Add(COL_COORDINATES[COL_COORDINATES.Count - 1].Count - 1);
                            }
                                COL[COL.Count - 1].Add(TA[t]);
                        }
                        else
                        {
                            if (!COL_COORDINATES[COL_COORDINATES.Count - 1][COL_COORDINATES[COL_COORDINATES.Count - 1].Count - 1].Contains(t))
                            {
                                COL_COORDINATES[COL_COORDINATES.Count - 1][COL[COL.Count - 1].IndexOf(TA[t])].Add(t);
                            }
                        }
                    }
                }

But this is all BS if i cant reassmble the thing properly

I see what’s missing
All the data will be missing

line 24 Needs to reference col MCV it only checks col on v + 1

noob mistake

I just divide > 99 by 10; and put a ‘-’ symbol so i know that it was divided by 10;

                        for (int v = 0; v < COL_COORDINATES[m][C].Count; v++)
                        {
                            var B = COL_COORDINATES[m][C][v].ToString().ToCharArray();

                            if (v != COL_COORDINATES[m][C].Count - 1)
                            {
                                if (v + 1 < COL_COORDINATES[m][C].Count)
                                {
                                    if (COL_COORDINATES[m][C][v + 1] == COL_COORDINATES[m][C][v])
                                        BYTE.Add(Convert.ToByte('+'));
                                    else
                                    {
                                        if (COL_COORDINATES[m][C][v] > 99)
                                        {
                                            BYTE.Add(Convert.ToByte('-'));
                                            BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v] / 10));
                                        }
                                        else
                                            BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v]));
                                    }
                                }
                            }
                            else
                            {
                                if (COL_COORDINATES[m][C][v] > 99)
                                {
                                    BYTE.Add(Convert.ToByte('-'));
                                    BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v] / 10));
                          
                                }
                                else
                                    BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v]));
                            }
                        }
                    }

After repair this here new Byte divider is the only compression occurring. Result is 1737 bytes. So we are back up again. i was too excited. We are back up again but i think i can almost live with it.
I am sorry for being so spammy. I just get a little bit over-excited, i should better control my urges.

Yes I believe I will live with it. and i will speak no more of it. 6x larger than a PNG i am confident it will re-assemble. No such more words i will speak of the matter. I will append that i cannot with this algorithm deal with a file larger than 999 pixels. Unless i give another ‘v’ to indicate i will divide by 1000, 10000, 100000 etc. WE are entering the realms of Power and Weakness (or power negative as they say)

if (COL_COORDINATES[m][C][v] > 99)
{
   if (COL_COORDINATES[m][C][v] > 999) // Remember this is an Index belonging to a COL
   {
         BYTE.Add(Convert.ToByte('@'));
         BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v] / 100));
   }
   else
   {
         BYTE.Add(Convert.ToByte('-'));
         BYTE.Add(Convert.ToByte(COL_COORDINATES[m][C][v] / 10));
   }
}

Nah, of course I took some liberty with the actual comparison. Technically, you are right, and in fact, there is no difference in complexity. I was just trying to handwave the “apparent” magnitude in difficulty when approaching the two:

  1. PNG is more obfuscated than TIFF, especially when trying to naively make any sense of it,
  2. TIFF is better documented, by all accounts,
  3. TIFF is much more straightforward as a container, PNG is more low-level and tightly-packed, with almost no observable structure,
  4. Raw TIFF is plain and simple, things only get complicated when you start looking at compressed binaries, but even then you can easily discern what tool was used, simply because the tool would leave marks, i.e. header. You can’t say the same for vanilla PNG, except maybe with palletized images which should be RLE with deflate.

This is all just how a person from a naive “hacking” position would evaluate the differences between the two. And frankly, I would always rather try to decode TIFF than PNG. It’s simply by nature of how these things are designed, from the looks of it. And I used to reverse engineer MP3 in late 90’s. It was actually harder to reverse engineer PNG than any other image format back in early 2000’s, and there were plenty of reasons behind that, but most of it is just incredible opacity and lack of internet access.

I can even semi-prove this position because even Photoshop had a few issues with how they treated PNG transparency in some edge-cases (through Save for Web module, which is highly specialized in JPEG/PNG codecs). 15 years ago or so, I remember having to either use an external plug-in (I think it was called FIXPNG or sth like that) to save PNG with a properly encoded alpha channel or to avoid the format altogether. For a format that was as ubiquitous, it’s kind of weird that even Adobe’s engineers had a hard time doing it properly. This is one of numerous instances that made me describe it the way I did.

So yes, PNG is more complex than TIFF, it’s just you and I are using the word complex to describe different things.

You’re thinking in engineering terms WHEN you now exactly what to look for, and PNG is pretty much an open book these days, and I’m thinking in terms of practical reverse engineering, and PNG was never easy to work with nor it was an open book. Of course when you really go in headlong, deflate = zip, yes.

Though I never said it was invented for any of the two, a zip is a zip, you probably misread something.
I used ARJ back in my time, DEFLATE and PKZIP came out in mid 90’s. I kind of know these things by heart.

Exactly, this is both pro and contra. You can at least tell how many containers TIFF has, and you can even tell what to expect from them, but yes this makes one hell of a combinatorial space, but that was the point of it, it’s TAGGED for a reason. PNG on the other hand, has no obligations toward you as a “hacker”. No tagging. You can only observe a stressfully tight stream of complete garbage, and unless you have ANY inside information about its intermediate contents (how on Earth you can tell which filters were applied if you don’t know how the encoder operates?), you’re doomed.

Yes it was (edit: I mean listen to your own words, they had to supersize what GIF has already achieved, they’ve stumbled into a new business model territory, and tried to cease this opportunity). I mentioned the apparent obsolescence already, but the real issue was that CompuServe was sued over LZW and development of GIF was stuck in a legal hell.

Here’s a snippet from WP and I’ve read a much more thorough overview of that era but sadly can’t remember where to look for it

I’ve bolded that word to highlight the apparent urgency. They had a deal with Netscape you see, and this was in the midst of the great browser war (or the start of it). So all of this hassle is mostly man-made, and had almost nothing to do with just improving the technology. This also partly explains a) the deliberate obfuscation of the PNG format, b) the passive-aggressive fight for dominance with JPEG, c) the subsequent and very silent non-adoption by Internet Explorer (which I remember well), and d) the first such successful monopoly of MP3 not that much further down the history line. I also personally know a guy who fully reverse engineered WMA and then got a job working for Nero (https://en.wikipedia.org/wiki/Nero_AAC_Codec , name redacted fully credited under History). Oh the details are juicy. I am still friends with his neighbor (he obviously moved to Germany and never looked back).

It was all part of the now famous “format war”.

I think you’re all used to having access to documentation, schools, mentors and proper culture for this sort of stuff. We’re still banned from Paypal in Serbia, obviously because of politics but also because we have some great hackers, and hacking is what I’m thinking of when I describe something as “easy” or “hard”. I learned all kinds of shit from having access to weird assemblies and demo scene files and whatnot, I have no school, or computer-related specialization and I have never read a book on computers except when I was 7, I remember my father brought me some introduction to BASIC and that was it. Everything I learned was from hands-on experience, my own mistakes, plenty of source code that was circulating in the underground, and also I’m a sponge when it comes to reading online. In the late 90’s I was already reverse-engineering the Adobe files, specifically by extracting the JPEG thumbnails as I would hijack Adobe Bridge to generate the thumbnails for me, that shit was easier than having to produce them in the first place, so I could automate entire image databases without doing much, this was before I had broadband internet; in the early 2000’s I used to earn for a living for a while by returning missing HDD partitions equipped only with a disk editor (it was literally one bad byte in FAT32 MBR /or was it FAT16 I don’t remember/ which I was able to pinpoint armed with some patience and series of good hunches; it was a wave of some malware, I still don’t know what caused it, but those photographers and designers paid well for my “wizardry”). This was my side job while I was producing 3 fully-featured luxuriously designed monthly-issued magazines in parallel. Crazy times.

Finally, I’m really intimate with TIFFs because I used to be a print professional myself, so maybe I’m a little biased lol. (And now you can perhaps understand my deeply rooted hatred of anything JPEG.)

As I said, you didn’t quite grasp what I tried to say, but nvm.
TIFF/ZIP was the latest addition, btw. Someone figured out it made sense in the mid 2000’s. Now it’s almost a standard. LZW, being a text compression algorithm isn’t that good for true color images, and in fact, does very little compared to general compression algorithms. But it was widely supported before, so it stuck. Who knows how many TIFF encoders exist in the wild, that thing is a jungle. By this metric, again, you’re right, PNG is much more straightforward. One format, one set of rules, sure.

Though I wish QOI would gain more traction. Such a fun format. Idk, we ought to move on to something that’s faster and more transparent (and open). FLAC did wonders for the audio industry.

Oh, btw, deflate is only the small part of the general PNG format. It comes basically after everything to squeeze just a little extra and obfuscate things further.

:smile: good advice, luckily I don’t use version control

dropping the single unit of the file to 6X a png I am over the moon with it

I opened tiff and looked at the data and it is rammed full of data. In comparison to png

the theory goes; well now if the files are all combined and then they are all linearised byte savings will be made on the compression of all files by index of file, index of colour which colour grid index references could potentially say all empty space for all files can be considered one byte per reference but I must admit proud as I am with such progress I will likely fall back on the open png filled directory in the end. I just know it.

And btw dune 2000 was one of my fav games. the entire Westwood franchize was my fav games. Didn’t buy the new one though. I modded the hell out of the originals added a snow pallets to dune all kinds of nerd behaviour. But I wasn’t a smart person back then. But I practiced a lot with graphics and copy and paste techniques. And I recognize no developer has attempted such things since. Not even in China. So I’m happy to be able to understand a fraction of the work they originally put into it. Of that ea was unable to replicate.

1 Like

I don’t really want to argue here. However are you sure you have even looked into the png format yourself?

Pretty much all of those points are upside down ^^.

Even though TIFF is much more complex and has much more features, the RFC of TIFF only has 23 pages and only explains some details. The PNG RFC has about 100 pages and explain pretty much every detail with clear examples. and pseudo code for a lot of things. TIFF uses integer tag IDs which are not really human readable or help identifying the structure. PNG on the other hand uses 4 byte text tags for each chunk which all represent a 4 character text which can be easily identified.

IHDR - header
PLTE - palette
IDAT - image data chunk
IEND - end marker

other common chunks
cHRM - chromaticity information
gAMA - gamma correction information
pHYs - physical dimensions (PPI)

You seem to only focus on trivial non compressed TIFF files. Sure they are simpler, but are only a tiny subset of what makes up the format. Writing a TIFF loader that can only load uncompressed files, which are always little endian, always use a certain bit depth, etc… is relatively easy to implement but would break down when you hit any other variation. The overall file structure of TIFF and PNG is exactly the same. They consists of several linked chunks. As a parser you don’t have to understand all of them. PNG has the 4 critical chunks that every PNG file must contain, I listed them above. Any other chunks are optional.

I think you somehow got scared by the video you linked above (I’ve seen this video in the past as well ^^). What was explained in the video was already everything there is to the PNG format. Here’s a blog article explaining how to load the image data including the 4 filter variants. All with sample code snippets in type script. So pulling everything together would probably give you a complete png loader :wink: Sure, they used zlib for the deflate, as it’s the most complex part, though since it’s standardized that’s not really an issue.

Note that we talk about loading the images, not creating them. Creating png images is quite a bit more complex if you want to do everything from scratch yourself. Finding the proper / best filter and implementing the inflate algorithm is a bit harder than decoding it.

Anyways, it’s already too late and I need some sleep :wink: As I said, I don’t wanna start a fight here. Everybody has their own reason(s) to like / dislike certain formats, companies, products, …

Here’s more good advice: start.

2 Likes

Cmon no fights.

I get your point, but you’re again quoting the RFC. Have you actually looked inside the file?

I am really talking about: F3 the file, try to make sense of what you see, if you do, cool, if you don’t you switch to hex view, match some patterns, find clues on headers, find locations of interest. Find pointers to locations of interest, demystify header, be on the lookout for 4-letter codes… Some algorithms will leave specific residues (as per some specification) which can provide a hint on what was ran on it, many compression algorithms have their own header, it’s rarely just pure data. There are also tags, frames, chunks, all kinds of separators. The structure itself reveals a lot about the process that went into it.

I’m talking about Sherlocking.

TIFF is intended for industrial interchange (and tool access), PNG is embedded and not made for any tools to analyze it, probe it, or dissect it in any way, it’s pretty much monolithic, that’s a huge difference in how I see file formats, and I’ve been doing this since I was a kid, literally since IFF and PCX times. I’ve made my own variants of PCX (because I was a kid), and it was a brutally honest format, employing only RLE. But this got me thinking about cryptography and so I naturally explored the idea further. I take all that as a learning process, and that was the core message behind my post intended for AnimalMan.

I don’t know what to say, maybe I wasn’t particularly lucky with my particular sample files at the time, but I remember that PNG wasn’t at all inviting. I wasn’t particularly persistent either, I have to admit, but with TIFFs I could at least parse what I saw, and I probably only ever worked with a couple of specific codecs, likely all by Adobe.

Anyway it was a long time ago the last time I did any of this, because I certainly did nothing of the sort in the last 18 years. I simply didn’t have to and my interests have moved elsewhere.

I don’t find that video too scary, none of it I find scary, not even the zip part (it’s just tedious), I do much scarier stuff occasionally; even JPEG is relatively simple to grasp, however: once you get to the bottom of the actual format on paper. But things are horribly different when you know next to nothing about the format, PNG is literally so unique that there isn’t nothing else to compare it with, it demands months of painstaking work to reverse engineer properly. TIFF on the other hand is a festival of trial and error and you’re pretty much guided the whole way, because it doesn’t need to obfuscate or compress anything but the actual container data. On the web even minified JS looks daunting. I’m not sure if you ever tried to go into this blindly, but when I say “hacking”, I really mean hacking, not programming, and even less so engineering. And it looked to me like AnimalMan was simply hacking for fun and credits.

I don’t know, maybe I should give you the benefit of the doubt, let’s leave it at that. I have no reason not to believe you.

I just don’t like to sound like I’m talking nonsense, because what would be the point, I have definitely formed this opinion after having certain experiences, it’s not just to write something willy-nilly. As I said, the differences in our viewpoints is only in how we perceive what is ‘complex’, but everything else surrounding the case I concur with. If you asked me to choose a decoder to build professionally, I would always choose PNG over TIFF exactly because of the things you mentioned; you simply have to consider reliability. But I still think it’s actually harder to do, because when you’re hacking TIFF (in hobby conditions, not in a corporate environment) you don’t really have to care about all the possible combinations out there in the wild. Not even Photoshop does that. All they care is raw, lzw, zip (gzip most likely), as the post-process compression, and several channel ordering schemes like serial, interleaved, something like that, because deflate is also used intermediately and ordering helps to align redundancy. You just stream the channels to buffers, process them, enclose the structure and save metadata. It is incredibly easy to decode once and figure out the exact process Photoshop does, because the image will likely come out scrambled, then you apply the changes and do it again.

Now think of PNG and tell me this is not the more banal case.

Only the very last stages. Graphics hardware is designed to keep texture data compressed and only decompress it when it is needed. The purpose of this is to maximize the data that can be stored and minimize the bandwidth that is consumed when transporting it.

That said the formats being discussed here are not the ones being used by the hardware. Graphics cards use much more specialized formats. When you set up a texture at runtime you can specify the format that it will use.

3 Likes

Good morning,

Did nobody sleep?

I see no objection to the way I handled compression. No fault ?
I am hoping this is not false confidence,
I’ll progress on that image rebuilder and see what the result is.

thanks for the useful responses everybody

if (File.Exists("C:/Users/SCHar/Desktop/OBJECTIVE/DATA/ART/" + "INFANTRY_46"))
        {

            var BYTE_ARRAY = File.ReadAllBytes("C:/Users/SCHar/Desktop/OBJECTIVE/DATA/ART/" + "INFANTRY_46");
            List<byte> BYTE = new List<byte>();
            int count = 0;
            for (int i = 0; i < BYTE_ARRAY.Length; i++)
            {
                BYTE.Add(BYTE_ARRAY[i]);
                count = count + 1;
                if (count > 3)
                {
                    OUTPUTSTRINGS.Add(BitConverter.ToString(BYTE.ToArray()));
                    BYTE = new List<byte>();
                    count = 0;
                }
            }
            OUTPUT = BitConverter.ToString(BYTE_ARRAY);
        }

I am un-educated.
The use case may be ridiculous overkill.
The space gain is relatively insignificant to returning readable data.
All values of output string on return are all 24 dash
So either byte convert screwed up the input,
or i am unable to string out the byte on byte read;

using -

 OUTPUTSTRINGS.Add(Convert.ToString(BYTE_ARRAY[i]));

it returns a list of 434 with all values being 36

OUTPUTSTRINGS.Add(BitConverter.ToString(BYTE.ToArray()));

it returns list of 434 where all values are ‘24’ alternate ‘-’

It is too scrambling for me before breakfast for file size gain to be useful. Considering the use case for this says that I am only loading the games entire variable data. As the game itself is a variable, all variables introduced from folders, including graphics, models and more finite data relating to their ingame functionality. The size of this file is insignificant. As it is not streamed over net, it is not run at runtime. I may aswell bail out and fall back on something that is clearly legible.
Do I need to be that much of a genius?
The answer is no.
Is anybody else that much of a genius?
there are wealthier people with worse published game designs that are much worse yes.
Considering this is all just initialize data.
This coding adventure is a waste of time until i acquire a meaningful context on what is going on, and what i am doing wrong.

:slight_smile:

I’m going to progress in other area’s of the code. And at a later date if i wanted to simplify my open texture2D directory i will return to the problem using a future theoretical universal directory wrapper.

Stream writer nets me just 1k more expensive for the same data in such a way. So its what 12x the size of a PNG? Who cares.

I didn’t examine it, so was in no position to comment either way.

1 Like

It’s def a job for end of development, when front end is complete ready for population and has been populated with a story some factions and the like. 1 day I learned enough. I could learn more, but it is not suitable time for it. I progressed with infantry import data, and animation sequence data instead. Production of infantry objects, who only remain to animate correctly as they move over the terrains. And now I stop for lunch.

I’ll confess I didn’t read line for line. I just don’t have the time to read those massive walls of text. Are you planning on loading the files at runtime? Or are you planning on importing while you create the game and let Unity create a build?

When you import an asset into the editor Unity creates a copy of that asset in the Library cache folder that is in a completely different format that it picks based on the platform you have selected in the build menu. When Unity makes a build that is the format it uses not the original format you imported into the editor.

You appear to be trying to make the files as small as possible but that’s only going to work if you load the files in at runtime. If you’re importing them into the editor for the purposes of making a build you’re going to lose all of those savings.