On 11/5/2019 at 3:20 PM, Sacaldur said:

First of all, you have to think about when it's used in the entire development cycle.

First of all, you should read the repo's main README. It starts by explaining exactly that. Why this format was created, and who is its target audience.

In short, this format aims to be used in engines, it is optimized for fast load times, but also to be a distribution format (you can embed the materials and textures and it is small, therefore saves storage space and network traffic). What it is not, a designer format that can store all aspects of a modeller software, I leave that job to the modeller software's native format (after all, they were developed for that only purpose).

If we want to compare it to 2D images, then this format is not a PSD, it does not store layers, history etc.; it is rather a PNG, which can store all aspects of the resulting product (dimensions, exif data, palette, etc.) with high efficiency and compression. You won't get an undo history by opening a PNG, but you'll get the image properly and entirely, there's no missing data (unlike with existing 3d model formats).

On 11/5/2019 at 3:20 PM, Sacaldur said:

either small size or (what most will prefer nowadays) fast load times

You speak of these if they were exclusive. The point is, they are not. Actually quite the contrary, a smaller file can be loaded faster from disk :-) With this Model 3D format, you can have both (that's why I invented it!).

For example, I've converted a high-resolution model of Bat Mobile which I've downloaded. The original was 22 Megabytes in binary FBX (and textures were separated files), but in M3D that's only 1.6 MBytes (with textures embedded). Obviously it is faster to load and parse 1.6M than 22M. But let's put deflate aside, the plain binary M3D is only 2.4 Mbytes (with textures), 90% smaller than the binary FBX just because it has better data representation and much higher information density. And I haven't spoken about the post-process yet, the FBX variant had to be converted a lot to finally display it, while the M3D version can be sent into a VBO directly (no in-memory conversion or further parsing needed at all).

Another example, in Minetest mobs_dwarves mod, there's a Blitz3D model with animation. It is 151 K. After converting into M3D, it is only 8 K (compressed to 0.05%!!!), and all information is there, nothing is missing, you have all the materials, their names, texture references etc., and you can replay all frames exactly as they were in the original. I can't stress enough how much work I put into this format to be better than any existing format. And it is better :-D

On 11/5/2019 at 3:20 PM, Sacaldur said:

only the engine developers are your target audience.

No. They are definitely targets, sure, but not the only one. This format is capable of storing all aspects of a model within a single file, because I hate that you can't download a complete model in a single file (textures are missing, mtl file is missing etc. which you can only see when you unzip). So it is also designed to be a model distribution format which does not need additional files and zip.

On 11/5/2019 at 3:20 PM, Sacaldur said:

For all of these, a text format is easier to handle

That's why it also has an ASCII variant. However I'd prefer to store the modeller's native format (BLEND for example) if I were in the designer's shoes, instead of any text-based format (those are for interchanging models between software, and not for storing all software-specific options. Hell, you can't even store a bone hierarchy in OBJ. And they are SLOOOW on load).

18 hours ago, Shaarigan said:

It is fast load times in nearly any case

Agreed. It is not pleasant to download 60 Mbytes, open it in a modeller software, wait a minute for the importer, just to find out that materials or texture are missing, named differently, so you can't actually use the model.

Let's not forget smaller files can be downloaded much faster (which is specially important if your game is server-based and you're sending models to the clients).

Because you haven't answered my question about the sprintf, I'll assume everything is fine with it (and you shouldn't use ASCII format in the first place :-) ).

Cheers,
bzt

6 hours ago, bzt said:

First of all, you should read the repo's main README. It starts by explaining exactly that. Why this format was created, and who is its target audience.

Yes, it states that other formats have certain drawbacks, and that's why this new format was created, but it doesn't explain where it wants to belong. 

6 hours ago, bzt said:

You speak of these if they were exclusive.

There's always a tradeoff. You can either optimize for calculation times, or you can optimize for memory usage. Yesm your file format is smaller than others and faster to load, but if someone now wanted to optimize it even further, they could either make changes to the format so the stored data more closely resembles the data structures used by the application, or the data is compressed even further. (Keep in mind that uncompressing and converting data needs time.)

Professional engines mostly have an own data compression anyways (or even an own data format that plays well with the compression) and are optimized to streaming the data on demand when it is needed. The question is if developers will use the format in their engines or if it stays a development time exclusive thing.

Then that's my point on this topic, it depends if it finally fits into their workflow and can be exported from Maya oder 3dsMax (because these software is industry standard). I honestly don't see people convert an art software export to the file format, just to convert it to an engine asset

Thank you guys for your interest! I have a feeling that you're seriously underestimating how prepared and experienced I am on this topic. But that's okay ?

4 hours ago, Sacaldur said:

but it doesn't explain where it wants to belong.

I think it does. Can you suggest a better phrasing that would make it clearer? I'd appreciate that.

4 hours ago, Sacaldur said:

they could either make changes to the format so the stored data more closely resembles the data structures used by the application

NO WAY! That's insane! Can you imagine that Photoshop or Gimp changes the PNG format as they please??? The M3D specification clearly states that the model file must store data related to the model, and only to the model, no application specific modifications allowed in basic data structures. For example, model stored with well-defined orientation (it does not care if your app is +Y up or -Z up, that has nothing to do with the model itself). If you need application specific data, you can have them in additional chunks, which does not interfere with other applications.

4 hours ago, Sacaldur said:

or the data is compressed even further.

Not possible either. I've spent months to study the topic and define the bare minimum information needed to store a model with skeletal animation, there's nothing you can take away without compromising data integrity. On the other hand, you can use a different stream compression algorithm in your engine if you like (or just use the plain binary without stream compression to speed up loading), no-one stops you from doing so. However distributed files are required to use RFC 1951 deflate only, the specification mandates that (and for a good reason).

And about data representation, I've used architecture limited quantities, you can't go below to shrink the model further. I've already introduced a quality setting in the model files, because high-resolution models (over 65556 polygons) may loose some surface details with the best compression (only noticeable if you zoom in to the model closely).

4 hours ago, Sacaldur said:

Keep in mind that uncompressing and converting data needs time.

I'm well aware, thank you. I've made performance measurements. That's why the SDK autodetects if the model is stream compressed, and works without deflate too, and that's why all of the conversions are done in a converter tool during export, so that loading only needs to deal with a simple, well-defined structure. Let me put it this way: I've moved all the complexity from the loader's shoulders away into the exporters.

3 hours ago, Shaarigan said:

Professional engines mostly have an own data compression anyways

And they s*ck. How big their installer is? 1G? 10G? Lempel-Zip-Welsh did a really good job in the 70's which is fast and compact enough. There's a reason why it is still being used 50 years later. Bzip2 or 7z may provide a better compression ratio, but require more resource and are much slower to uncompress. Homemade compressions are ALWAYS worse than LZW, if you want I can give you a mathematical proof of that (as a matter of fact LZW only fails in one case when the occurance distribution of samples to build Huffman-trees is actually the Fibonacci numbers, which is highly unlikely with real-life distribution).

3 hours ago, Shaarigan said:

can be exported from Maya oder 3dsMax (because these software is industry standard)

If you need those, you are welcome to implement them. Blender plugin is provided as a sample and Proof of Concept, and the license is MIT. Open Source rulez ? I'd really like to see you as a contributor to M3D!

3 hours ago, Shaarigan said:

I honestly don't see people convert an art software export to the file format, just to convert it to an engine asset

Professional engines all do this, converting into an engine asset. And all good Open Source engines too. For example the Unity editor imports models in formats that people exported from art software. Orge3D also converts models in art software format into .mesh. Megaglest likewise, but into .g3d. Shall I continue the list? M3D is no different, you can use the m3dconv tool to create .m3d files from .max, .blend, etc. and also from interchange model formats like .fbx, .dae, .ase, .obj, .stl, .ply etc. The only difference is, .m3d is also good for distribution and for your engine, therefore you can omit m3dconv from your build procedure without increasing the complexity of your engine's source. (Note: M3D can be your engine asset format, it was designed that way. If you need to convert a lot to fit its m3d_t in-memory structure into your engine's, then your engine is designed poorly and it will s*ck with VBOs no matter what (like Assimp does). Just stating the obvious).

Cheers,
bzt

Just to be clear, by compression I mean two different things:

1. the way how the abstract data is encoded in binary form, which I call data representation. This involves high level things like storing transformation matrices (12 floats min) or position+rotation quaternions (3+4 floats) instead, and also low level things like how many bits are used for a vertex index, how are strings encoded and such.

2. stream compression, which does not care about 1., rather encodes more often used patterns with smaller bit-chunks (typically the LZW algorithm)

The 1. point is specific to the format (and in which M3D is far better than the competition), while the 2. is optional (for that M3D uses standardized RFC 1951 deflate). Making binary representation (1.) smaller speeds up loading time, while using more complex algorithms for 2. to make the model smaller slows down loading time.

Cheers,
bzt

Hi,

I've put together a small mobile-friendly webpage for the Model 3D format. I hope this helps to clearify what this format is all about.

https://bztsrc.gitlab.io/model3d/

I've uploaded a handful of M3D models too, so that you can play with them. I took the original models from FOSS games with permissive license, or downloaded them from websites which claimed that the models are free to use. If you find your own model here, let me know so that I can properly attribute you (or remove the model if that's your wish).

Cheers,
bzt

Hi,

I'm proudly present that the M3D format is now officially finalized. It has all the features I wanted it to have. List of changes:

The M3D SDK has serious performance improvements, specially on export side. The API hasn't changed, but a few new arrays has been added to m3d_t. Test cases now cover all combinations of binary / ASCII format with bone transformation matrices generation and smooth normal vectors generation.

Support for CAD models have been added. Parameterized shapes, translatable UTF-8 annotations as well as a preview images are now supported.

The m3dconv utility now has its own OBJ parser, which can handle everything that Assimp and tinyobjloader, plus negative indices and free-form definitions too: polylines, Bezier and B-spline curves, Bezier surfaces and NURBS. (Just a side note, tinyobjloader has a C version too, which is almost as fast as this OBJ parser. It is only slower because it uses a float parser implemented in C, while m3dconv uses hardware-optimized, platform specific function from libc (both glib and musl has an Assembly optimized version)).

A dependency-free STEP (ISO-10303-21-4) parser is work in progress. With only 200 SLoC, it can read and tokenize any STEP file I could find; now it only needs a grammar parser (which should not be difficult considering we just need curves and surface definitions, CARTESIAN_POINTs are already converted to floats and indexed properly).

A new feature has been added, the m3dconv utility can now convert any texture format into PNG on-the-fly when creating distribution models (requires libpng and libimagequant).

The m3dview portable viewer has been updated to use VBOs that are compatible with OpenGL 3.3 vertex shaders (although I left the old, pure OpenGL command version too because I think it's code is more readable and more helpful.)

I've done some serious tests and measurements. For this, I've slightly modified the assview.c from asstools so that it exits immediately after the first render cycle. Similar modification was made to m3dview, so that these measurements include everything from model loading, parsing and rendering. Both viewers were linked against GLFW to use the same framework, and both are now building a VBO.

As a test subject, I've chosen SGC Atlantis, with more than 2 million triangles (model can be downloaded from the Model 3D website). The Assimp version parsed a COLLADA file, which had many references to the same parts of the mesh (therefore coordinates had to be transformed in run-time), and was 11 Megabytes in size. The M3D version was bigger, 21 MegaBytes, because it is important in M3D that a simple, only mesh-aware parser should be able to read the entire object, therefore it stores all over 2 million triangles directly (no run-time transformation needed). The COLLADA file had 2117 meshes, while the M3D only 23 material switches.

The results: to load and render the COLLADA file with Assimp, the average was 2.5 secs. The same with the M3D SDK was 1.2 secs. In both cases these results include the VBO generation, which only has to be done once after model loading. BTW that took 0.3 secs for the m3dview, but after all, more than 270 MegaBytes of RAM required for that many polygons, vertices and their attributes (more than 2 million triangles are quite a lot, the model is *very* detailed).

Cheers,
bzt

Looking at this, I want to touch back and ask about something asked earlier, and that's who this library is focused towards?  You talk about creators and engine developers being able to use this, but looking at your viewer you actually iterate over every face, then each vertex on the face, to copy that single bit into the target, then doing that over and over?  That is as far from optimal as you can get for the engine itself, especially doing this for numerous complex models.

For my own hobby project, I started out trying to integrate Assimp into my engine to load all the formats, but realized I was taking a lot of time with all the conversion that happened, both with Assimp loading things, then me loading from Assimp.  I don't see being able to do it any differently with your format, which defeats the purpose.  As is, I ended up creating my own basic engine format, focused on how the engine itself stores the information, so that I have as few loops/copies as possible.  In fact, I ended up with a single loop per mesh, and that results in creating a single vertex buffer and copying in the entire buffer from the format.  I don't need to break it down by faces/vertices because I already know that structure when I am converting to my final engine format.  I also precalculate as much as I possibly need at runtime and store it as a fixed header, this gives me the counts up front so that I don't need to iterate over anything (like you do in the viewer to get the number of material switches).

My question really is just this, why would I use this over Assimp?  I still don't see this as an engine format, which means I still need to create my own conversion utility to convert from this to my final engine optimized format.  At that point, why not just use Assimp, and use any 3D modeler format that Assimp supports as input?  I ask that genuinely, since if you expect engines to be implemented the same way that you have done in the viewer, then that is definitely a non-starter for me.

That gets back to a previous point about engines though, they have very specific internals.  If we are optimizing formats for the engine, then the format is likely always going to be different between engines.  Different engines need different pieces of data to operate, and may put that data in different orders depending on how it prioritizes things.  Expecting others to use what you view as the end all just doesn't work, I would rather see a way to layout custom final formats rather than needing it in a fixed order/layout/etc.  Having a 3D model SDK that standardizes how it reads and converts the input data, while still outputting to a custom format, would be amazing.

xycsoscyx said:

Looking at this, I want to touch back and ask about something asked earlier, and that's who this library is focused towards? You talk about creators and engine developers being able to use this, but looking at your viewer you actually iterate over every face, then each vertex on the face, to copy that single bit into the target, then doing that over and over? That is as far from optimal as you can get for the engine itself, especially doing this for numerous complex models.

For my own hobby project, I started out trying to integrate Assimp into my engine to load all the formats, but realized I was taking a lot of time with all the conversion that happened, both with Assimp loading things, then me loading from Assimp. I don't see being able to do it any differently with your format, which defeats the purpose. As is, I ended up creating my own basic engine format, focused on how the engine itself stores the information, so that I have as few loops/copies as possible. In fact, I ended up with a single loop per mesh, and that results in creating a single vertex buffer and copying in the entire buffer from the format. I don't need to break it down by faces/vertices because I already know that structure when I am converting to my final engine format. I also precalculate as much as I possibly need at runtime and store it as a fixed header, this gives me the counts up front so that I don't need to iterate over anything (like you do in the viewer to get the number of material switches).

My question really is just this, why would I use this over Assimp? I still don't see this as an engine format, which means I still need to create my own conversion utility to convert from this to my final engine optimized format. At that point, why not just use Assimp, and use any 3D modeler format that Assimp supports as input? I ask that genuinely, since if you expect engines to be implemented the same way that you have done in the viewer, then that is definitely a non-starter for me.

That gets back to a previous point about engines though, they have very specific internals. If we are optimizing formats for the engine, then the format is likely always going to be different between engines. Different engines need different pieces of data to operate, and may put that data in different orders depending on how it prioritizes things. Expecting others to use what you view as the end all just doesn't work, I would rather see a way to layout custom final formats rather than needing it in a fixed order/layout/etc. Having a 3D model SDK that standardizes how it reads and converts the input data, while still outputting to a custom format, would be amazing.



It looks like this new js-based page does not allow me to quote your post in parts. Please match my answers with your questions.

First, I must ask, are you serious, or are you just trolling? ALL of your questions has been answered already, or answered in great detail in the documentation. But telling you RTFM would be rude, so here we go again.

The library is a single header file without any dependency, which uses an indexing-only in-memory representation that resembles the GPU buffers. What do you think, who is its target? The Blender plugin adds support to M3D in Blender, what do you think, who is its target?
The vertices are NOT copied over-and-over again in the viewer as you said. Where did you get that? They are copied ONLY ONCE when the viewer starts, which is a must for animation, because if you would modify the original vertices for a frame, then you won't be able to calculate the next frame. I agree copying vertices over and over again wouldn't be optimal, that's why the viewer DOES NOT do that. And the viewer's renderer DOES NOT iterate through vertices either. It iterates through MATERIAL CONTEXT SWITCHes. Huge difference.
About complex models, I've provided loading and rendering benchmarks, also for a model with millions of polygons and more than 6 million vertices (took 900 millisec to load that much data). My SDK is a lot faster than others. What is your question here? Please read my previous posts or do the benchmark yourself. I encourage you to do the benchmark yourself, do not believe me just because I said so! See it for yourself!

If you don't see how to use the SDK with your hobby project, then maybe you're not up to the task of writing an engine. An experienced programmer can see (and a least-experienced can read the API documentation and API usage examples) and would know how to get the data from the in-memory model they need to pass to the GPU. Just a hint, that's easy, a LOT easier than with Assimp, and for static (non-animated) models you can pass the M3D in-memory model arrays directly to a VBO/EBO (no conversion or copying needed at all). But again, I've already told this in my previous posts, and above as well.
Yes, you could design your own format for your engine, but the purpose of this library is to save you from that, because most developers don't know how to design structures properly in an effective way. This SDK provides an in-memory representation of the model which aligns how the GPU expects data. If you need a new format because your engine diverges from that, then your engine s*cks, simply put. For performance, your engine must use a structure that resembles what the GPU needs, otherwise you'll end up converting a lot. Simple as that.

If you have to ask what is the benefit in handling only one format in real-time and convert the models in compile time over struggling with many different formats in run-time, then you're definitely not up to the task of writing a 3D game engine. Please don't feel offended, I'm telling you this as and advice to help you. You need to learn first, and get experienced with the basics.

And once again, last time, this format stores 3D MODELs. You DO NOT OPTIMIZE those for the engine, you OPTIMIZE them for the GPU. Model data is exactly the same for ALL engines, regardless what other data the engine needs and how it prioritizes, because raw model data goes to the GPU, and not handled by the engine directly (unless it's a CPU-only renderer). Yes, every engine needs different additional data, which you may add to the model files without breaking compatibility with other applications. The SDK provides a standard way to do that.

Cheers,
bzt