@theagentd: But that conversion from enum constant to backend specific integer is the same as my implementation, just using a static variable as opposed to an enum (although I would be interested in RAM usage with an enum as opposed to static class fields).
Although writing such a thin abstraction layer doesn’t really seem to serve a practical purpose. It just seems to introduce more problems than it solves.
It’s the standard method built into the JDK for an abstraction to find an implementation without having any dependency on it. eg https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
What I mean is that you build an initialization step into the abstraction API that looks up implementations at runtime using ServiceLoader. This usually involves an implementation of a provider class. That in itself isn’t specific to your enums. As part of that initialization the abstracted API (eg. code in the enum or same package) queries the provider implementation for each int value and caches it in the required enums. The implementations do not have to initialize the enums themselves, or know anything about what mechanism you’re using to cache the constant mapping. You manage the mechanism, initialization order, etc. in one place rather than many.
The code @Archive posted where each backend has to call back into each enum, know all the possible values, do things in the right order, etc. becomes tiresome quickly the more the API expands and the more backends you have.
I agree with that. I had another use case in mind. If you go for just those two backends and the mapping is n:m (with n <= m) then I’d go for the approach of CoDi^R or just map it directly in the switch/case statement.
I tend to think too long about such things ending up with solutions which doesn’t seem to improve it too much, but burned time away like hell.
I agree with this, which is why I want to go with enums for the abstraction to get compile time checking of arguments. My initial idea had the OpenGL and Vulkan constants in the enum, which was bad as it meant having backend specific data in the abstraction. However, I think putting an int field in the enum that the backend can use for mapping the enum to a constant isn’t the same thing and shouldn’t be bad design as it has the best performance and arguable the lowest complexity and maintenance requirements.
OpenGL and Vulkan are already abstractions… you probably don’t want to abstract over them any more.
Cas 
c.ontomions post is SPAM. He just copy/pasted theagents reply and added his URLs below.
Given the hoops you have to jump through to register that’s one determined spammer…
Cas
I don’t agree. This is common practice in the engine world. You write your own abstraction layer over the different APIs you want to support, mainly to support console specific APIs, and then write your engine on top of this abstraction. This makes your engine independent of the API you use.
The huge mistake that people make is that they see Vulkan and all its complexity and say “Damn, this is too complicated. I’m gonna write an abstraction that let’s me use Vulkan as if it’s OpenGL!”. Essentially, what this means is that they’ll be writing their own OpenGL driver on top of Vulkan, and trust me, nobody has more resources than Nvidia’s driver team when it comes to OpenGL drivers. You’re going to fail, it’s going to suck, it’s going to be a nightmare of complexity and it’s going to be slow.
The correct approach which I’m taking is to do the opposite thing: Write a Vulkan emulator on top of OpenGL. Basically, emulate command buffers and descriptor sets, discard all the extra information that Vulkan requires, etc. In fact, OpenGL either trivially maps to Vulkan in most cases, or is simpler than Vulkan. Vulkan render passes just contain a mapping from a list of render targets to a set of draw buffers, while Vulkan framebuffers (together with render passes) trivially map to an array of OpenGL FBOs. Emulating Vulkan on OpenGL is trivial.
The result however is not at all a simplification of the underlying libraries. Rather, it is the most explicit kind of abstraction you can imagine. You essentially have to provide all information that any of the underlying libraries would require for everything, with the unneeded information being silently discarded by each implementation. In practice, this means essentially coding for Vulkan and getting OpenGL/DirectX/Metal/WebGL/OpenGL ES for free.
That being said, you still get a lot of things for free with my abstraction. The most complicated parts of Vulkan is arguably the swapchain management, synchronization and queue management. My abstraction will completely hide the swapchain management, simplify and abstract the synchronization and automate the queue construction. In the end you only need to worry about the “fun” parts of Vulkan where you actually gain a shitload of performance and flexibility.
What I’m getting at is, why abstract this, when your “engine” API is already the abstraction? The underlying rendering API - Vulkan or OpenGL (or even DX12!) - is what your engine will be using to turn its own ideas about what it has to render into rendering commands for the specified low-level API. You write a specific plug-in to render what your “engine” contains using the specific API. There’s no real point in abstracting the rendering APIs and then using that abstraction to feed to your engine… do it directly. I’m probably not explaining this very well though.
Cas
It looks like @theagentd is developing a “middleware” for game engine developers.
In my opinion, it is probably of little use to actual “game developers”, who require a much higher level of abstraction from a game engine, to be able to say: “Here is my model file, please render that with a nice sky, a sun and beautiful reflecting water.” and possibly just modeling and scripting everything in a nice editor, like the CryEngine Sandbox editor. Those people hardly want to get down to such low-level details as “command buffers” and “swap chains”.
But for game engine developers, this could be a thing.
I think I understand what you’re saying, but I’ve concluded that it’s simply too much work to maintain. IMO, the graphics API is such a vital part of a game engine that the graphics needs to be well-integrated to have good performance. In addition, the graphics APIs actually have very much in common. The optimal strategy is essentially the same on all APIs, but how you accomplish that can differ a bit. The point of that is that regardless of the API, most of your logic will stay the same.
Obviously, the graphics is just a small part of any complete game engine, as there are lots of logic and API-independent functionality that the engine needs. Hence, it makes sense to at least compartmentalize the graphics API as a complete independent module of the game engine. Now, you could just write N different versions of this module, one for each API, but this has a number of big drawbacks. You will end up with a lot of code duplication, as a lot of functionality is almost independent of the API used, to the point where you simply want to call an API function with a different name to accomplish the same thing. Having to rewrite higher level functionality for each API is a lot of wasted time, increase in maintenance and increased risk of bugs. There is also a severe limitation in the extensibility of the engine by the user. There are lots of techniques that require very optimized and tailor-made rendering modules, for example advanced terrain systems, fluid/water rendering, etc, that require a level of control of draw calls that usually cannot be achieved effectively with a general purpose model renderer. By forcing the user to write one version for each graphics API they want to support, you both force them to learn all the APIs they want to support (which is exactly what you want to avoid), which in turn will
encourage users to limit their support to a few specific APIs. This is a major blow to cross platform compatibility, as something as trivial as rewriting a renderer from OpenGL to OpenGL-ES can be prohibitively time consuming. Essentially, such API integration would severely diminish the point of the entire engine.
I have a number of goals with my abstraction:
A completely different version of the engine would violate essentially all of these goals. To facilitate point 5, a low-level abstraction of the graphics API is needed, so that the user can take full control of the rendering if needed to accomplish some exotic rendering task. If I’m going to write a low-level abstraction of each API for the user anyway, it makes a lot of sense to base the entire engine on top of that abstraction too. This reduces the amount of code duplication and also forces me to test the abstraction fully, as a bug in the abstractions will show up when I implement the built-in renderers of the game engine.
As an example, consider texture streaming. In OpenGL, this involves creating a separate OpenGL context, mapping buffers, uploading texture data and managing GLsync objects. In Vulkan, it involves a transfer-specialized queue (if available), mapping buffers, uploading texture data and managing fences/semaphores. These concepts are easily abstracted 1-to-1 into a common interface. By abstracting that away at a very low level, I can both write a single texture streaming system based on this common interface and avoid tons of code duplication, and even allow others to write tailor-made texture streaming systems with the same performance as mine for their very own purpose, possibly based on GPU feedback on which texture data is missing, using sparse/virtual textures, etc.
This might be interesting: https://de.slideshare.net/DICEStudio/framegraph-extensible-rendering-architecture-in-frostbite
I think the concepts in JavaFX have it right.
Cas
In my code i have engine specific enums, so i would build your scene graph (as an example) with no back end specific values. When you then convert this to a data model that your back end renderer will use you then convert the enums to back end specific values. This way you have a distinct separation between rendering and the data model
Care to elaborate?
The entire point of this thread was to figure out how to do that efficiently. =P
Apologies theagentd, the OP seemed to be asking about the method rather than asking about the fastest way. Why would speed be an issue for rendering if you do the mapping at the point of initialisation which would mean all the mapping is done prerender?
Well, JavaFX does not in any way attempt to abstract both DX and GL into a single lookalike API. It resolutely has its own set of constants and commands and a scenegraph type of structure, and it has a completely pluggable back-end on it.
Unity is the same: it presents an API geared towards doing what you want to do, not precisely how to achieve it; it takes care of the stuff at the back end using a pluggable architecture. Even the shaders are in their own special Unity language and compiled to GL, GLES, DX or whatever, depending on the backend.
It seems that pretty much all successful, in-use APIs follow the same paradigm: don’t try and find a common ground between the backends you want to support and make the thinnest possible veneer between them; instead, totally hide it all away and present higher-level APIs to the end user.
Cas
@princec It certainly is possible to do a low level abstraction, and here is an example of it: https://github.com/livingcreative/xgs
[quote]1. I want to minimize the amount of work I have to put in.
2. I want to minimize the time taken to maintain and add new features to the system.
3. I want to minimize the risk of bugs by avoiding code duplication whenever possible.
4. I want to completely hide the graphics API used to avoid users having to learn all the APIs they want to support and subsequently locking themselves into a single API to save time.
5. I want the user to be able to work very close to the API when writing their own renderers for maximum performance and flexibility, without actually exposing the API being used.
[/quote]
@theagentd It seems like the best solution involves breaking each function/concept (i.e. Textures, Framebuffers, Shaders) into a class, then writing a bunch of static functions. The class would have a static final variable determining which backend, and then the functions would have an if-then-else based on the static final backend variable (to allow for optimizing the code if possible).
This approach:
For example,
public class Constants {
public static final int BACKEND = Integer.parseInt(System.getProperty("backend","0"));
public static final int POINTS;
static {
switch (BACKEND) {
case 0:
POINTS = GL11.GL_POINTS;
break;
}
}
//...
}
public class Texture {
public static int create() {
if (Constants.BACKEND == 0)
return GL11.glGenTextures();
//...
return -1;
}
public static void bind(int target, int id) {
if (Constants.BACKEND == 0)
GL11.glBindTexture(target, id);
}
//...
}
Nah, that’s way too frustrating, error-prone and un-extendable. If you want to add a backend with that system, you need to modify the core classes, meaning that if a user adds their own backend their code will be incompatible with future updates to the main code. It also essentially forces you to have all the backends in the same file, not to mention the huge amounts of switches/if-statements needed in literally every single function.
Using interfaces/abstract classes and having the backends implement them is much more robust, allows for extension without modifying the core classes and is equally fast, since with just one implementation of an interface loaded the function calls can be inlined perfectly.