Ok: I wasn’t going to say anything but I can’t resist. Why are you attempt to over-engineering your problem? Focus on what “must” happen, what you can fake and how data is manipulated. Much easier to think about and to write.
Don’t me distracted by the number of components - the actual code here is remarkably simple, mostly a handful of very decoupled systems. Similarly the data flow is simple, as is the actual object hierarchy. It’s just not as efficient as I’d like, and making it more efficient is probably going to sacrifice some of either the simplicity or the decoupled nature of the independent systems.
Just because I can talk verbosely about the particular design trade-offs currently in the system doesn’t mean that the current state of the code isn’t simple and straightforward. 
I have to agree with Roquen… I know what you’re doing is an experiment, but the hypothesis you’re testing with your experimentation is fatally flawed: you don’t have a system anywhere near remotely complex enough to justify the current implementation and design - so you have actually deliberately created complexity here to make it harder for yourself… to self-justify the original hypothesis that ECS make things easier for you.
Cas 
Greets… Been a while… ;D A little late to the topic here, but I’ll bite…
@OrangyTang
I’d like to start a discussion on how people are currently using entity systems, and how they’re structuring them.
I can give my recent insights over the past couple of years. I’ve been fully invested in the component architecture approach for everything including game ES usage since '10 with development primarily on Android since the get go; my larger effort is middleware oriented and is called TyphonRT. In the past year or so since Android 4.3 I’ve been focused on building next-gen hardware accelerated video engine / streaming middleware modules on top of the not always stable MediaCodec API. This includes an extensive GL based 2D post processing pipeline for compound effects / presets for video recording / editing; IE the post processing could easily be used in a game too along with the video recording capability to record game play directly in an app. That being said I haven’t gotten to apply some of my more recent thinking towards game ES use cases specifically, but do have some thoughts in that direction as well.
Cas, is always going to come back with the same line of reasoning re:
“…but the hypothesis you’re testing with your experimentation is fatally flawed: you don’t have a system anywhere near remotely complex enough to justify the current implementation and design - so you have actually deliberately created complexity here to make it harder for yourself… to self-justify the original hypothesis that ECS make things easier for you.”
or:
“I suppose it depends why you’re using an entity system… if it’s to make your code more readable and understandable and easy to maintain… then that’s a fail, as it doesn’t really do any of that at all”
Indeed, if one wants to release a simple to medium complexity game where all game mechanics are reasonably known in advance then plow ahead with traditional OOP and ship your game. This being JGO and indie game dev oriented that is indeed the general focus of most on this forum and the traditional wisdom. IE working on game dev / shipping games / who cares about experimentation.
Experiments are good though and one can become a better developer by experimenting; will it help ship games immediately, no… Until it does… but there is a valley to cross there.
In my case with creating general purpose performance middleware that runs across OSes (desktop / Android) I’ve found the component architecture direction really important and born out of necessity. It started in transitioning the generic traditional OOP entity system I had then I made the component architecture part a superset and applied it across the larger SDK.
I definitely disagree with the readable / understandable / easy to maintain angle. Of course I only have anecdotal evidence. One nice thing is a reduction of method proliferation since the core component architecture API provides a standard way to access data, etc.
The benefits of implicit composition greatly aids in modularity and code reuse in combination with runtime mechanisms that allow components to self wire themselves. Along the way I also adopted the EventBus pattern, but my own implementation different than Guava / Otto. It works with extensible enums to define a series of categories. Since in my component architecture API I can store components by extensible enums this works really well with the event bus since the same extensible enum storing a component can define its category in an EventBus implicitly, etc. This further helps separating communication between functional groups of components / modules without direct dependencies.
A big experiment of mine has been judiciously breaking up my middleware into very granular IDE modules / projects and I spent a lot of time doing this and minimizing the dependencies across all of them. It’s now up to about ~750 modules. The flexibility this brings is that I can configure different runtimes for specific purposes. IE a video recording / GL image post processing runtime vs a game development one. Maybe you want to combine both of them, etc. TyphonRT also declaratively loads the runtime from external configuration files. On Android this is pertinent because the config files can store conditional logic like load this component on OS < v4 otherwise load newer component > v4, etc. which is a lot better than embedding it all over a code base. One can even get down to device granularity and load potential workarounds for specific problem devices.
Again I’m talking about general purpose, but high performance middleware here and not a specific one off small to medium game.
The component architecture approach was the only way forward as traditional OOP buckled at its seams for me at least especially considering stability across the larger Android ecosystem.
I’m a fan of the MMM (modularity maturity model) http://www.infoq.com/news/2011/09/mmm-osgi and am working my way up the model and am roughly at stage 4.
Where is the launch? yeah yeah yeah… I get it… All of this has taken a while especially since I’m bootstrapping / have a day job at times. I’m actually quite excited to be moving toward launching the next-gen video engine for Android soon. I certainly could not have built it or had a chance to maintain it with traditional OOP as it would have been very difficult.
Regarding launching TyphonRT as a development framework I’m really trying to reach level 5 of the MMM before launching as it’s a difficult thing to organize otherwise. Hopefully time will make itself available and next year is the year… 
Now regarding game development specifically. As mentioned I haven’t been working with game demos for the past couple of years, so I have only prototyped what I mention below. As OrangyTang mentioned there are issues with Artemis and most ES implementations regarding performance in general. I haven’t had a chance to look at artemis-odb yet…
As things go most of the ES implementations out there for Java have been very narrowly focused on game dev and IMHO attempt optimizations at the framework level which make them rather rigid.
Often this leads to a global component pool which I believe is a big no no…
In the new ES efforts I will get around to sooner or later I use a built in extension mechanism of TyphonRT. I even described it here on JGO almost 3 years ago. That is a special system component that potentially executes when the component architecture API is invoked. This allows a basic component manager to modify its capabilities when components are added / removed. In my new ES efforts an “EntityManager” is just a “ComponentManager” with a special “EntityManagerController” which controls what is done when entities are added. Basically it splits up all of the child components and independently indexes them. At this point in time and this is rather dependent on the types of components it’s quite possible to pack all of the really important data (position / health / etc.) for all entities added to the manager into the most efficient data structure necessary to iterate over all entities from a packed array for instance versus calling an update method per entity with the downside of typical object / memory / cache issues. Remove an entity from the manager and the related EntityManagerController unpacks the entity and stores data in individual data components again. I guess you could say this could be called a JIT entity system… 
The nice thing is that if you don’t like the standard game dev components I provide or the way I pack / unpack the data one is 100% free to replace EntityManagerController with a custom implementation. IE all of this is not baked into the generic component architecture framework itself which is the most important part.
In short this is the direction I’m headed regarding ES design which was the original question.
I spent a great amount of time grinding my teeth over hierarchical inheritance in game programming. I ended up creating a Unit, then a ShootingUnit and then a MovingUnit, and then I had to make a ShootingMovingUnit and everything collapsed. ES approach blew those issues away. Why this is still debated simply is mind boggling.
Surely, my Artemis is just one approach, one implemention by one guy. It’s even quite dated. Artemis is not be-all and end-all, and frankly I’m quite surprised how popular it has become. There are many variations of Artemis out there by now that have improved upon the original design, in many languages. The Java Artemis quite influenced by the Java language design, so for other languages you might want other approaches.
The idea is not very complex. You want the shooting behavior and the moving behavior to be separate so that you can make shooting+moving unit, a shooting unit, and a moving unit without losing sleep. You can really only achieve that by decoupling the behaviors.
That doesn’t necessarily lead to entity systems. I have a completely normal sort of game object hierarchy here, but all the behaviours of things are governed by sub-objects, eg. brain, weapon, movement. If you’re classifying objects as ShootingUnit, MovingUnit, or ShootingMovingUnit, you’ve not quite grasped what it is you were trying to model in the first place. You’ve just got Units, and they may have Weapons and DriveUnits which are targeted by Brains and operated by Movements… but you don’t end up with an entity system.
Cas
I feel like we’re more discussing semantics, what these things are called and how “pure” it is.
ES is just the concept, not an implementation. It’s about splitting up your code into modules and executing it the way you want.
The tools regarding Java are generics (more specifically generic methods) and extensible enums. It took me ~4 years to finally get a grasp on advanced generic method usage after Java 5 dropped to move toward an implicit component architecture with Java. Extensible enums were the key for generic state handling along with extending the ability of storing multiple components in a manager of the same type while retaining type safety. Java doesn’t have built in ways to necessarily bake into an ES framework super efficiency (structs or the memory / Object[] matters), but the syntactical constructs do exist for modularity which provides an improvement over traditional OOP IMHO.
The performance trick as I described in the last post is more akin to creating an implicit system that can serialize disparate data components into the most efficient form for the moment.
An anecdotal account on how things at least for me led to a component architecture oriented entity system.
What Cas mentions is explicit composition of course, but even that direction breaks down eventually at least when considering a generic entity system or a game that is evolving. With a well defined and limited game mechanics explicit composition makes a more traditional OOP ES tenable up to a point. Outside of having a tendency to still have a god object base that potentially aggregates explicit composition aspects that may not be shared by all entities (say “Movement” for instance including accessor and manipulation methods in the parent object when some entities don’t move at all; then AI, then action / state handling etc.) this can still lead to inheritance with further explicit composition at various children levels unless one pushes all explicit composition up into the parent object expanding the bloat. The next gotcha that really made things untenable for me was state / input / control handling. Pushing this up to the parent object hard connects different areas and dependencies between explicitly composed subsystems can occur. Not so bad in a well defined game per se, but for a generic ES this causes consternation eventually. Using the movement example one then tends to push state handling just for movement into the Movement object. This also generally becomes untenable after various state handling is separated into unconnected areas. Even just separation of state handling between movement and some sort of action handler (shoot, jump, duck, etc.) can lead to general difficulties.
Implicit composition / component architectures just makes it possible via a consistent API to store / retrieve dynamically composed subsystems. On my side the “state handling Renaissance” came when I fully grasped extensible enums and by extension creation of new collections (extensible enum map & set) which basically is an EnumMap/Set that efficiently packs extensible enums. Extensible enums are great because state can be well defined within separate source code modules and shared between modules without hard dependencies via a common interface. There may be standard state for the engine / provided such as movement, but specific game state like (FLOATING, TOUCHING_NOSE, EATING_CUPCAKE) defined at the game / app level can be combined with unrelated engine state (MOVING_LEFT) accessible from one location. I suppose one could also employ this mechanism in an explicitly composed ES, but I made the jump to the component architecture approach along with moving to this state handling mechanism. I have a generic input handling mechanism that can define which keys / input sets / unsets which state extensible enums such that the input systems are not hard coded.
Another gotcha with traditional OOP ES is compound entities. Now this isn’t exactly addressed well at all with the initial Java component ES implementations (Artemis included), but is addressed in my efforts as one can nest components to any depth desired. Unified state handling becomes really awkward with compound entities in regard to the traditional OOP approach which is already strained between just explicit composition state handling of the parent and quite likely hacks / workarounds are involved that end in hard dependencies between sub-entities. How I get around this in my efforts is that the root entity has its own EventBus (In my efforts an EventBus is a system component and can be added to any component manager) and parent / child entities communicate to each other and other compound entities over the parent entities EventBus.
A further benefit of extensible enums is that I figured out a clever way of using them to store components implicitly while retaining type safety for the object type with the additional benefit of even being able to capture and ensure parameterized type safety for components (IE List). This gets around several problems including being able to store an infinite amount of the same component in a component manager by enum name instead of just being able to store a single component by class / interface type. In regard to compound entity state handling this is great because my EventBus works by a series of categories defined by extensible enums. A concrete example is lets say the player entity has left and right weapons which are sub-entites that can fire and potentially may be holding or not holding a weapon in either hand. When a weapon entity is added to the player entity it is stored under “LEFT_WEAPON” or “RIGHT_WEAPON” extensible enum. The weapon entity understands the “SHOOT” extensible enum, but has no idea directly about the “LEFT_WEAPON” type ID. On the EventBus thus the category for event distribution to the weapon becomes LEFT_WEAPON->SHOOT and on addition to the parent entity the weapon entity registers with the parent EventBus for the “LEFT_WEAPON” category which it can retrieve from the component architecture API since it is the type the sub-entity / component is stored under and the SHOOT sub category which is understood by the entity.
Now there is some sort of action handler at the parent entity level and it could be hard coded or more generic which can be accomplished in various manners. Let’s just say it’s hard coded (so I can stop adding words to this post!) and examines the extensible enum set for compound entity state. LEFT_WEAPON & SHOOT is defined so an event is created and fired over the EventBus to the LEFT_WEAPON->SHOOT category which the sub-entity receives as it implicitly registered for this category.
All this above is anecdotal experiences on how I moved to a component architecture ES.
It should be noted though that a lot of this applies more to generic engine / SDK development or an expansive / evolving game where the design is not finalized (which is most games right?). At a certain point even with explicit composition a rigid hierarchy and dependencies between composed components arise which make further modification / extension untenable and that only complicates things as this tends to occur toward the end of the project when there are a lot of late breaking changes in design. Whereas an implicitly composed ES does not face these problems especially if inter component / entity communication is handled in a generic manner and is much more flexible to changes in design later in the development cycle.
This may all seem like crazy specialization for an ES until one realizes that the above event distribution aspects and sub-component storage works for any sort of app / SDK development as well and applies architecture wide for the entire engine / framework beyond just an ES use case. I’d love to make a game with this tech or have the funding to do so at some point. In the meantime I’m focusing on mobile media / video engine tech as this is an underserved niche especially on Android whereas game engines especially now with bottom basement subscription model ala Crytek / Unreal Engine let alone Unity have gone the way of the dodo regarding any hope for monetization. I can’t imagine making the video engine tech and recording / editing app which is completely generic between GUI and GL post processing functionality without the techniques described above. It’d be a maintenance nightmare w/ traditional OOP.
While touched upon previously modularity is really key especially when creating testable larger SDK / frameworks / engines. By being able to only depend on a few core modules for the component architecture it’s possible to separate dependencies between modules quite finely. There is rarely any Class in my efforts that approaches 1k lines of code (if I had to guess ~5% of them) and modules often aren’t larger than ~5-10 components. This makes code understandable and by limiting the access / dependencies between modules it’s very possible to get things right and not have side effects / regressions occur with future development. I also like that the module gets a specific name and package thus while there are over ~3k classes in the larger TyphonRT effort (most not directly game oriented) there are only ~750 module names to remember and they are named very descriptively and form functional groupings which there are around ~80 of those.
Another massive benefit which I have yet to build from a tools perspective, but is something that I find very exciting is to be able to instrument the component architecture API and EventBus such that one can create a runtime viewer of the framework / engine and have all component architecture API calls and EventBus messages be sent over the network to a monitoring tool that visualizes the runtime and inter-module communication over various EventBus configurations. Instrumenting a component architecture is easily accomplished versus a traditional OOP engine.
Damn chitchat monster… back to work here…
This brings to mind the anecdote about the King and the Royal Toaster.
Cas
It’s gold right, the toaster; no, let’s make it solid platinum! ;D
And then there was Ashley… ;D
http://www.badlogicgames.com/wordpress/?p=3481
After a quick code review though it looks like it borrows the design completely from Artemis and will have the similar performance bottlenecks and general rigidity. It will be interesting if any additional performance can be squeezed out of it and where the folks behind it will take it given that it is tied to libgdx.
Yea, seems Ashley is derived mostly from Artemis.
Java is a very cumbersome language when dealing with inherently dynamic nature of ES. Probably 90% of the overhead and complexity is the ES implemention for Java (Artemis) stems from the rigidness of the Java language.
If you were writing in JavaScript, it would be almost effortless and natural to go the ES way.
There is a bit more flexibility regarding Java when it comes to supporting dynamic nature of component architectures. Seemingly focusing on the component architecture first and then extending it to build an ES versus wrapping basic CA techniques around a purpose built ES is a step to unlock a different design and more performance.
What I mean by that is the aspect, heh, of tying the CA to the notion of “aspects” or “family” as an organizational pattern impacts the flexibility of the CA API itself. In this case systems in Artemis and Ashley are organized as broad phase processing, but are limited to processing fully composed entities sequentially.
Years of work, 3K classes and only 750 module names to remember, effectively supporting video engine tech, which would otherwise have been a nightmare to maintain.
Surely this is a misleading conclusion, but this seems to be a lot of bootstrapping, creating this elaborate foundation, that has the potential to be awesome, yet the only outcome is a product that didn’t warrant this huge investment in time and effort. It seems like this Hollywood style Transformer washing the dishes.
I firmly believe generalization to this extent is counter productive. I can see the potential of Entity Systems, I just haven’t seen a use case, only people describing them as the ultimate answer to customizable Entities, while hardly any game ever created requires, or benefits from this complete lack of restrictions.
In return we get significant code overhead, memory overhead and performance overhead, to achieve this pinnacle that nobody really gets that excited about, while actually developing a game. People fall in love with the potential of clean code, but so far every game ever developed is a tangled mess, regardless of the framework it was built on.
Well… that.
The game I’m making at the moment is far more complex than anything ever produced here on JGO. Or for that matter more complex than any game I’ve ever yet seen made in Java, ever. Still doesn’t warrant it. But you know… toasters. Etc. No-one ever really learns until their livelihoods depend on it and someone’s head rolls.
Cas
Cool story :
From what I can see in the readme, Ashley looks like 99% Artemis. Only it uses different names for things? Where is the big difference here? Nice that it’s on gradle but we have an ArtemisGDX (or whatever it is called) there too. So, hmm, not sure why one should use Ashley now?
The overhead Entity Systems produce is really annoying, I agree with that. It’s unfortunate that Ashley couldn’t fix that. Maybe there is no way 
Well i am going to chime in and have my say…
I am using a Entity like system. Technically it is a Entity-attribute-value system. I think these things can work well for the right kind of problem.
The only 2 game cases i can think of are RPGs and MMOs. In both there are just huge amounts of different things yet the relative number of behaviours over all those things is small.
I am not using it for a game, but for work and have the same pattern. Lots and lots of different things where things can even change to be other things, yet only a fairly small set of behaviours over all possible things. I also need to be able to many cores.
So far its working out well. Java kind of suxs a little for cache coherency, but not much since the data size of entity’s can change at runtime and would probably need attributes referenced anyway.
“Surely this is a misleading conclusion” -> it is the one produced by the JGO / indie wisdom grinder.
For better or worse I started in this direction from a long term “blue sky” goal and adapted what I developed to try to move toward something of more immediate value which still is quite illusive in general, but close at this point or at least closer than the past. The bootstrapping part especially living in San Francisco has been arduous; behind on rent currently. Knowing that I passed on at least ~$2MM of income to this point if I simply took a traditional career path is more concerning than it ever has been…
Regarding the video engine / middleware it’s more like I had the right tools at the right time to start rapidly producing something of complexity the moment it was possible on Android that normally would be prohibitive from a traditional OOP perspective let alone a team of one. Not that having more devs reduces complexity / time to market especially considering OOP trappings. What I actually created is a solid middleware layer for advanced media handling on Android that is not available in the stock SDK. This is just one of many configurations of the tech involved and many apps can be created as a result in regard to product. I’m focusing initially on a prosumer video recording / editing product, but plan to release a suite of apps with some of them being more consumer friendly than others.
I also got through the tech evaluation process recently for licensing the video engine tech for streaming use cases with a company most have heard of alas they wanted the middleware tech, but didn’t actually want to pay for the requested SLA and other requirements like sublicensing. Boutique middleware development is not a good place to be in general and seemingly less so in the future in general. I’d really like to make the transition to being a product oriented company.
Besides expanded / final QA, packaging, and IAP / user extension testing things are ready to launch for the video editor. A final go to market goal is to enable users to buy a feature once and have it accessible across multiple apps in the suite. IE buy some presets or specific functionality like image stabilization or time lapse recording for the consumer app and you can use it in the prosumer / pro version, etc. I’m moving towards launching a Kickstarter since I’m 95% of the way done.
I guess one more offhand comment the chitchat monster compels me to tell is that even though I haven’t got a product to market the efforts I’ve done has always gotten me a job / client when I need one. These days I just have to pull out my phone and say, ever see that before?
As I mentioned generalization to this extent is counter productive until it isn’t… There is wider applicability of CA beyond ES use cases. In my case I had a large Java 5 codebase I immediately started porting to Android when the G1 hit my hands and rather quickly it became obvious the traditional OOP mechanisms that were marginally tenable at the time on the desktop with just a couple of variations between OSes simply did not map well onto supporting the larger Android ecosystem especially with how things proceeded and the aspect of integrating with the Android runtime.
The biggest problem with current available Java ES implementations is that they were created in the blind without an existing large scale game or codebase to test against for performance or actual flexibility. It would be much better if a high performance game rendering a lot of entities was created first then work backward to a more flexible structure while maintaining performance.
My CA efforts benefited greatly from having a large codebase to refine how it all fit together including performance concerns which were important to take into consideration given Android.
I totally get it that I need to release the framework or provide a knock out app to prove the case… Almost there… Though I’d like to defer as much as possible the chitchat monster on what I’m doing and move the discussion forward regarding CA / ES in general. I hope to participate with the larger community in a less anecdotal manner “soonish”…
Not sure where the code overhead is coming from as CA / ES not only allow smaller more intelligible Class / file sizes, but significantly reduces code duplication inherent to traditional OOP… Memory overhead is not much different than traditional OOP ES with existing available Java ES. The same general techniques to improve this from the OOP side of things can be applied to CA / ES, but with the benefit of CA doing these optimizations in a more dynamic manner. The performance faults of current ES implementations can be improved and what is currently publicly available should not deter anyone from working on a new design.
For a moment there I thought maybe you might be working on something ambitious… No disrespect… : :o
It’s a pretty ambitious free-to-play MMOG in which literally thousands of battledroids try to kill each other.
Cas
Right on… I’ll be in line to pick up a copy when you get it out there. For a moment there I wondered if you were being facetious to back up a point… ;D
The potential benefit for Ashley is libgdx integration with an expanded set of standard components that work with other areas of libgdx out of the box. The libgdx collections backing the ES implementation may be slightly faster as well.