@docstore

Just to be clear, @DocStore is the same as "Elastic Search" support which is on the list for removal in 15.x.

We use @DocStore often in conjunction with @DbJson

Here at FOCONIS, @DocStore means, that the EntityBean is stored in JSON format in an other @Entity - There are also many tests provided that do this.

Maybe we misused this annotation. The above should be possible with @Entity also (but we do not want a table)
Question may appear:
Q: Why do we need @DocStore at all and don't use plain Jackson/etc to serialize the bean.
A: We want the enhancement, we want to traverse the graph with the ebean descriptors and also want the old/new values. We also want to refer real entites from @DocStore

Maybe, the annotation can be renamed to @JsonEntity - which means that this is an @Entity but has no physical table in the database.

Ah ok, that is an interesting use. Good to know. A little bit like a @View based entity or maybe a special embeddable. Is it only used as an "embedded document" inside an actual @Entity?

Maybe, the annotation can be renamed to @JsonEntity - which means that this is an @entity but has no physical table in the database.

Yeah. Ultimately we don't want it confused with the ElasticSearch feature and that the name documentation tests makes it clear and obvious how it is expected to be used. [I don't recall tests for this, do we have tests for this?]

why you don't want to support Sets in the future?

Removing support for Set is saying that there are NO known good use cases where we recommend its use - NONE.

For Sets to be provide some value then the need some hashCode/equals implementation that effectively de-duplicates elements. The question becomes, is that ever a good idea. Is it ever a good idea to have rows in the database mapped into a Set and be implicitly de-duplicated in-memory on the application side always and automatically, as opposed to explicit de-duplication as and when needed.

The only cases that I had hit have been better served by using findMap() or explicit application code.

Said differently, I know of NO known good use cases for Set but I know of several ways that people can mess it up.

Just in the last week, we see #3173 and the question is, is that a good use of Set or actually a case where it should instead have been modelled as List and for the de-duplicate to be explicit? At this stage, I suspect when I see the failing test case for #3173 it will re-enforce the idea that Set is not good.

This automatically prevents having two identical elements in the collection.

I'd be interested to see what this looks like in practice and why it's thought of as a good idea, why was Set was chosen, why was it chosen to de-dup via the model rather than explicitly etc.

I looked through some code and Set is usually used with enums (EnumSet in constructor initialization, corresponding methods are Set based). There are also some occurrences of Set in combination with a SortedSet coming from application code but this doesn't make too much sense as it looses order when read from DB. Also some @OneToMany fields are Collection probably because IDE suggested it as no List specific method was ever used. Overall the vast majority of @OneToMany fields are List.

For Map it is usually used to speed up read access. The key is usually some application managed identifier, e.g. a String or Enum. Map is used to avoid searching all the time and and removing Map support would force introducing a transient Map field for reads. So it would make code look more ugly. As an example consider a questionnaire with questions and each question has a unique identifier. Answers are stored in a questionnaire result using that unique identifier. If you now have some application logic on answers it usually operates on these unique identifiers coming from the questionnaire model. So a Map for read access is desired.

I'd be interested to see what this looks like in practice and why it's thought of as a good idea, why was Set was chosen, why was it chosen to de-dup via the model rather than explicitly etc.

We often have code, that adds something, e.g with should be deduped. Mostly that are user objects, but here is a good example in a test case:

class Role {
    @ManyToMany
    Set<Permission> permissions;
}

If I want to add two permissions, I just call role.getPermissions().add(p1), role.getPermissions().add(p2)
If I have a list here, I have to convert it in a set first:

Set<Permission> tmp = new HashSet(role.getPermissions());
tmp.add(p1);
tmp.add(p2)
role.getPermissions().clear();
role.getPermissions().addAll(tmp);

Of course, I can also implement it with List.contains, but I always must iterate through all elements of the list.
List.contains has complexity of (O(n)) instead of the O(1) performance of Set.add

The problem is, if adding p1 and p2 is on different code places, I have to build the temp set multiple times.

It would be a good option, if we can implement some shadow fields:

class Role {
    @ManyToMany
    List<Permission> _permissions;

    @Transient
    Set<Permission> permissions;

    Set<Permission> getPermissions() {
       if (permissions == null) {
           permissions = new LinkedHashSet(_permissions); // or use a list, that invokes a callback on modification
       }
       return permissions;
    }

   void methodMustBeCalledBeforePersistEvenIfBeanIsNotDirty() {
       if (permissions != null) {
         _permissions.clear();
         _permissions.addAll(permissions);
      }
   }
}

This can be done for maps also, but ebean needs some mechanism to call a method, that writes back the state of the shadow field.
Unfortunately, this targets also dirty recognition, but would be doable.

Well, I think I'm wanting to see more accurately HOW Set is being used and take stock of the tests that we have. What exactly does Permission look like and especially the hashCode/equals implementations.

We could decide to keep support for Set if we are clear on how it can be used properly [perhaps we give Excellent guidance on hashCode/equals implementations].

What does a @OneToMany with an EnumSet look like? Care to share an example? I would have though that would be a @DbArray.

My bad. I mixed something up. EnumSet is used with @ElementCollection @CollectionTable and not @OneToMany. This is JPA code so no @DbArray. I also can't migrate to @DbArray easily because a lot of code still uses JPA. So database tables are shared by JPA entities and Ebean entities. Entities are duplicated because of weaving, which is really annoying but well. So there need to be some common ground between JPA and Ebean for a doable migration. Both libraries need to co-exist for a longer time and need to access the same db tables until all code paths have been rewritten in ebean. Big bang doesn't really work if databases are large and code base is large.

I did a quick search again and there are like 10 Set<Entity> fields that are either @OneToMany or @ManyToMany. Some of them because it makes sense for the Entity API to return a Set. Some of them to have an easy set.addAll call without checking for duplicates. There are also unique keys involved on DB level, especially on the @ManyToMany link tables.

I also can't migrate to @DBArray easily because a lot of code still uses JPA.

Hmm, bit of a blow that. ElementCollection might not be easy to deal with without built in support in the ORM itself. Hmm.

findFutureIds() findFutureList() ... No opinion, I never asked an ORM before to query something in a background thread.

FYI: 17 odd years ago this started as the idea that ebean could "Build object graphs in parallel" and this is effectively the remnants of that original idea. Today findFutureCount() is useful / used with findPagedList() [in parallel fetch the total row count]. As I see it, the original idea was basically a bad idea and I don't actually expect anyone to be using findFutureIds() or findFutureList() but they have never been removed - the last vestiges of that original idea that wasn't ultimately a good idea. Conceptually we could hide findFutureCount() so that it is only used as an implementation detail of findPagedList().

a good idea to look for synchronized blocks / object.wait() within ebean to avoid thread pinning

Already done. I did that ages ago and so there is no use of synchronised in ebean.

JDBC drivers will have trouble with this as well.

I'm also the guy that got the change into PostgreSQL JDBC Driver to remove the synchronised there (which is the JDBC driver I care most about). So yeah, I'm good to go with Virtual Threads. Just waiting on Helidon Nima now.

Just waiting on Helidon Nima now

Sounds nice but wondering how jetty with virtual threads compare against it on pure HTTP performance.

how jetty with virtual threads compare against it on pure HTTP performance.

Indeed. The implementation of Nima is really simple and uses VT per connection. Jetty 12 core now looks like its "async only" and to me that implies that with Jetty 12 there is always going to be the handoff from worker threads to Virtual Threads (unlike Jetty 11).

The Nima HTTP 1.0 & 1.1 implementation to me looks really nice and simple. The Helidon team have been benchmarking against Netty (Helidon 3.x is Netty based) and they are "All In" on Nima and Virtual Threads.

how jetty with virtual threads compare against it on pure HTTP performance

Oh, just to say this was the way I did all my original Loom / Virtual Threads performance testing. Ran Jetty using platform threads vs loom with various "simulations of IO wait" using LockSupport. That and using JDK HttpClient with Virtual Threads. This is how I got very comfortable with the performance of Loom.

I have not update those tests and re-ran them for ages. I believe Jetty 12 makes that harder so if I re-run those tests I'll probably stick to Jetty 11. Those tests could also now run with actual IO wait like actual JDBC calls to a DB rather than simulation via LockSupport.

@ElementCollection ... I guess you meant DBArray here as alternative instead of json

You'd use @DbArray with Postgres, YugabyteDB, CockroachDB, and H2 for collections of scalar types. You'd use JSON storage for the other databases and collections of embeddables.

Remove @Inheritance

Inheritance looks innocent enough but it has runtime costs and complexity costs. Ask the question, would you ever use inheritance if you were using SQL and JDBC? Going back to conceptual Logical Modelling and Physical Modelling then yes inheritance comes from that area and is pretty well defined BUT in 2023 no RDBMS that we care about has the concept and so using ORM @Inheritance adds an impedance mismatch on the application side - to some extent we are choosing to occasionally hit ourselves on the nose. At the core is that the PK is no longer enough most of the time and we instead need the PK AND discriminator ... and the complexity grows from there and touches pretty much all the internals of an ORM.

It will interesting to see how I feel with a diff that removes it wrt how much complexity that removes from the code base.

Hmm. We are using @Inheritance with the @docstore annotation when storing jsons. We do not use it anymore when storing db-beans.

I'll try to provide a testcase/showcase tomorrw, what we do exactly and what we really need. Then we can discuss, if there is a way to model that in ebean or we have to switch to alternatives (e. g. Jackson subtypes) .

@ElementCollection has not been removed (from the 15x branch).

To be clear, there is a ebean-15x branch that has all those other changes already made and that branch has been sitting there ready to go pretty much for a few weeks now. I've been keeping that branch up to date (rebasing). There will be 14.x releases that are pretty much the same as 13.x but with jakarta and javax reversed some deprecated methods removed some improved default values like default jdbc persistBatch size increased (from the current default of 20 which imo is too low).

Thank you so much for the clarification and the work you've put into the ebean!

We're stumbling onto (single table) inheritance being removed as well. Although I understand that inheritance often is overused, polymorphism is a core part of object oriented programming, and removing this will have a great impact on allowed architectures. There is nothing wrong with inheritance, if used correctly.

We've also been happily using Inheritance in some of our entities for many years in a fairly large JEE application ( - million loc) and will be impacted a lot by this removal.

Looking at the diff and the many properties, if statements and tests that were removed I understand the want to reduce complexity in Ebean. Inheritance is also not my preferred strategy to apply to most cases but it's part of the language and the JPA spec. Removing this will make it harder for other developers to make the move from Hibernate (or other JPA implementations) to Ebean.

I much prefer Ebean to Hibernate due to better query builders, explicit saves and the way it deals with loading relations. Still, I'm finding it harder to justify to my team to stay on Ebean.

So firstly, we do have some time to gather feedback and comments on @Inheritance. This means that we will continue to release in parallel 14.x, 14.x-javax, 15.x and these are all kept in sync in terms of bug fixes and feature changes. So 15.x is the same as 14.x but with those features removed including @Inheritance.

We are keeping these releases in parallel because generally speaking people are not reading release notes etc and so there is some expectation of some delay in terms of folks finally getting to update apps and only then coming to here at that point to give feedback - and that's ok but to some extent we can't wait forever for everyone to participate. The first 15.x release was Feb so that's already 7 months ago.

There is some possibility that inheritance makes it back into 15.x and we will probably know in Feb how it's going when 15.x would then have been out there for 1 year. We will have more comments added here, more participation, more feedback etc.

nothing wrong with inheritance

When we have a relationship To an entity bean with inheritance, we can no longer instantiate a reference bean for that without the discriminator. That is, the foreign key value by itself is no longer sufficient to create a reference bean and instead we additionally require the discriminator value and in order to get that ebean will add an extra sql join to that table [that has the inheritance] to obtain the discriminator value. We have this same scenario with database.reference(Class<?>, idValue) - this has to hit the database in the inheritance case [when normally this is just in-memory super cheap].

So this is the first extra cost we get when we use @Inheritance and in a strongly typed language like Java there really isn't any avoiding this issue unless we don't use @Inheritance.

Another thing we hit with single table inheritance is how we real with NOT NULL constraints on properties that are not on the base type. IMO there isn't really an optimal solution there.

Internally within ebean we get complexity when we have an inheritance hierarchy with different relationships [ManyToOne, OneToMany etc] on different parts of the inheritance hierarchy. [We have "different shapes" with "different paths"].

Alternatives to @Inheritance

An alternative approach to @Inheritance is for the ORM not to do it and instead deal with it in the application. For this, the inheritance is "flattened" on the ORM entity and then one approach is for the subtypes to be represented by interfaces that the now concrete entity class implements.

e.g. Vehicle with subtypes Car and Truck. All the properties of Car and Truck are flattened onto Vehicle and Vehicle becomes a concrete type and the only @Entity and Car and Truck become interfaces that Vehicle implements. A method can be added that will return the appropriate interface based on the discriminator value [but this is now owned by the application code].

This approach is along the lines that we pull out the @inheritance functionality from the ORM into the application and now there is literally no "orm functionality" to hide how this works in terms of SQL and the database. There is nothing hidden from the developer here [e.g. when the devs make design decisions like non null or extra relationships the implications of these design decisions are now more obvious]. The upside is that any developer coming here will see how in works and its a pretty direct one to one mapping between the orm model and the SQL database model.

This also means that creating a reference bean for Vehicle is now "cheap" again with no discriminator value required and no extra join required.

For myself, keen to keep any and all feedback coming and see how this plays out.

Remember that 14.x releases will continue and master branch is still 14.x and we are keeping that going for at least another 5 months [so all bug fixes and features still go into master branch which is 14.x and all changes there are merged back into 15.x branch].

I do not understand why single table inheritance would require an additional join. "Single table" says it already: there is only one table, what extra join can there be? All it should take is an in-memory map of discriminator values to actual class, and upon instantiation a lookup in that table which class to instantiate. I can see that partial selects are not an option anymore, because you first need to get the discriminator to decide what fields should be fetched, and with that select you might as well get the other fields as well. But that IMHO is a trade off for the developer to make.

And yes, using @inherited often has drawbacks; in the most optimal setup all the data is in a base class, and the extending classes only have differences on code level. Suppose Vehicle has a method calculateMaxCargo(), which differs only in code between Car and Truck. This is a natural fit for OO languages, where the use of inheritance is perfectly fine, and has no issues with any database constraints. Being forced to use interfaces and implementations is not a natural fit.

But even if there are database complexities, like columns that are NOT NULL but not mapped in all extending classes, it is not the job of the ORM to deal with that. It is a design consideration of the developer; either use Inherited and somehow deal with that NOT NULL, or use another pattern.

Nicely undermining the point I'm trying to make: one could argue that if you want inheritance in the domain model, trying to shoe-horn persistency onto the same model is not a good idea. JPA annotations are a way to re-use the domain model classes for persistency, which may not be at all suited for that. If the domain model classes cannot be easily used, a repository layer with DTO alike classes specifically for persistency (still using the annotations of course) may be needed. Maybe only for the one entity. When persisting a domain class, it first needs to be converted to repository layer classes, and vice versa. Not optimal, but IMHO better than not using natural OO architectures - if they are a good fit for a situation.

But the point remains: there is always a price to pay, in code complexity, in less optimal database constraints, everything is a trade-off. But it should be up to the developer to decide where that price is due; not supporting inheritance in an OO language is IMHO a bad idea.

why single table inheritance would require an additional join

Any table with a foreign key column to a table with inheritance.

e.g. Trip has a ManyToOne to Vehicle ... and Vehicle uses inheritance with Car and Truck subtypes.

The trip table as a fk column vehicle_id

When we query trips, the orm reads the vehicle_id column, but that is not sufficient to create a reference bean for vehicle because it is using inheritance and it needs to create the reference bean with the correct type. It needs the discriminator value, and that is over in the vehicle table. So when we query trip it adds a join to vehicle such that it can read the value from the vehicle_id column AND the discriminator value. This is the case that means we get an extra join required purely due to @Inheritance.