Thank you, blogspot

It was almost seven years ago when I started to write on blogpost (Blogger right now).

From the beginning it was a heavy technology blog about my software development, and I have never thought that someone would read these articles. Now, at the end of 2016 I have about 86.5K unique views, what means more than 30 visitors daily, on average. This is amazing. Thank you people (and bots :D)!

Anyway, I still plan to write and make a little more software. However I decided to move out from blogspot. Previously I wanted to write an article why I decided to leave blogspot, but I've recently found Trisha Gee post about why she is moving from blogspot to hugo. This is a perfect article, I can only repeat this explanation, so it's better to just put a link here :)

So, I'm closing this blog, and moving to hugo (my own choice is hugo - however other static site generators look great as well) and my own domain lifeinide.com.

Reads vs writes in production database of web application

I recently think about CQRS-like patterns we partially used in current application, and pros'n'cons of migrating more towards full CQRS, which is mostly about separating reads from writes. Previously I assumed that in usual application "we may have probably 100 times more reads than writes", what would qualify to have a value from these two models separation. I wanted to check it later on the production, but I forgot. Now we are close to start a big redesign of the whole application, and I've just remembered about that. So, how does it look for real production? This is data from postgres pg_stat_database of one of my projects:

tup_returned   | 58956902163

tup_inserted   | 3348809

tup_updated    | 4557408

tup_deleted    | 245501

Does it look that I made a big mistake previously and we have ~7200 times more reads than writes? It will be funny to investigate that in the near future...

How to avoid subclasses join in Hibernate model

This article will be about multiple joins in Hibernate model during super class fetch in JOINED inheritance mapping strategy, and generally how I significantly improved our system performance using some simple patterns. There's a lot of stuff on internet about this (usually wrong approach), so here I'll present my working solution for this problem.

Few words about motivation. People from Hibernate claim that this is not necessary to avoid joins, because join in nowaday databases is fast. This might be not a problem for, for example, 4 subclasses to make 4 join on each query. But what I personally really like in Hibernate, is that you can really model your world with classes and automatically map it to database. Really modeling your world usually means that you have a really lot of super and subclasses, and you may end up with for example 30-40 joins in simple from BaseEntity query, or even you may hit 61 joins limit (if you use mysql).

And here comes the problem. First and foremost, even if join is fast, I don't want to have 40 joins on simple queries. Even if it didn't influence the performance (but after my tests on big system - it does), I would like to be able to debug some queries in plain SQL, and I wouldn't like to have queries extending to 5 screens for that. So, here start things like adjusting the model to queries, what means that our real world model is no longer a real world model, but it becomes hibernate oriented model. That's second thing I hate to have in the project. I like to keep things clean.

OK, now we can delve into the problem.

Joined strategy defaults

Let's consider following simple model:

This is how it's reflected in database (I use postgresql for this example):

aero=> \d super;

              Table "public.super"

   Column   |          Type          | Modifiers 

------------+------------------------+-----------

 classname  | character varying(255) | not null

 id         | bigint                 | not null

 super_prop | character varying(255) | 

Indexes:

    "super_pkey" PRIMARY KEY, btree (id)

Referenced by:

    TABLE "suba" CONSTRAINT "fk_5yjvt9lkf5b24nyxv59n3kbgj" FOREIGN KEY (id) REFERENCES super(id)

    TABLE "subb" CONSTRAINT "fk_q4aajuitkvsk93mn07t66l0pi" FOREIGN KEY (id) REFERENCES super(id)

aero=> \d suba;

              Table "public.suba"

  Column  |          Type          | Modifiers 

----------+------------------------+-----------

 subaprop | character varying(255) | 

 id       | bigint                 | not null

Indexes:

    "suba_pkey" PRIMARY KEY, btree (id)

Foreign-key constraints:

    "fk_5yjvt9lkf5b24nyxv59n3kbgj" FOREIGN KEY (id) REFERENCES super(id)

aero=> \d subb;

              Table "public.subb"

  Column  |          Type          | Modifiers 

----------+------------------------+-----------

 subbprop | character varying(255) | 

 id       | bigint                 | not null

Indexes:

    "subb_pkey" PRIMARY KEY, btree (id)

Foreign-key constraints:

    "fk_q4aajuitkvsk93mn07t66l0pi" FOREIGN KEY (id) REFERENCES super(id)

So, we have a superclass table, and two subclasses tables (in JOINED inheritance mapping strategy).

I put some data there:

sessionFactory.getCurrentSession().save(new SubA());

sessionFactory.getCurrentSession().save(new SubB());

And can observe how this data is distributed in database:

aero=> select * from super; select * from suba; select * from subb;

 classname | id | super_prop 

-----------+----+------------

 SubA      |  1 | superProp

 SubB      |  2 | superProp

(2 rows)

 subaprop | id 

----------+----

 subAProp |  1

(1 row)

 subbprop | id 

----------+----

 subBProp |  2

(1 row)

So, as expected, the data is distributed around all tables, depending on subclass type.

Now I ask hibernate about superclasses list from database using from Super HQL query, and the final SQL query built by Hibernate looks as follows:

select [...] from SUPER super0_ left outer join SUBA super0_1_ on super0_.id=super0_1_.id left outer join SUBB super0_2_ on super0_.id=super0_2_.id

The example above shows the broached problem (we imagine now, that we have 50 subclasses here, of course :) ).

Explicit polymorphism

People on the internet often try to avoid multiple joins problem using explicit polymorphism. To be completely honest, I tried to use this hibernate feature to check if it solves the problem. The misunderstanding about this feature is probably located in Hibernate javadoc, and it claims that explicit polymorphism means: This entity is retrieved only if explicitly asked.

So if we don't ask about SubA class, for example, we should fetch only instances of Super. Let's try it. 

I tested two models, first one with explicit polymorphism on superclass:

And second one is with explicit polymorphism on subclass:

Both neither influence the table model, nor the data. And for both in response to from Super we have:

select [...] from SUPER super0_ left outer join SUBA super0_1_ on super0_.id=super0_1_.id left outer join SUBB super0_2_ on super0_.id=super0_2_.id

It just doesn't work as the expected.

So, what is the explicit polymorphism for? It looks that the only application of this features, is so called lightweight class pattern, and it may be used only in situation when two or more of classes are mapped to the same table. But it doesn't solve our N joins problem.

Single table + secondary tables

So how to solve the problem of N joins? I use following pattern, involving SINGLE_TABLE inheritance mapping strategy and secondary tables. Let's first look on the class model:

Everything is mapped to single table, but for all derived classes I define secondary table (using @SecondaryTable annotation) and tell Hibernate that this table should be fetched using additional select (using @Table annotation). The additional hassle here is that for all properties from subclasses, I need to mark them by @Column (or @JoinColumn - for single ended associations) to tell Hibernate to put this property to the secondary table. This really should be done automatically (if whole class has secondary table definition), but this is one of many other things that Hibernate people refuse to do.

Let's take a glance on the database model:

aero=> \d super; \d suba; \d subb;

              Table "public.super"

   Column   |          Type          | Modifiers 

------------+------------------------+-----------

 classname  | character varying(255) | not null

 id         | bigint                 | not null

 super_prop | character varying(255) | 

Indexes:

    "super_pkey" PRIMARY KEY, btree (id)

Referenced by:

    TABLE "suba" CONSTRAINT "fk_5yjvt9lkf5b24nyxv59n3kbgj" FOREIGN KEY (id) REFERENCES super(id)

    TABLE "subb" CONSTRAINT "fk_q4aajuitkvsk93mn07t66l0pi" FOREIGN KEY (id) REFERENCES super(id)

              Table "public.suba"

  Column  |          Type          | Modifiers 

----------+------------------------+-----------

 subaprop | character varying(255) | 

 id       | bigint                 | not null

Indexes:

    "suba_pkey" PRIMARY KEY, btree (id)

Foreign-key constraints:

    "fk_5yjvt9lkf5b24nyxv59n3kbgj" FOREIGN KEY (id) REFERENCES super(id)

              Table "public.subb"

  Column  |          Type          | Modifiers 

----------+------------------------+-----------

 subbprop | character varying(255) | 

 id       | bigint                 | not null

Indexes:

    "subb_pkey" PRIMARY KEY, btree (id)

Foreign-key constraints:

    "fk_q4aajuitkvsk93mn07t66l0pi" FOREIGN KEY (id) REFERENCES super(id)

The really good thing here is that the model is not changed at all, so if you experience problems with multiple joins, you may easily convert your JOINED hierarchy mapping strategy, to such model.

To be completely clean let's take a look on the data:

aero=> select * from super; select * from suba; select * from subb;

 classname | id | super_prop 

-----------+----+------------

 SubA      |  1 | superProp

 SubB      |  2 | superProp

(2 rows)

 subaprop | id 

----------+----

 subAProp |  1

(1 row)

 subbprop | id 

----------+----

 subBProp |  2

(1 row)

And finally on from Super HQL translated to SQL:

select [...] from SUPER super0_

select super_1_.subaprop as subaprop1_83_ from SUBA super_1_ where super_1_.id=1

select super_2_.subbprop as subbprop1_84_ from SUBB super_2_ where super_2_.id=2

There's no any joins here anymore.

Is this right?

If we don't want to have a big join, we need to consider that Hibernate needs to load all entities somehow. Here, we replaced the join with subclasses fetch in secondary selects, what is called the "N+1 select problem". So is this right or not to use this pattern?

I'd say: it depends. If you have 4-5 subclasses probably the join would be faster. If you have 50 joins, it probably wouldn't. Moreover if you hit 61 max joins in mysql, you cannot execute any query anymore, so this is the only sensible solution.

What I really do when I see such big joins in a project, I try to measure performance of both approaches, and choose the best solution for real database with real data and real queries we execute on underlying model. I cannot say that "a lot of joins" is an antipattern, or "N+1 select" is an antipattern. It just depends on some conditions. For example if you fetch data for the view with pagination, you usually don't fetch more that 10-20 entities in a single query, so N+1 select problem is very little here. If you want to fetch big amount of data, this probably will be a problem, but also there's probably something wrong with you app, if you need to fetch so much data from database.

So everything should be tested and used with thinking (what can be said about clearly everything in software development :) ).

Further improvements

This article I made after huge system refactor in current project. Translation from JOINED to the proposed model above was only one of the things I refactored. It turned out that uncontrolled eager fetch of various associations can also produce unbelievable queries. You may say that if one have uncontrolled eager fetch, one also have a bad model. That's true, this is why the refactor was needed - to convert bad model to best performing model. I always follow the pattern that we focus on business system value in a first place, and the optimization stage comes at the end.

So what I did more to optimize the model. First and foremost now I consider eager fetch for collections as a bad pattern. But this is not a problem, because all to-many associations are lazy by default, and if they become eager, this is a developer choice, so he should know what is he doing and why. But the problem is with single ended associations. All single ended associations are eager by default, and they can produce something really unwanted regarding application SQL.

You may control this "join depth" using hibernate.max_fetch_depth property. If you set it to 0, hibernate wouldn't make any deep joins fetching your entities. But this doesn't mean that you convert your single ended associations to lazily fetched. They still will be fetched (at least the first level of single-ended associations in relation to entity you want to fetch) using additional selects. To really convert them into lazy one, you also need to use additional annotation @LazyToOne. That really converts the objects in the relations to lazy proxy (you have then very ugly lazy proxies in these relation ends, but this still can be overcome using hibernate instrumentation in compilation stage).

So, is this right to have everything lazy and how does it influence on queries performance? No, it's not right to say so. I just don't believe in good perfomance, when you once can say "this entity should have these relations fetched lazily, and these relations we will use frequently and they should be fetched eagerly". This simply depends on the given entity usage. Sometimes, when you develop your model, it seems you can say something like this (usually thinking about fetching the list for the view, where this entity is displayed), but then it turns out that you also use list of the same entities in various different services, and you just don't need these eagerly fetched association in these queries. 

So I believe this is just not OK to define on your model what should be fetched eagerly and what should not. There's a different place where you can do it - in repository queries. For each query you make, you usually know the exact usage of this query, and you can define appropriate joins in HQL or by using setFetchMode(), when you use Criteria API. And this is the right place where it should be defined.

So what I did, when I was doing this refactoring, can be limited to these activities:

1. Convert some entity hierarchies from JOINED to SINGLE_TABLE with secondary table, to controll the subtree fetching.
2. Make everything lazy, and
3. Define appropriate joins on repository queries level, depending on the query usage.

After this job, with our high load database test we achieved about 560% of performance boost.

Statistics queries with time range for PostgreSQL and Hibernate

Few days ago I was asked to implement charts with some finance data using JS charting library, in the application working on Postgres database. There's a lot of finance stuff in the database stored every minute and I just needed to show this data on charts.

OK, but charts are nice if you can restrict amount of your data returned from the server to some limited number of items. Returning 1000 items would both - make chart very heavy and unreadable and consume a lot of net traffic. I assumed 50 as the average number of items that are nicely displayed on such charts and are also acceptable from the traffic point of view.

It would be nice to split time data to some periods that always assert no more than 50 items and then return only such statistics to the client side. And here I discovered very nice and applicable postgres function, which is date_trunc.

Equipped with this weapon I figured out universal solution to calculate such time-based statistics for postgres database using Hibernate. Firstly I designed a very simple enum to calculate right time-grouping criteria:

It'd be nice then to have everything working with whatever entity type and to be able to customize grouping query with Hibernate criteria, because this is right abstraction to build SQL query step by step from filters hypothetically associated with chart filters:

It's also quite simple to autodetect best data granulation for the chart (eg. if it is a hour, day, month or something else) depending on the base time range for the chart.

Modal windows available by URL binding in AngularJS with ui-bootstrap and ui-router

It's well known how ui-bootstrap supports modal windows. These modals don't have any URL and cannot be accessed by URL, though. This is very good, because they should stay as the main view subview only, don't mess with the browser history and first and foremost they shouldn't kill underlying main view controller, what would reset its current state (like filters applied, pagination, etc). This would be a problem using URL-based modals, especially with ui-router which seems to be a standard for now.

On the other hand it's good to have a URL-bound modals for one usage. When you send to people notification emails with links to perform the requested action. I mean such emails like "Attention, you have to do something in the application. You can do it with this link.". In typical scenario this action is already available in the application, usually on modal window. You just want to send this modal link to the user, to trigger requested action, and if you can't bind your modal to the URL, you can only send him some more or less awkward message, like "Use this link to login to the application, and then ... dig yourself to find where you can do it.".

Here is a little ui-router hack to achieve both things - normal, unobtrusive modals, and URL bindings.

NOTE, that this is not about having modals with ui-router URL address. There're many implementations on stackoverflow, and also example in ui-router FAQ. But these solutions have a very big flaw - their modals kill underlying main view controller, and when you're back, you're really back (like with the back button) on the main view with new controller instance and reset whole state, what is unnacceptable for real world applications. Maybe next time I'll think a little about how to achieve this point without this problem, but now it's about having normal bootstrap modals also bound to URL addresses in ui-router.

Firstly, you need to define in your main HTML the location where the modal will be rendered, if it's URL-bound. Let's make it simple as modal.html (all codes are inserted at the bottom of this post). You can define surrounding view as main view, and use it for the modal target (when the modal window is rendered) or the whole application layout target (for views other than modal windows) by appropriate router substates configuration and CSS.

Afterwards, let's define simple module.js with modal content, one URL parameter and resolve function that provides the resolved object to the modal controller, from the server side. At this point your modal is exposed at /modal/{ID} URL, and you may use it in your notification emails.

Now, let's support our modal as the regular ui-boostrap modal for other application views. This is done in modal.service.js by simple hack and moving parameters from ui-router configuration to ui-bootstrap $modal instance. In any view now, instead of calling $modal, you can call modalService(viewName, stateParams) to open your modal window previously defined in ui-router configuration, and at the same time you have this modal available by URL.

Source code

Distributed hibernate search with ApacheMQ and Spring

This catching title will be not about distributed hibernate search, but something really close :)

The case I've recently solved was a quite different. I have a frontend application that uses hibernate database and hibernate search, and then I needed to add additional application - let's call it an integration server, which exposes some API webservices for the overall system. Integration server uses the same database as the frontend application, and enables clients to put data to the database using its webservices. Both applications exist on two different physical servers, as well.

Everything looks simple unless you start thinking about hibernate search update from the integration server, while the index is located solely on the frontend application side, because only this part of the system uses it. When you put data to the database from the integration server, the frontend application full-text index is not updated, of course. I've been looking for simple solution to overcome this problem.

Firstly, let's take a look at what the hibernate search proposes. It supports distributed hibernate search index, with master-slave replication, where all nodes are connected using JMS. This solution was something I didn't really need because only one node uses the index for searching. Moreover this solution is based on periodical index replication, what causes the index is up-to-date on each node only after some interval. Finally, I didn't like this solution because it uses JNDI, what is not really Spring way to solve the problems (I don't really like JEE, I only like to work with lighweight Java application stacks).

So I figured out the solution with following prerequisites:

1. I don't want to use replication because I only need to use search on frontend application side.
2. I want to keep index physically on the side really using it, ie. on the frontend application side.
3. When the integration server updates database data, I need to update the index.
4. I don't want to use JNDI.
5. We are already using ApacheMQ, I want to use it for this solution as well. AMQ broker is already located on the integration server side.

OK, let's delve into the solution. Here is the spring config of important beans on the frontend side:

What do we have here? Standard hibernate session factory on which I'm showing the hibernate search config, that creates and uses local lucene index. Then AMQ connection factory, that connects to the broker running somewhere else. Nothing special. The only interesting bean is RemoteHibernateSearchController, which is derived from standard AbstractJMSHibernateSearchController, that already is a JMS message listener, and only needs to provide hibernate sesssion from our session factory:

Now let's take a look at integration server config, which is a little more interesting:

Same session factory, but configured in the other way. As the backend we use AMQBackendQueueProcessor - our own implementation, shown for a while, not the standard "lucene" implementation. Our implementation will delegate all hibernate search insert/update requests to the listnening frontend RemoteHibernateSearchController, through the JMS queue named "queue.search".

I need to mention here a thing. The integration server doesn't use hibernate search for searching at all (it is only insert/update oriented). But if we have enabled hibernate session for integration server, we need to have at least some index to work. This index won't be updated ever (AMQBackendQueueProcessor will delegate all updates to frontend index through JMS) and will never be read. So I decided to use "ram" provider, which holds whole this few-bytes index in RAM memory. You can, anyway, use any implementation you want - this is only a fake index.

AMQ configuration comes then, and we define only one queue here (this config is redundant, but you can add some parametrization to the config made this way). This is the part really starting the broker using TCP transport (in test environment both servers are run on localhost). Note, that AMQ connection factory connects to the (local) broker using VM transport, and doesn't start its own broker itself.

Now few words about the initialization order. We will use a little trick to bind AMQBackendQueueProcessor to Spring for a while, so the order is important. When session factory bean creates the session factory, hibernate search worker backend needs to be able to work right away. Our backend will work using AMQ, so AMQ needs to be initialized before the session factory bean is initialized. In the example it is done by depends-on attribute, and sessionFactory bean depends here on amqBroker bean (indirectly, the dependency goes through appContextProvider bean, which is also required to be initialized when session factory bean starts).

ApplicationContextProvider bean just accomplishes the commons hack to access Spring beans from non-spring aware code:

And finally AMQBackendQueueProcessor overrides JNDI-related code of standard JmsBackendQueueProcessor JMS connection factory lookup with spring-based implementation, using our container config and appContextProvider hack:

BIRT with Hibernate using POJO-s

In previous BIRT versions there were three ways to use BIRT with your Hibernate datasource:

1. Just plain SQL on Hibernate database.
2. Scripting data source.
3. Hibernate ODA data source.

Previously I've been using plain SQL, but with complex database model it becomes very difficult (a lot of properties, joins and other relationships). Scripting data source using javascript is a bit too awkward for me, and certainly adds another level of application complexity, and another level of layers where "the things may go wrong". Finally, Hibernate ODA data source from JBoss tools - this is what I wanted to use in older times, when I was using Eclipse, but for plain no-jboss application there were a lot of problem with these tools. They seem to be related to SEAM. Anyway AFAIR I had more problems with maintaining proper hibernate configuration in Eclipse (especially that it was partially annotation, and partially XML based) than the value of having working hibernate queries in IDE.

Recently, during a work on another application using BIRT, I started to think about another approach: how to delegate BIRT datasources to pure Java providers, that could be parametrized easily from runtime (without BIRT API) and which return my own arbitrary Java object models. It can be hibernate entity model, it can be something else, like for example DTO objects model, whatever I want to use in report and I consider better structured for particular reports. The most important part here is that these models would refactor easily with compiler support during the application development.

After a little research and tests it turned out feasible using BIRT POJO datasource, and now I consider it the best way of making BIRT reports from Java. The full source code of my example is on github, and the description is below.

Model + Data Set

First let's have a look on the model:

I wanted to use some associated objects in the model, to see how it works in BIRT, so what we have here is the company under which there are few departments, and our target report will be the list of departments grouped by companies, displayed directly from the database. To be more specific - directly from Hibernate queries.

The first thing is that we need some data set that can be used on BIRT design time. POJO data source is pretty badly documented and to reveal how to do it I needed to read some of BIRT sources. But what I discovered during this work is that the data set for the BIRT design time can be any class providing public Object next() method (it goes then through PojoDataSetFromCustomClass in the BIRT engine). So here is my mock data set implementation:

There are no bingings to Hibernate at all during this stage so we can prepare our object lists for the report design time independently from the database itself. My mock data set returns companies with "Mock" prefix to make it feasible to tell apart mock objects, from the real ones. Under every company there are three departments created (IT, HR and Sales).

Now we need to package everything (entity model and mock dataset) to a single jar that will be connected to BIRT for a while.

Report design

Now we can start preparing new BIRT report starting from the POJO data source:

Then we need to add our jar to data source config, and check the option below:

Now we need to configure POJO data set, giving our mock data set class name as the objects source:

The important part here is the key under which BIRT will look for the data set (APP_CONTEXT_KEY_MOCKCOMPANYDATASET). This is not important yet, but will be important in real runtime, where we will switch the mock data set to real one.

Now we can just select the POJO class, and configure column mappings:

I used property of associated Department entity here to see how the column mapping will be done. It looks good, and now on the Preview Results tab we can see how the column data is generated by BIRT from the underlying Company objects:

It fetches them as the list of departments that could be grouped by company, which seems to be good initial data set to create intended report.

On the layout view I just put the list bound to this data set, and grouped items by company id:

Now it's possible to generate mock report from the BIRT designer:

The runtime

In the runtime now what we want to do, is to replace the mock data set from the design stage, with some real data fetched from the database by Hibernate using entity model. A lot of this code is just to start the report engine, but I described below what's important here:

In line 2 we create BIRT application context, that is just a HashMap. To this map we need to pass our objects collection, under the key configured on design time. My companyRepository.findAll() method returns Hibernate Query. The "collection" passed to the application context can be Collection itself (so the list() is an option here), but I prefer to use Iterator (iterate() option), because for bigger reports it'd fetch data from database sequentially, not in a big single list.

For really inquisive people - BIRT can convert following objects to its internal IPojoDataSet representation:

1. Any object providing public Object next() method, using PojoDataSetFromCustomClass.
2. Collection, using PojoDataSetFromCollection.
3. Array of objects, using PojoDataSetFromArray.
4. Iterator, using PojoDataSetFromIterator.

In line 27, when the task is ready to run, we need to pass the application context to the task, to replace mock data set, with our real runtime data set.

The finally rendered report (all entities in database have "Company" prefix) looks this way:

Everything above can be easily built and tested from the command line from my birt-hibernate-example sources.

Database locks with Spring, Hibernate and Postgres

In the current project we faced the problem of concurrent changes to database, for the data that should be accessed sequentially. Imagine you have the customer's bank account where he can withdraw the money. If the customer is not a person, but company, and if he can have multiple users accessing the bank application, without any locks there's a chance for situation where two or more users depute transfers, that exceed the account balance, but because data is accessed concurrently, they both can make payoff. 

This is the simplest example, but one can imagine a lot of more such cases, that influence a lot of different applications. The lock for the single customer is simple, but imagine if we need lock for more than one customer in the same time. For example if two customers make some transaction between them, and the transaction depends on the account balance on both accounts. In such instance in single database transaction there are both sides required to be locked while the transaction lasts, to avoid the same problem.

Regarding such case we need also to be aware of deadlock problem. If the single row lock is an atomic operation for database, two locks are two operations, and this can produce following deadlock:

1. In transaction A customer 1 is locked.
2. In transaction B customer 2 is locked.
3. In transaction A customer 2 is locked (is waiting).
4. In transaction B customer 1 is locked (deadlock: A is waiting for 2, and B is waiting for 1, both are locked).

But first let's review the possibilities of what can we use for dealing with this problem in usual database application. Our exemplary stack here is standard Java (Spring + Transactional AOP-s + Hibernate) and Postgres database.

1. Language-level locks, by usage of object monitors or java.util.concurrent.locks package. Problems: doesn't work for clustered applications and dealing with deadlocks is not obvious and difficult.
2. Transaction isolation level. Here we'd have following opportunities (see postgres reference):
1. Read uncommited - this would be rather a gambling, but for postgres this level doesn't exist.
2. Read commited -  this is the default transaction isolation level causing the problem, because both transactions we consider see only data not commited by other transactions, so they can both work on the same balance "snapshot" from transaction start, and they can read the same value for subtracting from database. This level introduces locks on database level by default, but only for update, not for select. After acquiring the lock during update by transaction A, the transaction B is waiting with its update, but after the transaction A is finished, the B proceeds with its update anyway. So we may have the situation: A reads 100, B reads 100, A subtracts 20 from 100 and updates db with 80, then B subtracts 20 from 100 and updates db with 80. In the result the value in db is 80, while should be 60.
3. Repeatable read - looks even worse. Two transactions can never see their changes after they are started. But the update lock works in different way. If two transactions want to update the same row, one of them acquires the lock, and second one will fail with ERROR:  could not serialize access due to concurrent update.  So it should be feasible to deal with our problem using this level and update lock, and even better is...
4. Serializable. It works like repeatable read, but postgres tries to simulate sequential transaction execution. It tries to sort all queries in transaction, so that it looks that transactions are executed one by one. But this is only simulation, and transactions are really executed concurrently, though. Moreover if it comes across the situation where it can't perform two transactions "sequentially", it throw the same exception as for repeatable read.
3. Explicit database locking. There are many options, specified in postgres reference, but the most common is select for update. This solution may cause deadlocks in the way I described above. Postgres fortunately detects such deadlocks and throws appropriate exception when it happens.

So, it looks that we can use specific transaction isolation level here, or use the explicit locking. All these solutions require to have fail-safe scenario when exception is thrown (could not serialize or deadlock exception). But there are following things I don't like if I think about transaction isolation:

• They crash after doing a job, and the job can be significant. For example let's assume that the work costs 1 sec for each transaction, and on the end of this work we have this update that fails. If we have for example 10000 users concurrently, we can waste a lot of CPU.
• Serializable transaction level has its performance requirements. It consumes more RAM and CPU to fulfill its requirements. Again, assuming highly loaded application - do we really need to slow down the database with serializable isolation level to achieve our goals?

The solution devoid of these issues is explicit locking and I decided to carry on with this solution, what I'll describe in further part.

The lock

First let's implement the lock service. The select for update clause in postgres locks the concrete row in database against other select for update-s, but it is still unlocked for ordinary selects. So, for the code we need to acquire the lock we can use select for update, what esures that the rest of application works fine while operation in the meanwhile.

In the real world application we may have a lot of service methods that require lock, so I added here the code preventing to have more than one lock for given customer ID, using TransactionSynchronizationManager resources. On first lock there's transaction listener registered, which cleans these resources. If the deadlock happens in postgres, it is indicated by LockAcquisitionException in java code.

Retry on deadlock

OK, nothing really interesting happened for now, but here comes this interesting part. How can we handle deadlocks properly? It'd be the best to retry whole transaction on deadlock few times, until the lock can be acquired and second transaction finishes.

Here I need to mention the contributor from which I've caught some ideas for retrying transactions - Jelle Victoor. But I consider that his solution has some issues, so I extended it a little. I used the same idea, of using AOP aspect to repeat transaction, but I want to have this working without putting everywhere additional annotations, because in the proposed way I'd need to have a lot of double @Transactional and @DeadLockRetry annotations. So, finally I'd like to have it working transparently with @Transactional.

Moreover let's consider the situation of nested @Transactional services. The usual example is following:

• @Transactional ServiceA.doJob() -> calls...
• @Transactional ServiceB.doJob() -> calls...
• @Transactional ServiceC.doJob() -> and here comes the deadlock exception.

The same problem is when you use separate @DeadLockRetry, because the method with this annotation can be called from other service, and the real method that started the whole transaction is somewhere else, so repeating the code from method with @DeadLockRetry annotation may not work, because we want to retry whole transaction.

The trick is how to detect the @Transactional ServiceA.doJob() service execution and after rolling back whole transaction, retry overall operation. I used similar idea to Jelle - to use AOP proxy (with Aspect4J annotations so remember to add <aop:aspectj-autoproxy /> to your Spring config) and to have transaction manager order 100 (<tx:annotation-driven order="100" transaction-manager="transactionManager" />) and my deadlock aspect with order 99 to ensure it runs first.

If the deadlock aspect comes before transactional, we may detect where is our ServiceA.doJob() using the same TransactionSynchronizationManager as previously. If we are on the top level method, the transaction no longer exists because transactional AOP proxy already rolled it back. If the transaction still exists, this mean that we are not on the top level method, but this is nested transactional method.

The complete source:

Tracking object references in JavaScript

It is a common case in AngularJS to have some model loaded on the main view (like list of objects) and to use these objects in other controllers (like the object details view). Usually it's done by holding the reference of the list object in other controller scope, to interact with this reference. Until these both objects point to each other (both references point to the same object) it's very fine. The changes from the details controller are reflected in a list and contrarywise.

But I frequently come across the situation where at least one of these objects is refreshed from the server (eg. in async comet event) and this relationship is lost. A lot of case-by-case code is required to be written to support such instances on the client side.

Today I've been thinking about tracking the object references generally in JavaScript and how it can be done. Unfortunately it looks that it can't. JavaScript doesn't really support real references like eg. in C language, that could be used to achieve this. In JS function there's no way to have access to reference that carries the object as the function argument, and thus to modify this reference.

But indeed there's a way to have access to the reference - using the closure. Closure holds references to all objects belonging to the closure scope. After a little time of playing I figured out some solution for such hypothetical reference tracker:

And it works :)

Now, we have some more complex case. We are just re-assigning single object holding whole list, and we may have only the item (from the example above) assigned elsewhere. This is not as simple as the previous example, but feasible if we think about the model as persistent objects model. All persistent objects have some unique ID assigned. For example if you use UUID on server side, it can be UUID. If you use the numeric ID, it can be combination of ID and object type (class) etc. You can always figure out easily some other method to have unique ID for you objects some way.

In such instance, to achieve the goal we need to have the previous JSON structure, and to scan new one, looking for objects with the same ID, and then reuse our "reference tracker" above. Here is full source including the new usage:

To have it clean, I  also added cleanup() function to clean the objects from $ref reference, to have clean objects for JSON representation to be sent to server.

Tomcat, Atmosphere and Spring Security

Here I'd like to describe another interesting case I've been struggling with for recent few days. This involves the following use case: enable asynchronous events support for Tomcat/Spring multi-tenancy SaaS application, that can be pushed to listening client groups. To be specific, the event should be channeled to following groups: to specific user, to all users of specific tenant and to all users.

Atmosphere + Tomcat

The fancy new technology for async processing in Java world is Atmosphere, and I use it in this example. Unfortunately I started with horribly preconfigured Atmosphere which apparently locked Tomcat after making some number requests, and it wasn't able to serve more requests. It turned out that the working Tomcat + Atmosphere config is something not so obvious, so let's quickly describe all these problems to move on.

I started with the following maven dependency:

<dependency>

<groupId>org.atmosphere</groupId>

<artifactId>atmosphere-runtime</artifactId>

<version>2.1.7</version>

</dependency>

After a lot of struggling I came to conclusion that there's no way to properly run Tomcat with Atmosphere using this library (at least in 2.1.7 version). I started with standard Atmosphere configuration, which uses native Tomcat async implementation (Comet support). In this scenario there's a bug in Atmosphere which results in using Tomcat BIO support (blocking IO) instead of NIO (non-blocking IO). Finally, you have a thread created for each async request, which is then suspended and moved to waiting pool. When you reach the tomcat thread pool capacity (default is 200) you end up with completely frozen application.

Afterward I changed the implementation from native Tomcat async support to Servlet 3 specification, using following flags:

<init-param>

    <param-name>org.atmosphere.useNative</param-name>

    <param-value>false</param-value>

</init-param>

<init-param>

    <param-name>org.atmosphere.useWebSocketAndServlet3</param-name>

    <param-value>true</param-value>

</init-param>

Using this config, it started to work through Tomcat NIO, but the odd things started to happen as well. For example random freezes on standard request processing, and a lot of java.lang.IllegalStateException: Cannot forward after response has been committed exceptions. Something similar to this guy situation.

After a lot of debugging what is really happening in the Atmosphere and Tomcat threads I gave up and I found the solution with so called "native" atmosphere implementation, what apparently is the same lib with only one class changed with fixed native Tomcat support for Atmosphere (scenario 1), which is called:

<dependency>

<groupId>org.atmosphere</groupId>

<artifactId>atmosphere-runtime-native</artifactId>

<version>2.1.7</version>

</dependency>

And is describe here. It finally works well using Tomcat Comet support and/or native Tomcat websockets support. Additionally it requires /META-INF/context.xml with following content:

<Context>

<Loader delegate="true"/>

</Context>

Atmosphere + Spring

Now something which is simple and can be found in many examples on the net. How to configure Atmosphere so that it can route requests to Spring DispatcherServlet.  To skip unnecessary words, I'll make it quick:

Things that might be explained a little more are following:

1. org.atmosphere.useNative + org.atmosphere.useWebSocketAndServlet3 make it finally clear that we want to go using Tomcat native async support.
2. org.atmosphere.cpr.broadcaster.maxProcessingThreads - this is the limitation to 10 for Atmosphere threads. Atmosphere spawns some threads sweeping suspended requests (eg. by flushing their response buffers).
3. org.atmosphere.cpr.broadcasterLifeCyclePolicy=EMPTY_DESTROY is the lifecycle policy for Atmosphere Broadcaster objects. Usually Broadcaster has assigned some AtmosphereResource-s, representing opened async connections. When all connections for particular Broadcaster are closed, the Broadcaster object may still be held in memory and reused. For SaaS application, that may handle hundreds of tenants and thousands of users concurrently I consider it a bad pattern. EMPTY_DESTROY tells Atmosphere to relase all Broadcaster objects if they don't have assigned any resources, and remove them from memory.
4. org.atmosphere.cpr.AtmosphereInterceptor is the important one here, because after Atmosphere invokes broadcasting operation, the response buffers are flushed periodically with all data written, so they could contain more than a single message at one flush operation. In such instance your client would receive two or more messages in one event listener notification, what is usually unwanted. This can be overcome by using TrackMessageSizeInterceptor on the server side, and trackMessageLength parameter in Atmosphere client.
5. AtmosphereSpringControllerResolver enables direct AtmosphereResource injection to Spring controller.

Atmosphere + Spring Security

Now what we'd like to have is the Spring Security context injected to Atmosphere requests, in order to extract user from the SecurityContextHolder and to apply broadcasting operations on suspended requests. The answer on the question how to do it is simple: you can't.

There are two problems I came across with this subject. First the Spring Security filters aren't applied to MeteorServlet, because it's not a reguler servlet, but CometProcessor, supporting async requests. For such type of servlets only CometFilter can be applied, not a reguler Filter, which is implemented by Spring Security DelegatingFilterProxy. You can overcome this problem, though, by either wrapping the Spring Security filters with your own CometFilter-s, or by overriding the default FilterChain by your own implementation. Anyway, it doesn't work as well.

This is because the SecurityContextHolder default storage strategy is ThreadLocalSecurityContextHolderStrategy, which holds the SecurityContext in ThreadLocal (this is the only production implementation and one cannot imagine different working strategy for this problem). It works well for standard requests, processed in separate threads, but for suspended Atmosphere requests there's a problem. When the resources are swept and buffers are flushed, all this process happens in internal Atmosphere thread pool, and one thread supports many AtmosphereResource-s in single execution, so the SecurityContext can't be bound to the thread, because you end up with an exception, or much worse, with different user authorized than it should be.

So what I do, and I'll show in the further example, is how to extract user directly for HTTP session to be used with AtmosphereResource to create appropriate broadcasters.

There's another remark about this overall architecture. If you can run DispatcherServlet through Atmosphere, you might tend to run your whole application through MeteorServlet wrapper. But, when you consider above facts, that you can't apply normal filters to this servlet, and moreover you can't apply security filters on it, the conclusion is simple: just don't do it. Define your "async" Atmosphere servlet context separately from another regular "sync" DispatcherServlet, and everything will be fine.

Broadcast events to user groups

Before final implementation, we need to understand how Atmosphere and Atmosphere Broadcaster-s work internally. 

In regular request processing, when the request comes, Tomcat takes the free thread from the thread pool (or queues the job in the thread pool queue, if all threads from pool are busy), and delegates the request processing to this thread. The Spring Security filters extract the user from HTTP session and put him to the SecurityContext held in current request thread ThreadLocal. Then the work is delegated to your servlet, response buffers are filled, everything is cleaned out and thread is released back to the pool. The response buffer is then written to the client.

In async request processing Atmosphere waits for incoming requests with its own thread pool. When the request comes, one of these threads receives it (or, like in above situation, the job is queued waiting to release at least one thread from the pool), suspends it, and returns thread to the pool. The suspension figures on releasing the processing thread, while the TCP connection is still opened. All these suspended requests are stored in an internal storage, and can be accessed in any moment in application. For all of them the TCP connections are opened, and one can write to the opened Response object to send async events to the client.

But, how to tell apart one suspended request from another? For our example - how to find all requests sent from all logged users of specific tenant? For such use cases Atmosphere introduces the Broadcaster concept. With a single suspended request (AtmosphereResource) you can associate one or more broadcasters, and use these broadcasters to send events to choosen clients. With each AtmosphereResource there's a single Broadcaster created with random UUID. 

Using this idea and knowing this UUID you may send the async event to each suspended request separately, by choosing appropriate broadcaster. Another Atmosphere concept is MetaBroadcaster. It can be used to send event using all broadcasters fitting to the expression. For example:

1. User A connects to async service, the broadcaster with ID="/UUID-1" is created.
2. User B connects to async service, the broadcaster with ID="/UUID-2" is created.
3. Using MetaBroadcaster you may send data to either first or second user by broadcastTo("/UUID-1", event) or broadcastTo("/UUID-2", event).
4. Or you can send event to all users by broadcastTo("/*", event).

This well concept can be adapted to our use case. Let's assume we have a TENANT_ID and USER_ID, defining our tenant and its user. We need to assign only one broadcaster to each async request to achieve our goals:

1. User connects to async service, the broadcaster with ID="/TENANT_ID/USER_ID" is created.
2. To send event to this particular user, use broadcastTo("/TENANT_ID/USER_ID", event).
3. To send event to all logged users of specific tenant, use broadcastTo("/TENANT_ID/*", event).
4. To send event to all logged users, use broadcastTo("/*", event).

And here comes the implementation with all described above:

Finally, in AsyncDispatcher controller we just need to suspend request using AsyncService.suspend() method, to make it all working together.

Global state in AngularJS controllers

Suprisingly for me the controller state in AngularJS is not preserved between the controller invocations. I at least expected an option to switch it on and off on demand. For the classic application it was difficult to achieve that we may restore the state for a given view (eg. to be on the same page as we were leaving the view), what sounds great for me from the application usability point of view.

But, it's right there, so it can be implemented. However, I don't like using the $rootScope for this, what can be found in many examples on the net, in the same way I don't use globals, because they produce a mess. Here I'd like to propose an elegant solution for this.

I have the service that holds all controller states, and provide the initialization function that may be used when the state hasn't been cached yet. The code snippet:

Java Cryptography Architecture and common encryption usages

JCA can be really tricky to perform simple, common tasks. After latest usage of java cryptography I'd like to present below the simplest usage of cryptography methods, involving mainly symmetric (AES) and asymmetric (RSA/DSA) encryption plus some helper methods.

I don't use specific JCA provider, but let it choose appropriate one. If you want to select specific provider, you must provide additional parameters, usually to getInstance() methods of various algorithm elements. In examples I use RSA1024 (you might change it to DSA) and AES256.

Note about AES256 - to enable this for your VM you need Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. With default policy you can only use AES128.

The amount of "things can go wrong" is pretty big, so if you write your own cryptography util, there's a good thing to wrap JCA exception in something that can be easy caught in the user code.

public class EncryptionException extends Exception {
 
 public EncryptionException() {
 }
 
 public EncryptionException(String message) {
  super(message);
 }
 
 public EncryptionException(String message, Throwable cause) {
  super(message, cause);
 }
 
 public EncryptionException(Throwable cause) {
  super(cause);
 }
 
}

Firstly a simple thing, BASE64 encoding wrappers (to be able to switch implementation, what of course will be never used):

public static String encodeBase64(byte[] data) {
 return new BASE64Encoder().encode(data);
}
 
public static byte[] decodeBase64(String data) throws EncryptionException {
 try {
  return new BASE64Decoder().decodeBuffer(data);
 } catch (IOException e) {
  throw new EncryptionException("Error decoding base64", e);
 }
}

How to generate RSA 1024 keys with SecureRandom:

/**
 * Generates new keypair
 */
public static KeyPair generateKeys() throws EncryptionException {
 try {
  KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
  SecureRandom random = SecureRandom.getInstance("SHA1PRNG");
  keyGen.initialize(1024, random);
  return keyGen.generateKeyPair();
 } catch (Exception e) {
  throw new EncryptionException("Error generating keypair", e);
 }
}

How to encrypt with asymmetric key (public or private):

/**
 * Encrypts data with key
 */
public static byte[] encryptAsymmetric(Key key, byte[] data)
throws EncryptionException {
 try {
  Cipher cipher = Cipher.getInstance("RSA");
  cipher.init(Cipher.ENCRYPT_MODE, key);
  return cipher.doFinal(data);
 } catch (Exception e) {
  throw new EncryptionException("Error key encrypting", e);
 }
}

For above, how to decrypt with asymmetric key (public or private, the different one than used for encryption):

/**
 * Decrypts data with key
 */
public static byte[] decryptAsymmetric(Key key, byte[] data)
throws EncryptionException {
 try {
  Cipher cipher = Cipher.getInstance("RSA");
  cipher.init(Cipher.DECRYPT_MODE, key);
  return cipher.doFinal(data);
 } catch (Exception e) {
  throw new EncryptionException("Error key decrypting", e);
 }
}

How to build symmetric key for AES256 encryption:

/**
 * Builds a random secret key for symmetric algorithm
 */
public static Key buildSymmetricKey() throws EncryptionException {
 try {
  KeyGenerator keyGen = KeyGenerator.getInstance("AES");
  keyGen.init(256, SecureRandom.getInstance("SHA1PRNG"));
  return keyGen.generateKey();
 } catch (Exception e) {
  throw new EncryptionException("Error generating secret key", e);
 }
}

The building of the AES key based on user password is a little tricky. You need to have a salt (N-bytes array) that is used to generate a proper AES key (with a proper length). If you need to recover this key in the future, using just the same user password, you need to use exactly the same salt, so it's probably best to hardcode it somewhere and use for further keys generation (random 8 bytes):

private static byte[] SALT = new byte[]{
 (byte) 0xa1, (byte) 0x22, (byte) 0x33, (byte) 0xa4,
 (byte) 0x11, (byte) 0x22, (byte) 0x12, (byte) 0x22};
 
/**
 * Builds a secret key for symmetric algorithm recoverable by password
 */
public static Key buildSymmetricKey(String password)
throws EncryptionException {
 try {
  SecretKeyFactory factory =
   SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
  KeySpec spec = new PBEKeySpec(password.toCharArray(), SALT, 256,
   256);
  SecretKey tmp = factory.generateSecret(spec);
  return new SecretKeySpec(tmp.getEncoded(), "AES");
 } catch (Exception e) {
  throw new EncryptionException("Error encoding secret key", e);
 }
}

Now, how to encrypt the arbitrary length data block with AES:

/**
 * Encrypts data with symmetric algorithm and password
 */
public static byte[] encryptSymmetric(Key key, byte[] data)
throws EncryptionException {
 try {
  Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
  cipher.init(Cipher.ENCRYPT_MODE, key);
  return cipher.doFinal(data);
 } catch (Exception e) {
  throw new EncryptionException("Error symmetric encrypting", e);
 }
}

Note, that I use ECB as block cipher mode of operation. This is not the safest one, better would be CBC (refer wikipedia). But if you need to recover your data only by AES key, you can't do this. To recover from CBC you need to store your CBC initialization vector together with the password. So, I use ECB for this simple example, to make all it working only with keys.

The decryption for above symmetric encryption is similar:

/**
 * Decrypts data with symmetric algorithm and password
 */
public static byte[] decryptSymmetric(Key key, byte[] data)
throws EncryptionException {
 try {
  Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
  cipher.init(Cipher.DECRYPT_MODE, key);
  return cipher.doFinal(data);
 } catch (Exception e) {
  throw new EncryptionException("Error symmetric descrypting", e);
 }
}

This is all about working encryption/decryption. Now about storing the keys in db. If you'd like to use BASE64 encoding or even to hold everything in BLOB-s, methods below will be useful.

Converting the key to BASE64 is easy:

/**
 * Converts key to base64 encoded string
 */
public static String keyToString(Key key) {
 return encodeBase64(key.getEncoded());
}

But recovering the key from BASE64 or byte[] is another tricky part:

/**
 * Converts base64 encoded string to assymetric key
 *
 * @param publicKey if true returns public key, private key otherwise
 */
public static Key asymmetricKeyFromString(String s, boolean publicKey)
throws EncryptionException {
 return asymmetricKeyFromBytes(decodeBase64(s), publicKey);
}
 
/**
 * Converts bytes to assymetric key
 *
 * @param publicKey if true returns public key, private key otherwise
 */
public static Key asymmetricKeyFromBytes(byte[] bytes, boolean publicKey)
throws EncryptionException {
 try {
  if (publicKey) {
   return KeyFactory.getInstance("RSA").generatePublic(
    new X509EncodedKeySpec(bytes));
  } else {
   return KeyFactory.getInstance("RSA").generatePrivate(
    new PKCS8EncodedKeySpec(bytes));
  }
 } catch (Exception e) {
  throw new EncryptionException("Can't decode assymetric key", e);
 }
}

The same for symmetric key looks much easier:

/**
 * Converts base64 encoded string to symmetric key
 */
public static Key symmetricKeyFromString(String s) throws
EncryptionException {
 return symmetricKeyFromBytes(decodeBase64(s));
}
 
/**
 * Converts bytes to symmetric key
 */
public static Key symmetricKeyFromBytes(byte[] bytes)
throws EncryptionException {
 return new SecretKeySpec(bytes, "AES");
}

Sometimes you also need to convert your private/public keys to PEM format for exporting. Without bouncycastle you need your own method for this:

public static String getPem(Key key) {
 StringBuilder sb = new StringBuilder();
 if (key instanceof PrivateKey || key instanceof PublicKey)
  sb.append(String.format("-----BEGIN %s %s KEY-----\n", "RSA",
   key instanceof PublicKey ? "PUBLIC" : "PRIVATE"));
 else
  sb.append("-----BEGIN KEY-----");
 sb.append(encodeBase64(key.getEncoded()));
 if (key instanceof PrivateKey || key instanceof PublicKey)
  sb.append(String.format("\n-----END %s %s KEY-----", "RSA",
   key instanceof PublicKey ? "PUBLIC" : "PRIVATE"));
 else
  sb.append("\n-----END KEY-----");
 return sb.toString();
}

And this was just last example of simple Java cryptography API.