Tuesday, September 11, 2012

SonOfObsidian color scheme mod for IDEA 11

After latest Eclipse Juno installation, and few weeks of working in this, experiencing big performance downgrade and seeing the direction of current product development (like new fancy UI with animations) I realized that this great product has become an overloaded crap (with installation file growth from 90 MB for Helios to 220 MB for Juno - what can acknowledge this), and after almost 10 years of my Eclipse adventure I decided to look for another IDE. I recently tried to test Intellij IDEA 11 (commercial), because I've bought already their other product PhpStorm (really great IDE for PHP).

Few remarks about this IDE after a week of work:
  • very fast
  • great refactoring / code browsing / autocomplete capabilites
  • highly configurable from A to Z
  • it has everything I use besides java (html/css, groovy, javascript, properties editor, svn/git integration, tomcat, hibernate/spring suport, trac integration etc.)
And some worse things:
  • very hard to switch from Eclipse because of completely different keymap - but I accomplished fair comfort of work after this week
  • you need to spend at least a day to configure it for yourself, mainly because it has a big amount of code inspection hints displayed in editor, what makes it looking as the christmas tree (if everything is shown as w warning/hint, you cannot see anything worthful there)
  • I'd say that product is still wanting in some details, and reported few bugs/feature requests to them (but they are responsible)
But generally I find it a very useful and great IDE.

Unfortunately this IDE doesn't come with color schemes for editors, and only the default color scheme is available. I don't like to burn out my eyes with bright white and prefer some dark schemes. I've found one appropriate on the net - SonOfObsidian - translated from VisualStudio by some guy Roger (thanks, Roger).

Unfortunately after loading it into IDE, it turned out that only the Java editor is done well (in my opinion). Multiple editors have dark font on dark background. On the other hand other editors have a big amount of markers with light background, what completely kills the concept of "dark scheme". During this week of work I was trying to fix the most annoying parts, and make some standards between editors (eg. to have keywords, strings, number in similar colors in various editors), and I changed the font size as well to not to exhaust my eyes so much. Here are some examples of this work (before - after):

















Here are the sources for those who share my preferences. They need to be put into ~/.IntelliJIdea11/config/colors.

[UPDATE] Here is the update for Idea 13. 

Wednesday, August 22, 2012

javax.mail and plain, STARTTLS, SSL/TSL connection for POP3 and SMTP without keystore

Today my target was to implement various connections for POP3/SMTP email accounts. incuding:
  • plain connection with no encryption
  • STARTTLS
  • SSL/TSL
The key requirement here was to have it without java keystore file configured for storing and accepting server certificates. This should work transparently for the user.

Having dug through tons of not working examples on the net, and after trying figure out some solutions by trial and error, I decided to take my Plain Old Eclipse Debugger and dive into the javax.mail classes. Here are quick results of HOWTO.

The environment you need is Java Mail API 1.4.5 (1.4 doesn't work properly, due to inability to set custom socket factory for STARTTLS). On the other hand, I don't use plain SMTP connection, but Spring JavaMailSender, but it has exactly the same config (with properties).

AlwaysTrustSSLContextFactory

First thing to have for encrypted connection is to have SSLContextFactory, that trusts all certificates, to skip the verification stage (this is not required for my application). Here is the code:

package com.blogspot.lifeinide;

import java.io.IOException;
import java.net.InetAddress;
import java.net.Socket;
import java.net.UnknownHostException;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

import javax.net.SocketFactory;
import javax.net.ssl.KeyManager;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

/**
 * SSL socket factory accepting any certificate.
 */
public class AlwaysTrustSSLContextFactory extends SSLSocketFactory {

    private SSLSocketFactory factory;
        
    public static class DefaultTrustManager implements X509TrustManager {

        @Override
        public void checkClientTrusted(X509Certificate[] arg0, String arg1) 
        throws CertificateException {}

        @Override
        public void checkServerTrusted(X509Certificate[] arg0, String arg1)
        throws CertificateException {}

        @Override
        public X509Certificate[] getAcceptedIssuers() {
            return new X509Certificate[0];
        }
    }

    public AlwaysTrustSSLContextFactory() 
    throws NoSuchAlgorithmException, KeyManagementException {
        super();

        SSLContext ctx = SSLContext.getInstance("TLS");
        ctx.init(new KeyManager[0], new TrustManager[] {
            new DefaultTrustManager()}, new SecureRandom());
            factory = (SSLSocketFactory) ctx.getSocketFactory();
    }

    public static SocketFactory getDefault() {
        try {
            return new AlwaysTrustSSLContextFactory();
        } catch (Exception e) {
            throw new RuntimeException("Cannot instantiate default AlwaysTrustSSLContextFactory", e);
        }
    }

    public Socket createSocket() throws IOException {
        return factory.createSocket();
    }

    public Socket createSocket(InetAddress address, int port, 
    InetAddress localAddress, int localPort)
    throws IOException {
        return factory.createSocket(address, port, localAddress, localPort);
    }

    public Socket createSocket(InetAddress host, int port) throws IOException {
        return factory.createSocket(host, port);
    }

    public Socket createSocket(Socket s, String host, int port, 
    boolean autoClose) throws IOException {
        return factory.createSocket(s, host, port, autoClose);
    }

    public Socket createSocket(String host, int port, InetAddress localHost, 
    int localPort) throws IOException, UnknownHostException {
        return factory.createSocket(host, port, localHost, localPort);
    }

    public Socket createSocket(String host, int port) throws IOException, 
    UnknownHostException {
        return factory.createSocket(host, port);
    }

    public boolean equals(Object obj) {
        return factory.equals(obj);
    }

    public String[] getDefaultCipherSuites() {
        return factory.getDefaultCipherSuites();
    }

    public String[] getSupportedCipherSuites() {
        return factory.getSupportedCipherSuites();
    }

}

The funny part is that the getDefault() static method is crucial here, because javax.mail classes get the SSLContextFactory from this static method, and not by creating the object using constructor. This is a big trap here that can consume a lot of time to conceive. This also can reveal how the rest of sources are written, what is pretty bad for me (strange initializations by static methods and poorly described string properties, with which you can build a lot of combinations to try make it work).

Now, without a lot of complaining more, the revealed solutions. The thing we want to achieve is to get the connected javax.mail.Store from javax.mail.Session to work with POP3, and configured JavaMailSender (with javax.mail properties) for SMTP.

Plain connection

The easiest one. Implementation for POP3:

Session session = Session.getDefaultInstance(new Properties(), null);
Store store = session.getStore("pop3");
store.connect(host, port, username, password);

And SMTP:

JavaMailSenderImpl sender = new JavaMailSenderImpl(); 
sender.setHost(host);
sender.setUsername(username);
sender.setPassword(password);
sender.setPort(port);
                
Properties props = new Properties();
props.setProperty("mail.smtp.auth", "true");

sender.setJavaMailProperties(props);

SSL/TLS

Now the version working from the beginning through secure channel. POP3:

Properties props = new Properties();

props.setProperty("mail.pop3s.socketFactory.class", "com.blogspot.lifeinide.AlwaysTrustSSLContextFactory");
props.setProperty("mail.pop3s.socketFactory.port", port);
props.setProperty("mail.pop3.ssl.enable", "true"); 

URLName url = new URLName("pop3s", host, port, "", username, password);
Session session = Session.getInstance(props, null);
Store store = session.getStore(url);
store.connect();

And the SMTP likewise:

JavaMailSenderImpl sender = new JavaMailSenderImpl(); 
sender.setHost(host);
sender.setUsername(username);
sender.setPassword(password);
sender.setPort(port);

Properties props = new Properties();
sender.setProtocol("smtps");
props.setProperty("mail.smtps.auth", "true");
props.setProperty("mail.smtps.socketFactory.class", "com.blogspot.lifeinide.AlwaysTrustSSLContextFactory");
props.setProperty("mail.smtps.socketFactory.port", port);
props.setProperty("mail.smtp.ssl.enable", "true"); 

sender.setJavaMailProperties(props);

STARTTLS

The last one. POP3 version:

Properties props = new Properties();

props.setProperty("mail.pop3.ssl.socketFactory.class", "com.blogspot.lifeinide.AlwaysTrustSSLContextFactory");
props.setProperty("mail.pop3.ssl.socketFactory.port", port);
props.setProperty("mail.pop3.starttls.enable", "true");

URLName url = new URLName("pop3", host, port, "", username, password);
Session session = Session.getInstance(props, null);
Store store = session.getStore(url);
store.connect();

And SMTP:

JavaMailSenderImpl sender = new JavaMailSenderImpl(); 
sender.setHost(host);
sender.setUsername(username);
sender.setPassword(password);
sender.setPort(port);
                
Properties props = new Properties();
props.setProperty("mail.smtp.auth", "true");
props.setProperty("mail.smtp.ssl.socketFactory.class", "com.blogspot.lifeinide.AlwaysTrustSSLContextFactory");
props.setProperty("mail.smtp.ssl.socketFactory.port", port);
props.setProperty("mail.smtp.starttls.enable", "true");

sender.setJavaMailProperties(props);

Looking for amount of questions and not working examples on the net, and amount of time I needed to solve it, the guys implemented it this way should get the prize for one of the most obscure solutions in java I was working with.

Thursday, July 5, 2012

Spring 3 and Mule 2 initialized by SpringXmlConfigurationBuilder

If you have a Spring 2 application using MuleESB in 2.x version, and you initialize the mule using SpringXmlConfigurationBuilder, ie. something like this:
public void startMule() throws MuleException {
    SpringXmlConfigurationBuilder builder = 
      new SpringXmlConfigurationBuilder(muleConfigResources,applicationContext);
    MuleContextFactory muleContextFactory = new DefaultMuleContextFactory();
    muleContext = muleContextFactory.createMuleContext(builder);
    runtimeConfig(muleContext); 
    muleContext.start();
}
You may notice that this simply doesn't work in Spring 3;)
After a lot of struggling I've discovered that this is because the DefaultBeanDefinitionDocumentReader interface in spring has been changed from:
protected BeanDefinitionParserDelegate createHelper(XmlReaderContext readerContext, Element root) ...
to:
protected BeanDefinitionParserDelegate createHelper(XmlReaderContext readerContext, Element root, BeanDefinitionParserDelegate parentDelegate) ...
I don't want to migrate my whole application MuleESB 3.x, what seems to have Spring 3 compatible classes, so some fixes to existing mule initialization were required here. For those interested in the same, here is the solution:
public void startMule() throws MuleException {
    Spring3XmlConfigurationBuilder builder = 
      new Spring3XmlConfigurationBuilder(muleConfigResources,applicationContext);
    MuleContextFactory muleContextFactory = new DefaultMuleContextFactory();
    muleContext = muleContextFactory.createMuleContext(builder);
    runtimeConfig(muleContext); 
    muleContext.start();
}
For this you actually need to use my classes fixing this issue - they are available to download here.

Friday, June 22, 2012

High-availability architecure with mysql replication and C3P0

I've recently had a requirement of setting up the application to work in high-availability environment, which among the others involved hardiness for database (mysql) crash. We established with the customer that we want to have a second database server, where the mysql database mirror will be maintained, using master-slave mysql replication. In this model the master mysql server takes care of synchronizing records from itself to the slave server, and the slave server is available in fail-scenario for the application in read-only mode.

This is about database, but what about the code? If you establish a database connection each time the request comes, it sound easy, but we are talking about the Tomcat/Spring/Hibernate application with SessionFactory connected to polled JDBC datasource through c3p0 connection polling libraryHowever, I realized that c3p0 allows to change the JDBC connection params on-the-fly, and when it is done, it resets the existing poll. In such scenario, new connection acquiring requests will be directed to empty poll, and new connections will be established with overwritten parameters.

Therefore, the emerging idea was just to implement the connection supervisor, which can change the main c3p0 ComboPooledDataSource settings, when one of databases is unavailable. The implementation is below.

First class is only the connection configuration class:
import com.mchange.v2.c3p0.ComboPooledDataSource;

public class C3P0SupervisorConnectionConfig {

  protected String user;
  protected String password;
  protected String jdbcUrl;
  
  /** Whether connection is read-only. */
  protected boolean readonly = false;
  
  public C3P0SupervisorConnectionConfig() {
    super();
  }
  
  public C3P0SupervisorConnectionConfig(ComboPooledDataSource pds, 
      boolean readonly) {
    this.user = pds.getUser();
    this.password = pds.getPassword();
    this.jdbcUrl = pds.getJdbcUrl();
    this.readonly = readonly;
  }

  /** Let's omit here the getters and setters, doing nothing */

}
Then the supervisor itself.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Map;
import java.util.TreeMap;

import org.apache.log4j.Logger;
import org.quartz.JobDetail;
import org.springframework.beans.factory.BeanInitializationException;
import org.springframework.beans.factory.InitializingBean;

import com.mchange.v2.c3p0.ComboPooledDataSource;

public class C3P0Supervisor implements InitializingBean {
  
  public static final Logger logger = Logger.getLogger(C3P0Supervisor.class);
  
  /**
   * Connections configuration with priorities.
   */
  protected Map<Integer, C3P0SupervisorConnectionConfig> connections;
  
  /**
   * Main datasource. 
   */
  protected ComboPooledDataSource pooledDataSource;
  
  /**
   * Which one is live currently.
   */
  protected int live = 0;

  @Override
  public void afterPropertiesSet() throws Exception {
    if (pooledDataSource==null)
      throw new BeanInitializationException("Pooled datasource cannot be null for C3P0Supervisor");
    
    if (connections!=null)
      connections = new TreeMap<Integer, C3P0SupervisorConnectionConfig>(connections);
    
    if (connections.get(0)==null) 
      connections.put(0, new C3P0SupervisorConnectionConfig(pooledDataSource, false));
  }
  
  public void poll() {
    if (logger.isDebugEnabled())
      logger.debug("Starts polling JDBC connections...");
    
    for (Integer priority: connections.keySet()) {
      C3P0SupervisorConnectionConfig cfg = connections.get(priority); 
      if (test(cfg)) {
        
        // the connection was tested and it's current connection
        if (live == priority)
          return;
        
        // something happened
        else {
          if (priority<live) {
            if (logger.isInfoEnabled())
              logger.info(String.format("Higher priority connection: %d than " +
                  "current: %d is available again, reconnecting to: %s", 
                priority, live, cfg.getJdbcUrl()));
          } else {
            if (logger.isInfoEnabled())
              logger.info(String.format("Higher priority connection: %d is currently unavailable, " +
                  "switching to: %d and reconnecting to: %s", 
                live, priority, cfg.getJdbcUrl()));
          }
              
          switchTo(priority, cfg);
          return;
        }
        
      }
    }
    
    live = Integer.MAX_VALUE;
  }
  
  protected boolean test(C3P0SupervisorConnectionConfig cfg) {
    Connection c = null;
    
    try {
      c = DriverManager.getConnection(cfg.getJdbcUrl(), cfg.getUser(), cfg.getPassword());
    } catch (SQLException e) {
      return false;
    } finally {
      try {
        if (c!=null)
          c.close();
      } catch (Exception e) {
        logger.error("Cannot close connection", e);
        return false;
      }
    }
    
    return true;
  }
  
  protected void switchTo(int priority, C3P0SupervisorConnectionConfig cfg) {
    if (logger.isDebugEnabled())
      logger.debug(String.format("Switching to priority=%d: %s", priority, cfg.getJdbcUrl()));
    
    synchronized (pooledDataSource) {
      live = priority;
      pooledDataSource.setJdbcUrl(cfg.getJdbcUrl());
      pooledDataSource.setUser(cfg.getUser());
      pooledDataSource.setPassword(cfg.getPassword());
    }

    // TODO Handle readonly connections
  }

  public Map<Integer, C3P0SupervisorConnectionConfig> getConnections() {
    return connections;
  }

  public void setConnections(Map<Integer, C3P0SupervisorConnectionConfig> connections) {
    this.connections = connections;
  }

  public ComboPooledDataSource getPooledDataSource() {
    return pooledDataSource;
  }

  public void setPooledDataSource(ComboPooledDataSource pooledDataSource) {
    this.pooledDataSource = pooledDataSource;
  }

}
The supervisor takes the bunch of prioritized JDBC configurations, and when the 0-priotity connection fails (which is the source ComboPooledDataSource connection), it tries to connect to another connection, accordingly to their priorites. When the higher priority (lower priority number) connection is back, it reconnects main datasource to the higher priority connection.

The readonly parameter is for further usage in application. One should, depending on this param, "do something" in application, like eg. remove "Save" buttons, or generate some message, that application works in read-only mode.

Now, what remains is to configure the supervisor bean and the Quartz Scheduler job detail (or whatever scheduling technology the application uses). Here is the exemplary bean config together with quartz part:
  <!-- the main application datasource -->

  <bean id="mainDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
    <property name="user" value="${db.main.username}"/>
    <property name="password" value="${db.main.password}"/>
    <property name="jdbcUrl" value="${db.main.url}"/>
    <property name="driverClass" value="${db.driverClassName}"/>
  </bean>

  <!-- the supervisor bean -->

  <bean name="c3p0Supervisor" class="C3P0Supervisor">
    <property name="pooledDataSource" ref="mainDataSource"></property>
    <property name="connections">
    <map>
      <entry key="1">
        <bean class="C3P0SupervisorConnectionConfig">
        <property name="user" value="${db.spare.username}"/>
        <property name="password" value="${db.spare.password}"/>
        <property name="jdbcUrl" value="${db.spare.url}"/>
          <property name="readonly" value="true"></property>
        </bean>
      </entry>
    </map>
    </property>
  </bean>

  <!-- the quartz job and scheduler -->

  <bean id="c3p0SupervisorTask" 
      class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
    <property name="targetObject" ref="c3p0Supervisor" />
    <property name="targetMethod" value="poll" />
    <property name="concurrent" value="false" /> 
  </bean>
  <bean id="c3p0SupervisorTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerBean">
    <property name="jobDetail" ref="c3p0SupervisorTask" />
    <property name="repeatInterval" value="60000" />
  </bean>

  <bean id="schedulerFactoryBean" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="triggers">
    <list>
      <ref bean="c3p0SupervisorTrigger" />
    </list>
    </property>
  </bean>
Using the config above, after the first database failure, the application won't respond during 60secs and then will be switched to the second database. The "unavailibility time" depends on the scheduler repeatInterval settings.