Configuring Spring Application Using Configuration Annotation

A Spring Module called Java Config has been there for some time now. With the release of Spring 3.0 SpringSource has pulled this module into the core Spring Framework as @Configuration.

How can we use this @Configuration in our application? and should we use it when we have XML and annotations for configuration already?

In Spring 2.5 we find that there is a support for Property Externalization. Which was good in several ways than before. What it meant was, that the properties file could be modified for different build environments by the administrator.

One good thing about this approach is that the administrator need not edit the application context file. Editing verbose XML has plenty of room for errors. Editing the properties file on the other hand is a whole lot simpler and is less error prone.

For example for configuring the datasource this is what we do typically. We used Property externalization in Spring 2.5 using PropertyPlaceHolderConfigurer.

<context:property-placeholder location="classpath:datasource.properties" />
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
		<property name="driverClassName"><value>${dataSource.driverClassName}</value></property>
		<property name="url"><value>${dataSource.url}</value></property>
		<property name="username"><value>${dataSource.username}</value></property>
		<property name="password"><value>${dataSource.password}</value></property>
</bean>

Mailing Google Docs As Attachment On Google App Engine

Our application works with Google Spreadsheet API to update spreadsheets. The application now needed to send mail with an attachment of a google spreadsheet. We posted our query on google docs community here and on Google app engine here and there were no clear directions of implementing it.

Let’s see how we went about implementing this feature.

Google provides its spreadsheet API and it allows you to create and retrieve data in Google Spreadsheets, but it does not allow us to either create or manage their permissions. Google also has Google Documents List API which can be used to create and retrieve a list of Google Spreadsheets. For our case Google Documents List API provides exporting functionality such that we can export Google documents in common formats like pdf, rtf, xls and others. Google also provides java mail api and with its Multi-Part Messages feature we can send attachment as well.

For implementing this, we used spreadsheet API to retrieve spreadsheet entry. Using the spreadsheet entry we used Google Documents List API to export it in the specified format and we used java mail api to send exported data as an attachment.

We used play framework for our implementation but it can also be applied on other java frameworks as well. The class EmailSender first creates a google spreadsheet service and document service.

public class EmailSender extends Controller {

private static final String META_FEED_URL = "https://spreadsheets.google.com/feeds/spreadsheets/private/full";
private static String spreadSheetName = "spreadsheetname";
private static String yourApplicationName = "emailSender-version1";
private static String userEmail = "yourmail@gmail.com";
private static String recepientEmail = "recepientmail@gmail.com";
private static String password = "password";


public static void emailSpreadsheetAsAttachment() throws AuthenticationException, MessagingException, IOException {
	SpreadsheetService spreadSheetService = createSpreadSheetService();
	DocsService documentService = createDocumentService();
	SpreadsheetEntry spreadsheetEntry = loadSpreadSheet(spreadSheetService, spreadSheetName);
	byte[] spreadSheetData = getSpreadSheetData(spreadSheetService, documentService, spreadsheetEntry);
	sendMailWithAttachment(spreadSheetData);
	}
. . .
. . .
private static SpreadsheetService createSpreadSheetService() throws AuthenticationException {
	SpreadsheetService spreadSheetService = new SpreadsheetService(yourApplicationName);
	spreadSheetService.setUserCredentials(userEmail, password);
	return spreadSheetService;
	}

private static DocsService createDocumentService() throws AuthenticationException {
	DocsService docsService = new DocsService(yourApplicationName);
	docsService.setUserCredentials(userEmail, password);
	return docsService;
       }
. . .
. . .
}

Managing Wicket Serialization Problem On Google App Engine

There are several problems of using Wicket on Google App Engine. We are porting a Wicket application on Google App Engine and faced several issues. This post will look at the problems we may encounter while working on a Wicket project on Google App engine and how may we overcome them.

Google App engine will it play suggests that if you have a wicket project you follow this workaround your Wicket project will start working on app engine. Still, one of the error you may encounter while working on app engine are Wicket Serialization errors.

Wicket is very powerful in maintaining application/session state. This enables for example a nice usage of back/forward buttons. In a typical Wicket scenario, this is backed by the DiskPageStore implementation. But on Google App Engine we can’t use the DiskPageStore as it relies on writing to the filesystem. Therefore HttpSessionStore is used in Wicket Application class. In this implementation Wicket components and associated model will be saved in Session and therefore in datastore.

This may result in the session getting bloated till we encounter this error com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call datastore_v3.Put() was too large on datastore.

The way to manage this problem is that we use a different pageStore implementation which uses google memcache instead. If memcache implementation is used then the data: that is wicket components and its models will not be associated with the Session and therefore will not end up in datastore.

Memcache also has a limit on the data size limit and you may get an exception if data to be entered in it is high "Caused by: com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call memcache.Set() was too large."

MemcachePageStore that we used has a MAX_PAGES_PER_MAP instance variable and using this we can avoid the "memcache.Set() was too large" errors. Let’s look at how we went about implementing it. We tell our Wicket Application class to use Memcache based page store instead of HttpSessionStore.

public class EhourWebApplication extends AuthenticatedWebApplication {
       . . .
       @Override
	protected ISessionStore newSessionStore() {
		return new SecondLevelCacheSessionStore(this, new MemcachePageStore(3));
		// return new HttpSessionStore(this);
	}
       . . .
}

Managing Cold Start On Google App Engine

One of the major problem working on Google App engine is managing cold start. Well what is it?

Google App engine spins down the application when it is idle. On Google App engine, application does not get a permanent instance of java virtual machine. If there are no activities for a while then the JVM goes cold and application will have to start again to render a request. This is cold start and your application will be sluggish if you render a request to it after a while.

GAE team has accepted the issue and introduced “Ability to reserve instances to reduce application loading overhead” into their roadmap.

This is an issue we traditionally are not aware about while developing an application on Google App engine. If we are working on a greenfield project and we have a liberty of choosing the frameworks then we can stay away from heavy frameworks.

The application we are porting is Spring/Wicket/JPA tehnology stack and we like others were suffering from cold start. When the application warms up all the Spring beans are initialized again. Since we use JPA the EntityManagerFactory adds a significant amount of time for startup.

The quick fix for the cold start till Google App engine team figures it out, is to ping the application every few minutes. This will keep the application from going into the cold storage.

For this implementation to work we require a Servlet which essentially does nothing apart from receiving the request. We also require a cron job which renders request every few minutes. Let’s look at the servlet code listing and its mapping in web.xml.

public class PingServlet extends HttpServlet {

   private Logger logger = Logger.getLogger(PingServlet.class);

   @Override
   protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
	logger.info("Keeping the application warm");
   }

   @Override
   protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
	doGet(req, resp);
   }
}

Improve Performance By Using Batch Gets on Google App Engine

We are porting a JPA application to Google App Engine. One of the challenges we faced during this exercise was performance. There are simple and effective ways by which we can increase the performance of our application. Datastore batch get is one such optimization we used during porting the application.

If you are used to low level datastore api you must have used batch get. First of all what is it?

Batch gets are super-effecient way to load multiple entities, when you already have the keys of entities you want to load. Here is an example of low-level datastore API:

public Map getById(List keys) {
      DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
      return ds.get(keys);
}

Can we issue a batch get using JPA? yes we can.

Suppose you have an Entity TimesheetEntry and a JpaTimesheetEntryDao data access object which has methods to fetch data from datastore.

@Entity
public class TimesheetEntry {
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	private Key timesheetEntryKey;

	private Key projectAssignmentKey;

	private Float hours;

	private Date entryDate;
	. . .
}

Google App Engine More JPA Gotchas

We were in process of porting an application to GAE. The application used JPA for persistence and we decided to use the same for GAE as well. After deploying, the application worked fine on Google App Engine. Over a period of time we started getting errors, which seemed strange. All our tests were running fine and application worked without any issues with our seed data. Where was the problem!

The application had some IN queries in our JPA code which failed after the data changed on GAE production environment.

We have a JPA Entity TimesheetEntry and a Data Access Object JpaDetailedReportDao which fetches TimesheetEntry based on ProjectAssignment Keys. We used JPA IN Query to extract timesheet entries. Let’s look at the code listing.

@Entity
public class TimesheetEntry {
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	private Key timesheetEntryKey;

	private Key projectAssignmentKey;

	private Float hours;

	private Date entryDate;
	. . .
}

Google App Engine Some JPA Gotchas

While porting an application to Google App Engine we encountered several issues. Most of them where related to persistence. We were using JPA for persistence in the application. One of the most common mistake we did was in issuing a JPA Query where one or more parameters are of type com.google.appengine.datastore.api.Key class. Key class has two String representations which lead to some confusion.

Let’s look at an example. There is a Department entity and its associated JpaDepartmentDAO data access object. This DAO has a method which returns a Department based on departmentKey which is of type com.google.appengine.datastore.api.Key.

@Entity
public class Department implements Serializable {
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	private Key departmentKey;
	
	private String name;

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}

	public Key getDepartmentKey() {
		return departmentKey;
	}

	public void setDepartmentKey(Key departmentKey) {
		this.departmentKey = departmentKey;
	}
}

GAE Problems: Slim3 To The Rescue?

For the past few months we have been working on porting a time-sheet application on Google App Engine. The application we were trying to port had Spring for Dependency Injection, Hibernate for persistence and Wicket as the web framework. Looking at the will it play in app engine Hibernate was an easy call, it is not compatible on Google App Engine, wicket works with some workaround and Spring is fully supported. We decided to go with JPA as it is easier to convert a hibernate application to JPA than to other ORM framework. We decided to continue with Spring and Wicket.

We encountered several problems during porting the application. Most of the effort went in increasing the performance. We used Key-Only-Query, memcache to increase performance. We also had issues with handling consistency of datastore operations across entity groups.

When we select framework for application development, we think in terms of the application performance and the ease of application development. When we are not porting an application but doing a new development we can choose frameworks more freely. For new application development on Google App Engine Slim3 framework seems like a good option.

Google App Engine has a notion of entity groups. It is a very important concept and decides how the Entities will be mapped in JPA for the application.

There is a limitation that an Entity cannot have more than one parent. This can be a major issue while re-defining Entity associations for Google App Engine for the application. The Entity Groups also decide the transaction strategy for an application. Simply put you can only apply transaction on a single Entity Group. Is there a platform which allow us to provide transactions across Entity groups. Slim3 does allow us to work across entity groups.

Slim3 example for transferring amount across two accounts:

Implementing Multi-tenancy On Google App Engine

Google recently added support for multi-tenancy for applications via the Namespaces API. We are in process of porting an application to Google App engine. We were thinking of making our application multi-tenant. With the new Namespaces API in our arsenal we decided to give it a try. To our surprise implementing multi-tenancy is easy, we were done with it in couple of hours!

With multi-tenancy, multiple client organizations (or “tenants”) can all run on a same hosted application. In effect due to segregation of data using a unique namespace for each client. This allows us to serve the same application to different customers, with each customer seeing their own unique copy of the application.

Let’s see how we went about implementing it. First of all we needed a Filter it sets unique name-space for each request. Here is the code listing.

Improve Performance By Using Keys Only Query on Google App Engine

It has been around two months since we started to port an existing application on Google App engine. It used Hibernate for persistence, Wicket framework for web layer and Spring as an Ioc container. Looking at the will it play on appengine we had to change Hibernate to use either JPA or JDO. We decided on JPA purely on the basis of experience we had on JPA compared to JDO. Wicket is semi compatible and we made it work by the three standard workarounds described here.

Changing the persistence layer from Hibernate to JPA was the most challenging task. Not only we had to break the relationships between entities because of the Datastore’s notion of entity groups but also due to several performance optimizations we had to do later. We will have a look at how can we use keys-only query in JPA to increase performance.

We had trouble mapping the associations in JPA for Google App engine for which we blogged about the case for unowned keys and managing multiple parent problem. Understanding of Entity Groups and transactions are extremely important for implementing persistence on Google App Engine. This becomes especially important for porting an application to Google App engine.

In our application there were certain parts we could find were expensive either in terms of CPU usage or in Datastore calls. We were able to optimize application performance using keys-only queries.