Where do we get our data for comparison?

In case anyone missed it we have been developing a flight-comparison site for the last three months, so lets do some comparing. Hold on, there is something missing. The definition of a comparison is that there should be something to compare and that is where our suppliers should come in. Suppliers in this case can be airlines or travel agencies—in short anyone who can deliver flight data and have a site where you can book the tickets.

In a simplified and uncached form a search goes something like this:

1. The user starts a search on the Openjet site
2. The Openjet application tells the TripTrap meta-engine the details of the search
3. TripTrap translates the request for each supplier and sends it to them in their format
4. The suppliers return data to TripTrap which translates it to a unified format
5. The Openjet site fetches the unified data and does its magic

So for this to work we have to have contracts with suppliers and some sort of information. In my earlier work I have been dealing a lot with suppliers for our old meta-engine, and during that I have spent some time thinking on what we will need from them and what they will need from us. Two important areas come to mind immediately.

First comes the issue of technical documentation. In some cases this is quite good and comes with good examples and instructions on how to use a nice web service. In other cases the result may need to be retrieved in some obscure JavaScript-encoded format, directly from a web page, with no documentation whatsoever. To be able to efficiently develop new supplier connections this is something to think about. Would it be feasible to say “no” to suppliers without good documentation? Can we afford to not have a good supplier just because they do not have documentation?

Of course you could always have a contact person to talk to in case you run into problems when developing the supplier connection. This brings me to the second important area, which is to have a technical contact address that is not tied to a specific person. It should also be an address that is not liable to change if the structure of the company changes. There have been too many times in the past when I tried to get hold of a technical contact for a supplier which stopped working, only to find that no-one replied on that address or the mail ended up in marketing and got passed around for two weeks before I got a first reply.

There are of course other pieces of information that are important to have for us and for the supplier we use, but these two areas are the ones that have touched my work the most in the past and I feel that they are key to an efficient development process and bug-handling.

Cluster-wide search

Well, after some refactoring TripTrap is finally commited to the git repository.

Besides a lot of refactorings, rearrangements, broken tests fixes and optimizations the main improvement added recently is that each TripTrap instance in cluster environment is in some sense aware of what other nodes do. We had a principal problem with that issue: each cluster node can perform the same search, even if it's already launched at some other node. That's not a big issue as it's not so often, but such redundancy overloads the system, and it can cost us some money. Nobody likes paying money for exactly nothing.

Some synchronization through DB can be implemented, but that's not a scalable solution. The DB will become a bottleneck in a while. The solution, again, is to use memcached for it and place there a mark like "I'm searching this and that for that supplier, account and request". And delete that mark after search is over. Thus, for polling requests redundancy is avoided at all, for synchronous requests, which are waiting for all possible responses, there's some complex strategy like "if there's such searches on other nodes, first launch local ones, than check again, if some of remote searches completed" and so on. Memcached is rather fast and greatly scalable thing, so no bottlenecks is expected, I guess.

Now there are two different implementations of search, configured in Spring context: cluster-aware one, which uses cluster-wide synchronization and single-instance one, which is simplier and faster.

So it goes.

Martin mentioned the unique ID generation in previous post, so I recalled the problem which was solved in similar fashion. At the beginning of developing TripTrap the issue "how to find out if such request was already processed?" arose. Well, that's quite easy: store it to DB and next time check if it's already there. What becomes a problem is "how to do that fast?", especially when we have almost twenty request parameters (and thus such a number of conjunct conditions in SQL-query), also different types of requests possible, and also some of the parameters can be omitted, etc.

The solution was the following: count 64-bit integer hash for the request and store request to DB and to memcached together with that hash. Stored request is just the binary serialized and gzipped blob. So, in DB we have both BLOB field and BIGINT fields. The latter is very good indexed and is virtually unique, and it's very good key for memcached storage.

For each incoming request we count the hash, fetch the blob from cache or DB, deserialize the blob and compare it with the incoming one in Java-code. In the virtually impossible case of the same hashes for different requests the only problem is that we need to fetch, deserialize and compare more that one request blob from DB or the list of request blobs from memcached. 

Voila, for memcached with a lot of RAM we get the stuff working as fast as possible.

As I already mentioned, the main focuses in developing TripTrap were the speed, scalability and easy clustering. Some introduction to technical details now. 

There were some "not very common" decisions made while developing TripTrap.

1. To improve performance we store binary serialised Java objects in DB and memcached as well. That's fast, easy and universal for all kind of objects (in fact, TripTrap can be used not only for flight searches). The drawback of such approach is that DB is not self-sufficient, one cannot search anything in DB without TripTrap --- DB is a dumb persistent storage in our case. Nevertheless, up to the present moment we have not experienced any problems with that. One more reason: we try to move the work from DB to memcached as much as possible, and memcached is just a map, so we try to avoid involved data structures.
2. We use InnoDB engine for MySQL. Sounds a little strange, but we have a lot of read-write activity, also tests showed that MyISAM doesn't differ significantly. 

The most tricky thing in cluster environment is the synchronisation. In TripTrap it's done through memcached entries, thus we have relatively small overhead.

We can say TripTrap is good enough in what it made for. Anyway, there's still issues I'm not satisfied with:

• Load testing: it's done only partially and needs more test cases.
• More acceptance tests: a couple of dozens exist, but should be more.
  
• Threads: more parallel jobs should be used (for example, 16-core servers are more or less common, so TripTrap must use all available resources).
• Source code tree structure: it's time for review and moving things between packages and simplifying the stuff. We aimed at keeping the project structure as clear as possible.
• Documentation. Yep, TripTrap suffers from it too.
• Criticism. There is no ideal systems and solutions, no silver bullets and so on. Usage experience, testing and everything that can help us finding TripTrap problems to make it better. Or ideal :)

TripTrap is getting the feature of multi-point search currently (the search with not only origin and destination, but some intermediate points). That's a kind of challenging feature as it involves extraction of large amount of data to process and filter. Will be ready soon.

So it goes.

Trip and Trap

TripTrap is the meta-search engine behind the OpenJet providing Open Travel - conforming interface to the data gathered from numerous sites and sources. It also has JSON-based interface suited for AJAX client applications.

The main things were kept in mind while making the TripTrap are speed and good ability for scaling and clustering.

TripTrap developed using Java 1.6 and Spring framework with data stored in MySQL and memcached server used for faster data extraction.

The source code will be in git repository soon.

/Mikhail