http://mttech.mobitype.com Mobitype Tech Behind the curtain of Mobitype Tue, 04 Apr 2006 23:39:51 +0200 http://mttech.mobitype.com/mobitype_feed_refresh_pol.html Mobitype feed refresh policy Cedric from over-blog asked me a question today: how often does Mobitype refresh feeds published on Cedric from over-blog asked me a question today: how often does Mobitype refresh feeds published on other servers?
A rather simple question, but the answer is quite complex.
In fact, Mobitype refresh policy changed many times since first releases. The current feed base is about 20,000 feeds, we cannot refresh them too often, having server resources limited. But it is discouraging to consistently have stale feeds in the database ;)
So, first of all, all feeds are divided into two groups. There are alive feeds and dead feeds. A feed is alive when it has at least one subscriber, or it has at least 100 consultations per week.
For all alive feeds, an average time spacing between posts is calculated within last several posts (posts in future are not counted, since there are users who timestamp their posts in late 2099). Having this average spacing as A, the refresh time for the feed is calculated as A/20. Afterwards, it is bounded to be between 30 minutes and 1 day, to be reasonable.
The refresh time for dead feeds is about once every several days.
So when you browse lists of feeds, like tops, favorites or text search, the local database copy of the feed is displayed. However, when user clicks on "View post", the local copy is verified. If it has not been refreshed for more then 2 hours, the live request is being sent to the remote server at the time of the page rendering, and the feed eventually updated.]]> Tue, 04 Apr 2006 23:39:51 +0200 http://mttech.mobitype.com/new_in_mobitype__amr_podc.html New in Mobitype: AMR podcasting! Got a very fruitful conversation with Loïc le Meur this morning, about new features in Mobitype. Loïc Got a very fruitful conversation with Loïc le Meur this morning, about new features in Mobitype.
Loïc gave me an idea of some inexpensive, but extremely useful points. Some of them have been implemented this afternoon ;)))

Podcasting with AMR.

AMR is a new generation standard for audio/video encoding. Chances are that this is the standard in which your mobile phone saves microphone and conversation recordings. Widely used in the mobile world, it is far from being common to desktop users.
The idea itself was to convert .AMR to .mp3 on the fly as soon as they are uploaded to Mobitype. Some free libraries and some hack did the job ;)
In fact, Mobitype will keep both versions (.mp3 and .AMR), and cover the .mp3 file link with a nice podcast-looking thumbnail ;)

Post by mail confirmation.

Do you want to be sure your posting-by-mail have passed? Now yes, you'll get a confirmation mail, unless you desactivate it in the profile. The implementation is a bit lightweight by now, someone would like to be able to unsubscribe from annoying confirmation just by clicking to the link inside the mail, or recieve a live link to the new post... well, one cannot have everything at a time, that will appear a bit later.
Enjoy, and stay with Mobitype!
]]> Tue, 20 Dec 2005 19:59:03 +0100 http://mttech.mobitype.com/mobitype_and_caches__swit.html Mobitype and Caches: switching to OsCache Cache is critical for Mobitype. In fact, every RSS or Atom feed fetched from the Web should be cached Cache is critical for Mobitype.
In fact, every RSS or Atom feed fetched from the Web should be cached. This may avoid much of extra effort on verifying if feed have been changed or not. In fact, once the feed is in the cache, and if Mobitype does not know if the feed have been modified since last lookup, Mobitype may ask a question to the remote server in a form of "Please give me this feed if it has not been modified since last monday?" Most servers could report to this question shortly, saying either "Yes, here is the new version" , or "Just feel free to used your cached one, nothing changed" . This simple algorythm saves terabytes of trafic all over the world, and in fact just every known browser and web server uses it. But we definitely need cache for that.
Mobitype took off with EhCache - a simple non-clustered solution from SourceForge that can keep files in memory for fast access and swaps them to disk in case of memory owerfolw. Simple to configure, it showed up shortly with some decieving drawbacks :(
- Its disk owerflow mecanics is quite fragile. In fact, when a memory store reaches its limit, EhCache starts to enqueue objects to the in-memory spool, and there is a special process that flushes them to disk one by one. This process has, however, a vulnerability, that can be discovered by looking at the source code: whatever unexpected happens while flushing objects to disk, the spooling process may just die, and no more objects will be saved untill next server restart. The spooling queue just grows, untill it occupies all available memory, and server falls down ;[]
- All the cached entries are stored in one geant file, which grows progressively with every new entry, occupying enourmous space and heavily clustered. This file is indexed, and the index is stored in memory until the server is shut down. In case of abnormal server termination, all the index is lost ;( The geant cache file is however not cleaned up, and new entries are just appended th the end, thus wasting disk space.
Our next attempt is OsCache from Open Symphony .
At a glance, OsCache is twice complicated than EhCache, both in amount of source code and in configuration. In fact, there is a built-in clustering capability, wich may be not unnecessary if Mobitype grows up ;)) Same memory-overwlow-disk strategy, several caching algorythms (First-In-First-Out or Least-Recently-Used). Every disk-stored item is serialized in its own file, all files by default stored in the same directory. Expecting a very large number of files to store, I decided to override this behaviour to store files in a sort of a balanced tree of folders, which was surprisingly simple (just override the function which chooses a file name for an item, and change a configuration line).
As a first bad surprise, I didn't find any clue on how to remove expired elements from the disk, OsCache considers disk just unlimited . But having files stored in a simple directory structure, it seems to be simple to charge cron to sweep out very old files from time to time.
Time will show if this second attempt is fruitful.
]]> Fri, 25 Nov 2005 17:32:36 +0100 http://mttech.mobitype.com/mobitype___java___postgre.html Mobitype : java + postgress Mobitype is a Java project. Why Java? It is not an evident solution for this kind of open projects, m Mobitype is a Java project.
Why Java? It is not an evident solution for this kind of open projects, mostly written in PHP/MySQL or Perl .
Because I just know Java and I know that it works ;)
The main advantage of choosing Java is a possibility of a nicer architecture with classes and interfaces. The disadvantage is that it is a bit complex ;)
Mobitype uses PostgreSQL . Not very traditional, neither, but the only really free solution for a project that belongs to the commercial structure and cannot publish its source code ( http://mysql.org seems to be more popular and faster, but it is not truely free).
As of now, Mobitype is a single standalone monoprocessor server, with the database and Tomcat server onboard. In this configuration it should be able to support about several thousands of named users.
]]> Fri, 25 Nov 2005 13:48:11 +0100 http://mttech.mobitype.com/mobitype_technical_blog.html Mobitype technical blog In this blog I'll try to communicate, if I have time, some technical background and gotchas on the M In this blog I'll try to communicate, if I have time, some technical background and gotchas on the Mobitype functionnality and the underlying technology.
I choosed to do it in English, as it seems to be flooded with technical terms. Excuse my spelling in advance!
I'm Oleg Sidko, CTO of Tekora S.A., and the main developer for this project.
See you in this blog ;)
]]> Fri, 25 Nov 2005 11:50:15 +0100