Archive for May, 2007

h1

When visionaries are too much in advance.

May 15, 2007

There’s a company out there that always surprised me for always being in advance next to its competitors. This company is JetBrains (and no I do not work for JetBrains nor do I have stock options in it). But sometimes seeing in the future doesn’t always pay (in term of $!). Indeed maybe you remember back in 2004 Sergey Dmitriev, the cofounder and CEO of JetBrains Inc., published a paper about Language Oriented Programming: The Next Programming Paradigm.

Rather than solving problems in general-purpose programming languages, the programmer creates one or more domain-specific programming languages for the problem first, and solves the problem in those languages.

I remember reading that paper when it came out and finding it really promising and avantgardist, I thought it would be the future. Soon after I gave a try to MPS EAP version. Martin Fowler found it promising as well in 2005:

Although I’m not enough of a prognosticator to say whether they will succeed in their ambition, I do think that these tools are some of the most interesting things on the horizon of software development.

The old debates were already starting… Unfortunately, years passed and MPS became a commercial failure, and has been like discontinued since then. Nowadays everybody and his dog talk about DSL, maybe that’s the reason why it appears again on JetBrains website. Sergey had only 2 years in advance with his development software solution. People always complain about late delivery of their software, JetBrains is the only company delivering your software and features before you ever needed it!

h1

How to detect which language a text is written in? Or when science meets human!

May 13, 2007

As I mentioned earlier in my spam attack analysis, I wanted to know which language spams I receive are written in. My first bruteforce-like idea was to take each word one by one, and search in english/french/german/… dictionaries whether the words were in. But with this approach I would miss all the conjugated verbs (until I had a really nice dictionary like the one I have now in firefox plugin). Then I remember that languages could differ in the distribution of their alphabetical letters, but well I had no statistics about that…
That was it for my own brainstorming, I decided to have a look at what google thinks about this problem. I firstly landed on some online language detector… The easy solution would have been to abuse this service which must have some cool algorithms, but well I needed to know what kind of algorithms it could be, and I didn’t want to rely on any thirdparty web service. Finally I read about Evaluation of Language Identification Methods, of which the abstract seemed perfect:

Language identification plays a major role in several Natural Language Processing applications. It is mostly used as an important preprocessing step. Various approaches have been made to master the task. Recognition rates have tremendously increased. Today identification rates of up to 99 % can be reached even for small input. The following paper will give an overview about the approaches, explain how they work, and comment on their accuracy. In the remainder of the paper, three freely available language identification programs are tested and evaluated.

I found the N-gram approach on page 8 (chapter 4) rather interesting. The principle is to cut into defined pieces m long texts written in their respective language (english, french…), that we will call training texts, and count how much time each piece appeared; Do the same on the text you want to identify, and check the training text matching your text the best; This training text is most likely written in the same language as your text.
The pieces are the N-grams, ie for the word GARDEN the bi-grams (N=2) are: G, GA, AR, RD, DE, EN, N.
Now there are various way of finding the best matching text playing with the N-grams, distances, score…

N-gram comparison
I found an implementation from 1996 in C, here with sources. So I followed same algorithm and implemented it in Ruby. Those C sources reminded me of my C days where you had to implement your lists, hashes. Those sources are optimized for memory usage (10 years ago…)… At the end the Ruby code is a hundred line while the C was four times more, and the Ruby code is easier to read. Don’t take that as a demonstration, it is not!! I admit the C binary is maybe a bit faster (but not that much ;)). I’ll try to commit it on rubyforge when I have some time.

The results are excellent as shown in the paper:

N-gram results

Anyway actually in this story the most interesting was not the implementation but the method: It is funny that you can identify languages (so human population as well) by without requiring linguistic knowledge: ignoring grammar, senses of words (dictionary)… But by only analyzing letters and blocks of letters from Shakespeare or Baudelaire. N-grams can also be used in other areas, for example in music to predict which note is likely to follow.

h1

Spam attacks! When? How? What? … in Ruby

May 13, 2007

Today I was wondering about those spams I receive daily. GMail is doing a great work at detecting them, reason why I decided to forward several of my polluted personal mails to my GMail addie. I wanted to know more about those spams and additionally wanted to do that quickly and with fun. So I took my favorite Ruby IDE, I installed some ruby gems: gmailer, activerecord, gruff (together with Mouraf’s patch to extend legend as it was cut when too long on Gruff::Pie).
You should note that to make GMailUtils gem run, you should also have ‘net/https’ library installed on your pc, else you’ll end with a mysterious:
irb(main):001:0> require 'gmailer'
LoadError: no such file to load -- gmailer
from (irb):1:in `require'
from (irb):1

Solution :
sudo apt-get install libopenssl-ruby
Why I decided to use an api to get my mails while GMail allow pop3? Firstly I wanted to play with this gem, then you can’t get directly your spam through gmail pop3 (you’ll need some label tricks to finally put them in your ‘Inbox’, something that you could do in a first pass with GMailUtils), also the api goes through https thus bypassing usual firewalls that block pop3 port.
I wanted to know which language those f****** spams were in… So I decided to code (translate in Ruby even!) a language detector (am blogging about it here).
So some lines of code later I had what I wanted to know (at least for the last month as GMail has a Spam buffer):

Pie LanguageChart Hour

Pie SemanticMonthly Spam

Bars WeekDay Spam

What to conclude from this? Well actually nothing much… Except that 91% of spam is in english (followed by 7% of which are french but that is normal as I am french and I have french email addie forwarded to my gmail). 34% of the pollution is concerning viagra, sex… 16% about watch. Concerning the distribution of spam in time I thought I would find more observable period; It only seems as if I receive more spam on Thursday.


h1

Back!

May 13, 2007

Hi folks!
Sorry for being away that long. Actually I have been quite busy with work, Ruby, and commiting news for InfoQ Ruby community, writing about a Ruby IDE article for a french magazin (to be published when finished…) and now I should get involved in a Rails project.
But keep in touch, more to come about Spam analysis with Ruby, coding a text language detector, Ruby IDE comparison update…