A tale of two lambdas: The joys of working in a polyglot team.

If you take everything together, you get the picture in Figure 3.  Not in the picture are Graphite for metrics Elasticsearch for logs and ZooKeeper for configuration. It might appear a little messy at first sight, but it isn't. It is a straightforward, data-centric architecture. It uses the most pragmatic tools for any given task. All components are easily interchangeable. Last but not least: It is a lot of fun to work with.

I fear I may have bored you with all that contextual information, so lets get back on topic: the programming languages.

Scala.

Scala really is a great language. It has tons of interesting features. From a Java programmer's perspective, Scala feels like two huge steps forward. Data structures are immutable. Functional programming is well supported. There is pattern matching and countless other cool features. All that makes it very interesting and fun to work with.

At the same time it feels like a little step back. Scala is complicated. All too often a pair of developers will look at each other and say "Did we really just spend two hours getting the types right when converting that data representation into that other one?" Also, it will frequently give you a hard time trying to understand what some library code does. Chances are the authors of the library fancy a different set of language features and syntax options than you do.

I would not call myself an expert on Scala, so I will illustrate my feelings about it with three analogies:

• Scala is like C++:Powerful and Multifaceted. It is a very powerful language giving you the freedom to solve problems in a lot of different ways.
• Scala is like Haskell: Academic standards. Being subject of a lot of scientists' work results in well thought out (but possibly a little too academic) features like the powerful type system.
• Scala is like Gentoo Linux: A research lab for the best solution. Like Gentoo, Scala has a rather steep learning curve. Like Gentoo, Scala is the playground of a lot of language nerds that are in search for the best possible solutions.  Sooner or later everybody profits from that work, as the results migrate into the more user-friendly distributions. So if you are not that nerd, you do not have to use it.

Python.

Python is a nice and friendly little language that has been around for quite a while now. By using indentation instead of parentheses, it removes a lot of visual bloat. It is friendly to beginners and it is also understood by many, many techies that would not call themselves a software developer. I just asked one of our data scientists, Stephan, what he thinks about Python and he said:

"Python not only enables me to solve almost all of my computational problems but it is also, unlike R, a lingua franca to communicate with computer scientists."

That's what it is. It is a language equally well suited to support the rather exploratory work of a data scientists and to fulfill all the requirements of reliably operating production software. The support for Python in Spark makes it a first class language for us to build prototypes and to bring prototypes to production with little or no conversion overhead. I'm not sure we will ever find performance problems that are bad enough to require a migration to Scala, but if so, than that's what we will do.

Clojure.

Clojure It is by far our favourite language. Clojure has the intent to make simple easy. And it does. Listening to a pair say "Oh, I think we are done." is often followed by the disbelieving "But that was only five minutes!" And then they move on to the next task.

As a Lisp, Clojure features a syntax that is fundamentally different from everything most Java, Scala and Python programmers are accustomed to. This can be perceived as a hurdle when migrating to Clojure. In our experience it is only a low hurdle though. Most developers, when having crossed that bridge, are excited by the simplicity, brevity and expressiveness of Clojure code.

What is particularly puzzling on very first sight is the different use of parentheses. A pair of parentheses defines a list. That list can be either data or code. The first symbol in the list is treated as a function then, all the following are parameters. Sounds complicated? It Isn't. Here are some examples for how to translate code:

println("foo")becomes(println "foo"). 1 + 2becomes(+ 1 2)

The slightly more complicated Formula1 + 3 * 2becomes(+1 (* 3 2))which demonstrates the simplicity of the syntax: As the order of execution is clear, mathematical operators do not have to be complected with precedence rules. (Yes to complect is a word).

From my own experience and from that of many of my colleagues I can now confidently say: It is only Syntax. You get used to it quickly. Looking at the code of a library will frequently surprise you, too, but the good way. You will be surprised by how little code is actually necessary to do the trick.

There is no explicit type system in Clojure (although an optional one exists) and it turns out you don't need it. Clojure is data-centric instead. With immutability as the default, clojure offers a Software Transactional Memory to manage mutable state. This makes it much, much harder to shoot your own foot with shared mutable state problems. Also, it has very cool and simple concurrency features built in. Beeing homoiconic, macros can be written in the language itself which makes Clojure easily extendable. There are extensions for logic programming, matrix computation, pattern matching and many more.

I could go on with this list, but I will finish here. Just one more thing: All this goodness comes as a JVM-language. So the complete ecosystem of the Java world is only one import away (just like in Scala).

Should I try polyglot programming then?

Should you try polyglot programming? Yes you should. But isn't that inefficient? No it is not.

Let me say it with another analogy.

Polyglot programming is a lot like pair programming: If you have not tried it, it is intuitive to assume it means a lot of overhead and is of doubtable benefit. But as in pair programming the overall productiveness of a team is not diminished. Quite the contrary is true:

• As in pair programming looking at a problem from different perspectives makes you understand the problem better.
• As in pair programming that does not mean the most perfect, but the most pragmatic solution will be sought for.
• As in pair programming fiddling with the tools becomes less important and the business problems that need to be solved move into focus.
• As in pair programming it takes a little while to get used to it, but then you do not usually want to go back.
• As in pair programming polyglot programming makes work a lot more fun.

So go for it.

What's next?

There is always new stuff to try. Like I mentioned there is persistence technologies like Cassandra and Datomic. There is a lot of programming languages yet to be tried. And as soon as we feel a little safer with Spark and understand it better than we do now we will look at Clojure-APIs for it like e.g Sparkling.

Last year I visited the EuroClojure conference in the beautiful city of Kraków. It was by far the most useful conference I have yet visited. (see my reports of day one and day two This very week I will go to Barcelona where this year's EuroClojure takes place. This time I won't go alone, but take a few colleagues with me. I am really looking forward to that. See you there.