Hacking Devin Torres

Elixir: It’s Not About Syntax

Whenever there’s a discussion about Elixir, it soon becomes apparent that there’s still a lot of confusion regarding it’s purpose. Some developers have it in their heads that Elixir is merely some crazy new syntax that ex-Rubyists are using to avoid writing Erlang. Well, I’m going to try to dispell some of the myths and misconceptions with this blog post.

It’s not about syntax, stupid silly!

In The Excitement of Elixir, I responded to two of the most well-known criticisms of Erlang from real-world developers now that we have Elixir for comparison. I failed to drive home what Elixir truly represented. Elixir is not “just about syntax,” nor is that the only thing Elixir stands for. If Elixir’s goals were a pie chart, a friendlier syntax would represent a sliver of what makes Elixir a worthwhile investment.

Performance

Let’s nip this myth the bud right away: Performance of Elixir code should match or beat the performance of equivalent Erlang code. If you find that in your use case it doesn’t, you should immediately file it as a bug! Elixir, while incredibly expressive, still compiles to what the equivalent Erlang code would be. It’s still compiling to EVM bytecode at the end of the day. An Elixir function call is an Erlang function call—there is no overhead! Elixir’s powerful metaprogramming capabilities don’t come from e.g. runtime dispatching, but the fantastically powerful compiler. All this magic happens at compilation time, before your code even has to run. And this is the part that may blow your minds: Elixir will beat the performance of Erlang in some cases. We’ll get to that later, though.

Metaprogrammability

I’m done arguing: Elixir is strongly homoiconic.

1
2
3
4
5
6
7
8
9
10
11
12
13
iex> contents = quote do
...>   defmodule HelloWorld do
...>     def hello_world do
...>       IO.puts "Hello world!"
...>     end
...>   end
...> end
{:defmodule,[context: Elixir],[{:__aliases__,[alias: false],[:HelloWorld]},[do: {:def,[context: Elixir],[{:hello_world,[],Elixir},[do: {\{:.,[],[{:__aliases__,[alias: false],[:IO]},:puts]},[],["Hello world!"]}]]}]]}
iex> Code.eval_quoted contents
{{:module,HelloWorld,<<70,79,82,49,0,0,7,104,66,69,65,77,65,116,111,109,0,0,0,132,0,0,0,13,17,69,108,105,120,105,114,46,72,101,108,108,111,87,111,114,108,100,8,95,95,105,110,102,111,95,...>>,{:hello_world,0}},[]}
iex> HelloWorld.hello_world
Hello world!
:ok

Not only is it homoiconic, but it has one of the most powerful macro system I’ve been able to find in among other macro-capable languages I’ve used to date. There has been volumes written about the expressiveness and value of macros in Lisps. I don’t think I need to reiterate what’s already been written on that front, but FUD is nevertheless still disseminated.

A great, simple macro to briefly showcase what they look like would be the match?/2 macro from Elixir’s Kernel module:

1
2
3
4
5
6
7
8
9
10
defmacro match?(left, right) do
  quote do
    case unquote(right) do
      unquote(left) ->
        true
      _ ->
        false
    end
  end
end

It’s just incredibly easy to reason about what’s going on when you see it used in examples:

1
2
3
4
iex> list = [{:a,1},{:b,2},{:a,3}]
[a: 1, b: 2, a: 3]
iex> Enum.filter list, fn (thing) -> match?({:a, _}, thing) end
[a: 1, a: 3]

Let’s just clean that up with some partial application:

1
2
iex> Enum.filter list, match?({:a, _}, &1)
[a: 1, a: 3]

Macros are scary

Yes, they’re as scary as they are powerful. However, they’re the least kind of scary macros. Elixir macros are hygienic. This means that variables defined in a macro won’t interfere with variables defined in the local scope when you use the macro. Oh, and guess what? They are optionally unhygienic as well if you’re into that kind of thing. Oh, and you don’t have to lose line information either.

But…but macros can be hard to debug and can lead to rabbit holes of blah blah blah

Yeah, we get it. Macros shouldn’t be abused. That’s not a fault with the language, but something the user has to strike a balance with themselves. If you ever find José (the BDFL of Elixir) ever advocating the use of a macro when a simple function would do, I’d eat my own shorts.

Okay, but just how metaprogrammable are we talking?

Metaprogrammable enough to do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defmodule MimeTypes do
  HTTPotion.start
  HTTPotion.Response[body: body] = HTTPotion.get "http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types"

  Enum.each String.split(body, %r/\n/), fn (line) ->
    unless line == "" or line =~ %r/^#/ do
      [ mimetype | _exts ] = String.split(line)

      def is_valid?(unquote(mimetype)), do: true
    end
  end

  def is_valid?(_mimetype), do: false
end

MimeTypes.is_valid?("application/vnd.exn") #=> false
MimeTypes.is_valid?("application/json")    #=> true
What just happened?

We created a module that uses HTTPotion to download the public domain mime.types database from the Apache project, parses it, and creates a polymorphic function called is_valid?. This is exactly how String.Unicode is implemented in Elixir: it reads from an in-repo copy of the Unicode 6.2.0 database and statically compiles functions based on that database into the module. Extrapolate this power to anything, really. The public domain tzdata database for a timezone module, a currency database module, SQL query pre-generation, etc. This expressivity makes writing faster code easier, with less LOC.

The standard library and runtime

The Elixir standard library and runtime is where Elixir is really differentiating itself from Erlang. The Elixir standard library aims to dramatically increase the productivity of Elixir developers, while providing the extensibility and features Elixir developers expect from such a metaprogrammable language. A lot of Elixir newcomers really do themselves a disservice by merely wrapping functions in the Erlang stdlib to get a prettier module name and only adding a level of indirection. If your Elixir modules only wrap Erlang modules without provinding any real benefit, you’re doing it wrong. This is why Elixir interfaces to Erlang data types try to improve upon the Erlang interfaces by standardizing a noun-first API, providing enumerable support, enhancing with greater functionality, among many other things.

A brief glimpse at the standard library and runtime

Protocols

Inspired by Clojure protocols, Elixir protocols allow polymorphic interfaces to Elixir data types and user defined records. My favorite protocol is probably the simple EXN.Encoder protocol to represent a subset of Elixir for data representation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
defexception Exn.EncodeError, value: nil do
  def message(exception), do: "#{inspect exception.value} cannot be encoded"
end

defprotocol Exn.Encoder do
  def encode(term)
end

defimpl Exn.Encoder, for: [Atom, List, Number, BitString, Regex] do
  def encode(term), do: inspect(term)
end

defimpl Exn.Encoder, for: Range do
  def encode(Range[first: f, last: l]), do: "#{f}..#{l}"
end

defimpl Exn.Encoder, for: Tuple do
  def encode(term) do
    inspect(term, raw: true)
  end
end

defimpl Exn.Encoder, for: [PID, Function, Reference, Port] do
  def encode(term), do: raise Exn.EncodeError, value: term
end
1
2
3
4
iex> Exn.encode 1..5
"1..5"
iex> Exn.encode %r{.*}
"%r\".*\""

(EXN is a proof of concept answer to edn from the Clojure crowd.)

Reducers

The Enumerable protocol is based on reducers (inspired by Clojure Reducers), a functional, composable way to enumerate over your collections. Reducers allow for user defined enumerables to implement their own faster enumeration and also allows for lazy and parallel enumerations.

HashDict

Elixir’s HashDict is just such a great example of what Elixir’s standard library is trying to achieve that I want to mention it. Erlang has several dictionary-like modules for storing key-value pairs, each with their own performance characteristics depending on the size of the data set: dict, orddict, gb_trees. It’s up the programmer to profile and choose the best profile guided implementation for the data size they think their data set is going to need to remain performant. HashDict takes care of that for you automatically. Not only is it faster than the Erlang alternatives in almost any scenario, but HashDict will dynamically scale the underlying storage mechanism to be the fastest possible for the data set it’s working with.

Unafraid of change

The great thing about the Elixir standard library is that with each release it can provide features that Erlang developers clamor for everyday. We have Erlangers, Clojurists, Haskellers, Rubyists, and Pythonistas trying to incorporate useful features into Elixir every day. Elixir isn’t afraid of introducing functionality that improves the lives of Elixir developers, and everything is on the table: new data structures, real Unicode support, anything.

Tooling

The key to many developer’s hearts is tooling, and José understands this. Elixir tries to make tooling a big priority with the tools it provides.

A slight peek at the tooling

IEx

Everything is an expression:

1
2
3
4
5
6
iex> defmodule Foo do
...>   def bar, do: "bar!"
...> end
{:module,Foo,<<70,79,82,49,0,0,7,24,66,69,65,77,65,116,111,109,0,0,0,102,0,0,0,11,10,69,108,105,120,105,114,46,70,111,111,8,95,95,105,110,102,111,95,95,4,100,111,99,115,9,...>>,{:bar,0}}
iex> Foo.bar
"bar!"
  • Coloring! (Not pictured here.)
  • User defined helpers!
  • And more! What more could you ask for?

Doctests

Elixir takes inspiration from Python in the form of doctests. Interactive iex sessions are embeddable in Elixir’s first-class documentation and runnable from your ExUnit tests with a simple call to doctest.

An example @doc with doctests from the nil?/1 macro in the Kernel module:

1
2
3
4
5
6
7
8
9
10
11
12
@doc """
Checks if the given argument is nil or not.
Allowed in guard clauses.

## Examples

    iex> nil?(1)
    false
    iex> nil?(nil)
    true

"""

That IEx session is turned into two test cases run with the rest of your tests when you type mix test.

Mix

A fantastic build (and soon deployment) tool. Inspired by the much beloved Leiningen and centered around the notion of Mix.Tasks, Mix is by far the most pleasant answer to tooling I’ve worked with other than the awesome Leiningen.

Because I’m beginning to get lazy, I’ll just do a mix help for you:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ mix help
mix clean           # Clean generated application files
mix compile         # Compile source files
mix deps            # List dependencies and their status
mix deps.clean      # Remove dependencies files
mix deps.compile    # Compile dependencies
mix deps.get        # Get all out of date dependencies
mix deps.unlock     # Unlock the given dependencies
mix deps.update     # Update dependencies
mix do              # Executes the commands separated by comma
mix escriptize      # Generates an escript for the project
mix help            # Print help information for tasks
mix local           # List local tasks
mix local.install   # Install a task locally
mix local.rebar     # Install rebar locally
mix local.uninstall # Uninstall local tasks
mix new             # Creates a new Elixir project
mix run             # Run the given expression
mix test            # Run a project's tests

Elixir : Erlang :: Clojure : Java

Elixir isn’t the CoffeeScript of Erlang just as Clojure isn’t the CoffeeScript of Java. Just like Clojure, Elixir is more than a pretty face. Elixir is the power of it’s tooling, the expressiveness of it’s metaprogrammability, and the expansive feature set of it’s standard library while maintaining complete compatibility with—and heavily leveraging—OTP. Once again I have yet to adequately scratch the surface of what makes Elixir special, but I have more Elixir to write!

The Excitement of Elixir

Introduction

I’ve been an Erlanger since 2008 and I’m the author of a few Erlang projects of varying usefulness and popularity, including Poolboy. Poolboy is used by Basho in Riak, 2600hz in Kazoo, IRCCloud, ChicagoBoss, and other high profile projects.

Get on with it…

What I’m trying to say is that I’m (hopefully) not an Erlang noob, so with that context in mind I’d like to get this off my chest: programming in Erlang sucks. I’ve built dozens of services in Erlang and written fair amount of Erlang code. Don’t get me wrong, it’s not all bad! My love of pattern matching and polymorphic functions is limitless. However, everything else about Erlang (the language) is clunky and cumbersome. Now for the pitch: Elixir is everything good about Erlang and none — almost none — of the bad. That’s a bold statement, right? Elixir is what would happen if Erlang, Clojure, and Ruby somehow had a baby and it wasn’t an accident.

Let’s define terms

Déjà vu

We’ve all heard this before, haven’t we? It may sound to you like two of the most well-known criticisms of Erlang recently. The first, What Sucks About Erlang by Damien Katz, the original author of CouchDB. The other is The Trouble with Erlang (or Erlang is a ghetto) by Tony Arcieri the author of Reia (another BEAM based language). What if I told you that Elixir addressed almost all of their concerns? Let’s go through some of the points on these blog posts and see how Elixir handles it differently.

Syntax

What Damien has to say: Erlang’s syntax does away with nested statement terminators and instead uses expression separators everywhere. Lisp suffers the same problem, but Erlang doesn’t have the interesting properties of a completely uniform syntax and powerful macro system to redeem itself.

What Tony has to say: The syntax is atrocious.

How Elixir solves this: Elixir syntax is like a marriage of DSL friendly Ruby and the powerful hygenic macros of Clojure.

Expressions

What Damien has to say: Erlang ifs could be so much more useful if it would just return a sensible default when no conditionals match, like an empty list [] or the undefined atom. But instead it blows up with an exception. If Erlang were a side-effect free functional language, such a restriction would make sense. But it’s not side effect free, so instead it’s idiotic and painful.

What Tony has to say: In Clojure, I can write the following: (if false :youll-never-know). This implicitly returns nil because the condition was false. What’s the equivalent Erlang? Erlang forces you to specify a clause that always matches regardless of whether you care about the result or not. If no clause matches, you get the amazingly fun badmatch exception. In cases where you don’t care about the result, you’re still forced to add a nonsense clause which returns a void value just to prevent the runtime from raising an exception.

How Elixir solves this: Elixir if expressions, like most everything else in the language, are implemented using a macro.

1
if false, do: :youll_never_know #=> nil

Strings

What Damien has to say: The most obvious problem Erlang has for applications is sucky string handling. In Erlang, there is no string type, strings are just a list of integers, each integer being an encoded character value in the string.

What Tony has to say: The obvious solution here is to use binaries instead of lists of integers. Binaries are more compact and exist in a separate heap so they aren’t copied each time they’re sent in a message. The Erlang ecosystem seems to be gradually transitioning towards using binaries rather than strings. However, much of the tooling and string functions are designed to work with list-based strings. To leverage these functions, you have to convert a binary to a list before working with it. This just feels like unnecessary pain.

How Elixir solves this: Elixir strings are UTF8 binaries, with all the raw speed and memory savings that brings. Elixir has a String module with Unicode functionality built-in and is a great example of writing code that writes code. String.Unicode reads various Unicode database dumps such as UnicodeData.txt to dynamically generate Unicode functions for the String module built straight from that data!

1
2
<<?h, _ :: binary>> = "hello"
String.graphemes("hello") #=> ["h","e","l","l","o"]

Single assignment

What Damien has to say: Immutable variables in Erlang are hard to deal with when you have code that tends to change a lot, like user application code, where you are often performing a bunch of arbitrary steps that need to be changed as needs evolve […] Erlang’s context dependent expression separators and immutable variables end up being huge liabilities for certain types of code, and the result is far more line edits for otherwise simple code changes.

What Tony has to say: Erlang doesn’t allow destructive assignments of variables, instead variables can only be assigned once. Single assignment is often trotted out as a panacea for the woes of mistakenly rebinding a variable then using it later expecting you had the original value. […] Single assignment is often trotted out by the Erlang cargo cult as having something to do with Erlang’s concurrency model. This couldn’t be more mistaken. Reia compiled destructive assignments into Static Single Assignment (SSA) form. This form provides versioned variables in the same manner as most Erlang programmers end up doing manually. Furthermore, SSA is functional programming. While it may not jive with the general idealism of functional programming, the two forms (SSA and continuation passing style) have been formally proven identical.

How Elixir solves this: There is no context dependent expression separators and when you rebind a name all you’re doing is rebinding a new value to the name. Elixir is still immutable, it’s just not trying to pass off single assignment as immutability.

1
2
thing = doit(thing) #=> v0 = doit(thing)
thing = doit(thing) #=> v1 = doit(v0)

Records

What Damien has to say: The ‘records’ feature provides a C-ish structure facility, but it’s surprisingly limited and verbose, requiring you to state the type of the record for each reference in the code.

What Tony has to say: Erlang has a feature called ‘records’ which uses the preprocessor to give you something akin to a struct or map, i.e. a way to access named fields of a particular object/term within the system. As far as I can tell, there’s pretty much universal agreement within the community that this is a huge limitation.

How Elixir solves this: Elixir records are real records, and provide compile time pattern matching and a much more dynamic nature.

1
2
3
4
5
defrecord Foo, bar: "baz", quux: nil
x = Foo.new
x.bar #=> "baz"
x = x.quux "corge" #=> Foo[bar: "baz", quux: "corge"]
x.to_keywords[:bar] #=> "baz"

Standard library

What Damien has to say: The coding standards in the core Erlang libraries can differ widely, with different naming, argument ordering and return value conventions.

What Tony has to say: Should module names in the standard library be plural, like lists? Or should they be singular, like string? Should we count from 1, as in most of the functions found in things like the lists module, or should we count from 0 like the functions found in the array module?

How Elixir solves this: While you can’t escape using the Erlang standard library from time to time, Elixir’s standard library is trying to normalize zero-based access and noun-first argument ordering, along with a more consistent API.

1
2
elem({:a, :b, :c}, 0) #=> :a
List.member?([:a, :b, :c], :b) #=> true

Code organization

What Damien has to say: The only code organization offered is the source file module, there are no classes or namespaces. I don’t need inheritance or virtual methods or static checking or monkey patching. I’d just like some encapsulation, the ability to say here is a hunk of data and you can use these methods to taste the tootsie center. That would satisfy about 90% of my unmet project organization needs.

How Elixir solves this: Elixir modules provide a great way to encapsulate functionality. As many modules as you desire can be declared within the same file including private functionality (functions, macros, records, exceptions, fuzzybunnies, whatever).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
iex> defmodule Foo do
...>   defmodule Bar do
...>     @baz "hello"
...>     defp quux do
...>       @baz
...>     end
...>     def corge do
...>       quux
...>     end
...>   end
...> end
{:module,Foo,<<70,79,82,49,0,0,6,44,66,69,65,77,65,116,111,109,0,0,0,98,0,0,0,10,10,69,108,105,120,105,114,45,70,111,111,8,95,95,105,110,102,111,95,95,9,109,111,100,117,108,...>>,{:module,Foo.Bar,<<70,79,82,49,0,0,7,40,66,69,65,77,65,116,111,109,0,0,0,113,0,0,0,12,14,69,108,105,120,105,114,45,70,111,111,45,66,97,114,8,95,95,105,110,102,111,95,95,4,100,...>>,{:corge,0}}}
iex(2)> Foo.Bar.corge
"hello"

It’s the simple things

How many times have you wished for multiline strings in Erlang? It’s something so simple that ends up feeling like a godsend as soon as it’s available to you. What about a binary_to_float so you don’t have to keep doing list_to_float(binary_to_list(Bin))? Elixir’s got that too. No more proplists:get_value(username, Json), say hello to json[username].

Turtles all the way down

Elixir continues to blow me away everyday. The crazy thing is, it’s still just Erlang underneath. Theoretically, Elixir code is just as fast as Erlang code. Elixir function calls are just Erlang function calls. In fact, it may be easier to write more performant code in Elixir simply because of the power available to you to at compile time. For example, regular expressions in Elixir are compiled at compile time instead of runtime. The Dynamo web framework compiles routes to function heads, matching based on function guards. The latest Elixir will have a HashDict implementation significantly faster than Erlang’s dict.

Lisps traditionally empowered developers because you can eliminate anything that’s tedious through macros, and that power is really what people keep going back for.

— Rich Hickey

I’m done (for now)

I cannot possibly do Elixir any justice by just comparing it to Erlang. And I haven’t even begun to scratch the surface. Just give it a try.

Turn CSS Rules Into Inline Style Attributes Using jQuery

Gmail doesn’t support stylesheets or the style tag, but they allow inline style attributes. This snippet allows the browser to use it’s native facilities for building an HTML email for e.g. marketing campaigns by allowing you to write a document using linked stylesheets or inline style tags in the document. This could either be run within the context of the HTML email template itself or from a parent window that the HTML email is being built within. After the document has loaded and inlined it’s style definitions, the style and script tags that very few email clients support will be removed so you can get the innerHTML of the document and use it later as the HTML portion of your multipart MIME emails.

JavaScript: The Fastest Dynamic Language?

I remember programming for the web in the late 90’s. JavaScript engines were a lot worse then than they are now, and it was the last programming skill a programmer mentioned. Many programmers didn’t even regard the language as a real language, much the same way many of us today scoff when somebody mentions HTML as a programming skill. What many didn’t realize, though, is that JavaScript has always been a beautiful language when used correctly.

Original JavaScript engines were buggy and slow, which is why it’s taken this long for people to finally appreciate the language. We now have Nitro (SquirrelFish Extreme) from the Apple camp, V8 from the Google camp, and TraceMonkey from the Mozilla camp. Without getting into JavaScript engine wars, they’re all blazingly fast and, apart from Nitro which many take a bit more work, easy to integrate with any project.

I was mulling over how bright the future of JavaScript was when I read this week about a project called Narwhal on Ajaxian. Although I had heard of server-side JavaScript for awhile, I only imagined it being JavaScript without the DOM. I was unaware there was a move to create a true server-side JavaScript. This just means adding standard objects to interact with system-level services, such as the File object.

The implications of all of this is huge. While Ruby and Python have separate projects busy trying to implement fast JITed interpretters for their languages, JavaScript already has three! Imagine writing an entire Rails-like web framework in pure JavaScript. We can take it even further: Imagine writing a WxWidget application using pure JavaScript! Perhaps a cron job in pure JavaScript? Anything is possible. It has potential to become a first rate scripting language along with Python and Ruby. Better even, as any three of our new JavaScript engines could best them both in speed and possibly even memory. Even when Ruby bytecode is run on V8 it can be almost as fast as Ruby 1.9. Both Python and Ruby have problems scaling using threads because of  their GILs. Meanwhile, web worker threading is already implemented in all three engines.

Take V8, one of the many new ServerJS extensions to it, an interactive debugger like IPython such as the one already in the V8 on Python project, a package manager like RubyGems, and bundle it up for Windows, Mac, and Linux and you can call it whatever you like. The worlds fastest dynamic language.