Jabberwocky

snicker-snack!

Holiday Reading

| Comments

I’m about to go on holiday. It’s been ages, it’s my first proper break in a good while, so I’m really looking forward to it. As always, I’m going to go overboard and plan for a whole wash list of books to read – I mean, I’ll have time, right? We’re doing a road trip, so I’ll be driving a lot of the time, but surely I’ll have hours and hours to read?

Machine Learning for Hackers Nice practical overview of a number of machine learning use cases using R

Purely functional data structures I’m still bending my head round functional paradigms, so I’m hoping this provides more insights.

While we’re at it: I’m hoping Learn you some Haskell for Great Good will help rewiring my brain.

And I should finally finish Thinking, fast and slow which is an extraordinary book about cognition, which I pretty much recommend to everyone I meet.

And start on What I talk when I talk about running of Murakami, a long-time favourite writer. This is an essay, and I’m curious about what he has to say.

Speaking of which, I heartily recommend this video by Bradbury, who not only wrote great books but also sounds like a very likeable person: Talent is nothing without focus or endurance

Why I’m Not Using Clojurescript

| Comments

People all over the place have been promoting alternative syntaxes for javascript. The clojure community in particular has been working with clojurescript, the idea of using clojure in every part of the stack being very seductive. Here’s one reason why I’m not thrilled about using clojurescript:

The browser doesn’t speak clojure, it speaks javascript.

So switching to clojurescript adds a few constraints, which personally, I’m not thrilled about.

  • clojurescript is compiled into a fairly ugly bunch of obfuscated code, which is what one is dealing with whenever any debugging is in order.
  • it adds the google closure library to the dependencies, even if you’re just adding a couple of effects.
  • any moderately heavy client-side stuff usually needs some performance optimization, this means twiddly js stuff. Shall I use this function instead of this other? Shall I avoid this loop in favor of something else? This is made markedly more difficult when having to deal with a layer of indirection between the code you’re working on and the code actually running on the browser.
  • let’s not mention producing cross-browser compatible code, IE hacks are not nice and lispy, however you turn it.

Admittedly, the third point is also true for any high-level language – why not work in C instead of Ruby, Clojure, or others? However, javascript is no C, not in difficulty level. It’s not very pretty, it’s got warts that come with a long and twisted history, but it’s possible to produce fairly friendly-looking code. javascript by itself isn’t excruciating enough to justify the potential difficulty caused by using clojurescript.

I admit I’m sometimes a curmudgeon and resistant to change, so as always, happy to hear any arguments as to why I might be glaringly wrong.

(Note: do read the comments by Stuart Sierra and David Nolen, for a different point of view)

The Art of Speaking at Conferences

| Comments

Speaking at conferences, and public speaking in general, is not a straightforward matter. The following subjective diagram illustrates what I mean: public speaking dimensions

Explanation to the matrix:

  • content: purely informative, with interesting content, but an unimaginative, factual, and possibly slightly boring delivery.
  • entertainment: bringing entertainment to the party, moving the souls of all involved, raising the spirits, taking them for an enjoyable ride, but people will wonder afterwards what the presentation was about.
  • the golden quadrant, content + entertainment: both interesting and rich content, presented in a way that makes it a joy for the people to hear.
  • let’s not speak about the bottom left quadrant. You don’t want to be there.

This is probably more a spectrum than a clearly defined matrix, you can be closer to one edge of the cell than the other, the limits are fuzzy, but let’s go with this for simplicity’s sake.

I personally think that a skilful presenter will land squarely in the content + entertainment quadrant.

A conference is not a university course. Conferences are often full-day or multiple-day affairs. There is no way you can present too much condensed content to an audience and expect them to stay alert and interested. The idea of a conference talk is to make people curious, encourage them to have a closer look at a topic, library or concept, not to force-feed them information. As we all know, giving an emotional payload to content makes it more memorable. The entertainment part of a presentation is intended to fulfill that role.

On the other hand, they do come to your talk to be informed, so having pure fluff is not going to cut it either. Treat your audience like the bunch of intelligent developers they are, if they wanted to watch pure entertainment they would probably do so elsewhere.

The best recipe is to see the presentation not as a summary of information, but as a story to tell the audience, with a beginning and an end, downfalls and victories. Using gimmicks and effects is allowed: I love Jim Weirich’s presenter style, he has a deep story teller’s voice, and I’ll always remember that presentation he started by whipping out a ukulele and playing a song. Adding personal anecdotes and conveying enthousiasm and emotion can do no harm to your cause. Slides can be colorful and attractive.

Interestingly, conference attendants also situate themselves on different parts of the quadrant. I had a discussion with Konstantin Haase and Josh Kalderimis on this very topic last year, about a presentation that was clearly in the content quadrant. The presenter had interesting material but droned on a bit, and had clearly expended no effort on his slides. Konstantin had enjoyed the presentation, while Josh was very bored and thought it was pretty bad.

Personally, I probably fall a bit too much on the content part of the quadrant. I like to research my topics to the extreme, and my presentation are probably slightly too packed with facts and devoid of ukuleles. One lives to learn.

Speakers of the world, leave your bullet points at home. Bring food to the table, but let your creative minds loose. You’ll only make your presentations better.

! ‘Startups in Europe: Hurdles’

| Comments

I’ve been a tourist in startup world for a little while: I attempted to co-found one, and subsequently sold my services as an IT freelancer to spin-offs and innovative ventures. This doesn’t make me an expert, but it certainly gives me a perspective on the topic of this post: why are there proportionally fewer startups in Europe, as opposed to the constant creative supernova that is Silicon Valley?

Tax

Allow me first to debunk the popular liberal myth that higher tax rates are holding us back. The proponents of this theory argue that tax makes business in Europe anticompetitive compared to the States or other lower-taxed countries. I come from a country, Belgium, that has exceptionnally high tax rates, and I’ve been known to complain about it, but I don’t believe it holds us back from innovating. On the contrary: all this tax goes to health care, cheap education, decent infrastructure, a social net. Sure, there are cases of mismanagement, but the end result is this: a huge decrease in risk for the average citizen. Even if your startup doesn’t make it, you’ll still have a minimum to live on, your child will be able to go to school, your family will receive health care where needed. Compare this to the US system where without medical insurance, you’re screwed, where good education is prohibitively expensive. It could make people hesitate to let go of a stable job.

Secondly, a small part of that tax money is going to grants for innovative businesses – if you’re on the ball, you might be able to get a good boost from the government, and it will be a better deal than what you get from the average seed/venture capitalist.

Tax is not necessarily anticompetitive for startups, if the government makes good use of it.

Administration

Depending on which country you’re in, starting a business might have prohibitive costs and take a lot of paperwork. In Belgium, for instance, it takes about a month to get all the paperwork going, costs you a hefty fee for a sollicitor, and you have to have a capital of 12000eur in the bank. All these rules obviously date from an age when you needed Infrastructure (capital I) to start a business. For an internet business, where all you need is a personal computer, an internet connection and living costs, it’s ridiculous and frustrating.

In the UK the situation is somewhat better, starting a limited company is done online and in one week, and in theory, you’re in business. The tax papers are somewhat more complicated but at least you can go ahead and start trading.

I want to give you money

But if you want an online payment set up, this is where the problems starts. There are several formulas available:

  • the merchant account: if you’re selling a good volume of products, and you’re expecting to do so for a while, this is the best option: a higher overhead cost usually, but lower transaction costs. However, all people I’ve spoken to have told me this is a nightmare to obtain. To get a merchant account you have to demonstrate six months to a year’s trading figures to a bank. This is impossible if you’re a budding startup, so another solution is needed. Even if you have a solid financial background, I’m told banks are still reticent.
  • payment service providers (PSP): higher transaction costs, lower overhead, so ideal if you’re starting up. Paypal is the well-known one, but it has awful customer service and seriously bad PR at the moment. A lot of them still require a merchant account (see above), some of them don’t, but might require an account with a certain bank. They’re still notoriously skittish to accept new business.

The aspect that makes no sense in this is that all you’re asking the bank is to accept money, you’re not asking them to grant you a loan or an overdraft. Of course, you still have the risk of fraud or money-laundering schemes – keeping an eye on the ball for odd transactions can mitigate that risk. The risk is usually put at the seller’s door anyway. What is holding the banks back? And how are internet based startups supposed to make money? It’s not uncommon to get a partner in the States (like Travis are doing), just to cover that aspect.

Failure is the end

In the US, most successful entrepreneurs are serial entrepreneurs. They have a string of more or less successful ventures behind them, some less than more – and that’s considered normal. Failed ventures constitute an apprenticeship. Not so in Europe. In Europe, bankruptcy affects your credit rating aversely. Once your credit rating is bad, the game is up: you’re limited on bank accounts, credit cards, and say bye bye to mortgages or reasonable loans. It’s a long and difficult process to reverse an adverse credit rating. 6 years until this mark is erased according to the website.

In short, you have one attempt, and one attempt only: you better make sure your idea has legs.

Solutions

The administration issue is something that has to be solved via legal routes: talk to your representatives, get things moving. Starting a business should be the easy part of starting a venture. Governments are usually aware of this issue, and are generally making positive changes, but a prod can definitely help.

For online payments we need innovative solutions, like Stripe in the states. Personal and small business banking is due an overhaul, and definitely competition. Several startups have picked up on this, like GoCardless, and we hope to see more solutions in the future. I’m convinced that any solutions that removes this pain will be insanely successful.

The last part is the hardest: the whole credit rating system. I understand that in some cases it makes sense for people to have a generally bad rating – however it definitely feels like the banks are erring on the side of caution, to their own advantage and to the public’s loss. The system should allow for failed ventures. When there was no gross mismanagement and the founders acted with common sense, but the plan just didn’t work out, they should be able to appeal to reverse and bad marks. Otherwise we cannot come to a system where an experienced founder is allowed to start again. I have no clear idea of how to change this, but maybe the legal route is in order here as well.

What do you think?

Transparency

| Comments

2 weeks ago, we watched some of the Strata preview from the Office. Strata is a conference about bhe business of big data which takes place every year.

Michael Nelson’s story: after Wikileaks broke some highly sensitive stories, not only governments but also organizations decided to clamp down on every possible leak and information source. Michael Nelson argued that this is a bad approach – a much better oneis to strive for maximum transparency. The brazilian company SEMCO, striving for total transparency, was one of his examples.

He posited that in most organizations there is a kind of Pareto principle of sensitivity of information: a large majority of data could be disclosed without any damage, only a few percentage are of strategic importance.

Even better, disclosing that non-sensitive information creates a sense of trust and familiarity in the customers. Being open is simply good PR.

I do agree with those points, especially the last one: as examples see the blog post by a Rand Fishkin, and the Peldi’s blog. Both company founders talk about milestones, decisions, joys and disappointments in an earnest way. Their sharing creates a personal connection. The reader feels like cheering for their successes, and lamenting their losses, which does no end of good to their brand.

A more cynical part of me also senses that if you look transparent, people won’t look that closely because they think they know all there is to know. Also, one could be drowning out relevant information in noise, as it were. Not to mention that any displayed infomation can be given the spin it requires.

On the other hand, I feel that a lot of organizations would still be reluctant to go that far: unfortunately, companies can be dysfunctional enough that they wouldn’t like to wash their dirty laundry in public.

While in fact, transparency might be very beneficial for these organizations: public shaming is a strong motivator for change. Something they might want to come to terms with, because in the age of porous boundaries, disclosure might happen whether they want to or not.

Interlude

| Comments

I’ve been slack on the blogging front for several months now – been fairly busy:

I’m slowly regrouping and have a little bit more time to settle here in London. Time to start blogging again – plenty of material to do so.

Getting Cosy With MRI Ruby

| Comments

Have you ever wondered what is going on in the entrails of MRI Ruby ? I certainly have. I started out studying engineering because I wanted to know how everything works. As it turned out, every fact just generated more questions, but that won’t keep me from trying.

A little bit of poring through the source code (ruby 1.9.2) yielded some tools which can provide helpful insights for this research.

You’re probably already aware that a Ruby program is executed like so:

  • parsing: the program is tokenized, lexed and interpreted into an Abstract Syntax Tree (AST). This AST is basically the source code reduced to a tree form, which looks vaguely lispy in construction.
  • compilation: this tree is compiled to a set of instructions for the Ruby virtual machine.
  • execution: the instructions are then executed by the ruby virtual machine.

Here are some tools, built-in in ruby, to see what happens to a bit of code.

Parsing

1
ruby --dump parsetree code.rb

say code.rb contains just one statement:

1
answer = 42

The corresponding AST yielded by this option:

1
2
3
4
5
6
7
8
9
10
# @ NODE_SCOPE (line: 1)
# +- nd_tbl: :answer
# +- nd_args:
# |   (null node)
# +- nd_body:
#     @ NODE_DASGN_CURR (line: 1)
#     +- nd_vid: :answer
#     +- nd_value:
#         @ NODE_LIT (line: 1)
#         +- nd_lit: 42

This gives us an insight in what constitutes an AST node for Ruby.

Compilation

2 ways to see the instructions produced by compilation.

The first one is a dump parameter like the one above:

ruby --dump insns code.rb

outputs:

1
2
3
4
5
6
7
8
== disasm: <RubyVM::InstructionSequence:<main>@test>====================
local table (size: 2, argc: 0 [opts: 0, rest: -1, post: 0, block: -1] s1)
[ 2] answer
0000 trace            1                                               (   1)
0002 putobject        42
0004 dup
0005 setdynamic       answer, 0
0008 leave

The second way to get information somewhat more tricky – in this case a C macro needs to be ‘turned on’ for it to work. A hint in the comments of compile.c:

1
2
3
4
5
6
7
 * debug level:
 *  0: no debug output
 *  1: show node type
 *  2: show node important parameters
 *  ...
 *  5: show other parameters
 * 10: show every AST array

There are two ways to set CPDEBUG: you can add a parameter to the compilation in the makefile -DCPDEBUG=5 Or you can change the default value directly in compile.c

1
2
3
#ifndef CPDEBUG
#define CPDEBUG 5
#endif

In both cases, recompilation is necessary, and will produce more output than usual. Running the same minimal program code.rb yields a whole lot of output about the generated instructions, maybe more than is useful, depending on how deep you want to go.

Execution in the VM

If you want to see output at the execution level: pointers, heaps, frame pointers, there’s another debug constant you can activate. In vm.c, change the value of PROCDEBUG to a non-zero value:

1
#define PROCDEBUG 1

Apparently, this causes segmentation faults out of the box, but by commenting out some stuff I was able to get it compiled. This causes an output like so:

1
2
3
4
5
6
7
8
---
envptr: 0x100482f48
orphan: 0x100887358
inheap: 0x0
lfp:    0x100482f48
dfp:    0x100482f48
<internal:gem_prelude>:1:in `require': cannot load such file -- rubygems.rb (LoadError)
  from <internal:gem_prelude>:1:in `<compiled>'

OK, not very enlightening still. Waiting for the ruby-core’s feedback on that one.


By running these few commands on extremely simple programs, it seems to me that we can get a better insight into how it works.

Note: setting tabstop=8 (in vim, or tab = 8 spaces in other editors) gives you the proper incrementation for the Ruby source code. I had to find out.

The Rise and Fall of Methodologies

| Comments

In my years as a software developer, I’ve seen several methodologies primed as the next big thing, the one way that would ensure a higher success rate of all software projects.

Let’s see:

  • waterfall in large enterprise (ah, the scores of UML diagrams)
  • XP-scrum-agile
  • Kanban

There’s an eternal hype curve, as with technology. Oh look, this will solve all our problems. Then some years laters, outcries because the results don’t match the expectations.

Recently, there’s been some backlash against dogmatic agile. Dan North’s deliberate discovery is one, Fred George’s Programming Anarchy (both good talks, both coincidentally ex-Thoughtworkers), or Zed Shaw’s macho ‘Programming, Motherfucker’, a (hopefully) less serious one.

Here’s the thing:

It’s NEVER easy to drive a project.

Every project needs to be considered on its own, the task to accomplish, the shareholders, the team. Time needs to go into pondering how to handle each of these, how to navigate the ecosystem.

I know we, as developers, would like to be given the golden receipe. Laziness is our virtue, and we would like to have a clear algorithm so that we can just concentrate on the fun parts. And this makes us easy targets.

Gurus will not tell you about complexity.

I’m sure most gurus are genuinely convinced of what they’re selling. They have worked in software, stumbled upon a good idea, and in their own context, applied it with the necessary creativity (hell, they were inventing), it works.

But they’re also in the business of convincing (and maybe to sell books, courses, talks etc). That requires one-liners, clear and simple ideas. Theories that can be grasped immediately. They won’t add multiple caveats to every simple statements, because that could be confusing and off-putting.

There are no easy receipes. Every practice needs to be assessed with a critical eye, and reassessed constantly. In some environments, with some people they’ll work, and sometimes they won’t, and then we need to think, ponder and improvise instead of sticking to the letter of any book.

Not saying that there is no value in existing methodologies. But the only methodology that works lives in the negative space between methodologies.
Read. Pick and mix. Use brains.

A bit like growing up – time to start relying on our own hearts and minds.

HTML5, Pubsub and Browser Push

| Comments

I’m currently part of the team working on the prototype of an interesting web application. One of the features of this application is a constant message stream (think Twitter) pushed to the user.

The browser

In a first phase of the project we can count on a HTML5 enabled browser, so that allows us to use one of the options HTML5 offers for browser push, basically functionalities built in the browser which can be accessed with javascript. These are:

  • websockets: these sockets allow bidirectional communication
1
2
3
4
5
6
7
8
9
10
11
12
13
 var ws = new WebSocket("ws://websocket_host/websocket");

 ws.onmessage = function(evt) {
   doSomethingWithMessage(evt);
 }

 ws.onopen = function() {
   subscribe();
 }

 ws.onclose = function() {
   alert('whoops !  lost connection');
 }

obviously, the events callbacks should be modified to the desired action.

  • eventsource: one directional, push only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
var source = new EventSource('/eventsource');

source.onmessage = function (event)
{
  doSomethingWithMessage(evt);
};

source.onerror = function (event)
{
  alert('whoops !  lost connection');
};

source.onopen = function (event)
{
  subscribe();
};

For older browsers (anything that is not Chrome, Firefox 4, Safari or Opera at the time of writing) the push mechanism needs to be replaced by long polling, i.e. a regular check for the resource.

The server

The interaction between browser and back-end is different from our usual client-server relationship. Essentially, the browser establishes a subscribe relationship to the server. Websockets create a bidirectional relationship, in that they both subscribe to something on server side but also publish back to the server (hence pubsub).

Your standard apache is not equipped for this, because it only handles classic request-response (as far as I can tell, I might be missing something). A server needs to be able to react to events: what is called an evented server. Added to that a publish-subscribe mechanism should be used, to dialogue with the browser.

I investigated two possible configurations, in Ruby: mongrel2 with ZeroMQ, and thin with cramp and redis. Node.js would certainly have been another option, but I stuck with Ruby since it might be easier for maintenance, for the time being at least (node might still be an alternative later).

Mongrel2 with 0MQ

I fell for 0MQ immediately, and will certainly be considering it for future projects, because it has a nice bare-bones feel to it. It aims to supercharge existing sockets and connections. It also has easy ruby bindings.

A 0MQ socket is what you get when you take a normal TCP socket, inject it with a mix of radioactive isotopes stolen from a secret Soviet atomic research project, bombard it with 1950-era cosmic rays, and put it into the hands of a drug-addled comic book author with a badly-disguised fetish for bulging muscles clad in spandex.

Mongrel2 is another story. While I like what it promises, the documentation is a bit sparse, and I had to dig through the internets to get my configuration right. Mongrel2 integrates with 0MQ. It offers handlers, which can be managed with Ruby. Mongrel2 is language agnostic, and stores its configuration in sqlite3. People who have looked at the code (I haven’t yet) tell me it’s clean and well-structured.

Mongrel2’s basic config offers publish and subscribe handler, to have the bidirectional communication with the websocket. Good examples here.

So I got it running, but it took a certain amount of pain, especially in getting routing right.

But it works: mongrel2 creates 2 handlers (0mq queues with pubsub) to interact with the websocket. These 0mq sockets can be connected to from any other process, and that other process can publish any data it needs to push the browser, and handle any messages that come in. Excerpt of mongrel configuration:

1
2
3
4
5
6
7
8
9
10
root_dir = Dir(base='html/', index_file='index.html', default_ctype='text/plain')

esupdates = Handler(send_spec='tcp://127.0.0.1:9999', send_ident='54c6755b-9628-40a4-9a2d-cc82a816345e',
                    recv_spec='tcp://127.0.0.1:9998', recv_ident='')

routes = {
    '/websocket': esupdates,
    '/': root_dir
}
(...)

Since the code of the corresponding ruby handler is a bit long, I put a bare-bones commented handler in this gist.

Redis with Cramp and thin

In the meantime, I’d found another, pure ruby example in the Redis documentation.

Basically, cramp is a layer on top of sinatra which adds the functionality to talk with websockets. It makes writing a controller for the websocket blindingly simple, by adding a couple of handy callbacks. Sinatra (well, Rack) can then run on top of thin, which is an evented server. You need to use the edge version of cramp at the moment, because older versions have MySQL dependencies. The ORM has been removed in newer versions to just leave the controller DSL.

This choice ended up winning, because Redis is also used in another local project so the know-how is there. The fact that configuration and use were extremely simple and in ruby contributed to the choice. It also turned out that having named pubsub channels and the ability to subscribe based on a regular expression fitted well with our problem domain.

Redis has a lot of good functionality combined with a friendly programmer interface: we also use the sets, sorted sets and the key-value store. It’s blazingly fast. Check out the interactive tutorial if you haven’t played with it yet. The maintainer, Salvatore Sanflippo is a nice guy (being paid by VMWare to work on his stuff full time), which always helps.

The principle is basically the same as for the mongrel2 setup: starting a sinatra-cramp app on Thin with eventmachine, which will listen to the browser and publish to it. Cramp provides a number of callbacks to handle incoming messages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
require 'sinatra/base'
require 'cramp'
require 'yajl'
require 'daemons'

class ApplicationStreamController < Cramp::Websocket
  on_start :create_redis
  on_finish :destroy_redis
  on_data :received_data

  def create_redis
    @pub = EventedRedis.connect
    @sub = EventedRedis.connect
  end
  def destroy_redis
    @pub.close_connection_after_writing
    @sub.close_connection_after_writing
  end

  def received_data(data)
    msg = parse_json(data)

    # handle received data here
    case msg[:action]
    when 'join'
      handle_join(msg)
    end
  end

  # message channel is the redis channel which the socket subscribes to
  def handle_join
    @sub.subscribe(msg[:channel]) do |type, channel, message|
       render message
    end
  end

end

Redis is used in this example as the publish-subscribe pipe to make any other process send information to this process, which is then pushed to the browser. I’d like to add the functionality to talk with Eventsource to Cramp (instead of websockets), since we only push to the browser – I’ll see if I find the time.

Update: people suggest using em-hiredis for non-blocking redis, and the slimmer em-websocket instead of going through the layers of cramp. Sounds interesting, and I’ll definitely investigate. socket.io is rumored to provide graceful degradation from the html5 features.

So

Ingredients for browser push:

  • javascript websocket or eventsource in the browser
  • back-end evented server to interact/react to pubsub
  • back-end mechanism that speaks the right http to push messages to the browser

Two possible configurations were presented here, one with mongrel2 and zeromq, the other with sinatra/cramp on thin and redis.

Tail Call Optimization

| Comments

Tail call optimization: when in recursion, you can just reuse the stack of the recursive function you’re calling again and again, instead of creating a new stack every time. This means saving memory and gaining speed, and that count for a lot when you’re, say, calculating a large fibonacci number.

Now I’d been told time and again that Ruby doesn’t do that out of the box, unlike languages like Lisp and Erlang.

Today, I saw this tweet by Jim Weirich, where he pointed to a blog post implementing tail call in Ruby. The code is very clever – it took me a good few minutes to parse it. Have a look for yourself, it’s worth it !

Coincidentally (note: not so coincidentally after all, the person reporting the bug, Magnus Holm, is also the person having posted the blog), this very morning ruby-core closed an issue, showing a feature on ruby 1.9 I didn’t even know existed: the possibility to tell the VM you want tail call !

To use the same example used in the beautiful ruby code I referred to “` ruby RubyVM::InstructionSequence.compile_option = { :tailcall_optimization => true, :trace_instruction => false }

RubyVM::InstructionSequence.new(<<-EOF).eval class TCOTest # tail-recursive factorial def fact( n, acc = 1 )

if n < 2 then acc else fact( n-1, n*acc ) end

end

# length of factorial def fact_size( n )

fact( n ).size

rescue

$!

end
end EOF

1
2
3
4
5
6
7
8

The performance of the vm-based tail call seems to be slightly better than the ruby one, as could be expected.

How does it work:<br/>
The main difference seems to be in compile.c, in the function responding to the friendly name of iseq_peephole_optimize.<br/>
As far as I understand, this function deals with a linked list of binary instructions (corresponding to the program), and tries to optimize its execution by introducing some shortcuts where appropriate.

The relevant code (iobj is the linked list of instructions):

c if (do_tailcallopt && iobj->insn_id == BIN(leave)) { / * send … * leave * => * send …, … | VM_CALL_TAILCALL_BIT, … * leave # unreachable / INSN piobj = (INSN )get_prev_insn((INSN *)list);

if (piobj->insn_id == BIN(send) &&

  piobj->operands[2] == 0 /* block */
  ) {
  piobj->operands[3] = FIXNUM_OR(piobj->operands[3], VM_CALL_TAILCALL_BIT);

} }

1
2
3

If we're in the right situation, the instruction gets a VM_CALL_TAILCALL_BIT.
We find our friend the tailcall bit in vm_insnhelper.c in vm_setup_method

c VALUE sp, rsp = cfp->sp – argc; // rb_iseq_t iseq = me->def->body.iseq; / … */ sp = rsp + iseq->arg_size;

if (LIKELY(!(flag & VM_CALL_TAILCALL_BIT))) { // / clear local variables / for (i = 0; i < iseq->local_size – iseq->arg_size; i++) {

  *sp++ = Qnil;

} vm_push_frame(th, iseq,

      VM_FRAME_MAGIC_METHOD, recv, (VALUE) blockptr,
      iseq->iseq_encoded + opt_pc, sp, 0, 0);
}

cfp->sp = rsp – 1 / recv /;

} else {

VALUE p_rsp; th->cfp++; / pop cf */ p_rsp = th->cfp->sp;

/ copy arguments / for (i=0; i < (sp – rsp); i++) {

  p_rsp[i] = rsp[i];

}

sp –= rsp – p_rsp; vm_push_frame(th, iseq,

      VM_FRAME_MAGIC_METHOD, recv, (VALUE) blockptr,
      iseq->iseq_encoded + opt_pc, sp, 0, 0);
}

} “`

iseq contains the method instruction sequence. rsp contains the stack pointer of the control frame minus the size of the given arguments. sp contains the rsp augmented with the new methods’ arguments size.
In the case of tailcall optimization (else) the next frame gets rewritten to the method instructions, else a frame gets pushed normally.
As far as I understand. I’m not blown away by the naming of the variables or the structure, though I can’t claim I would have done an any better job since I’m not that familiar with the subject matter.

Since this tailcall optimization is an option and not a default, and since it’s not one that is widely publicized (to my knowledge) (although I did find some threads when googling), I’m guessing that it’s either experimental or that there are some downsides to it. Does someone know ?

Update: Magnus Holm (@judofyr) added notes in his blog post about this feature, and also benchmarking between about 5 ways to implement this ! nice.