Jay Phillips and Mark Spencer Square Off Over Asterisk Critique

Jay Phillips, the creator of Adhearsion, a RUBY development framework for Asterisk, recently called out Digiumfor failing to make Asterisk sufficiently accessible to developers. While it is clear that Phillips has his own entrepreneurial agenda, he asserts that Asterisk alienates would-be developers by presenting an often cryptic, non-intuitive interface.

Jay goes on in detail concerning a laundry list of issues that he believes are limiting the proliferation of Asterisk beyond first adopter types.

In response to Jay’s diatribe, Mark Spencer of Digium responded with his own lengthy counterpoint, the text of which I have reposted below. As I said, Mark’s retort is extensive, but I found it to be a pretty fascinating collision of viewpoints.

Recently Jay Phillips of Adhearsion fame sent me
a blog he’s written about Asterisk and asked me
to respond to it, which of course I’m happy to
do. First, I have to say that I met Jay at
Astricon and was immediately impressed with his
energy and excitement around Adhearsion. To say
Jay is an out of box thinker is probably
something of an understatement, and it is great
to have people working with Asterisk building the
types of tools that Jay is developing. Jay and I
have talked quite a bit with each other on
subjects regarding Asterisk and Adhearsion, and I
recall an article on Adhearsion was written
by Jay on the dock of my weekend place. I hope
that we can continue this discussion as a
conversation of ideas regarding how Asterisk can
continue to evolve, even if we disagree on some
of the notions that each of us bring to the
table. While I may not entirely understand some
of Jay’s issues he outlines in his posting,
perhaps my comments on some of what I believe
are his points will help us both better
understand where improvements or clarifications
can be made.

I’m going to try to address Jay’s blog point by
point, but it’s made complicated by Jay’s having
mixed so many business metaphors and by leaving
terms like “market” unspecified or variable.
Also, Jay, like my mother, seems to have issues
differentiating between “fact” and “his personal
opinion”. Much as mom thinks I should like
salmon because “it’s good”, Jay makes a number of
assertions which are simply his opinion, but he
states them as fact. I’ll try to point some of
these out, and where possible I will try to use
actual metrics to either support or refute those
claims, and where not possible, some research
would be required to be able to support them.
Lets start with Jay’s largest “one thing” he
wants us to understand, that “Asterisk Has Not
Crossed the Chasm Yet”. The first, most
fundamental problem with trying to address such a
statement is that Jay hasn’t actually chosen a
market. It’s much easier to try to decide if the
“iPhone” has crossed the chasm or not because its
purpose is generally clear. It’s a phone, and
the market is consumers. But what exactly *is*
Asterisk? Most people think of Asterisk as a
PBX, even though it is a great deal more today.
Is he saying that it has not crossed the chasm as
a PBX? Jay is particularly interested in
Asterisk as a platform underlying his technology
for developing applications. Is it that Asterisk
has not crossed the chasm of being the defacto
development platform of choice for IVR
applications? For conferencing?

Jay says he feels like Digium has been caught up
in the misconception of the growth of the
visionary market. In reality though, Digium’s
primary market is not Jay’s market, because
Digium’s *commercial* focus is primarily on the
SMB and to some degree Enterprise PBX market –
almost certainly the largest *single market*
telecom products that you can build with
Asterisk. Digium’s *primary* business is not to
sell to developers like Jay, it’s to sell to
customers like Joe’s Laundry down the street,
through resellers and channels that almost all
fit in the early and late majority category.
Digium’s work on expanding Switchvox and channels
clearly indicates that we *clearly* understand
the importance of that market and precisely *are*
doing the right things to cross the chasm in the
PBX market. We understand that the early and
late majority customers are looking for a PBX
product that is feature rich, inexpensive and
easy to use, and that’s precisely what we’re
building. You’re not going to be able to take
regular open source Asterisk without any
packaging and try to take it to your average end
user. It needs to look, feel and smell like
something they can actually use, and doing so
demands the creation of a derivative work focused
on the market you’re going for, and *that’s* what
you’re going to cross the chasm with. Jay says
that Asterisk isn’t sticky because it’s too hard
for an end user to use — and for many end users
that may be true, but that’s not the product that
*should* be bringing them — it’s AsteriskNOW or
Switchvox or some other, more cooked product,
that is the right thing if you want a PBX.
That’s like saying that Chevy 350 small block is
a lousy product because most people don’t use it.
It’s not a bad product, it’s just that people
want cars, not just motors.

Having said that though, the direct market for
Asterisk, as a project, is of course not early
and late majority PBX customers — it’s
developers who want to build products that
customers in turn will actually use. I *think*
this is the “market” that Jay is actually
addressing with his “Otto” example, and it is a
market Digium is trying to pursue for Asterisk as
the benevolent maintainer of the software — not
as its primary commercial interest (although
don’t get me wrong, we’re happy to sell our wares
to developers as well!) Clearly if he was a
traditional PBX customer without technical
background, downloading Asterisk and building /
installing it on his own newly installed Linux is
not likely the best place for him to start. If
Otto is a developer, then that’s probably a good
way for him start (although AsteriskNOW might be
a little better place to start). Jay calls
asterisk “prohibitively unintuitive” and if
you’ve never used Linux or a command line, that
may be true, but if you’re a developer, you’ve
configured tons of things before, and something
like AsteriskGUI can get you off to a running
start from which you can start fooling around
with some of the developer technologies, although
currently the best single source of those tools
may be Google.

Jay drives his opinion that “the average amount
of time a developer plays with Asterisk is less
than one week” and that “the developer retention
rate of Asterisk is less than 1%” but in spite of
being in bold print, these are not statistics but
merely his opinion. To my knowledge, no one has
ever really researched this, and I’m uncertain
even anecdotal evidence of “sticky” numbers of
developers is significantly different than any
other major project that sees experimental use by
the overall OSS developer community. In any
case, Jay’s premise seems to be that Asterisk is
simply too difficult for a developer to build
from. Fortunately for us (and unfortunately for
his argument), such a premise is largely in
conflict with reality. For example, lets look
just at a series of recent projects built at the
Interactive Telecommunications Program (ITP) at
New York University:

* Botanicalls – Your plants call you when they’re
low on water. Each plant has its own personality
based on its species. Further development is to
include other environmental factors.

* I-PLATE-U – Record and retrieve messages based
on someones license plate. Heavily uses speech
recognition to avoid having to type on the
keyboard

* Popularity Dialer – Calls you and plays one of
5 different pre-recorded “half conversations”
including the “Cousin needs help” call, to help
you get out of a sticky situation and the
original “popularity” call where your friend
calls trying to get you to join your other
friends, but you decline because you’re on a
wonderful date.

* Cause Caller – Automates the process of
allowing you to call in to politicians in support
of various political causes.

* Booty Dialer – Automatically calls each entry
on your “booty call” list until someone responds
positively to the interrogatory, at which point
your phone rings and you are connected with them
directly, saving lots of time.

* Canal Street Station – Interactive telephony
mystery which is played out by calling a toll
free number from various public pay phones in the
canal street station

* Rap Happy – Allows you to use your phone to
record rap music (presumably other music) via
your phone.

Now, obviously these are extremely creative
applications of Asterisk, but what is perhaps
most interesting about these is that the ITP
program is not an engineering program — nor are
the creators of these applications engineers –
they’re artists, in the art department. This
program has taught them to use Asterisk as a
medium for creating telephonic art, or at least
taught them how to accomplish telephony tasks
without having a CS degree or years of
development and sysadmin experience. Of course
there are lots of other business examples that
actually generate revenue, but the point is that
lots of people make real applications with
Asterisk today, even in its current form.

So, do I think there’s no truth to anything Jay
is suggesting on the API? First of all, like
Jay, both Digium and myself want to see the
advancement of the technology worldwide — that’s
why we invest so heavily in Asterisk. How about
this for a fact and — the largest single
engineering team at Digium is the team dedicated
to Open Source Asterisk development. Certainly I
agree there are improvements that can be made.
Asterisk has at least three different ways to
“program” it, including the manager API, the AGI
interface and the various extensions languages
like AEL and LUA. Like Jay says, Asterisk and
open source telephony are virtually synonymous
because Asterisk has been around for such a long
time that it is able to talk to virtually every
VoIP and TDM telephony technology. It has
virtually every core traditional telephony
application (like voicemail, call queues, ivr,
conferencing, etc) already integrated and has the
maturity and stability to be relied upon by a
wide variety of OEMs, enterprises and businesses.
What Jay, Digium and I all want to do is to make
those technologies as accessible as possible to
the widest variety of developers who are going to
take this core out to new places and create more
applications that haven’t existed before.

Jay proceeds to enumerate several specific
concerns about Asterisk which again I’ll try to
address. Firstly, he mentions that if you build
an Asterisk system, you have to be a developer.
This is no more true of Asterisk than it is of
Apache or even SSHD for that matter. Each of
those applications has various configuration
files that control its behavior. In each case,
you can either use a graphical interface or you
can choose to configure the system by manually
editing the files. That’s a matter of personal
preference. By Jay’s suggestion, anyone editing
a config file is a developer. There is probably
a continuum between developer and user wherein as
a system becomes more flexible, and more options
are presented to the end user, their need for
understanding and abstracting concepts becomes
higher. This is precisely why Asterisk is a
product for developers, and products like
AsteriskNOW or Switchvox are for end users, and
simply utilize Asterisk as the functionality
provider. I know Jay is annoyed that
extensions.conf is Turing equivalent, but hey,
that’s not how it was intended to be used, and if
you choose to use that even when there are plenty
of other, easier to understand, easier to read
ways of configuring Asterisk, that’s your
prerogative just like you can choose to write in
assembly language if you wanted to. Just because
it’s there doesn’t mean you have to use it that
way. What extensions.conf *is* useful for is
listing a bunch of extensions — precisely its
original purpose. And if you don’t want to use
it, use the Asterisk GUI. Although Jay describes
this as “irrefutable” evidence that configuring
Asterisk makes you a developer, it no more makes
you a developer to configure Asterisk than it
does to use the bash command line, which by the
way is substantially more complex, also is Turing
complete, and people also use primarily for the
purpose of simple things (launching applications
for example) and not sophisticated software
development.

Jay also seems to be concerned that Asterisk is
written in C. One of Asterisk’s core missions is
to scale and be efficient. Writing in a language
like C that supports that goal is important to
allowing these creative apps that live on top of
Asterisk to be able to scale to the number of
users that end up wanting to actually use them.
Yes, programming in C requires you to be careful
in your programming, but I think it’s worth the
extra effort to write a good program once to
allow the end user fantastically better
scalability in the long run.

Leveraging Asterisk beyond just being a PBX is
typically done at different levels depending on
the sophistication of what is desired. If you’re
just doing a basic IVR, you can largely do that
within AEL (or heaven forbid, extensions.conf)
using many of the applications and variables
included within Asterisk. Granted, this is
learning another language, but it’s a language
very specifically designed for operating within a
switch, and its efficiency is tremendous because
it’s so tightly coupled to the operation of the
system. Beyond the basic IVR’s, people start
wanting to use a full-fledged general purpose
programming language. Which language to use,
however, is clearly not an area of agreement, as
evidenced by the AGI and AMI bindings for Perl,
Java, PHP, C, .NET, and many moreŠ and of course
Adhearsion for Ruby. One important thing that I
think we can try to do within Asterisk is to make
primitives that more easily support binding of
other languages. Jay talks a lot about framework
– and framework is a great concept — but there
are lots of languages out there, and they largely
each have different frameworks and programming
paradigms about them, and each attract different
sorts of developers and lend themselves to
different ideal applications.

Asterisk, as with most telephony software,
suffers from misperceptions about goals, and
there are complaints no matter which side of the
double-edged sword is being swung. “It’s too
hard to configure!” is the cry that I hear from
this post. But equally loud is the group who
yells “I want to control all the details.” We
agree that Asterisk is complex; but telephony is
complex in the details. This is not dissimilar
to Linux, firewalls, routers, mobile phones,
Apache, databases, or a host of other complex
systems which have various and sometimes
competing layers of control put in place over
them to simplify adoption and integration.
It is possible to abstract complex sets of
primitive elements once you have a full
collection of those primitive elements. It is
*not* possible to go the other way around.
Asterisk (the project) is working towards a
(more) complete set of basic functionality which
allows a developer to achieve as many telephony
tasks as possible, even when some of those tasks
may be complex to develop if you are creating
them from scratch. When we say “developer” in
this instance, it could mean either someone
directly building an application, or someone
building an API into another programming
environment. We are hoping and encouraging
projects like Adhearsion, FreePBX, the various
language libraries, and others to create
abstraction layers and configuration layers on
top of the Asterisk toolset in order to
accomplish specific goals which may be far
outside our ability to comprehensively predict or
develop. That is what we envision as our path
towards the “Tipping Point” for application
developers.

Jay continues by trying to describe a view of how
an Asterisk System and an Asterisk Application
are different. If you get at the core of the
concern though, it’s that he thinks people build
Asterisk systems in ways that aren’t the ways he
wants to do them. He says they’re built the way
they are because of deficiencies in the
extensions.conf configuration format. Only
problem with that logic is that if the
extensions.conf format was deficient, they would
obviously have used one of the other wide variety
of programming techniques available (maybe even
Adhearsion). If someone chooses to use AEL, or
even extensions.conf to create an application of
Asterisk, it is precisely because they *were*
able to do it with those systems that they chose
to do so, not because they couldn’t — again,
having those systems doesn’t take away from your
ability to use any one of the native languages
and frameworks that are available for building
applications on Asterisk. Said another way:
Don’t complain about what languages people use
(even if you want to call extensions.conf a
language). Use what you like best, and let
people choose what language they want to use!
There certainly are a lot of them! Asterisk is
*already* being used as an underlying layer by a
huge number of abstraction libraries – and we’re
hoping to continue that trend to make programming
telephony applications easier by developers in
their own environments. You don’t need to be a
dialplan wizard to write applications that use
Asterisk: “There’s more than one way to do it.”

Now, another point that Jay makes that I think is
valid (and one Digium is trying to address) has
to do with how you install additional features
and applications in an Asterisk-based
environment. The Asterisk Marketplace is
precisely intended to be a place to allow people
to have applications more readily distributed,
including through graphical interfaces, to be
installed not just on Asterisk but on PBX
products like AsteriskNOW and Switchvox. Sorry
to be vague on this one; stay tuned.

To continue on: I think the “The Long Tail”
illustration cited could not be more perfect. If
you look at the green portion of the curve, that
is where Digium’s primary business focus is
placed, and if you look at the yellow portion,
that’s where Digium’s primary focus for Asterisk
is placed. The former is best address by a
product and a company, and the latter by a
project (in this case sponsored by a company),
where we enable lots of small micro-markets that
can be attacked by a variety of developers
enabled by technologies like Asterisk either
directly or through interfaces like Adhearsion
and others. We actually agree on this point-
there are more end users (ultimately) on the long
tail, even though that long tail is very long and
thin. Asterisk, the project, is doing all it can
to help with building apps out on that long tail
by creating a free toolset that developers can
use to address their infinite niche markets.
This “long tail” will exist because of tools like
Asterisk, not the other way around. Digium, the
company, is creating a set of products based on
or around Asterisk that address the head of the
market, which by far is the largest single niche
in the diagram by far. One end of the diagram
supports the other – the products that Digium
creates around the largest and leftmost portion
of the curve creates and supports the tools that
will enable the proliferation of independent
applications that fill the “tail” on the right.

After reviewing this article a few times, I am
more unclear on what the issues are that Jay is
actually raising. This perhaps is due to my own
myopia when looking closely at Asterisk and also
having a pre-conceived notion of what I feel are
the shortcomings of the project today. As a
possible continuance of this discussion, perhaps
a succinct list of problem statements would help
me in refocusing what we need to do to solve what
are obviously to Jay issues of concern.
Certainly I wouldn’t claim Asterisk is perfect,
but the developers in the community and within
Digium are interested in hearing the things that
are necessary, and then as they are needed, the
code can be written to fit the solution. I
appreciate that the requirements are changing as
Asterisk becomes more mature – we’ll try to stay
on or ahead of the curve based on the
perceptions, requests, and most importantly, the
work and contributions of the community.

Reading between the lines in this post, it’s
clear that Jay has what he believes is a great
idea with Adhearsion. I don’t disagree – it’s a
very neat solution for people working with Ruby
On Rails, and simplifies life for developers in
that particular arena. Perhaps working so
closely with high-level abstractions makes the
details of telephony a chore – this is
understandable, but I believe that additional
tools have been and will be built that allow us
to cross “the chasm” for both developers and end
users – I think we’re crossing that chasm
already. Perhaps after waiting a few months it
will be the case that Jay’s company will
announce a more commercial solution to the set of
problems he ambiguously identifies today; if so,
I would caution him not to become myopic in his
own way when identifying the uniqueness of his
own solution and how it resolves perceived
shortcomings of Asterisk. We’re happy to see
more interest building around the Asterisk
ecosystem, and we hope to help Adhearsion and
other firms build on Asterisk as the engine for
their telephony applications.

In summary, I’d say that Jay and I share the same
goal of making Asterisk a more accessible
technology both for developers and for end users,
and I’ve invited his participation in the future
of Asterisk at the next Astricon 2008 (and
subsequent developer meeting), which he has
eagerly accepted to particpate in! I look
forward as always to working with him.

Mark