Modules, the basic building block

A Zotonic web site is made of different functional parts:

• Controllers, in fact Webmachine resources.
• Templates, all what you see. Templates can include other templates.
• Screen components, scomps, those are templates with their own little controller.
• Actions, everything that you can do with buttons, forms, jQuery effects etc.
• Validators, used to check submitted input elements.
• Css, Javascript, static images. Those are all put in lib directories.
• Dispatch rules. They map a request url to the correct controller.
• And a bunch of supporting Erlang code to make it all work.

A module contains one or more of the above items. All templates, actions, validations, css, javascript, dispatch rules and scomps are contained in modules. Modules have a priority and can overrule components of modules with lower priority.

Modules are directories with at least one Erlang module implementing a gen_server that manages the module. Subdirectories in the module contain all the templates, Webmachine resources, dispatch rules, actions etc.

Modules communicate with each other using notifications. Modules can be activated or deactivated using the admin interface (which is implemented using multiple modules).

Data Model

The data model is inspired by semantic web triple stores. But then adapted to make it work with a relational database and usual CMS interfaces and methods.

All (ok, most) data in Zotonic is a resource. Resources are connected with edges, creating a directed graph of information. Every edge has also a predicate. The predicate defines the meaning of the edge. For example author, as in the author of an article.

For example, a news article is a resource. The image of the news article is another resource, connected to the article with an edge, and so is the author of the article, the place the article is happening etc.

You could say that Zotonic has only two tables. A resource table and an edge table.

Resources are typed using a hierarchical category system. Some predefined resource categories are: text, news, image, video, sound, person, group, predicate, category and other.

A Data Resource

A resource has some fixed fields and can have any additional fields. The fixed fields are:

• Id, an unique integer.
• Title.
• Slug. Used to make understandable uris
• Introduction, a short abstract used in lists.
• Body, the textual body of a resource. A body contains HTML.
• Unique name, used by the system to find resources by name instead of id.
• Featured flag, to show things at important places on your site.
• Published flag.
• Publication period. From when till when a resource is published.
• Uri. The originating uri of the resource when it is imported from another site.
• Authoritative flag. Defines if the resource is imported or created locally.
• Category, defines what the resource represents. For example a person, article, event, image or video.
• Group. Editors work together in groups, all content belonging to a group can be edited by editors in that group.
• Visibility level. Defines if the resource is viewable by the group, community or the world.

All textual fields (except name and uri) can be multilingual. Any resource can have multiple extra data fields. Think of a date range, name or address.

In the next article I will show a bit of a template.

Zotonic enables hosting very popular sites on simple hardware.

I will write a couple of articles highlighting aspects of Zotonic. Today I focus on the reason we built a new CMS and on performance aspects. In other articles I will highlight the general design of Zotonic, the template system, modules, screen components, form handling, and adding logic (actions) to the templates.

Why?

Yet another CMS? Why? Because the current crop of open source content management systems don’t do what we want them to do. What do we want to do? We have a simple list of requirements:

• Flexible and easy to learn template system;
• Separation of logic, viewer and data store (MVC);
• Rich data model, easy to add knowledge to the data;
• Easy to make highly interactive web sites;
• High performance from a single machine for simple hosting;
• Future proof, easy to integrate with XMPP and other services;
• Easily extensible with modules;
• Able to handle push notifications with Comet;
• Easy and fun to program the back end.

The Open Source building blocks of Zotonic

Zotonic would not be possible without the contributions of many open source projects:

• Django like template engine (an adapted version of ErlyDTL);
• HTTP protocol handler and abstraction with Webmachine;
• Event driven user interface based on Nitrogen;
• The best scalable open source database around: PostgreSQL;
• A great CSS framework: Atatonic;
• jQuery for the user interface.
• epgsql For connecting to the database; and
• esmtp For sending e-mail.

Zotonic is written in the programming language Erlang. In the next section I explain why we use this programming language.

What is a web site anyway? Or why we choose Erlang

Once upon a time a website was a collection of manually edited HTML pages and some images. All was fine.

Then search was added to the site, necessitating extra programs running on the server on behalf of a web site visitor.

Then those pages started to be filled with content from a database, nothing changed much except that editing and searching got a lot easier.

And then we started to let the visitor react, interact and upload information to the server. Not only the visitor was allowed to interact, also machines got their own interface to talk to the server. Web 2.0 with APIs and user generated content took off.

Nowadays a web site is talking in many different ways to humans and machines. It is collecting information from multiple sources and actively submitting new articles to other sites or machines. The web server gives the visitor more and more ways to access the stored information. People are using RSS readers, widgets, desktop applications, other web sites and so on.

The web server evolved from a simple machine serving some pages to a complex switchboard connected to many different servers and services. The web server constantly sifts through information and storing it on behalf of its users. It constantly generates different representations of its own stored information, let is be as HTML pages, RDF documents, Atom entries, XMPP messages or API result sets.

This new web server is an always on machine. Not a simple server reacting to requests from a web browser.

This new web server is way more dynamic than yesterdays database based HTML generator. It needs to stay connected with other servers, it needs to have all those connections open and active at all times. It even needs to push information to the web browsers of visitors.

This new web server needs a different approach and programming model which becomes more and more like a information switchboard with thousands of open connections to other machines and web browsers.

Erlang was created to program telephone switches. Its programming model and features are a natural match to this new kind of web/information server. Besides its natural match to the problem domain it was also created to make very robust systems that are always on. No need to reboot an Erlang system for a program update.

That is why we choose Erlang. To create a robust web server that can be like a spider in the web of information. And one that is still easy to program and extend.

Performance: Scale up? Scale out? Scale in!

The current generation of computers are so fast that we can play multi user video games on a single machine, whilst surfing the web, checking our mail and what not. Why should we be happy with generating a couple of web pages per second from a machine that is so capable? How green is that?

We have quite a lot of experience building content management systems and web sites in general. What did we learn? For one thing: when you want high performance and flexibility and less machines, then you are in a lot of trouble.

Usage scenario

Our usage scenario is:

• A fairly popular web site experiencing periodic surges of visitors;
• The web site uses modern technologies like Ajax and Comet; and
• Multiple web sites on a single server.

So we are not targeting the new Flickr or similar sites. They need custom built software, not an off the shelf CMS. We are targeting the other 99 or so percent of the web sites.

PHP

PHP is the most popular language for building web sites.

For every request PHP wakes up and starts finding out where he is, where the code is, what the code is, where the database is, etc. etc. It does this again, and again, and again. Even when nothing is changed. This share nothing approach is nice when handling a single request, or when scaling out (adding more machines). It is not nice when you want to handle lots of requests from a single machine.

Ruby on Rails and Django

Other scaling problems arise with Python (Django) and Ruby (on Rails). Typically they are single threaded, that is they use only one thread of those 32 or more hardware threads that are available on modern hardware. That is solved by having more independent processes, each handling requests one by one. Those processes need to communicate with each other, adding more processes and complexity for caching systems etc.

Zotonic

Thanks to Erlang, Zotonic is completely multi threaded and scales linearly when adding more cores. All information about the web site is kept in memory and easily shared between requests. When two requests need the same html fragment then that fragment can be only rendered once. Note that this is not caching, which Zotonic handles as well, but a simple prevention for doing the same thing twice at the same time.

Zotonic typically serves 100 to 500 dynamic pages per second on normal of the shelf hardware. Which is enough for the fast majority of web sites.

Zotonic can serve a complete page before a Joomla/Drupal/WordPress site has even loaded its PHP code. With a Zotonic server you can serve either many more sites, or many more visitors per site.

Later more about the architecture of Zotonic and how to use Zotonic.

WordPress resizes uploaded images to two different sizes. A thumbnail and a medium format. In most designs you would really like to control how an image is used on the page. Sometimes it will be a small cropped thumbnail, another time a bit bigger, or a nice medium format on the post pages themselves.

The wok-imager plugin enables this without you resizing the images for WordPress.

Tested with PHP 5.2 and WordPress 2.7 and 2.6.3. Many thanks to Tim Benniks for testing it with WordPress 2.7.

Installation

Short version:

cd wp-content/plugins/
svn checkout http://wp-wok-imager.googlecode.com/svn/trunk/ wok-imager
Go to the admin page of your site and install the plugin.


How it works

Just add an image element in your page:


<img src="/wp-content/uploads/file.jpg" width="100" height="100" filter="crop,gray" />


The wok-imager will scan your generated page upon serving, apply the width, height and filters to the image and change the tag into something like:


<img src="/wp-content/uploads/wok-imager/file.jpg/4ee630a60ec7feb605516ac5b98972d1.jpg" width="100" height="100" />


When you don’t supply the crop filter then the width and height are the maximum bounding rectangle of the image. The resulting image has either the given width or the given height but is never bigger than the given width and height. You can also supply just the width or the height to make the resized image unbounded in respectively the height or the width.

Available filters

You can combine multiple filters by separating then with a comma. For example “crop,flip,gray”.

crop

Crops the image to the size given by the width and height attribute. Default is a center crop. You can have a crop aligning to the top, bottom, left or right by adding two extra letters (cropcc). The first letter defines how to align vertically and the second how to align horizontally. Valid values are t,c,b and l,c,r respectively.

Examples:

crop center crop
cropcc center crop
croptl upper left corner
cropbr lower right corner

scale

Allow the image to be upscaled. Per default wok-imager does not allow an image to be resized to bigger dimensions than the width and height of the original image.

flip

Horizontal flip of the image, swaps left and right.

flop

Vertical flop of the image, swaps top and bottom.

blur

Gaussian blur of the image.

smooth

Smoothen the resulting image. Default is a smooth factor of 6 is used. You can supply another with eg. smooth=4.

bright

Correct the brightness. Default is a brightness factor of 50. You can supply another with eg. bright=100.

contrast

Correct the contrast. Default is a brightness factor of -10. You can supply another with eg. contrast=-50.

gray

Convert the image to grayscale.

sepia

Convert the image to sepia.

colorize

You have to supply the three rgb colors with this filter. For example: filter=”colorize=100:100:100″

negate

Negate all colors, makes the image look like a color negative,

q=quality

Sets the quality of the jpeg conversion. When you don’t supply the quality then the quality depends on the size of the resulting image. The smaller the image the higher the quality setting.

Imager plugins

You can add your own imager plugins. There are two examples in the imager-plugin directory.

deleted

Adds a translucent red cross on top of your image.

example

Adds a small text on top of your image.

I maintain an implementation of OAuth. Whilst checking existing code and implementing my own version I wondered how some implementations will differ.

OAuth uses RFC3986 for the encoding of URI parameters. Another often used encoding scheme is RFC1738 or RFC2396 (updated in RFC2732).

The differences? Quite a bit.

The unreserved characters in RFC3986 are:

unreserved = ALPHA, DIGIT, '-', '.', '_', '~'

And in RFC17138:

unreserved     = alpha | digit | safe | extra
safe           = "$" | "-" | "_" | "." | "+"
extra          = "!" | "*" | "'" | "(" | ")" | ","

And in RFC2396/2732

"A" .. "Z", "a" .. "z", "0" .. "9",
"-", "_", ".", "!", "~", "*", "'", "(", ")"

What do some programming languages use?

PHP

PHP has two options. Use urlencode or rawurlencode. The difference between the two is the escaping of the “+” character.

rawurlencode claim to fame is to be compatible with RFC1738. In fact it is not. It encodes all characters as in RFC3986, and then also the “~”. So rawurlencode comes close. It escapes everything, except:

A-Za-z0-9\-_.

So the only difference here is the “~” character. The correct funtion will be:

function oauth_urlencode ( $s )
{
    return str_replace('%7E', '~', rawurlencode($s);
}

Perl

CPAN implements URI:Escape. This one uses RFC2396/2732 as the basis for its character encoding. The default for this function is to escape everything except:

A-Za-z0-9\-_.!~*'()

Javascript

Yet another encoding method. Most javascript programmers will use the function escape. This one encoded everything except:

A-Za-z0-9\-_.+*/@

Addition from John Kristian: when you want to encode with Javascript you can take a look atOAuth.percentEncode in oauth.googlecode.com/svn/code/javascript/oauth.js

It is a mess, what to do?

I assume that programmers are lazy and will re-use their good old trusty encoding functions. They should, it is how we train programmers. This will give some interesting problems with OAuth though, as OAuth insists on encoding and sometimes double encoding parameters. And uses those encoded values to calculate the signature of a message.

This insistance on using a very specific implementation of encoding is, in my opinion, the Achilles heel of OAuth.

When checking the signature we have basically two options:

1. Not decode whatever comes in before recalculating the signature. In that way we won’t make any false assumptions about parameters being encoded one way or the other.
2. Recode all incoming parameter values and names, in that way we make sure that we use the correct encoding (according to the spec) for our signature calculation.

However, we must assume that the encoding used in calculating the signature base string and the key from the consumer secret and the token secret is correct according to RFC3986.

Conclusion

We, as a community using OAuth, need to create test sets that will test all edge cases, otherwise we are in for a rough ride. And, to help other programmers, there is an immediate need for correct implementations of the RFC3986 parameter encoding.

Update

The OAuth community has a nice set of testcases online at wiki.oauth.net/TestCases Let’s all use them and make sure that our implementations are correct!

All these networks have one thing in common: they are walled gardens, islands that force their users to rediscover their friends and re-explain who they are.

At Mediamatic Lab I have been exploring methods to connect all those islands, creating an archipelago where you can travel from island to island.

Social Networks Should be Fun

A common interest is what fuels most sites. It brings people together, it leads to discussions, creativity and close relationships between people. Groups, hobbies, clubs, it is the glue of society. Social networks bring that glue to the internet, make it possible to interact from our homes to interact with the very same people we know from the sport field, that conference or some photo we shared online.

We have more than one interest. We might have more than one career. And we definitely have more than one group of friends. All those groups will most definitely not be present on one single website, and probably will have a multitude of different online places where they meet. The website of the rowing club, the website of a cultural foundation, the website of the family.

Here the problem starts.

Social Networks Are Walled Gardens

Creating and maintaining profiles on social network sites leads to a very real social network fatigue. More and more people simply refuse to become member of yet another o-so-cool website. I am one of those people who refuse to become member, it is just too much. And I definitely do not maintain my profiles.

People not maintaining their profiles is a shame. It makes sites less interesting, and gives less incentive to create an active community around a common interest. And that active community was the goal in the first place.

Established sites try to boast their user base into the millions of people. Just to make the chances bigger that your friends are on that specific site as well. But what if my active friend group is somewhere else? What if my organisation wants to have their own identity on the web? Their own editorial content? Their own responsibilities? Do you think your organisation wants to be a Facebook or Hyves group? No way, they want to have their own site. Their own people to stay in contact with, their own identity on the world wide web.

A Network of Sites For People With Common Interests

Organisations want to retain their own identity and keep control over that identity. That is why they mostly have a website of their own. Those organisational websites are slowly transformed into social networks to connect with interested people, members, associates and other people.

To be able to get people participate in your site you have to make it easy to become member. There is technology for that, it is called OpenID. OpenID is a way to authenticate at a website using the credentials of another website. Say signing on on Mediamatic.net, using your yahoo.com username.

Besides authentication there are also initiatives on the way to make it easy to distribute updates to your profile to other websites you are member of. And the same for your friend lists, why make friends for the 2nd, 3rd, 4th time? You know those people already, let the websites figure out the rest.

So you have your presence on multiple sites, you have a central profile, and you are using your OpenID to sign on. And now you switch your blog from Blogger to your own site. You want to move your nice photos along with you, your articles, your bookmarks, your whole online blogging world.

How to do that?

It Is Your Data, Your Personal Network

It should be possible to move all your data from one site to another. Leaving behind references at the old location to the new location. It is all your data, you made it, you created it, you own it. So you should be able to pick it up and move it to any other website or maybe even multiple sites.

The same for you account (or profile). When you leave a site, pick it up and move it somewhere else. Leaving a reference to the new location so that people (your friends!) can still find you.

In this way we are talking more about your personal network that is facilitated by a multitude of websites. Your photos on Flickr, your blog on wordpress, your musings on your own site, your career on LinkedIn. All of those sites working together, using the photos on Flickr on your blog, your CV on LinkedIn on your personal site.

The idea is that you become a member of a site because the for you interesting people are there. When your interests change, your location changes. You can move houses, let us make it possible to move your data as well.

What Now?

We are working hard to make it possible to have your own personal network, to give you the ownership of your data. Mediamatic Lab hosted workshops where we have discussed the technologies and user scenarios involved with other partners in the industry. You can read an article about the technologies we propose.

We quickly found out that the problem was related the IP address of the user. As some users’ laptops did work at their office, but not at their home.

Weird.

Testing Long Urls

We then checked our server logs, and found that the machines of the affected users did not even try to load some JavaScript libraries.

Some library and css files were requested, some were not at all. No mention of the longer urls in our logs whatsoever, none, nada.

Different user agents, using different machines, from the same IP address all showed the same problem.

So we started to make a test script that tried to load JavaScript libraries using different urls. After some versions of our script we found an interesting pattern.

What is the difference between these two urls:

www.example.com/jstest/f/TEST/library/Ajax/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/test21.js

and:

www.example.com/jstest/f/TEST/library/Ajax/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/test22.js

The big difference is that the first one does load and the second one does not load. Ok, the last one is a bit longer, so rest assured that the following url does load as well:

www.example.com/jstest/f/TEST/library/Ajax/somelangurltext-somelangurltext-somelangurltext-somelangurltext-somelangurltext-somelangurltext-somelangurltext-somelangurltext-somelangurltext/test12.js

Conclusion

It is not the length of the url, but the number of directories in the url. The break point seems to be around 19 directory levels.

It all seems to be related to the Web Browsing Interception feature of SpeedTouch modems. Besides that that option makes the modem around 30% slower, it also intercepts some long javascript or css url lookups.

We updated our CMS to use less slashes in the urls…