Modules, the basic building block

A Zotonic web site is made of different functional parts:

• Controllers, in fact Webmachine resources.
• Templates, all what you see. Templates can include other templates.
• Screen components, scomps, those are templates with their own little controller.
• Actions, everything that you can do with buttons, forms, jQuery effects etc.
• Validators, used to check submitted input elements.
• Css, Javascript, static images. Those are all put in lib directories.
• Dispatch rules. They map a request url to the correct controller.
• And a bunch of supporting Erlang code to make it all work.

A module contains one or more of the above items. All templates, actions, validations, css, javascript, dispatch rules and scomps are contained in modules. Modules have a priority and can overrule components of modules with lower priority.

Modules are directories with at least one Erlang module implementing a gen_server that manages the module. Subdirectories in the module contain all the templates, Webmachine resources, dispatch rules, actions etc.

Modules communicate with each other using notifications. Modules can be activated or deactivated using the admin interface (which is implemented using multiple modules).

Data Model

The data model is inspired by semantic web triple stores. But then adapted to make it work with a relational database and usual CMS interfaces and methods.

All (ok, most) data in Zotonic is a resource. Resources are connected with edges, creating a directed graph of information. Every edge has also a predicate. The predicate defines the meaning of the edge. For example author, as in the author of an article.

For example, a news article is a resource. The image of the news article is another resource, connected to the article with an edge, and so is the author of the article, the place the article is happening etc.

You could say that Zotonic has only two tables. A resource table and an edge table.

Resources are typed using a hierarchical category system. Some predefined resource categories are: text, news, image, video, sound, person, group, predicate, category and other.

A Data Resource

A resource has some fixed fields and can have any additional fields. The fixed fields are:

• Id, an unique integer.
• Title.
• Slug. Used to make understandable uris
• Introduction, a short abstract used in lists.
• Body, the textual body of a resource. A body contains HTML.
• Unique name, used by the system to find resources by name instead of id.
• Featured flag, to show things at important places on your site.
• Published flag.
• Publication period. From when till when a resource is published.
• Uri. The originating uri of the resource when it is imported from another site.
• Authoritative flag. Defines if the resource is imported or created locally.
• Category, defines what the resource represents. For example a person, article, event, image or video.
• Group. Editors work together in groups, all content belonging to a group can be edited by editors in that group.
• Visibility level. Defines if the resource is viewable by the group, community or the world.

All textual fields (except name and uri) can be multilingual. Any resource can have multiple extra data fields. Think of a date range, name or address.

In the next article I will show a bit of a template.

Zotonic enables hosting very popular sites on simple hardware.

I will write a couple of articles highlighting aspects of Zotonic. Today I focus on the reason we built a new CMS and on performance aspects. In other articles I will highlight the general design of Zotonic, the template system, modules, screen components, form handling, and adding logic (actions) to the templates.

Why?

Yet another CMS? Why? Because the current crop of open source content management systems don’t do what we want them to do. What do we want to do? We have a simple list of requirements:

• Flexible and easy to learn template system;
• Separation of logic, viewer and data store (MVC);
• Rich data model, easy to add knowledge to the data;
• Easy to make highly interactive web sites;
• High performance from a single machine for simple hosting;
• Future proof, easy to integrate with XMPP and other services;
• Easily extensible with modules;
• Able to handle push notifications with Comet;
• Easy and fun to program the back end.

The Open Source building blocks of Zotonic

Zotonic would not be possible without the contributions of many open source projects:

• Django like template engine (an adapted version of ErlyDTL);
• HTTP protocol handler and abstraction with Webmachine;
• Event driven user interface based on Nitrogen;
• The best scalable open source database around: PostgreSQL;
• A great CSS framework: Atatonic;
• jQuery for the user interface.
• epgsql For connecting to the database; and
• esmtp For sending e-mail.

Zotonic is written in the programming language Erlang. In the next section I explain why we use this programming language.

What is a web site anyway? Or why we choose Erlang

Once upon a time a website was a collection of manually edited HTML pages and some images. All was fine.

Then search was added to the site, necessitating extra programs running on the server on behalf of a web site visitor.

Then those pages started to be filled with content from a database, nothing changed much except that editing and searching got a lot easier.

And then we started to let the visitor react, interact and upload information to the server. Not only the visitor was allowed to interact, also machines got their own interface to talk to the server. Web 2.0 with APIs and user generated content took off.

Nowadays a web site is talking in many different ways to humans and machines. It is collecting information from multiple sources and actively submitting new articles to other sites or machines. The web server gives the visitor more and more ways to access the stored information. People are using RSS readers, widgets, desktop applications, other web sites and so on.

The web server evolved from a simple machine serving some pages to a complex switchboard connected to many different servers and services. The web server constantly sifts through information and storing it on behalf of its users. It constantly generates different representations of its own stored information, let is be as HTML pages, RDF documents, Atom entries, XMPP messages or API result sets.

This new web server is an always on machine. Not a simple server reacting to requests from a web browser.

This new web server is way more dynamic than yesterdays database based HTML generator. It needs to stay connected with other servers, it needs to have all those connections open and active at all times. It even needs to push information to the web browsers of visitors.

This new web server needs a different approach and programming model which becomes more and more like a information switchboard with thousands of open connections to other machines and web browsers.

Erlang was created to program telephone switches. Its programming model and features are a natural match to this new kind of web/information server. Besides its natural match to the problem domain it was also created to make very robust systems that are always on. No need to reboot an Erlang system for a program update.

That is why we choose Erlang. To create a robust web server that can be like a spider in the web of information. And one that is still easy to program and extend.

Performance: Scale up? Scale out? Scale in!

The current generation of computers are so fast that we can play multi user video games on a single machine, whilst surfing the web, checking our mail and what not. Why should we be happy with generating a couple of web pages per second from a machine that is so capable? How green is that?

We have quite a lot of experience building content management systems and web sites in general. What did we learn? For one thing: when you want high performance and flexibility and less machines, then you are in a lot of trouble.

Usage scenario

Our usage scenario is:

• A fairly popular web site experiencing periodic surges of visitors;
• The web site uses modern technologies like Ajax and Comet; and
• Multiple web sites on a single server.

So we are not targeting the new Flickr or similar sites. They need custom built software, not an off the shelf CMS. We are targeting the other 99 or so percent of the web sites.

PHP

PHP is the most popular language for building web sites.

For every request PHP wakes up and starts finding out where he is, where the code is, what the code is, where the database is, etc. etc. It does this again, and again, and again. Even when nothing is changed. This share nothing approach is nice when handling a single request, or when scaling out (adding more machines). It is not nice when you want to handle lots of requests from a single machine.

Ruby on Rails and Django

Other scaling problems arise with Python (Django) and Ruby (on Rails). Typically they are single threaded, that is they use only one thread of those 32 or more hardware threads that are available on modern hardware. That is solved by having more independent processes, each handling requests one by one. Those processes need to communicate with each other, adding more processes and complexity for caching systems etc.

Zotonic

Thanks to Erlang, Zotonic is completely multi threaded and scales linearly when adding more cores. All information about the web site is kept in memory and easily shared between requests. When two requests need the same html fragment then that fragment can be only rendered once. Note that this is not caching, which Zotonic handles as well, but a simple prevention for doing the same thing twice at the same time.

Zotonic typically serves 100 to 500 dynamic pages per second on normal of the shelf hardware. Which is enough for the fast majority of web sites.

Zotonic can serve a complete page before a Joomla/Drupal/WordPress site has even loaded its PHP code. With a Zotonic server you can serve either many more sites, or many more visitors per site.

Later more about the architecture of Zotonic and how to use Zotonic.

WordPress resizes uploaded images to two different sizes. A thumbnail and a medium format. In most designs you would really like to control how an image is used on the page. Sometimes it will be a small cropped thumbnail, another time a bit bigger, or a nice medium format on the post pages themselves.

The wok-imager plugin enables this without you resizing the images for WordPress.

Tested with PHP 5.2 and WordPress 2.7 and 2.6.3. Many thanks to Tim Benniks for testing it with WordPress 2.7.

Installation

Short version:

cd wp-content/plugins/
svn checkout http://wp-wok-imager.googlecode.com/svn/trunk/ wok-imager
Go to the admin page of your site and install the plugin.


How it works

Just add an image element in your page:


<img src="/wp-content/uploads/file.jpg" width="100" height="100" filter="crop,gray" />


The wok-imager will scan your generated page upon serving, apply the width, height and filters to the image and change the tag into something like:


<img src="/wp-content/uploads/wok-imager/file.jpg/4ee630a60ec7feb605516ac5b98972d1.jpg" width="100" height="100" />


When you don’t supply the crop filter then the width and height are the maximum bounding rectangle of the image. The resulting image has either the given width or the given height but is never bigger than the given width and height. You can also supply just the width or the height to make the resized image unbounded in respectively the height or the width.

Available filters

You can combine multiple filters by separating then with a comma. For example “crop,flip,gray”.

crop

Crops the image to the size given by the width and height attribute. Default is a center crop. You can have a crop aligning to the top, bottom, left or right by adding two extra letters (cropcc). The first letter defines how to align vertically and the second how to align horizontally. Valid values are t,c,b and l,c,r respectively.

Examples:

crop center crop
cropcc center crop
croptl upper left corner
cropbr lower right corner

scale

Allow the image to be upscaled. Per default wok-imager does not allow an image to be resized to bigger dimensions than the width and height of the original image.

flip

Horizontal flip of the image, swaps left and right.

flop

Vertical flop of the image, swaps top and bottom.

blur

Gaussian blur of the image.

smooth

Smoothen the resulting image. Default is a smooth factor of 6 is used. You can supply another with eg. smooth=4.

bright

Correct the brightness. Default is a brightness factor of 50. You can supply another with eg. bright=100.

contrast

Correct the contrast. Default is a brightness factor of -10. You can supply another with eg. contrast=-50.

gray

Convert the image to grayscale.

sepia

Convert the image to sepia.

colorize

You have to supply the three rgb colors with this filter. For example: filter=”colorize=100:100:100″

negate

Negate all colors, makes the image look like a color negative,

q=quality

Sets the quality of the jpeg conversion. When you don’t supply the quality then the quality depends on the size of the resulting image. The smaller the image the higher the quality setting.

Imager plugins

You can add your own imager plugins. There are two examples in the imager-plugin directory.

deleted

Adds a translucent red cross on top of your image.

example

Adds a small text on top of your image.

OAuth-PHP

OAuth-PHP is a PHP library implementing both the consumer and server side of OAuth. You can easily integrate it in your web software to either accept or make OAuth authenticated requests.

Verso Wiki

A wiki library that translates Wiki to XHTML and XHTML back to Wiki. It is written in PHP and is useful when you want to have a fast way of publishing texts with Wiki markup.

Depcached

A variant of memcached, with added dependency keys, queues and garbage collector.

wok-imager

Wok-imager is a WordPress plugin for automatic resizing, cropping and editing of images.

I maintain an implementation of OAuth. Whilst checking existing code and implementing my own version I wondered how some implementations will differ.

OAuth uses RFC3986 for the encoding of URI parameters. Another often used encoding scheme is RFC1738 or RFC2396 (updated in RFC2732).

The differences? Quite a bit.

The unreserved characters in RFC3986 are:

unreserved = ALPHA, DIGIT, '-', '.', '_', '~'

And in RFC17138:

unreserved     = alpha | digit | safe | extra
safe           = "$" | "-" | "_" | "." | "+"
extra          = "!" | "*" | "'" | "(" | ")" | ","

And in RFC2396/2732

"A" .. "Z", "a" .. "z", "0" .. "9",
"-", "_", ".", "!", "~", "*", "'", "(", ")"

What do some programming languages use?

PHP

PHP has two options. Use urlencode or rawurlencode. The difference between the two is the escaping of the “+” character.

rawurlencode claim to fame is to be compatible with RFC1738. In fact it is not. It encodes all characters as in RFC3986, and then also the “~”. So rawurlencode comes close. It escapes everything, except:

A-Za-z0-9\-_.

So the only difference here is the “~” character. The correct funtion will be:

function oauth_urlencode ( $s )
{
    return str_replace('%7E', '~', rawurlencode($s);
}

Perl

CPAN implements URI:Escape. This one uses RFC2396/2732 as the basis for its character encoding. The default for this function is to escape everything except:

A-Za-z0-9\-_.!~*'()

Javascript

Yet another encoding method. Most javascript programmers will use the function escape. This one encoded everything except:

A-Za-z0-9\-_.+*/@

Addition from John Kristian: when you want to encode with Javascript you can take a look atOAuth.percentEncode in oauth.googlecode.com/svn/code/javascript/oauth.js

It is a mess, what to do?

I assume that programmers are lazy and will re-use their good old trusty encoding functions. They should, it is how we train programmers. This will give some interesting problems with OAuth though, as OAuth insists on encoding and sometimes double encoding parameters. And uses those encoded values to calculate the signature of a message.

This insistance on using a very specific implementation of encoding is, in my opinion, the Achilles heel of OAuth.

When checking the signature we have basically two options:

1. Not decode whatever comes in before recalculating the signature. In that way we won’t make any false assumptions about parameters being encoded one way or the other.
2. Recode all incoming parameter values and names, in that way we make sure that we use the correct encoding (according to the spec) for our signature calculation.

However, we must assume that the encoding used in calculating the signature base string and the key from the consumer secret and the token secret is correct according to RFC3986.

Conclusion

We, as a community using OAuth, need to create test sets that will test all edge cases, otherwise we are in for a rough ride. And, to help other programmers, there is an immediate need for correct implementations of the RFC3986 parameter encoding.

Update

The OAuth community has a nice set of testcases online at wiki.oauth.net/TestCases Let’s all use them and make sure that our implementations are correct!

I was the principal architect and software engineer of anyMeta. In four major revisions it grew from a small library to full featured content management system with over 90 different modules. It is also the first CMS that actively shares information between different web sites using push updates and cross site editorial protocols.

The system allows visitors to actively participate on the website, while the resulting information automatically appears in the right place, within a relevant context. This makes anyMeta extremely suitable for social networks, knowledge management, collaborations, and story telling.

Semantic

As a semantic network anyMeta uses meta data, which means the system is ahead of the current internet developments. This meta data is used to let information automatically appear in a consistent and clear structure, within a relevant context. This will save editors a lot of time. It also helps and stimulates visitors to navigate the website, because the associations between different kinds of information are directly shown.

Social

There are many options for visitors to actively participate on anyMeta websites. They can register as a user and create a personal profile. Administrators can grant users several rights to certain parts of the website. This allows users to add all kinds of information to the web site themselves.

Flexible

The system allow anyone with an internet connection to easily update and maintain the website. Despite the automatic structuring, editors are not totally dependent on the system and its users. Information can also be ordered in a custom way. Besides that, editors can actively manipulate the use of meta data, by offering a set of predetermined keywords. This makes anyMeta very flexible, because the automatic structuring can be supported with an editorial signature.

Open-CI

Especially interesting about anyMeta is the option to exchange information with other web sites. With the use of open standards, the system is able to share information with other systems, without losing its own identity. So the system is not only capable of creating relevant connections within one network, but also between multiple related web sites.

Sustainable

Web sites based on anyMeta can handle large numbers of visitors at once. Pages and information load fast, and editors can publish new information directly on the website. The powerful search functionality and automatic structuring help managing large amounts of information. Because of its semantic structure, it is easy to add new meta data to the system when changes to the website are necessary.

Janssen, G., G. Nieuwdorp, C. Worrell, M. Worrell, Proefproject “Handhavingen Industriele Veiligheid in het Havengebied” - PolCop, 80 pp (No. 95005).

The input specification language was designed with flexibility and ease-of-use in mind, requiring no special skills or training. It supports any architecture, from 4 bit to 64 bit, and is widely programmable to support any instruction-set, syntax and opcode format (including little/big endian, or mixed encoding).

The NOVAgen technology does support a number of powerful optimizations that are extremely important for microcontrollers and DSPs:

• Instruction scheduling
  where the order of instructions can be changed in order to minimize pipeline delays.
• Instruction Parallelization
  where instruction sequences are replaced by their parallelized counterparts.
• Jump optimization
  where the shortest possible coding scheme is selected based on the address range.
• Peephole optimization
  where instruction sequences are identified that are to be replaced by shorter ones.

The language and the automatic generation process have been very successfully used in combination with instruction-set databases. In this case, the instruction-set is derived either from a hardware design model, or from a specification document, and can directly be used to generate input description files for the NOVAgen generation suite.

The first version was build using Haskell and C.