This report provides a comparative evaluation of two versions of a popular content management system (CMS) developed by eZ Systems: eZ publish v 2.2.8 (referred to as eZ publish 2 or version 2) and eZ publish v.3.0-2 (referred to as eZ publish 3 or version 3).

This article was also published on Sitepoint.com.

1. Introduction
This report provides a comparative evaluation of two versions of a popular content management system (CMS) developed by eZ Systems: eZ publish v 2.2.8 (referred to as eZ publish 2 or version 2) and eZ publish v.3.0-2 (referred to as eZ publish 3 or version 3). eZ publish 2 is one of the most popular open content management systems on the web today. Its successor, eZ publish 3, is a content management system and development framework that fundamentally differs from version 2.

At present there is no utility for updating automatically from version 2 to version 3, although eZ Systems has planned to release an upgrade script. Until then the decision to adopt either version 2 or 3 will have far-reaching implications for the future development of a website. eZ publish 2 is a stable, well-tested system, but further development of version 2 might be discontinued in the future. At the moment eZ Systems still supports version 2, but with limited resources only, doing bug releases once in a while. The development of eZ publish version 3 is in full swing, with all the associated advantages and disadvantages for early adopters. Version 3 is high-maintenance because it contains more bugs and offers less 'out-of-the-box' functionality than the previous version, but it offers more opportunities for the future development of a site.

This report provides a general framework that will allow the reader to make an informed choice between the two versions of the eZ publish CMS. The report does not attempt to establish which of the two versions should be considered the preferred option. The suitability of either version will depend on the functional requirements, plans for future site development and time/project constraints.

2. eZ publish
eZ publish, a CMS released under an open source licence, has recently become very popular. The system includes all the features of most commercially licensed systems and offers many optional extras. The eZ publish CMS has been developed by eZ systems, a small company located in Skien, Norway.

eZ publish 2.2.x is a professional open source CMS that gives easy access to Internet sites through a user-friendly and reliable program. It is platform and database independent and module based, with a total of 26 different modules available. Version 2 was first released in November 2000; the most recent version of the 2 series is 2.2.8.

eZ publish 3 is a content management system and development framework. The CMS is based on a fully customizable and extendable content model. Its stand-alone libraries can be used as a development framework for cross-platform, database-independent PHP projects. Version 3 was released on 25 March 2003. The first update of version 3 has already been released (3.0-2), fixing some bugs in the kernel.

As the numbering of versions suggests, there are some fundamental differences between eZ publish versions 2 and 3. eZ publish 3 broadens the capabilities of version 2 towards a general application framework, providing basic data handling and display mechanisms and expanding the possibilities for creating customized content within the system indefinitely. Whereas version 2 works with a 'limited' set of 26 modules, version 3 allows users to identify their own content classes and attributes. Some examples of content classes are article, forum, product, publication, organization or user account. A simple 'article' content class may consist of attributes such as author, title, intro and body. New classes and attributes can be simply added through the administrative back-end - no additional coding is required.

3. Evaluation criteria
The two versions of eZ publish were evaluated on the basis of the following criteria:

• functionality;
• flexibility;
• performance;
• stability.

Usability was not included as a criterion because both the front- and back-end of the eZ publish application can be fully customized to meet the needs of users and content managers.

3.1 Functionality
The bigger the gap between functional requirements and actual functionality available in the default install of eZ publish, the more resources (staff, money and time) will have to be spent on site development. In the best case scenario, customization is limited to the enhancement of the design or existing functionality. In the worst case, advanced customization requires the development of new functionality from scratch.

Advanced but poorly documented customization may turn the eZ publish system into a legacy solution that no one can extend, extract data from and fix bugs on. Once the system is no longer truly ‘open’, the organization may be tied into an exclusive long-term contract with the company that customized the system. Moreover, advanced customization may seriously hamper the install of future updates, with the result that the organization could miss out on important security updates.

eZ publish 2 is a system with extensive, built-in functionality that meets the needs of most small to medium-sized organizations and campaigns. The customization of existing modules is usually limited to adapting and improving eZ Article (multiple topics, alternative links, additional frontpage rows), eZ Filemanager (sorting method), eZ Search (searching additional modules) and eZ Link (sorting method). In many cases, customization also involves adding extra fields for information to a module. For example, in the case of eZ Filemanager this could be the field ‘author’ for a file. More advanced customization is often required for multilingual support, eZ Calendar, eZ Contact, or in cases where eZ Filemanager has to be converted into an ‘e-library’.

At first glance, eZ publish 3 appears to offer much less functionality than version 2. However, as mentioned above, version 3 allows users to identify their own content classes and attributes through the administrative back-end. Much of the functionality of the version 2 modules is available in some form in version 3 (see also How to implement ez publish 22 functionality in 30 on the eZ publish website). A number of standard functional enhancements required for version 2 are available in version 3 by default, and additional information fields (attributes) can be added to objects without coding. This means that advanced customization of version 3 is possible while keeping the source code comprehensible or ‘open’, thus avoiding dependency upon a single service provider. Version 3 has a compact design and the kernel does not need to be altered for most customizations. As a result future upgrades are possible even if the system is highly customized. Moreover, eZ Systems is working on more example implementations in version 3, and making it easier for others to contribute their work.

Conclusion. eZ publish 2 offers more in terms of ‘out-of-the-box’ functionality, allowing organizations with limited resources and time constraints to ‘jump start’ their Internet projects while keeping costs under control. Version 2 is flexible enough to allow functional extensions as and when required, but advanced development of this system may lead to dependency on an external service provider for technical maintenance. eZ publish 3 is ideally suited for projects that require functionality beyond that provided in the ‘standard’ modules. It allows for extensive customization while avoiding external dependency. However, due to its flexibility, the lack of accurate and comprehensive functional specifications may cause the development process to spiral out of control. This ‘grown-up’ system needs a ‘grown-up’ development approach, which might not suit time- and resource pressed organizations.

3.2 Flexibility
The flexibility of eZ publish versions 2 and 3 was compared in terms of the creation of custom content classes and page customization and layout. With regard to custom content classes, as mentioned above, eZ publish 3 allows users to identify their own content classes and attributes through the administrative back-end. In version 2 this required making manual changes to the database, coding new classes and functions and creating the matching front- and back-end templates.

In eZ publish, page customization and layout are separate from content, code multilingual functionality. Every page is generated dynamically by means of a template. A template is one or more files with plain HTML which contain template variables. The variables are filled in by eZ publish when a page is generated. In eZ publish 2 the template sets are spread out between all modules in eZ publish. Each module has at least a subdirectory called ‘admin’ and usually a directory called ‘user’. In eZ publish 3 all standard templates are stored in a single folder. The basic principle for creating custom designs in version 3 is to create override templates in the custom design folders that replace the standard templates. eZ publish 3 also introduces template operators and functions that allow advanced template customization.

Conclusion. The process of creating new content classes and attributes is much easier in version 3 than in version 2. Version 3 has the advantage of allowing users to identify their own content classes and attributes without coding or manual database updates. The only task that remains is to develop the necessary front-end templates for the new content class. In version 3 the template system has become much more advanced and flexible, but it is also more cumbersome to implement. In comparison to version 2, the version 3 user needs to have a greater knowledge of the eZ publish application in order to be able to set up or change design templates. This could lead to a situation where an organization could become dependent on an external service provider for even minor changes to the look-and-feel of the site.

3.3 Performance
Scalability and performance are key factors in the evaluation of any web platform. In this evaluation, eZ httpbench was used to get an indication of the differences in the performance of eZ publish versions 2 and 3. The experiences of users as submitted to the forums on the eZ publish website were also reviewed.

Using eZ httpbench, a benchmark tool for HTTP pages developed by eZ Systems, the average download times of pages with a size of approximately 12.0 KB were compared for the two versions. The following pages were fetched 25 times by the eZ httpbench benchmark tool: http://www.contactivity.com/ezpublish/index.php/filemanager/list/ (14.47 KB, eZ publish 2) and http://www.contactivity.com/ezpublish3/index.php/demo/content/view/full/62/ (11.78 KB, eZ publish 3). The results are shown in table 1 below. Unfortunately, eZ httpbench is no longer available as a download from the eZ Systems site itself, but it can still be downloaded from 3sign.de.

Note that these statistics are not typical for all installs of eZ publish. Throughput may vary according to server specifications, the overall server load, the configuration of the eZ publish application, and the use of accelerators. The results of a more extensive performance test of eZ publish version 3 are available on the eZ publish website.

Table 1. Comparison of eZ publish versions 2 and 3: average download times of pages with a size of approximately 12.0 KB

On the developer forums on the eZ publish website, user responses with regard to the performance of versions 2 and 3 are mixed. Some users find that version 2 performs better, while others prefer version 3. The use of eZ publish caching and PHP caching applications has recently received a lot of attention, and useful suggestions have been made on how to use caching to improve performance.

According to one forum entry of an eZ publish developer ‘[…] we've just started with optimizations and you should see tremendous speed improvements from the 2.x series’. Another experienced user advises to use ‘2 GHz dual cpu servers + php accelerator for a heavy load eZ publish 3 server’. Users report that both versions contain several poorly written sql statements that slow down some parts of the application.

eZ Systems claims that eZ publish 3 is much more scalable than version 2. However, because of the flexibility of version 3, it is also easier to misconfigure the system and thus get poorer performance. eZ Systems has made this issue a focus point for the future development of version 3. We have not attempted to verify the claim of scalability, but for more information on this topic, see ez 30 scalability on the eZ publish website.

Conclusion. From the eZ httpbench tests it appears that eZ publish 3 is slightly slower than version 2. However, the performance of both versions 2 and 3 on a standard server is still more than adequate. The performance of both versions may be further improved by tuning the site with eZ publish caching enabled or by using PHP caching applications, such as the ionCube PHP Accelerator, Zend, Alternativ PHP Cache or Turck MMCache.

3.4 Stability
For the purposes of this evaluation, ‘stability’ is defined as the number of bugs reported over a fixed period of time. This basically means: the smaller the number of bugs, the more stable the system should be considered. Obviously this working definition can be said to measure other things besides ‘stability’, such as the commitment or involvement of the user community to one particular version, or the degree to which the available functionality matches the users’ expectations or requirements. In any case, it is safe to say that a system with a large number of reported bugs will require more maintenance than one with fewer reported bugs. If an organization does not have the resources to monitor developments and perform all required updates, an advanced system may soon develop into a slow, unreliable, error-ridden and potentially insecure website.

An overview of the eZ publish bugs can be found at eZ publish 2.2 / bug reports and eZ publish 3 / bug reports . Table 2 below lists the numbers of bugs reported during the period 26 March to 26 May 2003, categorized as follows: database, user interface, generic logic problems, permissions, hangs and crashes, and other. Bugs that presented security problems are also listed.

Table 2. Comparison of eZ publish versions 2 and 3: numbers of bugs in each category reported during the period 26 March to 26 May 2003

Conclusion. eZ publish 2 is an application that requires relatively little maintenance. The system is well-tested, stable and apparently secure. An organization opting for an eZ publish 2 application with fairly limited customization would be relatively independent from external developers. Although eZ publish 3 has an advantage over version 2 in terms of expanding the possibilities for creating customized content within the system, it is still a high-maintenance application. Even though the core functionality of the version 3 is much more stable than version 2, the amount of bugs in version 3 is still considerable. Vigilance on the part of the internal IT team or the external developers is constantly required, at least to ensure that all potential (security) problems are resolved. However, a systematic approach with regard to the handling of bugs might be beyond the scope of organizations with limited resources.

4. Conclusions
In general we would advise any organization to be realistic and use common sense when developing an Internet strategy, taking into account the current requirements and future plans for a website, what resources (budget, staff, developers, support) are available, and possible time constraints. For example, do the functional requirements demand the use of the latest, cutting-edge, web-based technology, or does the organization simply need something to ‘jump start’ an Internet project while keeping other options open?

eZ publish 2 is an ideal solution for an organization with limited resources and under time constraints that requires a low-maintenance, user-friendly application with sufficient ‘out-of-the-box’ functionality. The server requirements for running version 2 are relatively modest, and enable the organization to opt for a cheap hosting solution. eZ publish 2 has a comprehensive set of functional modules that require little modification to meet the demands of most small to medium-sized organizations. The back-end is user-friendly and easy to understand, even for novices with no web publishing experience. Changing the look-end-feel of the front-end is relatively simple, and does not require extensive knowledge of the eZ publish application. Due to its flexible, modular setup, version 2 allows advanced customization to meet the needs of the organization, but this might come at the cost of dependency on an external service provider and problems with the install of (security) updates or modifications by third parties.

eZ publish 3 is ideally suited for organizations whose functional requirements that go beyond what is provided in the ‘standard’ modules of version 2, or that have ambitious plans for future site development. eZ publish 3 is more flexible than version 2 and allows for extensive customization while avoiding dependency on a single external service provider. On the downside, the development of a website based on eZ publish 3 will require more from an organization (or external service provider) in terms of involvement in the development cycle, project planning skills, application maintenance and training of web designers/developers and content managers. Because version 3 is more flexible, it has a steeper learning curve than in version 2. This disadvantage is offset by the quality of the available manuals and community contributed documentation. The performance level of eZ Publish 3 is acceptable and is likely to improve in the near future, but the use of PHP caching applications could put some restrictions on the choice of hosting accounts.