File/program/lib/updatelib.php

Description

/program/lib/updatelib.php - update wizard

This file handles all system updates. The basic idea is as follows.

We assume that a previous version of Website@School is/was already correctly installed using the code in /program/install.php. Using this version the school has added many, many hours of work in entering data.

Now a new version is installed, i.e. the new files (or just the updated files) are copied/uploaded to the webserver, including the file version.php and perhaps also updated versions of modules, themes, etc. This yields an error message for visitors (the infamous 'error 050') because the database version and the file version no longer match. The user logging in into admin.php is forced to attend to the job 'update', arriving here in job_update().

There an overview is presented of the core version and versions of all subsystems, with the option to upgrade those that qualify. The actual work for the core version is done in update_core() in this file. The actual work for modules and themes are done via the code in those subsystems. However, these are called from here.

The user is more or less _forced_ to perform the upgrade: all ways lead to job_upgrade() while the internal (database) version does not match the file version.

The upgrade should perform all necessary steps to upgrade, ending with updating the internal version number to match the file version. After that the error message '050' for visitors is gone, and admin.php no longer forces the user to come here. It is possible, however, to manually arrive here to check the updates of modules, themes, etc. but I consider that less important. That is: an upgrade of a module or theme will NOT be forced upon the user. It is wise, though to upgrade, but that is up to the user.

Constants
TASK_INSTALL_LANGUAGE = install_language (line 66)
TASK_INSTALL_MODULE = install_module (line 69)
TASK_INSTALL_THEME = install_theme (line 72)
TASK_UPDATE_CORE = core (line 64)
TASK_UPDATE_LANGUAGE = update_language (line 67)
TASK_UPDATE_MODULE = update_module (line 70)
TASK_UPDATE_OVERVIEW = overview (line 63)
TASK_UPDATE_THEME = update_theme (line 73)
Functions
get_manifests (line 1048)

retrieve an array of manifests for modules, themes or languages

this examines the file system starting in the directory $path, looking for manifest files. These manifest files are named after the subdirectory they are in as follows. Example: If $path is /program/modules, this routine steps through that directory and may find subdirectories 'htmlpage', 'guestbook' and 'forum'. Eventually these manfest files are include()'d: /program/modules/htmlpage/htmlpage_manifest.php, /program/modules/guestbook/guestbook_manifest.php and /program/modules/forum/forum_manifest.php.

Every manifest file must describe the module (or language or theme) via the following construct:

    $manifests['htmlpage'] = array('name' => 'htmlpage', ...., 'cron_interval' => 0);

After processing all the subdirectories of $path, the resulting array $manifests is returned. Note that pseudo-directories like '.' and '..' are not considered. Also, subdirectories 'foo' without the file 'foo_manifest.php' are also ignored.

Note that the name of the manifest file itself is also stored in the array, but excluding the subdirectory name.

Note: a similar routine is used in the installation script install.php.

  • return: zero or more arrays comprising manifests
array get_manifests (string $path)
  • string $path: top directory for the search for manifest files
install_language (line 343)

install an additional language pack

this routine attempts to insert the information from the manifest of language $language_key into the database. The routine displays the result (error or success) in a message in $output. Details can be found in the logs.

The language_key is validated by reading all existing manifests. This is quite expensive, but that is not important because we do not use this routine very often anyway.

Note that we assume that the actual translations of the language pack are already unpacked into the correct directories. The corresponding manifest should exist in the directory /program/languages/$language_key.

  • return: results are returned as output in $output
void install_language (object &$output,  $language_key, string $lanuage_key)
  • object &$output: collects the html output
  • string $lanuage_key: primary key for language record in database AND name of the /program/languages subdirectory
  • $language_key
install_module (line 475)

install an additional module

this routine attempts to insert the information from the manifest of module $module_key into the database. The routine displays the result (error or success) in a message in $output. Details can be found in the logs.

The module_key is validated by reading all existing module manifests. This is quite expensive, but that is not important because we do not use this routine very often anyway.

Note that we assume that the actual modules are already unpacked into the correct directories. The corresponding manifest should exist in the directory /program/modules/$module_key.

  • return: results are returned as output in $output
  • todo: we should refactor and combine install_theme() and install_module()
  • uses: $CFG
void install_module (object &$output, string $module_key)
  • object &$output: collects the html output
  • string $module_key: primary key for module record in database AND name of the /program/modules subdirectory
install_theme (line 696)

install an additional theme

this routine attempts to insert the information from the manifest of theme $theme_key into the database. The routine displays the result (error or success) in a message in $output. Details can be found in the logs.

The theme_key is validated by reading all existing module manifests. This is quite expensive, but that is not important because we do not use this routine very often anyway.

Note that we assume that the actual thenes are already unpacked into the correct directories. The corresponding manifest should exist in the directory /program/themes/$theme_key.

  • return: results are returned as output in $output
  • todo: we should refactor and combine install_theme() and install_module()
  • uses: $CFG
void install_theme (object &$output, string $theme_key)
  • object &$output: collects the html output
  • string $theme_key: primary key for theme record in database AND name of the /program/themes subdirectory
job_update (line 103)

main entry point for update wizard (called from /program/main_admin.php)

This routine takes care of executing update routines for both the core program and modules, themes, etc. It is called automagically whenever the core program version in the database is different from the version in the file {@lnk version.php} (see also main_admin()).

It can also be called manually via 'job=update'. When no specific task is specified, this routine shows the overview of versions for core, modules, themes, etc. Whenever a component is NOT up to date, an [Update] button is displayed. If a component IS up to date, we simply display the word 'OK'. This implies that when everything is up to date, the overview simply displays a list of OK's and the user is 'free to go'.

The actual updates for modules, themes, etc. is done via the various subsystems themselves, e.g. by calling htmlpage_upgrade() in the file /program/modules/htmlpage/htmlpage_install.php. The updates for the core program are actually performed from this file right here, see below for an example.

Note that we give a core update high priority: if the core is not up to date, nothing will work, except updating the core.

  • return: results are returned as output in $output
void job_update (object &$output)
  • object &$output: collects the html output
update_core (line 891)

update the core version in the database to the version in the version.php file (the 'manifest' version)

  • return: results are returned as output in $output
void update_core (object &$output)
  • object &$output: collects the html output
update_core_2010120800 (line 1119)

perform actual update to version 2010120800

  • return: TRUE on success, FALSE otherwise
bool update_core_2010120800 (object &$output)
  • object &$output: collects the html output
update_core_2010122100 (line 1137)

perform actual update to version 2010122100

  • return: TRUE on success, FALSE otherwise
bool update_core_2010122100 (object &$output)
  • object &$output: collects the html output
update_core_2011020100 (line 1155)

perform actual update to version 2011020100

  • return: TRUE on success, FALSE otherwise
bool update_core_2011020100 (object &$output)
  • object &$output: collects the html output
update_core_2011051100 (line 1218)

perform actual update to version 2011051100

this is a substantial change in the database: we (finally) standardise on UTF-8 including the database. Up until now we still only have a choice of exactly one database driver: MySQL. Therefore the upgrade we do here can be more or less MySQL-specific. (So much for database-independency).

What needs to be done here?

The most important task (in fact: the only task) is to change the collation (and implicitly the default charset) to utf8_unicode_ci (4.1.x <= MySQL < 5.5.2) or utf8mb4_unicode_ci (MySQL 5.5.3+). See mysql.class.php for more information on these UTF-8 & MySQL issues.

Strategy here is as follows.

    for all 'our' tables (ie. "LIKE '{$prefix}%'" do    if table default charset is already utf8 (or utf8mb4)        continue;    for appropriate columns in this table        change column type to binary        change column type back to non-binary with correct charset and collation    if no trouble sofar        change default charset/collation of the table too    else        return failure return success

This way we _might_ be able to work our way through huge tables: if the PHP max processing time kicks in, we can rerun the upgrade and start (again) with the table we had in our hands the previous time. I don't expect this to happen, but it still the way to do it IMHO.

Note that I assume that I cannot change the default charset of the DATABASE for the same reason the Installation Wizard expects the database to be ready before installation commences. (I cannot be sure that I have the privilege to execute 'ALTER DATABASE $db_name DEFAULT CHARSET utf8 COLLATE utf8_unicode_ci').

A useful reference for solving this problem of converting to utf8 can be found here: http://codex.wordpress.org/Converting_Database_Character_Sets.

In the case of W@S we do not have to deal with enum-fields because those are not used at this time. In fact it boils down to changing char, varchar, text and longtext.

Let goforit...

  • return: TRUE on success, FALSE otherwise
bool update_core_2011051100 (object &$output)
  • object &$output: collects the html output
update_core_2011093000 (line 1547)

perform actual update to version 2011093000

this is yet another substantial change in the database: after we (finally) standardised on UTF-8 the last time (see update_core_2011051100() a number of problems occurred with new installations.

This specifically occurs with MySQL (currently the only supported database). In all their wisdom Oracle decided to change the default database engine from MyISAM to InnoDB in MySQL version 5.5.5. Bad move to do that somewhere in a sub-sub-release. Anyway. New installations with the default InnoDB engine AND with the 4-byte utf8mb4 character set (available since sub-sub-release 5.5.3) now generate serious trouble, because

  • there is a hard-coded limit of 767 bytes for a key (index) in InnoDB, and
  • every utf8mb4 character counts as four bytes never mind the actual content.
Note: the limit of 767 bytes stems from a utf8 (or utf8mb3 as it is now called) string of max. 255 characters and 1 16-bit string length. 255 * 3 + 2 = 767 bytes. I wonder why UTF-8 wasn't implemented correctly (ie. with 1 to 4 bytes) to begin with and the key limit increased to 4 * 255 + 2 = 1022 bytes. The limited UTF-8 support (only the BMP) now poses substantial problems. Yet another reason to start looking for an alternative database solution. BTW: the key limit in MyISAM is 1000 bytes.

These two conditions (InnoDB and utf8mb4) limit the length of a key (index) to 767 bytes / 4 bytes-per-char = 191 utf8mb4 characters. As it happens, some tables in WebsiteAtSchool used keyfields like varchar(240) and even varchar(255). These key sizes fail in InnoDB/utf8mb4 and the latter even fails with MyISAM/utf8mb4 because 255 * 4 + 2 = 1022 bytes > 1000 bytes. What a mess...

So there you have it: all keys MUST be shortened to 191 characters max. in order to prevent stupid error messages about key too long. The alternative (forcing another character set such as 'ascii' or 'latin1' for some fields) doesn't cut it IMHO.

*sigh*

We still have a choice of exactly one database driver: MySQL. Therefore the upgrade we do here can be more or less MySQL-specific (so much for database-independency), as it has to be, because the syntax of ALTER TABLE is -- unsuprisingly -- MySQL-specific.

The good news is that we are still in beta, so a major change in the data definition is less painful than with hundreds of production servers...

Another issue is the use of foreign keys. We used to have a FK in the nodes tabledef along the lines of this construct: FOREIGN KEY parentnode (parent_id) REFERENCES nodes (node_id); Upto now this could not possibly have worked with InnoDB because adding a node would at the top level of an area would not satisfy this constraint. Since MyISAM silently ignores any foreign key definition it 'simply works' in that case. So, because this FK must be removed from earlier installations we need to DROP the FOREIGN KEY. However, since the whole program never installed using InnoDB, there is no need to drop this foreign key that wasn't even recorded (in a MyISAM database) in the first place. The same applies to a number of other FK's too: these are now removed from the various tabledefs but do no need to be DROPped in this update routine.

What needs to be done here?

For existing tables some fields must be shortened from varchar(255) or varchar(240) to something like varchar(191) or even less. This MUST be done for key (index) fields. However, while we are at it some more fields SHOULD (or COULD) be shortened too. Here is what we do.

    for all affected table.fields do    if a record exists with current data length > proposed new length then        tell the user about it    endif next if there were data length errors then     tell the user about manually fixing it     bail out with result FALSE (= not upgraded) endif for all affected table.fields do     change field definition to new length     if errors         tell the user about it (but carry on)     endif next return results (TRUE on success, FALSE on 1 or more errors)

Below is a discussion of all affected fields and the rationale for picking the new lengths less than 191 characters.

    config.name: varchar(240) => varchar(80) modules_properties.name: varchar(240) => varchar(80) themes_properties.name: varchar(240) => varchar(80) themes_areas_properties.name: varchar(240) => varchar(80) users_properties.name: varchar(240) => varchar(80) users_properties.section: varchar(240) => varchar(80)

Currently the longest parameter name in use is 27 characters, so I have to admit that the arbitrary size of 240 is a little bit too much. I'll reduce these fields to a size of 80, which seems a little more realistic. As an additional bonus, this allows for a compound key using 'section' and 'name' in users_properties while staying within the limit of 767 bytes or 191 characters.

    areas.path: varchar(240) => varchar(60) groups.path: varchar(240) => varchar(60) users.path: varchar(240) => varchar(60)

and

    groups.groupname: varchar(255) => varchar(60) users.username: varchar(255) => varchar(60)

The length of username or groupname was arbitrary set to 255. Different systems have different limits, e.g. 8, 14, 15, 16, 20, 32, 64 or 128. Since W@S is a stand-alone system we are more or less free to choose whatever we want (as long as it is less than 191 of course).

Since a username or groupname is only used to distinguish one user from another but at the same time giving at least some readability, a length of 255 is way too long. An arbitrary but hopefully more realistic choice is 60 characters.

The path for a user or group is derived from the corresponding name so it makes sense to make both fields the same length.

    log_messages.remote_addr: varchar(255) => varchar(150) login_failures.remote_addr: varchar(255) => varchar(150)

A remote address of type IPv4 generally looks like this: 'ddd.ddd.ddd.ddd' => length 15 It is not so easy to determine the length of an IPv6 address, because many valid variants exist. 'xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx' => length 39 '0000:0000:0000:0000:0000:0000:ddd.ddd.ddd.ddd' => length 45 '[0000:0000:0000:0000:0000:0000:ddd.ddd.ddd.ddd'] => length 47 (RFC3989)

Adding to the complexity and confusion are link-local addresses with zone indices: a percent-sign followed by an interface number (e.g. '%1') or interface name (e.g. '%eth0') appended to the raw address. This adds 2 or 5 or even more characters to the address. And then we of course have the reverse DNS-variant like 'x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.ip6.arpa.' => length 73 or the special Microsoft trick to shoehorn a literal address in a UNC path: 'xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx.ipv6-literal.net' => length 56 or 'xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxs1.ipv6-literal.net' => length 58+ (with zone index)

Of course there several 'simplifications' such as omitting leading zeros in the hexquads and replacing the longest sequece of 0-hexquads with '::' that add to the confusion. RFC5952 adds the definition of a 'canonical representation' of IPv6 addresses to the party. Mmmm, see http://xkcd.com/927

My conclusion is: this whole IPv6-idea suffers from the Second System Syndrome (see F. Brooks' Mythical Man Month) and unfortunately we have to deal with it.

*sigh*

I will reduce the length of these fields from 255 to 150 for no other reason than that it is 10 times the length of a dotted-decimal IPv4 address and sufficient to accomodate a reverse DNS address twice (2 x 73 = 146).

    sessions.session_key: varchar(255) => varchar(172)

This field stores a session key, currently constructed using md5() which yields a string with 32 (lowercase) hexadecimal characters. In the future a different digest could be used to provice a session_key, e.g. SHA-1 (40 hexdigits) or SHA-512 (128 hexdigits). Another option would be to use a UUID: 128 bits represented in 32 hexdigits in the form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (string of 36 bytes). Alternatively, the SHA-512 could be encoded in base64 yielding a string of 512 / 6 = 86 bytes. In this context, a field of size 255 seems a little over the top, not to mention problematic with 4-byte UTF-8 characters combined with the infamous MySQL / InnoDB-limit of 767 bytes for keyfields. I guess I will settle for a field size of 172 characters which is not too much for InnoDB keys + utf8mb4 and exactly enough to store a 1024 bit number in base64.

  • return: TRUE on success, FALSE otherwise
bool update_core_2011093000 (object &$output)
  • object &$output: collects the html output
update_core_2012041900 (line 1874)

perform actual update to version 2012041900

Changes between 2011093000 and 2012041900:

  • addition of the ckeditor-option in the site configuration table

  • return: TRUE on success, FALSE otherwise
bool update_core_2012041900 (object &$output)
  • object &$output: collects the html output
update_core_2013071100 (line 1979)

perform actual update to version 2013071100

Changes between 2012041900 and 2013071100:

  • addition of a new core table 'tokens'

  • return: TRUE on success, FALSE otherwise
bool update_core_2013071100 (object &$output)
  • object &$output: collects the html output
update_core_2014111700 (line 2017)

perform actual update to version 2014111700

Changes between 2013071100 and 2014111700:

  • addition of extra field in modules table
  • addition of new site config parameter 'resize_dimension'
  • addition of editor option 'ckeditor3' in site configuration

  • return: TRUE on success, FALSE otherwise
bool update_core_2014111700 (object &$output)
  • object &$output: collects the html output
update_core_2016062900 (line 2167)

perform actual update to version 2016062900

Changes between 2013071100 and 2016062900:

  • addition of new site config parameter 'cron_interval'
  • addition of new site config parameter 'cron_next'
  • addition of new site config parameter 'favicon'
  • removal of foreign key constraint in 'nodes' table

  • return: TRUE on success, FALSE otherwise
bool update_core_2016062900 (object &$output)
  • object &$output: collects the html output
update_core_version (line 955)

record the specified version number in the config table AND in $CFG->version

This utility routine records the new version number in the config table and also adjusts the version number already in core (in $CFG->version).

  • return: TRUE on success, FALSE otherwise
bool update_core_version (object &$output, int $version)
  • object &$output: collects the html output
  • int $version: the new version number to store in config table
update_create_table (line 1096)

create table in database from an individual tabledef

  • return: TRUE on success, FALSE otherwise + message written to logger
  • uses: $DB
bool update_create_table (array $tabledef)
  • array $tabledef: contains the definition of a single table
update_create_tables (line 1072)

create tables in database via include()'ing a file with tabledefs

  • return: TRUE on success, FALSE otherwise + messages written to logger
bool update_create_tables (string $filename)
  • string $filename: contains the table definitions
update_language (line 406)

update a language in the database

this routine tries to update the information in the database with the information in the language manifest of the selected language $language_key. The event is logged via logger().

Note that an upgrade of a language is not at all interesting because there is nothing to do except to update the data in the databse with that from the manifest. However, we still do it this way in order for the user to grow accustomed to it so we can complexicate this routine in the future without the user having to learn new tricks.

  • return: results are returned as output in $output
void update_language (object &$output,  $language_key, string $lanuage_key)
  • object &$output: collects the html output
  • string $lanuage_key: primary key for language record in database AND name of the /program/languages subdirectory
  • $language_key
update_module (line 587)

call the module-specific upgrade routine

this routine tries to execute the correct upgrade script/function for module $module_key. If all goes well, a success message is written to $output (and the update is performed), otherwise an error message is written to $output Either way the event is logged via logger().

Note that we take care not to load spurious files and execute non-existing functions. However, at some point we do have to have some trust in the file system...

  • return: results are returned as output in $output
void update_module (object &$output, string $module_key)
  • object &$output: collects the html output
  • string $module_key: unique secondary key for module record in modules table in database
update_remove_obsolete_files (line 1834)

attempt to remove or at least flag obsolete files

this routine can grow bigger on every update when perhaps more files are obsoleted. We always check all files (even the older ones) because the user might not have removed them yet. If we can delete the files, we do so. If not, we log it and also show a message to the user via $output.

  • return: TRUE on success, FALSE otherwise
bool update_remove_obsolete_files (object &$output)
  • object &$output: collects output
update_show_overview (line 166)

display an introductory text for update + status overview

  • return: results are returned as output in $output
void update_show_overview (object &$output)
  • object &$output: collects the html output
update_status_anchor (line 930)

return an anchor tag with link to the specific update function

This utility routine returns a ready to user HTML anchor tag.

  • return: ready to use HTML-code
array update_status_anchor ([string $task = NULL], [string||null $key = NULL], [string $anchor = NULL])
  • string $task: which update task do we need to do?
  • string||null $key: which module/theme/etc. (NULL for core)
  • string $anchor: text to show in link
update_status_table_close (line 1012)

close the status overview HTML-table we opened before

this is the companion routine for update_status_table_open(); it closes the open HTML-table

  • return: results are returned as output in $output
void update_status_table_close (object &$output)
  • object &$output: collects the html output
update_status_table_open (line 986)

open a status overview HTML-table including column headers

this routine opens an HTML-table in prepration for a status overview of the system or a subsystem (languages, modules, themes). The optional title is used as the header of the first column.

The width of the first column is 25% and the remaining 5 columns area 15% each which creates an orderly display of name, internal version, external version, releasedate, release and status.

  • return: results are returned as output in $output
void update_status_table_open (object &$output, [string $title = ''])
  • object &$output: collects the html output
  • string $title: is the header of the first column
update_theme (line 802)

call the theme-specific upgrade routine

this routine tries to execute the correct upgrade script/function for theme $theme_id. If all goes well, a success message is written to $output (and the update is performed), otherwise an error message is written to $output Either way the event is logged via logger().

Note that we take care not to load spurious files and execute non-existing functions. However, at some point we do have to have some trust in the file system...

  • return: results are returned as output in $output
void update_theme (object &$output, string $theme_key)
  • object &$output: collects the html output
  • string $theme_key: unique secondary key for theme record in themes table in database

Documentation generated on Tue, 28 Jun 2016 19:12:25 +0200 by phpDocumentor 1.4.0