Since the early days of Textpattern’s existence, it used what was colloquially known as the RPC server for various upgrade tasks. The server drove every external interaction that Textpattern core made with regards its updates, language provision and popup help topics. It served its purpose for well over a decade, but large numbers of requests (such as at upgrade time) along with changes to the way some hosts limit access to remote resources from PHP, put extra load on the server and caused issues such as “Cannot connect to update server”.

From v4.7.0, reliance on that resource was designed out of Textpattern completely: language Textpacks (i.e. the translated text for supported languages) ship with Textpattern in its lang directory. This has a number of important ramifications:

  1. No distinction between updating languages from the RPC server vs from a file: all language handling is from the file system.
  2. No direct provision to “upgrade” language strings between version releases by clicking Update from the Languages panel.
  3. No need for us to maintain old, unused strings for previous versions, which makes the language table in your databases potentially smaller, and faster to process.
  4. No more incorrect language strings because someone changed one to support a future or legacy version of Textpattern.
  5. Language strings are automatically updated on upgrade.

Thus, the strings you have in your database when you download and install Textpattern are the only ones you’ll get. You may think that’s a bad thing… a regression in functionality. Well, yes and no. You’ll still be able to get language pack updates between releases if you wish, but there’s no real need to do so, since all the strings Textpattern needs are shipped with each release.

If you do wish to update the language pack (for example, to bring down new or altered strings in response to a change or bug fix), you perform a three-step process:

  • download the language pack
  • drop it in your installation’s lang directory
  • apply the changes from the Languages panel to import it into your database

Far from being a regression, this gives you greater control over the process than relying on the server telling you that updates have been made; updates that may have not applied to the version you have installed, or that may adversely affect the user experience.

A leaner database table

Bundling the strings directly in core does mean that the overall size of the uncompressed Textpattern distribution has increased significantly – almost double – because all languages are present instead of just en-gb. But the benefit of having all languages instantly available to install without any external server involvement outweighs the file size bump.

We continue to prune legacy strings. To put this in context, a typical lang table in Textpattern 4.6.2 had over 1300 core strings per language. Each string takes up a database row. Install a couple of additional well-represented languages and you’re looking at almost 4000 rows. That’s a lot of data. Since many of the strings simply weren’t necessary – merely there to support legacy versions – we’ve removed a few hundred strings per language from the out-of-the-box installation.

We haven’t yet taken the step of forcibly removing old strings from your database on upgrade because we don’t wish to risk deleting plugin strings by mistake, but new installations (and those people that delete and reinstall an entire language pack) will benefit from a leaner database. It’s a step that’s well worth taking if you know there aren’t any custom strings you wish to keep.

Less opinionated languages

Taken in isolation, the above changes might seem unnecessary or a backwards step, but they’re part of a broader spectrum of alterations.

One major change from 4.7.0 onward is that we no longer use en-gb as the default language. It’s now en, or Oxford English, which is an international standard mashup of en-gb and en-us.

We have also opted for reducing how opinionated our languages are. So instead of the full language designators such as fr-fr or ja-jp, we have used the simpler language identifier where appropriate (fr, ja, etc). Plugins and code that use the old identifiers will be seamlessly converted to the new ones, though we encourage plugin authors to start using the new designators in code going forward.

To make things simpler for Textpack management, language designators may be listed in a pack. So if strings in your plugin can apply equally to a bunch of languages, you can define them once and benefit from the same strings being loaded for any of the given languages:

#@owner your-name-or-plugin
#@language en, en-gb, en-us
english_key1 => value1
english_key2 => value2
#@language fr
french_key1 => value1
french_key2 => value2

The number of languages continues to increase, taking our official supported tally to 54 as of this writing, including the latest addition to the flock: Fulah. Our translators work behind the scenes on Textpattern translations at CrowdIn, and we welcome contributions from anyone for our current languages or new ones. Please get involved if you can to help the majority of these bar charts near 100%. Every translation helps countless administrators and users around the world.

Separate admin-side and site languages

As of this article, Textattern supports a single front-end website language, chosen from any of those installed. There are techniques you can employ via plugins or code snippets to deliver content in multiple languages but it’s not a core feature yet. We’re actively working towards true multi-lingual content.

On the back-end, however, authors and administrators can select the language in which to display the admin panels. Each user can choose any language that suits them, selected from those that an administrator has installed. Administrators also get to see how many authors are using each language via a counter alongside the language designator. This is a handy visual aid if you wish to remove a language, as you can gauge its reach among your user base.

Two additional features are:

  1. Users can choose their interface language directly from the Login panel. Whichever language is chosen there becomes the back-end language for that user immediately upon login, and remains that way until it is changed.
  2. Administrators with access to the Admin>Users panel can set a language for existing or new user accounts. For new users, this will trigger the welcome message in that chosen language. It will also render the login panel in the given language (provided a translation exists for the strings it uses), which is a much friendlier workflow.

String handling and plugins

In an effort to reduce the memory load on each page, all language strings are represented according to their events. That means we have strings that only appear on the Preferences panel in a [prefs] group. Strings that only appear on the Files panel are in a [files] group, and so on. The group is the name of the panel given in the URL after ?event=.

On any given panel, only the strings that are required for that panel are loaded, plus the [common] ones (see below for an overview). This system extends to plugins, so if you want strings to appear on your own plugin panel, make sure they are assigned to a group that matches [abc_your_plugin_event].

There are several special groups of string:

  • [admin-side] strings are loaded on every admin-side panel. So anything that appears in the menu (such as your plugin panel’s name), header or footer goes here.
  • [public] strings are loaded only on the public website.
  • [common] strings are loaded on both the public website and every admin-side panel.

So if your plugin is rendering abc_plugin_string instead of the correct translation, check it’s in the correct group. The smd_babel plugin is a handy back-end tool for checking, altering and applying language translations to any language for any string.

It may seem tempting to stuff all your strings in [common] for simplicity, but please don’t do this as it’ll increase the memory footprint of the public site too, and every admin-side panel. Instead, if you need access to strings from another group that’s not one of the special groups mentioned above, employ code like this:

// Fetch a reference to the language class if you don't already have one.
$langObject = \Txp::get('\Textpattern\L10n\Lang');

// Extract strings from the given groups in an installed pack.
$strings = $langObject->extract('<langDesignator>', array('article', 'image', 'prefs', ...));

// Set the internal strings to use them.
// true = append to what's already loaded, false = replace entirely.
$langObject->setPack($strings, true);

Language installation and plugins, with fallbacks

Textpacks are handled seamlessly in plugins. Previously, Textpacks were only installed for the current language when a plugin was installed. If you subsequently installed a different language—even if it was represented in the plugin—tough.

When you install plugins in Textpattern 4.7.0+, you are shown its language pack(s) at the Plugin Preview step, and all these packs are stored along with the plugin itself in the database. This means that if you have French and English languages available in Textpattern at plugin installation time, you get the plugin’s French and English strings installed. But if you later install German and the plugin had a bundled German Textpack, it will be installed as well at that time.

This is a massive help for plugin authors as you can embed all relevant strings in any languages in your plugins and have Textpattern automatically install them for any language. If an administrator removes a language and reinstalls it, the plugin strings come along for the ride and are automatically reinstalled.

Further, if a language is missing from a plugin Textpack, or only partially translated, the strings from the default language will be used to fill the gaps. This is known as your fallback language and it’s set by default to en. It can be changed in your config.php by setting the constant TEXTPATTERN_DEFAULT_LANG but it’s probably not advisable unless you know what you’re doing: you should certainly choose one that is well-represented, preferably one that has 100% of its strings translated or you’ll get untranslated strings in your interface.

The above language segregation means it is imperative that you designate the #@language in your Textpacks for correct language assignment. You can no longer rely on the ‘default’ (current) language being the first one in the list if it doesn’t have a designator. Language strings in a pack without one or more designators will always be assigned to the default fallback language (en in most cases), which may not be desired.

This also applies to plugins loaded from the plugin cache directory. Any strings for the currently-used admin-side language of the user, or the language for the public site, will be loaded on-the-fly from the plugin file.

Exciting language improvements on the horizon

As we progressively add features, we’re working towards bringing a full multi-lingual experience to Textpattern. We’ve already made iterative steps in that direction by:

  • Adding high level callbacks to intercept URLs.
  • Bundling languages with each release.
  • Customising URLs per section.
  • Decoupling admin- and front-side languages.
  • Expanding the Textpattern tag suite to interact more closely with URL parameters.

As it stands, a combination of plugin code and a little clever form processing can get you most of the way. There are a few wrinkles around the edges, such as implementing a mechanism for natively tying content that differs only by language, and syndicating it. That’s all less than optimal right now and requires planned database table changes and a tweaked UI workflow to tie everything together.

But with some ingenuity, it’s possible to deliver a multi-lingual front-end experience in the latest version of Textpattern; at least for two or three languages before it becomes unwieldy to manage. There have been various discussions around multi-linguality including one on language handling at the tag level, and a full proposal document (Google Doc).

As with any proposal, community members form a key part of the process. So if anyone has any questions, comments or ideas, please raise them here, on the forum or in the proposal document (whichever medium is most appropriate) so it can be considered. We are especially interested in the views of people who routinely make (or who have tried to make) multi-lingual content websites in Textpattern.

Being able to adapt the interface and content management goals for the most flexibility using suitable conventions to simplify the process is the aim. Thank you in advance for any thoughts you may have.