Update 05 Nov 2012: Please note that this post has been superceded.
Calling all programmers: we need your help!
As you may well know, Textpattern 5 is undergoing a major rewrite; moving from the old function-based approach to a more modern MVC framework: Spark/Plug. Since the announcement, things have gone slow. No, actually slower. Various projects and other commitments have meant that we couldn’t divert the resource required to do the project justice.
While public movement of Textpattern 5 has been glacial, in the background three things have happened:
- I (Bloke) have been learning the ropes of this MVC malarky and documenting key Spark/Plug functionality as it relates to Textpattern. This gives potential patchers, developers and plugin authors a leg-up on what the underlying framework offers. It’s still pretty scant and a long-term work-in-progress that will probably transfer to the wiki at some point, but it offers some insight into the naming conventions and core features that make working under S/P a joy.
- In tandem, Sam has been answering awkward questions and hammering updates into Spark/Plug which have actually justified the crawl. For example, the framework now supports shiny CSRF protection for starters, which means porting over Txp 4.4.1’s security features is, well, not necessary in the code since the framework does it for free (as long as the conventions are adhered to).
- Phil’s admin theme has matured to the point where it’s a pretty slick replacement skin for today’s Classic theme, so it’s been incorporated at this stage. We know there are hundreds of iterations to go as it’s tweaked and refined to bring in new features which improve workflow, but it’s a solid base upon which to build.
One screen to rule them all
To demonstrate the way of working, one (admittedly simple) tab has been coded pretty much end-to-end: Presentation Styles. This shows off a lot of functionality and best practice working with Spark/Plug. There’s a little work to go, but it pretty much functions as it does in 4.4.1 today.
Part of the reason this screen was chosen as the blueprint was because it required no database changes. Things like ‘rename sheet’ need to be factored into the UI yet, but that relies on a few changes to the workflow and a schema change or two, so it’ll be folded in further down the line once the install/update mechanism has been started and Phil has had a chance to reconcile the functionality into the theme.
As it stands today, you can use a stock Textpattern 4.4.1 database with Textpattern 5’s Styles panel. All the other tabs are just HTML mock-ups.
Hobbits unite
The real work now begins. While we could continue to hammer away at this behind closed doors, the spirit of community involvement and the benefits of using Mercurial as a VCS are that clones and branches are fairly slick to manage. So instead of the old “here’s a patch” methodology, a more visible “I’ve cloned the branch, here’s a pull request” approach is being taken.
The distinction is subtle but the ramifications are huge. No longer will one person submit a patch to a dev list and wait: you’ll publicly clone the master branch and work on your edits in public. If anyone else wants to join the party, they can clone the clone. Peer review of the clones is public. Anyone can download any clone direct from the repo and use it, offer advice, and shape it until it suits everyone. We’ll all be involved in that: devs (gatekeepers if you will), enthusiasts and people who want to push Textpattern forward. Once a clone has been tested to destruction by the community and it’s ready, a simple Mercurial pull request notifies us so the functionality can be folded into the core.
The first job is to port all the existing functionality of Textpattern’s admin side over to the Spark/Plug framework using the Presentation Styles panel as a blueprint.
To make all this work, we do need some kind of master list of who’s working on what so we don’t duplicate effort. A simple public spreadsheet ought to do it for now. If you want to tackle a tab, jot your name down, maybe team up with someone, clone and get cracking.
Over time we can expand this so if you’re thinking of tackling a new feature you can add it to the list to signify your intention, which will also attract like-minded people to download, test and/or develop your clone. Cloning a repo adds it to the Google Code Clone page so maybe there’s a better way than a spreadsheet to keep everyone on the same page. Please speak up if you know of one, or can donate some Basecamp bandwidth.
The road to Mordor
There’s not much else to say. Get in touch with one of us to gain access to the developer guidelines doc and spreadsheet (they’re both Google Docs so can’t be made blanket public, unfortunately—maybe they should be moved to the wiki sooner rather than later).
At the back of your mind, keep a copy of the Textpattern platform mantra:
- Powerful
- Lightweight
- Flexible
- Plugin/theme extensibility over core bloat (i.e. lowest common denominator as default functionality with hooks for easy enhancement)
Please clone and help make Textpattern 5 the CMS you love. Let’s fling Txp 4 into the volcano and forge a better, smarter ring from the molten metal. It’s what Dean Frodo would have wanted.
Comments
CH
Good luck guys, I still love TXP after all these years.
Jukka Svahn
All I can say is “meh”. I’m disappointed. Not everyone wants a rewrite, and to be honest, I’m one of those guys. I do want a lot work to be done on TXP 4.x base, but this sluggish rewire isn’t it.
But while you are trying to do TXP 5 something shiny and cool (“meh”), I would suggest not using PHP’s old datetime functions, but the newer datetime class. Old datetime functions use signed integers, which do not go that far. I’m not implying that TXP 5 never comes out or takes 20 years, but still.
Phil Wareham
@Jukka
What’s you take on what TXP5 is going to be, and your arguments against a rewrite (not being a programmer, I don’t know exactly what is being rewritten/changed for TXP5).
You are the type of collaborator that the devs should be attracting to help on the project (be it TXP4 or TXP5) from your outstanding work on plugins and forum advice.
My personal involvement is to try and improve the admin-side layout and HTML codebase to make it more logical, flexible and themeable. Practical over shiny and cool.
Jukka Svahn
@Phil
All power to you. Interface changes are very welcome. I have nothing against making the interface better. As you were asking how much is going to be rewritten… completely I suppose.
It’s just that I do not like TXP5’s direction. To me Textpattern is just an empty canvas with very minimal footprint. Small library, no real fun and dandy huge frameworks. Only thing that is essential is the tag parser (tags) and template/session/preference handling. It’s a hub to host software on, doing projects that need flexibility. No expectations, no frameworks, no nothing. That’s the good thing. And TXP5… is something else. I feel it’s too complicated.
I don’t know if it’s Stef being playful, but when he says that he doesn’t understand something and needs to ask guidance from Sam… that doesn’t seem right. You should understand your code. That gives me the bad wipes about bad design or something. I’m also worried whether everything is used correctly, and testing and most importantly security.
Instead of it being that complete rewrite, I would like to see improvements on the current codebase. For example changing the tag parser so that it accepts array of attributes, has actual tag hander (registering), supports classes (static and otherwise) and methods (in addition to global functions). Best part is that with tag registering and classes come magic methods; every PHP function can be made a tag writing minimal lines of code, and all singular tags (site_name, site_url etc) can use the same code.
I have nothing against changing the code layout to OOP (that would be awesome even). It can be done. With compatibility. Without changing the complete model. I would love if the existing codebase moved forward little by little. Starting by making an actual roadmap. Writing down what will be removed, when deprecated code will be removed, how long some compatibility layer is kept. Then doing it by doing task#1, task#2, task#3. Then 3 months later we have TXP 4.5 or what will be it. Then TXP 4.6. Then 4.7 that drops some deprecated code and so on.
John Hennessy
Hearing that this is a total rewrite makes one wonder if upgrading will break websites already running TXP. Will Textile work in the same way and what impact will there be on plugins?
Giuseppe
It is interesting. I didn’t know about TXP 5 and its completely rewriting.
I only hope devs are going to create something like Expression Engine, i.e. a CMS which is “form agnostic”. That’s the only way to create a real CMS which would be able to adapt itself to the always different content needed by each website since each is different.
I really loved TXP, but even with the excellent plugin to create custom fields (and above all all the guys at the forum), I was tired to use “images captions fields” to create “description text for features of a product” and to manually set another custom field where to set “the category of those images” to be able to associate them to their product page… we have all better ways to spend our time, do we?
So, if TXP 5 will be that kind of CMS, long life to TXP!
Stef Dawson
@Jukka: noted. Yes, definitely datetime class. Seeing that PHP 5.3 or 5.4(ish) is probably going to be minimum spec, that’s a good excuse to use it.
Wish you’d spoken up earlier. You’re the sort of person I want to push us onward and upward, especially when it comes to the tag parser. You keep us all sharp.
Thing is, we couldn’t see a way to get to a more OOP approach without ending up going through the Crockery thing again so a clean break seemed like a good way to go. And as Spark/Plug was available and small compared to some other frameworks, it seemed a good way to improve the plugin power of Textpattern (among other things). As you imply, there may have been a way to do OOP on the existing codebase but as Jeff noted a while ago, Textpattern has almost outgrown its own code. If you thought otherwise I should have listened, but I never spotted your strong opposition to the move. Perhaps I’m the one who needs a new Observable Class in my head :-p
Regarding it being sluggish, I can’t really comment yet because everything’s sluggish on my machine — even Txp 4. I’m sure I spotted some places yesterday where optimisations could be made by passing vars between methods instead of fetching them again, but it was 4am and I stopped looking. Remember the screen(s) I’ve done so far are just clones of Txp 4 functionality. When Phil’s factored in the ajax workflow and made space for some other functionality like renaming things, and displaying where other items are used (like adi_forms / adi_pages does today), the streamlined operation will help improve the perceived speed.
Other things like observer/observable will be a huge boon for plugin authors and admin-side themers as you have finer grained control over what areas you affect, and the enforced security rules/naming conventions in Spark/Plug mean that plugins are automatically more robust than today’s ones. We can employ RESTful states too if we choose.
As for me asking guidance from Sam: 90% of the time it’s been because I just didn’t “get” MVC as a concept. Nothing to do with Spark/Plug or Txp itself, just me being dim in the same way it took me a lot of back-and-forth before I understood OOP first time round. It just ‘clicked’ one day, and now I see the power of separating the three areas it’s really opened my eyes.
Let’s face it there are parts of Txp 4 I don’t get and I leave well alone (tag parser, for instance). Doesn’t stop me doing things in other places and leaving the parts I find voodoo to clever folk like yourself and Ruud.
A parallel: I don’t understand how jQuery does its stuff but I use it almost every day (well, maybe 10% of it). Am I a bad programmer in that regard? Yes, because my nature has always been to understand something fully before using it. But in jQuery’s case, does it keep me awake at night? Not often, because I tired of doing stuff with raw DOM manipulation as it’s cumbersome. Moreover, I trust the jQuery team to take those headaches away. Same with Spark/Plug. The bits I need, I look up and learn new techniques as I go. As new stuff comes into the product (jQuery or S/P) I change my habits and app code gets leaner.
Having said all that if you can show me/us how to take the existing codebase and make Txp more powerful, more flexible, and more secure without requiring any major restructuring or any framework then for goodness sake speak up! I’d rather do that than waste my life writing something that you know is going to be crap, whereby you can turn around and say “I told you so” :-|
Stef Dawson
@John Hennessy: Yes, Textile will still be supported. You’ll also be able to install any other system you or your clients may prefer, like Markdown.
As for plugins, there’s always collateral damage in every release. The transition is planned to be smoothed with a compatibility layer that will stay active for a good few revisions of the core. That will present a Txp4ish interface to the world, mapping the old functions and variables to the new core.
The documentation that myself and Phil are preparing on the admin side DOM hooks and features of the core will help plugin authors migrate to the more powerful engine as time permits. The tag parser will undergo a separate revision in a later phase of the project: for now we’re focusing on the admin side.
Walker Hamilton
Guys, this is a bit disappointing, having asked for code access back when the change was announced. I was told that you’d be getting as much done on the transition before releasing the code.
Then this comes. It feels like a cop out. I know this is an open-source project, but I’ve long since abandoned it.
Also, you couldn’t put it on github? I’d be much more likely to fork this project if it lived there.
I wish you well, but count me out.
Stef Dawson
@Walker: That was the plan. Then life got in the way. Selfish of me to not admit this earlier? Yes. But what would have happened if we’d released access to the repo with no code at all? Probably get equally flamed. And getting flamed for something I do as a hobby isn’t my idea of fun (that’s why I don’t do roadmaps, btw: if it slips by even one day I’m culpable, and since I can’t guarantee a sustained level of dedication to the cause, it seems more like lies than just doing it as and when time permits).
As for github: wasn’t my call. Mercurial’s not that bad once you get used to its little foibles.
Honestly, I didn’t want to be writing posts such as this. I didn’t want to delay things in the first place. I didn’t want to have to make decisions on the future of Txp when I wasn’t sure if what we were doing was right. So I wanted to try and make sure; to document things; to start off the libraries; to wait for Spark/Plug to mature further; to reach a state where at least I was happy Txp 5 was possible and then open things up so the work could be shared and done quickly.
I don’t want closed doors. I don’t want a dev list using a 1990s majordomo. I want a place where anyone who can wield PHP or HTML or CSS is able to contribute to the project in public. I want to attract talented coders and designers to Txp through honesty and non-elitism. To let people play in our streets, try things, prove things, generate attention and testers through social channels, refine and see which ideas get the greatest traction then just roll them into the core without fuss. If it’s proved and tested under public scrutiny and there’s a case for doing it to strengthen the CMS then that’s what it’s about.
I’ve been this close (indicates tiny distance between thumb and forefinger) to chucking the towel in on a few occasions last year and just tinkering away on Txp as a personal project. The only reason I’m still here (besides being thick skinned and able to admit when I’m often wrong) is because I believe in the goals of Textpattern as a CMS and I believe wholeheartedly in the community that surrounds it.
But if my actions such as posting here and doing what I can, when I can are dividing the community or are not helping the CMS — counter to my intended goals — then perhaps it’s time I let someone else who has a better grasp of code and project management do a better job.
Marc Carson
Stef, you’re doing a great job of communicating here. Thank you for keeping us all posted. I do get the sense that you’re communicating what you feel is in some ways a less-than-ideal development model.
Perhaps we should be hearing from the rest of the dev team, too—what their opinions are, why they think things are lagging, and what time they will be able to commit to this project—otherwise I’d say you’re 100% correct in feeling frustrated.
I was really pleased to see Spark/Plug announced, and as a MVC newbie myself I can’t see not using the MVC concept in most of my PHP projects anymore. But there are some good reasons why, as someone new to MVC, I’m not using Spark/Plug. :-) If it’s any sort of roadblock, it seems like that should be discussed. Do docs need to be written? Better examples than pointing at source code? I don’t know the answers, but it doesn’t seem well that a volunteer dev team with plenty of 4.x momentum should be blocked by any of that if it exists.
I also wondered about Mercurial…Git seems almost a no-brainer these days, especially if we desire community involvement, but again I don’t know what discussions have come before.
Thanks again for the update and for all of your work toward making Textpattern a strong CMS.
Dale Chapman
Remember, txpeeps, that no one is getting paid to build txp, and guys like stef contribute a lot of time to the effort.
I’m eager to see what v5 looks like, but as I have a significant investment in custom txp plugins I’m also very interested in continued 4.x development and or a computability layer for plugins.
I nominate Jukka as a dev to continue 4.x development.
Stef. Don’t be discouraged. Your work and the momentum you have brought to txp’s development have been astounding. You’ve also brought a transparency to the process which is refreshing. I wonder sometimes where you get the time for a day job and a social life.
So thanks again!, Stef.
Marc Carson
That’s odd, was my comment deleted? Sorry if I offended anyone…
Patrick Woods
I know that re-writes are tempting but they take a lot fo time to get you exactly where you were before. There is a lot of clean up that could happen in 4 that could lead to a more modular system. The way the tabs are broken up into files atm almost works like a controller setup with the event/step system serving as a simple router. Work could be done to separate presentation from the functions, alot of which is already done in the xml-rpc lib, it’s just not reused in the core. From there moving the DB calls into organized models moves the system into MVC.
Remember MVC is just a pattern it doesn’t need some fancy underlying framework.
That being said Spark/Plug looks nice and Escher is pretty cool, but I see shades of Crokery 2. I really liked the continuous improvement we were seeing and having it all stall for the grand rewrite is kind of sad.
I love the switch to mercurial and the open invite for pull requests. (would have loved to see a move to Github but that’s just bikeshedding). Need to get a clone and start poking.
Keep up the great work, it’s not easy.
Stef Dawson
OK, so here we are then. Three or four of the community members who I hold in highest regard in terms of coding are saying that moving to Spark/Plug is not a good thing and that the Txp 4 core has, in fact, not outgrown its usefulness and can be resuscitated and extended with a bit of love. Someone has even suggested moving to ProcessWire which actually has some merit!
I like Spark/Plug and enjoy working under the framework. It takes a lot of the headaches away. But equally I don’t want to do things “wrong” as seen by the rest of the community, nor do I want to drive our current talent away.
As long as you are all 100% convinced that it’s not just fear of the unknown(!) and the current core can in fact be modernised, then we should do that. Right now. Forget what’s been done on Textpattern 5 to date on Mercurial. I really don’t care: you won’t hurt my feelings. Just please, please show me the light so I can see how to proceed. Show us how to move the current codebase to an MVCish pattern so we can bring all the things we want into the CMS.
Take this discussion to the forum/G+/FB if you prefer. Check out the current 4.x SVN and make changes to demonstrate it’s possible to modernise the core. Last year we couldn’t see a way to do it cleanly and thought S/P was the ticket. Things have moved on since then so maybe there is another way.
Keep in mind these things that we would like to achieve:
* unlimited cats / taxonomy / custom fields
* a more normalised DB structure
* support for DBs other than just MySQL
* more installation options, e.g. choosing a default theme package or pre-packaged baseline to install
* namespacing so we can integrate more easily with other systems (e.g. having a global called “DB” is silly. It should at least be TXP_DB, but a proper namespace for the project would be preferable).
* a more secure tag parser (secondpass: Gocom!)
* better security for plugins, e.g. forcing variable types in core functions like lAtts() so plugins don’t have to do it, which will help reduce runtime errors and reduce opening unwitting security holes in plugins
* prevent any old PHP function being used as a tag (rah_function notwithstanding: that’s extended functionality which is acceptable, because it’s caveat utilitor; but it shouldn’t be default behaviour in the core, imo)
* better prefs handling
* simpler admin side interface building (at the moment it’s a real chore to build a screen under extensions — some API calls to simplify things (think: admin-side ‘tags’) would be a real help)
* improved user management API — remove reliance on flat file for privs/groups
* a more AJAXy workflow where it makes sense
* improved image/file support and workflow
* yahde yahde
As long as you think that’s all doable under the current codebase, then go go go! We’ll worry about shifting development to a different VCS at a later date so we can take advantage of better branch/clone techniques and pull requests etc.
Just please don’t sit there and tell me I’m wrong. Show me I’m wrong so I can help make it right.
Phil Wareham
Whatever route you decide to take (v4.x incremental improvements or v5 total rewrite), I can still work on the admin-side HTML/theme rewrite – it needs to be done whatever.
One thing though: Please please please can we reconsider using Github for developing Textpattern 5 (or 4 or whatever number you finally decide on) instead of Mercurial. Or implement a Git-Mercurial bridge (yes they exist).
christoph
being a marketing professional, i have almost no clue of programming. but maybe i can contribute one perspective from my studies in economics: according to many, many studies, it is an empirical proven matter of fact, that the concept of “constant improvement” is always superior and more efficient than the “start from scratch”. maybe there exist certain limits in programming that require a platform change if one wishes to achieve certain functionality, i don’t know. but i would balance my reasons very accurate before i decide for such a restart. considering this, i second what jukka said.
Patrick Woods
Sorry guys, I think I’m letting some work thoughts overflow into TXP. We’ve chased silver bullets a lot there and we’ve ended up with various fractured systems that never moved the system forward. Currently I’m falling much more on the refactor towards a goal idea as a way to move a code base forward.
This of course does not imply that TXP5 will fall to the same trappings, but these ideas were coloring my comment.
Refactoring does require having an end goal/idea imo, and that end goal can still be getting TXP on Spark/Plug. The challenge is how do you get there in smaller incremental steps.
One fear is that rewrites can give the illusion of lack of progress or movement. TXP was very quiet for a long time and recently has had a good amount of movement which increases the project’s visibility. I would hate to have that visible movement disappear just as Destry is doing so much work on FB, G+ and TXPMag.
Marc Carson
Regarding ProcessWire, Ryan Cramer, the developer of ProcessWire, was generous enough to offer to maintain a pure framework branch of ProcessWire (for TXP and others to build on) should it be needed.
Since there are quite a few current and former TXP users who use ProcessWire, I’ve put out some feelers over at the ProcessWire forum.
Marc Carson
Just posted a summary of my feelings on the use of ProcessWire for TXP5.
Giuseppe
About GIT vs. Mercurial debate, I used both for long time and, IMHO, I think Mercurial is better. I have no time to write you why, but someone explained why Mercurial is better (or at least, different) then GIT better then I ever would be able to.
That said, we have to admit that GitHub is the real GIT. A really well made social repositories service but for Mercurial users (and now also for GIT ones) there is Bitbucket which is getting better and better. And, yes, it is free also for private repos…! ?
Lawrence
The way I see it, Texpattern works wonderfully in its current state but it’s not going to be long before it’s elegant simplicity will start looking and feeling tired. Even if it isn’t works wonderfully the perception clients have is just that – based on some great interfaces (front and back end) of other CMSs.
I’ve actually started to look at WP for the new version of my main project because of some of Textpattern’s limitations. Blasphemy, I know.
So I’m keen to see TXP developed so that it futureproofs itself and remains competitive.
6sigma
Caught up on a bit of this over the weekend. After all the talk, have the following decisions been made?
1. Rewrite or not?
2. Spark/Plug framework or not?
3. Processwire or not?
4. Mercurial or Github?
BB
I use textpattern from RC1 and I still love it too ?
Commenting has expired for this article.