Progress has slowed a bit while I migrated to a new server and cleaned things up, and rehabilitated my Git infrastructure. Prior to that, I’d been doing a lot of research about how I’m going to proceed with TightURL.
I’ve been seeding hooks throughout the code, and a concentration of them ended up in the template code. One of the things I’ve always been uncomfortable about in TightURL is the template system. One of my long time pet-peeves is about everyone creating their own PHP template system, and I ended up doing the same here, just because I really wanted the most lightweight template system I could get. But the discomfort remains to this day, and as a result, if you’re so inclined, you’ll be able to use a plugin to replace either parts or the entire template system.
After cleanup, I’ve got three template hooks, all of which are filters:
- template_fetch : Retrieves a template, can only be used to replace the template retrieval process. Use this if for example, you want to store templates in the database instead of the filesystem. If no plugin causes HTML to be returned by this hook, the template will be searched for in the file system as usual. For the time being, you’ll have to be careful not to use more than one plugin that replaces the template retrieval process, as whichever one runs last will be the one whose HTML is returned.
- template_process: Processes template. If you want to set or change template variables, you set those here by filtering the template variables. If you wish to replace TightURL’s template processing system, you do so here by doing “final” processing and returning the HTML to be displayed. Here again, if you were to try to use more than one plugin that replaces the template processing system, the results would be inconsistent (bad).
- template_filter: Filters the processed template, just prior to display. If you only wish to modify template output before it is sent to the user, this is the filter to use. As with all other TightURL filters named something_filter, more than one plugin can hook into template_filter and modify the output.
The template_fetch and template_process filters combined, will allow plugin developers to completely replace the TightURL template system, and to do so in a way that preserves other plugin developers ability to filter the output before it is sent to the user.
Still working on TightURL, except most of the last couple of weeks has been spent looking at other people’s code, examining frameworks, and finally doing some serious reading about object-oriented PHP (something I’ve been meaning to do for at least 10 years!).
I am still forward-porting the code from the running copy at tighturl.com into the development version, which will be the first of two or three re-writes. The goal of the first rewrite is to adjust the coding style that was used to how I write things today, incorporate changes from the production code into the development version, and begin breaking down the bloated save_URL function into smaller functions with the anti-abuse code separated out. The template system will be enhanced in anticipation of plugins. Work on a REST API will begin.
The anticipated goal of the second re-write will be to fully incorporate the plugin and cron systems. I may include internationalization in this re-write. (if not it will come in the third re-write, or be added after the wholesale re-writing of the application)
Finally, if there’s a third re-write, it will be to correct earlier design errors.
Because I have already begun to add support for them, these features will emerge by the time the code is released:
- Localization (translations)
Unfortunately it seems to be a real hassle dealing with internationalized URLs. I’ll do my best here. Translations seem to be difficult as well, but I’ll make a best effort here too.
- Internal cron jobs
TightURL will be able to run scheduled tasks via standard (Vixie) cron syntax. The TightURL internal cron system can run either at the end of HTTP requests to the application, or you can have your system task scheduler run these in the background. Cron jobs can be created and managed via plugins.
- Additional Squisher Engine support
TightURL will come standard with two Squisher Engines: Default, and Default+. The default squisher will create traditional case-insensitive TightURL IDs, and default plus creates case-sensitive IDs.
Additional policy, template variables, Squisher Engines, and other functionality can be added to TightURL via plugins.
This is actually less ambitious than it probably sounds, a lot of this work was started over the last couple of years, and I just need to tie it all together now.
After being interrupted nearly a year ago, work has resumed on TightURL, prompted, as usual, by abuse taking place at tighturl.com.
The cron system has had to be replaced again.
I don’t want to make any promises, so I’m not going to say too much more. There will be major changes. You’ll need PHP 5.3+. The database is going to change. Your urls table will now need to be named “urls”. I still haven’t found a framework/library for templates or APIs that I like, so I’m probably keeping the existing template system. If there’s an API, I’ll have to write the protocol code myself. These “lightweight” things (frameworks) still contain more code than all of TightURL, making them unsuitable for use.
I haven’t mentioned the plugin system for a while, because I’ve been working on the production version of TightURL rather than the “next” version. However, I’m already starting to use callbacks in the anti-abuse system, which amounts to nascent support for plugins. The first place this is showing up is in the URIBL checks. Since neither SURBL nor URIBL.COM has a stable API (the querying method changes, in URIBL.COM’s case, in undocumented ways in realtime *sigh*) I’m going to implement URIBL support through plugins. This will allow users to upgrade their URIBL plugins as needed without upgrading the rest of TightURL, and it will allow users to add support for additional URIBLs through their own or 3rd party plugins.
Since I point interested users to this devblog, I wanted to ping the blog and let everyone know I’ve been working on TightURL every day for the last couple of weeks. I haven’t been able to test any of the new code yet though, because I’m still changing it and in so many places it’s not practical to try and actually run it.
The good news is, I think I’m getting close to having a settled way to handle the anti-abuse system. I’m still getting useful data from the last set of changes I put into production, and I’m trying to incorporate lessons drawn from that abuse data into the anti-abuse system. As I think I mentioned before, it’s currently very crude and it tends to over-block things. It needs to be more nuanced before I can trust it myself, let alone share it.
I’m still ripping the anti-abuse code to shreds and putting it back together again, over and over. Eventually when I get it right and done, it’s going to represent a lot of work, and it’s going to be built upon the production version of TightURL, not the “new” version.
What this ends up meaning is I know there are several or more people waiting for an update for various reasons. I also know these aren’t the changes they’ve been waiting for either. But these are the changes I’ve been forced to implement first, and I feel it’s important to offer them to all the users of the TightURL source code.
So there will be another release of the production version (the copy running at tighturl.com) series of TightURL. I’ll have to write some crude admin pages to update various things in the database that are now being stored in the database instead of .php files, but I think this is the right thing to do.
After that however, I’m going to return focus to the “new” version, because that too, contains a lot of features/changes that I think it’s important for people to have, and it really is very difficult for me to do development on two different versions at the same time.
Courtesy of the audit log I’ve built into the production copy of TightURL in the last couple of weeks, I’ve discovered TightURL has been rejecting Twitter status URLs with an exclamation point in them.
The fix is to swap out the file tighturl.urlpattern.inc.php with the one found inside here.
Still working on the Policy system, which is evolving through work on the production copy of TightURL. Since I’m still hoping for it to be easy for people to write their own TightURL front-ends, and because it’s too risky to run a public URL shortener without the anti-abuse services, I’m integrating the Policy system into the back-end code as much as possible.
Although the Policy system is evolving under heavy development, I’ve already replaced the hard-coded filtering in the New Additions Report run by the killbot, and the self-referential checks with code that uses the new policy tables.
Still a bit disrupted by the move. More work has gone into the new database-based black/white listings. From now on, I’m going to refer to that as the Policy System, since that’s what I think of it as, and that’s what it really is.
The enhancements I’ve mentioned previously have worked so well, I was unaware of the latest abuse attack for several days. The system at present is too crude and incomplete to share yet, but I hope to share it as a continuation of the release version of TightURL. I don’t know if I’ve got a PHP 5.x dependency yet. I hope everyone’s running on PHP 5 by now…
After 78 days (seventy eight days!), dozens of inquiries and no response from them whatsoever, Spamhaus took tighturl.com off their DBL. You really don’t want to expose your mail to the mercy of Spamhaus. I recommend anyone running a public URL shortener to get a domain you only use for the URL shortener. The Spamhaus listing only made my users suffer since the only mail I send from tighturl.com is firstname.lastname@example.org and the support and developers mailing lists. I don’t think Spamhaus cares about the irony.
The enhanced (database-based) blocklisting functionality I installed prior to my move and short trip out of town turns out to have pretty much blocked all of my regular users. Or so I think. Maybe.
So now I’m adding an audit log. The need for this became apparent during initial development, but again, necessity forced it into being.
It’s apparent that the blocklisting is going to take up my next available block of time, but it’s important to get that done, and done correctly.
It’s also important to stress again, that running a public URL shortener requires constant monitoring. I’m trying to make it so that it’s as small a time commitment as possible, but if you want to run a URL shortener that accepts new URLs from the public you have to be reachable by e-mail most of the day, every single day, in case your URL shortener or your abuse address or your upstream provider needs to tell you something.