I haven’t mentioned the plugin system for a while, because I’ve been working on the production version of TightURL rather than the “next” version. However, I’m already starting to use callbacks in the anti-abuse system, which amounts to nascent support for plugins. The first place this is showing up is in the URIBL checks. Since neither SURBL nor URIBL.COM has a stable API (the querying method changes, in URIBL.COM’s case, in undocumented ways in realtime *sigh*) I’m going to implement URIBL support through plugins. This will allow users to upgrade their URIBL plugins as needed without upgrading the rest of TightURL, and it will allow users to add support for additional URIBLs through their own or 3rd party plugins.
Since I point interested users to this devblog, I wanted to ping the blog and let everyone know I’ve been working on TightURL every day for the last couple of weeks. I haven’t been able to test any of the new code yet though, because I’m still changing it and in so many places it’s not practical to try and actually run it.
The good news is, I think I’m getting close to having a settled way to handle the anti-abuse system. I’m still getting useful data from the last set of changes I put into production, and I’m trying to incorporate lessons drawn from that abuse data into the anti-abuse system. As I think I mentioned before, it’s currently very crude and it tends to over-block things. It needs to be more nuanced before I can trust it myself, let alone share it.
I’m still ripping the anti-abuse code to shreds and putting it back together again, over and over. Eventually when I get it right and done, it’s going to represent a lot of work, and it’s going to be built upon the production version of TightURL, not the “new” version.
What this ends up meaning is I know there are several or more people waiting for an update for various reasons. I also know these aren’t the changes they’ve been waiting for either. But these are the changes I’ve been forced to implement first, and I feel it’s important to offer them to all the users of the TightURL source code.
So there will be another release of the production version (the copy running at tighturl.com) series of TightURL. I’ll have to write some crude admin pages to update various things in the database that are now being stored in the database instead of .php files, but I think this is the right thing to do.
After that however, I’m going to return focus to the “new” version, because that too, contains a lot of features/changes that I think it’s important for people to have, and it really is very difficult for me to do development on two different versions at the same time.
Courtesy of the audit log I’ve built into the production copy of TightURL in the last couple of weeks, I’ve discovered TightURL has been rejecting Twitter status URLs with an exclamation point in them.
The fix is to swap out the file tighturl.urlpattern.inc.php with the one found inside here.
Still working on the Policy system, which is evolving through work on the production copy of TightURL. Since I’m still hoping for it to be easy for people to write their own TightURL front-ends, and because it’s too risky to run a public URL shortener without the anti-abuse services, I’m integrating the Policy system into the back-end code as much as possible.
Although the Policy system is evolving under heavy development, I’ve already replaced the hard-coded filtering in the New Additions Report run by the killbot, and the self-referential checks with code that uses the new policy tables.
Still a bit disrupted by the move. More work has gone into the new database-based black/white listings. From now on, I’m going to refer to that as the Policy System, since that’s what I think of it as, and that’s what it really is.
The enhancements I’ve mentioned previously have worked so well, I was unaware of the latest abuse attack for several days. The system at present is too crude and incomplete to share yet, but I hope to share it as a continuation of the release version of TightURL. I don’t know if I’ve got a PHP 5.x dependency yet. I hope everyone’s running on PHP 5 by now…
After 78 days (seventy eight days!), dozens of inquiries and no response from them whatsoever, Spamhaus took tighturl.com off their DBL. You really don’t want to expose your mail to the mercy of Spamhaus. I recommend anyone running a public URL shortener to get a domain you only use for the URL shortener. The Spamhaus listing only made my users suffer since the only mail I send from tighturl.com is email@example.com and the support and developers mailing lists. I don’t think Spamhaus cares about the irony.
The enhanced (database-based) blocklisting functionality I installed prior to my move and short trip out of town turns out to have pretty much blocked all of my regular users. Or so I think. Maybe.
So now I’m adding an audit log. The need for this became apparent during initial development, but again, necessity forced it into being.
It’s apparent that the blocklisting is going to take up my next available block of time, but it’s important to get that done, and done correctly.
It’s also important to stress again, that running a public URL shortener requires constant monitoring. I’m trying to make it so that it’s as small a time commitment as possible, but if you want to run a URL shortener that accepts new URLs from the public you have to be reachable by e-mail most of the day, every single day, in case your URL shortener or your abuse address or your upstream provider needs to tell you something.
I have moved (again). This is very disruptive, and thus I haven’t done a lick of work on TightURL since.
That doesn’t mean I didn’t do anything before I moved though. I have finally gotten into the thick of things as far as moving the blocklist data out of PHP and into MySQL is concerned. As is so often the case with TightURL, abuse taking place over at tighturl.com required immediate action, and thus the beginnings of the new, much more sophisticated blocklisting system were laid down in the production version. Unfortunately that also means replicating that code in the “new” version and the “next” (based on the production code) version. I’ve got it into the “new” version, but not yet into “next”.
At this time, I have no UI for the black and white listing functionality. I will get that into the “new” version. Whether or not it gets into the “next” version is an open question right now.
Bad Behavior 2.2.3
Anyone who upgraded their Bad Behavior using the updated Bad Behavior for TightURL connector should *NOT* upgrade to Bad Behavior 2.2.3 . Unfortunately 2.2.3 contains a very rare API change that I will have to make in the connector before you can upgrade. I’ll announce it when this is ready.
UPDATE: I read the update notice for Bad Behavior a little too quickly. You can continue upgrading Bad Behavior.
For those interested in an API, these are the latest hopes: http://luracast.com/products/restler/
I only just discovered both of these, but either looks promising.
Since Fat Free Framework shows me procedural PHP examples, that’s a plus in its favor.
However, since we’re probably going to need an API-key based system, I’m going to need to get an Admin section up that includes user accounts and user profiles. That also ties into the question of templates. I really want to make a final decision about a new template system (or not, and which to use if so) before then so the admin section can be templated also.
Fat Free Framework as it happens, has a template system. That said, I don’t want to make TightURL itself dependent on any framework, even if it’s tiny.
Which brings up another matter… I haven’t decided how to connect the API to TightURL. My original thought was to keep everything in the same namespace, such that you would have something like:
One of the original intents behind TightURL was for people to run small-scale and/or private URL shorteners. The first approach probably works best for those people since they may not be able to add subdomains. The second approach allows you to split API traffic off from the main web site.
Although I like the “cleanness” of the first approach (and that’s an entirely subjective opinion), TightURL can no longer be run on the simplest hosting setups. It’s going to require PHP 5.3 or newer, and it already needs to be able to do DNS lookups and open network connections to fetch the contents of URLs. This precludes really low end or free hosting. I’m not exactly bothered that you can’t run TightURL on free hosting, given that virtually all the URLs submitted to tighturl.com from free hosting services are abusive links.
Work slowed a bit due to illness. It’s going to slow again as I am moving yet again to a new home.
In the meantime, I am going to be porting the TightURL interface for Bad Behavior to the Bad Behavior 2.2 series. This will be made available to all TightURL users (current version and “next major version”).
EDIT (2012-03-02 21:58 EST): Now available!