Bad Behavior 2.2.15

TightURL has been updated to Bad Behavior 2.2.15 .

I’ve been testing the TightURL plugin system code, and I’m relieved that it hasn’t needed much additional effort.  I hope to get back to working on TightURL soon.

Posted in Uncategorized | 1 Comment

Utility Code

I haven’t touched the TightURL code in a couple of months, it seems.  But I have been working on code that I need for TightURL.  It’s been slow going.  I would say based on the progress made over the last year, that I’m probably going to be releasing a “tenth anniversary edition” of TightURL sometime in late 2014.

Posted in Uncategorized | Leave a comment

AWStats plugin

I have created a plugin that adds the bit of JavaScript that AWStats needs to gather the extra information about things like screen sizes of visitors.

And then because the two plugins are similar, I tinkered a bit in the code for the Bad Behavior plugin.  That’s not done yet though.  I did get a quick response from the always responsive Michael Hampton about a wishlist request for Bad Behavior.  The next major version of Bad Behavior will allow me to control both the output and action of Bad Behavior.  This is going to be very helpful going forward.

Posted in Uncategorized | Tagged , , | Leave a comment

Development and the demise of the “404 Method” in TightURL

Work on TightURL springs partly from a general burst of maintenance and development going on in my life right now.  Related in a minor way to the work on TightURL, I’ve been doing some equally long delayed work on my own blocklisting infrastructure.

Work on this code revealed a pretty bad bug in either Apache or PHP.  Which one seems more likely to you?  Yeah, let’s just call it a PHP bug and save ourselves the time.  The bug is that PHP isn’t returning a proper $_REQUEST, $_GET, or $_POST reliably when using the “404 method” whereby our script intercepts the 404 error, and then takes action and changes the code as appropriate.

While I could probably work around this the same way you work around PHP’s failure to present $_PUT, it seems like a good time to retire the venerable “404 method” in TightURL, since the functionality it relies upon is apparently not well tested or used.

To use TightURL with similar functionality (“pretty URLs”) going forward, you’ll need to do URL rewriting  or aliasing or script aliasing in your web server.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Rate-limiting, REST Hooks, and the API

Thoughts

In a bit of a change of mind, I’ve decided now that an API is an abuse control measure, not an abuse-enabling method.  This bodes well for delivery of the API with the first release of the re-written TightURL.

REST Hooks (and how they enhance site reliability and performance)

I came to this conclusion during consideration of REST Hooks, an implementation of a method for pushing data via REST instead of polling.  I will be implementing something along the lines of REST Hooks in the TightURL API.  Concurrently, the TightURL UI will become rate-limited, forcing automated bulk additions to go through the API.  In turn, at least certain parts of the API will require an API key.  The use of an API key allows TightURL site operators much better control over how users use the service.

The primary motivation of REST Hooks is to eliminate the vast waste associated with polling.  If you eliminate the necessity of making all those polling calls, you can drastically reduce the number of calls any given API key is allowed to make.

Rate Limiting

The rate limiting in TightURL will (likely) revolve around four totals:

  1. Usage of the User Interface (web page)
  2. API calls to add a URI
  3. API calls to lookup a TightURL ID
  4. API calls for everything other than items 2 and 3.
Posted in Uncategorized | Tagged , , , , | Leave a comment

Anti-abuse progress/plugins

Since last update, I’ve been working on the anti-abuse system in TightURL.  Or more accurately, stripping it out of TightURL, and then adding it back in as plugins.

As a result of this kind of work, I’ve also started adding some additional anti-abuse plugins.  Keeping in mind that I may drop some (if they’re not effective) and add others, for now, the list of anti-abuse plugins looks like this:

  • SURBL
  • URIBL
  • Bad Behavior
  • blocklist.de
  • AHBL
  • Google Safe Browsing

The DNS-based blocklist plugins (SURBL, URIBL, AHBL, blocklist.de) are in fact all the same plugin template with different configuration settings.  The functionality that actually checks DNS-based blocklists remains internal to TightURL, making it simple to add new DNS-based blocklisting services to TightURL by copying the plugin template and changing a few settings at the top.

Bad Behavior has been part of TightURL for a long time, but the Google Safe Browsing service is a new addition to our anti-abuse arsenal.   You’ll need an API key for each of these plugins to get the most out of them (Bad Behavior benefits from a Http:BL key).

Posted in Uncategorized | Tagged | Leave a comment

DNSBLs

It hadn’t seemed important when TightURL was developed, and thus support for DNSBLs had never been added until now.    DNSBL checking is a great deal simpler than URIBL checking is, so adding support for these is actually a subset of URIBL support.

Posted in Uncategorized | Tagged | Leave a comment

URIBLs [updated]

Haven’t done much since I last updated here.  Sat down to look at the code though, and I’ve pretty well decided how URIBLs will be handled going forward.

I had said here a few dsays ago that you’d be able to add new URIBLs via config file changes.  Upon reflection, this isn’t workable for me, and it would have been bad for site admins.

URI blocklists all pretty much follow the same general technical rules.  The primary difference is the list of domains where you’re supposed to check the domain name at a finer-grained level, and whether or not you should check IP address domains in URLs.  Note the emphasis.  Blocklist operators have a long history of violating Wil Wheaton’s Law when they shut down, and some violate it as a matter of standard operating procedure.  So it’s important when you add a new URI blocklist, that you know whether or not to check IP address domains, and account for that when you use that list.

To make it easy to add new URIBLs to TightURL, I’m going to provide a plugin template that site admins will be able to edit and then activate.  Rather than being a model for how templates should be written this template will be designed to be easy to edit for the purpose of adding new URIBLs.

I’m also adding a special hook for URIBLs, to minimize the amount of code required in the plugin, since URIBL checks can be sufficiently abstracted.

Posted in Uncategorized | Tagged , , | Leave a comment

API location

If anyone still follows this blog, I’d like to know your thoughts on where to put the API in TightURL.

  1. http://example.com/api
  2. http://api.example.com

My guess is the main argument for choice 2 is that it makes scaling easier, which shouldn’t be a concern for TightURL users, none of whom are to my knowledge running a major URL shortening service.*

The main argument in favor of choice 1, as far as I’m concerned, is that it keeps with an original design principle in TightURL, that all URLs on a TightURL site branch off the web root.  As it turns out, this seems to be a design principle in REST as well.

Thoughts?

[*] TightURL has for some time, been developed under the belief that the demonstration/reference implementation copy running at tighturl.com is the one with the most traffic, and that at any given time, if TightURL is able to handle the load at tighturl.com, then it should handle the load for all the other users as well.

Posted in Uncategorized | Tagged , | Leave a comment

Progress Report

A quick status update.  Progress has been made in the following areas:

  • Templates: The template system has been re-written to accommodate the plugin system; unless something unexpected turns up, this is now finished.
  • API: The first code ever, has been added for the (REST) API.  Everything about it is still undefined, but the most requested feature is now underway.

No progress on the second most requested feature, localization.  I have been looking into it this development cycle, but I suspect this is going to be among the last things that gets in.  I’m starting to write object-0riented code, and I think the localization will be easier to get done via OO code, but I’m not quite ready to deal with that yet.

Posted in Uncategorized | Tagged , , | Leave a comment