TightURL Development Blog

I recently added the Spamhaus DBL to TightURL’s blocklist checking code. Shortly after, I discovered tighturl.com is on the DBL. We’ve at about day 17, and at least a half dozen inquiries gone unanswered.

This will affect deliverability of the support and developer’s mailing lists. Most importantly, it reflects poorly on Spamhaus that it is run in such an irresponsible manner. The fight against abuse is taken far more seriously by me, the sole developer of TightURL than it is by Spamhaus, a business that takes in money and whose data is relied upon by so many.

I have been setting up my “new” development system. I’m going back to developing TightURL under Linux. I can’t say that there’s been any compatibility issues that came up while I was developing under Windows, so it is likely that TightURL will continue to be compatible with WAMP (Windows, Apache, MySQL, PHP) stacks.

EDIT: 24 days, no response from Spamhaus. Apparently the lights are on, but nobody’s home.

Just to update those who are interested in this sort of thing…

Work continues on a new release. This will be a major release that incorporates a number of things that needed to be done. How many of these things get done remains to be seen, but I’m trying not to get bogged down with too many things being done at the same time. So far, the changes of note and work under active development are:

• PHP 5.0 now required
• Black/White listing moved to the database
• Cron-like task scheduling and execution
• Scheduled new additions report
• Scheduled anti-abuse detector
• Abuse mail processing script

Things likely to be included:

• TightURL Library
• Localization
• New release and security notifications

Things that would be nice but aren’t yet likely to be in the next release:

• Improved templating
• Improved API

Input from users of both tighturl.com and the TightURL source code is welcome. The TightURL Library is an attempt to separate the front-end of TightURL from the back-end, making it easier to create your own URL shortening front-ends that use TightURL’s library to shorten URLs, handle abuse, etc.

I know many of you are interested in an improved API, but so far I haven’t found a simple way to do this or any way to do it that doesn’t require me to shoehorn TightURL into some framework that’s many times the size of TightURL’s code itself. I’m still hoping to figure out a simple way to add a REST API.

I’ve avoided doing anything to make TightURL require PHP 5 up until now, but PHP 5 is no longer “new” by any stretch of the imagination, and I need to make use of code that requires PHP 5.

So there you have it. The next major release of TightURL will require at least PHP 5.0. For the time being, I’m going to try not to force the required version up to PHP 5.3.

Although I’ve got an awful lot of other things in my life that I have to take care of, admittedly much of the latest delay is due to getting a little thrown off TightURL development by the discovery that I’d sank so much time into a non-working PHP cron system I found in Google. I can report now though that I’ve replaced the broken code with something that works correctly, and it passes tests.

I’m back to the point now where I was when I discovered the other code didn’t work. That means a little integration work and a little testing of the job processor, and I can finally get away from a major, if necessary, distraction.

The next thing I have to get back to work on is the blacklisting system. It’s been half-written since January, when I realized I was going to need the cron system for the blacklisting system, and stopped the blacklisting system development to quickly (ha ha ha) bang out the cron code.

Every time I write something to read e-mails, I forget about quoted printable. I have some kind of mental block about it. It is the bane of my e-mail reading existence. I got abuse mail from Engadget the other day. First I was surprised they use Google for mail. I don’t know why this still surprises me, but I find it hard to take a company seriously who does this with their mail. It tells me that what something’s perceived cost is, is really the only thing that matters to them. Second, the mail contained a Quoted Printable part. My poor regex was laid low by Quoted Printable.

What’s this in regard to? A future release of TightURL will include a script that analyzes inbound abuse complaints by e-mail and replies to the sender when appropriate. Sadly after realizing it was going to be a major undertaking, and looking around for acceptable PHP code to properly parse the complicated mish-mash that is our beloved Internet mail, I ended up with another class from PHPClasses, 80K of BSD licensed PHP code to parse an e-mail. It continues to bother me that frequently for the sake of something like parsing an e-mail, something which used to be pretty simple, I end up with a class or library that’s several times the size of the script that uses it.

I am going to use PHP Parse cron strings and compute schedules as my cron parser/calculator. At this point I’m not sure anyone’s got code that works, but this looks good so far. I’d have preferred to have GPL code so there would only be one license over the whole of TightURL, but the MIT license is not incompatible, so we’re good.

I should be able to write a job processing system to use it pretty quickly by re-using the code I wrote in the previous attempt.

Well, Pseudo-cron does not work correctly. At least not with PHP 5, if it ever worked correctly. It’s a shame I didn’t notice this in earlier testing, but I also can’t figure out how that code is supposed to work in the first place. So I’m ripping pseudo-cron out, and will resume writing my own as the whole point of using pseudo-cron was for the sake of using the scheduler contained within. Which seems to be horribly broken. *sigh* It really annoys me that it took me so long to notice. I can’t and won’t complain that there’s a bug or bugs in Pseudo-cron since I’m sure there’s bugs in TightURL too. I can and will kick myself for a while though, that I spent a month working with code I didn’t check carefully enough to see if it worked correctly.

After initially rejecting it, I am now adapting and possibly in the process of re-writing pseudocron to be run by the TightURL maintenance task scheduler. This will let me use my own code as the “master scheduler”, but whose only task is to decide when to run the modified pseudocron subsystem.

The pseudocron system will be responsible for running the anti-abuse tasks and any reports.

I hope this will all be efficient enough.

UPDATE: This has proved to be more time consuming than I anticipated, since converting the script to use a database has required me to understand how it works in much more detail than I would have liked. Still, I think it was the right decision since I will be able to let users have more fine-grained control over when their jobs (reports mostly) run.

Nobody ever gives me any feedback about TightURL except when they’re having trouble getting it to run.  So in order to get useful information about how well it works, I run a public copy at http://tighturl.com and that’s how I learn everything I know about how TightURL performs out in the world.  One of the reasons I started a devblog (development blog) was to try and get some more input from my [code] users.

In that vein, how's your abuse? Mine's been showing the first serious signs of bot abuse since summer of 2010.  I'm also finding that Kaspersky Internet Security flags a lot of the abusive URLs.  So the next enhancement to the TightURL anti-abuse system will probably be to add additional data sources to the URL checking routines.

Does anyone have experience with Google Safe Browsing?  It looks like that's probably the only free data available that I'm not already using via SURBL.