September 2004

Email is one of the biggest advantages of the internet. It’s one of the most prolific advances in global communication the world has seen since the invention of the telephone. It’s liberating and very convenient to be able to send of a virtual letter and know that it’ll be received almost immediately.

Email is one technology that hasn’t really changed since it’s conception. From the moment Ray Tomlinson finished his neat little hack in 1971 to allow one to send a message to another computer right up until this morning when I downloaded my daily invites to porn mailing lists and herbal enlargement pills, email has been a stalwart in the digital world.

The big question is simply “Is it time for a change?”. I’m not just talking about spam (I’ve done enough of that on here), I’m talking about several key flaws in the email concept. Key flaws which can destroy trust in the medium. Experienced ‘netters (I wonder if that’ll catch on) know how to spot a fake email from a distance of 100 pixels, I mean paces - but the net-novice isn’t so blessed.

One of the biggest flaws in email is the reliant on trust. Trust was a big factor of the early internet. With geeks throwing bits and bytes at each other’s modems there was no reason to question the validity of the information. When you said that the email came from “bob@isp.com” you believed it - and why shouldn’t you? There was no need to ensure that it actually came from good ol’ Bob.

Nowadays it’s a different story. You can send an email with any “from” field and it’ll appear in your receipients mail box from whomever you choose. There’s no way to actually check this is valid and there’s no way to stop someone sending emails out in your name. This can lead to some potentially difficult situations.

My email address has been cloned and used millions of times. It’s been used to tempt people into downloading the .exe attachment and it’s been used to tempt people to purchase a porn subscription. That’s midly irritating because (A) some people may actually believe I secretly run adult websites and have a hankering for viruses; (B) I get all the mail failed, mail bounced, autoresponders; (C) trust is broken with that email address and (D) my email address is reported to spamcop and I can no longer rely on it to send emails.

That’s bad enough - but how many people have been convinced that the email address from “noreply@paypal.com” which requires them to provide their PayPal username and password is genuine? Quite a few. How many people have been convinced that the email from “admin@somebank.com” which requires them to confirm their credit card number and PIN is genuine? Even more if you read the news.

Even worse is the ability to chip away at a companies reputation. At some point over the weekend a mass mailer was sent out inviting people to join a porn site with the from address of our company. Now, of course, any net-savvy citizen could run a WHOIS on the email header, but that didn’t stop a flood of emails from angry ‘net users demaning to be taken off our “smutty” email list.

As usual for this blog, I don’t have any answers - just a bunch of questions. Some will say that it’s no different in real life. “Con” men phone up people all the time an claim to be someone they’re not. This is true but people are usually more skeptical of this as the media has made them aware that this threat exists. So do we need more media to promote “safe” emailing or do we need a method that doesn’t allow my email address to be used unless it comes from my IP address? That’s a nice dream but until the advent of static IP addresses, it’s unrealistic.

This is where a dedicated ‘net user ID would come in very handy and virtually (no pun intended) solve all these problems overnight.

The rumour machine is in full swing over at the Google camp.

The New York Post published an article about the speculation that Google is recruiting some of the best brains on the internet (explains why I didn’t get asked) - specifically “the” java guy and several Internet Explorer developers. They can afford it to as they are some $1.67 billion richer from their recent stock market floatation.

This is either really good news or really bad news depending on how it all pans out.

The initial thought is that the browser will rival Mozilla (Firefox, Camino, et all) and Internet Explorer. The question is very simply this “Is there room for another one?”. Google has a very strong brand which people are familiar with, but do we need another web browser? I don’t really think we do. Especially from a developer’s point of view. The last thing we need is yet another engine with its own quirks for us to write work arounds for.

I doubt they are going to base it on another browser, such as Mozilla as they have recruited four Internet Explorer developers. That doesn’t make much sense. I figure they are going to start over from scratch. This could be bad news. Lets go the whole “tin-foil-wearing” hog here and speculate that if they have their own browser code they can use proprietry methods and if they can use proprietry methods, they can have special code that won’t allow other browsers to access the Google.com search engine and the Gmail.com web-mail software.

This will give them a monopoly to rival Microsoft. Without questions 90% of us couldn’t live without Google on a daily basis and they have specifically pitched their Gmail system to be just as popular with it’s generous space allowance and informal service terms. Should Mozilla and Internet Explorer start to worry?

The recruitment of Joshua Bloch, one of the main developers of Java is an interesting choice. I’m guessing they are planning on having a very tightly integrated java engine within their browser which is something that IE definitely lacks and Mozilla could be better at. It’s either that or they are planning on writing a web-based browser in Java to encapsulate their online services. An easier way to launch Google and Gmail is a welcome thought.

Which is more likely: A java web-based browsing system or a brand new downloadable web browser?

Who knows, but it’s going to be interesting finding out. Of course, this could all be another marketing gag from those Googlers to ensure that Google gets written about in all the major newspapers and in thousands of blogs around the world.

No, not the baked chocolate chip kind. They will never suck. Ever. If the entire world was sucked into a black hole, the only thing left would be chocolate chip cookies. Because they never suck.

I mean the digital variety that don’t sound even as bit as tasty. Sometimes they work, sometimes they don’t Sometimes browsers accept them, sometimes they don’t. Sometimes javascript creates them, sometimes javascript doesn’t create them. Sometimes a blank value will delete them, sometimes it won’t. All in all it’s smash-me-over-the-head fantastic. I love them. They are brilliant. They adhere very closely to what the internet calls “standards” in that they dance around and mock you and don’t do what they are told.

Actually it’s probably my lously programming but if a bad workman can blame his tools, a bad programmer can find someone or something to blame. Lets blame Internet Explorer.

The only thing that sucks more than cookies are deadlines. I think they are called deadlines because it’s like a finishing line strewn with dead developers. If you stand on the deadline, you can hear faint moaning. That could be the impatient users though. Although they usually come armed with projectiles and pitchforks.

Div’s are annoying too. They float all over the place until you use “float” which is when they deny knowing what you mean and stick 400px to the left of where you want them to go. Would anyone notice if I used a table? Probably. Still, all in the name of advancement. I blame Internet Explorer.

And today is a good day.

I am in the middle of writing a new email system to replace the current email system.

It’s guaranteed spam free, virus free and it’s never wrong. I’m calling it Femail.

On the surface it works just like an ordinary email system. The basic concept is the same. The real beauty of the system is in the Femail logic processing.

Becaue the Femail logic processing system is widly dependent on factors that are not documented, it’s almost impossible to create an email that will get through the spam filter. The rules are constantly changing too, without documentation, making it even harder to send a virus through the system. As there is no consistent logic and no documentation, spam is a thing of the past.

Unfortunately, there are a few downsides to the system which are very hard to debug. It seems very temperamental and not entirely user friendly. I’ve narrowed it down to the following bugs:

#1: It keeps deleting conversations so you’ve no proof they existed. Trying to find proof is futile.
#2: It keeps on sending the email over and over again until it gets a reply from the recipient.
#3: Randomly re-arranges the user interface without warning.
#4: If it’s in “non-verbose” mode, you’re unable to log in. We like to call this “not talking” mode around the office.
#5: The message filtering system is odd. It filters not according to subject or date but color, writing style and “like-ability”.
#6: Ocassionaly resents being told what to do and refuses to do it until you ask again nicely.
#7: Refuses to empty the trash folder.

We do feel that it’s worth developing further though. There are some good key features such as the limitless memory; it’s able to pull conversations from disk based storage at will and is very adept at emoticon usage - even when you don’t want to show them.

I’ve attached a flow diagram to better explain how the logic processing works.

(Click image to enlarge)

There aren’t many brave developers out there anymore. We’re too scared to break convention - even if it’s begging to be broken - because we don’t want to be the pioneers that start a new trend. We only hear about the successful pioneers; the ones that really made a difference. We rarely hear about the ones that failed along the way.

Typically, the average program user will be used to a way of working. Despite the differences between Macintosh and Windows, they all pretty much do the same thing. You double click in a folder and another one opens. You click on the “close window” widget and the window closes. You right click (yes! even us mac owners can have two mouse buttons) and a little menu pops up. This means that even when we use a different operating system it behaves in pretty much the same way. Just like every car has the same basic functionality operated by the same basic interface.

<Digression>
I remember reading a while ago about an inventor that combined the accelerator and the brake pedal of a car into one unit to make driving more intuitive. He made it so that if you put your foot on the pedal and used your heel as a hinge, you could pivot the pedal towards the floor to accelerate the car. To use the brake, you simple pushed the pedal downwards. Very clever and those who test drove it agreed that it made perfect sense and it made braking quicker. It’s unlikely to be used simply because millions of us are used to using the more standard layout.
&lt/ Digression>

Even when we’re faced with the perfect opportunity to actually change a commonly used but bad piece of user interface we hesitate. Whilst “if it ain’t broke, don’t fix it” springs to mind; it’s more a case of “even if it’s broke but commonly used and accepted, don’t fix it”.

For us humble web developers we’re often at a crossroad trying to decide whether to pioneer a new GUI for a familiar genre and hope-to-god it works or take the more travelled and safer path. The web is a great medium for trying new things out due to it’s instant and easily changeable medium - but with established and well used products it’s hard to throw caution to the wind and try something different.

I think that to truly create something fresh and new, you’re better off not knowing what the competition do and you’re probably better off not having any contact with other products that fit into the genre you’re trying to work on. Either that or you have to have a very clear vision of what you want to achieve, have a lot of experience in your field and have a burning desire to make things better.

I’ve just discovered an amazing new tool!

It’s ideal for all your organizational and development needs. It sounds too good to be true, but it is! It is great for leaving memos to other people, great for leaving memos to yourself, great for reminders, ideal for note taking, perfect for roughing out designs and ideal for problem solving. It’s also great for basic mathematics.

“Where can I buy it?”, “How much does it cost?”, “WHERE CAN I DOWNLOAD IT FROM???” I hear you cry!

You don’t need to purchase it or download it.

It’s a bit of paper and a pencil.

To my amazement, this new tool came in very handy today and I was blown away by its ease of use. Not only was it very easy to learn, it was very intiuative and one of the most portable devices I’ve seen for a long time. It came into its own today when I was stuck on a little design / code / structure problem for the CMS system I’m developing. With it’s easy learning curve, I was immediately able to pick it up and work out my problem in only a few minutes!

It was such a great experience, I just had to share it. Don’t believe me? I’ve attached PROOF on how it helped me resolve my problem in half the time and at 100% less cost than other methods I’ve used in the past.

(Click image to enlarge)

Unless you’ve been living in a cave or abandoned on a desert island, you’ll know that Google has cobbled together a rather nifty web-email system entitled “GMail” (geddit? Gmail - sounds like Email?)

Now, before anyone starts replying with comments about how “spooky” Gmail is and how EVIL they are for having the ability to electronically scan email: I don’t care.

I don’t care if Google scans the contents of my emails to present me with new “purchasing opportunities”. I don’t care if they compile a fact-file of my pathetic online life to be used in some EVIL marketing ploy. Quite frankly anyone naive enough to believe that their normal emails are secure and not read electronically is living in some pink plastic Barbie fantasy land.

If it’s un-encrypted assume that it’s being read by someone else. End of story.

Gmail doesn’t do anything more than your government has been doing / has the ability to do. If you value your conversations with Auntie Mildred then write her a letter or use an encrypted mail service. Just don’t whine on about how EVIL Google is. If you don’t like the it - don’t use it. It’s that simple.

Anyhoo, tinfoil hats aside - it looks like someone finally got around to spamming my Gmail account which leads me on to my favourite feature of Gmail - the ability to “preview” the message before opening it.

Let’s play: spot the SPAM.

(Click to enlarge - real emails removed from screenshot)

This blog started off as a two line entry about my pet hate for people starting bug reports off with “This isn’t a bug, but…”. However, as I ate my lunch I remembered some more irritating habits I’ve witnessed over my three million year tenure as a lowly programmer who has to deal with bug reports daily.

Ways to “bug” a developer in their bug tracker system.

Start a bug report with: “This isn’t a bug, but….” and then ramble on aimlessly about why you think feature X should be changed / added / removed. That’s the best way to get the developers attention.

If you get told your reported bug has already been fixed or that it’s not actually a bug - continue to argue for as long as you can hold out. The developers only wrote the software so what the heck do they know!

There are many more examples, but I’ve just depressed myself. For the sake of programmers and project managers everywhere, don’t do it!

Thanks to the internet being a wholly global affair, owning your own domain name might not be as straight forward as you’d hope.

Traditionally, people gravitate towards “.com” first out of habit and then use their own country’s top level domain second. Most businesses tend to register their own country’s domain because usually the .com is already taken and because their country’s domain - such as “.co.uk” - encapsulates that the business is a fairly local one. That loveable scamp Rob Manuel of B3ta.com fame hilariously registered a “.co.uk” address after a UK company went straight for the jugular and registered “IntroducingMonday.com” only. Rob managed to literally stick his fingers up at them with IntroducingMonday.co.uk.

It gets even more (conf|am)using when local goverments pitch in and try to get some action on the interweb by using their fancy “.gov” domain names.

Recently an enterprising chap called Thomas couldn’t believe his luck after the UK government launched their new site preparingforemergencies.gov.uk to act as a companion to their free information booklet. As the UK government had registered the “.gov.uk” address but not the “.co.uk” address, Thomas swooped in and created a parody site that probably gets more hits because of its address: preparingforemergencies.co.uk.

Personally, I think Thomas’ site is much more informative and useful than the governments attempt to patronise us even further. Here are some gems from the site:

So kids, the lesson here is this: register just about as many .TLDs as you can afford before someone else does.

As some of you may know, I use Macintosh to write my code and my text editor of choice is BBEdit. I’ve used BBEdit since version 6 and every major new release brings great enhancements to the software.

BBEdit are of the “if it ain’t broke, why fix it?” stable and it works for them. The core of BBEdit - including the interface - hasn’t changed much over the last few years and from someone who uses BBEdit for 8 hours+ a day 5+ days a week, this is a good thing as you can instantly get to grips with the new version and continue working without having to re-learn anything.

BBEdit 8 is the new release and it’s now in the must-have category if you’re a serious programmer. It uses the best bits of OS X now that it’s free from OS 9 - such as the window tray to enable you to effectively ‘tab’ several documents in one window. This an ingenious step and will definitely save me on some screen clutter. The new cursor-line highlight is a neat little addition too.

The search window has benefited from the addition of a window tray to allow you to select which document to search in without closing the search window and going back in on another document. It also allows you to select more than one document to search in at a time without setting up the batch search - ingenious!

I’m still having a little play around with it but I’m already blown away with this update and that rarely happens to me with updates to software. If you’ve got BBEdit 7 - update now and if you’re not using BBEdit - go purchase it now!

Here’s a little screen-shot (click it to enlarge!)

Being a web developer is often “hit-me-over-the-head-with-something-heavy” fantastic.

I mean, there are days when you would quite happily sell your computer for a one way ticket to the Gambian rain forests to live out your days swinging through the trees with the monkeys. Actually, that sounds pretty good. How much would you get for a slightly used iMac these days?

These are the days that people with faith call “testing”. You know, the ones where you wake up to find that a major new version of apache / PHP / MySQL / etc has been released and eager webhosts have immediately upgraded without testing backwards compatibility which forces their customers to ask for support on their previously working software. This often lands squarely at the feet of the development team who wrote the product.

Now, the web moves at a breakneck speed. Internet time is like dog years. I actually believe that for every hour you spend on the internet, you age seven hours. So in real life it could take years to develop a new application. Try doing that on the internet and by the time your finished, the scripting engine / database engine you’re developing with will be totally outdated. It’s hard enough as it is trying to develop for one platform when the next version of that platform is near ready for release.

This puts us firmly on the bleeding edge when we often don’t even want to be anywhere near it. As a programmer you have to try and work with what you have but always keep in mind what’s due out next. This can often lead to some very obfuscated and redundant code.

Then there are the new version blues. Those shaky x.0.1 releases that scare even the most hardened system administrators. Hands up who remembers the hilarious CPanel / MySQL 4.0.0 upgrade? It’s funny now - sure but for a few weeks if you listened carefully you could hear the rhythmic pounding of a million system adminstrators heads hitting their desks in frustration.

I write this on the cusp of the new PHP 5 release which promises exciting developments to our favourite scripting engine. It also brings a whole new bucket load of broken scripts, incompatibilies and support tickets. Throw in a few PHP bugs for good measure along with eager-to-please hosts upgrading without a care in the world and you’ve got a receipe for late nights.

Still, at least we don’t have that problem with perl anymore.

This leads on from yesterdays blog and touches on a very sore point for many web developers.

Full Disclosure
[This is where] full details of the vulnerability are disclosed to the public, often through Bugtraq or similar means. This must include disclosure of the details of the vulnerability (including how to detect and exploit it). More controversially, it may also involve release of sample code or an executable tool to exploit the problem.

That’s how the wonderful WikiPedia describes full disclosure which is a method of alerting developers to bugs and vulnerabilties in their code.

Before we explore why this is bad for web business, lets travel back in time a little. Way back in 1995 the web was a different place. There was growing interest for this new medium but the internet community seemed much more positive. Web sites were run by enthusiasts with a lot of technical knowledge and they relied on few scripts not written by them.
This was a time before PHP and before perl hit the big time so there wasn’t the selection of ‘upload, CHMOD and play’ scripts that we have today. This meant that there were few common components of a website. With no publicly available scripting and relatively crude server operating systems there wasn’t a lot to exploit.
Back in those dark days shareware, open source and freeware ruled the roost. Popular programs were managed and developed by hundreds of people with often no clear management or structure. Most exploits were found by enthusiasts who fixed the source code and gave the fix back into the open source pool.

The internet is a very different medium now. The market for pre-written scripts has exploded and nearly every single website will run a piece of software that is publicly available to download and the source code available to read. This makes searching for exploits a game. Rather than simply fix problems you come across these exploits are posted as some kind of trophy. Newsgroups that actively encourage posting of expoits are used by script kiddies to simply cause mischief.

Posting full details of the vulnerabilty and how to exploit vulnerabilties is bad news for everyone. Even when the vendor issues a fix, not everyone applies it immediately so the mischievious script kiddies just have to search for people still running old software - and with powerful search engines such as Google that’s not much of a challenge either.

Lets look at a quick example. If a vulnerabilty is found in Windows (the most common operating system of home computer users) that allows one to gain access to the computer - and this vulnerabilty is posted with full details it won’t take the average script kiddie long before they’re using thousands of computers owned by slow upgraders to launch a full scale DDoS attack on their chosen target.

I have no issue with people posting that they’ve found a weakness in the script publicly - but I don’t see the need to post HOW to exploit it. The open source movement isn’t as popular as it was and most popular scripts are owned by a single company with a very clear management structure. I don’t see that it’s of any value to post a full disclosure report on a vulnerability found in a commercially owned script. The only people that can fix and issue a fix are the software owners. These days most project managers worth their salt take security seriously and won’t delay in getting a fix ready so the “scare them into doing something” approach with full disclosure isn’t really relevant anymore.

The most common argument I hear is “Well, if you coded properly…”. That’s a good argument but it’s flawed. There are far more people trying to find a weakness in a script than people coding that script so it’s just a matter of time. Also, changes in server OS and scripting / database engines will make previously “safe” code unsafe. A good example of this is the recent upgrade in MySQL. SQL injection attacks were fairly rare and relied on very poor programming to work. Now that MySQL 4 supports “UNION” to join queries, suddenly every point at which data is used in an SQL query has to be examined. Software that was written for MySQL 3 didn’t have to and as such most programmers weren’t anal about ensuring correct data types. There’s no excuse for poor programming but you can excuse programmers for not foreseeing the future or not accounting for changes in software outside of their control.

I’ve seen a lot change with security over the last five years or so. When I started out, all you had to do was make sure that the flat-file database you were using didn’t allow anyone to tamper with the path so malicious users couldn’t read and write to otherwise unreachable files. Then, with the advances in database technology and the uptake of the popular PHP/MySQL bundle you had to watch out for SQL injections. Now XSS (cross site scripting) is probably the most common form of exploit. This relies on vulnerabilties in the code which allow the attacker to inject HTML into the normal output of a script to catch viewers cookies with then allow them to spoof this information to fool the script into logging them in as another user.

It seems that just as soon as programmers catch up on the latest techniques those pesky crackers are already onto the next method which will spread like wildfire across different software thanks to full disclosure reports.

What can we do about it? If you find an exploit please report the full disclosure privately to the company and then publish a limited disclosure report publicly. There’s no need to give script kiddies more material to work with. Make sure you keep your software on your home computer and website as up to date as you can and don’t rely on “it won’t happen to me” to keep you safe because invariably, it will happen to you. These days any monkey with a 486 and a subscription to Bugtraq has the ability and the tools to have a good go at hacking your website.