Entries by Jonathan Street

Following my initial discussion on using geo-targeting to predict timezones I've finally found time to play around with some of these ideas.

The idea

Simplify the user experience by predicting the timezone someone is in and auto-populating a registration form accordingly.

The solution

At the time I wasn't really looking for any solutions other than matching an IP address up to an actual location. This is certainly possible and I'll show you how below but first I'll highlight the alternative. If you are reasonably confident that the timezone will be set correctly on the users computer it is possible to access that value via javascript.

GeoIP to Timezone

If you want to avoid relying on javascript then you still have some options.

Maxmind: The PEAR pacakage supporting maxminds geoip databases doesn't support timezone lookup but the script offered by maxmind directly does support timezone lookup. The usage seems a little convoluted but it is there.

IP2Location: The timezone information is integrated directly in the database once you reach 'DB11', their 11th database offering. The usage should be straightforward. There is only one drawback - DB11 costs $649/year. Personally this is more than I would be willing to spend but their client list is impressive so if this is within your budget spread the moolah and use my affiliate link .

Free solution: There are still a few options here. Although I haven't tried it I believe the maxmind code should work with their GeoLite products. Alternatively once you have a location, either using maxminds free databases or hostip.info you could try getting to a timezone using the 'world time engine' class available from phpclasses.org. The drawback with this approach is that even after the database lookup you still need to query two web services. That is going to introduce a significant lack in response time.

My preference

Although I started off looking at GeoIP services I would much prefer to be able to tackle this problem using javascript. It's not an ideal solution, some people will have their timezone set incorrectly while others will have javascript disabled but on balance I think it is good enough. This is the icing on the cake rather than core functionality after all.

I've recently been playing around with geoip databases looking at implementing the type of timezone prediction I previously discussed. I'll be writing a blog post to cover that in the next week or so but first wanted to mention something which has been puzzling me.

Why don't geoip services accept feedback?

A good example of accepting feedback is Akismet. There you can report wrongly missed messages as spam and messages incorrectly labelled as spam as ham. I couldn't find an official statement on their accuracy but figures as high as 99.9% are mentioned. Defensio, a direct competitor, gives an accuracy rate of 99.77%. With my own experience using akismet an error rate of 1 in every 1000 seems about right.

In contrast, maxmind who is, to my knowledge, the commercial leader in geoip, states their accuracy as,

Over 99% accurate on a country level, 90% accurate on a state level, 81% accurate for the US within a 25 mile radius.

I don't have enough information to query their accuracy in the US but for the rest of the world it falls significantly from personal experience.

With feedback from their users I think this accuracy could be substantially increased.

Hostip.info

Hostip.info, which makes their database freely available does accept feedback from users. It's a simple process with only three screens and a CAPTCHA to get past. Oh, you also need cookies enabled. Not many people are going to use this. What is really needed is an automated process and this system is set up to prevent automated processes.

My 2 cents

Geoip services have the potential to more efficiently utilise their user communities than even the anti-spam services can. Internet users regularly submit their location on forums, social networks and whenever they make a purchase. These are all situations where predictions based on geoip lookups can create a more pleasant user experience. If the user doesn't change the predicted location then all is well. If the user does change the location though then that information should be sent back to the geoip service so the database can be updated.

Analysis of hostnames will only get you so far. To really boost the accuracy you need human reviewers. If you can convert ordinary internet users into reviewers without inconveniencing them then that's even better.

This blog has been quiet for a little over a month now as 'real world' events have consumed all my spare time. Overall it has been a month well spent.

PEAR bug triage

I was able to set aside a little time one weekend for the inaugural PEAR bug triage event. I learnt a lot even if I didn't really achieve much. Definitely something I want to set aside some more time for in the (near) future.

Edinburgh International Science Festival

I spent a couple of days (attempting) to exhaust those perpetual motion machines most commonly known as children. I believe I failed entirely. The department I'm based in ran a stall focusing on the heart and healthy living. The experience left me exhausted but with renewed hope that the next generation are not entirely the devils the media would have us believe.

Munich

I spent a long weekend in Munich. It seemed like a really nice city. Best of all the trip was free which always adds an extra delight to any experience.

Tai Chi seminar

For the second time I went to a weekend seminar on Tai Chi. I've been attending weekly classes for ~ 18 months now but find the focus of these weekend seminars to be valuable. Draining but valuable.

My next project

Over the past few days I've finally found time to start work on a new project I've been thinking about for a couple of months.

A new start

With a new project starting I feel it's a good time to reflect on experiences with past projects and take a look at what are currently considered to be 'best practices'.

Frameworks

The first decision I made was to take a closer look at the plethora of frameworks which have sprung up over the past year or so. I decided to give zend framework a try and so far have found it to offer what I need. My main concern was that it would be too inflexible. I've wanted to deviate from the default three or four times now in what I would consider to be non-trivial ways and found that by hitting my code base with the manual a couple of times it would eventually yield to my will. There is certainly a learning curve to master but on balance I believe the benefits will be worth it.

Source Control

Previously my source control has been embarrassingly bad. I still have backups which contain backups which contain old code archives to remind me. I suspect there is useful code in there somewhere but it has reached the point now where finding it is so demoralising that I prefer to pretend it doesn't exist and start again.

Recently I've been playing with subversion. It's definitely something I want to continue using. Currently I have it running from a slightly flaky old computer I turn on when needed. It still isn't really an ideal solution. What I really want is an always on service I can connect to from anywhere. To that end I've been looking around for subversion hosting.

Following the suggestions in an year old post by Jonathan Snook I've been comparing the offerings available. I've put together an excel spreadsheet which you can download here or view in Google docs here.

Hopefully that will save someone a little work. Personally I'm edging towards assembla which makes 500 Mb of svn space available for free and seems reasonably priced should my needs grow. I would be delighted to hear from anyone who currently uses them or has in the past.

My 'day job' has nothing to do with PHP. It has nothing to do with any form of programming. I graduated in 2006 with a degree in Biochemistry and went on to do a MSc and now PhD in cardiovascular biology. The closest most of my colleagues come to programming is a formula in an Excel spreadsheet.

It was actually Excel which prompted this post. Yesterday I was analysing some data and bemoaning the poor search functionality that Excel makes available. I had already expanded the small set of experimental data I had with some values pulled from a web service using a quickly hacked together PHP script and it got me to wondering how much better things could be if I just stuck with PHP.

Where's the science?

This train of thought led on to whether PHP has been used all that often for scientific projects. There is an accelerating trend in Biology to make data and tools available via web interfaces. In my opinion this is an environment where PHP excels and yet all the literature I've seen discussing the development of these services uses Perl or occasionally Java.

Searching a little harder for PHP projects yields an equally depressing outlook. In PEAR Jesus Castagnetto released the Science_Chemistry and Math_Stats packages back in 2003. For my purposes though the Chemistry package is a little too 'chemical' and the stats package is a little too basic. In sourceforge there is a package named BioPHP which looks promising but again there has been no activity since 2003. A lot has happened since then.

Biology is increasingly data generative. There is going to be a steadily increasing need for tools to analyse all this data. These are likely to be centralised and made available via web interfaces.

Anyone out there?

I suspect I'm going to be increasingly creating automated solutions to remove some of the repetition involved in processing the, relatively, small amounts of data that I generate. A PHP toolkit able to leverage the latest online databases and perform 'advanced' statistics would be immensely valuable.

So my question is this. Is anyone out there using PHP in a scientific environment? Are there resources available which I've missed?

The web service for fetching contacts from MSN messenger / Windows Live Messenger has been hit by a series of problems over the past few days. First I hit the bandwidth limit for the account used to host the service. Next, the server used was hit by a DDoS attack. Finally after all that was sorted out the bandwidth limit again caused problems.

Hopefully everything should now be back to normal and the service will be stable.

As always, if you do have any problems, my contact details are available under the 'About' tab at the top of the page.

I have a love hate relationship with geo-targeting. The web wasn't designed with making it easy to get the geographical location of connected computers in mind. A users geographical location is interesting and potentially valuable though and so methods have been developed to make it (almost) possible.

These methods typically involve something akin to a brute force attack. Figure out where enough IP addresses have been assigned and you can get a good idea of where a user is from their IP address. Other methods involve identifying the computers through which they are communicating with you and assuming the user is in the surrounding geographical area. Neither method is perfect but in the majority of cases you can know which country a user is in with reasonable accuracy.

What I hate about geo-targeting is how some sites think they can locate you more accurately than the country you are in. Maxmind, which is probably the commercial leader, thinks it can guess your location to your nearest city with an accuracy of 81% in the US. Outside of the US I suspect this drops considerably. I'm seeing fewer sites than I once did trying to tell me where I'm connecting to the internet from (and getting it wrong) so I'll skip forward to what I love about geo-targeting.

Continue reading "Geotargeting in forms"

Over a full month after the fact I present my summary of the BarcampScotland2008 event.

The event was split over two days. It kicked off on the Friday evening (1st Feb) in the main room of Alison House at the School of Architecture. It was just the one room with a decent communal feeling. I think most people were holding back with their presentations for the following day. Despite this two presentations did take place. James Littlejohn made the first presentation after the welcome session and talked about data portability.



It was a good summary of the current situation. I also happen to agree with most of his positions. He has taken the decision of making his homepage the hub of his social network. Although I think he has perhaps taken things a little too far my main criticism is the implementation. Looking at the site it took me about 20 seconds to figure out that aboyne is where he is based and not his surname. I couldn't find his surname anywhere on his homepage. After navigating around for a while I found it in the byline for his blog. It wasn't an easy process. I highly doubt that data was machine readable despite the importance he attached to this during his talk.

Ewan Spence was next up with an improvised talk he largely made up on the spot. This rapidly migrated to a conversation with some interesting points raised.

Following on from this Dave McClure set a small competition going. A excessive number of random words were gathered from the audience which then broke up into 5 groups to brainstorm company ideas around any pair of words. Somehow the team I was in won with sexydyslexia.com, a couple that takes standard prose and converts it into netspeak and vice-versa. I notice that the domain name is still available so although rated as the best apparently no one in the audience wanted to run with it.

All the details on the second day after the jump . . .

Continue reading "BarCampScotland2008 Roundup"

The second BarCampScotland event finished yesterday. This was the first BarCamp event I've attended and I have to say I was impressed.

Almost without exception I would describe the talks I attended as interesting or very interesting. I plan to post summaries and link to slides where possible in a later post.

The facilities were also quite impressive. Appleton Tower always looks to me to be a rather drab building from the outside but the concourse, where BarCamp was based, was a deceptively impressive space and certainly met our needs. Having said that I don't know whether they turn the heating off on the weekend but it was definitely chilly.

There were five large (easily seating 100+ people) lecture theatres available on two levels which were again well equipped. For the number of laptops I saw out on display the power sockets beneath every other seat would have been invaluable. It was only in the penultimate session I realised they were there but as that was when I started being concerned about power it all worked out well.

The Slides

I spoke in the morning about contact importers and where I felt they were going in the future.

I've embedded the slidecast below. Feel free to link to it or embed it elsewhere.

I plan on attending the second BarCamp event in Edinburgh at the start of next month. If you haven't heard of Barcamp before take a look at their site.

The first BarCampScotland even took place last year but I didn't know about it at the time so this will be the first I have attended. The intention with BarCamp seems to be to generate as much online content as possible. Videos on youtube, photos on flickr, presentations on slideshare. Wanting to know what I was letting myself in for I decided to set some time today to see what has previously been posted.

There isn't really much content, except photos, from the first BarCampScotland so I started to expand my search to BarCamp in general. There has been some decent content presented by various people. Below are some I found particularly interesting (though in no particular order . . .

DIY User Research
Leisa Reichelt

This isn't really an area I know much about though though have the uncomfortable feeling that I should. It is a nice introduction though be prepared for the slides running out before the 'story' is complete.

Leisa also presented at FOWA. Her slides and audio are also available.



How to Scale
George Palmer

This is perhaps the most complete overview of scaling a web application I've seen. I suspect the actual presentation would have felt something like a whirlwind going overhead but as we can take our time over each slide it's manageable, and very informative.



Introduction to SlideShare
Kapil Mohan

An introduction to SildeShare at BarCamp and posted to SlideShare. Naturally.



In my opinion they were the best 3 of the first 60. There were 275 in total. If you want to see more you can continue the search.

BarCampScotland

There were not many but there were some . . .







p.s. SlideShare supports audio. When did that happen? Lack of audio support was my main objection to the idea. Having said that none of the presentations have audio which is a shame.

The big facebook story this week has been the decision to boot Robert Scoble for running a scraper over his account so that he can move his 'social graph', more commonly know as his friends list, to other services. This was against their terms of service - naughty Scoble!

I'm not going to get into the moral issues about whether he breached the privacy of his friends by attempting to view their email addresses which they had granted him permission to view. My work on a MSN messenger contact grabber and highlighting gmail contact and yahoo contact grabbers probably speaks for itself.

Instead let's see if we can make sense of the terms of service Robert Scoble decided didn't apply to him.

I've taken relevant extracts from the terms of service at 3 websites. Try and guess which belongs to facebook.

Site 1

In addition, you agree not to use the Service or the Site to:

- harvest or collect email addresses or other contact information of other users from the Service or the Site by electronic or other means for the purposes of sending unsolicited emails or other unsolicited communications;

Provided that you are eligible for use of the Site, you are granted a limited license to access and use the Site and the Site Content and to download or print a copy of any portion of the Site Content to which you have properly gained access solely for your personal, non-commercial use, provided that you keep all copyright or other proprietary notices intact. Except for your own User Content, you may not upload or republish Site Content on any Internet, Intranet or Extranet site or incorporate the information in any other database or compilation, and any other use of the Site Content is strictly prohibited. Such license is subject to these Terms of Use and does not include use of any data mining, robots or similar data gathering or extraction methods.

Site 2

In using the service, you may not:

[...]
* use any automated process or service to access and/or use the service (such as a BOT, a spider, periodic caching of information stored by Microsoft, or "meta-searching");

Site 3

You agree not to access the Service by any means other than through the interface that is provided by Yahoo! for use in accessing the Service.

Social networks are built on scraping content

The first site is facebook. Robert Scoble clearly breached the terms of service. But here is where it gets interesting.

Site 2 is hotmail for which facebook provides a automated service, which could also be called a BOT, to grab contacts.

Site 3 is yahoo for which facebook provides a automated service to grab contacts.

Terms of service be damned! Mark Zuckerberg has a fortune to make and if that means being an accessory to millions of users breaching their contracts with other companies then so be it.

Many of the companies which facebook provides bots for are moving to support porting users data between services. On Thursday facebook was invited to do the same. I would ask that they give it serious thought.

There have been some requests to revisit the contacting a contact list tutorial and include additional features. Some of the requests have been for features mentioned in my original follow up dealing with some of the potential problems. That's great but until now I've avoided writing such an article. The main reason being that much of the functionality would need to be deeply integrated with an entire site and so everyone would need a custom solution. I've finally decided that a generic example may make it easier for people to implement their own custom solutions and as such here it is.

I feel the improvements I'm going to include are really important so to ensure that everyone can make use of this tutorial I'm going to take a step back and develop it in PHP4. I suspect there may still be some using PHP4. This will likely be the last time I worry about compatibility with PHP4. In the new year it will be PHP 5 all the way.

The Original

For those who haven't seen the original article it

1) Requested login details for gmail or msn messenger (not the same as hotmail)
2) Logged in to the service and fetched the contact details
3) Listed all contacts and enabled the user to choose which should be contacted
4) Sent an email to all requested contacts.

Improvements

This updated tutorial will show how to the above and also the following

1) Defend against malicious attacks
2) Prevent duplicate messages from being sent
3) Allow recipients to opt out of future messages

If you have read the follow up post to the original tutorial you'll see that the improvements are focused around minimising the problems surrounding unsolicited email rather than improving the efficiency of the process. Those potential improvements are really beyond what can sensibly be included in a generic tutorial like this one.

So, without further introduction lets get started. Continue reading "Contacting a contact list: A tutorial - revisited"

I'm privileged in that I'm able to 'guide' the choice of some of the gifts I'm likely to receive over Christmas. Given the equally dire state of both my understanding of OOP and the programming sections of my local libraries I feel this is somewhere a good book could help.

From various places on the web I've found three books which may be helpful and would love to get some feedback from the PHP community.

For background, programming is not a full time activity for me and although I can easily knock up a custom cms I wouldn't describe myself as a seasoned pro. Also, although it has not always been the case, I almost work exclusively in PHP and expect this to be the case for the foreseeable future.

The Books

PHP|architect's Guide to PHP Design Patterns by Jason E. Sweat.

From the reviews it looks like this one may go somewhat off-topic.

PHP 5 Objects, Patterns, and Practice by Matt Zandstra

The reviews seem positive. Probably the most likely candidate at this point.

The Object Oriented Thought Process (Developer's Library) by Matt Weisfeld

From the reviews it may be a too brief introduction.

Have you bought any of the three books above? How useful was it to you? Any other suggestions?

p.s. A cookie for anyone who can name the film I've paraphrased in the post title.

A couple of hours ago I was playing around scraping some content from a website. All was going well until suddenly I couldn't get my script to fetch meaningful content. I could view the content perfectly through my browser, on the same IP address, but through PHP is was a no go.

The first thing I did was stop visiting the site for 15 minutes or so and then increase the time between requests. It briefly worked again but quickly stopped. Next, I opened up php.ini and checked what useragent PHP was using. It turned out to be 'PHP'. I changed that and for the past 3 hours (almost) the script has been working perfectly.

Moral of the story: When scraping content from the web don't make it obvious

It's worth noting that I can't say for sure that it was changing the user agent which fixed the problem, it could have just been coincidence, but it's an easy fix and why make it obvious that you're scraping content?

Options

In this instance I was just working from my development server so I had access to php.ini but I had several options.

I could have added a line to my .htaccess file



or used ini_set.



Curl also allows you to specify the useragent.



If you want to take cloaking the useragent further with curl this comment in the PHP manual may be useful.

As mentioned earlier this is the second, and final, post going into greater detail on one of the sessions at the recent (it's been sitting in my drafts folder for a week) FOWA conference in London.

Matt Biddulph discussed interacting with 3rd party sites and services and the portable social graph. This session was particularly interesting to me given my interest in handling contacts from MSN messenger, Gmail, AOL messenger (AIM) and Yahoo!.

Matt's talk focused around how we can move beyond the use of such scripts with their multitude of risks to a situation were a user can join a new site, release the minimum data necessary and quickly identify their friends already using the service.

For those wondering what risks I am talking about the current situation means that for a site to view your contacts in a service they need total access. This means they could view your mail and send mail through your account. Where a username/password combination is used to access other services on a site these would also be compromised, so to take Google as an example you also grant access to Google Checkout, Adsense, Analytics etc. This is the situation today. Even Dopplr, the contact importing functionality of which the conference chairs spoke about with nothing but praise, requires your username and password to import your Gmail contacts.

Luckily, this situation is on the cusp of improving. For services where you are happy to make your list of friends publicly viewable marking them up with microformats is a simple way to allow other services to make sense of your friends list. Matt mentioned that twitter was also thinking of taking this a step further and supporting openid so that a user could prove that a friends list really was their friends list. I'm not terribly familiar with openid but I assume this wouldn't lead to once again proliferating login details and you simply delegate to your actual provider. Correct me if I'm wrong.

This works well when you're happy for your contacts to be public but you're probably going to want to keep at least some of the contacts in your email address book private so another solution needs to be found. Thankfully a standard way of achieving this is being developed. In fact, Matt talked about five.

Yahoo! BBAuth
Google AuthSub
flickr authentication
AOL OpenAuth
OAuth

I've also talked previously about Windows Live Contacts Control which although more limited in scope for the purposes of this discussion it does much the same thing.

All these services, and hopefully there will be convergence in the standards, open up parts of that glorious social graph and they do it in a safe manner. They also work without an absolute demand for javascript which was one of my main criticisms of Windows Live Contacts when I first spoke about it. Even the Microsoft service now has a RESTful API, either I missed that when I first took a look or it's new.

Hopefully it won't be too long before most of the code on this site is nothing more than an historical curiosity. I don't think we are there yet but soon. . .

The slides from Matt's talk are also now available.

I've previously mentioned the work Steve Souders is doing evangelizing high performance websites at Yahoo! and I was very pleased to be able to hear him speak at the FOWA conference. Sadly the video from that conference isn't going to be freely available and although the audio and slides will be freely available they are not yet available.

Luckily he has previously recorded a video of a very similar (though not identical) presentation and I've embedded it below.



It's 37 minutes so for those in a hurry here are my notes from his presentation at FOWA.
The key points have also been discussed in a series of blog posts so where possible I'll link out to the relevant posts.

Importance of the backend

The first point raised is that the user perception of load time is more important than the actual load time. This means that the relevant metric is not how fast can the html document be returned to the browser but how quickly that html is rendered in the browser. In measuring how quickly a page renders it was quickly realised that the backend performance, returning the html for the page, accounted for only about 5% of the overall time it took to render the page. Even with a full cache the backend was still only 13% of overall time.

Of the top 10 sites in the US only the backend for Google accounted for more than 20% of the load time with a full cache. The Google homepage is so spartan that with a primed cache only two HTTP requests need to be made.

Cache Usage

The importance of the cache was then discussed with data presented discussing how many people at a site had a primed cache. They inserted a one pixel image on the Yahoo! homepage and then monitored the number of HTTP requests with a 200 header (empty cache) and a 304 header (primed header).

It was found that 50% of daily users have an empty cache which accounts for 20% of daily pageviews. This varies depending on the type of site, for example an empty cache accounts for fewer pageviews in a webmail site where each user will view multiple pages, but is broadly accurate. The data highlights the importance of catering for those users without a primed cache. Excessive use of images can't be justified by the assumption that once they are loaded the cached versions can be used. 50% of your users every day will be arriving at your site with an empty cache.

iFrame

Next he talked briefly about iFrames and how they can cause a 40-50 ms delay. onLoad doesn't work until the iFrame source responds which can cause a problem with 3rd party content.

YSlow

Next he discussed YSlow which grades a website based on the 14 rules developed through their research. YSlow is an extension for Firebug the popular development extension for the Firefox browser. It looks at how the page was built. Despite looking at the content rather than the response time its score correlates well with the rendering time. As such it could be a valuable tool during development to predict the speed of a site prior to its launch.

Another issue which YSlow apparently solves is a bug in how Firebug charts HTTP requests. Apparently Firebug will show queries to the cache as HTTP requests and YSlow patches this.

That's all I made notes on. I've got vague memories of stepping HTTP requests to increase download speed and cookies are always worth considering but I picked up a nasty cold in London and it's all a bit fuzzy.