I’m sure an application focusing more on URLs is somebody’s 10% project somewhere sitting on the backburner until they have time to finish it.  It seems like almost anything you’d want to do has a corresponding web application.  When I come across a need that doesn’t I just feel like that void should soon be filled.  I’m going to outline the application that I believe would fill this void.  Maybe in future posts I’ll also walkthrough my attempts to do it as my personal .5% project.

Doesn’t this already exist?

When giving the quick summary of what I envisioned for the project the first question I get is, “Doesn’t Google Cache already do that?”  My initial reaction is that if it did, I would already be using it for that purpose.  After thinking about it, it doesn’t seem like that far of a stretch for Wayback Machine or Google Cache to archive URLs as well.  I feel like the real kicker is that in order to make it useful there would have to be extra processing and utilities for the URL archive.  The information is already there, so what needs to be done is create an interface for analyzing the URL a site has.  Before talking about the ideal solution, first let’s look at what is offered by Google Cache and Wayback Machine already to compare.

To get this sort of functionality out of Google Cache as far as I know is to use a “site:” search.  So, typing in the search, “site:arena.cse.sc.edu”, I get the following:

Result for site:arena.cse.sc.edu

I know that our research group’s website has lists of most of the pdfs that we have published as well a number of static pages that total more than the 17 that are listed in these results.  Of course, this site isn’t really optimized for search engines, so that might also not be helping.  Searching for just pdfs returns a measly 7 results.  Some of our papers are listed as links to other sites such as IEEE, which causes them to not get listed in these results.  Alternatively, if I could somehow easily browse all the URLs that exists for this domain, I might be able to see more information about what the website actually contains internally (more on this later).  Another experiment was to search for, “site:chase.ratchetsoftware.com”, and because this site is optimized for search engines, most of the URLs that exist were found pretty easily.  There is still no way to really tell what the site contains as an overview without going to the site or reading the provided description.  Put another way, it would be interesting to be able to see a something like a content tree that lets a viewer easily see what is contained in the site with a quick glance.  These are all things I believe can be solved and would be useful, but Google Cache and Wayback Machine are not the tools to do it.

What would an ideal solution look like?

I’ve given a few examples where Google Cache and Wayback Machine don’t offer what I am looking for.  I think to improve my argument it is best to talk about what the URL archive system would be in a perfect world.  It’s hard to see the difference in something that doesn’t yet exist.

I am going to list the features I have in mind and then explain any that aren’t obvious:

1. Provide a list of all URLs ever found for a given domain.
2. When a user views a list, it should check and display whether that URL is still valid.
3. Organize the list of URLs so that it is tractable by a normal user.  It should have collapsible SubURLs so that only top level URLs are in the master list.
4. Provide a filter so that only URLs matching the given filter are shown.
5. Provide the ability to click on a URL to make that the new root URL for our list.
6. A utility that can display a graphical tree view of the page and its links.  If the page links to an outside page, then show the link with the icon to that page if available.  This view would ideally provide enough information and “graphics” to give the user a good idea of what is available on the site.
7. If the site uses dynamic URLs, perhaps for a site written in rails or php, there should be some sort of analysis of the GET arguments from what the system can figure out.  This might be an area where users can submit additional information to improve the analysis of a web applications generated URL addresses.
8. One of the coolest ones.  A firefox plugin that automatically helps you when trying to figure out URLs based on what is stored in the URL Archive.  So if you’re trying to modify a URL slightly, you can get real-time help and see which URLs are still valid, etc.  Firefox is good right now with it’s new location bar that automatically searches your history and all that stuff.  It’d be even better if it could help you with long URLs that you haven’t typed in before!
9. Another plugin idea would be to provide screen readers easier access to a site’s content.  If all we had are the URLs and perhaps keywords for those URLs, a blind person might be able to more easily find content quicker.
10. One more plugin would be to have a sidebar while browsing sites, and as you browse you get an updated list of URLs that have existed at your current location on the site.  This would put a sidebar on your left with all the links you might see on that site and all the URLs that you might not see but existed in the past.  This would be useful for quickly navigating huge sites, especially ones that are moving their links around a lot.
11. The last idea I can think of right now is that the system could analyze how the URLs are changing and suggest to users URLs that may be used in the future or are currently used but not linked to based on the patterns in  the current URLs and their change history.  This might be useful if someone is trying to URL hack but doesn’t have any idea where to start.

I feel like number six is the one that needs further explanation in order to seem as cool as it does to me.  I believe there are already existing tools that will show you a tree view of a website.  There are definitely tools to show you this information for your own site, cause I’ve used them before.  So how does number six differ from these tools?  The answer is that it doesn’t differ dramatically, but hopefully it would offer more information because the purpose is slightly different.  Let’s give an example of a user that is looking for a site that offers ebooks in pdf form online somewhere.  The user might go to various sites by searching through Google and waste time at many sites that seem to offer ebooks but don’t have that large of a selection.  If the user could easily view a graph of each site before visiting, and turned on a filter for “.pdf”, he could hopefully see a tree that branches out into many different URLs leading to ebooks he’s interested in.  He should be able to quickly get an idea of the quantity, quality, and link validity of the site’s offerings.  Each node could be clicked on to make that node the new root node in the tree to perhaps get more specific information about a certain path.   There are many possibilities for graphically browsing a site’s URLs and URL relationships.  To me, this seems like a separate application entirely.  It might be that the site should be developed with a SOAP or REST interface and another application could be developed that provides this tree view of websites.

Potential problems and Questions

Not all good things come from having a URL Archive.  All the security concerns that come with Google Cache and Wayback Machine will also affect the URL Archive.  Some information that people wished to remove from the internet by unlinking it might be affected in a negative way by a system that indexes URLs forever.  Analyzng the GET parameters of a web application might be good for some users, but a malicious user could probably use this information to make the website work as it wasn’t intended.  Most of these problems come with developing this type of application.  There are tradeoffs and we just need to try to do our best to mitigate them while still providing a good service.

Another issue is the fact that I have no idea how to store such a large amount of information as this system would require.  I don’t even know where to begin with the indexing of URLs on the Internet.  It seems like it would take way too long with this little virtual machine on a server somewhere that I have.  I’d love to hear some comments about this.  How would one create an application like this that has to store such large amounts of information and process so much new data all the time?  Some resources relating to this would probably make for some great bathroom pulp .

Call for help

In my current situation I have way too much on my plate to try to develop an application for which there would be no immediate gain.  I need to work on my own business to try to get that going before I have to get a real job .  I also have an seemingly endless amount of Ph.D. work.  Despite this, I would more than willing to contribute to a project that was working on this solution.  I would even be happy to set up a Rails application on this server and with a Git repository if there were a couple developers interested in working on it.  All I’d need to distract me from my other work is a little motivation from other people’s interest, so please let me know in the comments if this is something you’d like to use or have needed in the past.

Thanks for reading,

- Chase Gray