I searched thoroughly for an easy way to retrieve information from this set of .ADT files I was dealing with. Sybase provides tools to work with the data, but all of them require using their software or drivers developed by them. I would have settled for this if it wasn’t such a hassle to figure out how to use them. I asked the question on Stackoverflow.com and got an answer from one of the employee’s of Sybase. While helpful, the suggested Perl driver was one of the things I was avoiding. I did attempt it and after a little hassle, moved on to creating a simpler solution.

One solution was to simply use Sybase’s provided “Advantage Data Architect” software to open the .ADT files and export them all to SQL. This is possible with their software, but it would not work for my situation. The developers of the program outputting these .adt files used a separate embedded database for each new report that was being worked on. This meant there was hundreds of folders, each containing 30-40 .ADT files making up their own embedded database. I needed something that could automate the extraction of this data.

Data Format

To cut this short, I decided to open up the .ADT files with a hex editor and see if I could get the information I needed out. I only needed the column names, their types, and the actual rows of data. To aid anybody else doing something similar to this, I’ll include some of my notes about the format here:

• The data is stored at the very end of the file.
• The byte offset when the data starts is located at byte 32 and is 32-bits
• Byte 24 contains a 32-bit integer specifying the number of rows
• Byte 36 contains the a 32-bit number representing the number of bytes each record consumes
• The header is 400 bytes
• Column information is after the header, before the data. Each column entry is 200 bytes.
• The number of columns can be calculated (data_offset-400)/200
• Column names consume the first 128 of each column entry.
• Byte 130 contains a 16-bit integer representing the type of data.
• Byte 136 contains a 16-bit integer, the length of the field in bytes
• Character = 04, Double = 0A, Integer = 0B, Shortint = 0C, CICharacter = 14, Date = 03, Time = 0D, Timestamp = 0E, Autoinc = 0F

This was enough information to get what I wanted.

Ruby-ADT Gem

I took the existing DBF gem by Keith Morrison. Most of the credit for this gem goes to him as I borrowed the structure heavily from his gem.

I’ll provide an example of what is now possible with my new ADT gem. No drivers are required and it’s as simple as installing the gem. I’ll include a snippet of the readme from my Ruby-ADT gem on github, borrowed heavily again from the DBF gem.

Installation

gem install ruby-adt

Basic Usage

Load an ADT file:

require 'rubygems'
require 'adt'
 
table = ADT::Table.new("test.adt")

Enumerate all records

table.each do |record|          
	puts record.name          
	puts record.email
end

Load a single record using record or find

table.record(6)
table.find(6)

Attributes can also be accessed through the attributes hash in original or underscored form or as an accessor method using the underscored name.

table.record(4).attributes["PhoneBook"]
table.record(4).attributes["phone_book"]
table.record(4).phone_book

Search for records using a simple hash format. Multiple search criteria are ANDed. Use the block form of find if the resulting recordset could be large otherwise all records will be loaded into memory.

# find all records with first_name equal to Keith
 
table.find(:all, :first_name => 'Keith') do |record|  
	puts record.last_name
end
 
# find all records with first_name equal to Keith and last_name equal to Morrison
table.find(:all, :first_name => 'Keith', :last_name => 'Morrison') do |record|   
	puts record.last_name
end
 
# find the first record with first_name equal to Keith
 
table.find :first, :first_name => 'Keith'
 
# find record number 10 
 
table.find(10)

Migrating to ActiveRecord

An example of migrating a DBF book table to ActiveRecord using a migration:

require 'adt'
 
class CreateBooks < ActiveRecord::Migration      
	def self.up            
		table = ADT::Table.new('db/adt/books.adt')            
		eval(table.schema)        
		table.each do |record|                  
			Book.create(record.attributes)            
		end      
	end  
	def self.down        
		drop_table :books    
	end
end

Limitations and known bugs

• ADT is read-only
• External index files are not used
• Dates are not currently handled correctly.

Future Directions

I am going to continue working on this library. I have already heard from a few others that found this useful immediately after I posted it online. I was surprised people could find it so quickly. I’m sure there are bugs. I would appreciate any problems to be reported to the Ruby-ADT Github page. Emails are also welcome, but any issues should be directed to the github page and I’ll get to them when I can.

Hope this helps somebody out there dealing with this format.

- Chase Gray

Prerequisites

The first thing I needed to do was to successfully configure and compile mdbtools. This probably took the longest as there are a few different version out there. I had the most success with these instructions from the author’s git repository, http://www.automatthew.com/2008/02/how-to-compile-mdbtools-on-mac-os-x.html?showComment=1219246620000. I was also able to compile this version on my ubuntu machine pretty easily. The biggest hangup was with a couple of dependencies. I also had a weird error where the autogen.sh was not detecting my libtools library, but I knew this was installed. I commented out this check in the autogen.sh and things all worked, luckily.

Understanding mdb-tools was not very difficult. It contains various libaries for interfacing with it through C, etc. The only thing I cared about were the command line tools that it provided, listed here. The commands I will use in this post are mdb-tables, mdb-schema, mdb-export.

The other requirement is to have a MySQL table created ready with permissions, etc. that will end up holding the data from the combined .mdb files.

Step 1: Extracting the Schema

One important thing to note about my situation is that all the .mdb files have the same schema. Knowing this, I was able to simply find a random .mdb file and base my MySQL schema on this.

I have my .mdb files stored in a shared directory and so within my Rake task I used the following code to grab the first one (I’m sure there is a better way).

first_mdb_file = ''
#get first mdb file
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    first_mdb_file = "#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file}"
    break
  end
end

Knowing the path to the first .mdb file the schema can be extracted by getting the result of the following shell command (the code surrounded in backticks is executed on the command line of the system). The -S flag tells it to santize some of the table names by replacing spaces with underscore, etc. I needed this for my tables because there were some problems without it.

schema_sql = `mdb-schema -S #{first_mdb_file} mysql`

Step 2: Building the Tables

Since I’m doing this in a Rake task in Rails (you could do this in plain ruby), I am using an ActiveRecord connection to interface with my MySQL database. I have a class called ExternalDatabase that allows me to connect to a different database without modifying the one that my main Rails program uses. I will list it below, I don’t remember exactly where I got it from.

#Class used to connect to external databases without modifying the exiisting connection used by Rails
class Data::ExternalDatabase < ActiveRecord::Base
  self.abstract_class = true
 
  def self.set_source(configuration_name)
    source = ActiveRecord::Base.configurations["sources"][configuration_name.to_s]
    establish_connection source
  end
 
  def self.set_configuration(config)
    source = config
    establish_connection source
  end
end

As you can see, I also have a section in my database.yml where i list connection details for various source databases. This isn’t related to this post though.

So, assuming we have this class, we can use the following code to load our new schema into out MySQL database.

Data::ExternalDatabase.set_configuration(:adapter  => "mysql",
:host     => "localhost",
:username => YOUR_DATABASE_USERNAME,
:password => YOUR_DATABASE_PASSWORD,
:database => YOUR_DATABASE_NAME,
)
db =  Data::ExternalDatabase.connection 
 
puts "adding if exists clause to drop tables"
 
schema_sql = schema_sql.gsub(/DROP TABLE (.*);/, 'DROP TABLE IF EXISTS \1;')
schema_sql = schema_sql.gsub(/--.*$/, '')
puts "executing schema sql"
lines = schema_sql.split(';')
 
lines.each do |sql_line|
  next if sql_line.blank?
  db.execute(sql_line)
end

The first part builds my database configuration hash that I pass in to ExternalDatabase. I save the connection into a variable called ‘db’. One of the important things to note about this and mdb-tools is that it does not check if the table currently exists before trying to drop the table. So I added a snippet of code that removes the comments and adds ‘IF EXISTS’ to the ‘DROP TABLE’ clause in the schema sql.

For some reason, trying to execute this entire schema sql string with db.execute causes various errors that I couldn’t figure out. By separating the schema sql into individual sql statements terminated with semicolons seemed to fix this problem. The last bit of code does the separation into lines and executes the sql on the database. After all this, the database should have the correct schema.

Step 3: Load Data

One of the last things we want to do is actually load all the data from the different mdb files into the single MySQL database.

tables = (`mdb-tables #{first_mdb_file}`).split(' ')
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    puts "Current file: #{mdb_file}"
    tables.each do |table|
      puts "Current table: #{table}"
      insert_sql = `mdb-export -I -S #{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file} #{table}`
      insert_sql = insert_sql.gsub(/;/, ',')
      insert_sql = insert_sql.gsub(/(.*?\))(\s+INSERT)/, '\1;\2')
      lines = insert_sql.split(';')
      lines.each do |sql_line|
        next if sql_line.blank?
        db.execute(sql_line)
      end
    end
  end
end

So the first line runs mdb-tools command line tool ‘mdb-tables’ to get a list of the tables separated by spaces. I could have done this through my previously created schema as well.

This code goes through every .mdb file in my directory and uses mdb-tools to extract the data as sql ‘INSERT’ statements from the command-line.

In the inner-most loop, I clean up the returned value from mdb-export because it does not include semi-colons after each INSERT statement. I also remove any semi-colons before adding any because some of my fields had semi-colons in them. Again, I separate each INSERT statement into a single statement and execute these individually in the last section of the code.

After all this you should have all your data combined into a single MySQL database. One problem might exists though. Certain tables, such as tables that contain general purpose labels, etc. for other tables to refer to, might contain duplicates. This wouldn’t have caused a big problem for me because I just needed to extract data, but if you needed to continue using this database you would need to remove the duplicates.

Final Step: Removing Duplicate Rows

There seem to be a lot of different ways to remove duplicate rows from a MySQL table. I wish there was a simple solution, but I couldn’t find one. The following code does the following:

1. Create a temporary table that has the same schema as the table we are removing duplicates from.
2. Insert all distinct rows into this new temporary table.
3. remove all values from the original table.
4. Insert all of the rows from the temporary table back into the original table.
5. Finally, drop the temporary table.

tables.each do |table|
  db.execute("CREATE TABLE temp LIKE #{table}")
  db.execute("INSERT INTO temp select distinct * FROM #{table}")
  db.execute("TRUNCATE #{table}")
  db.execute("INSERT INTO #{table} select * from temp")
  db.execute("DROP TABLE temp")
end

Thanks for reading, I hope this was helpful to others that have some similar problems.

- Chase

Problem 1: How to let Google index my data that requires a login

A prerequisite for this step is to submit sitemaps for your site’s content to Google Webmaster Tools.  Once you have done this, at some point in the future Googlebot will attempt to traverse your page and index its contents (hopefully).

The content on my site requires a user to pay a subscription fee in order to have free reign and browse through whatever they please.  I’m not very concerned about any single piece of information getting leaked, instead I’m protecting the resource the site provides as a whole.  By letting Google index your site, you’re obviously opening up your content to outside viewers that are not logged in.  The extent that you need to protect your content could vary from what my needs are.

What I want is for a user to be able to type their name into Google search engine and find out that they are referenced in the data provided by Myhealthcaresource.com.  So, I need to somehow let Googlebot traverse my site, even through it requires a login by a normal user.

I’m using Restful Authentication and a role requirement system, so this is how I include this exception.

class FacilitiesController < ApplicationController
 require_role :basic, :only => :show, :unless => request.env['HTTP_USER_AGENT'] =~ /.*Googlebot.*/
end

This could vary depending on your authentication system, but the part that will remain the same is the check if the user agent string contains ‘Googlebot’.

That’s all it takes in order to allow Googlebot to traverse my controlelr that previously required a login.  A potential problem that could come up is that users could ‘fake’ their user agent to say ‘Googlebot’ and gain access to the site.  This could be a problem for some sites, and we would keep an eye out for this in our logs or through analytics software.  We provide a free trial to all our users before they decide to pay, so uers get a free look at the data already and there is no reason to try to sneak around the site to see it.  We’re not concerned with this problem, but it could be an issue for others.

Problem 2: How to prevent Google from caching private pages

Once Google starts indexes pages of your site that require a login for a normal user, it will also start caching all the sites that it visits.  If you are worried about your internal content getting cached by Google you will want to prevent Google or any other robots from caching.  This is simple and is accomplished by putting the following in the heading of your main layout file.

<META name="robots" content="index,follow,noarchive" />
<META NAME="googleBOT" CONTENT="noarchive" />

When Googlebot reaches your page and sees this tag, it will not cache it.  I have this on the Myhealthcaresource layout, and by searching myhealthcaresource in google you can see there are no cache links.

Problem 3: Allowing Users Referred From Google View Content Without Logging In

When a user sees a link to myhealthcaresource on a set of Google search results, they are most likely to the detailed facility reports that I submitted in a sitemap.  They probably searched their company name or their own and found that they were referred to by my site’s information.  When someone clicks this link in Google, by default they would get redirected to a page asking them to log in.  This would make almost every user referred from Google resort to their back button immediately.  I noticed this behavior during the period when my site functioned this way.  These users from Google have no idea what kind of information our site offers and they might be a potential customer, so we don’t want them to leave because we present a log in form when they first arrive.

An alternative is to allow users referred to our site through a Google search to view the entire page.  We decided to do just this.  When they click the link in Google’s search results they are not asked to log in and they are shown the same page a logged in user would see.  What happens when they try to browse the rest of the private content?  They are asked to log in, but we already have their interest at this point and they are more likely to stay if they found something that was useful to them on the page they landed on.

So how do we accomplish this?  In a similar way that we allowed Googlebot to traverse our pages, we can allow users that were referred from Google to access our page:

class FacilitiesController < ApplicationController
  require_role :basic, :only => :show, :unless => request.referer =~ /.*google.*/ || request.env['HTTP_USER_AGENT'] =~ /.*Googlebot.*/
end

Of course this code would vary slightly if you are using a different authentication scheme.  The core idea would be the same though.  Now, the potential problem that could come up with this is that a malicious user uses Google to repeatedly access your site by creating clever search queries.  In our case, if a user is this intent on stealing the data, we weren’t likely to win them as a customer in the first place.  We can also monitor this through analytics tools and if it is abused we will look into an alternative solution.  It could be possible to restrict the number of Google referrals a certain IP can have in a time period or something.

I hope that some of this work can be useful to people who are looking for ways to leverage Google for their dynamically generated login-only content.  We see people come from Google searches every day, and now they stay much longer as a result of these changes.

In a future post, I will briefly talk about another method our site uses to send links to users that allow them to browse the site freely without logging in.  After a certain time period these links expire and the users must sign up to browse the site further.

I found myself working with some data in a hash generated from scraping a website. The website had some duplicate content and I didn’t feel like it was necessary to clutter my scraping code with tests for duplicate content while it was scraping. After scraping the data and turning it into a hash, I wished to remove the duplicates with some post-processing. I felt like this should be pretty simple, except that the elements I wanted to test for uniqueness contained a deeply nested structure of hashes and arrays. Here is an example an element in an array I wanted to uniqify:

{:name => 'bob', :posts => [{:title => 'First Post!' :comments => [{:commenter => 'Alice', :text => 'Hi bob!'}]}]}

The #uniq method does not work as expected when there is a Hash element in the array, it only uses the object_id to determine uniqueness. There are some solutions online for dealing with this.

If you know that each element inside the array is a hash with no sub-hashes you can use the solution presented in the answer of this stackoverflow.com question. The first solution uses #inject to build a new array with only unique items. Whether an item is included in the array already or not is determined with the Hash’s #include method.

a.inject([]) { |result,h| result << h unless result.include?(h); result }

The second solution presented in the same post:

a.map {|h| h.to_a[0]}.uniq.map {|k,v| {k => v}}

This flattens all of your hashes into an array. It then calls uniq on the resulting array after all the hashes are flattened. This means that for an item to be unique it must have the same key AND value. After finding the unique elements, it then maps the items back into a hash.

In this post, Mike Burns solve this problem by redefining #hash and #eql?. If you are doing this kind of comparison often, this would probably be a better approach. He does note though, that by doing this the #dup method no longer works as expected.

So it was obvious through searching that I was going to have to figure this out on my own. I was a little worried that I would have to implement an ugly recursive solution that searches the entire tree to find all duplicates. While looking into this solution, I felt that it would be very nice if #uniq would accept a block that would allow me to return true if the given parameter is a duplicate, thus removing it from the returned array. This is not implemented in my current ruby version (1.8.6), but there is an example of how to do this on this blog post. Hopefully, more (all) of these methods in the ruby core will accept blocks someday.

I began to write a method that would search my tree-like structure of hashes and arrays and I felt like it was a very ugly solution. I wanted something simple that would just take a line or two and get the job done. So I showed an example of the problem to my non-computer scientist wife and she said, “why don’t you just compare how the two lists look when printed out?” I was a little shocked that I hadn’t thought of this. It isn’t the most efficient solution and it relies on the #inspect method, which is probably pretty slow because it has to determine the structure of all the hashes in order to print them in a readable format. The old solution I was cooking up would also have to traverse the tree, similar to what I expected #inspect was doing. In addition to the #inspect call, there would be a string comparison for each element. I was willing to accept both of these for a short elegant solution to my problem.

The issue that came to my mind first was whether hashes get printed out the same way every time. In Ruby 1.9, it seems that Hashes are going to be stored in the same order that they were entered. In previous versions, I’m assuming that they are stored in the order of their generated hash. This could cause a problem if the data someone was entering could be in different orders. For my purposes though, the data would be entered in the same order if it was a duplicate.

So, after all of that, to uniqify an array containing a deeply nested structure, I present the following solution:

class Array
  def nested_uniq
    self.delete_if {|item| (self - [item]).any? {|other_item| item.inspect == other_item.inspect}}
  end
  def nested_uniq!
    self.replace self.nested_uniq
  end
end

I’ll walk through it because it may seem a bit hard to understand at first. I go through every element in the array with delete_if to determine if the element should be deleted from the array. Then, for all elements except the one in question, I check to see if #any? of them are equivalent to the element in question. This equivalence is determined by doing a string comparison of what the #inspect method returns for that element.

Here is an IRB test

>>  test = [{:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}, {:to => '1', :yeah => '8'}, {:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}]
=> [{:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}, {:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]
>> test.delete_if {|item| (test - [item]).any? {|other_item| item.inspect == other_item.inspect}}
=> [{:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]

Hope this is helpful to someone! I am very interested to hear suggestions and comments on alternative ways to uniqify an array with elements that have a nested structure.

-Chase

alternate_data:
    adapter: mysql
    database: alternate_database
    username: user
    password: pass
    host: localhost

I was accessing it with something similar to the following:

ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations["#{state}_data"])
results = connection.execute description_params[:extraction_sql];
#This wasn't actually hard coded, but this is just an example
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations["development"])

The problems with this was that I had to repeat this ugly code everywhere, and if there was an exception thrown while accessing the alternate database it would not revert back to the original before exiting.  So the next request to the server thought the default database was the alternate database.  What I decided would be useful would be to have an “establish_connection” method that accepted a block and handle these issues.

I am going to write about this because I planned to before realizing that in Rails 2.2 the exact same issue is solved already with a much better implementation (of course).  You can find how to solve this problem very elegantly in Rails 2.2 at http://www.igvita.com/2008/10/27/scaling-activerecord-with-mysqlplus/.

My implementation is much simpler and I hope that anybody stuck with Rails 2.1 (like me for now) or any previous version can use it temporarily until they can upgrade in the future.

I just wanted a simple block establish_connection so I could write code like the following:

   establish_temporal_database_connection("#{state}_data") do |connection|
      results = connection.execute description_params[:extraction_sql];
    end

It would reset the old database connection if something went wrong or when the block was over.

I’m just going to list a simple module that does this, if anybody wants this in plugin form that actually overrides establish_connection then it isn’t that difficult to turn this into that:

module ConnectionWithBlock  
  def self.extended(object)
    class << object
        extend ClassMethods
    end
  end
  module ClassMethods
    def establish_temporal_database_connection(configuration)
       #save the old connection
       previous_connection = ActiveRecord::Base.connection
       #establish connection handling a possible exception and going back to original connection
       begin
              ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations[configuration.to_s])
              yield(ActiveRecord::Base.connection)
       rescue Exception => e
              raise e
       ensure
              ActiveRecord::Base.establish_connection(ConnectionSpecification.new(previous_connection.config, previous_connection.adapter_method))
       end   
     end
  end
end

Chase

I’ve been very happy with the results.  Very server intense requests are responding instantly.  This is all great, but caching is usually only appropriate for pages with content that is static for a period of time.  The biggest problem I was having is that there are very small pieces of the pages I wished to cache that are dynamic.  For instance, I want to have a bar at the top that tells the user who is logged in and provides a set of buttons to do things to their profile, etc.  An example of this is shown below:

Small Dynamic Block Showing User Profile Links

So what I needed was an easy way to specify that this small block would be dynamic while still allowing the rest of the view to take advantage of action caching.  I’m sure there is some plugin that does this and more.  If there is a very good plugin that does just this, I would love to know about it in the comments.  For now, though I am going to provide my simple solution to the problem.

My solution was to create a partial for the profile summary.  I already had the block as a partial to make my view code more DRY.  Instead of this:

<%=render :partial => 'profile_summary'%>

in my view, I replaced it with an AJAX call using Prototype (updated to use Richard Poirier’s suggestion in the comments):

    <div id="info_panel">
        <%= javascript_tag remote_function(:url => { :controller => 'profile', :action => 'summary' }) %>
    </div>

I then have a controller that can repond to this AJAX request for the profile.

def summary
  render :update do |page|
    page.replace_html 'info_panel', :partial => '/profile/summary'
  end
end

Now, every time the page loads, it will load the entire page using the cached version.  It will then make an additional POST request to get the dymanic content.  This would become more complicated if there were many dynamic sections that I needed to load.  I feel like if it was wrapped into a plugin that could automatically handle special dynamic fragments inside a block like this:

<%=dynamic_fragment do%>
   <%=render :partial => 'profile_summary'%>
<%end%>

it would make it very easy and intuitive to insert dynamic fragments into an otherwise static page.  I do not have the need to implement something like this yet, I am satisfied with my simple solution.  I would be very happy to find a solution where you can specify the dynamic sections of the view instead of only specifying the static portions.  Then the system would magically build the server-side AJAX responses without the user having to do anything.  This allows the use of action_caches which are supposed to be much faster than fragment caching.  While I’m super busy with my project, stopping to create plugins is not on the priority list, but by writing it all down here I might come back to the best ideas.

I hope anybody looking to insert a small section of dynamic code in an otherwise action cacheable page finds this information useful.

Chase

I’m sure an application focusing more on URLs is somebody’s 10% project somewhere sitting on the backburner until they have time to finish it.  It seems like almost anything you’d want to do has a corresponding web application.  When I come across a need that doesn’t I just feel like that void should soon be filled.  I’m going to outline the application that I believe would fill this void.  Maybe in future posts I’ll also walkthrough my attempts to do it as my personal .5% project.

Doesn’t this already exist?

When giving the quick summary of what I envisioned for the project the first question I get is, “Doesn’t Google Cache already do that?”  My initial reaction is that if it did, I would already be using it for that purpose.  After thinking about it, it doesn’t seem like that far of a stretch for Wayback Machine or Google Cache to archive URLs as well.  I feel like the real kicker is that in order to make it useful there would have to be extra processing and utilities for the URL archive.  The information is already there, so what needs to be done is create an interface for analyzing the URL a site has.  Before talking about the ideal solution, first let’s look at what is offered by Google Cache and Wayback Machine already to compare.

To get this sort of functionality out of Google Cache as far as I know is to use a “site:” search.  So, typing in the search, “site:arena.cse.sc.edu”, I get the following:

Result for site:arena.cse.sc.edu

I know that our research group’s website has lists of most of the pdfs that we have published as well a number of static pages that total more than the 17 that are listed in these results.  Of course, this site isn’t really optimized for search engines, so that might also not be helping.  Searching for just pdfs returns a measly 7 results.  Some of our papers are listed as links to other sites such as IEEE, which causes them to not get listed in these results.  Alternatively, if I could somehow easily browse all the URLs that exists for this domain, I might be able to see more information about what the website actually contains internally (more on this later).  Another experiment was to search for, “site:chase.ratchetsoftware.com”, and because this site is optimized for search engines, most of the URLs that exist were found pretty easily.  There is still no way to really tell what the site contains as an overview without going to the site or reading the provided description.  Put another way, it would be interesting to be able to see a something like a content tree that lets a viewer easily see what is contained in the site with a quick glance.  These are all things I believe can be solved and would be useful, but Google Cache and Wayback Machine are not the tools to do it.

What would an ideal solution look like?

I’ve given a few examples where Google Cache and Wayback Machine don’t offer what I am looking for.  I think to improve my argument it is best to talk about what the URL archive system would be in a perfect world.  It’s hard to see the difference in something that doesn’t yet exist.

I am going to list the features I have in mind and then explain any that aren’t obvious:

1. Provide a list of all URLs ever found for a given domain.
2. When a user views a list, it should check and display whether that URL is still valid.
3. Organize the list of URLs so that it is tractable by a normal user.  It should have collapsible SubURLs so that only top level URLs are in the master list.
4. Provide a filter so that only URLs matching the given filter are shown.
5. Provide the ability to click on a URL to make that the new root URL for our list.
6. A utility that can display a graphical tree view of the page and its links.  If the page links to an outside page, then show the link with the icon to that page if available.  This view would ideally provide enough information and “graphics” to give the user a good idea of what is available on the site.
7. If the site uses dynamic URLs, perhaps for a site written in rails or php, there should be some sort of analysis of the GET arguments from what the system can figure out.  This might be an area where users can submit additional information to improve the analysis of a web applications generated URL addresses.
8. One of the coolest ones.  A firefox plugin that automatically helps you when trying to figure out URLs based on what is stored in the URL Archive.  So if you’re trying to modify a URL slightly, you can get real-time help and see which URLs are still valid, etc.  Firefox is good right now with it’s new location bar that automatically searches your history and all that stuff.  It’d be even better if it could help you with long URLs that you haven’t typed in before!
9. Another plugin idea would be to provide screen readers easier access to a site’s content.  If all we had are the URLs and perhaps keywords for those URLs, a blind person might be able to more easily find content quicker.
10. One more plugin would be to have a sidebar while browsing sites, and as you browse you get an updated list of URLs that have existed at your current location on the site.  This would put a sidebar on your left with all the links you might see on that site and all the URLs that you might not see but existed in the past.  This would be useful for quickly navigating huge sites, especially ones that are moving their links around a lot.
11. The last idea I can think of right now is that the system could analyze how the URLs are changing and suggest to users URLs that may be used in the future or are currently used but not linked to based on the patterns in  the current URLs and their change history.  This might be useful if someone is trying to URL hack but doesn’t have any idea where to start.

I feel like number six is the one that needs further explanation in order to seem as cool as it does to me.  I believe there are already existing tools that will show you a tree view of a website.  There are definitely tools to show you this information for your own site, cause I’ve used them before.  So how does number six differ from these tools?  The answer is that it doesn’t differ dramatically, but hopefully it would offer more information because the purpose is slightly different.  Let’s give an example of a user that is looking for a site that offers ebooks in pdf form online somewhere.  The user might go to various sites by searching through Google and waste time at many sites that seem to offer ebooks but don’t have that large of a selection.  If the user could easily view a graph of each site before visiting, and turned on a filter for “.pdf”, he could hopefully see a tree that branches out into many different URLs leading to ebooks he’s interested in.  He should be able to quickly get an idea of the quantity, quality, and link validity of the site’s offerings.  Each node could be clicked on to make that node the new root node in the tree to perhaps get more specific information about a certain path.   There are many possibilities for graphically browsing a site’s URLs and URL relationships.  To me, this seems like a separate application entirely.  It might be that the site should be developed with a SOAP or REST interface and another application could be developed that provides this tree view of websites.

Potential problems and Questions

Not all good things come from having a URL Archive.  All the security concerns that come with Google Cache and Wayback Machine will also affect the URL Archive.  Some information that people wished to remove from the internet by unlinking it might be affected in a negative way by a system that indexes URLs forever.  Analyzng the GET parameters of a web application might be good for some users, but a malicious user could probably use this information to make the website work as it wasn’t intended.  Most of these problems come with developing this type of application.  There are tradeoffs and we just need to try to do our best to mitigate them while still providing a good service.

Another issue is the fact that I have no idea how to store such a large amount of information as this system would require.  I don’t even know where to begin with the indexing of URLs on the Internet.  It seems like it would take way too long with this little virtual machine on a server somewhere that I have.  I’d love to hear some comments about this.  How would one create an application like this that has to store such large amounts of information and process so much new data all the time?  Some resources relating to this would probably make for some great bathroom pulp .

Call for help

In my current situation I have way too much on my plate to try to develop an application for which there would be no immediate gain.  I need to work on my own business to try to get that going before I have to get a real job .  I also have an seemingly endless amount of Ph.D. work.  Despite this, I would more than willing to contribute to a project that was working on this solution.  I would even be happy to set up a Rails application on this server and with a Git repository if there were a couple developers interested in working on it.  All I’d need to distract me from my other work is a little motivation from other people’s interest, so please let me know in the comments if this is something you’d like to use or have needed in the past.

Thanks for reading,

- Chase Gray

Failed Attempts

I probably began hoping that I could one day work at Google at the very end of High School.  Back then I was focusing on Microsoft, but stories of the Google workplace soon overshadowed my dreams of working at Microsoft.  Looking back, I think each place has its pros and cons.  Without going to Microsoft for an interview, I don’t know if I can fairly compare the two.  I will say that I would enjoy making Google’s products more than Microsoft’s.  I feel like Google is making products that are pushing things forward, while Microsoft is stuck maintaining it’s current product line. Over time there began to be various videos and articles focusing on what it was like working at Google. I’ll include a couple here instead of wasting time talking about them.

CNN Covers Working at Google

Oprah Talks about Working at Google

It seems pretty hard to resist the ideal working environment for a young computer scientist.  I believe I applied to only Microsoft while I was still in high school because at the time it had the most popular(infamous) internship program.  Although it wouldn’t be long before Google interview stories began to become synonymous with Microsoft’s.  Of course looking back there was absolutely no reason for Microsoft to decide to interview me besides a very excited sounding cover letter.  I didn’t have stellar grades, I didn’t have any completed projects to show off, and I didn’t have any recommendations from anybody that had worked at Microsoft.  So of course I got a generic rejection letter from Microsoft a few months later.

Throughout college I could tell pretty much the exact same story except that my resume was growing every so slightly and I believe I quit applying to Microsoft after my sophomore year of college or around that time.  I focused on the Google internship program because I felt that was where the exciting things were happening.  At first, if I remember correctly, the internship at Google was available to all college students.  At one points though, probably around my sophomore year again, I noticed that they started requiring you to be a Master’s or Ph.D. candidate.   Regardless, I still applied each Summer just in case I could get lucky.

Landing the Interview

As you can see in my introduction post, I started doing research with my academic adviser, Srihari Nelakuditi, during the Summer of my sophomore year as well.  This was when things seemed to be coming together and I was getting a little recognition for all the reading and programming I had been doing up until this point.  I was a step or two above the average peer in my classes, and I’m glad that Srihari could see some sort of potential.  I continued to apply every Summer for an internship at Google and also at Fog Creek Software.  I had hopes that with my growing resume and research experience I might slip into an interview at one of these internships.  If I did not get the internships I was funded by my adviser to continue doing research in the ARENA lab over the Summer.  I continued at the University of South Carolina into my Graduate studies and got to the point where I was about to complete my Master’s thesis.  During my second to last semester as a Master’s student I got a lucky break.

Apparently my adviser had a few hookups at some of the companies I had been applying to over the years and was waiting until I was ready to refer me to them.  So, he told me one day that he gave me a reference to Google and I would be hearing from them soon.  It seemed kind of surprising that it should be so easy, but I’ve always read that these good jobs were about connections.  Sure enough…a few weeks later I got an email from a woman at Google recruiting.  This was the same location that had sent me all those generated rejection letters in the past.  I had a pretty similar resume this time, my experience only consisted of six extra months since the previous rejection, only this time I had someone else say, “Hey, hire this guy”.  It makes me wonder if I could have had only the reference and still be contacted by the recruiting department.

So, the first interesting thing someone might wonder who’s been attempting to get a job at a place like this is what do they say in an non-rejection email.  Well, here is the first email I got from the recruitment woman.

We continued talking about what kind of position I was looking for through a few email exchanges.  They were very nice and she even asked me about things in my extra-curriculum activities that were not related to Google, it seemed we had some similar interests.  For various reasons, I decided I didn’t want to go to California and instead let her know that I was interested in a Software Engineer full-time position at the Boston office.   I sent her my two page curriculum vitae and my current unofficial transcript and I was moved on to the next step.

So what came next was an email from a recruiter from the Google -- Boston office.  He immediately asked for a good time that he could call me and talk about various opportunities at his office.   Somehow I missed his first call because I believe he called my old phone for some reason.  The only explanation I had was that they had me in their database from previous resumes, and these old resumes had my old phone number on them.  So, all those old resumes had been getting into their system after all . This happened on more than one occasion and still confuses me why they kept using the old number.  He was a cheerful sounding guy who seemed to make every effort to put me at ease during the whole process.   I was worried at first this might involve some technical questions but I found out that it was just to talk to me and make some plans for the interview process.

The first step of most Google interviews is the infamous phone interview.  In this process you are supposedly asked questions by one or more engineers over the phone and you are supposed to talk your way through the problem.  There are a lot of examples of these online, so that’s where I’d recommend looking for them.  They are usually questions that aren’t exactly computer science, or slightly related, but lend themselves to solutions that could be spoken out over the phone.  The reason I don’t have any personal examples is because I got pretty lucky while talking to this guy on the phone.  It just so happened that my girlfriend and her parents were going to be going to Boston about a month later.  He mentioned something about a phone interview sometime later and I said that I would probably already be visiting Boston around that time.  I waited and hoped he would make the same conclusion that I had and he did.  He asked me if I would prefer to just come on in to the office when I was visiting then.  I was ecstatic , both because I didn’t have to take the phone interview and because I was going to Google’s office in Boston in less than a month!  I don’t know if I could recommend the same process to anyone else, but it worked for me.  I was going to have to pay for my hotel because I was arriving one day earlier than my girlfriend but it was OK with me.  They said I should come in at about 10:00AM and my interviews would last until about 2:15PM including lunch.  I asked about what to wear and I was told it’s pretty casual in the office so I should probably wear something semi-formal like a collared shirt and slacks.  Now all I had to do until the big day was to try and prepare for whatever questions might be asked of me during my technical interview.

Preparation

Like most people preparing for a technical interview, I didn’t have any idea where to begin.  It seemed like they could ask me anything.  Looking up examples of past questions from Microsoft and Google only made me more worried.  The questions I found were mostly reasonable, but some were just so weird and I had no idea how to solve them.  Regardless, one of the best ways to prepare was to study these example questions.  Not so much to memorize the answer, but to understand the thought process behind the answer.  Most of the questions people seem to face have similar fundamental problems.  The biggest challenge is to take a problem you are presented and trying to put it into terms similar to a problem you already know how to solve.  Many of the traditional problems could be found in most algorithms books.  It seems like algorithms pretty much dominate the questions, but all other areas of computer science are fair game.

I’ll try to list some of the resources I found.  I didn’t exaustively try to understand every problem, but I spent a while going over things that I didn’t understand and refreshing mysefl with fuzzy areas.  Again, the idea was to try and be capable of coming up with my own solution based on the knowledge I had, not to memorize answers.  Aside from studying algorithms and previous questions, my adviser also gave me a book with quick hacking tricks used in programming.  These ranged from how count the number of ones in a binary number quickly to various memory hacks.  I’ve seen that people were asked questions relating to these dirty hacks but they are mostly there to increase your bag of tricks.

• Ace The Interview
• Some good general tips for a technical interview from a Microsoft interviewer
• Huge list of compiled questions from interviews.
• http://www.allinterview.com/

How to Hack the Technical Interview

After reading algorithms and technical interview questions for a few weeks during my free time.  I was finally getting prepared to hop on the plane to Boston for my interview.  I was very excited to get to see Boston for the first time.  It was also going to be when we told my girlfriend’s parents about our engagement, so it was going to go well even if my interview went bad.

The Boston Office

Arriving in Boston I got into my hotel near the airport and planned to stay in the entire night reviewing some basic algorithms.  I started to make sure I had everything to wear for the next day.  A nice collared shirt and black pants.  I realized at this point that I had forgotten to bring a belt, and the pants didn’t fit perfectly around my waist so it was a required piece.  I started to get pretty worried and was looking for a quick solution.  I decided to look online for the closest store.  I was near the airport so there was nothing at all.  I just got out of the hotel and walked on foot through East Boston for about 30-45 minutes until I came across an area that had some sort of stores in it.  I had very little hope though that I would find a belt for my outfit in what looked to be a mexican area of East Boston.   To my surprise I found a small mexican shop that sold clothes.  When looking through their limited supply of belts, I found that they all looked like this:

Tacky Belt

This would obviously not do for my interview!  They did have one generic brown belt. I asked for it without trying it on at all and left.  I went to some pasta place for dinner and brought it back to the hotel.  When trying the belt on later that night I realized it was GIGANTIC!  I had to figure out a way to cut a hole in it and figure out a way to make it stay on and hold my pants up throughout the all day interview.  This was the biggest issue I faced during my entire time at Boston, so I think that makes my trip a relative success.

I woke up the next morning pretty early so I had time to figure out the train system and where I had to go.  I needed to get from East Boston over to Cambridge.  Obviously this wasn’t too much of an issue and pretty soon I was right next to the MIT campus looking for which building Google was located in.  The lady had given me the address, “One Broadway, Cambridge, MA”.  Apparently it was this big building here:

Google Boston office from across the street

Walking up closer you can see that it is just a typical office building.  There was nothing flashy or fancy about the outside, but Google was located on the 7th floor (and I would learn also another floor as well).  I expected at least the 7th floor to have the “Google look”.

Google Boston office front doors. Then 7th floor.

For those of you who are truly interested, here is an embedded map of the location:

View Larger Map

I then used my extra time to sit at a nearby Cosi restaurant and get some breakfast and relax.  I was going to be in interviews for the next 4 or 5 hours.  So I finally went inside the building and up to the 7th floor to see what awaited me.

The Interview (and Lunch)

I walked off of the elevator to a big happy Google logo in the main hallway.  Then there were two glass doors that needed a card to unlock them.  Luckily someone was walking out and I walked in.  There was a lady at a desk to greet visitors and she had me sign in on a computer with a little application they had.  I had read somewhere that the Mountain View office has a big screen with real-time searches being displayed.  I had also read someone’s blog that the Boston office did not have one of these screens.  I was pleasantly surprised to find that there it was in all its glory to entertain me while I sat on one of the fancy Google colored chairs amidst the various toys and Google exercise balls on the floor.  As I looked around there were a couple offices (one with guitar hero) and a surprising amount of women.  As I would later learn, this was primarily the sales floor, which explains the large number of women.  Finally, my recruiter came out and greeted me and took me up to a higher floor where they had more of their engineers hiding.

The upper floor was still very nice and people were not too cramped at their desks.  There were not an cubicles, mostly open space.  Everyone had two monitors and either a PC or a Mac.  Some people had portable video conferencing devices so they could work with people from other offices with constant communication.  Everyone I ran into seemed very nice.  It looked like the upper echelon of computer scientists.  That’s a weird way to put it, but most people reading this probably know what the lower portion of computer scientists consist of.  I’m talking social skills of course, not technical.  I was told that this office was becoming too small and they would soon be moving to a larger building.  Apparently, they would soon be allowed to bring dogs to work on one of the floors of the new building!  I always loved the thought of bringing my dogs to work with me instead of leaving them at home to be bored all day. After the quick walk through the upstairs office I was brought into this little room with a desk, a chair, and a whiteboard.

The interview began with a conversation between me and my recruiter who I had talked with on the phone one month prior.  We talked about what it was like to work at Google -- Boston.  He wanted to answer any sort of question I might have.  So I asked about all the various things their office did, the normal workday, the time commitment, etc.  Overall I got the feeling that the Boston office was more consistent in their eight hour workdays because the average age of employees was higher and they had families at home, etc.  I was very worried about all the articles I read about online about Google employees working to death without much compensation.  He then told me how the interview process was going to proceed.  I was going to be interviewed by four different engineers for 30 minutes each.  In the middle I was going to have lunch with another engineer.  With that he said his goodbyes and good lucks and introduced me to my first interviewer.

In order to avoid any problems, I’m only going to generically talk about the problems any of the engineers gave me.

The first engineer was a small asian guy who seemed very friendly.  He sat down across from me and asked a few questions about my research experience and tried to try to get a handle on some of my previous projects.  He then asked if I had any questions about Google for him and we moved on to the technical questions.  He had two questions for me.  Given an array of stock prices, what would be the best buy and sell point to make the most money.  I had read about problems similar to this while studying.  I had to modify it a bit in order to get the solution.  The similar problem I had studied was finding the largest sum of an array of positive and negative numbers.  For the stock question, I had to change each number to instead be the gain or loss for that day.  Then, the problem reduced to the largest sum of contiguous elements or Kadane’s algorithm.  The second question was given a set of N terms, you have N linked lists each containing an IDs of a webpages that match the term the list corresponds to.  The important thing was that each list was in sorted order. I was to come up with a method to AND multiple terms together and retrieve the matching pages.  This was not similar to a problem I had studied and I had to come up with a couple ideas before settling on one.  The fact that they are in sorted order allows you to quickly determine if you need to continue a search or not in the second list.  The biggest challenge was when the interviewer then asked me to implement it.  I stumbled a bit with some exceptional cases and the fact that the board was so tiny.  We ran out of time before it was perfect, but I felt that it was sufficient.

The second engineer came in immediately after the first.  He was a larger bearded American guy. He also seemed nice, but much more focused and serious.  We started with the discussion about my experience just as the last engineer did, but we talked in depth about one of my networking research projects with sensor networks.  I was very surprised with how well the engineers seemed to have read my resume, they were very prepared and I appreciated that.  We then moved on to the questions.  The first one was tricky at first but I was able to get it without too many hints from the interviewer.  He was mostly giving positive sounds when I talked about something that was close to the right solution. The question was given a random number generator that generates the numbers between 1 and 7, how do you create a random number generator that generates 1 through 10 uniformly?  This was similar to questions I had read online about biased coin flipping.  You basically have to generate twice and take all the combinations that can occur, ie (1, 1), (1, 2), of which there should be 7*7=49.  Divide these sets evenly into 10 sets.  Then if you generate a combination that falls within one of these 10 sets, you have a number uniformly between 1 and 10.  There will be some sets that are left over, and if you get one of these you must regenerate your numbers.  He extended the question by asking what is the probability that you will have to regenerate your random numbers.  Then he further extended the question to ask if you could reuse the previous rolls so you only had to regenerate one random number.  So I’ll leave this one to you.  I then had to determine the probability of not getting a uniform number between 1 and 10 after 7 attempts or something like that.  His second question was much more discussion oriented.  He asked me to talk about how networking would be implemented in a simple board game.  He wanted to know how the data was sent over the network.  We focused on connect four because I had experience programming this game before.  We went into all the details of what each player must know about their opponents in order for them to play the game.  We then talked about what a player would need in order to be an observer, or to join a game in the middle, etc.  There wasn’t any tricks to this question, we just talked about game networking basically until our time ran out.

Finally, I was out of the first two interviews and they seemed to have gone relatively well.  I was now introduced to my third engineer and he was supposed to have lunch with me.  There are a lot of stories about the food services at Google.  I wasn’t sure what the setup would be in the Boston office, but I was hoping it was still free.  Sure enough, free lunches are the norm even at a smaller office (30 engineers) like this one.  It wasn’t one quite as big a scale as the Mountain View campus is, but it was a buffet of above average food brought down from a catering service on an upper floor of the same building.  I don’t remember what I had, but it was pretty healthy and tasty.  I was very excited about the giant glass doored fridge they had filled with all kinds of free sodas and water bottles.  I had a pretty tasty sparkling lemon drink that was covered with a foil top that you had to remove before opening it.  I don’t remember the name but I wish I did so I could get some.  The guy said they replaced the Sprite in the office with the lemon drink.  During lunch we talked a lot about the life at Google.  The guy had a lot to say about the people you get to work with, the types of projects and languages used, and his own experiences during the interview.  There were a couple other Googlers that sat down with us and joined in the discussion about life at Google.  Everyone seemed pretty happy while they enjoyed their free lunch (and free cake).  There were a bunch of board games in this room that looked like they were played when people were not eating.  It was a pretty small room for a cafeteria, like a large conference room that they placed lunch tables in.  I was pretty surprised to hear that you could bring friends and family to the lunch with you for free as long as you let them know they were coming ahead of time.  Another interesting thing I learned was that most of the engineers at this office were from MIT.  So much so that they all had stories of when they were in college together and partied together.  I had read previously that Google had a preference for Ivy league schools, and this kind of worried me about my own educational background.  After our talk we walked around the office for a more formal tour.  I saw most of the office the first go around because it was pretty small, but they said they would be moving pretty soon to a bigger location.  We ended back at the small office where I would continue the last two technical interviews.

I don’t know how to describe the next engineer except as spunky.  He was very happy and excited to be working there.  He explained to me that every Google employee spends about three months or so at the main campus in Mountain View, CA for training.  They said it’s a pretty crazy experience unlike anything else.  So we started back with the standard talk about my experience and moved into the technical questions.   I’m sorry but I can only remember one of the questions posed by this interviewer.  He asked me to describe all the details involved in implementing something similar to the wireless access points at the airports.  When you first log on it redirects your browser to a page that forces you to either pay or sign in.  We talked about various security concerns and problems with the current approach.  He then asked me a question I did on the white board but can’t remember it for the life of me.  It might have been the stock quote question that I thought was in the first interview.  It really doesn’t matter though as I think all the questions I was asked are in this post somewhere.

The last interviewer was another large bearded American guy.  We again talked about my experience and any questions I might have about Google.  The questions posed by this engineer were the ones that stumped me the most.  One was very simple but I was too quick to come up with a solution and it had some flaws.  The question was given a binary tree, determine if it is a binary search tree.  This is a typical problem and I should have had an immediate answer.  My answer did not work on a special case and he didn’t tell me this before asking me to implement it.  I then ran into trouble while implementing it and we talked about why and figured out what my problem was.  This interviewer didn’t seem near as excited or happy to be interviewing me as the previous engineers.  The second question he asked was an event scheduling problem.  Basically you were given events with a start time and an end time.  You were then to determine if there were any conflicts in any of the events.  He would then extend the question to find a O(n) solution.  I don’t think I came up with a solution to his satisfaction during the interview.  My solution involved a hashing technique so we could determine collisions between events if the same hash was found.  I’m sure there is a much better solution, but this was at the end of the day and the guy was not being very helpful with hints.  I  felt the worst about this interview and so I left on a bad note.

I was then taken downstairs by another engineer and we made sure everything was in order and he said goodbye.  I felt kind of rushed out the door without any kind of thank you for coming or anything like that.  I felt good about most of the interview, but the last interview would bother me for the next few weeks.

Aftermath

So I was finally back home and couldn’t stop worrying about the results of the interview.  I waited a few weeks but didn’t hear anything back.  I had sent a thank you email a couple days after the email but I didn’t get any reply.  After about three weeks my secondary phone rang (again I don’t know why they have this number).  It the recruiter from the Boston office and he was letting me know that I didn’t get the job.  He said my background was not matching what they needed at that time.  I thought this was a very odd way of putting it.  This made me think that the rumors about them preferring certain colleges were at least partially true.  I’ll never know the real reasons why, but I did feel that I did pretty well during most of my interview.

Since then, I’ve come to terms that I might not be a good fit for a company like Google.  I feel like I should be putting my talents to actually create something useful and start my own company.  I know it’s a common dream, but I feel like I have a few good ideas and I’m working to make them a reality.  So, I haven’t applied to any new companies since then because I’ve been doing Ph.D. work and working on my own company Ratchet Software.

I hope that this story about my experience interviewing at Google will be of some sort of help to someone.  I know when I was interviewing I was looking for every resource I could find on the topic.  I had only found one blog post that mentioned anything about the Boston office.  I know there are a ton of people trying to get their foot in the door at Google.  I just hope that my story helps them do that.  Good luck to all the future Googlers out there.  I own some Google stock and would love to see some of you young motivated engineers come up with the next Gmail. 

- Chase

Of course the first question I had was what physics libraries are available for Flash these days.  I had some experience in the past using Newton and various wrappers in C++ for a game project.  My experience with Newton would end up helping me pick up a new physics engine relatively easy.

There were a couple of choices to choose from and I started my search by reading some reviews found on GeekGlue. From this I was able to reduce my search to the Actionscript Physics Engine (APE) and Box2DFlashAS3.  APE seemed much easier to get started with, but I wasn’t sure if it would accomplish what I wanted without a lot of massaging.  After further reading on some forums, I found that the rope simulation in APE is much more limited because there is only one type of joint available at this time.  Box2D has 6 different types of joints and their provided rope bridge looked a little better.  The fact that Box2D had ragdoll physics examples made me feel that I would have as much control as I needed to create the necklace simulation I wanted.

After choosing Box2D I wanted to take their rope “springy” bridge demo and see if that could be easily turned into some sort of free falling rope.  This was not as simple as adding links and changing a few variables like I had hoped.  If you simply add links to the rope, after adding too many the iterative solver breaks the joints with the default configuration.  I had to search for a while to figure out why this was.  After some searching, I found that a joint can only handle about a 10:1 weight ratio before giving out.  The solution to this is the increase the number of iterations of decrease the step time.  This of course will slow down your simulation depending on how much you have to increase the number of iterations.  I had to change mine from about 10 to 40.  My simulation was still running at a decent speed but in the future I am not sure how this will hold up with more physics bodies flying around.  This was a huge problem for a while for me and I was very glad to finally find a solution.  If you have joints that are breaking due to too much force being applied to them is too much without adding more iterations to the engine so it can accurately solve the joint forces.  I hope someone with a similar problem can find this post because it took me a while to find a solution on a forum somewhere.

I am not concerned with creating a nice looking application at this time since this was just a weekend project I was doing on the side.  The goals for the prototype of my necklace simulator I wanted to have the following:

1. A rope like necklace that could be moved around by the user with realistic physics.
2. Two end pieces on the necklace that were different than the rest of the necklace and could be connected to things.
3. Two or more connectors in the world that could have a necklace endpiece connected to them and unattached when the user wants to move the necklace again.
4. A prototype of a bead which can be moved by the user and placed on the necklace.
5. A background of a neck or something so that it is easy to see what the necklace would look like on a person.

For my resulting application, I simply modified the default Test class provided with Box2DFlashAS3 to have instances of the objects in the world such as the rope and beads.  Previously it was a base class for various tests.  This might not be the best solution, but it was fine for my prototype to see if things would work.  I’ll include the entire source here and highlight my changes as well.

Playground.as

/*
* Copyright (c) 2006-2007 Erin Catto http://www.gphysics.com
*
* This software is provided 'as-is', without any express or implied
* warranty.  In no event will the authors be held liable for any damages
* arising from the use of this software.
* Permission is granted to anyone to use this software for any purpose,
* including commercial applications, and to alter it and redistribute it
* freely, subject to the following restrictions:
* 1. The origin of this software must not be misrepresented; you must not
* claim that you wrote the original software. If you use this software
* in a product, an acknowledgment in the product documentation would be
* appreciated but is not required.
* 2. Altered source versions must be plainly marked as such, and must not be
* misrepresented as being the original software.
* 3. This notice may not be removed or altered from any source distribution.
*/
 
package TestBed{
 
	import Box2D.Dynamics.*;
	import Box2D.Collision.*;
	import Box2D.Collision.Shapes.*;
	import Box2D.Dynamics.Joints.*;
	import Box2D.Dynamics.Contacts.*;
	import Box2D.Common.*;
	import Box2D.Common.Math.*;
	import Main;
	import General.Input;
 
	import flash.utils.getTimer
	import flash.display.*;
 
	public class Playground{
 
		public function Playground(){
 
			m_sprite = Main.m_sprite;
 
			var worldAABB:b2AABB = new b2AABB();
			worldAABB.lowerBound.Set(-1000.0, -1000.0);
			worldAABB.upperBound.Set(1000.0, 1000.0);
 
			// Define the gravity vector
			var gravity:b2Vec2 = new b2Vec2(0.0, 10.0);
 
			// Allow bodies to sleep
			var doSleep:Boolean = true;
 
			// Construct a world object
			m_world = new b2World(worldAABB, gravity, doSleep);
			// set debug draw
			var dbgDraw:b2DebugDraw = new b2DebugDraw();
			//var dbgSprite:Sprite = new Sprite();
			//m_sprite.addChild(dbgSprite);
			dbgDraw.m_sprite = m_sprite;
			dbgDraw.m_drawScale = 30.0;
			dbgDraw.m_fillAlpha = 0.3;
			dbgDraw.m_lineThickness = 1.0;
			dbgDraw.m_drawFlags = b2DebugDraw.e_shapeBit | b2DebugDraw.e_jointBit;
			m_world.SetDebugDraw(dbgDraw);
 
			// Create border of boxes
			var wallSd:b2PolygonDef = new b2PolygonDef();
			var wallBd:b2BodyDef = new b2BodyDef();
			var wallB:b2Body;
 
			// Left
			wallBd.position.Set(-95 / m_physScale, 360/m_physScale/2);
			wallSd.SetAsBox(100/m_physScale, 400/m_physScale/2);
			wallB = m_world.CreateBody(wallBd);
			wallB.CreateShape(wallSd);
			wallB.SetMassFromShapes();
			// Right
			wallBd.position.Set((640+95) / m_physScale, 360/m_physScale/2);
			wallB = m_world.CreateBody(wallBd);
			wallB.CreateShape(wallSd);
			wallB.SetMassFromShapes();
			// Top
			wallBd.position.Set(640/m_physScale/2, -95/m_physScale);
			wallSd.SetAsBox(680/m_physScale/2, 100/m_physScale);
			//wallB = m_world.CreateBody(wallBd);
			//wallB.CreateShape(wallSd);
			//wallB.SetMassFromShapes();
			// Bottom
			wallBd.position.Set(640/m_physScale/2, (360+95)/m_physScale);
			wallB = m_world.CreateBody(wallBd);
			wallB.CreateShape(wallSd);
			wallB.SetMassFromShapes();
 
			m_rope = new Rope(m_world, m_physScale);
 
			m_leftHookJoint = null;
			m_rightHookJoint = null;
 
			m_leftHook = new Hook(m_world, m_physScale, new b2Vec2(175/m_physScale, 50/m_physScale), 10/m_physScale);
			m_rightHook = new Hook(m_world, m_physScale, new b2Vec2(500/m_physScale, 50/m_physScale), 10/m_physScale);
			m_hookJointDef = new b2RevoluteJointDef();
 
			m_bead = new Bead(m_world, m_physScale, 7.5/m_physScale);
 
		}
 
		public function Update():void{
 
			// Update mouse joint
			UpdateMouseWorld()
			MouseDestroy();
			MouseDrag();
 
			// Update physics
			var physStart:uint = getTimer();
			m_world.Step(m_timeStep, m_iterations);
			Main.m_fpsCounter.updatePhys(physStart);
 
		}
 
		//======================
		// Member Data
		//======================
		public var m_world:b2World;
 
		public var m_mouseJoint:b2MouseJoint;
		public var m_iterations:int = 40;
		public var m_timeStep:Number = 1/30;
		public var m_physScale:Number = 30;
		// world mouse position
		static public var mouseXWorldPhys:Number;
		static public var mouseYWorldPhys:Number;
		static public var mouseXWorld:Number;
		static public var mouseYWorld:Number;
		// Sprite to draw in to
		public var m_sprite:Sprite;
		public var m_rope:Rope;
		public var m_hookJointDef:b2RevoluteJointDef;
		public var m_leftHookJoint:b2Joint;
		public var m_rightHookJoint:b2Joint;
		public var m_leftHook:Hook;
		public var m_rightHook:Hook;
		public var m_bead:Bead;
 
		//======================
		// Update mouseWorld
		//======================
		public function UpdateMouseWorld():void{
			mouseXWorldPhys = (Input.mouseX)/m_physScale;
			mouseYWorldPhys = (Input.mouseY)/m_physScale; 
 
			mouseXWorld = (Input.mouseX);
			mouseYWorld = (Input.mouseY);
		}
 
		//======================
		// Mouse Drag
		//======================
		public function MouseDrag():void{
			// mouse press
			if (Input.mouseDown && !m_mouseJoint){
 
				var body:b2Body = GetBodyAtMouse();
 
				if (body)
				{
					var md:b2MouseJointDef = new b2MouseJointDef();
					md.body1 = m_world.GetGroundBody();
					md.body2 = body;
					md.target.Set(mouseXWorldPhys, mouseYWorldPhys);
					md.maxForce = 600.0 * body.GetMass();
					md.timeStep = m_timeStep;
					m_mouseJoint = m_world.CreateJoint(md) as b2MouseJoint;
					body.WakeUp();
					if (body == m_rope.m_leftHook || body == m_rope.m_rightHook){
						if (m_leftHookJoint && body == m_leftHookJoint.GetBody2())
						{
								m_world.DestroyJoint(m_leftHookJoint);
								m_leftHookJoint = null;
						}
						if (m_rightHookJoint && body == m_rightHookJoint.GetBody2())
						{
								m_world.DestroyJoint(m_rightHookJoint);
								m_rightHookJoint = null;
						}
					}
					if (body == m_bead.m_body)
					{
						m_bead.SetCollidable(false);
					}
				}
			}
 
			// mouse release
			if (!Input.mouseDown){
				if (m_mouseJoint)
				{
					//get body being clicked on
					var clickedBody:b2Body = m_mouseJoint.GetBody2();
					m_world.DestroyJoint(m_mouseJoint);
					m_mouseJoint = null;
					if (clickedBody == m_bead.m_body)
					{
						m_bead.SetCollidable(true);
					}
					if (m_rope.IsAttachable( clickedBody))
					{
 
						var ground:b2Body = m_world.GetGroundBody();
						if (m_leftHook.IsTouching(clickedBody))
						{
							trace("test");
							m_hookJointDef.Initialize(m_leftHook.m_body,  clickedBody, new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
							if (m_leftHookJoint){
								m_world.DestroyJoint(m_leftHookJoint);
								m_leftHookJoint = null;
							}
							m_leftHookJoint = m_world.CreateJoint(m_hookJointDef);
						}
						else if (m_rightHook.IsTouching( clickedBody))
						{
							if (m_rightHookJoint){
								m_world.DestroyJoint(m_rightHookJoint);
								m_rightHookJoint = null;
							}
							m_hookJointDef.Initialize(m_rightHook.m_body,  clickedBody,  new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
							m_rightHookJoint = m_world.CreateJoint(m_hookJointDef);
						}
					}
				}
			}
 
			// mouse move
			if (m_mouseJoint)
			{
				var p2:b2Vec2 = new b2Vec2(mouseXWorldPhys, mouseYWorldPhys);
				m_mouseJoint.SetTarget(p2);
			}
		}
 
		//======================
		// Mouse Destroy
		//======================
		public function MouseDestroy():void{
			// mouse press
			if (!Input.mouseDown && Input.isKeyPressed(68/*D*/)){
 
				var body:b2Body = GetBodyAtMouse(true);
 
				if (body)
				{
					m_world.DestroyBody(body);
					return;
				}
			}
		}
 
		//======================
		// GetBodyAtMouse
		//======================
		private var mousePVec:b2Vec2 = new b2Vec2();
		public function GetBodyAtMouse(includeStatic:Boolean=false):b2Body{
			// Make a small box.
			mousePVec.Set(mouseXWorldPhys, mouseYWorldPhys);
			var aabb:b2AABB = new b2AABB();
			aabb.lowerBound.Set(mouseXWorldPhys - 0.001, mouseYWorldPhys - 0.001);
			aabb.upperBound.Set(mouseXWorldPhys + 0.001, mouseYWorldPhys + 0.001);
 
			// Query the world for overlapping shapes.
			var k_maxCount:int = 10;
			var shapes:Array = new Array();
			var count:int = m_world.Query(aabb, shapes, k_maxCount);
			var body:b2Body = null;
			for (var i:int = 0; i < count; ++i)
			{
				if (shapes[i].GetBody().IsStatic() == false || includeStatic)
				{
					var tShape:b2Shape = shapes[i] as b2Shape;
					var inside:Boolean = tShape.TestPoint(tShape.GetBody().GetXForm(), mousePVec);
					if (inside)
					{
						body = tShape.GetBody();
						break;
					}
				}
			}
			return body;
		}
 
	}
 
}

The first change is at the end of the constructor where I create the items that are in the world.

m_rope = new Rope(m_world, m_physScale);
 
m_leftHookJoint = null;
m_rightHookJoint = null;
 
m_leftHook = new Hook(m_world, m_physScale, new b2Vec2(175/m_physScale, 50/m_physScale), 10/m_physScale);
m_rightHook = new Hook(m_world, m_physScale, new b2Vec2(500/m_physScale, 50/m_physScale), 10/m_physScale);
m_hookJointDef = new b2RevoluteJointDef();
 
m_bead = new Bead(m_world, m_physScale, 7.5/m_physScale);

The second noticable difference is the addition of the member variables.

public var m_rope:Rope;
public var m_hookJointDef:b2RevoluteJointDef;
public var m_leftHookJoint:b2Joint;
public var m_rightHookJoint:b2Joint;
public var m_leftHook:Hook;
public var m_rightHook:Hook;
public var m_bead:Bead;

In the mouse drag function are the most important additions. In the first part of this function we are focusing on mouse presses, or an initiated drag. The first thing we do is check to see if we clicked on a joint at one of our hooking points, if so we want to destroy that joint and let the mouse move the necklace wherever it wants to. We also want the user to be able to click a bead and place it on the necklace without colliding with the necklace while trying to drag the bead, this is also check for here, we turn off collisions for a bead if it is being dragged.

\The second part of the mouse drag function is when the mouse button is relased after dragging has occurred. When this happens, we check to see if we were holding a bead, if we were we set it to be collidable, so that it will stay on the necklace if it was placed there. We are also checking to see if the right or left hook are clicked on with one of the end pieces of the necklace. If they are then we create a new joint between the hook and the necklace endpiece.

//======================
//======================
// Mouse Drag
//======================
public function MouseDrag():void{
	// mouse press
	if (Input.mouseDown && !m_mouseJoint){
 
		var body:b2Body = GetBodyAtMouse();
 
		if (body)
		{
			var md:b2MouseJointDef = new b2MouseJointDef();
			md.body1 = m_world.GetGroundBody();
			md.body2 = body;
			md.target.Set(mouseXWorldPhys, mouseYWorldPhys);
			md.maxForce = 600.0 * body.GetMass();
			md.timeStep = m_timeStep;
			m_mouseJoint = m_world.CreateJoint(md) as b2MouseJoint;
			body.WakeUp();
			if (body == m_rope.m_leftHook || body == m_rope.m_rightHook){
				if (m_leftHookJoint && body == m_leftHookJoint.GetBody2())
				{
						m_world.DestroyJoint(m_leftHookJoint);
						m_leftHookJoint = null;
				}
				if (m_rightHookJoint && body == m_rightHookJoint.GetBody2())
				{
						m_world.DestroyJoint(m_rightHookJoint);
						m_rightHookJoint = null;
				}
			}
			if (body == m_bead.m_body)
			{
				m_bead.SetCollidable(false);
			}
		}
	}
 
	// mouse release
	if (!Input.mouseDown){
		if (m_mouseJoint)
		{
			//get body being clicked on
			var clickedBody:b2Body = m_mouseJoint.GetBody2();
			m_world.DestroyJoint(m_mouseJoint);
			m_mouseJoint = null;
			if (clickedBody == m_bead.m_body)
			{
				m_bead.SetCollidable(true);
			}
			if (m_rope.IsAttachable( clickedBody))
			{
 
				var ground:b2Body = m_world.GetGroundBody();
				if (m_leftHook.IsTouching(clickedBody))
				{
					trace("test");
					m_hookJointDef.Initialize(m_leftHook.m_body,  clickedBody, new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
					if (m_leftHookJoint){
						m_world.DestroyJoint(m_leftHookJoint);
						m_leftHookJoint = null;
					}
					m_leftHookJoint = m_world.CreateJoint(m_hookJointDef);
				}
				else if (m_rightHook.IsTouching( clickedBody))
				{
					if (m_rightHookJoint){
						m_world.DestroyJoint(m_rightHookJoint);
						m_rightHookJoint = null;
					}
					m_hookJointDef.Initialize(m_rightHook.m_body,  clickedBody,  new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
					m_rightHookJoint = m_world.CreateJoint(m_hookJointDef);
				}
			}
		}
	}
 
	// mouse move
	if (m_mouseJoint)
	{
		var p2:b2Vec2 = new b2Vec2(mouseXWorldPhys, mouseYWorldPhys);
		m_mouseJoint.SetTarget(p2);
	}
}

There are three types that are used in the Playground.as file that we haven’t seen and are not in Box2D.  Those are the Rope, Bead, and Hook classes.

The rope is the most complicated and has the most new code.  When instantiated it builds itself vertically at a hard coded location.  It has an endpiece on each end that is different than the rest of the necklace.   Initially I made all pieces of the necklace a rectangle and connected them end to end.  This proved to be a mistake because when there was too much bending or twisting between two rectangle shapes at a joint they would sometimes not know how to fix themselves.  It would look like a permanent crease between the joint.  I was unable to figure out how to fix this problem with rectangles and I figured collisions between spheres is always cheaper so I moved everything to circles.

So the result is that the rope is made up of anywhere between 10 and 60 circles.  The values for the number of circles and the size of them is currently hard coded in the rope.as file.  Most of what is happening during the building of the rope is pretty obvious.  I changed a couple of the collision properties to make things look more realistic.

Each end piece is special and can be attached to a connector in the stage.  The isAttachable function checks to see if a given body can be attached to something.  I did this so it was easy to change which things could be attached in the future.

One last thing to note is that I made the end pieces collidable with the necklace pieces but the necklace pieces are not collidable with themselves.  This saves a lot of computation I’m sure but the main reason was so that you could more easily find an endpiece among a pile of necklace since it should be sitting on top.

Rope.as

import Box2D.Dynamics.*;
import Box2D.Collision.*;
import Box2D.Collision.Shapes.*;
import Box2D.Dynamics.Joints.*;
import Box2D.Dynamics.Contacts.*;
import Box2D.Common.*;
import Box2D.Common.Math.*;
 
public class Rope{
 
	public function Rope(world:b2World, physScale:Number){
 
		// Set block length & width & overlap(to make block connections look smoother)
		m_blockLength =3.5;
		m_blockWidth = 3.5;
		m_overlap = -4.5;
		// Set end piece paramaters
		m_circleRadius = 3;
 
		// Starting height for necklace
		m_startingHeight = -200; 
 
		var vel:b2Vec2 = new b2Vec2();
		vel.Set(5, 0); 
 
		var ground:b2Body = world.GetGroundBody();
		var i:int;
		var anchor:b2Vec2 = new b2Vec2();
		var body:b2Body;
 
		// Bridge
		{
			var sd:b2PolygonDef = new b2PolygonDef();
			sd.SetAsBox(m_blockLength / physScale, m_blockWidth / physScale);
			sd.density = 20;
			sd.friction = .8;
			sd.restitution = 0.0;
			sd.filter.groupIndex = -2;
 
			var cd:b2CircleDef = new b2CircleDef();
			cd.density = 20.0;
			cd.radius = (m_circleRadius) / physScale;
			cd.restitution = 0.0;
			cd.friction=0.8;
			cd.filter.groupIndex = -3;
 
			var bd:b2BodyDef = new b2BodyDef();
			bd.angle = -90 / (180/Math.PI);
			var jd:b2RevoluteJointDef = new b2RevoluteJointDef();
			const numPlanks:int = 50;
 
			jd.lowerAngle = -10 / (180/Math.PI);
			jd.upperAngle = 10 / (180/Math.PI);
			jd.enableLimit = false;
			jd.collideConnected = false;
 
			var prevBody:b2Body = ground;
 
			bd.position.Set(100 / physScale, (m_startingHeight + m_blockLength*2)/physScale)
			//bd.isBullet = true;
			body = world.CreateBody(bd);
			body.CreateShape(sd);
			body.SetMassFromShapes();
			body.SetLinearVelocity(vel);
			m_leftHook = prevBody = body;
			for (i = 0; i < numPlanks; ++i)
			{
				bd.position.Set(100 / physScale, (m_startingHeight + m_blockLength + (m_blockLength*2*i) - m_overlap*i) / physScale);
				bd.isBullet = false;
				bd.linearDamping = 0.1;
				bd.angularDamping = 0.5;
				body = world.CreateBody(bd);
				body.CreateShape(cd);
				body.SetMassFromShapes();
 
				anchor.Set(100 / physScale, (m_startingHeight + (m_blockLength*2*i) - m_overlap*i) / physScale);
 
				jd.Initialize(prevBody, body, anchor);
 
				world.CreateJoint(jd);
 
				prevBody = body;
			}
			bd.linearDamping = 0;
			bd.angularDamping = 0;
			//bd.isBullet = true;
			bd.position.Set(100/physScale, (m_startingHeight + m_blockLength + m_blockLength*2*numPlanks - m_overlap*numPlanks) / physScale);
			body = world.CreateBody(bd);
			body.CreateShape(sd);
			body.SetMassFromShapes();
			m_rightHook = body;
			anchor.Set(100 / physScale, (m_startingHeight + (m_blockLength*2*numPlanks) - m_overlap*numPlanks) / physScale);
			jd.Initialize(prevBody, body, anchor);
			world.CreateJoint(jd);
 
		}
 
	}
 
	public function IsAttachable(body:b2Body) : Boolean {
		if (body == m_leftHook || body == m_rightHook)
		{
			return true;
		}
		else
		{
			return false;
		}
	}
 
	//======================
	// Member Data
	//======================
	public var m_blockLength:Number;
	public var m_blockWidth:Number;
	public var m_circleRadius:Number;
	public var m_overlap:Number;
	public var m_startingHeight:Number;
	public var m_leftHook:b2Body;
	public var m_rightHook:b2Body;
 
}

Creating a bead class that could be placed on the necklace was one thing I was unsure of how to do initially. The end result for my prototype was to create three circle shapes on a single body and to make the center one a sensor. A sensor is something that can collide with other objects but doesn’t cause any actual physical response. It is essentially not collidable but still keeps track of its collisions.

So the constructor creates this complex body. I found very little documentation on how to make a simple complex body out of simple shapes online. It was very easy though and if you look at the bead.as constructor you can see how I did it. The most confusing part was how I was supposed to offset the different shapes in the body. This is accomplished via the Shape class’ localPosition attribute. By modifying this attribute I could offset the shape in the body it will eventually be a part of.

The last thing in this class is the setCollidable function. I needed a way to easily make all of the parts of the bead a sensor temporarily. This was so that the user could drag the bead and not collide with anything and then let go and it would become collidable again.
Bead.as

import Box2D.Dynamics.*;
import Box2D.Collision.*;
import Box2D.Collision.Shapes.*;
import Box2D.Dynamics.Joints.*;
import Box2D.Dynamics.Contacts.*;
import Box2D.Common.*;
import Box2D.Common.Math.*;
 
public class Bead{
 
	public function Bead(world:b2World, physScale:Number, radius:Number){
 
		m_circleRadius = radius;
 
		var cd:b2CircleDef = new b2CircleDef();
		cd.density = 0.01;
		cd.radius = (m_circleRadius);
		cd.friction = 0.1;
		cd.restitution = 0.1;
 
		var bd:b2BodyDef = new b2BodyDef();
 
		cd.localPosition.x = -15 / physScale;
 
		bd.position.Set(400/physScale, 200/physScale)
		bd.isBullet = true;
		m_body = world.CreateBody(bd);
		m_body.CreateShape(cd);
		cd.localPosition.x = 15 / physScale;
		m_body.CreateShape(cd);
		cd.localPosition.x=0;
		cd.isSensor = true;
		m_centerShape = m_body.CreateShape(cd);
		m_body.SetMassFromShapes();
 
	}
 
	public function SetCollidable(toggle:Boolean) : void{
		var currentShape:b2Shape = m_body.GetShapeList();
 
		for(currentShape; currentShape; currentShape = currentShape.m_next){
			if (currentShape == m_centerShape){
				continue;
			}
			currentShape.m_isSensor = !toggle;
		}
	}
 
	//======================
	// Member Data
	//======================
 
	public var m_circleRadius:Number;
	public var m_position:b2Vec2;
	public var m_body:b2Body;
	public var m_centerShape:b2Shape;
 
}

The hook class is very simple and just creates a circle in the location where it is supposed to be. I temporarily just used the physics circle to make it easy to test. I don’t plan to have a physics body be the connector piece in the future unless I am using so sort of callback function. I think it is easy enough to use the isTouching function to check if a body is within touching range of this hook.
Hook.as

import Box2D.Dynamics.*;
import Box2D.Collision.*;
import Box2D.Collision.Shapes.*;
import Box2D.Dynamics.Joints.*;
import Box2D.Dynamics.Contacts.*;
import Box2D.Common.*;
import Box2D.Common.Math.*;
 
public class Hook{
 
	public function Hook(world:b2World, physScale:Number,  position:b2Vec2, radius:Number){
 
		m_circleRadius = radius;
		m_position = position;
 
		var cd:b2CircleDef = new b2CircleDef();
			cd.density = 1.0;
			cd.radius = (m_circleRadius);
			cd.isSensor = true;
			var bd:b2BodyDef = new b2BodyDef();
 
			bd.position.Set(m_position.x, m_position.y)
			m_body = world.CreateBody(bd);
			m_body.CreateShape(cd);
 
	}
 
	public function IsTouching(body:b2Body):Boolean{
		var position = body.GetWorldCenter().Copy();
 
		position.Subtract(m_position);
		position.Abs();
		trace(position.x + " x units away.");
		trace("radius of circle is " + m_circleRadius);
		if (position.x < m_circleRadius && position.y < m_circleRadius)
		{
			return true;
		}
		else
		{
			return false;
		}
	}
	//======================
	// Member Data
	//======================
 
	public var m_circleRadius:Number;
	public var m_position:b2Vec2;
	public var m_body:b2Body;
 
}

The resulting application turned out pretty good. I was not totally satisfied with placing a bead on the necklace, but I ran out of time and had to do real work by that time. I think I could make the bead automatically orient correctly and highlight itself when it is being placed on the right spot on the necklace. Besides that problem I am satisfied with the it as a prototype for a future application I might develop.

Here it is:

Beadr Prototype 1:

The biggest problem for this week was actually making myself sit down and write something on here.  I don’t think that would be very interesting to readers or my future self so I’ll fore go the discussion of that.

I think the first thing I should do is explain my background.  I am a Ph.D. candidate at the University of South Carolina (USC) working in the ARENA for Research on Emerging Networks and Applications (ARENA) lab.  Here I focus on developing new wireless network protocols in an endless quest to publish papers.  I have been with Dr. Srihari Nelakuditi since about 2004, when he hired me as an undergraduate researcher.  We have presented multiple papers together.  One was presented at MobiCom and can be found in the Mobile Computer Communications Review (MCCR), it is on security in Mobile Ad Hoc NETworks (MANETs).  Another notable paper was presented at the WiMesh workshop 2008 at SECON, it is not yet available online and focused on bit-rate selection in oppotunistic routing.  The second paper was also the focus of my Master’s Thesis.

During my time as a student at USC, I have worked on quite a few side projects.  Some of these were related to school and some could be seen as distractions.  I spent some time doing contract work on a health-care data system for a director of marketing of a contract company for nursing homes.  I have since used my experience in this field to try to create a large scale public web application providing similar information.  The project is being developed in Ruby on Rails, my first large scale project using the framework.  I have been very happy with it for the most part, and I hope that I can provide some useful insight into problems I have faced while programming with it.

During this Summer of 2008, though, I took a break and had a tour at a nameless agency in the Maryland area doing computer science research .   This was very fun, but nonetheless the day to day government work motivated me even further to pursue my goal of having my own business.  I know (and am told very frequently) that this is a pipe dream and every young computer scientist wants to have their own company, but I feel like I have the motivation and drive to bring a reasonable idea to fruition.  If the completion of my plans does not bring any type of success then I will simply move on.  The key point is that I do plan to have an operational business before deciding that it is not successful.  I believe this is the reason most technical start-ups fail.  They don’t realize how much work is involved in creating their envisioned product.

I have also spent some time while at school developing games.  See, this was my first love with computers.  I fully expected to be a video game developer by the time I graduated college.  There are multiple things that changed my mind about this goal.  The first is that I feel like the industry has totally changed.  This might be due to many reasons.  My observations lead me to believe it is a combination of the following

1. The large emerging market of the casual gamer, usually on a console.
2. Companies like EA buying up every company they can and losing all creativity in their games for the sake of optimization.
3. The failing PC market for games, which prevents the small developers from getting a market unless they can get their game on one of the major consoles.

So, I feel like the video game market is no longer somewhere I would like to work, besides a select few companies.  The competition for getting into the industry is enormous.  I feel like I could find a very enjoyable job in normal software development that pays much better, has less stress, and does not require me to do a little dance for the hiring guys culling through hundreds of potential code monkeys.  Regardless, I still took a game development class this past semester.  Despite having a five person team, it ended up giving me much needed experience with slack teammates.  I’ve had slack teammates before, but most of the time it was because they were either incapable of creating decent code, or they just didn’t care about school in general.  This graduate class was different though, in that my teammates were perfectly capable of accomplishing what I tasked them with.  I believe that the problem was that they saw that I was also capable of doing it myself, so any work they didn’t finish was not going to cause them to get a bad grade.  The class ended well, our game turned out OK, but the team was very unsatisfied with themselves for being so uninvolved during the semester.  I’m generalizing about their lack of work, not every team member was horrible, I can think of two that I would have given F’s to though.

If you’ve read this far, you might have noticed something about the way I write and think.  I move very quickly from one topic to the next.  I often find myself talking and something comes up in my head and it seems like it would be more time efficient to talk about that instead so I simply move to that topic without a very smooth transition.  I’m afraid if I simply free write that is how it will turn out.  If I’m writing a technical article, I can rewrite it as many times as I need to.  So there will probably be two different types of writing on this blog.

Thank you for reading.  In my next post I am hoping to go over some problems I have faced in Rails and my current solutions to the problems.  Hopefully I can get some feedback and maybe make some improvements.

-Chase