I searched thoroughly for an easy way to retrieve information from this set of .ADT files I was dealing with. Sybase provides tools to work with the data, but all of them require using their software or drivers developed by them. I would have settled for this if it wasn’t such a hassle to figure out how to use them. I asked the question on Stackoverflow.com and got an answer from one of the employee’s of Sybase. While helpful, the suggested Perl driver was one of the things I was avoiding. I did attempt it and after a little hassle, moved on to creating a simpler solution.

One solution was to simply use Sybase’s provided “Advantage Data Architect” software to open the .ADT files and export them all to SQL. This is possible with their software, but it would not work for my situation. The developers of the program outputting these .adt files used a separate embedded database for each new report that was being worked on. This meant there was hundreds of folders, each containing 30-40 .ADT files making up their own embedded database. I needed something that could automate the extraction of this data.

Data Format

To cut this short, I decided to open up the .ADT files with a hex editor and see if I could get the information I needed out. I only needed the column names, their types, and the actual rows of data. To aid anybody else doing something similar to this, I’ll include some of my notes about the format here:

• The data is stored at the very end of the file.
• The byte offset when the data starts is located at byte 32 and is 32-bits
• Byte 24 contains a 32-bit integer specifying the number of rows
• Byte 36 contains the a 32-bit number representing the number of bytes each record consumes
• The header is 400 bytes
• Column information is after the header, before the data. Each column entry is 200 bytes.
• The number of columns can be calculated (data_offset-400)/200
• Column names consume the first 128 of each column entry.
• Byte 130 contains a 16-bit integer representing the type of data.
• Byte 136 contains a 16-bit integer, the length of the field in bytes
• Character = 04, Double = 0A, Integer = 0B, Shortint = 0C, CICharacter = 14, Date = 03, Time = 0D, Timestamp = 0E, Autoinc = 0F

This was enough information to get what I wanted.

Ruby-ADT Gem

I took the existing DBF gem by Keith Morrison. Most of the credit for this gem goes to him as I borrowed the structure heavily from his gem.

I’ll provide an example of what is now possible with my new ADT gem. No drivers are required and it’s as simple as installing the gem. I’ll include a snippet of the readme from my Ruby-ADT gem on github, borrowed heavily again from the DBF gem.

Installation

gem install ruby-adt

Basic Usage

Load an ADT file:

require 'rubygems'
require 'adt'
 
table = ADT::Table.new("test.adt")

Enumerate all records

table.each do |record|          
	puts record.name          
	puts record.email
end

Load a single record using record or find

table.record(6)
table.find(6)

Attributes can also be accessed through the attributes hash in original or underscored form or as an accessor method using the underscored name.

table.record(4).attributes["PhoneBook"]
table.record(4).attributes["phone_book"]
table.record(4).phone_book

Search for records using a simple hash format. Multiple search criteria are ANDed. Use the block form of find if the resulting recordset could be large otherwise all records will be loaded into memory.

# find all records with first_name equal to Keith
 
table.find(:all, :first_name => 'Keith') do |record|  
	puts record.last_name
end
 
# find all records with first_name equal to Keith and last_name equal to Morrison
table.find(:all, :first_name => 'Keith', :last_name => 'Morrison') do |record|   
	puts record.last_name
end
 
# find the first record with first_name equal to Keith
 
table.find :first, :first_name => 'Keith'
 
# find record number 10 
 
table.find(10)

Migrating to ActiveRecord

An example of migrating a DBF book table to ActiveRecord using a migration:

require 'adt'
 
class CreateBooks < ActiveRecord::Migration      
	def self.up            
		table = ADT::Table.new('db/adt/books.adt')            
		eval(table.schema)        
		table.each do |record|                  
			Book.create(record.attributes)            
		end      
	end  
	def self.down        
		drop_table :books    
	end
end

Limitations and known bugs

• ADT is read-only
• External index files are not used
• Dates are not currently handled correctly.

Future Directions

I am going to continue working on this library. I have already heard from a few others that found this useful immediately after I posted it online. I was surprised people could find it so quickly. I’m sure there are bugs. I would appreciate any problems to be reported to the Ruby-ADT Github page. Emails are also welcome, but any issues should be directed to the github page and I’ll get to them when I can.

Hope this helps somebody out there dealing with this format.

- Chase Gray

Prerequisites

The first thing I needed to do was to successfully configure and compile mdbtools. This probably took the longest as there are a few different version out there. I had the most success with these instructions from the author’s git repository, http://www.automatthew.com/2008/02/how-to-compile-mdbtools-on-mac-os-x.html?showComment=1219246620000. I was also able to compile this version on my ubuntu machine pretty easily. The biggest hangup was with a couple of dependencies. I also had a weird error where the autogen.sh was not detecting my libtools library, but I knew this was installed. I commented out this check in the autogen.sh and things all worked, luckily.

Understanding mdb-tools was not very difficult. It contains various libaries for interfacing with it through C, etc. The only thing I cared about were the command line tools that it provided, listed here. The commands I will use in this post are mdb-tables, mdb-schema, mdb-export.

The other requirement is to have a MySQL table created ready with permissions, etc. that will end up holding the data from the combined .mdb files.

Step 1: Extracting the Schema

One important thing to note about my situation is that all the .mdb files have the same schema. Knowing this, I was able to simply find a random .mdb file and base my MySQL schema on this.

I have my .mdb files stored in a shared directory and so within my Rake task I used the following code to grab the first one (I’m sure there is a better way).

first_mdb_file = ''
#get first mdb file
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    first_mdb_file = "#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file}"
    break
  end
end

Knowing the path to the first .mdb file the schema can be extracted by getting the result of the following shell command (the code surrounded in backticks is executed on the command line of the system). The -S flag tells it to santize some of the table names by replacing spaces with underscore, etc. I needed this for my tables because there were some problems without it.

schema_sql = `mdb-schema -S #{first_mdb_file} mysql`

Step 2: Building the Tables

Since I’m doing this in a Rake task in Rails (you could do this in plain ruby), I am using an ActiveRecord connection to interface with my MySQL database. I have a class called ExternalDatabase that allows me to connect to a different database without modifying the one that my main Rails program uses. I will list it below, I don’t remember exactly where I got it from.

#Class used to connect to external databases without modifying the exiisting connection used by Rails
class Data::ExternalDatabase < ActiveRecord::Base
  self.abstract_class = true
 
  def self.set_source(configuration_name)
    source = ActiveRecord::Base.configurations["sources"][configuration_name.to_s]
    establish_connection source
  end
 
  def self.set_configuration(config)
    source = config
    establish_connection source
  end
end

As you can see, I also have a section in my database.yml where i list connection details for various source databases. This isn’t related to this post though.

So, assuming we have this class, we can use the following code to load our new schema into out MySQL database.

Data::ExternalDatabase.set_configuration(:adapter  => "mysql",
:host     => "localhost",
:username => YOUR_DATABASE_USERNAME,
:password => YOUR_DATABASE_PASSWORD,
:database => YOUR_DATABASE_NAME,
)
db =  Data::ExternalDatabase.connection 
 
puts "adding if exists clause to drop tables"
 
schema_sql = schema_sql.gsub(/DROP TABLE (.*);/, 'DROP TABLE IF EXISTS \1;')
schema_sql = schema_sql.gsub(/--.*$/, '')
puts "executing schema sql"
lines = schema_sql.split(';')
 
lines.each do |sql_line|
  next if sql_line.blank?
  db.execute(sql_line)
end

The first part builds my database configuration hash that I pass in to ExternalDatabase. I save the connection into a variable called ‘db’. One of the important things to note about this and mdb-tools is that it does not check if the table currently exists before trying to drop the table. So I added a snippet of code that removes the comments and adds ‘IF EXISTS’ to the ‘DROP TABLE’ clause in the schema sql.

For some reason, trying to execute this entire schema sql string with db.execute causes various errors that I couldn’t figure out. By separating the schema sql into individual sql statements terminated with semicolons seemed to fix this problem. The last bit of code does the separation into lines and executes the sql on the database. After all this, the database should have the correct schema.

Step 3: Load Data

One of the last things we want to do is actually load all the data from the different mdb files into the single MySQL database.

tables = (`mdb-tables #{first_mdb_file}`).split(' ')
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    puts "Current file: #{mdb_file}"
    tables.each do |table|
      puts "Current table: #{table}"
      insert_sql = `mdb-export -I -S #{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file} #{table}`
      insert_sql = insert_sql.gsub(/;/, ',')
      insert_sql = insert_sql.gsub(/(.*?\))(\s+INSERT)/, '\1;\2')
      lines = insert_sql.split(';')
      lines.each do |sql_line|
        next if sql_line.blank?
        db.execute(sql_line)
      end
    end
  end
end

So the first line runs mdb-tools command line tool ‘mdb-tables’ to get a list of the tables separated by spaces. I could have done this through my previously created schema as well.

This code goes through every .mdb file in my directory and uses mdb-tools to extract the data as sql ‘INSERT’ statements from the command-line.

In the inner-most loop, I clean up the returned value from mdb-export because it does not include semi-colons after each INSERT statement. I also remove any semi-colons before adding any because some of my fields had semi-colons in them. Again, I separate each INSERT statement into a single statement and execute these individually in the last section of the code.

After all this you should have all your data combined into a single MySQL database. One problem might exists though. Certain tables, such as tables that contain general purpose labels, etc. for other tables to refer to, might contain duplicates. This wouldn’t have caused a big problem for me because I just needed to extract data, but if you needed to continue using this database you would need to remove the duplicates.

Final Step: Removing Duplicate Rows

There seem to be a lot of different ways to remove duplicate rows from a MySQL table. I wish there was a simple solution, but I couldn’t find one. The following code does the following:

1. Create a temporary table that has the same schema as the table we are removing duplicates from.
2. Insert all distinct rows into this new temporary table.
3. remove all values from the original table.
4. Insert all of the rows from the temporary table back into the original table.
5. Finally, drop the temporary table.

tables.each do |table|
  db.execute("CREATE TABLE temp LIKE #{table}")
  db.execute("INSERT INTO temp select distinct * FROM #{table}")
  db.execute("TRUNCATE #{table}")
  db.execute("INSERT INTO #{table} select * from temp")
  db.execute("DROP TABLE temp")
end

Thanks for reading, I hope this was helpful to others that have some similar problems.

- Chase

I found myself working with some data in a hash generated from scraping a website. The website had some duplicate content and I didn’t feel like it was necessary to clutter my scraping code with tests for duplicate content while it was scraping. After scraping the data and turning it into a hash, I wished to remove the duplicates with some post-processing. I felt like this should be pretty simple, except that the elements I wanted to test for uniqueness contained a deeply nested structure of hashes and arrays. Here is an example an element in an array I wanted to uniqify:

{:name => 'bob', :posts => [{:title => 'First Post!' :comments => [{:commenter => 'Alice', :text => 'Hi bob!'}]}]}

The #uniq method does not work as expected when there is a Hash element in the array, it only uses the object_id to determine uniqueness. There are some solutions online for dealing with this.

If you know that each element inside the array is a hash with no sub-hashes you can use the solution presented in the answer of this stackoverflow.com question. The first solution uses #inject to build a new array with only unique items. Whether an item is included in the array already or not is determined with the Hash’s #include method.

a.inject([]) { |result,h| result << h unless result.include?(h); result }

The second solution presented in the same post:

a.map {|h| h.to_a[0]}.uniq.map {|k,v| {k => v}}

This flattens all of your hashes into an array. It then calls uniq on the resulting array after all the hashes are flattened. This means that for an item to be unique it must have the same key AND value. After finding the unique elements, it then maps the items back into a hash.

In this post, Mike Burns solve this problem by redefining #hash and #eql?. If you are doing this kind of comparison often, this would probably be a better approach. He does note though, that by doing this the #dup method no longer works as expected.

So it was obvious through searching that I was going to have to figure this out on my own. I was a little worried that I would have to implement an ugly recursive solution that searches the entire tree to find all duplicates. While looking into this solution, I felt that it would be very nice if #uniq would accept a block that would allow me to return true if the given parameter is a duplicate, thus removing it from the returned array. This is not implemented in my current ruby version (1.8.6), but there is an example of how to do this on this blog post. Hopefully, more (all) of these methods in the ruby core will accept blocks someday.

I began to write a method that would search my tree-like structure of hashes and arrays and I felt like it was a very ugly solution. I wanted something simple that would just take a line or two and get the job done. So I showed an example of the problem to my non-computer scientist wife and she said, “why don’t you just compare how the two lists look when printed out?” I was a little shocked that I hadn’t thought of this. It isn’t the most efficient solution and it relies on the #inspect method, which is probably pretty slow because it has to determine the structure of all the hashes in order to print them in a readable format. The old solution I was cooking up would also have to traverse the tree, similar to what I expected #inspect was doing. In addition to the #inspect call, there would be a string comparison for each element. I was willing to accept both of these for a short elegant solution to my problem.

The issue that came to my mind first was whether hashes get printed out the same way every time. In Ruby 1.9, it seems that Hashes are going to be stored in the same order that they were entered. In previous versions, I’m assuming that they are stored in the order of their generated hash. This could cause a problem if the data someone was entering could be in different orders. For my purposes though, the data would be entered in the same order if it was a duplicate.

So, after all of that, to uniqify an array containing a deeply nested structure, I present the following solution:

class Array
  def nested_uniq
    self.delete_if {|item| (self - [item]).any? {|other_item| item.inspect == other_item.inspect}}
  end
  def nested_uniq!
    self.replace self.nested_uniq
  end
end

I’ll walk through it because it may seem a bit hard to understand at first. I go through every element in the array with delete_if to determine if the element should be deleted from the array. Then, for all elements except the one in question, I check to see if #any? of them are equivalent to the element in question. This equivalence is determined by doing a string comparison of what the #inspect method returns for that element.

Here is an IRB test

>>  test = [{:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}, {:to => '1', :yeah => '8'}, {:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}]
=> [{:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}, {:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]
>> test.delete_if {|item| (test - [item]).any? {|other_item| item.inspect == other_item.inspect}}
=> [{:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]

Hope this is helpful to someone! I am very interested to hear suggestions and comments on alternative ways to uniqify an array with elements that have a nested structure.

-Chase

I’m sure an application focusing more on URLs is somebody’s 10% project somewhere sitting on the backburner until they have time to finish it.  It seems like almost anything you’d want to do has a corresponding web application.  When I come across a need that doesn’t I just feel like that void should soon be filled.  I’m going to outline the application that I believe would fill this void.  Maybe in future posts I’ll also walkthrough my attempts to do it as my personal .5% project.

Doesn’t this already exist?

When giving the quick summary of what I envisioned for the project the first question I get is, “Doesn’t Google Cache already do that?”  My initial reaction is that if it did, I would already be using it for that purpose.  After thinking about it, it doesn’t seem like that far of a stretch for Wayback Machine or Google Cache to archive URLs as well.  I feel like the real kicker is that in order to make it useful there would have to be extra processing and utilities for the URL archive.  The information is already there, so what needs to be done is create an interface for analyzing the URL a site has.  Before talking about the ideal solution, first let’s look at what is offered by Google Cache and Wayback Machine already to compare.

To get this sort of functionality out of Google Cache as far as I know is to use a “site:” search.  So, typing in the search, “site:arena.cse.sc.edu”, I get the following:

Result for site:arena.cse.sc.edu

I know that our research group’s website has lists of most of the pdfs that we have published as well a number of static pages that total more than the 17 that are listed in these results.  Of course, this site isn’t really optimized for search engines, so that might also not be helping.  Searching for just pdfs returns a measly 7 results.  Some of our papers are listed as links to other sites such as IEEE, which causes them to not get listed in these results.  Alternatively, if I could somehow easily browse all the URLs that exists for this domain, I might be able to see more information about what the website actually contains internally (more on this later).  Another experiment was to search for, “site:chase.ratchetsoftware.com”, and because this site is optimized for search engines, most of the URLs that exist were found pretty easily.  There is still no way to really tell what the site contains as an overview without going to the site or reading the provided description.  Put another way, it would be interesting to be able to see a something like a content tree that lets a viewer easily see what is contained in the site with a quick glance.  These are all things I believe can be solved and would be useful, but Google Cache and Wayback Machine are not the tools to do it.

What would an ideal solution look like?

I’ve given a few examples where Google Cache and Wayback Machine don’t offer what I am looking for.  I think to improve my argument it is best to talk about what the URL archive system would be in a perfect world.  It’s hard to see the difference in something that doesn’t yet exist.

I am going to list the features I have in mind and then explain any that aren’t obvious:

1. Provide a list of all URLs ever found for a given domain.
2. When a user views a list, it should check and display whether that URL is still valid.
3. Organize the list of URLs so that it is tractable by a normal user.  It should have collapsible SubURLs so that only top level URLs are in the master list.
4. Provide a filter so that only URLs matching the given filter are shown.
5. Provide the ability to click on a URL to make that the new root URL for our list.
6. A utility that can display a graphical tree view of the page and its links.  If the page links to an outside page, then show the link with the icon to that page if available.  This view would ideally provide enough information and “graphics” to give the user a good idea of what is available on the site.
7. If the site uses dynamic URLs, perhaps for a site written in rails or php, there should be some sort of analysis of the GET arguments from what the system can figure out.  This might be an area where users can submit additional information to improve the analysis of a web applications generated URL addresses.
8. One of the coolest ones.  A firefox plugin that automatically helps you when trying to figure out URLs based on what is stored in the URL Archive.  So if you’re trying to modify a URL slightly, you can get real-time help and see which URLs are still valid, etc.  Firefox is good right now with it’s new location bar that automatically searches your history and all that stuff.  It’d be even better if it could help you with long URLs that you haven’t typed in before!
9. Another plugin idea would be to provide screen readers easier access to a site’s content.  If all we had are the URLs and perhaps keywords for those URLs, a blind person might be able to more easily find content quicker.
10. One more plugin would be to have a sidebar while browsing sites, and as you browse you get an updated list of URLs that have existed at your current location on the site.  This would put a sidebar on your left with all the links you might see on that site and all the URLs that you might not see but existed in the past.  This would be useful for quickly navigating huge sites, especially ones that are moving their links around a lot.
11. The last idea I can think of right now is that the system could analyze how the URLs are changing and suggest to users URLs that may be used in the future or are currently used but not linked to based on the patterns in  the current URLs and their change history.  This might be useful if someone is trying to URL hack but doesn’t have any idea where to start.

I feel like number six is the one that needs further explanation in order to seem as cool as it does to me.  I believe there are already existing tools that will show you a tree view of a website.  There are definitely tools to show you this information for your own site, cause I’ve used them before.  So how does number six differ from these tools?  The answer is that it doesn’t differ dramatically, but hopefully it would offer more information because the purpose is slightly different.  Let’s give an example of a user that is looking for a site that offers ebooks in pdf form online somewhere.  The user might go to various sites by searching through Google and waste time at many sites that seem to offer ebooks but don’t have that large of a selection.  If the user could easily view a graph of each site before visiting, and turned on a filter for “.pdf”, he could hopefully see a tree that branches out into many different URLs leading to ebooks he’s interested in.  He should be able to quickly get an idea of the quantity, quality, and link validity of the site’s offerings.  Each node could be clicked on to make that node the new root node in the tree to perhaps get more specific information about a certain path.   There are many possibilities for graphically browsing a site’s URLs and URL relationships.  To me, this seems like a separate application entirely.  It might be that the site should be developed with a SOAP or REST interface and another application could be developed that provides this tree view of websites.

Potential problems and Questions

Not all good things come from having a URL Archive.  All the security concerns that come with Google Cache and Wayback Machine will also affect the URL Archive.  Some information that people wished to remove from the internet by unlinking it might be affected in a negative way by a system that indexes URLs forever.  Analyzng the GET parameters of a web application might be good for some users, but a malicious user could probably use this information to make the website work as it wasn’t intended.  Most of these problems come with developing this type of application.  There are tradeoffs and we just need to try to do our best to mitigate them while still providing a good service.

Another issue is the fact that I have no idea how to store such a large amount of information as this system would require.  I don’t even know where to begin with the indexing of URLs on the Internet.  It seems like it would take way too long with this little virtual machine on a server somewhere that I have.  I’d love to hear some comments about this.  How would one create an application like this that has to store such large amounts of information and process so much new data all the time?  Some resources relating to this would probably make for some great bathroom pulp .

Call for help

In my current situation I have way too much on my plate to try to develop an application for which there would be no immediate gain.  I need to work on my own business to try to get that going before I have to get a real job .  I also have an seemingly endless amount of Ph.D. work.  Despite this, I would more than willing to contribute to a project that was working on this solution.  I would even be happy to set up a Rails application on this server and with a Git repository if there were a couple developers interested in working on it.  All I’d need to distract me from my other work is a little motivation from other people’s interest, so please let me know in the comments if this is something you’d like to use or have needed in the past.

Thanks for reading,

- Chase Gray

Of course the first question I had was what physics libraries are available for Flash these days.  I had some experience in the past using Newton and various wrappers in C++ for a game project.  My experience with Newton would end up helping me pick up a new physics engine relatively easy.

There were a couple of choices to choose from and I started my search by reading some reviews found on GeekGlue. From this I was able to reduce my search to the Actionscript Physics Engine (APE) and Box2DFlashAS3.  APE seemed much easier to get started with, but I wasn’t sure if it would accomplish what I wanted without a lot of massaging.  After further reading on some forums, I found that the rope simulation in APE is much more limited because there is only one type of joint available at this time.  Box2D has 6 different types of joints and their provided rope bridge looked a little better.  The fact that Box2D had ragdoll physics examples made me feel that I would have as much control as I needed to create the necklace simulation I wanted.

After choosing Box2D I wanted to take their rope “springy” bridge demo and see if that could be easily turned into some sort of free falling rope.  This was not as simple as adding links and changing a few variables like I had hoped.  If you simply add links to the rope, after adding too many the iterative solver breaks the joints with the default configuration.  I had to search for a while to figure out why this was.  After some searching, I found that a joint can only handle about a 10:1 weight ratio before giving out.  The solution to this is the increase the number of iterations of decrease the step time.  This of course will slow down your simulation depending on how much you have to increase the number of iterations.  I had to change mine from about 10 to 40.  My simulation was still running at a decent speed but in the future I am not sure how this will hold up with more physics bodies flying around.  This was a huge problem for a while for me and I was very glad to finally find a solution.  If you have joints that are breaking due to too much force being applied to them is too much without adding more iterations to the engine so it can accurately solve the joint forces.  I hope someone with a similar problem can find this post because it took me a while to find a solution on a forum somewhere.

I am not concerned with creating a nice looking application at this time since this was just a weekend project I was doing on the side.  The goals for the prototype of my necklace simulator I wanted to have the following:

1. A rope like necklace that could be moved around by the user with realistic physics.
2. Two end pieces on the necklace that were different than the rest of the necklace and could be connected to things.
3. Two or more connectors in the world that could have a necklace endpiece connected to them and unattached when the user wants to move the necklace again.
4. A prototype of a bead which can be moved by the user and placed on the necklace.
5. A background of a neck or something so that it is easy to see what the necklace would look like on a person.

For my resulting application, I simply modified the default Test class provided with Box2DFlashAS3 to have instances of the objects in the world such as the rope and beads.  Previously it was a base class for various tests.  This might not be the best solution, but it was fine for my prototype to see if things would work.  I’ll include the entire source here and highlight my changes as well.

Playground.as

/*
* Copyright (c) 2006-2007 Erin Catto http://www.gphysics.com
*
* This software is provided 'as-is', without any express or implied
* warranty.  In no event will the authors be held liable for any damages
* arising from the use of this software.
* Permission is granted to anyone to use this software for any purpose,
* including commercial applications, and to alter it and redistribute it
* freely, subject to the following restrictions:
* 1. The origin of this software must not be misrepresented; you must not
* claim that you wrote the original software. If you use this software
* in a product, an acknowledgment in the product documentation would be
* appreciated but is not required.
* 2. Altered source versions must be plainly marked as such, and must not be
* misrepresented as being the original software.
* 3. This notice may not be removed or altered from any source distribution.
*/
 
package TestBed{
 
	import Box2D.Dynamics.*;
	import Box2D.Collision.*;
	import Box2D.Collision.Shapes.*;
	import Box2D.Dynamics.Joints.*;
	import Box2D.Dynamics.Contacts.*;
	import Box2D.Common.*;
	import Box2D.Common.Math.*;
	import Main;
	import General.Input;
 
	import flash.utils.getTimer
	import flash.display.*;
 
	public class Playground{
 
		public function Playground(){
 
			m_sprite = Main.m_sprite;
 
			var worldAABB:b2AABB = new b2AABB();
			worldAABB.lowerBound.Set(-1000.0, -1000.0);
			worldAABB.upperBound.Set(1000.0, 1000.0);
 
			// Define the gravity vector
			var gravity:b2Vec2 = new b2Vec2(0.0, 10.0);
 
			// Allow bodies to sleep
			var doSleep:Boolean = true;
 
			// Construct a world object
			m_world = new b2World(worldAABB, gravity, doSleep);
			// set debug draw
			var dbgDraw:b2DebugDraw = new b2DebugDraw();
			//var dbgSprite:Sprite = new Sprite();
			//m_sprite.addChild(dbgSprite);
			dbgDraw.m_sprite = m_sprite;
			dbgDraw.m_drawScale = 30.0;
			dbgDraw.m_fillAlpha = 0.3;
			dbgDraw.m_lineThickness = 1.0;
			dbgDraw.m_drawFlags = b2DebugDraw.e_shapeBit | b2DebugDraw.e_jointBit;
			m_world.SetDebugDraw(dbgDraw);
 
			// Create border of boxes
			var wallSd:b2PolygonDef = new b2PolygonDef();
			var wallBd:b2BodyDef = new b2BodyDef();
			var wallB:b2Body;
 
			// Left
			wallBd.position.Set(-95 / m_physScale, 360/m_physScale/2);
			wallSd.SetAsBox(100/m_physScale, 400/m_physScale/2);
			wallB = m_world.CreateBody(wallBd);
			wallB.CreateShape(wallSd);
			wallB.SetMassFromShapes();
			// Right
			wallBd.position.Set((640+95) / m_physScale, 360/m_physScale/2);
			wallB = m_world.CreateBody(wallBd);
			wallB.CreateShape(wallSd);
			wallB.SetMassFromShapes();
			// Top
			wallBd.position.Set(640/m_physScale/2, -95/m_physScale);
			wallSd.SetAsBox(680/m_physScale/2, 100/m_physScale);
			//wallB = m_world.CreateBody(wallBd);
			//wallB.CreateShape(wallSd);
			//wallB.SetMassFromShapes();
			// Bottom
			wallBd.position.Set(640/m_physScale/2, (360+95)/m_physScale);
			wallB = m_world.CreateBody(wallBd);
			wallB.CreateShape(wallSd);
			wallB.SetMassFromShapes();
 
			m_rope = new Rope(m_world, m_physScale);
 
			m_leftHookJoint = null;
			m_rightHookJoint = null;
 
			m_leftHook = new Hook(m_world, m_physScale, new b2Vec2(175/m_physScale, 50/m_physScale), 10/m_physScale);
			m_rightHook = new Hook(m_world, m_physScale, new b2Vec2(500/m_physScale, 50/m_physScale), 10/m_physScale);
			m_hookJointDef = new b2RevoluteJointDef();
 
			m_bead = new Bead(m_world, m_physScale, 7.5/m_physScale);
 
		}
 
		public function Update():void{
 
			// Update mouse joint
			UpdateMouseWorld()
			MouseDestroy();
			MouseDrag();
 
			// Update physics
			var physStart:uint = getTimer();
			m_world.Step(m_timeStep, m_iterations);
			Main.m_fpsCounter.updatePhys(physStart);
 
		}
 
		//======================
		// Member Data
		//======================
		public var m_world:b2World;
 
		public var m_mouseJoint:b2MouseJoint;
		public var m_iterations:int = 40;
		public var m_timeStep:Number = 1/30;
		public var m_physScale:Number = 30;
		// world mouse position
		static public var mouseXWorldPhys:Number;
		static public var mouseYWorldPhys:Number;
		static public var mouseXWorld:Number;
		static public var mouseYWorld:Number;
		// Sprite to draw in to
		public var m_sprite:Sprite;
		public var m_rope:Rope;
		public var m_hookJointDef:b2RevoluteJointDef;
		public var m_leftHookJoint:b2Joint;
		public var m_rightHookJoint:b2Joint;
		public var m_leftHook:Hook;
		public var m_rightHook:Hook;
		public var m_bead:Bead;
 
		//======================
		// Update mouseWorld
		//======================
		public function UpdateMouseWorld():void{
			mouseXWorldPhys = (Input.mouseX)/m_physScale;
			mouseYWorldPhys = (Input.mouseY)/m_physScale; 
 
			mouseXWorld = (Input.mouseX);
			mouseYWorld = (Input.mouseY);
		}
 
		//======================
		// Mouse Drag
		//======================
		public function MouseDrag():void{
			// mouse press
			if (Input.mouseDown && !m_mouseJoint){
 
				var body:b2Body = GetBodyAtMouse();
 
				if (body)
				{
					var md:b2MouseJointDef = new b2MouseJointDef();
					md.body1 = m_world.GetGroundBody();
					md.body2 = body;
					md.target.Set(mouseXWorldPhys, mouseYWorldPhys);
					md.maxForce = 600.0 * body.GetMass();
					md.timeStep = m_timeStep;
					m_mouseJoint = m_world.CreateJoint(md) as b2MouseJoint;
					body.WakeUp();
					if (body == m_rope.m_leftHook || body == m_rope.m_rightHook){
						if (m_leftHookJoint && body == m_leftHookJoint.GetBody2())
						{
								m_world.DestroyJoint(m_leftHookJoint);
								m_leftHookJoint = null;
						}
						if (m_rightHookJoint && body == m_rightHookJoint.GetBody2())
						{
								m_world.DestroyJoint(m_rightHookJoint);
								m_rightHookJoint = null;
						}
					}
					if (body == m_bead.m_body)
					{
						m_bead.SetCollidable(false);
					}
				}
			}
 
			// mouse release
			if (!Input.mouseDown){
				if (m_mouseJoint)
				{
					//get body being clicked on
					var clickedBody:b2Body = m_mouseJoint.GetBody2();
					m_world.DestroyJoint(m_mouseJoint);
					m_mouseJoint = null;
					if (clickedBody == m_bead.m_body)
					{
						m_bead.SetCollidable(true);
					}
					if (m_rope.IsAttachable( clickedBody))
					{
 
						var ground:b2Body = m_world.GetGroundBody();
						if (m_leftHook.IsTouching(clickedBody))
						{
							trace("test");
							m_hookJointDef.Initialize(m_leftHook.m_body,  clickedBody, new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
							if (m_leftHookJoint){
								m_world.DestroyJoint(m_leftHookJoint);
								m_leftHookJoint = null;
							}
							m_leftHookJoint = m_world.CreateJoint(m_hookJointDef);
						}
						else if (m_rightHook.IsTouching( clickedBody))
						{
							if (m_rightHookJoint){
								m_world.DestroyJoint(m_rightHookJoint);
								m_rightHookJoint = null;
							}
							m_hookJointDef.Initialize(m_rightHook.m_body,  clickedBody,  new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
							m_rightHookJoint = m_world.CreateJoint(m_hookJointDef);
						}
					}
				}
			}
 
			// mouse move
			if (m_mouseJoint)
			{
				var p2:b2Vec2 = new b2Vec2(mouseXWorldPhys, mouseYWorldPhys);
				m_mouseJoint.SetTarget(p2);
			}
		}
 
		//======================
		// Mouse Destroy
		//======================
		public function MouseDestroy():void{
			// mouse press
			if (!Input.mouseDown && Input.isKeyPressed(68/*D*/)){
 
				var body:b2Body = GetBodyAtMouse(true);
 
				if (body)
				{
					m_world.DestroyBody(body);
					return;
				}
			}
		}
 
		//======================
		// GetBodyAtMouse
		//======================
		private var mousePVec:b2Vec2 = new b2Vec2();
		public function GetBodyAtMouse(includeStatic:Boolean=false):b2Body{
			// Make a small box.
			mousePVec.Set(mouseXWorldPhys, mouseYWorldPhys);
			var aabb:b2AABB = new b2AABB();
			aabb.lowerBound.Set(mouseXWorldPhys - 0.001, mouseYWorldPhys - 0.001);
			aabb.upperBound.Set(mouseXWorldPhys + 0.001, mouseYWorldPhys + 0.001);
 
			// Query the world for overlapping shapes.
			var k_maxCount:int = 10;
			var shapes:Array = new Array();
			var count:int = m_world.Query(aabb, shapes, k_maxCount);
			var body:b2Body = null;
			for (var i:int = 0; i < count; ++i)
			{
				if (shapes[i].GetBody().IsStatic() == false || includeStatic)
				{
					var tShape:b2Shape = shapes[i] as b2Shape;
					var inside:Boolean = tShape.TestPoint(tShape.GetBody().GetXForm(), mousePVec);
					if (inside)
					{
						body = tShape.GetBody();
						break;
					}
				}
			}
			return body;
		}
 
	}
 
}

The first change is at the end of the constructor where I create the items that are in the world.

m_rope = new Rope(m_world, m_physScale);
 
m_leftHookJoint = null;
m_rightHookJoint = null;
 
m_leftHook = new Hook(m_world, m_physScale, new b2Vec2(175/m_physScale, 50/m_physScale), 10/m_physScale);
m_rightHook = new Hook(m_world, m_physScale, new b2Vec2(500/m_physScale, 50/m_physScale), 10/m_physScale);
m_hookJointDef = new b2RevoluteJointDef();
 
m_bead = new Bead(m_world, m_physScale, 7.5/m_physScale);

The second noticable difference is the addition of the member variables.

public var m_rope:Rope;
public var m_hookJointDef:b2RevoluteJointDef;
public var m_leftHookJoint:b2Joint;
public var m_rightHookJoint:b2Joint;
public var m_leftHook:Hook;
public var m_rightHook:Hook;
public var m_bead:Bead;

In the mouse drag function are the most important additions. In the first part of this function we are focusing on mouse presses, or an initiated drag. The first thing we do is check to see if we clicked on a joint at one of our hooking points, if so we want to destroy that joint and let the mouse move the necklace wherever it wants to. We also want the user to be able to click a bead and place it on the necklace without colliding with the necklace while trying to drag the bead, this is also check for here, we turn off collisions for a bead if it is being dragged.

\The second part of the mouse drag function is when the mouse button is relased after dragging has occurred. When this happens, we check to see if we were holding a bead, if we were we set it to be collidable, so that it will stay on the necklace if it was placed there. We are also checking to see if the right or left hook are clicked on with one of the end pieces of the necklace. If they are then we create a new joint between the hook and the necklace endpiece.

//======================
//======================
// Mouse Drag
//======================
public function MouseDrag():void{
	// mouse press
	if (Input.mouseDown && !m_mouseJoint){
 
		var body:b2Body = GetBodyAtMouse();
 
		if (body)
		{
			var md:b2MouseJointDef = new b2MouseJointDef();
			md.body1 = m_world.GetGroundBody();
			md.body2 = body;
			md.target.Set(mouseXWorldPhys, mouseYWorldPhys);
			md.maxForce = 600.0 * body.GetMass();
			md.timeStep = m_timeStep;
			m_mouseJoint = m_world.CreateJoint(md) as b2MouseJoint;
			body.WakeUp();
			if (body == m_rope.m_leftHook || body == m_rope.m_rightHook){
				if (m_leftHookJoint && body == m_leftHookJoint.GetBody2())
				{
						m_world.DestroyJoint(m_leftHookJoint);
						m_leftHookJoint = null;
				}
				if (m_rightHookJoint && body == m_rightHookJoint.GetBody2())
				{
						m_world.DestroyJoint(m_rightHookJoint);
						m_rightHookJoint = null;
				}
			}
			if (body == m_bead.m_body)
			{
				m_bead.SetCollidable(false);
			}
		}
	}
 
	// mouse release
	if (!Input.mouseDown){
		if (m_mouseJoint)
		{
			//get body being clicked on
			var clickedBody:b2Body = m_mouseJoint.GetBody2();
			m_world.DestroyJoint(m_mouseJoint);
			m_mouseJoint = null;
			if (clickedBody == m_bead.m_body)
			{
				m_bead.SetCollidable(true);
			}
			if (m_rope.IsAttachable( clickedBody))
			{
 
				var ground:b2Body = m_world.GetGroundBody();
				if (m_leftHook.IsTouching(clickedBody))
				{
					trace("test");
					m_hookJointDef.Initialize(m_leftHook.m_body,  clickedBody, new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
					if (m_leftHookJoint){
						m_world.DestroyJoint(m_leftHookJoint);
						m_leftHookJoint = null;
					}
					m_leftHookJoint = m_world.CreateJoint(m_hookJointDef);
				}
				else if (m_rightHook.IsTouching( clickedBody))
				{
					if (m_rightHookJoint){
						m_world.DestroyJoint(m_rightHookJoint);
						m_rightHookJoint = null;
					}
					m_hookJointDef.Initialize(m_rightHook.m_body,  clickedBody,  new b2Vec2(mouseXWorldPhys, mouseYWorldPhys));
					m_rightHookJoint = m_world.CreateJoint(m_hookJointDef);
				}
			}
		}
	}
 
	// mouse move
	if (m_mouseJoint)
	{
		var p2:b2Vec2 = new b2Vec2(mouseXWorldPhys, mouseYWorldPhys);
		m_mouseJoint.SetTarget(p2);
	}
}

There are three types that are used in the Playground.as file that we haven’t seen and are not in Box2D.  Those are the Rope, Bead, and Hook classes.

The rope is the most complicated and has the most new code.  When instantiated it builds itself vertically at a hard coded location.  It has an endpiece on each end that is different than the rest of the necklace.   Initially I made all pieces of the necklace a rectangle and connected them end to end.  This proved to be a mistake because when there was too much bending or twisting between two rectangle shapes at a joint they would sometimes not know how to fix themselves.  It would look like a permanent crease between the joint.  I was unable to figure out how to fix this problem with rectangles and I figured collisions between spheres is always cheaper so I moved everything to circles.

So the result is that the rope is made up of anywhere between 10 and 60 circles.  The values for the number of circles and the size of them is currently hard coded in the rope.as file.  Most of what is happening during the building of the rope is pretty obvious.  I changed a couple of the collision properties to make things look more realistic.

Each end piece is special and can be attached to a connector in the stage.  The isAttachable function checks to see if a given body can be attached to something.  I did this so it was easy to change which things could be attached in the future.

One last thing to note is that I made the end pieces collidable with the necklace pieces but the necklace pieces are not collidable with themselves.  This saves a lot of computation I’m sure but the main reason was so that you could more easily find an endpiece among a pile of necklace since it should be sitting on top.

Rope.as

import Box2D.Dynamics.*;
import Box2D.Collision.*;
import Box2D.Collision.Shapes.*;
import Box2D.Dynamics.Joints.*;
import Box2D.Dynamics.Contacts.*;
import Box2D.Common.*;
import Box2D.Common.Math.*;
 
public class Rope{
 
	public function Rope(world:b2World, physScale:Number){
 
		// Set block length & width & overlap(to make block connections look smoother)
		m_blockLength =3.5;
		m_blockWidth = 3.5;
		m_overlap = -4.5;
		// Set end piece paramaters
		m_circleRadius = 3;
 
		// Starting height for necklace
		m_startingHeight = -200; 
 
		var vel:b2Vec2 = new b2Vec2();
		vel.Set(5, 0); 
 
		var ground:b2Body = world.GetGroundBody();
		var i:int;
		var anchor:b2Vec2 = new b2Vec2();
		var body:b2Body;
 
		// Bridge
		{
			var sd:b2PolygonDef = new b2PolygonDef();
			sd.SetAsBox(m_blockLength / physScale, m_blockWidth / physScale);
			sd.density = 20;
			sd.friction = .8;
			sd.restitution = 0.0;
			sd.filter.groupIndex = -2;
 
			var cd:b2CircleDef = new b2CircleDef();
			cd.density = 20.0;
			cd.radius = (m_circleRadius) / physScale;
			cd.restitution = 0.0;
			cd.friction=0.8;
			cd.filter.groupIndex = -3;
 
			var bd:b2BodyDef = new b2BodyDef();
			bd.angle = -90 / (180/Math.PI);
			var jd:b2RevoluteJointDef = new b2RevoluteJointDef();
			const numPlanks:int = 50;
 
			jd.lowerAngle = -10 / (180/Math.PI);
			jd.upperAngle = 10 / (180/Math.PI);
			jd.enableLimit = false;
			jd.collideConnected = false;
 
			var prevBody:b2Body = ground;
 
			bd.position.Set(100 / physScale, (m_startingHeight + m_blockLength*2)/physScale)
			//bd.isBullet = true;
			body = world.CreateBody(bd);
			body.CreateShape(sd);
			body.SetMassFromShapes();
			body.SetLinearVelocity(vel);
			m_leftHook = prevBody = body;
			for (i = 0; i < numPlanks; ++i)
			{
				bd.position.Set(100 / physScale, (m_startingHeight + m_blockLength + (m_blockLength*2*i) - m_overlap*i) / physScale);
				bd.isBullet = false;
				bd.linearDamping = 0.1;
				bd.angularDamping = 0.5;
				body = world.CreateBody(bd);
				body.CreateShape(cd);
				body.SetMassFromShapes();
 
				anchor.Set(100 / physScale, (m_startingHeight + (m_blockLength*2*i) - m_overlap*i) / physScale);
 
				jd.Initialize(prevBody, body, anchor);
 
				world.CreateJoint(jd);
 
				prevBody = body;
			}
			bd.linearDamping = 0;
			bd.angularDamping = 0;
			//bd.isBullet = true;
			bd.position.Set(100/physScale, (m_startingHeight + m_blockLength + m_blockLength*2*numPlanks - m_overlap*numPlanks) / physScale);
			body = world.CreateBody(bd);
			body.CreateShape(sd);
			body.SetMassFromShapes();
			m_rightHook = body;
			anchor.Set(100 / physScale, (m_startingHeight + (m_blockLength*2*numPlanks) - m_overlap*numPlanks) / physScale);
			jd.Initialize(prevBody, body, anchor);
			world.CreateJoint(jd);
 
		}
 
	}
 
	public function IsAttachable(body:b2Body) : Boolean {
		if (body == m_leftHook || body == m_rightHook)
		{
			return true;
		}
		else
		{
			return false;
		}
	}
 
	//======================
	// Member Data
	//======================
	public var m_blockLength:Number;
	public var m_blockWidth:Number;
	public var m_circleRadius:Number;
	public var m_overlap:Number;
	public var m_startingHeight:Number;
	public var m_leftHook:b2Body;
	public var m_rightHook:b2Body;
 
}

Creating a bead class that could be placed on the necklace was one thing I was unsure of how to do initially. The end result for my prototype was to create three circle shapes on a single body and to make the center one a sensor. A sensor is something that can collide with other objects but doesn’t cause any actual physical response. It is essentially not collidable but still keeps track of its collisions.

So the constructor creates this complex body. I found very little documentation on how to make a simple complex body out of simple shapes online. It was very easy though and if you look at the bead.as constructor you can see how I did it. The most confusing part was how I was supposed to offset the different shapes in the body. This is accomplished via the Shape class’ localPosition attribute. By modifying this attribute I could offset the shape in the body it will eventually be a part of.

The last thing in this class is the setCollidable function. I needed a way to easily make all of the parts of the bead a sensor temporarily. This was so that the user could drag the bead and not collide with anything and then let go and it would become collidable again.
Bead.as

import Box2D.Dynamics.*;
import Box2D.Collision.*;
import Box2D.Collision.Shapes.*;
import Box2D.Dynamics.Joints.*;
import Box2D.Dynamics.Contacts.*;
import Box2D.Common.*;
import Box2D.Common.Math.*;
 
public class Bead{
 
	public function Bead(world:b2World, physScale:Number, radius:Number){
 
		m_circleRadius = radius;
 
		var cd:b2CircleDef = new b2CircleDef();
		cd.density = 0.01;
		cd.radius = (m_circleRadius);
		cd.friction = 0.1;
		cd.restitution = 0.1;
 
		var bd:b2BodyDef = new b2BodyDef();
 
		cd.localPosition.x = -15 / physScale;
 
		bd.position.Set(400/physScale, 200/physScale)
		bd.isBullet = true;
		m_body = world.CreateBody(bd);
		m_body.CreateShape(cd);
		cd.localPosition.x = 15 / physScale;
		m_body.CreateShape(cd);
		cd.localPosition.x=0;
		cd.isSensor = true;
		m_centerShape = m_body.CreateShape(cd);
		m_body.SetMassFromShapes();
 
	}
 
	public function SetCollidable(toggle:Boolean) : void{
		var currentShape:b2Shape = m_body.GetShapeList();
 
		for(currentShape; currentShape; currentShape = currentShape.m_next){
			if (currentShape == m_centerShape){
				continue;
			}
			currentShape.m_isSensor = !toggle;
		}
	}
 
	//======================
	// Member Data
	//======================
 
	public var m_circleRadius:Number;
	public var m_position:b2Vec2;
	public var m_body:b2Body;
	public var m_centerShape:b2Shape;
 
}

The hook class is very simple and just creates a circle in the location where it is supposed to be. I temporarily just used the physics circle to make it easy to test. I don’t plan to have a physics body be the connector piece in the future unless I am using so sort of callback function. I think it is easy enough to use the isTouching function to check if a body is within touching range of this hook.
Hook.as

import Box2D.Dynamics.*;
import Box2D.Collision.*;
import Box2D.Collision.Shapes.*;
import Box2D.Dynamics.Joints.*;
import Box2D.Dynamics.Contacts.*;
import Box2D.Common.*;
import Box2D.Common.Math.*;
 
public class Hook{
 
	public function Hook(world:b2World, physScale:Number,  position:b2Vec2, radius:Number){
 
		m_circleRadius = radius;
		m_position = position;
 
		var cd:b2CircleDef = new b2CircleDef();
			cd.density = 1.0;
			cd.radius = (m_circleRadius);
			cd.isSensor = true;
			var bd:b2BodyDef = new b2BodyDef();
 
			bd.position.Set(m_position.x, m_position.y)
			m_body = world.CreateBody(bd);
			m_body.CreateShape(cd);
 
	}
 
	public function IsTouching(body:b2Body):Boolean{
		var position = body.GetWorldCenter().Copy();
 
		position.Subtract(m_position);
		position.Abs();
		trace(position.x + " x units away.");
		trace("radius of circle is " + m_circleRadius);
		if (position.x < m_circleRadius && position.y < m_circleRadius)
		{
			return true;
		}
		else
		{
			return false;
		}
	}
	//======================
	// Member Data
	//======================
 
	public var m_circleRadius:Number;
	public var m_position:b2Vec2;
	public var m_body:b2Body;
 
}

The resulting application turned out pretty good. I was not totally satisfied with placing a bead on the necklace, but I ran out of time and had to do real work by that time. I think I could make the bead automatically orient correctly and highlight itself when it is being placed on the right spot on the necklace. Besides that problem I am satisfied with the it as a prototype for a future application I might develop.

Here it is:

Beadr Prototype 1: