I searched thoroughly for an easy way to retrieve information from this set of .ADT files I was dealing with. Sybase provides tools to work with the data, but all of them require using their software or drivers developed by them. I would have settled for this if it wasn’t such a hassle to figure out how to use them. I asked the question on Stackoverflow.com and got an answer from one of the employee’s of Sybase. While helpful, the suggested Perl driver was one of the things I was avoiding. I did attempt it and after a little hassle, moved on to creating a simpler solution.

One solution was to simply use Sybase’s provided “Advantage Data Architect” software to open the .ADT files and export them all to SQL. This is possible with their software, but it would not work for my situation. The developers of the program outputting these .adt files used a separate embedded database for each new report that was being worked on. This meant there was hundreds of folders, each containing 30-40 .ADT files making up their own embedded database. I needed something that could automate the extraction of this data.

Data Format

To cut this short, I decided to open up the .ADT files with a hex editor and see if I could get the information I needed out. I only needed the column names, their types, and the actual rows of data. To aid anybody else doing something similar to this, I’ll include some of my notes about the format here:

• The data is stored at the very end of the file.
• The byte offset when the data starts is located at byte 32 and is 32-bits
• Byte 24 contains a 32-bit integer specifying the number of rows
• Byte 36 contains the a 32-bit number representing the number of bytes each record consumes
• The header is 400 bytes
• Column information is after the header, before the data. Each column entry is 200 bytes.
• The number of columns can be calculated (data_offset-400)/200
• Column names consume the first 128 of each column entry.
• Byte 130 contains a 16-bit integer representing the type of data.
• Byte 136 contains a 16-bit integer, the length of the field in bytes
• Character = 04, Double = 0A, Integer = 0B, Shortint = 0C, CICharacter = 14, Date = 03, Time = 0D, Timestamp = 0E, Autoinc = 0F

This was enough information to get what I wanted.

Ruby-ADT Gem

I took the existing DBF gem by Keith Morrison. Most of the credit for this gem goes to him as I borrowed the structure heavily from his gem.

I’ll provide an example of what is now possible with my new ADT gem. No drivers are required and it’s as simple as installing the gem. I’ll include a snippet of the readme from my Ruby-ADT gem on github, borrowed heavily again from the DBF gem.

Installation

gem install ruby-adt

Basic Usage

Load an ADT file:

require 'rubygems'
require 'adt'
 
table = ADT::Table.new("test.adt")

Enumerate all records

table.each do |record|          
	puts record.name          
	puts record.email
end

Load a single record using record or find

table.record(6)
table.find(6)

Attributes can also be accessed through the attributes hash in original or underscored form or as an accessor method using the underscored name.

table.record(4).attributes["PhoneBook"]
table.record(4).attributes["phone_book"]
table.record(4).phone_book

Search for records using a simple hash format. Multiple search criteria are ANDed. Use the block form of find if the resulting recordset could be large otherwise all records will be loaded into memory.

# find all records with first_name equal to Keith
 
table.find(:all, :first_name => 'Keith') do |record|  
	puts record.last_name
end
 
# find all records with first_name equal to Keith and last_name equal to Morrison
table.find(:all, :first_name => 'Keith', :last_name => 'Morrison') do |record|   
	puts record.last_name
end
 
# find the first record with first_name equal to Keith
 
table.find :first, :first_name => 'Keith'
 
# find record number 10 
 
table.find(10)

Migrating to ActiveRecord

An example of migrating a DBF book table to ActiveRecord using a migration:

require 'adt'
 
class CreateBooks < ActiveRecord::Migration      
	def self.up            
		table = ADT::Table.new('db/adt/books.adt')            
		eval(table.schema)        
		table.each do |record|                  
			Book.create(record.attributes)            
		end      
	end  
	def self.down        
		drop_table :books    
	end
end

Limitations and known bugs

• ADT is read-only
• External index files are not used
• Dates are not currently handled correctly.

Future Directions

I am going to continue working on this library. I have already heard from a few others that found this useful immediately after I posted it online. I was surprised people could find it so quickly. I’m sure there are bugs. I would appreciate any problems to be reported to the Ruby-ADT Github page. Emails are also welcome, but any issues should be directed to the github page and I’ll get to them when I can.

Hope this helps somebody out there dealing with this format.

- Chase Gray

Prerequisites

The first thing I needed to do was to successfully configure and compile mdbtools. This probably took the longest as there are a few different version out there. I had the most success with these instructions from the author’s git repository, http://www.automatthew.com/2008/02/how-to-compile-mdbtools-on-mac-os-x.html?showComment=1219246620000. I was also able to compile this version on my ubuntu machine pretty easily. The biggest hangup was with a couple of dependencies. I also had a weird error where the autogen.sh was not detecting my libtools library, but I knew this was installed. I commented out this check in the autogen.sh and things all worked, luckily.

Understanding mdb-tools was not very difficult. It contains various libaries for interfacing with it through C, etc. The only thing I cared about were the command line tools that it provided, listed here. The commands I will use in this post are mdb-tables, mdb-schema, mdb-export.

The other requirement is to have a MySQL table created ready with permissions, etc. that will end up holding the data from the combined .mdb files.

Step 1: Extracting the Schema

One important thing to note about my situation is that all the .mdb files have the same schema. Knowing this, I was able to simply find a random .mdb file and base my MySQL schema on this.

I have my .mdb files stored in a shared directory and so within my Rake task I used the following code to grab the first one (I’m sure there is a better way).

first_mdb_file = ''
#get first mdb file
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    first_mdb_file = "#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file}"
    break
  end
end

Knowing the path to the first .mdb file the schema can be extracted by getting the result of the following shell command (the code surrounded in backticks is executed on the command line of the system). The -S flag tells it to santize some of the table names by replacing spaces with underscore, etc. I needed this for my tables because there were some problems without it.

schema_sql = `mdb-schema -S #{first_mdb_file} mysql`

Step 2: Building the Tables

Since I’m doing this in a Rake task in Rails (you could do this in plain ruby), I am using an ActiveRecord connection to interface with my MySQL database. I have a class called ExternalDatabase that allows me to connect to a different database without modifying the one that my main Rails program uses. I will list it below, I don’t remember exactly where I got it from.

#Class used to connect to external databases without modifying the exiisting connection used by Rails
class Data::ExternalDatabase < ActiveRecord::Base
  self.abstract_class = true
 
  def self.set_source(configuration_name)
    source = ActiveRecord::Base.configurations["sources"][configuration_name.to_s]
    establish_connection source
  end
 
  def self.set_configuration(config)
    source = config
    establish_connection source
  end
end

As you can see, I also have a section in my database.yml where i list connection details for various source databases. This isn’t related to this post though.

So, assuming we have this class, we can use the following code to load our new schema into out MySQL database.

Data::ExternalDatabase.set_configuration(:adapter  => "mysql",
:host     => "localhost",
:username => YOUR_DATABASE_USERNAME,
:password => YOUR_DATABASE_PASSWORD,
:database => YOUR_DATABASE_NAME,
)
db =  Data::ExternalDatabase.connection 
 
puts "adding if exists clause to drop tables"
 
schema_sql = schema_sql.gsub(/DROP TABLE (.*);/, 'DROP TABLE IF EXISTS \1;')
schema_sql = schema_sql.gsub(/--.*$/, '')
puts "executing schema sql"
lines = schema_sql.split(';')
 
lines.each do |sql_line|
  next if sql_line.blank?
  db.execute(sql_line)
end

The first part builds my database configuration hash that I pass in to ExternalDatabase. I save the connection into a variable called ‘db’. One of the important things to note about this and mdb-tools is that it does not check if the table currently exists before trying to drop the table. So I added a snippet of code that removes the comments and adds ‘IF EXISTS’ to the ‘DROP TABLE’ clause in the schema sql.

For some reason, trying to execute this entire schema sql string with db.execute causes various errors that I couldn’t figure out. By separating the schema sql into individual sql statements terminated with semicolons seemed to fix this problem. The last bit of code does the separation into lines and executes the sql on the database. After all this, the database should have the correct schema.

Step 3: Load Data

One of the last things we want to do is actually load all the data from the different mdb files into the single MySQL database.

tables = (`mdb-tables #{first_mdb_file}`).split(' ')
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    puts "Current file: #{mdb_file}"
    tables.each do |table|
      puts "Current table: #{table}"
      insert_sql = `mdb-export -I -S #{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file} #{table}`
      insert_sql = insert_sql.gsub(/;/, ',')
      insert_sql = insert_sql.gsub(/(.*?\))(\s+INSERT)/, '\1;\2')
      lines = insert_sql.split(';')
      lines.each do |sql_line|
        next if sql_line.blank?
        db.execute(sql_line)
      end
    end
  end
end

So the first line runs mdb-tools command line tool ‘mdb-tables’ to get a list of the tables separated by spaces. I could have done this through my previously created schema as well.

This code goes through every .mdb file in my directory and uses mdb-tools to extract the data as sql ‘INSERT’ statements from the command-line.

In the inner-most loop, I clean up the returned value from mdb-export because it does not include semi-colons after each INSERT statement. I also remove any semi-colons before adding any because some of my fields had semi-colons in them. Again, I separate each INSERT statement into a single statement and execute these individually in the last section of the code.

After all this you should have all your data combined into a single MySQL database. One problem might exists though. Certain tables, such as tables that contain general purpose labels, etc. for other tables to refer to, might contain duplicates. This wouldn’t have caused a big problem for me because I just needed to extract data, but if you needed to continue using this database you would need to remove the duplicates.

Final Step: Removing Duplicate Rows

There seem to be a lot of different ways to remove duplicate rows from a MySQL table. I wish there was a simple solution, but I couldn’t find one. The following code does the following:

1. Create a temporary table that has the same schema as the table we are removing duplicates from.
2. Insert all distinct rows into this new temporary table.
3. remove all values from the original table.
4. Insert all of the rows from the temporary table back into the original table.
5. Finally, drop the temporary table.

tables.each do |table|
  db.execute("CREATE TABLE temp LIKE #{table}")
  db.execute("INSERT INTO temp select distinct * FROM #{table}")
  db.execute("TRUNCATE #{table}")
  db.execute("INSERT INTO #{table} select * from temp")
  db.execute("DROP TABLE temp")
end

Thanks for reading, I hope this was helpful to others that have some similar problems.

- Chase

Problem 1: How to let Google index my data that requires a login

A prerequisite for this step is to submit sitemaps for your site’s content to Google Webmaster Tools.  Once you have done this, at some point in the future Googlebot will attempt to traverse your page and index its contents (hopefully).

The content on my site requires a user to pay a subscription fee in order to have free reign and browse through whatever they please.  I’m not very concerned about any single piece of information getting leaked, instead I’m protecting the resource the site provides as a whole.  By letting Google index your site, you’re obviously opening up your content to outside viewers that are not logged in.  The extent that you need to protect your content could vary from what my needs are.

What I want is for a user to be able to type their name into Google search engine and find out that they are referenced in the data provided by Myhealthcaresource.com.  So, I need to somehow let Googlebot traverse my site, even through it requires a login by a normal user.

I’m using Restful Authentication and a role requirement system, so this is how I include this exception.

class FacilitiesController < ApplicationController
 require_role :basic, :only => :show, :unless => request.env['HTTP_USER_AGENT'] =~ /.*Googlebot.*/
end

This could vary depending on your authentication system, but the part that will remain the same is the check if the user agent string contains ‘Googlebot’.

That’s all it takes in order to allow Googlebot to traverse my controlelr that previously required a login.  A potential problem that could come up is that users could ‘fake’ their user agent to say ‘Googlebot’ and gain access to the site.  This could be a problem for some sites, and we would keep an eye out for this in our logs or through analytics software.  We provide a free trial to all our users before they decide to pay, so uers get a free look at the data already and there is no reason to try to sneak around the site to see it.  We’re not concerned with this problem, but it could be an issue for others.

Problem 2: How to prevent Google from caching private pages

Once Google starts indexes pages of your site that require a login for a normal user, it will also start caching all the sites that it visits.  If you are worried about your internal content getting cached by Google you will want to prevent Google or any other robots from caching.  This is simple and is accomplished by putting the following in the heading of your main layout file.

<META name="robots" content="index,follow,noarchive" />
<META NAME="googleBOT" CONTENT="noarchive" />

When Googlebot reaches your page and sees this tag, it will not cache it.  I have this on the Myhealthcaresource layout, and by searching myhealthcaresource in google you can see there are no cache links.

Problem 3: Allowing Users Referred From Google View Content Without Logging In

When a user sees a link to myhealthcaresource on a set of Google search results, they are most likely to the detailed facility reports that I submitted in a sitemap.  They probably searched their company name or their own and found that they were referred to by my site’s information.  When someone clicks this link in Google, by default they would get redirected to a page asking them to log in.  This would make almost every user referred from Google resort to their back button immediately.  I noticed this behavior during the period when my site functioned this way.  These users from Google have no idea what kind of information our site offers and they might be a potential customer, so we don’t want them to leave because we present a log in form when they first arrive.

An alternative is to allow users referred to our site through a Google search to view the entire page.  We decided to do just this.  When they click the link in Google’s search results they are not asked to log in and they are shown the same page a logged in user would see.  What happens when they try to browse the rest of the private content?  They are asked to log in, but we already have their interest at this point and they are more likely to stay if they found something that was useful to them on the page they landed on.

So how do we accomplish this?  In a similar way that we allowed Googlebot to traverse our pages, we can allow users that were referred from Google to access our page:

class FacilitiesController < ApplicationController
  require_role :basic, :only => :show, :unless => request.referer =~ /.*google.*/ || request.env['HTTP_USER_AGENT'] =~ /.*Googlebot.*/
end

Of course this code would vary slightly if you are using a different authentication scheme.  The core idea would be the same though.  Now, the potential problem that could come up with this is that a malicious user uses Google to repeatedly access your site by creating clever search queries.  In our case, if a user is this intent on stealing the data, we weren’t likely to win them as a customer in the first place.  We can also monitor this through analytics tools and if it is abused we will look into an alternative solution.  It could be possible to restrict the number of Google referrals a certain IP can have in a time period or something.

I hope that some of this work can be useful to people who are looking for ways to leverage Google for their dynamically generated login-only content.  We see people come from Google searches every day, and now they stay much longer as a result of these changes.

In a future post, I will briefly talk about another method our site uses to send links to users that allow them to browse the site freely without logging in.  After a certain time period these links expire and the users must sign up to browse the site further.

I found myself working with some data in a hash generated from scraping a website. The website had some duplicate content and I didn’t feel like it was necessary to clutter my scraping code with tests for duplicate content while it was scraping. After scraping the data and turning it into a hash, I wished to remove the duplicates with some post-processing. I felt like this should be pretty simple, except that the elements I wanted to test for uniqueness contained a deeply nested structure of hashes and arrays. Here is an example an element in an array I wanted to uniqify:

{:name => 'bob', :posts => [{:title => 'First Post!' :comments => [{:commenter => 'Alice', :text => 'Hi bob!'}]}]}

The #uniq method does not work as expected when there is a Hash element in the array, it only uses the object_id to determine uniqueness. There are some solutions online for dealing with this.

If you know that each element inside the array is a hash with no sub-hashes you can use the solution presented in the answer of this stackoverflow.com question. The first solution uses #inject to build a new array with only unique items. Whether an item is included in the array already or not is determined with the Hash’s #include method.

a.inject([]) { |result,h| result << h unless result.include?(h); result }

The second solution presented in the same post:

a.map {|h| h.to_a[0]}.uniq.map {|k,v| {k => v}}

This flattens all of your hashes into an array. It then calls uniq on the resulting array after all the hashes are flattened. This means that for an item to be unique it must have the same key AND value. After finding the unique elements, it then maps the items back into a hash.

In this post, Mike Burns solve this problem by redefining #hash and #eql?. If you are doing this kind of comparison often, this would probably be a better approach. He does note though, that by doing this the #dup method no longer works as expected.

So it was obvious through searching that I was going to have to figure this out on my own. I was a little worried that I would have to implement an ugly recursive solution that searches the entire tree to find all duplicates. While looking into this solution, I felt that it would be very nice if #uniq would accept a block that would allow me to return true if the given parameter is a duplicate, thus removing it from the returned array. This is not implemented in my current ruby version (1.8.6), but there is an example of how to do this on this blog post. Hopefully, more (all) of these methods in the ruby core will accept blocks someday.

I began to write a method that would search my tree-like structure of hashes and arrays and I felt like it was a very ugly solution. I wanted something simple that would just take a line or two and get the job done. So I showed an example of the problem to my non-computer scientist wife and she said, “why don’t you just compare how the two lists look when printed out?” I was a little shocked that I hadn’t thought of this. It isn’t the most efficient solution and it relies on the #inspect method, which is probably pretty slow because it has to determine the structure of all the hashes in order to print them in a readable format. The old solution I was cooking up would also have to traverse the tree, similar to what I expected #inspect was doing. In addition to the #inspect call, there would be a string comparison for each element. I was willing to accept both of these for a short elegant solution to my problem.

The issue that came to my mind first was whether hashes get printed out the same way every time. In Ruby 1.9, it seems that Hashes are going to be stored in the same order that they were entered. In previous versions, I’m assuming that they are stored in the order of their generated hash. This could cause a problem if the data someone was entering could be in different orders. For my purposes though, the data would be entered in the same order if it was a duplicate.

So, after all of that, to uniqify an array containing a deeply nested structure, I present the following solution:

class Array
  def nested_uniq
    self.delete_if {|item| (self - [item]).any? {|other_item| item.inspect == other_item.inspect}}
  end
  def nested_uniq!
    self.replace self.nested_uniq
  end
end

I’ll walk through it because it may seem a bit hard to understand at first. I go through every element in the array with delete_if to determine if the element should be deleted from the array. Then, for all elements except the one in question, I check to see if #any? of them are equivalent to the element in question. This equivalence is determined by doing a string comparison of what the #inspect method returns for that element.

Here is an IRB test

>>  test = [{:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}, {:to => '1', :yeah => '8'}, {:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}]
=> [{:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}, {:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]
>> test.delete_if {|item| (test - [item]).any? {|other_item| item.inspect == other_item.inspect}}
=> [{:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]

Hope this is helpful to someone! I am very interested to hear suggestions and comments on alternative ways to uniqify an array with elements that have a nested structure.

-Chase

alternate_data:
    adapter: mysql
    database: alternate_database
    username: user
    password: pass
    host: localhost

I was accessing it with something similar to the following:

ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations["#{state}_data"])
results = connection.execute description_params[:extraction_sql];
#This wasn't actually hard coded, but this is just an example
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations["development"])

The problems with this was that I had to repeat this ugly code everywhere, and if there was an exception thrown while accessing the alternate database it would not revert back to the original before exiting.  So the next request to the server thought the default database was the alternate database.  What I decided would be useful would be to have an “establish_connection” method that accepted a block and handle these issues.

I am going to write about this because I planned to before realizing that in Rails 2.2 the exact same issue is solved already with a much better implementation (of course).  You can find how to solve this problem very elegantly in Rails 2.2 at http://www.igvita.com/2008/10/27/scaling-activerecord-with-mysqlplus/.

My implementation is much simpler and I hope that anybody stuck with Rails 2.1 (like me for now) or any previous version can use it temporarily until they can upgrade in the future.

I just wanted a simple block establish_connection so I could write code like the following:

   establish_temporal_database_connection("#{state}_data") do |connection|
      results = connection.execute description_params[:extraction_sql];
    end

It would reset the old database connection if something went wrong or when the block was over.

I’m just going to list a simple module that does this, if anybody wants this in plugin form that actually overrides establish_connection then it isn’t that difficult to turn this into that:

module ConnectionWithBlock  
  def self.extended(object)
    class << object
        extend ClassMethods
    end
  end
  module ClassMethods
    def establish_temporal_database_connection(configuration)
       #save the old connection
       previous_connection = ActiveRecord::Base.connection
       #establish connection handling a possible exception and going back to original connection
       begin
              ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations[configuration.to_s])
              yield(ActiveRecord::Base.connection)
       rescue Exception => e
              raise e
       ensure
              ActiveRecord::Base.establish_connection(ConnectionSpecification.new(previous_connection.config, previous_connection.adapter_method))
       end   
     end
  end
end

Chase

I’ve been very happy with the results.  Very server intense requests are responding instantly.  This is all great, but caching is usually only appropriate for pages with content that is static for a period of time.  The biggest problem I was having is that there are very small pieces of the pages I wished to cache that are dynamic.  For instance, I want to have a bar at the top that tells the user who is logged in and provides a set of buttons to do things to their profile, etc.  An example of this is shown below:

Small Dynamic Block Showing User Profile Links

So what I needed was an easy way to specify that this small block would be dynamic while still allowing the rest of the view to take advantage of action caching.  I’m sure there is some plugin that does this and more.  If there is a very good plugin that does just this, I would love to know about it in the comments.  For now, though I am going to provide my simple solution to the problem.

My solution was to create a partial for the profile summary.  I already had the block as a partial to make my view code more DRY.  Instead of this:

<%=render :partial => 'profile_summary'%>

in my view, I replaced it with an AJAX call using Prototype (updated to use Richard Poirier’s suggestion in the comments):

    <div id="info_panel">
        <%= javascript_tag remote_function(:url => { :controller => 'profile', :action => 'summary' }) %>
    </div>

I then have a controller that can repond to this AJAX request for the profile.

def summary
  render :update do |page|
    page.replace_html 'info_panel', :partial => '/profile/summary'
  end
end

Now, every time the page loads, it will load the entire page using the cached version.  It will then make an additional POST request to get the dymanic content.  This would become more complicated if there were many dynamic sections that I needed to load.  I feel like if it was wrapped into a plugin that could automatically handle special dynamic fragments inside a block like this:

<%=dynamic_fragment do%>
   <%=render :partial => 'profile_summary'%>
<%end%>

it would make it very easy and intuitive to insert dynamic fragments into an otherwise static page.  I do not have the need to implement something like this yet, I am satisfied with my simple solution.  I would be very happy to find a solution where you can specify the dynamic sections of the view instead of only specifying the static portions.  Then the system would magically build the server-side AJAX responses without the user having to do anything.  This allows the use of action_caches which are supposed to be much faster than fragment caching.  While I’m super busy with my project, stopping to create plugins is not on the priority list, but by writing it all down here I might come back to the best ideas.

I hope anybody looking to insert a small section of dynamic code in an otherwise action cacheable page finds this information useful.

Chase