I own a small company that specializes in obtaining information from public and proprietary sources.  We then take this information and try to make it easier for the average user to browse and work with.  For this service, subscribers pay a small monthly fee.  Most of the information we work with is is related to Medicare and Medicaid financial data.

We have obtained the information we have requested from most of the U.S. states with very few problems arising.  We try to make it as easy as possible for the state agencies and offload most of the processing and parsing of the data to our developers.  The agencies usually comply within a week or two and send everything we asked for on a CD, or if they are really good they’ll send it through an online file transfer service.  Throughout all of these transactions, the worst thing that’s happened is a confused employee that wasn’t sure if they had the requested data.  That is, until contacting the Alabama Medicaid Agency.

The request for data from Alabama began as they have for all the other states.  As with all the other states, it’s unusually difficult to find the right person to talk to in order to get the data you want.  Usually after 10-15 minutes of referrals or continuous promises to have someone call you back, you finally find the right person.  After finding the right person, you still have to ask for the data the way they are accustomed to referring to it, which of course I don’t know initially. I finally found the person in charge of nursing home auditing, Keith Boswell, who was also in charge of the data I wanted to request.  This of course would be the beginning of a 3-4 month ordeal that is still unresolved and I believe there have been many violations of the Alabama Open Records Act in the Alabama Code throughout this entire period.

I’m going to now provide as much detail as possible about these 3-4 months, from which it should be obvious that there has been a violation of Alabama law as well as plenty of government inefficiency and waste.

Cast

Me – Chase Gray
Carol H. Steckel – Alabama Medicaid Commissioner – 334-242-5600 – Carol.Steckel@medicaid.alabama.gov
Stephanie Mcgee Azar – Deputy General Counsel, Alabama Medicaid Agency – 334-242-5126 – Stephanie.Azar@medicaid.alabama.gov
Mattie Jackson – Executive Assistant – 334-353-4361 – Mattie.Jackson@medicaid.alabama.gov
Keith Boswell – Provider Audit/Reimbursement – 334-242-2311 – Keith.Boswell@medicaid.alabama.gov
Terrell Flowers (only passively involved) – Information Systems – Terrell.Flowers@medicaid.alabama.gov

Org Chart for Alabama Medicaid Agency in order to see where these state employees are in relation to each other.

Initial Interaction

As mentioned earlier, I initially spoke with Keith Boswell in charge of provider audit/reimbursement.  He seemed to know which data I was interested in, but they only provided it through paper copies.  This has been a problem in the past when the person I was speaking with never had to provide anything except the paper copies.   Usually through a little discussion of their system it has been simple to determine what types of electronic files they have in order to make the right request. It is actually much less time-intensive and the agency saves a lot of resources by simply providing a data dump instead of making thousands of copies and tying up a state employee’s time for so long.

Through my discussion with Keith I was able to determine that they did indeed have the data in an electronic form, but they did not have the expertise to export it in any way. The state agency had some software developed for them and so there was no internal knowledge about how it stored its data.  I’ve come across this a lot and it appears to be a bit of a recurring problem.  I don’t remember exactly but to summarize, the conversation went a bit like this:

Me: I’m looking for data from the medicaid reports nursing homes submit to your agency.
Keith: We do provide those in paper form, it will be $30 per report, if you want them all that would be a big task and could cost thousands of dollars.
Me: Do you have the data in any sort of digital or electronic format?
Keith: No sir, we only can provide it in paper form.
Me: How do you view the data?
Keith: On my computer, we have this program…
Me: Then what I am requesting is the database that program is using.
Keith: I’m sorry, but we don’t have access to those, they are inside the program.  Also, there are some social security numbers on one of the reports which we normally manually redact from the paper reports.
Me: I thought as long as you had the data available in a certain form, it was could be requested under the open records act.
Keith: You can speak with our legal counsel, Stephanie Azar….

Of course this is from memory and there was much more drilling down on my part to figure out just what they had available and what the difficulty was in extracting it.

The next step seemed to be to contact Ms. Azar.  I called and left a voicemail, with no reply.  I also then followed up with an email.  I assumed I would get a reply in some sort of reasonable time frame.  This was not true unfortunately, and I set aside Alabama for the time being and moved on to other work.  I wouldn’t get back to it for several months.

Months Later

I finally came back to the missing state data of Alabama a few months later.  This time it was the only state left that we hadn’t made any progress on.  I called up Keith Boswell again to refresh my memory about what I needed to do to put in a request.  From him I learned that they use custom software called Medimax, from a company of the same name.   Since they did not have the ability to export data from their software, I offered my company’s services for free to create a tool that could export this data. He seemed receptive at first to the possibility of this, and I began by taking a look at the Medimax software.

Medimax

I didn’t have too high hopes for the software when I noticed the web site, especially the little animated gif . Simply by installing and running the program, it was apparant that it stored all of the data in a folder called ‘data’ in .ADT files.  I came to learn that .ADT files were created by an embedded Advantage Database Server library in the application.  I even gave the developers a call and tried to talk with them a little about my situation.  It was a long phone call, but I lost hope near the beginning when there were lots of late 90s buzz words and ‘enterprise’ talk.  I almost felt like I was trying to be sold the services of this other software company who had a website with a little animated flame gif.  I had the information I needed though.  These .ADT files contained the information I needed, even though there wasn’t a clear way on how to automate the process of parsing all of it (see my previous post on extracting data from ADT files).  So there was no reason to create a tool to extract the data, when the ‘data’ folder contained everything I was interested in.  It was even separated very cleanly into files by section so the sensitive SSN data could be removed easily.

First Unfair Denial

So I felt confident now knowing exactly what I was requesting and hopeful that it would be easy for me and the agency to simply burn them and send them on a CD.  I called Keith back, I was already beginning to feel like a bother, and explained that I simply wanted to obtain these files.  I said it would be very easy  to get these and remove any sensitive data.  He seemed a bit frustrated now, and seemed to close up with a generic response such as, “We’re not required to provide electronic copies”.  I was asking for files that represented a database of course, not copies of anything that existed before them.  By this time I had spent some time with my attorney friend and went over the Alabama low briefly to make sure that I wasn’t in the wrong by asking for these files. I had a basic understanding of what was required, which I thought would have been sufficient.  At the first mention of any of the Alabama code or previous case law Keith became kind of hostile, in my opinion.  He got very defensive and raised his voice and told me, “You’re going to have to speak to our legal department.”  I tried to explain that I wasn’t trying to cause trouble, but just make a perfectly allowed request.  He responded by telling me, “Please stop calling, you are bothering me and wasting my time.” I was pretty amazed to hear this from a state employee, especially since I wasn’t being aggressive in the slightest.  Obviously I was going to have to go to someone else before getting anywhere.  Back to Stephanie Azar.

To summarize, I called and emailed Azar several times over the next week or two without a single response.  Finally I got a call from someone named Mattie Jackson whom I had been referred to since Stephanie Azar had apparently been receiving all my messages.

I had a very similar conversation with Mattie as I did with Keith.  She was much more formal about it and referred to “policy” that they don’t provide “electronic copies” and only have the paper reports.  I quickly asked for an email so I could send in a detailed request explaining it thoroughly.

Before beginning the email exchange I decided to defer some more general questions to some higher offices. I talked with several people at the governor’s office and the attorney general’s office.  Some of them were very helpful and understood my problem.  One of the attorney’s at the AG office answered my questions by listing off all the circuit courts that I might have to use in order to file a lawsuit.  I think it’s a really messed up system where the only way to question a possibly uninformed state employee’s decision is to file a lawsuit and waste 10s of thousands of dollars in attorney fees that will mostly not be reimbursed. That is a much bigger problem though.  I felt a sliver of hope when someone at the governor’s office informed me that the commissioner of the Alabama Medicaid Agency, Carol Steckel,  might be able to answer my questions and provide assistance.

What follows is the emails that were sent back and forth over a couple months between Mattie Jackson, Myself, Stephanie Azar, Keith Boswell, and Carol Steckel.  There has been no telephone conversation since.

Email Exchange

Feb 19th 2010 : Chase Gray –> Mattie Jackson

Hello,

Thanks for returning my call.

I own a company called Ratchet Software LLC.  We specialize in providing software that analyzes and organizes medicaid and medicare cost report data for vendors and nursing homes.   For example, if a dietary contract provider wanted to determine which county they should focus their next sales effort, they could make a better informed decision if they knew the statistics about dietary spending around the state.

Because we do data processing, printed reports or PDF digital copies of reports are not useful.  This is why we request the database of information because it contains structure and contextual data that is lost when converted to a printed form.  We are interested in this structural data because it allows us to automate the processing of the data.  Because of this, we would argue that the electronic database files and the actual cost reports do not contain the same data.

The Alabama Medicaid Agency uses Medimax software to process submitted cost reports.  This software uses database files ending in a .ADT extension which are not proprietary to Medimax, these are Advantage Database Server files.  These are the files that contain the data as well as the structure that we are interested in.   One problem when requesting the data is that the users of the Medimax software do not know that the data is stored in files inside the Medimax directory.

One issue that arose is that the cost reports contain a schedule that contains sensitive information.  This schedule is contained inside three specific files that correlate with the schedule name.  The schedule is D9 and the files are D-91.ADT, D-92.ADT, and D-93.ADT.

We are making a request for the following:

1. On a computer that has the Medimax software with all of the cost reports from 2008 uploaded to it.  This means that the user can open up the Medimax ALCR viewer and browse the cost report folders for 2008. 
2. On this computer the Medimax software is most likely under this directory, C:\Program Files\MediMax\ALCRV19  since this is the default installation directory.
3. In this directory there is a folder called Data.  This is where all the uploaded data gets stored.  
4. We are requesting any “folders” inside this data directory with the files D-91.ADT, D-92.ADT, and D-93.ADT deleted from each folder as they contain sensitive information.

Those folders each represent the data for a single facility’s submitted cost report.  Each .ADT file is for a schedule or a section of a schedule.

We would like the modified folders to all be burnt to a CD (they should fit fine) or sent over the web if someone knows how to do that there.  They would be too large for email most likely.

Please let me know before incurring any cost over $100.

Thank you,

Chase Gray
Ratchet Software LLC

I received a pretty quick acknowledgment. I would soon learn this would only happen for things they wanted to answer. Otherwise it would take weeks just to provide usually a one sentence answer.

Feb 19th 2010 : Mattie Jackson –> Chase Gray

Thanks, I will get back to you next week.

Mattie M. Jackson  |  Administrator, Office of the Commissioner /Alabama Medicaid Agency/ 501 Dexter Avenue  Montgomery, Alabama 36104 |  P:334.353.4361/ F:242.5097/ C:467.7105/ mattie.jackson@medicaid.alabama.gov

The following Monday I would receive what I consider a pretty ridiculous response. Mattie cites legislature being in session as one of the reasons for declining my request?  There are many reasons they could try to use to decline my simple request, but is this one even legal? There do appear to be legal consequences for declining information requests when they should have legally been fulfilled.  I’m sure they never happen, but I just can’t believe this was the response.  Isn’t legislature in session a significant portion of the year? In Alabama the normal session lasts 105 days according to Wikipedia. This was a total brush off and from this point on it was apparent that I wasn’t going to be taken very seriously.

Feb 22nd 2010 : Mattie Jackson –> Chase Gray

The Alabama Medicaid Agency is very proud of the services we provide to our clients and appreciates your interest in helping us to better serve the needs of our clients.  Unfortunately, with our Legislature in Session and many other initiatives, staff time and resources are limited; therefore, we will not be able to honor this request.

Thanks, Mattie

Feb 22nd 2010 : Chase Gray –> Mattie Jackson

Hello Mattie,

Would you be able to clarify that the information is available/requestable and the only reason the request is denied is due to time current time constraints?

Thanks,

Chase Gray

What’s interesting about the following response is that they declined my request without actually determining whether the data exists or is able to be requested.  From the following reply I get the impression that there was no attempt to fulfill the request at all.

Feb 22nd 2010 : Mattie Jackson –> Chase Gray

No, we did not go as far as to make that determination.

Thanks

Mattie M. Jackson  |  Administrator, Office of the Commissioner /Alabama Medicaid Agency/ 501 Dexter Avenue  Montgomery, Alabama 36104 |  P:334.353.4361/ F:242.5097/ C:467.7105/ mattie.jackson@medicaid.alabama.gov

Of course now I’m very interested in determining for what legal reasons my request is being declined.  I would accept a clear reason citing some case law or something from the Alabama Code.

Feb 22nd 2010 : Chase Gray –> Mattie Jackson

Can you cite where in the Alabama code I can read how my request can be denied completely due to time and not simply delayed a reasonable amount of time?

Feb 22nd 2010 : Chase Gray –> Mattie Jackson

Hello Mattie,

Do you have a fax number that our attorneys or myself can send documents to?

Thanks,

Chase Gray

I never got a response citing any legal reasons why my request was declined.  I did receive the following reply immediately when asking for their fax number.

Feb 22nd 2010 : Mattie Jackson –> Chase Gray

Certainly, 334-242-5097

Mattie M. Jackson  |  Administrator, Office of the Commissioner /Alabama Medicaid Agency/ 501 Dexter Avenue  Montgomery, Alabama 36104 |  P:334.353.4361/ F:242.5097/ C:467.7105/ mattie.jackson@medicaid.alabama.gov

The original supposed reason my request was declined was due to time constraints, so this is all I had to work with.  I felt that copying a folder off of a computer and deleting a few files within them would be much less time consuming than making thousands of copies.  These copies then have to be redacted.

Feb 22nd 2010 : Chase Gray –> Mattie Jackson

Hello Mattie,

Can you specify the difference in time spent from a standard request for paper cost reports, which I was told was a standard request that is fulfilled regularly.

Your initial email implies that the request would take longer than a standard one so it must be denied.  A standard request requires printing every report out and manually redacting information on all of them.

I’m just hoping for clarification on which part of the process was going to consume a large amount of time.

Thanks,

Chase Gray

I don’t know exactly why I was sent the following information if there was never any intention to fulfill my request.  I was given some false hope thinking that things were finally starting to work out like they should.

Feb 22nd 2010 : Mattie Jackson –> Chase Gray

See information extracted from the Agency’s Code regarding data request:  We will be more than happy to provide your requested data at our staff’s convenience if you want to have this data copied or have the data ran by our Statistical Support Unit at the costs specified below:

A request for one to ten copies of one-sided copier imprints shall be provided for a minimum charge of $5.00.  When more than ten copies are requested, the minimum charge of $5.00 shall be assessed for the initial ten copies plus fifty cents for each additional imprint.  This schedule of charges will cover the administrative cost of copying the requested material.

In addition to making the actual copies of printed documents, there is often a considerable amount of staff time involved in research and compiling of information.  A record of the staff time spent on each project will be kept and charged at the rate of $15.00 per hour.

Less than 30 minutes:              no charge

First 30 – 60 minutes:               $15.00

Each additional 30 minutes:     $  7.50

Additionally, there is a $38.00 per hour charge if our Statistical Support Division runs the data.

Please let me know how you’d like for us to proceed, or if you want staff to compute the cost to run the data requested. Also, I am not certain if what you are asking for is available electronically, but will find out tomorrow from the director of this unit and let you know.

Hope this helps, Mattie.

Mattie M. Jackson  |  Administrator, Office of the Commissioner /Alabama Medicaid Agency/ 501 Dexter Avenue  Montgomery, Alabama 36104 |  P:334.353.4361/ F:242.5097/ C:467.7105/ mattie.jackson@medicaid.alabama.gov

Feb 25th 2010 : Chase Gray –> Mattie Jackson

Hello Mattie,

Are you able to provide an estimate on price and time that it will take to obtain the requested data?

Thanks,

Chase Gray

Mar 1st 2010 : Mattie Jackson –> Chase Gray

Staff is reviewing this request.  Hope to have something to you this afternoon or early a.m.

Thanks.

Mattie M. Jackson  |  Administrator, Office of the Commissioner /Alabama Medicaid Agency/ 501 Dexter Avenue  Montgomery, Alabama 36104 |  P:334.353.4361/ F:242.5097/ C:467.7105/ mattie.jackson@medicaid.alabama.gov

Mar 9th 2010 : Chase Gray –> Mattie Jackson

Hi Mattie,

I was wondering what the status is of the information request?  In the previous email you said you would get back to me within a day or so. 
The information doesn’t need to be converted in any way, just files burnt to a CD with the sensitive information removed as detailed in the original email.

Let me know if there are any issues,

Thanks,

Chase Gray

My request is declined again.  This time they inform me that the data is not available in the form I requested.  At this point I could not say 100% that it was there, but I was pretty positive their Medimax software stored all of the data in ADT files.  For some reason programming languages are brought up, which would only be required if they were attempting to automate the process I suppose.  Pointing and clicking on folders and moving them to the burn folder is what I was hoping would be done to fulfill my request, despite this being inefficient.

Mar 9th 2010 : Mattie Jackson –> Chase Gray

I apologize for the delay in responding.  With limited staff due to the statewide budget crisis affecting Medicaid as well as the state of Alabama, there are various staff constraints I am certain you can understand.

You requested the cost report in electronic format.  Since the report contains protected health information, we can only provide a hard copy after redacting the PHI.  There’s a $30 fee for each copy.  To provide the data you are requesting in a hard copy form requires massive copying after removing the PHI.  This information is not available in the form you requested.  Additionally, we do not have neither the expertise nor the time to learn the programming language to get this information in the format you requested.

The information I provided in the email below regarding cost does not relate to cost reports, so disregard.  It only applies to request to run and copy queries.

Thanks, Mattie

Mattie M. Jackson | Medicaid Administrator| Office of Commissioner Steckel |Alabama Medicaid Agency |501 Dexter Avenue |Montgomery, Alabama 36104|  P:334.353.4361| F:242.5097 | C:467.7105| email: mattie.jackson@medicaid.alabama.gov

Mar 9th 2010 : Chase Gray –> Mattie Jackson

Hi again,

I am not interested in digital copies of the cost report.  I do understand that these contain sensitive information.  I am interested in the electronic files that already exist that make up the database.  Like I specified in the original email, each filename refers to a schedule.  The sensitive information that is usually redacted is in schedule D-9.  The files that contain this information are D-91.ADT, D-92.ADT, and D-93.ADT.    I will copy the original request below for convenience.

Because we do data processing, printed reports or PDF digital copies of reports are not useful.  This is why we request the database of information because it contains structure and contextual data that is lost when converted to a printed form.  We are interested in this structural data because it allows us to automate the processing of the data.  Because of this, we would argue that the electronic database files and the actual cost reports do not contain the same data.

 

 

 

 

 

The Alabama Medicaid Agency uses Medimax software to process submitted cost reports.  This software uses database files ending in a .ADT extension which are not proprietary to Medimax, these are Advantage Database Server files.  These are the files that contain the data as well as the structure that we are interested in.   One problem when requesting the data is that the users of the Medimax software do not know that the data is stored in files inside the Medimax directory.

One issue that arose is that the cost reports contain a schedule that contains sensitive information.  This schedule is contained inside three specific files that correlate with the schedule name.  The schedule is D9 and the files are D-91.ADT, D-92.ADT, and D-93.ADT.

We are making a request for the following:

1. On a computer that has the Medimax software with all of the cost reports from 2008 uploaded to it.  This means that the user can open up the Medimax ALCR viewer and browse the cost report folders for 2008. 
2. On this computer the Medimax software is most likely under this directory, C:\Program Files\MediMax\ALCRV19  since this is the default installation directory.
3. In this directory there is a folder called Data.  This is where all the uploaded data gets stored.  
4. We are requesting any “folders” inside this data directory with the files D-91.ADT, D-92.ADT, and D-93.ADT deleted from each folder as they contain sensitive information.

Those folders each represent the data for a single facility’s submitted cost report.  Each .ADT file is for a schedule or a section of a schedule.

We would like the modified folders to all be burnt to a CD (they should fit fine) or sent over the web if someone knows how to do that there.  They would be too large for email most likely.

Please let me know if there are any problems,

Thanks,

Chase Gray

Mar 17th 2010 : Chase Gray –> Mattie Jackson

Hi Mattie,

I was just checking on the status of the information request.

Is there somebody else I should be speaking with that has more direct interaction with the data?  Maybe somebody in data processing or IT?  There is no programming or “languages” involved in my request.

Would it be better for me to start with a request verifying the existence of the files?

Let me know how we can move this along,

Thanks,

Chase Gray

New Direction

I had to give up on the direction things were going.  I decided that I would email the attorney and the commissioner of the medicaid agency.  This is the same attorney, Stephanie Azar, that I had so much trouble reaching in the past.  Luckily I received a reply from her, possibly due to also contacting Carol Steckel, the commissioner.

I thought that things would go smoother now that someone who understood the law was working with me.  So what follows is a set of emails that were CC’ed to everybody listed in the “Cast” section above.  Replies were much more consistent but still a little slow, possibly due to the accountability of everyone getting a copy.

In order to try and solve the problem of the agency lacking the expertise to extract the files from the computer, I thought I’d take an easier approach.  Each nursing facility sends in a diskette with the .adt files on it.  These files must be stored for a number of years by the agency.  I decided to form my request around the data on these diskettes instead.

Mar 23rd 2010 : Chase Gray –> Mattie Jackson, Stephanie Azar

Hi Mattie,

I have not heard from you in over a week or two.  I am just checking in on the status of things.  If I need to take a different approach please let me know.

In your response you stated that you didn’t believe anybody had the expertise to obtain the files.  If there is a problem finding the files on the actual computer, the files are also located on the disks submitted to the agency by each nursing facility.   So if the existence of the files on a computer in your office is in question, they also are on each of the submitted disks that facilities send in.

On each disk there is  a folder called BACKUP1, inside of this there is a zipped folder called Data.  This “Data” folder contains all of the files that I am requesting.

As I mentioned before, due to the sensitive data included in schedule D-9 contains sensitive information.  I don’t need this schedule.
This schedule is contained inside three specific files that correlate with the schedule name.  The schedule is D9 and the files are D-91.ADT, D-92.ADT, and D-93.ADT.  If this must be verified by Medimax before you can release it, then let me know of the costs involved.   I believe it would more time efficient to get the data off of a PC, but the disks work just as well.

If each of these disks data could be included on a single CD, that would be perfect.  As long as the folders are separate it’s not necessary that they are labeled correctly, as we should be able to determine which facility it is based on the data files.  The data shouldn’t take up more than 100-200MB.

One of our partners is actually coming to your office for different reasons related to their other business around April 2nd I believe.  If you would like to speak with her, then please let me know.  She may have some questions of her own relating to this matter, since she hasn’t been involved until recently.

Please let me know if there are any problems or unusually high costs.

Thanks,

Chase Gray

Mar 24th 2010 : Stephanie Azar –> Chase Gray

Mr. Gray,

The Alabama Medicaid Agency (Medicaid) has received your data request.  The data you are specifically requesting does not currently exist in a written public document which can be provided to you for copying and/or electronically sent to you.  To provide such information would require that Medicaid utilize staff time and state resources to extrapolate and create the document that you are requesting.  Under the Alabama Opens Record Act, Ala. Code 36-12-1, et al., an Alabama citizen has a right to inspect and copy any public writing of the State.  Your request is not a request that Medicaid is required to provide under the Opens Record Act because it is not information in a “writing” that currently exists at Medicaid.   There is no requirement that Medicaid create the document that contains the information/data that your are requesting.

Medicaid appreciates your contacting us to see if this information is available for inspection, but since this information does not currently exist in the form you are seeking Medicaid cannot honor your request.   As stated to you previously, Medicaid will make available for your inspection and copying the cost report information.  If you desire to have this information then please contact Keith Boswell at 334-242-2311.   Thank you.

*************************************

Stephanie McGee Azar

General Counsel

Alabama Medicaid Agency

Office of General Counsel

As soon as I felt like I was making progress, it now seemed like we took a step backwards.  I consulted our legal consultant for the company.  We put together a letter addressing every one of their concerns with relevant case law and attorney general opinions.  After putting this letter together I felt confident that there was no legal reason that my request could be declined.  I should give love to the attorney who helped me with this at 4am one night while I purchased us McDonald’s breakfast. Our attorney is Brian Payne of Small Business Law Firm, who is now attempting to move many of his services online so he may be affordable and accessible to those willing to use legal services online. I will embed the letter as a PDF to maintain the formatting.

Mar 25th 2010 : Chase Gray –> Stephanie Azar, Carol Steckel, Mattie Jackson, Keith Boswell

Ratchet – Ltr. to Azar – 3-25-10

This response from Stephanie is one of the most interesting. In it she agrees that the information must be already compiled and organized in a database or electronic form.  The request I am making is for the files created by the Advantage Database Server embedded in their Medimax software.  The .ADT files I’m requesting represent the database.  Basically, there is a single .ADT file for each schedule in the report for each facility. All of the data is already compiled in this format and they are separated in different files for further organizational benefit. I would actually prefer that the agency not do any extra compilation or organization, because the database server has done a great job at it already.  So one of the most interesting questions is raised in this email.  Is burning specific folders from a hard drive onto a CD considered “compiling and organizing” in the way that this case law intended?  I’ll revisit this at the end of the post. The next interesting thing about this reply is that there is finally some sort of verification that the data I am requesting does exist.  She refers to the 218 separate folders, which would be one per facility.  This means that somebody finally did find the folder I was asking about and it was indeed there. She refers to the cost report information available on the web site after April 1st. They will be posting the PDF copies of the reports on their web site.  This is again not helpful at all because information is lost when converted into a PDF from its original database form. Honestly, I felt like this last decline email was really reaching for any possibly reason to issue a denial.  The compilation and organization argument is pretty shaky when put up to any test.  How many files can I request before their limit is reached and its considered compiling and organizing?

Mar 30th 2010 : Stephanie Azar –> Chase Gray

Mr. Gray,

The Alabama Medicaid Agency (Medicaid) has received your request for reconsideration regarding Medicaid not being able to provide you with the information you are requesting due to the information not being available for inspection or copying.    Medicaid appreciates your response and Medicaid is aware of the case law and the additional information you provided in your response.  It is not Medicaid’s position that the information requested must be in a physical written document.  However, it is Medicaid’s position that the information has to be already compiled and organized in a written form, electronic form, or data base.  This is in keeping with the case law that you cited in your response.  As stated previously, your request is not compiled and available for your inspection or copying in any form.

Medicaid appreciates you clarifying your information request but Medicaid understands what you are requesting.  Such a request will require Medicaid to extract the files you are requesting from 218 separate folders and once such information is extracted will have to be compiled and organized into a document.  Since the information is not already compiled and organized, Medicaid is not required to prepare such information for your inspection and copying.   Medicaid appreciates your request for this information but unfortunately is unable to honor your request for the reasons stated above.  However, the cost report information for each Medicaid nursing facility in Alabama will be available on the Medicaid Agency website on April 1^st at www.medicaid.alabama.gov.  Thank you.

 

*************************************

Stephanie McGee Azar

General Counsel

Alabama Medicaid Agency

Office of General Counsel

In my final request I provide them with two commands that will extract the data folders and copy them all to the desktop ready to be burned.  I also provide a single command that will recursively delete the files containing any sensitive information.  It appalls me that this could be considered more time consuming and difficult than printing out all of the thousands of pages and redacting them all.  In the event that this was still declined, I decided to test my previous theory and simply request a single folder.  A single folder should not be subject to “compiling and organizing.”

Mar 31st 2010 : Chase Gray –> Stephanie Azar, Carol Steckel, Mattie Jackson, Terrell Flowers, Keith Boswell

Ms. Azar:

Thank you for reconsidering my request.

The information that is going to be put online is not in a parsable form, meaning that by putting it into PDF the structure and utility of the database is lost.  We are not requesting the cost reports, but the database, which includes the structure that is required.

My request does not require that Medicaid “compile and organize” any of the data.  The data, in the form that it already exists, is sufficient to determine which facility it is for and work with.  Because the data is on a computer, the act of manually compiling and organizing should no longer be a requirement, since automation is one of the most significant advantages of having electronic data.   I explain below in more detail how it is as easy as double clicking a file.

In a standard paper request, the department photocopies all of the reports and manually redacts information on schedule D-9.  This seems to fit the definition of compiling and organizing more than my request.  In the standard request, each report must be printed or found (similar to finding an electronic folder).  Information must be redacted from each report (similar to deleting a file or two).  These reports, that are made up of a significant amount of pages, must be organized and separated based on which facility they are for.  They must all be put into one location for shipping to the requester (similar to burning the redacted folders to a CD).

I understand that I will be charged $15/hour for staff time to accomplish this.  I would like to be informed if the cost is to exceed $150, but otherwise it’s fine.

Below I provide a link a simple batch file which allows you to simply double click it and the result will be a single folder on the desktop called “MedimaxData” that can be burned to a CD.  The resulting folder will have the correctly deleted files and there is no “compiling or organization” required.  If you need to verify that it indeed does this, it should be a straightforward task for anyone who regularly deals with computer issues such as your IT employee, Terrell Flowers.  If the only issue is the labor involved in copying files, then this would make it as simple as clicking on this file.  It consists of two operations, copy all directories, then delete the D-9 schedules.  This is done succinctly in two commands (shown below):

xcopy /I /S “C:\Program Files\MediMax\ALCRV19\Data” “%UserProfile%\Desktop\MedimaxData”

del /s “%UserProfile%\Desktop\MedimaxData\D-9*”

Simply double clicking the batch file will execute these and will result in exactly what I am requesting, fully redacted, on the desktop.  It just must be run on a Medicaid PC that has these “218 folders” for the latest fiscal year.  These two commands are the extent of any compiling of data, and they are automated further by becoming a batch file that can be double clicked.   I don’t see how this is more time and trouble than working with paper documents.

The file can be downloaded here http://dl.dropbox.com/u/112959/ExtractData.bat

If you are still unable to fulfill this request, then I would like to rephrase my request.   I would like to simply request ONE of the folders under the directory “C:\Program Files\MediMax\ALCRV19\Data\”  that does not contain the word “SAMPLE” in the folder name.  Of course, removing the D9 files is still required for redaction purposes.  Because I am only requesting one folder, there is no need to compile or organize any data.  If possible I would appreciate this folder to simply be emailed since it will be around 350KB.  This would save any of the trouble of burning a CD or mailing anything.  I am only requesting this if the previous request is unable to be fulfilled.

Thanks again for reconsidering my revised request,

Chase Gray

I received an interesting reply the following morning. The commissioner, Carol Steckel finally sent a reply to me.  At first I was excited, but then I noticed the email was empty except for the “Sent from my iPhone” signature. It appears that I received this email by mistake from Ms. Steckel while viewing her emails on her iPhone.  I was surprised though to see that someone who can appreciate the advantages that an iPhone brings doesn’t support the advantages that a electronic database can have.  Regardless, this at least shows that my emails were being seen by Ms. Steckel and either read or forwarded or something.

Mar 31st 2010 : Carol Steckel –> Chase Gray

I believe I received this email by mistake, but at least it shows that Carol was seeing my emails and as early as 7:30am apparently.

Finally, to close the deal, Stephanie sent me a reply addressing none of my questions.  She simply stated that there is no requirement they provide this information.  In the case law we provided in our formal letter, we showed that other agencies have been required to provide their databases and files, why are they different?

April 9th 2010 : Stephanie Azar –> Chase Gray

Mr. Gray,

The cost report for each Alabama Medicaid nursing facility is available on the Alabama Medicaid website.  As stated previously, there is no requirement that Medicaid provide you this information in a different format.  Thank you.

Concluding Questions

So this whole ordeal brings up quite a few questions:

1. Is the agency required to keep the electronic information I am requesting?

The answer to this is definitely yes.  The state has an organization that creates a document specifying how long each agency must retain their files.  For the Alabama Medicaid Agency, they certainly must retain even temporary files for at least 3 years if I read it correctly.

2. What information is lost when moving from a database to a paper/PDF form?

There are many ways to answer this question.  It is definitely not a question asking whether information is lost, it obviously is. I’m going to answer it as simple as possible in a technical way. Each specific piece of information is explicitly referenced by two identifiers, the row ID and the column name.  The row ID in this case identifies each facility uniquely, and the column specifies which piece of information it is.  When this information is in a database, we have the ability to request a single piece of information for all facilities (rows).  This request would be time consuming for paper or PDF reports.  We can also make the same request on a single column for facilities in a specific county.  Now there are conditions that have to be checked before including a piece of data, which would add even more time to the manual paper/PDF task.  With the database form, these queries could get as complex as the programmer could handle.  It is obvious that we lose this ability as the queries get more complex with paper and PDF files.  The Alabama court has acknowledged that a database contains utility that is not available without it.

3. Can my request be declined before even looking into the details of it?

Initially my request was declined because they said the agency was busy.  As far as I can tell from the Alabama Code, there is no justification for declining a request because the agency is too busy.  If the request would take an unreasonable amount of time and effort, then it can be declined.  I would think that this would actually apply to requesting thousands of paper  copies, instead of digital files.  The Alabama senate actually even passed a bill stating that an agency has 15 days to comply with a request once it has been made.  It used to be that there was no real time limit in Alabama for information requests.

4. Throughout this ordeal, many of the state employees refer to their policy of not providing electronic copies or files.  Is this policy recorded somewhere?  Are they allowed to have a policy that trumps the Open Records Act?

I don’t have a good answer to this.  I couldn’t find any policy on open records from the Alabama Medicaid Agency.  I would think that the Open Records Act trumps any decisions made locally, otherwise each agency could do what they wanted.  This supposed policy seemed to change as I brought up case law or the Open Records Act.

5. If I cannot request 218 folders because it requires “compiling and organizing into a document” then can I request 1 file?  Could I request 1 folder?  At what point does the number of files require “compiling and organizing?”

This is a really interesting question.  The last request I made I asked for a single folder instead of all of them.  I felt that this required no “compiling and organizing.”  In the original denial, it seemed that the properties of my specific request were squeezed to fit into the only denial reason left which was that they had to “compile and organize the files into a document.”  I don’t know how document is defined here, but the only definition that makes sense to me is CD.  So the folders are the files and they have to copy them to a CD which requires compilation into a document?  I don’t really understand this one at all, and I would really love to hear some  clarification on this topic especially.  I might resubmit my request for a single file since it hasn’t been addressed yet.  This is something I’d like to hear feedback about from the attorneys and lawyers out there.

6. Was their purpose to stall until they could try and fall back on the PDFs they were planning to put online?

As their deadline for putting the reports online in PDF form drew closer, it did appear that the responses were getting further and further apart.  Finally, after that deadline passed, I received the most terse email of all of them stating that they will not comply with my request without addressing any of my questions.  It does appear as though they did stall, but it may just be coincidence.  As stated in item one, these PDFs are nowhere near the same thing as what I am requesting.

Call to Action

Finally, I would like to ask that anybody who reads this and find it just as troubling as I do, please click the Digg or Reddit button below.  The more people that can read about this experience the better.  If you have any news contacts in the Alabama area then please refer this to them or them to me.  If you have a blog relating to FOIA issues, then I would be happy to let you take anything you want from this post.  I’ve thought about legal action, and it may be worth the effort but at this point I am moving on to some other work for a bit.  If you are an attorney with any advice, I’d love to hear it.  If you have a connection to somebody involved with Alabama Government then please bring this to their attention.

• Upvote this article on Digg or Reddit, you can sign in using your Facebook login so you shouldn’t have to register.
• Pass the link along to anybody you know that has experience with these issues or Alabama Government.
• If you know somebody at an Alabama press organization, they might have some interest in this ordeal.
• Email or call the following people
  • Attorney General Troy King – Twitter: @TKtheAG – Facebook: http://www.facebook.com/tktheag – Contact: http://www.ago.state.al.us/contact_online.cfm
  • Governor Bob Riley – Twitter: @governorRiley – Contact: http://www.governor.state.al.us/contact/contact_form.aspx
  • Contacting any of the people in the ‘Cast’ section above, especially the commissioner Carol Steckel

I appreciate all the support I’ve received already,

Feel free to email me if you have questions of feedback, my email is available at the bottom or the right bar.

Chase Gray

I searched thoroughly for an easy way to retrieve information from this set of .ADT files I was dealing with. Sybase provides tools to work with the data, but all of them require using their software or drivers developed by them. I would have settled for this if it wasn’t such a hassle to figure out how to use them. I asked the question on Stackoverflow.com and got an answer from one of the employee’s of Sybase. While helpful, the suggested Perl driver was one of the things I was avoiding. I did attempt it and after a little hassle, moved on to creating a simpler solution.

One solution was to simply use Sybase’s provided “Advantage Data Architect” software to open the .ADT files and export them all to SQL. This is possible with their software, but it would not work for my situation. The developers of the program outputting these .adt files used a separate embedded database for each new report that was being worked on. This meant there was hundreds of folders, each containing 30-40 .ADT files making up their own embedded database. I needed something that could automate the extraction of this data.

Data Format

To cut this short, I decided to open up the .ADT files with a hex editor and see if I could get the information I needed out. I only needed the column names, their types, and the actual rows of data. To aid anybody else doing something similar to this, I’ll include some of my notes about the format here:

• The data is stored at the very end of the file.
• The byte offset when the data starts is located at byte 32 and is 32-bits
• Byte 24 contains a 32-bit integer specifying the number of rows
• Byte 36 contains the a 32-bit number representing the number of bytes each record consumes
• The header is 400 bytes
• Column information is after the header, before the data. Each column entry is 200 bytes.
• The number of columns can be calculated (data_offset-400)/200
• Column names consume the first 128 of each column entry.
• Byte 130 contains a 16-bit integer representing the type of data.
• Byte 136 contains a 16-bit integer, the length of the field in bytes
• Character = 04, Double = 0A, Integer = 0B, Shortint = 0C, CICharacter = 14, Date = 03, Time = 0D, Timestamp = 0E, Autoinc = 0F

This was enough information to get what I wanted.

Ruby-ADT Gem

I took the existing DBF gem by Keith Morrison. Most of the credit for this gem goes to him as I borrowed the structure heavily from his gem.

I’ll provide an example of what is now possible with my new ADT gem. No drivers are required and it’s as simple as installing the gem. I’ll include a snippet of the readme from my Ruby-ADT gem on github, borrowed heavily again from the DBF gem.

Installation

gem install ruby-adt

Basic Usage

Load an ADT file:

require 'rubygems'
require 'adt'
 
table = ADT::Table.new("test.adt")

Enumerate all records

table.each do |record|          
	puts record.name          
	puts record.email
end

Load a single record using record or find

table.record(6)
table.find(6)

Attributes can also be accessed through the attributes hash in original or underscored form or as an accessor method using the underscored name.

table.record(4).attributes["PhoneBook"]
table.record(4).attributes["phone_book"]
table.record(4).phone_book

Search for records using a simple hash format. Multiple search criteria are ANDed. Use the block form of find if the resulting recordset could be large otherwise all records will be loaded into memory.

# find all records with first_name equal to Keith
 
table.find(:all, :first_name => 'Keith') do |record|  
	puts record.last_name
end
 
# find all records with first_name equal to Keith and last_name equal to Morrison
table.find(:all, :first_name => 'Keith', :last_name => 'Morrison') do |record|   
	puts record.last_name
end
 
# find the first record with first_name equal to Keith
 
table.find :first, :first_name => 'Keith'
 
# find record number 10 
 
table.find(10)

Migrating to ActiveRecord

An example of migrating a DBF book table to ActiveRecord using a migration:

require 'adt'
 
class CreateBooks < ActiveRecord::Migration      
	def self.up            
		table = ADT::Table.new('db/adt/books.adt')            
		eval(table.schema)        
		table.each do |record|                  
			Book.create(record.attributes)            
		end      
	end  
	def self.down        
		drop_table :books    
	end
end

Limitations and known bugs

• ADT is read-only
• External index files are not used
• Dates are not currently handled correctly.

Future Directions

I am going to continue working on this library. I have already heard from a few others that found this useful immediately after I posted it online. I was surprised people could find it so quickly. I’m sure there are bugs. I would appreciate any problems to be reported to the Ruby-ADT Github page. Emails are also welcome, but any issues should be directed to the github page and I’ll get to them when I can.

Hope this helps somebody out there dealing with this format.

- Chase Gray

Prerequisites

The first thing I needed to do was to successfully configure and compile mdbtools. This probably took the longest as there are a few different version out there. I had the most success with these instructions from the author’s git repository, http://www.automatthew.com/2008/02/how-to-compile-mdbtools-on-mac-os-x.html?showComment=1219246620000. I was also able to compile this version on my ubuntu machine pretty easily. The biggest hangup was with a couple of dependencies. I also had a weird error where the autogen.sh was not detecting my libtools library, but I knew this was installed. I commented out this check in the autogen.sh and things all worked, luckily.

Understanding mdb-tools was not very difficult. It contains various libaries for interfacing with it through C, etc. The only thing I cared about were the command line tools that it provided, listed here. The commands I will use in this post are mdb-tables, mdb-schema, mdb-export.

The other requirement is to have a MySQL table created ready with permissions, etc. that will end up holding the data from the combined .mdb files.

Step 1: Extracting the Schema

One important thing to note about my situation is that all the .mdb files have the same schema. Knowing this, I was able to simply find a random .mdb file and base my MySQL schema on this.

I have my .mdb files stored in a shared directory and so within my Rake task I used the following code to grab the first one (I’m sure there is a better way).

first_mdb_file = ''
#get first mdb file
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    first_mdb_file = "#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file}"
    break
  end
end

Knowing the path to the first .mdb file the schema can be extracted by getting the result of the following shell command (the code surrounded in backticks is executed on the command line of the system). The -S flag tells it to santize some of the table names by replacing spaces with underscore, etc. I needed this for my tables because there were some problems without it.

schema_sql = `mdb-schema -S #{first_mdb_file} mysql`

Step 2: Building the Tables

Since I’m doing this in a Rake task in Rails (you could do this in plain ruby), I am using an ActiveRecord connection to interface with my MySQL database. I have a class called ExternalDatabase that allows me to connect to a different database without modifying the one that my main Rails program uses. I will list it below, I don’t remember exactly where I got it from.

#Class used to connect to external databases without modifying the exiisting connection used by Rails
class Data::ExternalDatabase < ActiveRecord::Base
  self.abstract_class = true
 
  def self.set_source(configuration_name)
    source = ActiveRecord::Base.configurations["sources"][configuration_name.to_s]
    establish_connection source
  end
 
  def self.set_configuration(config)
    source = config
    establish_connection source
  end
end

As you can see, I also have a section in my database.yml where i list connection details for various source databases. This isn’t related to this post though.

So, assuming we have this class, we can use the following code to load our new schema into out MySQL database.

Data::ExternalDatabase.set_configuration(:adapter  => "mysql",
:host     => "localhost",
:username => YOUR_DATABASE_USERNAME,
:password => YOUR_DATABASE_PASSWORD,
:database => YOUR_DATABASE_NAME,
)
db =  Data::ExternalDatabase.connection 
 
puts "adding if exists clause to drop tables"
 
schema_sql = schema_sql.gsub(/DROP TABLE (.*);/, 'DROP TABLE IF EXISTS \1;')
schema_sql = schema_sql.gsub(/--.*$/, '')
puts "executing schema sql"
lines = schema_sql.split(';')
 
lines.each do |sql_line|
  next if sql_line.blank?
  db.execute(sql_line)
end

The first part builds my database configuration hash that I pass in to ExternalDatabase. I save the connection into a variable called ‘db’. One of the important things to note about this and mdb-tools is that it does not check if the table currently exists before trying to drop the table. So I added a snippet of code that removes the comments and adds ‘IF EXISTS’ to the ‘DROP TABLE’ clause in the schema sql.

For some reason, trying to execute this entire schema sql string with db.execute causes various errors that I couldn’t figure out. By separating the schema sql into individual sql statements terminated with semicolons seemed to fix this problem. The last bit of code does the separation into lines and executes the sql on the database. After all this, the database should have the correct schema.

Step 3: Load Data

One of the last things we want to do is actually load all the data from the different mdb files into the single MySQL database.

tables = (`mdb-tables #{first_mdb_file}`).split(' ')
Dir.entries("#{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/").each do |mdb_file|
  if mdb_file =~ /\.mdb/
    puts "Current file: #{mdb_file}"
    tables.each do |table|
      puts "Current table: #{table}"
      insert_sql = `mdb-export -I -S #{RAILS_ROOT}/../shared/system/data_sources/SC/sc_mdb_files/#{mdb_file} #{table}`
      insert_sql = insert_sql.gsub(/;/, ',')
      insert_sql = insert_sql.gsub(/(.*?\))(\s+INSERT)/, '\1;\2')
      lines = insert_sql.split(';')
      lines.each do |sql_line|
        next if sql_line.blank?
        db.execute(sql_line)
      end
    end
  end
end

So the first line runs mdb-tools command line tool ‘mdb-tables’ to get a list of the tables separated by spaces. I could have done this through my previously created schema as well.

This code goes through every .mdb file in my directory and uses mdb-tools to extract the data as sql ‘INSERT’ statements from the command-line.

In the inner-most loop, I clean up the returned value from mdb-export because it does not include semi-colons after each INSERT statement. I also remove any semi-colons before adding any because some of my fields had semi-colons in them. Again, I separate each INSERT statement into a single statement and execute these individually in the last section of the code.

After all this you should have all your data combined into a single MySQL database. One problem might exists though. Certain tables, such as tables that contain general purpose labels, etc. for other tables to refer to, might contain duplicates. This wouldn’t have caused a big problem for me because I just needed to extract data, but if you needed to continue using this database you would need to remove the duplicates.

Final Step: Removing Duplicate Rows

There seem to be a lot of different ways to remove duplicate rows from a MySQL table. I wish there was a simple solution, but I couldn’t find one. The following code does the following:

1. Create a temporary table that has the same schema as the table we are removing duplicates from.
2. Insert all distinct rows into this new temporary table.
3. remove all values from the original table.
4. Insert all of the rows from the temporary table back into the original table.
5. Finally, drop the temporary table.

tables.each do |table|
  db.execute("CREATE TABLE temp LIKE #{table}")
  db.execute("INSERT INTO temp select distinct * FROM #{table}")
  db.execute("TRUNCATE #{table}")
  db.execute("INSERT INTO #{table} select * from temp")
  db.execute("DROP TABLE temp")
end

Thanks for reading, I hope this was helpful to others that have some similar problems.

- Chase

Problem 1: How to let Google index my data that requires a login

A prerequisite for this step is to submit sitemaps for your site’s content to Google Webmaster Tools.  Once you have done this, at some point in the future Googlebot will attempt to traverse your page and index its contents (hopefully).

The content on my site requires a user to pay a subscription fee in order to have free reign and browse through whatever they please.  I’m not very concerned about any single piece of information getting leaked, instead I’m protecting the resource the site provides as a whole.  By letting Google index your site, you’re obviously opening up your content to outside viewers that are not logged in.  The extent that you need to protect your content could vary from what my needs are.

What I want is for a user to be able to type their name into Google search engine and find out that they are referenced in the data provided by Myhealthcaresource.com.  So, I need to somehow let Googlebot traverse my site, even through it requires a login by a normal user.

I’m using Restful Authentication and a role requirement system, so this is how I include this exception.

class FacilitiesController < ApplicationController
 require_role :basic, :only => :show, :unless => request.env['HTTP_USER_AGENT'] =~ /.*Googlebot.*/
end

This could vary depending on your authentication system, but the part that will remain the same is the check if the user agent string contains ‘Googlebot’.

That’s all it takes in order to allow Googlebot to traverse my controlelr that previously required a login.  A potential problem that could come up is that users could ‘fake’ their user agent to say ‘Googlebot’ and gain access to the site.  This could be a problem for some sites, and we would keep an eye out for this in our logs or through analytics software.  We provide a free trial to all our users before they decide to pay, so uers get a free look at the data already and there is no reason to try to sneak around the site to see it.  We’re not concerned with this problem, but it could be an issue for others.

Problem 2: How to prevent Google from caching private pages

Once Google starts indexes pages of your site that require a login for a normal user, it will also start caching all the sites that it visits.  If you are worried about your internal content getting cached by Google you will want to prevent Google or any other robots from caching.  This is simple and is accomplished by putting the following in the heading of your main layout file.

<META name="robots" content="index,follow,noarchive" />
<META NAME="googleBOT" CONTENT="noarchive" />

When Googlebot reaches your page and sees this tag, it will not cache it.  I have this on the Myhealthcaresource layout, and by searching myhealthcaresource in google you can see there are no cache links.

Problem 3: Allowing Users Referred From Google View Content Without Logging In

When a user sees a link to myhealthcaresource on a set of Google search results, they are most likely to the detailed facility reports that I submitted in a sitemap.  They probably searched their company name or their own and found that they were referred to by my site’s information.  When someone clicks this link in Google, by default they would get redirected to a page asking them to log in.  This would make almost every user referred from Google resort to their back button immediately.  I noticed this behavior during the period when my site functioned this way.  These users from Google have no idea what kind of information our site offers and they might be a potential customer, so we don’t want them to leave because we present a log in form when they first arrive.

An alternative is to allow users referred to our site through a Google search to view the entire page.  We decided to do just this.  When they click the link in Google’s search results they are not asked to log in and they are shown the same page a logged in user would see.  What happens when they try to browse the rest of the private content?  They are asked to log in, but we already have their interest at this point and they are more likely to stay if they found something that was useful to them on the page they landed on.

So how do we accomplish this?  In a similar way that we allowed Googlebot to traverse our pages, we can allow users that were referred from Google to access our page:

class FacilitiesController < ApplicationController
  require_role :basic, :only => :show, :unless => request.referer =~ /.*google.*/ || request.env['HTTP_USER_AGENT'] =~ /.*Googlebot.*/
end

Of course this code would vary slightly if you are using a different authentication scheme.  The core idea would be the same though.  Now, the potential problem that could come up with this is that a malicious user uses Google to repeatedly access your site by creating clever search queries.  In our case, if a user is this intent on stealing the data, we weren’t likely to win them as a customer in the first place.  We can also monitor this through analytics tools and if it is abused we will look into an alternative solution.  It could be possible to restrict the number of Google referrals a certain IP can have in a time period or something.

I hope that some of this work can be useful to people who are looking for ways to leverage Google for their dynamically generated login-only content.  We see people come from Google searches every day, and now they stay much longer as a result of these changes.

In a future post, I will briefly talk about another method our site uses to send links to users that allow them to browse the site freely without logging in.  After a certain time period these links expire and the users must sign up to browse the site further.

I found myself working with some data in a hash generated from scraping a website. The website had some duplicate content and I didn’t feel like it was necessary to clutter my scraping code with tests for duplicate content while it was scraping. After scraping the data and turning it into a hash, I wished to remove the duplicates with some post-processing. I felt like this should be pretty simple, except that the elements I wanted to test for uniqueness contained a deeply nested structure of hashes and arrays. Here is an example an element in an array I wanted to uniqify:

{:name => 'bob', :posts => [{:title => 'First Post!' :comments => [{:commenter => 'Alice', :text => 'Hi bob!'}]}]}

The #uniq method does not work as expected when there is a Hash element in the array, it only uses the object_id to determine uniqueness. There are some solutions online for dealing with this.

If you know that each element inside the array is a hash with no sub-hashes you can use the solution presented in the answer of this stackoverflow.com question. The first solution uses #inject to build a new array with only unique items. Whether an item is included in the array already or not is determined with the Hash’s #include method.

a.inject([]) { |result,h| result << h unless result.include?(h); result }

The second solution presented in the same post:

a.map {|h| h.to_a[0]}.uniq.map {|k,v| {k => v}}

This flattens all of your hashes into an array. It then calls uniq on the resulting array after all the hashes are flattened. This means that for an item to be unique it must have the same key AND value. After finding the unique elements, it then maps the items back into a hash.

In this post, Mike Burns solve this problem by redefining #hash and #eql?. If you are doing this kind of comparison often, this would probably be a better approach. He does note though, that by doing this the #dup method no longer works as expected.

So it was obvious through searching that I was going to have to figure this out on my own. I was a little worried that I would have to implement an ugly recursive solution that searches the entire tree to find all duplicates. While looking into this solution, I felt that it would be very nice if #uniq would accept a block that would allow me to return true if the given parameter is a duplicate, thus removing it from the returned array. This is not implemented in my current ruby version (1.8.6), but there is an example of how to do this on this blog post. Hopefully, more (all) of these methods in the ruby core will accept blocks someday.

I began to write a method that would search my tree-like structure of hashes and arrays and I felt like it was a very ugly solution. I wanted something simple that would just take a line or two and get the job done. So I showed an example of the problem to my non-computer scientist wife and she said, “why don’t you just compare how the two lists look when printed out?” I was a little shocked that I hadn’t thought of this. It isn’t the most efficient solution and it relies on the #inspect method, which is probably pretty slow because it has to determine the structure of all the hashes in order to print them in a readable format. The old solution I was cooking up would also have to traverse the tree, similar to what I expected #inspect was doing. In addition to the #inspect call, there would be a string comparison for each element. I was willing to accept both of these for a short elegant solution to my problem.

The issue that came to my mind first was whether hashes get printed out the same way every time. In Ruby 1.9, it seems that Hashes are going to be stored in the same order that they were entered. In previous versions, I’m assuming that they are stored in the order of their generated hash. This could cause a problem if the data someone was entering could be in different orders. For my purposes though, the data would be entered in the same order if it was a duplicate.

So, after all of that, to uniqify an array containing a deeply nested structure, I present the following solution:

class Array
  def nested_uniq
    self.delete_if {|item| (self - [item]).any? {|other_item| item.inspect == other_item.inspect}}
  end
  def nested_uniq!
    self.replace self.nested_uniq
  end
end

I’ll walk through it because it may seem a bit hard to understand at first. I go through every element in the array with delete_if to determine if the element should be deleted from the array. Then, for all elements except the one in question, I check to see if #any? of them are equivalent to the element in question. This equivalence is determined by doing a string comparison of what the #inspect method returns for that element.

Here is an IRB test

>>  test = [{:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}, {:to => '1', :yeah => '8'}, {:hello => '5', :test => [{:hello => '10', :hell => '5'}, {:yeah => '2'}]}]
=> [{:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}, {:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]
>> test.delete_if {|item| (test - [item]).any? {|other_item| item.inspect == other_item.inspect}}
=> [{:to=>"1", :yeah=>"8"}, {:test=>[{:hello=>"10", :hell=>"5"}, {:yeah=>"2"}], :hello=>"5"}]

Hope this is helpful to someone! I am very interested to hear suggestions and comments on alternative ways to uniqify an array with elements that have a nested structure.

-Chase

alternate_data:
    adapter: mysql
    database: alternate_database
    username: user
    password: pass
    host: localhost

I was accessing it with something similar to the following:

ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations["#{state}_data"])
results = connection.execute description_params[:extraction_sql];
#This wasn't actually hard coded, but this is just an example
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations["development"])

The problems with this was that I had to repeat this ugly code everywhere, and if there was an exception thrown while accessing the alternate database it would not revert back to the original before exiting.  So the next request to the server thought the default database was the alternate database.  What I decided would be useful would be to have an “establish_connection” method that accepted a block and handle these issues.

I am going to write about this because I planned to before realizing that in Rails 2.2 the exact same issue is solved already with a much better implementation (of course).  You can find how to solve this problem very elegantly in Rails 2.2 at http://www.igvita.com/2008/10/27/scaling-activerecord-with-mysqlplus/.

My implementation is much simpler and I hope that anybody stuck with Rails 2.1 (like me for now) or any previous version can use it temporarily until they can upgrade in the future.

I just wanted a simple block establish_connection so I could write code like the following:

   establish_temporal_database_connection("#{state}_data") do |connection|
      results = connection.execute description_params[:extraction_sql];
    end

It would reset the old database connection if something went wrong or when the block was over.

I’m just going to list a simple module that does this, if anybody wants this in plugin form that actually overrides establish_connection then it isn’t that difficult to turn this into that:

module ConnectionWithBlock  
  def self.extended(object)
    class << object
        extend ClassMethods
    end
  end
  module ClassMethods
    def establish_temporal_database_connection(configuration)
       #save the old connection
       previous_connection = ActiveRecord::Base.connection
       #establish connection handling a possible exception and going back to original connection
       begin
              ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations[configuration.to_s])
              yield(ActiveRecord::Base.connection)
       rescue Exception => e
              raise e
       ensure
              ActiveRecord::Base.establish_connection(ConnectionSpecification.new(previous_connection.config, previous_connection.adapter_method))
       end   
     end
  end
end

Chase

I’ve been very happy with the results.  Very server intense requests are responding instantly.  This is all great, but caching is usually only appropriate for pages with content that is static for a period of time.  The biggest problem I was having is that there are very small pieces of the pages I wished to cache that are dynamic.  For instance, I want to have a bar at the top that tells the user who is logged in and provides a set of buttons to do things to their profile, etc.  An example of this is shown below:

Small Dynamic Block Showing User Profile Links

So what I needed was an easy way to specify that this small block would be dynamic while still allowing the rest of the view to take advantage of action caching.  I’m sure there is some plugin that does this and more.  If there is a very good plugin that does just this, I would love to know about it in the comments.  For now, though I am going to provide my simple solution to the problem.

My solution was to create a partial for the profile summary.  I already had the block as a partial to make my view code more DRY.  Instead of this:

<%=render :partial => 'profile_summary'%>

in my view, I replaced it with an AJAX call using Prototype (updated to use Richard Poirier’s suggestion in the comments):

    <div id="info_panel">
        <%= javascript_tag remote_function(:url => { :controller => 'profile', :action => 'summary' }) %>
    </div>

I then have a controller that can repond to this AJAX request for the profile.

def summary
  render :update do |page|
    page.replace_html 'info_panel', :partial => '/profile/summary'
  end
end

Now, every time the page loads, it will load the entire page using the cached version.  It will then make an additional POST request to get the dymanic content.  This would become more complicated if there were many dynamic sections that I needed to load.  I feel like if it was wrapped into a plugin that could automatically handle special dynamic fragments inside a block like this:

<%=dynamic_fragment do%>
   <%=render :partial => 'profile_summary'%>
<%end%>

it would make it very easy and intuitive to insert dynamic fragments into an otherwise static page.  I do not have the need to implement something like this yet, I am satisfied with my simple solution.  I would be very happy to find a solution where you can specify the dynamic sections of the view instead of only specifying the static portions.  Then the system would magically build the server-side AJAX responses without the user having to do anything.  This allows the use of action_caches which are supposed to be much faster than fragment caching.  While I’m super busy with my project, stopping to create plugins is not on the priority list, but by writing it all down here I might come back to the best ideas.

I hope anybody looking to insert a small section of dynamic code in an otherwise action cacheable page finds this information useful.

Chase

I’m sure an application focusing more on URLs is somebody’s 10% project somewhere sitting on the backburner until they have time to finish it.  It seems like almost anything you’d want to do has a corresponding web application.  When I come across a need that doesn’t I just feel like that void should soon be filled.  I’m going to outline the application that I believe would fill this void.  Maybe in future posts I’ll also walkthrough my attempts to do it as my personal .5% project.

Doesn’t this already exist?

When giving the quick summary of what I envisioned for the project the first question I get is, “Doesn’t Google Cache already do that?”  My initial reaction is that if it did, I would already be using it for that purpose.  After thinking about it, it doesn’t seem like that far of a stretch for Wayback Machine or Google Cache to archive URLs as well.  I feel like the real kicker is that in order to make it useful there would have to be extra processing and utilities for the URL archive.  The information is already there, so what needs to be done is create an interface for analyzing the URL a site has.  Before talking about the ideal solution, first let’s look at what is offered by Google Cache and Wayback Machine already to compare.

To get this sort of functionality out of Google Cache as far as I know is to use a “site:” search.  So, typing in the search, “site:arena.cse.sc.edu”, I get the following:

Result for site:arena.cse.sc.edu

I know that our research group’s website has lists of most of the pdfs that we have published as well a number of static pages that total more than the 17 that are listed in these results.  Of course, this site isn’t really optimized for search engines, so that might also not be helping.  Searching for just pdfs returns a measly 7 results.  Some of our papers are listed as links to other sites such as IEEE, which causes them to not get listed in these results.  Alternatively, if I could somehow easily browse all the URLs that exists for this domain, I might be able to see more information about what the website actually contains internally (more on this later).  Another experiment was to search for, “site:chase.ratchetsoftware.com”, and because this site is optimized for search engines, most of the URLs that exist were found pretty easily.  There is still no way to really tell what the site contains as an overview without going to the site or reading the provided description.  Put another way, it would be interesting to be able to see a something like a content tree that lets a viewer easily see what is contained in the site with a quick glance.  These are all things I believe can be solved and would be useful, but Google Cache and Wayback Machine are not the tools to do it.

What would an ideal solution look like?

I’ve given a few examples where Google Cache and Wayback Machine don’t offer what I am looking for.  I think to improve my argument it is best to talk about what the URL archive system would be in a perfect world.  It’s hard to see the difference in something that doesn’t yet exist.

I am going to list the features I have in mind and then explain any that aren’t obvious:

1. Provide a list of all URLs ever found for a given domain.
2. When a user views a list, it should check and display whether that URL is still valid.
3. Organize the list of URLs so that it is tractable by a normal user.  It should have collapsible SubURLs so that only top level URLs are in the master list.
4. Provide a filter so that only URLs matching the given filter are shown.
5. Provide the ability to click on a URL to make that the new root URL for our list.
6. A utility that can display a graphical tree view of the page and its links.  If the page links to an outside page, then show the link with the icon to that page if available.  This view would ideally provide enough information and “graphics” to give the user a good idea of what is available on the site.
7. If the site uses dynamic URLs, perhaps for a site written in rails or php, there should be some sort of analysis of the GET arguments from what the system can figure out.  This might be an area where users can submit additional information to improve the analysis of a web applications generated URL addresses.
8. One of the coolest ones.  A firefox plugin that automatically helps you when trying to figure out URLs based on what is stored in the URL Archive.  So if you’re trying to modify a URL slightly, you can get real-time help and see which URLs are still valid, etc.  Firefox is good right now with it’s new location bar that automatically searches your history and all that stuff.  It’d be even better if it could help you with long URLs that you haven’t typed in before!
9. Another plugin idea would be to provide screen readers easier access to a site’s content.  If all we had are the URLs and perhaps keywords for those URLs, a blind person might be able to more easily find content quicker.
10. One more plugin would be to have a sidebar while browsing sites, and as you browse you get an updated list of URLs that have existed at your current location on the site.  This would put a sidebar on your left with all the links you might see on that site and all the URLs that you might not see but existed in the past.  This would be useful for quickly navigating huge sites, especially ones that are moving their links around a lot.
11. The last idea I can think of right now is that the system could analyze how the URLs are changing and suggest to users URLs that may be used in the future or are currently used but not linked to based on the patterns in  the current URLs and their change history.  This might be useful if someone is trying to URL hack but doesn’t have any idea where to start.

I feel like number six is the one that needs further explanation in order to seem as cool as it does to me.  I believe there are already existing tools that will show you a tree view of a website.  There are definitely tools to show you this information for your own site, cause I’ve used them before.  So how does number six differ from these tools?  The answer is that it doesn’t differ dramatically, but hopefully it would offer more information because the purpose is slightly different.  Let’s give an example of a user that is looking for a site that offers ebooks in pdf form online somewhere.  The user might go to various sites by searching through Google and waste time at many sites that seem to offer ebooks but don’t have that large of a selection.  If the user could easily view a graph of each site before visiting, and turned on a filter for “.pdf”, he could hopefully see a tree that branches out into many different URLs leading to ebooks he’s interested in.  He should be able to quickly get an idea of the quantity, quality, and link validity of the site’s offerings.  Each node could be clicked on to make that node the new root node in the tree to perhaps get more specific information about a certain path.   There are many possibilities for graphically browsing a site’s URLs and URL relationships.  To me, this seems like a separate application entirely.  It might be that the site should be developed with a SOAP or REST interface and another application could be developed that provides this tree view of websites.

Potential problems and Questions

Not all good things come from having a URL Archive.  All the security concerns that come with Google Cache and Wayback Machine will also affect the URL Archive.  Some information that people wished to remove from the internet by unlinking it might be affected in a negative way by a system that indexes URLs forever.  Analyzng the GET parameters of a web application might be good for some users, but a malicious user could probably use this information to make the website work as it wasn’t intended.  Most of these problems come with developing this type of application.  There are tradeoffs and we just need to try to do our best to mitigate them while still providing a good service.

Another issue is the fact that I have no idea how to store such a large amount of information as this system would require.  I don’t even know where to begin with the indexing of URLs on the Internet.  It seems like it would take way too long with this little virtual machine on a server somewhere that I have.  I’d love to hear some comments about this.  How would one create an application like this that has to store such large amounts of information and process so much new data all the time?  Some resources relating to this would probably make for some great bathroom pulp .

Call for help

In my current situation I have way too much on my plate to try to develop an application for which there would be no immediate gain.  I need to work on my own business to try to get that going before I have to get a real job .  I also have an seemingly endless amount of Ph.D. work.  Despite this, I would more than willing to contribute to a project that was working on this solution.  I would even be happy to set up a Rails application on this server and with a Git repository if there were a couple developers interested in working on it.  All I’d need to distract me from my other work is a little motivation from other people’s interest, so please let me know in the comments if this is something you’d like to use or have needed in the past.

Thanks for reading,

- Chase Gray

Failed Attempts

I probably began hoping that I could one day work at Google at the very end of High School.  Back then I was focusing on Microsoft, but stories of the Google workplace soon overshadowed my dreams of working at Microsoft.  Looking back, I think each place has its pros and cons.  Without going to Microsoft for an interview, I don’t know if I can fairly compare the two.  I will say that I would enjoy making Google’s products more than Microsoft’s.  I feel like Google is making products that are pushing things forward, while Microsoft is stuck maintaining it’s current product line. Over time there began to be various videos and articles focusing on what it was like working at Google. I’ll include a couple here instead of wasting time talking about them.

CNN Covers Working at Google

www.youtube.com/watch?v=4YA9QWceEfE

Oprah Talks about Working at Google

www.youtube.com/watch?v=vd6BPhJjYL4

It seems pretty hard to resist the ideal working environment for a young computer scientist.  I believe I applied to only Microsoft while I was still in high school because at the time it had the most popular(infamous) internship program.  Although it wouldn’t be long before Google interview stories began to become synonymous with Microsoft’s.  Of course looking back there was absolutely no reason for Microsoft to decide to interview me besides a very excited sounding cover letter.  I didn’t have stellar grades, I didn’t have any completed projects to show off, and I didn’t have any recommendations from anybody that had worked at Microsoft.  So of course I got a generic rejection letter from Microsoft a few months later.

Throughout college I could tell pretty much the exact same story except that my resume was growing every so slightly and I believe I quit applying to Microsoft after my sophomore year of college or around that time.  I focused on the Google internship program because I felt that was where the exciting things were happening.  At first, if I remember correctly, the internship at Google was available to all college students.  At one points though, probably around my sophomore year again, I noticed that they started requiring you to be a Master’s or Ph.D. candidate.   Regardless, I still applied each Summer just in case I could get lucky.

Landing the Interview

As you can see in my introduction post, I started doing research with my academic adviser, Srihari Nelakuditi, during the Summer of my sophomore year as well.  This was when things seemed to be coming together and I was getting a little recognition for all the reading and programming I had been doing up until this point.  I was a step or two above the average peer in my classes, and I’m glad that Srihari could see some sort of potential.  I continued to apply every Summer for an internship at Google and also at Fog Creek Software.  I had hopes that with my growing resume and research experience I might slip into an interview at one of these internships.  If I did not get the internships I was funded by my adviser to continue doing research in the ARENA lab over the Summer.  I continued at the University of South Carolina into my Graduate studies and got to the point where I was about to complete my Master’s thesis.  During my second to last semester as a Master’s student I got a lucky break.

Apparently my adviser had a few hookups at some of the companies I had been applying to over the years and was waiting until I was ready to refer me to them.  So, he told me one day that he gave me a reference to Google and I would be hearing from them soon.  It seemed kind of surprising that it should be so easy, but I’ve always read that these good jobs were about connections.  Sure enough…a few weeks later I got an email from a woman at Google recruiting.  This was the same location that had sent me all those generated rejection letters in the past.  I had a pretty similar resume this time, my experience only consisted of six extra months since the previous rejection, only this time I had someone else say, “Hey, hire this guy”.  It makes me wonder if I could have had only the reference and still be contacted by the recruiting department.

So, the first interesting thing someone might wonder who’s been attempting to get a job at a place like this is what do they say in an non-rejection email.  Well, here is the first email I got from the recruitment woman.

We continued talking about what kind of position I was looking for through a few email exchanges.  They were very nice and she even asked me about things in my extra-curriculum activities that were not related to Google, it seemed we had some similar interests.  For various reasons, I decided I didn’t want to go to California and instead let her know that I was interested in a Software Engineer full-time position at the Boston office.   I sent her my two page curriculum vitae and my current unofficial transcript and I was moved on to the next step.

So what came next was an email from a recruiter from the Google -- Boston office.  He immediately asked for a good time that he could call me and talk about various opportunities at his office.   Somehow I missed his first call because I believe he called my old phone for some reason.  The only explanation I had was that they had me in their database from previous resumes, and these old resumes had my old phone number on them.  So, all those old resumes had been getting into their system after all . This happened on more than one occasion and still confuses me why they kept using the old number.  He was a cheerful sounding guy who seemed to make every effort to put me at ease during the whole process.   I was worried at first this might involve some technical questions but I found out that it was just to talk to me and make some plans for the interview process.

The first step of most Google interviews is the infamous phone interview.  In this process you are supposedly asked questions by one or more engineers over the phone and you are supposed to talk your way through the problem.  There are a lot of examples of these online, so that’s where I’d recommend looking for them.  They are usually questions that aren’t exactly computer science, or slightly related, but lend themselves to solutions that could be spoken out over the phone.  The reason I don’t have any personal examples is because I got pretty lucky while talking to this guy on the phone.  It just so happened that my girlfriend and her parents were going to be going to Boston about a month later.  He mentioned something about a phone interview sometime later and I said that I would probably already be visiting Boston around that time.  I waited and hoped he would make the same conclusion that I had and he did.  He asked me if I would prefer to just come on in to the office when I was visiting then.  I was ecstatic , both because I didn’t have to take the phone interview and because I was going to Google’s office in Boston in less than a month!  I don’t know if I could recommend the same process to anyone else, but it worked for me.  I was going to have to pay for my hotel because I was arriving one day earlier than my girlfriend but it was OK with me.  They said I should come in at about 10:00AM and my interviews would last until about 2:15PM including lunch.  I asked about what to wear and I was told it’s pretty casual in the office so I should probably wear something semi-formal like a collared shirt and slacks.  Now all I had to do until the big day was to try and prepare for whatever questions might be asked of me during my technical interview.

Preparation

Like most people preparing for a technical interview, I didn’t have any idea where to begin.  It seemed like they could ask me anything.  Looking up examples of past questions from Microsoft and Google only made me more worried.  The questions I found were mostly reasonable, but some were just so weird and I had no idea how to solve them.  Regardless, one of the best ways to prepare was to study these example questions.  Not so much to memorize the answer, but to understand the thought process behind the answer.  Most of the questions people seem to face have similar fundamental problems.  The biggest challenge is to take a problem you are presented and trying to put it into terms similar to a problem you already know how to solve.  Many of the traditional problems could be found in most algorithms books.  It seems like algorithms pretty much dominate the questions, but all other areas of computer science are fair game.

I’ll try to list some of the resources I found.  I didn’t exaustively try to understand every problem, but I spent a while going over things that I didn’t understand and refreshing mysefl with fuzzy areas.  Again, the idea was to try and be capable of coming up with my own solution based on the knowledge I had, not to memorize answers.  Aside from studying algorithms and previous questions, my adviser also gave me a book with quick hacking tricks used in programming.  These ranged from how count the number of ones in a binary number quickly to various memory hacks.  I’ve seen that people were asked questions relating to these dirty hacks but they are mostly there to increase your bag of tricks.

• Ace The Interview
• Some good general tips for a technical interview from a Microsoft interviewer
• Huge list of compiled questions from interviews.
• http://www.allinterview.com/

How to Hack the Technical Interview

www.youtube.com/watch?v=4KyCPiJPx48

After reading algorithms and technical interview questions for a few weeks during my free time.  I was finally getting prepared to hop on the plane to Boston for my interview.  I was very excited to get to see Boston for the first time.  It was also going to be when we told my girlfriend’s parents about our engagement, so it was going to go well even if my interview went bad.

The Boston Office

Arriving in Boston I got into my hotel near the airport and planned to stay in the entire night reviewing some basic algorithms.  I started to make sure I had everything to wear for the next day.  A nice collared shirt and black pants.  I realized at this point that I had forgotten to bring a belt, and the pants didn’t fit perfectly around my waist so it was a required piece.  I started to get pretty worried and was looking for a quick solution.  I decided to look online for the closest store.  I was near the airport so there was nothing at all.  I just got out of the hotel and walked on foot through East Boston for about 30-45 minutes until I came across an area that had some sort of stores in it.  I had very little hope though that I would find a belt for my outfit in what looked to be a mexican area of East Boston.   To my surprise I found a small mexican shop that sold clothes.  When looking through their limited supply of belts, I found that they all looked like this:

Tacky Belt

This would obviously not do for my interview!  They did have one generic brown belt. I asked for it without trying it on at all and left.  I went to some pasta place for dinner and brought it back to the hotel.  When trying the belt on later that night I realized it was GIGANTIC!  I had to figure out a way to cut a hole in it and figure out a way to make it stay on and hold my pants up throughout the all day interview.  This was the biggest issue I faced during my entire time at Boston, so I think that makes my trip a relative success.

I woke up the next morning pretty early so I had time to figure out the train system and where I had to go.  I needed to get from East Boston over to Cambridge.  Obviously this wasn’t too much of an issue and pretty soon I was right next to the MIT campus looking for which building Google was located in.  The lady had given me the address, “One Broadway, Cambridge, MA”.  Apparently it was this big building here:

Google Boston office from across the street

Walking up closer you can see that it is just a typical office building.  There was nothing flashy or fancy about the outside, but Google was located on the 7th floor (and I would learn also another floor as well).  I expected at least the 7th floor to have the “Google look”.

Google Boston office front doors. Then 7th floor.

For those of you who are truly interested, here is an embedded map of the location:

View Larger Map

I then used my extra time to sit at a nearby Cosi restaurant and get some breakfast and relax.  I was going to be in interviews for the next 4 or 5 hours.  So I finally went inside the building and up to the 7th floor to see what awaited me.

The Interview (and Lunch)

I walked off of the elevator to a big happy Google logo in the main hallway.  Then there were two glass doors that needed a card to unlock them.  Luckily someone was walking out and I walked in.  There was a lady at a desk to greet visitors and she had me sign in on a computer with a little application they had.  I had read somewhere that the Mountain View office has a big screen with real-time searches being displayed.  I had also read someone’s blog that the Boston office did not have one of these screens.  I was pleasantly surprised to find that there it was in all its glory to entertain me while I sat on one of the fancy Google colored chairs amidst the various toys and Google exercise balls on the floor.  As I looked around there were a couple offices (one with guitar hero) and a surprising amount of women.  As I would later learn, this was primarily the sales floor, which explains the large number of women.  Finally, my recruiter came out and greeted me and took me up to a higher floor where they had more of their engineers hiding.

The upper floor was still very nice and people were not too cramped at their desks.  There were not an cubicles, mostly open space.  Everyone had two monitors and either a PC or a Mac.  Some people had portable video conferencing devices so they could work with people from other offices with constant communication.  Everyone I ran into seemed very nice.  It looked like the upper echelon of computer scientists.  That’s a weird way to put it, but most people reading this probably know what the lower portion of computer scientists consist of.  I’m talking social skills of course, not technical.  I was told that this office was becoming too small and they would soon be moving to a larger building.  Apparently, they would soon be allowed to bring dogs to work on one of the floors of the new building!  I always loved the thought of bringing my dogs to work with me instead of leaving them at home to be bored all day. After the quick walk through the upstairs office I was brought into this little room with a desk, a chair, and a whiteboard.

The interview began with a conversation between me and my recruiter who I had talked with on the phone one month prior.  We talked about what it was like to work at Google -- Boston.  He wanted to answer any sort of question I might have.  So I asked about all the various things their office did, the normal workday, the time commitment, etc.  Overall I got the feeling that the Boston office was more consistent in their eight hour workdays because the average age of employees was higher and they had families at home, etc.  I was very worried about all the articles I read about online about Google employees working to death without much compensation.  He then told me how the interview process was going to proceed.  I was going to be interviewed by four different engineers for 30 minutes each.  In the middle I was going to have lunch with another engineer.  With that he said his goodbyes and good lucks and introduced me to my first interviewer.

In order to avoid any problems, I’m only going to generically talk about the problems any of the engineers gave me.

The first engineer was a small asian guy who seemed very friendly.  He sat down across from me and asked a few questions about my research experience and tried to try to get a handle on some of my previous projects.  He then asked if I had any questions about Google for him and we moved on to the technical questions.  He had two questions for me.  Given an array of stock prices, what would be the best buy and sell point to make the most money.  I had read about problems similar to this while studying.  I had to modify it a bit in order to get the solution.  The similar problem I had studied was finding the largest sum of an array of positive and negative numbers.  For the stock question, I had to change each number to instead be the gain or loss for that day.  Then, the problem reduced to the largest sum of contiguous elements or Kadane’s algorithm.  The second question was given a set of N terms, you have N linked lists each containing an IDs of a webpages that match the term the list corresponds to.  The important thing was that each list was in sorted order. I was to come up with a method to AND multiple terms together and retrieve the matching pages.  This was not similar to a problem I had studied and I had to come up with a couple ideas before settling on one.  The fact that they are in sorted order allows you to quickly determine if you need to continue a search or not in the second list.  The biggest challenge was when the interviewer then asked me to implement it.  I stumbled a bit with some exceptional cases and the fact that the board was so tiny.  We ran out of time before it was perfect, but I felt that it was sufficient.

The second engineer came in immediately after the first.  He was a larger bearded American guy. He also seemed nice, but much more focused and serious.  We started with the discussion about my experience just as the last engineer did, but we talked in depth about one of my networking research projects with sensor networks.  I was very surprised with how well the engineers seemed to have read my resume, they were very prepared and I appreciated that.  We then moved on to the questions.  The first one was tricky at first but I was able to get it without too many hints from the interviewer.  He was mostly giving positive sounds when I talked about something that was close to the right solution. The question was given a random number generator that generates the numbers between 1 and 7, how do you create a random number generator that generates 1 through 10 uniformly?  This was similar to questions I had read online about biased coin flipping.  You basically have to generate twice and take all the combinations that can occur, ie (1, 1), (1, 2), of which there should be 7*7=49.  Divide these sets evenly into 10 sets.  Then if you generate a combination that falls within one of these 10 sets, you have a number uniformly between 1 and 10.  There will be some sets that are left over, and if you get one of these you must regenerate your numbers.  He extended the question by asking what is the probability that you will have to regenerate your random numbers.  Then he further extended the question to ask if you could reuse the previous rolls so you only had to regenerate one random number.  So I’ll leave this one to you.  I then had to determine the probability of not getting a uniform number between 1 and 10 after 7 attempts or something like that.  His second question was much more discussion oriented.  He asked me to talk about how networking would be implemented in a simple board game.  He wanted to know how the data was sent over the network.  We focused on connect four because I had experience programming this game before.  We went into all the details of what each player must know about their opponents in order for them to play the game.  We then talked about what a player would need in order to be an observer, or to join a game in the middle, etc.  There wasn’t any tricks to this question, we just talked about game networking basically until our time ran out.

Finally, I was out of the first two interviews and they seemed to have gone relatively well.  I was now introduced to my third engineer and he was supposed to have lunch with me.  There are a lot of stories about the food services at Google.  I wasn’t sure what the setup would be in the Boston office, but I was hoping it was still free.  Sure enough, free lunches are the norm even at a smaller office (30 engineers) like this one.  It wasn’t one quite as big a scale as the Mountain View campus is, but it was a buffet of above average food brought down from a catering service on an upper floor of the same building.  I don’t remember what I had, but it was pretty healthy and tasty.  I was very excited about the giant glass doored fridge they had filled with all kinds of free sodas and water bottles.  I had a pretty tasty sparkling lemon drink that was covered with a foil top that you had to remove before opening it.  I don’t remember the name but I wish I did so I could get some.  The guy said they replaced the Sprite in the office with the lemon drink.  During lunch we talked a lot about the life at Google.  The guy had a lot to say about the people you get to work with, the types of projects and languages used, and his own experiences during the interview.  There were a couple other Googlers that sat down with us and joined in the discussion about life at Google.  Everyone seemed pretty happy while they enjoyed their free lunch (and free cake).  There were a bunch of board games in this room that looked like they were played when people were not eating.  It was a pretty small room for a cafeteria, like a large conference room that they placed lunch tables in.  I was pretty surprised to hear that you could bring friends and family to the lunch with you for free as long as you let them know they were coming ahead of time.  Another interesting thing I learned was that most of the engineers at this office were from MIT.  So much so that they all had stories of when they were in college together and partied together.  I had read previously that Google had a preference for Ivy league schools, and this kind of worried me about my own educational background.  After our talk we walked around the office for a more formal tour.  I saw most of the office the first go around because it was pretty small, but they said they would be moving pretty soon to a bigger location.  We ended back at the small office where I would continue the last two technical interviews.

I don’t know how to describe the next engineer except as spunky.  He was very happy and excited to be working there.  He explained to me that every Google employee spends about three months or so at the main campus in Mountain View, CA for training.  They said it’s a pretty crazy experience unlike anything else.  So we started back with the standard talk about my experience and moved into the technical questions.   I’m sorry but I can only remember one of the questions posed by this interviewer.  He asked me to describe all the details involved in implementing something similar to the wireless access points at the airports.  When you first log on it redirects your browser to a page that forces you to either pay or sign in.  We talked about various security concerns and problems with the current approach.  He then asked me a question I did on the white board but can’t remember it for the life of me.  It might have been the stock quote question that I thought was in the first interview.  It really doesn’t matter though as I think all the questions I was asked are in this post somewhere.

The last interviewer was another large bearded American guy.  We again talked about my experience and any questions I might have about Google.  The questions posed by this engineer were the ones that stumped me the most.  One was very simple but I was too quick to come up with a solution and it had some flaws.  The question was given a binary tree, determine if it is a binary search tree.  This is a typical problem and I should have had an immediate answer.  My answer did not work on a special case and he didn’t tell me this before asking me to implement it.  I then ran into trouble while implementing it and we talked about why and figured out what my problem was.  This interviewer didn’t seem near as excited or happy to be interviewing me as the previous engineers.  The second question he asked was an event scheduling problem.  Basically you were given events with a start time and an end time.  You were then to determine if there were any conflicts in any of the events.  He would then extend the question to find a O(n) solution.  I don’t think I came up with a solution to his satisfaction during the interview.  My solution involved a hashing technique so we could determine collisions between events if the same hash was found.  I’m sure there is a much better solution, but this was at the end of the day and the guy was not being very helpful with hints.  I  felt the worst about this interview and so I left on a bad note.

I was then taken downstairs by another engineer and we made sure everything was in order and he said goodbye.  I felt kind of rushed out the door without any kind of thank you for coming or anything like that.  I felt good about most of the interview, but the last interview would bother me for the next few weeks.

Aftermath

So I was finally back home and couldn’t stop worrying about the results of the interview.  I waited a few weeks but didn’t hear anything back.  I had sent a thank you email a couple days after the email but I didn’t get any reply.  After about three weeks my secondary phone rang (again I don’t know why they have this number).  It the recruiter from the Boston office and he was letting me know that I didn’t get the job.  He said my background was not matching what they needed at that time.  I thought this was a very odd way of putting it.  This made me think that the rumors about them preferring certain colleges were at least partially true.  I’ll never know the real reasons why, but I did feel that I did pretty well during most of my interview.

Since then, I’ve come to terms that I might not be a good fit for a company like Google.  I feel like I should be putting my talents to actually create something useful and start my own company.  I know it’s a common dream, but I feel like I have a few good ideas and I’m working to make them a reality.  So, I haven’t applied to any new companies since then because I’ve been doing Ph.D. work and working on my own company Ratchet Software.

I hope that this story about my experience interviewing at Google will be of some sort of help to someone.  I know when I was interviewing I was looking for every resource I could find on the topic.  I had only found one blog post that mentioned anything about the Boston office.  I know there are a ton of people trying to get their foot in the door at Google.  I just hope that my story helps them do that.  Good luck to all the future Googlers out there.  I own some Google stock and would love to see some of you young motivated engineers come up with the next Gmail. 

- Chase