I agree that data should be broadly accessible for analysis, review, or value-added applications–especially publicly funded information. However, I don’t agree that your proposal is the best approach.

There are three main issues:

Provenance–the data comes from a reliable source that people accept as being an authority.

Reliability–the data can be reliable and reproducible, and therefore conclusions based on information derived from the data can be accurately reproduced.

Historical Volatility–once the data is published, the need to update previously stated information is well understood. The NYT best-seller list should have little-to-no volatility, but a financial report database might require frequent updates as company restate earnings based on new accounting principles, laws, or audits.

Data that must have provenance or that is historically volatile, but requires reliability are good candidates for API-level access, rather than downloaded databases. A downloaded SQLite database can be modified and the data and redistributed. Depending on how it is done, it can create problems with provenance, reliability, and can suffer from volatility issues that may not be taken into account. Furthermore, this can happen both maliciously, or accidentally, but in either case it creates a number of problems for the data publisher, and downstream consumers.

It’s unlikely that anyone who has a vested interest in maintaining reliability and provenance of their data is going to allow it to be downloaded in this manner.

APIs provide a way for publishers to ensure accuracy and ownership of data, but still allow people to create derivative content. If you need more access to the data than the API access agreements offer you, you are more likely to make headway with the data providers by asking them to increase the rate-limit for the service, instead of asking them to relinquish all control of the data.

But, it’s also important to understand the data that you are requesting and the way in which you will use it, so that you determine what the appropriate rate-limit should be. Let’s look at the statements that you present as an example:

* NYT Bestsellers data is fully available
* NYT Bestsellers data changes infrequently (~4/per week/list)

For someone using the data in an online application, or even on research effort, querying March 2000, Hardcover Fiction is going to get the same data every query. Given that, you should be caching that query, rather than passing it to the NYT API. Given that the data changes infrequently, once you have historical data backfilled, querying the data more frequently than it updates and caching the result should lead to accurate data.

Given that, the NYT API query limit is at least 1000x times more than anyone would need on an ongoing basis. I’m sure that the limit was put into place to avoid their API services from being beaten to death by poorly architected applications.

Even in the case of research, a two day wait to get access to the data isn’t really that onerous of a requirement, assuming there was no other way to get that data.

In the case of government data, they are concerned about data reliability, and possibly volatility: that is they want the ability to provide accurate, reliable data that everyone can access and get the same results while incorporating possible updates, if needed. APIs do a better job of ensuring that than distributed data sources.

For public information that is published and static, a downloadable format is great, and many government agencies provide access to data in downloadable format in machine processable formats such as MS Excel, CSV, XML, or text. Others provide data that is intentionally scrape-able from their site.

Of course, all of this could be resolved by ensuring that data that is consumed is appropriately attributed, that modifications to data are properly described, and that methodologies are both documented and technically sound. However, there are people who improperly interpret data (intentionally or accidentally, in good faith and bad), and there are non-disclosed agendas, and people use data to covertly influence others.

So, until Utopia arrives we have to address the sometimes conflicting needs and available resources of publishers and consumers.

For our Open Policitics API “Deutschland API” (http://deutschland-api.de) we’re not limiting requests and provide all kinds of Query Languages (for ex. YQL).

Publishing the full data set as well is a good suggestion though!

At Sunlight Labs (we advocate for Gov open data), the concept of we’re advocating for bulk access to data. Generally, what we think government ought to be doing is the following– for just about every data-driven website it creates:

1. Provide bulk access to the data that powers the site in some kind of machine readable format. CSV, SQLite, JSON, XML– just use something that makes some kind of sense. Not a .tiff file or a .pdf (unless it has the source documents appended)

2. If the datasets are significantly large– it’s cool to provide a functional, registration-less API on top of that dataset, but once you go into registration-land and I have to give you my email address to get a key to that dataset– well, no. I already paid for the data. It’s public domain. Make it free with no strings attached.

3. Make your website powered by the same bulk data or API you’re providing. Don’t provide bulk data, then build your own website with some better, cleaner copy of that data. That’ll make it so you take care of the bulk data like you take care of the graphics on your website.

It’s remarkable how many gov agencies want to jump into API land without providing bulk access *first*, but generally I don’t think it’s because of “information control” issues, rather, it’s because API is a three letter acronym– and the government thinks three letter acronyms are really sexy.