Comments on: We have an API http://nat.org/blog/2010/03/we-have-an-api/ Mon, 07 May 2012 20:05:57 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Nat Friedman http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6534 Nat Friedman Fri, 02 Apr 2010 23:04:56 +0000 http://nat.org/blog/?p=1565#comment-6534 Thanks for responding again. Time/resources - that's what I figured. I felt like your web signup form was a good thing because it did ask for a fair amount of info about the app and why I am using your API. So I would expect that you have decent info there. Thanks for responding again. Time/resources – that’s what I figured.

I felt like your web signup form was a good thing because it did ask for a fair amount of info about the app and why I am using your API. So I would expect that you have decent info there.

]]> By: Derek Gottfrid http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6532 Derek Gottfrid Fri, 02 Apr 2010 12:51:28 +0000 http://nat.org/blog/?p=1565#comment-6532 It is a two fold problem one is simply finding the time. It is isn't much work - and I do love the idea of a sqllite db (we have used it here internally for a very very long time) but setting aside the time amidst all the other things we are trying to do. One of the things we want to do and haven't entirely solved is tracking where our data/apis are being used. It is important for the business to know where the data is flowing so we can identify new opportunities and be an actually useful partner w/ developers. It is a two fold problem one is simply finding the time. It is isn’t much work – and I do love the idea of a sqllite db (we have used it here internally for a very very long time) but setting aside the time amidst all the other things we are trying to do.

One of the things we want to do and haven’t entirely solved is tracking where our data/apis are being used. It is important for the business to know where the data is flowing so we can identify new opportunities and be an actually useful partner w/ developers.

]]> By: mattmole http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6530 mattmole Thu, 01 Apr 2010 10:40:51 +0000 http://nat.org/blog/?p=1565#comment-6530 Why not provide the API in couchDB format. That way the built in replication feature could be used to pull down new data at specified times. This could also count as the full allocation of requests for a day Why not provide the API in couchDB format. That way the built in replication feature could be used to pull down new data at specified times. This could also count as the full allocation of requests for a day

]]> By: Niel http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6529 Niel Wed, 31 Mar 2010 14:58:23 +0000 http://nat.org/blog/?p=1565#comment-6529 Just in the last couple of days I've been researching some claims by our local school board member that 70% of the voters in our county are 62 or older. Trying to find the data has been an interesting experiment in open data. It turns out that the elections commission only offers the data in a PDF of a printed report, and that comes from the state anyway. I would much prefer to have been able to download raw data, I don't want an API at all. Oh, and it turns out that the number is 17%, not 70%. So now I'm not sure if my source misheard or if the board member misspoke, but either way it's good to be able to dig into the raw data myself. Just in the last couple of days I’ve been researching some claims by our local school board member that 70% of the voters in our county are 62 or older. Trying to find the data has been an interesting experiment in open data. It turns out that the elections commission only offers the data in a PDF of a printed report, and that comes from the state anyway. I would much prefer to have been able to download raw data, I don’t want an API at all.

Oh, and it turns out that the number is 17%, not 70%. So now I’m not sure if my source misheard or if the board member misspoke, but either way it’s good to be able to dig into the raw data myself.

]]> By: Nat Torkington http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6528 Nat Torkington Wed, 31 Mar 2010 02:37:56 +0000 http://nat.org/blog/?p=1565#comment-6528 @Clay if "API" is sexy and "bulk data download" is not, perhaps we need to talk about offering "BD2 Compatibility". (half-serious) @Clay if “API” is sexy and “bulk data download” is not, perhaps we need to talk about offering “BD2 Compatibility”. (half-serious)

]]> By: Nat Friedman http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6527 Nat Friedman Wed, 31 Mar 2010 02:32:16 +0000 http://nat.org/blog/?p=1565#comment-6527 > It’s remarkable how many gov agencies want to jump into API land without providing bulk access *first*, but generally I don’t think it’s because of “information control” issues, rather, it’s because API is a three letter acronym– and the government thinks three letter acronyms are really sexy. That's what I figured too. But it's so much HARDER in many cases to provide an API than to just put up a database dump. So it's funny that they get it backwards. (Thanks for commenting, I'm a big fan of Sunlight Labs!) > It’s remarkable how many gov agencies want to jump into API land without providing bulk access *first*, but generally I don’t think it’s because of “information control” issues, rather, it’s because API is a three letter acronym– and the government thinks three letter acronyms are really sexy.

That’s what I figured too. But it’s so much HARDER in many cases to provide an API than to just put up a database dump. So it’s funny that they get it backwards.

(Thanks for commenting, I’m a big fan of Sunlight Labs!)

]]> By: Nat Friedman http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6526 Nat Friedman Wed, 31 Mar 2010 02:27:32 +0000 http://nat.org/blog/?p=1565#comment-6526 Thanks for writing Derek! For the record, I am very impressed and happy that NYTimes provides this data in a computer-readable form at all. If you do add a downloadable list in the future, that would be great - and I'd be curious if you do have any business reasons not to do that, or if it's just a matter of time and priorities. Thanks for writing Derek! For the record, I am very impressed and happy that NYTimes provides this data in a computer-readable form at all.

If you do add a downloadable list in the future, that would be great – and I’d be curious if you do have any business reasons not to do that, or if it’s just a matter of time and priorities.

]]> By: Nat Friedman http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6525 Nat Friedman Wed, 31 Mar 2010 02:25:05 +0000 http://nat.org/blog/?p=1565#comment-6525 Well put Francis. I'm not against API, I just want an additional entry point that gives me a dump of all the data. Well put Francis. I’m not against API, I just want an additional entry point that gives me a dump of all the data.

]]> By: Michael http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6524 Michael Wed, 31 Mar 2010 00:48:48 +0000 http://nat.org/blog/?p=1565#comment-6524 Hard to say that they are or not - they definitely make money on the list itself. I think rate limits are fine for most people. If you need to have it raised just send an email. Hard to say that they are or not – they definitely make money on the list itself. I think rate limits are fine for most people. If you need to have it raised just send an email.

]]> By: Oliver Charles http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6523 Oliver Charles Tue, 30 Mar 2010 21:27:58 +0000 http://nat.org/blog/?p=1565#comment-6523 Amen! At MusicBrainz, we offer a public database dump that is regenerated weekly I think. It does require some PostgreSQL specific extensions, but it's the best we can do. We also offer a live replication stream (paid for) with hourly replication packets. And then on top of that we also offer a web service with a 1 query per second limit. They all get heavily used, and we certainly wouldn't ever think about removing one of the options now. Amen! At MusicBrainz, we offer a public database dump that is regenerated weekly I think. It does require some PostgreSQL specific extensions, but it’s the best we can do. We also offer a live replication stream (paid for) with hourly replication packets. And then on top of that we also offer a web service with a 1 query per second limit. They all get heavily used, and we certainly wouldn’t ever think about removing one of the options now.

]]> By: Clay Johnson http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6522 Clay Johnson Tue, 30 Mar 2010 20:51:07 +0000 http://nat.org/blog/?p=1565#comment-6522 At Sunlight Labs (we advocate for Gov open data), the concept of we're advocating for bulk access to data. Generally, what we think government ought to be doing is the following-- for just about every data-driven website it creates: 1. Provide bulk access to the data that powers the site in some kind of machine readable format. CSV, SQLite, JSON, XML-- just use something that makes some kind of sense. Not a .tiff file or a .pdf (unless it has the source documents appended) 2. If the datasets are significantly large-- it's cool to provide a functional, registration-less API on top of that dataset, but once you go into registration-land and I have to give you my email address to get a key to that dataset-- well, no. I already paid for the data. It's public domain. Make it free with no strings attached. 3. Make your website powered by the same bulk data or API you're providing. Don't provide bulk data, then build your own website with some better, cleaner copy of that data. That'll make it so you take care of the bulk data like you take care of the graphics on your website. It's remarkable how many gov agencies want to jump into API land without providing bulk access *first*, but generally I don't think it's because of "information control" issues, rather, it's because API is a three letter acronym-- and the government thinks three letter acronyms are really sexy. At Sunlight Labs (we advocate for Gov open data), the concept of we’re advocating for bulk access to data. Generally, what we think government ought to be doing is the following– for just about every data-driven website it creates:

1. Provide bulk access to the data that powers the site in some kind of machine readable format. CSV, SQLite, JSON, XML– just use something that makes some kind of sense. Not a .tiff file or a .pdf (unless it has the source documents appended)

2. If the datasets are significantly large– it’s cool to provide a functional, registration-less API on top of that dataset, but once you go into registration-land and I have to give you my email address to get a key to that dataset– well, no. I already paid for the data. It’s public domain. Make it free with no strings attached.

3. Make your website powered by the same bulk data or API you’re providing. Don’t provide bulk data, then build your own website with some better, cleaner copy of that data. That’ll make it so you take care of the bulk data like you take care of the graphics on your website.

It’s remarkable how many gov agencies want to jump into API land without providing bulk access *first*, but generally I don’t think it’s because of “information control” issues, rather, it’s because API is a three letter acronym– and the government thinks three letter acronyms are really sexy.

]]> By: Chris Anderson http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6519 Chris Anderson Tue, 30 Mar 2010 16:53:49 +0000 http://nat.org/blog/?p=1565#comment-6519 You might be interested in looking at CouchDB's replication support. It was designed with exactly this use case in mind. You might be interested in looking at CouchDB’s replication support. It was designed with exactly this use case in mind.

]]> By: Nader http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6518 Nader Tue, 30 Mar 2010 15:04:51 +0000 http://nat.org/blog/?p=1565#comment-6518 For our Open Policitics API "Deutschland API" (http://deutschland-api.de) we're not limiting requests and provide all kinds of Query Languages (for ex. YQL). Publishing the full data set as well is a good suggestion though! For our Open Policitics API “Deutschland API” (http://deutschland-api.de) we’re not limiting requests and provide all kinds of Query Languages (for ex. YQL).

Publishing the full data set as well is a good suggestion though!

]]> By: Derek Gottfrid http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6517 Derek Gottfrid Tue, 30 Mar 2010 14:56:24 +0000 http://nat.org/blog/?p=1565#comment-6517 Great stuff here. I like Nat's point and the discussion that follows. Ultimately, we want to provide data in a variety of different means. It is still early on for nytimes and apis/data - stay tuned and keep the thoughtful discussion going because we are listening. Great stuff here. I like Nat’s point and the discussion that follows. Ultimately, we want to provide data in a variety of different means. It is still early on for nytimes and apis/data – stay tuned and keep the thoughtful discussion going because we are listening.

]]> By: Jay Woods http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6516 Jay Woods Tue, 30 Mar 2010 14:46:35 +0000 http://nat.org/blog/?p=1565#comment-6516 To me this looks like the reason that RSS came into existance. It is not necessary that the copy of the database needs to be available from the original database. One person can take it upon themselves to have a sync'd copy and makes available a downloadable copy when called for and updates as requested. We seldom ask why a person would subscribe to an RSS feed rather than an email list nor is it fruitful here to ask why someone would want a sync'd copy of the database. They just do. Overall it is easier to handle the results of an RSS feed than the multiple formats of the received emails. Likewise, it is easier for the recipient to squirrel away the results of a sync'd copy than the multiple formats of the received queries against a lot of different databases. To me this looks like the reason that RSS came into existance. It is not necessary that the copy of the database needs to be available from the original database. One person can take it upon themselves to have a sync’d copy and makes available a downloadable copy when called for and updates as requested.

We seldom ask why a person would subscribe to an RSS feed rather than an email list nor is it fruitful here to ask why someone would want a sync’d copy of the database. They just do. Overall it is easier to handle the results of an RSS feed than the multiple formats of the received emails. Likewise, it is easier for the recipient to squirrel away the results of a sync’d copy than the multiple formats of the received queries against a lot of different databases.

]]> By: Pete Warden http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6515 Pete Warden Tue, 30 Mar 2010 14:43:04 +0000 http://nat.org/blog/?p=1565#comment-6515 I'm on a crusade to create analyzable data sets too, but to a lot of companies it's quite a threatening act. You can see the result of my planned release of crawled public profile data from Facebook here: http://petewarden.typepad.com/searchbrowser/2010/03/facebook-data-destruction.html You should also check out Infochimps, they're trying to build a business around distributing these sort of data sets: http://infochimps.com/ I’m on a crusade to create analyzable data sets too, but to a lot of companies it’s quite a threatening act. You can see the result of my planned release of crawled public profile data from Facebook here:
http://petewarden.typepad.com/searchbrowser/2010/03/facebook-data-destruction.html

You should also check out Infochimps, they’re trying to build a business around distributing these sort of data sets:
http://infochimps.com/

]]> By: Mike http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6514 Mike Tue, 30 Mar 2010 14:10:02 +0000 http://nat.org/blog/?p=1565#comment-6514 You could have just used Tor to make your connections originate from a large number of different IPs - http://www.torproject.org/ You could have just used Tor to make your connections originate from a large number of different IPs – http://www.torproject.org/

]]> By: Hans Granqvist http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6513 Hans Granqvist Tue, 30 Mar 2010 13:57:51 +0000 http://nat.org/blog/?p=1565#comment-6513 Good APIs let you download the main bulk and let you do the deltas. It makes sense for all. Here's how we do it for Netflix: http://developer.netflix.com/docs/REST_API_Reference#0_42335 Good APIs let you download the main bulk and let you do the deltas. It makes sense for all. Here’s how we do it for Netflix: http://developer.netflix.com/docs/REST_API_Reference#0_42335

]]> By: Jack http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6512 Jack Mon, 29 Mar 2010 22:26:55 +0000 http://nat.org/blog/?p=1565#comment-6512 @Miguel de Icaza "There is also the Dallas market place that allows people to publish their oData sources and decide how they want to charge for it: https://www.sqlazureservices.com/Catalog.aspx" One has to admire Microsoft for doing again something ambiguous. That page is filled with *free* things that in fact are *TRIAL OFFERS*; even one from a government. @Miguel de Icaza

“There is also the Dallas market place that allows people to publish their oData sources and decide how they want to charge for it:

https://www.sqlazureservices.com/Catalog.aspx”

One has to admire Microsoft for doing again something ambiguous. That page is filled with *free* things that in fact are *TRIAL OFFERS*; even one from a government.

]]> By: Nat Friedman http://nat.org/blog/2010/03/we-have-an-api/comment-page-1/#comment-6511 Nat Friedman Mon, 29 Mar 2010 19:15:01 +0000 http://nat.org/blog/?p=1565#comment-6511 Doesn't have to be sqlite. I just find it to be a really handy way to pass around large datasets. But CSV or JSON or XML is fine too. Actually CSV is not that great since no one implements it properly - the others are fine though. I'm curious why you dislike sqlite though... Doesn’t have to be sqlite. I just find it to be a really handy way to pass around large datasets. But CSV or JSON or XML is fine too. Actually CSV is not that great since no one implements it properly – the others are fine though.

I’m curious why you dislike sqlite though…

]]>