{"id":292,"date":"2004-03-03T13:04:00","date_gmt":"2004-03-03T17:04:00","guid":{"rendered":"http:\/\/www.markbaker.ca\/wp\/?p=314"},"modified":"2004-03-03T13:04:00","modified_gmt":"2004-03-03T17:04:00","slug":"tim-bray-on-googlebot","status":"publish","type":"post","link":"http:\/\/www.markbaker.ca\/blog\/2004\/03\/tim-bray-on-googlebot\/","title":{"rendered":"Tim Bray on Googlebot"},"content":{"rendered":"<p><a href=\"http:\/\/www.tbray.org\/ongoing\/When\/200x\/2004\/03\/02\/GoogleStorm\">Tim notes<\/a>\nthat Googlebot is frequenting his server, and costing him real money.<\/p>\n\n<p>A quick investigation on an <a href=\"http:\/\/www.tbray.org\/ongoing\/When\/200x\/2003\/08\/11\/\">old page<\/a>\nfrom his log reveals;<\/p>\n\n<pre>\nHTTP\/1.1 200 OK\nDate: Wed, 03 Mar 2004 14:20:02 GMT\nServer: Apache\/1.3.26 (Unix) Debian GNU\/Linux\nLast-Modified: Wed, 03 Mar 2004 08:00:54 GMT\nETag: \"1b404e-f1d-404590b6\"\nAccept-Ranges: bytes\nContent-Length: 3869\nKeep-Alive: timeout=15, max=20\nConnection: Keep-Alive\nContent-Type: text\/html; charset=utf-8\n<\/pre>\n\n<p>Like many other agents and caches, Googlebot presumably uses some\n&#8220;freshness&#8221; heuristic based on Last-Modified.  As you can see above, Tim&#8217;s\nserver is telling the world that even his archived content changes\nfrequently.  Ergo, Google hits him frequently.  Conclusion; don&#8217;t do that! 8-)<\/p>\n\n<p>Full disclosure; my weblog isn&#8217;t cacheable at all &#8211; not even any\nLast-Modified headers &#8211; and I have little motivation to fix it because my bandwidth\nisn&#8217;t metered.<\/p>","protected":false},"excerpt":{"rendered":"Tim notes that Googlebot is frequenting his server, and costing him real money. A quick investigation on an old page from his log reveals; HTTP\/1.1 200 OK Date: Wed, 03 Mar 2004 14:20:02 GMT Server: Apache\/1.3.26 (Unix) Debian GNU\/Linux Last-Modified: Wed, 03 Mar 2004 08:00:54 GMT ETag: &#8220;1b404e-f1d-404590b6&#8221; Accept-Ranges: bytes Content-Length: 3869 Keep-Alive: timeout=15, max=20 [&hellip;]","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-292","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/posts\/292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/comments?post=292"}],"version-history":[{"count":0,"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/posts\/292\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/media?parent=292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/categories?post=292"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.markbaker.ca\/blog\/wp-json\/wp\/v2\/tags?post=292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}