{"id":4953,"date":"2025-12-14T18:26:00","date_gmt":"2025-12-14T09:26:00","guid":{"rendered":"https:\/\/matoken.org\/blog\/?p=4953"},"modified":"2025-12-14T18:12:04","modified_gmt":"2025-12-14T09:12:04","slug":"snac-robots-txt","status":"publish","type":"post","link":"https:\/\/matoken.org\/blog\/2025\/12\/14\/snac-robots-txt\/","title":{"rendered":"snac\u306erobots.txt"},"content":{"rendered":"<div id=\"preamble\">\n<div class=\"sectionbody\">\n<div class=\"paragraph\">\n<p><span class=\"image\"><img decoding=\"async\" src=\"https:\/\/matoken.org\/blog\/wp-content\/uploads\/2025\/12\/susie64-cool.png\" alt=\"susie64 cool\" width=\"48\"><\/span><\/p>\n<\/div>\n<div class=\"paragraph\">\n<p>\u3055\u304f\u3089\u306e\u30ca\u30ec\u30c3\u30b8\u306b\u8efd\u91cfActivityPub Server \u306esnac \u306b\u3064\u3044\u3066\u306e\u8a18\u4e8b\u3092\u5bc4\u7a3f\u3057\u307e\u3057\u305f\uff0e<\/p>\n<\/div>\n<div class=\"ulist\">\n<ul>\n<li>\n<p><a href=\"https:\/\/matoken.org\/blog\/2025\/12\/09\/sakura-knowledge\/\">\u3055\u304f\u3089\u30a4\u30f3\u30bf\u30fc\u30cd\u30c3\u30c8\u306e\u3055\u304f\u3089\u306e\u30ca\u30ec\u30c3\u30b8\u306bsnac\u306e\u8a18\u4e8b\u3092\u5bc4\u7a3f \u2013 matoken&#8217;s blog<\/a><\/p>\n<\/li>\n<\/ul>\n<\/div>\n<div class=\"paragraph\">\n<p>\u3053\u3053\u306b\u66f8\u304b\u306a\u304b\u3063\u305fsnac \u306e\u3053\u3068\u3092\u3044\u304f\u3064\u304b\u66f8\u3044\u3066\u3044\u3053\u3046\u3068\u601d\u3044\u307e\u3059\uff0e<br \/>\n\u4eca\u65e5\u306fsnac \u3067\u306erobots.txt \u306b\u3064\u3044\u3066\uff0e<\/p>\n<\/div>\n<p><!--more--><\/p>\n<div class=\"paragraph\">\n<p>\u691c\u7d22\u30a8\u30f3\u30b8\u30f3\u3067\u300csite:snac.kagolug.org\u300d\u306e\u3088\u3046\u306b\u691c\u7d22\u3057\u3066\u307f\u308b\u3068\u4f55\u3082\u51fa\u3066\u304d\u307e\u305b\u3093\uff0e\u904b\u7528\u958b\u59cb\u304b\u3089\u305d\u3046\u6642\u9593\u304c\u7d4c\u3063\u3066\u3044\u306a\u3044\u304b\u3089\u304b\u306a?\u3068\u601d\u3044\u3064\u3064\u5ff5\u306e\u70ba <code>robots.txt<\/code> \u3092\u78ba\u8a8d\u3059\u308b\u3068\u3053\u3093\u306a\u611f\u3058\u3067\u5168\u3066\u62d2\u5426\u308b\u3059\u3088\u3046\u306b\u306a\u3063\u3066\u3044\u307e\u3057\u305f\uff0e<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ w3m -dump https:\/\/snac.kagolug.org\/robots.txt\nUser-agent: *\nDisallow: \/<\/pre>\n<\/div>\n<\/div>\n<div class=\"paragraph\">\n<p>source \u3092\u78ba\u8a8d\u3059\u308b\u3068 <code>httpd.c<\/code> \u306e\u4e2d\u3067\u30cf\u30fc\u30c9\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u3055\u308c\u3066\u3044\u308b\u3088\u3046\u3067\u3059\uff0e<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ grep -n -A5 robots.txt httpd.c\n321:    if (strcmp(q_path, \"\/robots.txt\") == 0) {\n322-        status = HTTP_STATUS_OK;\n323-        *ctype = \"text\/plain\";\n324-        *body  = xs_str_new(\"User-agent: *\\n\"\n325-                            \"Disallow: \/\\n\");\n326-    }<\/pre>\n<\/div>\n<\/div>\n<div class=\"paragraph\">\n<p>snac \u306e\u8a2d\u5b9a\u3067\u3069\u3046\u306b\u304b\u3067\u304d\u308c\u3070\u3044\u3044\u3067\u3059\u304c\u6050\u3089\u304f\u7121\u7406\uff0eISSUE \u3092\u63a2\u3059\u3068\u4ee5\u4e0b\u306e\u3082\u306e\u304c\uff0e404\u30a8\u30e9\u30fc\u304c\u90aa\u9b54\u306a\u306e\u3067\u7528\u610f\u3057\u3066\u3044\u308b\u3060\u3051\u3067\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u3057\u305f\u3044\u4eba\u306f\u524d\u6bb5\u306ehttp proxy \u3067\u30eb\u30fc\u30eb\u3092\u8ffd\u52a0\u3059\u308b\u611f\u3058\u306e\u3088\u3046\u3067\u3059\uff0e<\/p>\n<\/div>\n<div class=\"ulist\">\n<ul>\n<li>\n<p><a href=\"https:\/\/codeberg.org\/grunfink\/snac2\/issues\/223\">#223 &#8211; default <code>robots.txt<\/code> breaks integration with fedi-fetcher &#8211; grunfink\/snac2 &#8211; Codeberg.org<\/a><\/p>\n<\/li>\n<\/ul>\n<\/div>\n<div class=\"paragraph\">\n<p>ISSUE \u3092\u7acb\u3066\u305f\u65b9\u306fnginx \u3067\u30ab\u30b9\u30bf\u30e0robots.txt \u3092\u8fd4\u3059\u3088\u3046\u306b\u3057\u3066\u3044\u308b\u3088\u3046\u3067\u3059\uff0e<\/p>\n<\/div>\n<div class=\"quoteblock\">\n<blockquote>\n<div class=\"paragraph\">\n<p>Actually, I am able to work around this by returning a custom robots.txt with nginx:<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>location = \/robots.txt {\n\t\treturn 200 'User-agent: FediFetcher\\nAllow: \/\\nUser-agent: *\\nDisallow: \/\\n';\n}<\/pre>\n<\/div>\n<\/div>\n<\/blockquote>\n<\/div>\n<div class=\"paragraph\">\n<p>\u81ea\u5206\u306e\u74b0\u5883\u306fApache2 httpd \u3067\u3059\u304c\u540c\u3058\u3088\u3046\u306a\u8a2d\u5b9a\u306f\u3067\u304d\u308b\u306e\u3067\u8a2d\u5b9a\u3057\u3066\u307f\u307e\u3059\uff0e<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"sect1\">\n<h2 id=\"_apache2_httpd_\u306ealias_\u3067robots_txt_\u3092\u8a2d\u5b9a\">Apache2 httpd \u306eAlias \u3067robots.txt \u3092\u8a2d\u5b9a<\/h2>\n<div class=\"sectionbody\">\n<div class=\"paragraph\">\n<p>\u4ee5\u524dNitter \u3092\u516c\u958b\u3057\u3066\u3044\u305f\u3068\u304d\u306b\u3082\u3084\u3063\u3066\u3044\u305f\u306e\u3067\u3053\u308c\u3092\u53c2\u8003\u306b\u8a2d\u5b9a\u3057\u307e\u3059\uff0e<\/p>\n<\/div>\n<div class=\"ulist\">\n<ul>\n<li>\n<p><a href=\"https:\/\/matoken.org\/blog\/2021\/05\/31\/set-robots-txt-in-nitter-alias-setting-in-reverse-proxy-environment-of-apache-httpd\/\">Nitter\u306brobots.txt\u3092\u8a2d\u5b9a(Apache httpd\u306ereverse proxy\u74b0\u5883\u3067Alias\u8a2d\u5b9a) \u2013 matoken&#8217;s blog<\/a><\/p>\n<\/li>\n<\/ul>\n<\/div>\n<div class=\"paragraph\">\n<p>Apache2 httpd \u306esnac \u90e8\u5206\u306e\u8a2d\u5b9a\u3092\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u4fee\u6b63\u3057\u307e\u3057\u305f\uff0e\u3053\u308c\u3067Linux \u30d5\u30a1\u30a4\u30eb\u30b7\u30b9\u30c6\u30e0\u5185\u306e <code>\/var\/lib\/snac2\/robots.txt<\/code> \u304c <code>\/robots.txt<\/code> \u306b\u306a\u308b\u306f\u305a\u3067\u3059\uff0e<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ sudo git diff \/etc\/apache2\/sites-available\/011-snac.kagolug.org.conf\ndiff --git a\/apache2\/sites-available\/011-snac.kagolug.org.conf b\/apache2\/sites-available\/011-snac.kagolug.org.conf\nindex f5b5c7f..7bb72c1 100644\n--- a\/apache2\/sites-available\/011-snac.kagolug.org.conf\n+++ b\/apache2\/sites-available\/011-snac.kagolug.org.conf\n@@ -58,6 +58,11 @@\n        &lt;\/Location&gt;\n\n        Alias \/static \/var\/www\/static\n+       Alias \/robots.txt \/var\/lib\/snac2\/robots.txt\n+       &lt;Location \"\/robots.txt\"&gt;\n+               ProxyPass !\n+               Require all granted\n+       &lt;\/Location&gt;\n\n        # Possible values include: debug, info, notice, warn, error, crit,\n        # alert, emerg.<\/pre>\n<\/div>\n<\/div>\n<div class=\"paragraph\">\n<p>\u8a2d\u5b9a\u3092\u78ba\u8a8d\u3057\u3066\u304b\u3089\u53cd\u6620\u3057\u307e\u3059\uff0e<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ sudo apache2ctl configtest\n$ sudo systemctl reload apache2<\/pre>\n<\/div>\n<\/div>\n<div class=\"paragraph\">\n<p>\u3053\u306e\u30d5\u30a1\u30a4\u30eb\u304c\uff0c<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ sudo -u www-data cat \/var\/lib\/snac2\/robots.txt\nUser-agent: *\nAllow: \/<\/pre>\n<\/div>\n<\/div>\n<div class=\"paragraph\">\n<p>\/robots.txt \u306b\u53cd\u6620\u3055\u308c\u307e\u3057\u305f\uff0e<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ w3m -dump https:\/\/snac.kagolug.org\/robots.txt\nUser-agent: *\nAllow: \/<\/pre>\n<\/div>\n<\/div>\n<div class=\"paragraph\">\n<p>\u3053\u308c\u3067\u5404\u7a2ebot \u304b\u3089\u306e\u30a2\u30af\u30bb\u30b9\u304c\u8a31\u53ef\u3055\u308c\u307e\u3057\u305f\uff0e<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"sect1\">\n<h2 id=\"_source_\u66f8\u304d\u63db\u3048\">source \u66f8\u304d\u63db\u3048<\/h2>\n<div class=\"sectionbody\">\n<div class=\"paragraph\">\n<p>source \u304b\u3089build \u3057\u3066\u3044\u308b\u74b0\u5883\u3067\u306fsource \u66f8\u304d\u63db\u3048\u3067\u3082robots.txt \u3092\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u3067\u304d\u307e\u3057\u305f\uff0e\u3067\u3082\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u306e\u305f\u3073\u306b\u30d1\u30c3\u30c1\u3092\u5f53\u3066\u308b\u306e\u306f\u9762\u5012\u306a\u306e\u3067\u3084\u306f\u308aproxy \u3067\u8a2d\u5b9a\u3059\u308b\u307b\u3046\u304c\u826f\u3055\u305d\u3046\u3067\u3059\uff0e<\/p>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"title\">robots.txt \u3092\u7121\u304f\u3059(404 \u30a8\u30e9\u30fc\u304c\u305f\u304f\u3055\u3093\u306a\u306e\u3067\u3042\u307e\u308a\u826f\u304f\u7121\u3055\u305d\u3046)<\/div>\n<div class=\"content\">\n<pre>$ git diff httpd.c\ndiff --git a\/httpd.c b\/httpd.c\nindex a8cd849..976fdd6 100644\n--- a\/httpd.c\n+++ b\/httpd.c\n@@ -318,13 +318,6 @@ int server_get_handler(xs_dict *req, const char *q_path,\n         *body  = xs_json_dumps(j, 4);\n     }\n     else\n-    if (strcmp(q_path, \"\/robots.txt\") == 0) {\n-        status = HTTP_STATUS_OK;\n-        *ctype = \"text\/plain\";\n-        *body  = xs_str_new(\"User-agent: *\\n\"\n-                            \"Disallow: \/\\n\");\n-    }\n-    else\n     if (strcmp(q_path, \"\/style.css\") == 0) {\n         FILE *f;\n         xs *css_fn = xs_fmt(\"%s\/style.css\", srv_basedir);<\/pre>\n<\/div>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ w3m -dump https:\/\/snac.matoken.org\/robots.txt\n404 Not Found (snac\/2.85)<\/pre>\n<\/div>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"title\">\u5168\u3066\u8a31\u53ef\u306b\u66f8\u304d\u63db\u3048<\/div>\n<div class=\"content\">\n<pre>$ git diff httpd.c\ndiff --git a\/httpd.c b\/httpd.c\nindex a8cd849..c0bcaef 100644\n--- a\/httpd.c\n+++ b\/httpd.c\n@@ -322,7 +322,7 @@ int server_get_handler(xs_dict *req, const char *q_path,\n         status = HTTP_STATUS_OK;\n         *ctype = \"text\/plain\";\n         *body  = xs_str_new(\"User-agent: *\\n\"\n-                            \"Disallow: \/\\n\");\n+                            \"Allow: \/\\n\");\n     }\n     else\n     if (strcmp(q_path, \"\/style.css\") == 0) {<\/pre>\n<\/div>\n<\/div>\n<div class=\"listingblock\">\n<div class=\"content\">\n<pre>$ w3m -dump https:\/\/snac.matoken.org\/robots.txt\nUser-agent: *\nAllow: \/<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>\u3055\u304f\u3089\u306e\u30ca\u30ec\u30c3\u30b8\u306b\u8efd\u91cfActivityPub Server \u306esnac \u306b\u3064\u3044\u3066\u306e\u8a18\u4e8b\u3092\u5bc4\u7a3f\u3057\u307e\u3057\u305f\uff0e \u3055\u304f\u3089\u30a4\u30f3\u30bf\u30fc\u30cd\u30c3\u30c8\u306e\u3055\u304f\u3089\u306e\u30ca\u30ec\u30c3\u30b8\u306bsnac\u306e\u8a18\u4e8b\u3092\u5bc4\u7a3f \u2013 matoken&#8217;s blog \u3053\u3053\u306b\u66f8 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"webmentions_disabled_pings":false,"webmentions_disabled":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"federated","footnotes":""},"categories":[7,6,1231],"tags":[493,1248,1247,1243],"class_list":["post-4953","post","type-post","status-publish","format-standard","hentry","category-debian-linux","category-linux","category-trixie","tag-activitypub","tag-apache2-httpd","tag-robots-txt","tag-snac"],"_links":{"self":[{"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/posts\/4953","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/comments?post=4953"}],"version-history":[{"count":2,"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/posts\/4953\/revisions"}],"predecessor-version":[{"id":4956,"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/posts\/4953\/revisions\/4956"}],"wp:attachment":[{"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/media?parent=4953"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/categories?post=4953"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/matoken.org\/blog\/wp-json\/wp\/v2\/tags?post=4953"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}