"\uAD6C\uAE00\uBD07(Googlebot)\uC740 \uAD6C\uAE00\uC774 \uC0AC\uC6A9\uD558\uB294 \uAC80\uC0C9 \uBD07 \uC18C\uD504\uD2B8\uC6E8\uC5B4\uC774\uBA70 \uC6F9\uC73C\uB85C\uBD80\uD130 \uBB38\uC11C\uB97C \uC218\uC9D1\uD558\uC5EC \uAD6C\uAE00 \uAC80\uC0C9 \uC5D4\uC9C4\uC744 \uC704\uD55C \uAC80\uC0C9 \uAC00\uB2A5\uD55C \uC0C9\uC778\uC744 \uB9CC\uB4E0\uB2E4. \uC774 \uC774\uB984\uC740 \uC2E4\uC81C\uB85C \uAC01\uAE30 \uB2E4\uB978 \uB450 \uC885\uB958\uC758 \uC6F9 \uD06C\uB864\uB7EC\uB97C \uAC00\uB9AC\uD0A4\uAE30 \uC704\uD574 \uC0AC\uC6A9\uB41C\uB2E4: \uB370\uC2A4\uD06C\uD1B1 \uD06C\uB864\uB7EC(\uB370\uC2A4\uD06C\uD1B1 \uC0AC\uC6A9\uC790\uB97C \uC2DC\uBBAC\uB808\uC774\uD2B8\uD558\uAE30 \uC704\uD568), \uBAA8\uBC14\uC77C \uD06C\uB864\uB7EC(\uBAA8\uBC14\uC77C \uC0AC\uC6A9\uC790\uB97C \uC2DC\uBBAC\uB808\uC774\uD2B8\uD558\uAE30 \uC704\uD568) \uC6F9\uC0AC\uC774\uD2B8\uB294 \uAD6C\uAE00\uBD07 \uB370\uC2A4\uD06C\uD1B1\uACFC \uAD6C\uAE00\uBD07 \uBAA8\uBC14\uC77C\uC5D0 \uC758\uD574 \uD06C\uB864\uB9C1\uB420 \uAC00\uB2A5\uC131\uC774 \uC788\uB2E4. \uC694\uCCAD \uC911 \uC0AC\uC6A9\uC790 \uC5D0\uC774\uC804\uD2B8 \uBB38\uC790\uC5F4\uC744 \uC0B4\uD3B4\uBD04\uC73C\uB85C\uC368 \uAD6C\uAE00\uBD07\uC758 \uC11C\uBE0C\uD0C0\uC785\uC744 \uC2DD\uBCC4\uD560 \uC218 \uC788\uB2E4. \uADF8\uB7EC\uB098 \uB450 \uC885\uB958\uC758 \uD06C\uB864\uB7EC \uBAA8\uB450 robots.txt\uC758 \uB3D9\uC77C\uD55C \uC81C\uD488 \uD1A0\uD070(useent \uD1A0\uD070)\uC744 \uC900\uC218\uD558\uBBC0\uB85C robots.txt\uB97C \uC0AC\uC6A9\uD558\uC5EC \uAD6C\uAE00\uBD07 \uBAA8\uBC14\uC77C \uB610\uB294 \uAD6C\uAE00\uBD07 \uB370\uC2A4\uD06C\uD1B1\uC744 \uC120\uBCC4\uC801\uC73C\uB85C \uC9C0\uC815\uD560 \uC218 \uC5C6\uB2E4."@ko . . "Googlebot"@it . "Googlebot ist ein Webcrawler des US-amerikanischen Unternehmens Google LLC. Das Computerprogramm l\u00E4dt World-Wide-Web-Inhalte eigenst\u00E4ndig herunter und f\u00FChrt diese der herstellereigenen Suchmaschine Google zu."@de . "Googlebot"@pt . . "298642"^^ . . "\u062C\u0648\u062C\u0644 \u0628\u0648\u062A \u0623\u0648 \u063A\u0648\u063A\u0644 \u0628\u0648\u062A (\u0632\u0648\u0627\u062D\u0641 \u062C\u0648\u062C\u0644 \u0623\u0648 \u0639\u0646\u0627\u0643\u0628 \u062C\u0648\u062C\u0644) (\u0628\u0627\u0644\u0625\u0646\u062C\u0644\u064A\u0632\u064A\u0629: Googlebot) \u0647\u0648 \u0628\u0648\u062A \u0648\u0638\u064A\u0641\u062A\u0647 \u0627\u0644\u062A\u063A\u0644\u063A\u0644 \u062F\u0627\u062E\u0644 \u0645\u0648\u0627\u0642\u0639 \u0627\u0644\u0648\u064A\u0628 \u0648\u0641\u0647\u0631\u0633\u0629 \u0627\u0644\u0643\u0644\u0645\u0627\u062A \u0627\u0644\u0645\u0648\u062C\u0648\u062F\u0629 \u0641\u064A\u0647\u0627\u060C \u0628\u062D\u064A\u062B \u064A\u0638\u0647\u0631 \u0627\u0644\u0645\u0648\u0642\u0639 \u0641\u064A \u0646\u062A\u0627\u0626\u062C \u0645\u062D\u0631\u0643 \u0627\u0644\u0628\u062D\u062B \u062C\u0648\u062C\u0644 \u0625\u0630\u0627 \u062A\u0648\u0641\u0631\u062A \u0641\u064A\u0647 \u0627\u0644\u0643\u0644\u0645\u0629 \u0627\u0644\u0645\u0628\u062D\u0648\u062B \u0639\u0646\u0647\u0627. \u0623\u063A\u0644\u0628 \u0645\u0648\u0627\u0642\u0639 \u0645\u062D\u0631\u0643\u0627\u062A \u0627\u0644\u0628\u062D\u062B \u062A\u0642\u0648\u0645 \u0628\u0648\u0627\u0633\u0637\u0629 \u0628\u0631\u0646\u0627\u0645\u062C \u064A\u0633\u0645\u0649 \u00ABweb spider\u00BB \u0628\u062A\u062C\u0645\u064A\u0639 \u0642\u0648\u0627\u0626\u0645 \u0628\u0627\u0644\u0643\u0644\u0645\u0627\u062A \u0627\u0644\u0645\u0648\u062C\u0648\u062F\u0629 \u0639\u0644\u0649 \u0627\u0644\u0634\u0628\u0643\u0629 \u0648\u062A\u0633\u0645\u0649 \u0647\u0630\u0647 \u0627\u0644\u0639\u0645\u0644\u064A\u0629 \u00ABweb crawling\u00BB\u060C \u0648\u0628\u0630\u0644\u0643 \u0641\u0639\u0646\u062F\u0645\u0627 \u062A\u0628\u062D\u062B \u0639\u0646 \u0643\u0644\u0645\u0629 \u0645\u0639\u064A\u0646\u0629 \u0641\u0623\u0646\u0647\u0627 \u062A\u0628\u062D\u062B \u0639\u0646\u0647\u0627 \u0641\u064A \u062A\u0644\u0643 \u0627\u0644\u0642\u0648\u0627\u0626\u0645 \u0627\u0644\u0645\u0639\u062F\u0629 \u0645\u0646 \u0642\u0628\u0644 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062A. \u0644\u0648 \u0623\u0631\u0627\u062F \u0645\u0634\u0631\u0641 \u0645\u062D\u0631\u0643 \u0628\u062D\u062B \u00ABwebmaster\u00BB \u062A\u0642\u064A\u062F \u0628\u0639\u0636 \u0627\u0644\u0645\u0639\u0644\u0648\u0645\u0627\u062A \u0639\u0644\u0649 \u0627\u0644\u0645\u0648\u0627\u0642\u0639 \u0627\u0644\u062A\u064A \u064A\u0634\u0631\u0641 \u0639\u0644\u064A\u0647\u0627 \u0628\u0645\u0646\u0639 \u0632\u0648\u0627\u062D\u0641 \u0645\u062D\u0631\u0643\u0627\u062A \u0627\u0644\u0628\u062D\u062B \u0623\u0648 \u0627\u0644\u0639\u0646\u0627\u0643\u0628 \u0645\u062B\u0644 \u063A\u0648\u063A\u0644 \u0628\u0648\u062A \u0641\u0625\u0646\u0647 \u064A\u0642\u0648\u0645 \u0628\u0630\u0644\u0643 \u0639\u0646 \u0637\u0631\u064A\u0642 \u0627\u0644\u062A\u0648\u062C\u064A\u0647\u0627\u062A \u0627\u0644\u0645\u0646\u0627\u0633\u0628\u0629 \u0645\u0646 \u062E\u0644\u0627\u0644 \u0645\u0644\u0641 \u00ABrobot.txt\u00BB \u0648\u0647\u0648 \u0645\u0639\u064A\u0627\u0631 \u0627\u0633\u062A\u0628\u0639\u0627\u062F \u0627\u0644\u0631\u0648\u0628\u0648\u062A\u0627\u062A\u060C \u0623\u0648 \u0639\u0646 \u0637\u0631\u064A\u0642 \u0625\u0636\u0627\u0641\u0629 \u00ABMeta Tag\u00BB \u0627\u0644\u062A\u0627\u0644\u064A \u0644\u0635\u0641\u062D\u0629 \u0627\u0644\u0648\u064A\u0628 \u0627\u0644\u0645\u0637\u0644\u0648\u0628\u0629 : "@ar . . . "Googlebot\uFF08\u30B0\u30FC\u30B0\u30EB\u30DC\u30C3\u30C8\uFF09\u306F\u3001Google\u304C\u63A1\u7528\u3057\u3066\u3044\u308B\u30AF\u30ED\u30FC\u30E9\uFF08Web\u30ED\u30DC\u30C3\u30C8\uFF09\u30BD\u30D5\u30C8\u30A6\u30A7\u30A2\u3067\u3042\u308B\u3002"@ja . . . . "Googlebot is the web crawler software used by Google, which collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler (to simulate desktop users) and a mobile crawler (to simulate a mobile user)."@en . "Googlebot je vyhled\u00E1vac\u00ED robot softwarov\u00E9 spole\u010Dnosti Google, kter\u00FD indexuje webov\u00E9 str\u00E1nky, a pak je lze dohledat p\u0159es vyhled\u00E1va\u010D Google. Sb\u011Br prob\u00EDh\u00E1 z velk\u00E9 \u0159ady IP adres a samostatn\u00FDch po\u010D\u00EDta\u010D\u016F, proto je sb\u011Br opravdu velmi rychl\u00FD a tak si m\u016F\u017Ee Google dovolit aktu\u00E1lnost i p\u00E1r hodin. Na n\u011Bkter\u00E9 weby nen\u00ED vhodn\u00E9 roboty pou\u0161t\u011Bt z d\u016Fvodu citliv\u00FDch \u00FAdaj\u016F, kter\u00E9 by se nem\u011Bly b\u011B\u017En\u011B zobrazovat ve v\u00FDsledc\u00EDch hled\u00E1n\u00ED. Vstup lze zak\u00E1zat p\u0159es soubor robots.txt, a nebo p\u0159es speci\u00E1ln\u00ED meta hlavi\u010Dku: Googlebot se do logu zapisuje v\u011Bt\u0161inou takto: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"@cs . . . . . . . . "Googlebot je vyhled\u00E1vac\u00ED robot softwarov\u00E9 spole\u010Dnosti Google, kter\u00FD indexuje webov\u00E9 str\u00E1nky, a pak je lze dohledat p\u0159es vyhled\u00E1va\u010D Google. Sb\u011Br prob\u00EDh\u00E1 z velk\u00E9 \u0159ady IP adres a samostatn\u00FDch po\u010D\u00EDta\u010D\u016F, proto je sb\u011Br opravdu velmi rychl\u00FD a tak si m\u016F\u017Ee Google dovolit aktu\u00E1lnost i p\u00E1r hodin. Na n\u011Bkter\u00E9 weby nen\u00ED vhodn\u00E9 roboty pou\u0161t\u011Bt z d\u016Fvodu citliv\u00FDch \u00FAdaj\u016F, kter\u00E9 by se nem\u011Bly b\u011B\u017En\u011B zobrazovat ve v\u00FDsledc\u00EDch hled\u00E1n\u00ED. Vstup lze zak\u00E1zat p\u0159es soubor robots.txt, a nebo p\u0159es speci\u00E1ln\u00ED meta hlavi\u010Dku: Googlebot se do logu zapisuje v\u011Bt\u0161inou takto:"@cs . . . . . . . . . "\uAD6C\uAE00\uBD07(Googlebot)\uC740 \uAD6C\uAE00\uC774 \uC0AC\uC6A9\uD558\uB294 \uAC80\uC0C9 \uBD07 \uC18C\uD504\uD2B8\uC6E8\uC5B4\uC774\uBA70 \uC6F9\uC73C\uB85C\uBD80\uD130 \uBB38\uC11C\uB97C \uC218\uC9D1\uD558\uC5EC \uAD6C\uAE00 \uAC80\uC0C9 \uC5D4\uC9C4\uC744 \uC704\uD55C \uAC80\uC0C9 \uAC00\uB2A5\uD55C \uC0C9\uC778\uC744 \uB9CC\uB4E0\uB2E4. \uC774 \uC774\uB984\uC740 \uC2E4\uC81C\uB85C \uAC01\uAE30 \uB2E4\uB978 \uB450 \uC885\uB958\uC758 \uC6F9 \uD06C\uB864\uB7EC\uB97C \uAC00\uB9AC\uD0A4\uAE30 \uC704\uD574 \uC0AC\uC6A9\uB41C\uB2E4: \uB370\uC2A4\uD06C\uD1B1 \uD06C\uB864\uB7EC(\uB370\uC2A4\uD06C\uD1B1 \uC0AC\uC6A9\uC790\uB97C \uC2DC\uBBAC\uB808\uC774\uD2B8\uD558\uAE30 \uC704\uD568), \uBAA8\uBC14\uC77C \uD06C\uB864\uB7EC(\uBAA8\uBC14\uC77C \uC0AC\uC6A9\uC790\uB97C \uC2DC\uBBAC\uB808\uC774\uD2B8\uD558\uAE30 \uC704\uD568) \uC6F9\uC0AC\uC774\uD2B8\uB294 \uAD6C\uAE00\uBD07 \uB370\uC2A4\uD06C\uD1B1\uACFC \uAD6C\uAE00\uBD07 \uBAA8\uBC14\uC77C\uC5D0 \uC758\uD574 \uD06C\uB864\uB9C1\uB420 \uAC00\uB2A5\uC131\uC774 \uC788\uB2E4. \uC694\uCCAD \uC911 \uC0AC\uC6A9\uC790 \uC5D0\uC774\uC804\uD2B8 \uBB38\uC790\uC5F4\uC744 \uC0B4\uD3B4\uBD04\uC73C\uB85C\uC368 \uAD6C\uAE00\uBD07\uC758 \uC11C\uBE0C\uD0C0\uC785\uC744 \uC2DD\uBCC4\uD560 \uC218 \uC788\uB2E4. \uADF8\uB7EC\uB098 \uB450 \uC885\uB958\uC758 \uD06C\uB864\uB7EC \uBAA8\uB450 robots.txt\uC758 \uB3D9\uC77C\uD55C \uC81C\uD488 \uD1A0\uD070(useent \uD1A0\uD070)\uC744 \uC900\uC218\uD558\uBBC0\uB85C robots.txt\uB97C \uC0AC\uC6A9\uD558\uC5EC \uAD6C\uAE00\uBD07 \uBAA8\uBC14\uC77C \uB610\uB294 \uAD6C\uAE00\uBD07 \uB370\uC2A4\uD06C\uD1B1\uC744 \uC120\uBCC4\uC801\uC73C\uB85C \uC9C0\uC815\uD560 \uC218 \uC5C6\uB2E4. \uC6F9\uB9C8\uC2A4\uD130\uAC00 \uAD6C\uAE00\uBD07\uC774\uB098 \uB2E4\uB978 \uC2A4\uD30C\uC774\uB354\uC5D0 \uC774\uC6A9\uD560 \uC218 \uC788\uB294 \uC0AC\uC774\uD2B8\uC758 \uC815\uBCF4\uB97C \uC81C\uD55C\uD558\uACE0 \uC2F6\uB2E4\uBA74 robots.txt \uD30C\uC77C\uC5D0 \uC801\uC808\uD55C \uB514\uB809\uD2F0\uBE0C\uB97C \uC9C0\uC815\uD558\uC5EC \uD574\uB2F9 \uC791\uC5C5\uC744 \uC218\uD589\uD560 \uC218 \uC788\uC73C\uBA70, \uBA54\uD0C0 \uD0DC\uADF8 \uB97C \uC6F9 \uD398\uC774\uC9C0\uC5D0 \uCD94\uAC00\uD558\uC5EC \uC9C4\uD589\uD560 \uC218\uB3C4 \uC788\uB2E4. \uC6F9 \uC11C\uBC84 \uB300\uD55C \uAD6C\uAE00\uBD07\uC758 \uC694\uCCAD\uC740 \"Googlebot\"\uC744 \uD3EC\uD568\uD558\uB294 \uC0AC\uC6A9\uC790 \uC5D0\uC774\uC804\uD2B8 \uBB38\uC790\uC5F4, \"googlebot.com\"\uC744 \uD3EC\uD568\uD558\uB294 \uD638\uC2A4\uD2B8 \uC8FC\uC18C\uB85C \uC2DD\uBCC4\uC774 \uAC00\uB2A5\uD558\uB2E4. \uD604\uC7AC \uAD6C\uAE00\uBD07\uC740 HREF \uB9C1\uD06C\uC640 SRC \uB9C1\uD06C\uB97C \uB530\uB978\uB2E4. \uAD6C\uAE00\uBD07\uC774 \uC790\uBC14\uC2A4\uD06C\uB9BD\uD2B8\uB97C \uC2E4\uD589\uD558\uACE0 Ajax \uD638\uCD9C\uC5D0 \uC758\uD574 \uC0DD\uC131\uB41C \uCF58\uD150\uCE20\uC758 \uAD6C\uBB38\uC744 \uBD84\uC11D\uD560 \uC218 \uC788\uB2E4\uB294 \uC815\uD669\uC774 \uC788\uB2E4. \uC9C4\uBCF4\uB41C \uAD6C\uAE00\uBD07\uC758 \uAE30\uB2A5\uC744 \uC124\uBA85\uD558\uB294 \uC5EC\uB7EC \uC774\uB860\uB4E4\uC744 \uB4B7\uBC1B\uCE68\uD558\uB294 \uAC83\uC740 \uC790\uBC14\uC2A4\uD06C\uB9BD\uD2B8\uB97C \uCC98\uB9AC\uD558\uB294 \uAC83\uC774\uB2E4. \uD604\uC7AC \uAD6C\uAE00\uBD07\uC740 \uD06C\uB86C 41(M41) \uAE30\uBC18\uC758 \uC6F9 \uB80C\uB354\uB9C1 \uC11C\uBE44\uC2A4(WRS)\uB97C \uC0AC\uC6A9\uD55C\uB2E4. \uAD6C\uAE00\uBD07\uC740 \uBAA8\uB4E0 \uBB38\uC11C\uC758 \uBAA8\uB4E0 \uB9C1\uD06C\uB97C \uCC3E\uC544\uB4E4\uC5B4\uAC00\uBA74\uC11C \uD398\uC774\uC9C0\uB97C \uBC1C\uACAC\uD55C\uB2E4. \uADF8 \uB2E4\uC74C \uB2E4\uB978 \uC6F9 \uD398\uC774\uC9C0\uC758 \uC774 \uB9C1\uD06C\uB4E4\uC744 \uB530\uB77C\uAC04\uB2E4. \uC0C8 \uC6F9 \uD398\uC774\uC9C0\uB4E4\uC740 \uC6F9 \uC0C1\uC5D0 \uC54C\uB824\uC9C4 \uB2E4\uB978 \uBB38\uC11C\uB4E4\uACFC \uC5F0\uACB0\uB418\uC5B4 \uC788\uC5B4\uC57C \uC6F9\uB9C8\uC2A4\uD130\uAC00 \uD06C\uB864\uB9C1, \uC0C9\uC778, \uC218\uB3D9 \uC81C\uCD9C\uC744 \uD560 \uC218 \uC788\uAC8C \uB41C\uB2E4."@ko . . "Googlebot \u0446\u0435 \u043F\u043E\u0448\u0443\u043A\u043E\u0432\u0438\u0439 \u0440\u043E\u0431\u043E\u0442, \u0441\u0442\u0432\u043E\u0440\u0435\u043D\u0438\u0439 Google, \u044F\u043A\u0438\u0439 \u0437\u0431\u0438\u0440\u0430\u0454 \u0434\u043E\u043A\u0443\u043C\u0435\u043D\u0442\u0438 \u0437 \u0456\u043D\u0442\u0435\u0440\u043D\u0435\u0442\u0443 \u0434\u043B\u044F \u0431\u0443\u0434\u0456\u0432\u043D\u0438\u0446\u0442\u0432\u0430 \u043F\u043E\u0448\u0443\u043A\u043E\u0432\u043E\u0433\u043E \u0456\u043D\u0434\u0435\u043A\u0441\u0443 \u043F\u043E\u0448\u0443\u043A\u043E\u0432\u0438\u043A\u0430 Google. \u042F\u043A\u0449\u043E \u0432\u0435\u0431\u043C\u0430\u0439\u0441\u0442\u0435\u0440 \u0431\u0430\u0436\u0430\u0454 \u043E\u0431\u043C\u0435\u0436\u0438\u0442\u0438 \u0456\u043D\u0444\u043E\u0440\u043C\u0430\u0446\u0456\u044E \u043D\u0430 \u0441\u0432\u043E\u0454\u043C\u0443 \u0441\u0430\u0439\u0442\u0456 \u0434\u043B\u044F Googlebot, \u0432\u0456\u043D \u043C\u043E\u0436\u0435 \u0432\u0438\u043A\u043E\u0440\u0438\u0441\u0442\u0430\u0442\u0438 \u0444\u0430\u0439\u043B robots.txt \u0430\u0431\u043E \u043C\u0435\u0442\u0430-\u0442\u0435\u0433 noindex. Googlebot \u0456\u0434\u0435\u043D\u0442\u0438\u0444\u0456\u043A\u0443\u0454 \u0441\u0435\u0431\u0435 \u0447\u0435\u0440\u0435\u0437 \u0441\u0442\u0440\u0456\u0447\u043A\u0443 user agent, \u0432 \u044F\u043A\u0456\u0439 \u043C\u0456\u0441\u0442\u0438\u0442\u044C\u0441\u044F \u0441\u043B\u043E\u0432\u043E googlebot."@uk . . "Googlebot"@uk . "Googlebot \u00E9 o rob\u00F4 indexador usado pelo Google para indexar p\u00E1ginas e documentos da web para construir um \u00EDndice de buscas para o Google Search. Existem dois tipos de rastreadores: o Googlebot Desktop e o Googlebot Mobile para dispositivos m\u00F3veis. Ambos simulam a a\u00E7\u00E3o dos usu\u00E1rios nesses dois ambientes."@pt . "Googlebot \u2014 \u043F\u043E\u0438\u0441\u043A\u043E\u0432\u044B\u0439 \u0440\u043E\u0431\u043E\u0442 \u043A\u043E\u043C\u043F\u0430\u043D\u0438\u0438 Google, \u0438\u0441\u043F\u043E\u043B\u044C\u0437\u0443\u0435\u043C\u044B\u0439 \u0434\u043B\u044F \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u044F \u043C\u0438\u0440\u043E\u0432\u043E\u0439 \u0441\u0435\u0442\u0438 \u0438 \u043F\u043E\u043F\u043E\u043B\u043D\u0435\u043D\u0438\u044F \u043F\u043E\u0438\u0441\u043A\u043E\u0432\u043E\u0433\u043E \u0438\u043D\u0434\u0435\u043A\u0441\u0430 \u0441\u0438\u0441\u0442\u0435\u043C\u044B Google. \u0420\u043E\u0431\u043E\u0442 \u043D\u0430\u0447\u0438\u043D\u0430\u0435\u0442 \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u0435 \u043A\u0430\u0436\u0434\u043E\u0433\u043E \u043E\u0442\u0434\u0435\u043B\u044C\u043D\u043E\u0433\u043E \u0441\u0430\u0439\u0442\u0430 \u0441 \u043F\u043E\u043B\u0443\u0447\u0435\u043D\u0438\u044F \u0441\u043F\u0438\u0441\u043A\u0430 URL-\u0430\u0434\u0440\u0435\u0441\u043E\u0432 \u0432\u0435\u0431-\u0441\u0442\u0440\u0430\u043D\u0438\u0446, \u0441\u043E\u0437\u0434\u0430\u043D\u043D\u043E\u0433\u043E \u043F\u043E \u0440\u0435\u0437\u0443\u043B\u044C\u0442\u0430\u0442\u0430\u043C \u043F\u0440\u0435\u0434\u044B\u0434\u0443\u0449\u0438\u0445 \u0441\u0435\u0430\u043D\u0441\u043E\u0432, \u0438 \u043F\u043E\u043F\u043E\u043B\u043D\u044F\u0435\u0442 \u0435\u0433\u043E \u0434\u0430\u043D\u043D\u044B\u043C\u0438 \u0438\u0437 \u0444\u0430\u0439\u043B\u043E\u0432 Sitemap; \u043F\u0440\u043E\u0441\u043C\u0430\u0442\u0440\u0438\u0432\u0430\u044F \u0441\u0442\u0440\u0430\u043D\u0438\u0446\u044B, \u0440\u043E\u0431\u043E\u0442 \u0438\u0449\u0435\u0442 \u0441\u0441\u044B\u043B\u043A\u0438 (SRC \u0438 HREF) \u0438 \u0434\u043E\u0431\u0430\u0432\u043B\u044F\u0435\u0442 \u0438\u0445 \u0432 \u0441\u043F\u0438\u0441\u043E\u043A \u0441\u0442\u0440\u0430\u043D\u0438\u0446, \u043F\u043E\u0434\u043B\u0435\u0436\u0430\u0449\u0438\u0445 \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u044E."@ru . . . "Googlebot \u00E4r en s\u00F6krobot som anv\u00E4nds av s\u00F6kmotorn Google. Den h\u00E4mtar dokument fr\u00E5n Internet, fr\u00E4mst World Wide Web, f\u00F6r att bygga upp en databas f\u00F6r Googles s\u00F6kmotor. Googlebot finns i tv\u00E5 olika versioner, Freshbot och Deepbot. Deepbot f\u00F6ljer alla l\u00E4nkar den hittar och laddar ner s\u00E5 mycket den kan till Googles indexerare. En runda f\u00F6r Deepbot tar cirka en m\u00E5nad att slutf\u00F6ra. Freshbot far runt p\u00E5 Internet och letar efter nytt material. Den bes\u00F6ker enskilda webbsidor beroende p\u00E5 hur mycket de \u00E4ndras. Den kan bes\u00F6ka en del webbplatser s\u00E5som utpr\u00E4glade nyhetssajter upp till sju g\u00E5nger om dagen. Alla Googles s\u00F6krobotar har v\u00E4rdmasken crawl-xxx-xxx-xxx-xxx.googlebot.com, d\u00E4r alla xxx representerar det IP-nummer s\u00F6kroboten har."@sv . . . . "Googlebot \u00E9 o rob\u00F4 indexador usado pelo Google para indexar p\u00E1ginas e documentos da web para construir um \u00EDndice de buscas para o Google Search. Existem dois tipos de rastreadores: o Googlebot Desktop e o Googlebot Mobile para dispositivos m\u00F3veis. Ambos simulam a a\u00E7\u00E3o dos usu\u00E1rios nesses dois ambientes."@pt . "Googlebot"@fr . . "7792"^^ . . "Googlebot"@en . "Googlebot"@de . . . . "Les googlebots sont des robots d'indexation utilis\u00E9s par le moteur de recherche Google afin de recenser et indexer les pages web. Depuis 2019, Google offre une version \"evergreen\" de Google Bot bas\u00E9 sur Chromium. Google Bot va donc \u00EAtre mis \u00E0 jour beaucoup plus r\u00E9guli\u00E8rement. La nouvelle version de Google Bot apporte plus d'un millier de nouvelles fonctionnalit\u00E9s."@fr . "\uAD6C\uAE00\uBD07"@ko . . . "Googlebot"@cs . . "Googlebot \u2013 robot internetowy, u\u017Cywany przez Google. Przechodz\u0105c z jednej strony na drug\u0105 za pomoc\u0105 odno\u015Bnik\u00F3w indeksuje je tym samym tworz\u0105c indeks dla wyszukiwarki Google. Je\u015Bli webmaster za\u017Cyczy sobie, aby informacje na jego stronie nie by\u0142y dost\u0119pne dla Googlebota lub innego robota internetowego, mo\u017Ce wykona\u0107 to za pomoc\u0105 pliku robots.txt. Googlebot dzia\u0142a w dw\u00F3ch wersjach; wykonuje tak zwany deep crawl i fresh crawl. Deep crawl czyli pe\u0142zanie g\u0142\u0119bokie ma za zadanie przechodzenie na ka\u017Cdy link jaki znajdzie na \"ogl\u0105danych\" przez siebie stronach i dodanie tylu, ile mo\u017Ce, do indeksu. Ten proces jest ko\u0144czony i powtarzany mniej wi\u0119cej co miesi\u0105c. W trybie Fresh crawl surfuje po sieci i odwiedza strony zmieniaj\u0105ce si\u0119 cz\u0119sto; w zale\u017Cno\u015Bci od tego, jak cz\u0119sto s\u0105 aktualizowane. Operacja ta wykonywana jest w celu od\u015Bwie\u017Cenia cz\u0119sto zmieniaj\u0105cej si\u0119 zawarto\u015Bci niekt\u00F3rych stron. Googlebot przegl\u0105da strony korzystaj\u0105c ze wszystkich link\u00F3w, jakie znajdzie na ka\u017Cdej wcze\u015Bniej odwiedzonej przez siebie stronie. Nowe strony musz\u0105 by\u0107 zalinkowane od innej, znanej ju\u017C Googlebotowi strony, aby m\u00F3g\u0142 j\u0105 odwiedzi\u0107 i zaindeksowa\u0107. Problem, z kt\u00F3rym cz\u0119sto borykaj\u0105 si\u0119 webmasterzy, to du\u017Ce zu\u017Cycie transferu przez Googlebota. Mo\u017Ce on powodowa\u0107, \u017Ce strony wykorzystaj\u0105 sw\u00F3j limit transferu i zostan\u0105 na pewien czas zawieszone. To problem dotycz\u0105cy szczeg\u00F3lnie stron mirroruj\u0105cych, kt\u00F3re przechowuj\u0105 gigabajty danych. Google umo\u017Cliwia dost\u0119p do \"Webmaster Tools\", kt\u00F3ra umo\u017Cliwia posiadaczom stron dopasowanie \"nat\u0119\u017Cenia\" odwiedzin Googlebota na stronie."@pl . . . . "\u062C\u0648\u062C\u0644 \u0628\u0648\u062A \u0623\u0648 \u063A\u0648\u063A\u0644 \u0628\u0648\u062A (\u0632\u0648\u0627\u062D\u0641 \u062C\u0648\u062C\u0644 \u0623\u0648 \u0639\u0646\u0627\u0643\u0628 \u062C\u0648\u062C\u0644) (\u0628\u0627\u0644\u0625\u0646\u062C\u0644\u064A\u0632\u064A\u0629: Googlebot) \u0647\u0648 \u0628\u0648\u062A \u0648\u0638\u064A\u0641\u062A\u0647 \u0627\u0644\u062A\u063A\u0644\u063A\u0644 \u062F\u0627\u062E\u0644 \u0645\u0648\u0627\u0642\u0639 \u0627\u0644\u0648\u064A\u0628 \u0648\u0641\u0647\u0631\u0633\u0629 \u0627\u0644\u0643\u0644\u0645\u0627\u062A \u0627\u0644\u0645\u0648\u062C\u0648\u062F\u0629 \u0641\u064A\u0647\u0627\u060C \u0628\u062D\u064A\u062B \u064A\u0638\u0647\u0631 \u0627\u0644\u0645\u0648\u0642\u0639 \u0641\u064A \u0646\u062A\u0627\u0626\u062C \u0645\u062D\u0631\u0643 \u0627\u0644\u0628\u062D\u062B \u062C\u0648\u062C\u0644 \u0625\u0630\u0627 \u062A\u0648\u0641\u0631\u062A \u0641\u064A\u0647 \u0627\u0644\u0643\u0644\u0645\u0629 \u0627\u0644\u0645\u0628\u062D\u0648\u062B \u0639\u0646\u0647\u0627. \u0623\u063A\u0644\u0628 \u0645\u0648\u0627\u0642\u0639 \u0645\u062D\u0631\u0643\u0627\u062A \u0627\u0644\u0628\u062D\u062B \u062A\u0642\u0648\u0645 \u0628\u0648\u0627\u0633\u0637\u0629 \u0628\u0631\u0646\u0627\u0645\u062C \u064A\u0633\u0645\u0649 \u00ABweb spider\u00BB \u0628\u062A\u062C\u0645\u064A\u0639 \u0642\u0648\u0627\u0626\u0645 \u0628\u0627\u0644\u0643\u0644\u0645\u0627\u062A \u0627\u0644\u0645\u0648\u062C\u0648\u062F\u0629 \u0639\u0644\u0649 \u0627\u0644\u0634\u0628\u0643\u0629 \u0648\u062A\u0633\u0645\u0649 \u0647\u0630\u0647 \u0627\u0644\u0639\u0645\u0644\u064A\u0629 \u00ABweb crawling\u00BB\u060C \u0648\u0628\u0630\u0644\u0643 \u0641\u0639\u0646\u062F\u0645\u0627 \u062A\u0628\u062D\u062B \u0639\u0646 \u0643\u0644\u0645\u0629 \u0645\u0639\u064A\u0646\u0629 \u0641\u0623\u0646\u0647\u0627 \u062A\u0628\u062D\u062B \u0639\u0646\u0647\u0627 \u0641\u064A \u062A\u0644\u0643 \u0627\u0644\u0642\u0648\u0627\u0626\u0645 \u0627\u0644\u0645\u0639\u062F\u0629 \u0645\u0646 \u0642\u0628\u0644 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062A. "@ar . . . . . . . . . "Googlebot"@pl . . . . . . . "Googlebot"@ja . . . . . . . . . . . . . . . . . "Googlebot \u2013 robot internetowy, u\u017Cywany przez Google. Przechodz\u0105c z jednej strony na drug\u0105 za pomoc\u0105 odno\u015Bnik\u00F3w indeksuje je tym samym tworz\u0105c indeks dla wyszukiwarki Google. Je\u015Bli webmaster za\u017Cyczy sobie, aby informacje na jego stronie nie by\u0142y dost\u0119pne dla Googlebota lub innego robota internetowego, mo\u017Ce wykona\u0107 to za pomoc\u0105 pliku robots.txt. Googlebot przegl\u0105da strony korzystaj\u0105c ze wszystkich link\u00F3w, jakie znajdzie na ka\u017Cdej wcze\u015Bniej odwiedzonej przez siebie stronie. Nowe strony musz\u0105 by\u0107 zalinkowane od innej, znanej ju\u017C Googlebotowi strony, aby m\u00F3g\u0142 j\u0105 odwiedzi\u0107 i zaindeksowa\u0107."@pl . . . . . . . . . . . . . . . "Googlebot \u2014 \u043F\u043E\u0438\u0441\u043A\u043E\u0432\u044B\u0439 \u0440\u043E\u0431\u043E\u0442 \u043A\u043E\u043C\u043F\u0430\u043D\u0438\u0438 Google, \u0438\u0441\u043F\u043E\u043B\u044C\u0437\u0443\u0435\u043C\u044B\u0439 \u0434\u043B\u044F \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u044F \u043C\u0438\u0440\u043E\u0432\u043E\u0439 \u0441\u0435\u0442\u0438 \u0438 \u043F\u043E\u043F\u043E\u043B\u043D\u0435\u043D\u0438\u044F \u043F\u043E\u0438\u0441\u043A\u043E\u0432\u043E\u0433\u043E \u0438\u043D\u0434\u0435\u043A\u0441\u0430 \u0441\u0438\u0441\u0442\u0435\u043C\u044B Google. \u0420\u043E\u0431\u043E\u0442 \u043D\u0430\u0447\u0438\u043D\u0430\u0435\u0442 \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u0435 \u043A\u0430\u0436\u0434\u043E\u0433\u043E \u043E\u0442\u0434\u0435\u043B\u044C\u043D\u043E\u0433\u043E \u0441\u0430\u0439\u0442\u0430 \u0441 \u043F\u043E\u043B\u0443\u0447\u0435\u043D\u0438\u044F \u0441\u043F\u0438\u0441\u043A\u0430 URL-\u0430\u0434\u0440\u0435\u0441\u043E\u0432 \u0432\u0435\u0431-\u0441\u0442\u0440\u0430\u043D\u0438\u0446, \u0441\u043E\u0437\u0434\u0430\u043D\u043D\u043E\u0433\u043E \u043F\u043E \u0440\u0435\u0437\u0443\u043B\u044C\u0442\u0430\u0442\u0430\u043C \u043F\u0440\u0435\u0434\u044B\u0434\u0443\u0449\u0438\u0445 \u0441\u0435\u0430\u043D\u0441\u043E\u0432, \u0438 \u043F\u043E\u043F\u043E\u043B\u043D\u044F\u0435\u0442 \u0435\u0433\u043E \u0434\u0430\u043D\u043D\u044B\u043C\u0438 \u0438\u0437 \u0444\u0430\u0439\u043B\u043E\u0432 Sitemap; \u043F\u0440\u043E\u0441\u043C\u0430\u0442\u0440\u0438\u0432\u0430\u044F \u0441\u0442\u0440\u0430\u043D\u0438\u0446\u044B, \u0440\u043E\u0431\u043E\u0442 \u0438\u0449\u0435\u0442 \u0441\u0441\u044B\u043B\u043A\u0438 (SRC \u0438 HREF) \u0438 \u0434\u043E\u0431\u0430\u0432\u043B\u044F\u0435\u0442 \u0438\u0445 \u0432 \u0441\u043F\u0438\u0441\u043E\u043A \u0441\u0442\u0440\u0430\u043D\u0438\u0446, \u043F\u043E\u0434\u043B\u0435\u0436\u0430\u0449\u0438\u0445 \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u044E. \u0412\u0435\u0431-\u043C\u0430\u0441\u0442\u0435\u0440\u0430 \u043C\u043E\u0433\u0443\u0442 \u0443\u043F\u0440\u0430\u0432\u043B\u044F\u0442\u044C \u0440\u043E\u0431\u043E\u0442\u043E\u043C \u0441 \u043F\u043E\u043C\u043E\u0449\u044C\u044E \u043A\u043E\u043C\u0430\u043D\u0434\u044B \u0432 \u0444\u0430\u0439\u043B\u0435 robots.txt \u0438\u043B\u0438 \u043C\u0435\u0442\u0430-\u0442\u0435\u0433\u0430 ; Google \u0432 \u0441\u043F\u0438\u0441\u043A\u0435 \u0441\u0432\u043E\u0438\u0445 \u0438\u043D\u0441\u0442\u0440\u0443\u043C\u0435\u043D\u0442\u043E\u0432 \u0434\u043B\u044F \u0432\u0435\u0431-\u043C\u0430\u0441\u0442\u0435\u0440\u043E\u0432 \u0434\u0430\u0451\u0442 \u0432\u043E\u0437\u043C\u043E\u0436\u043D\u043E\u0441\u0442\u044C \u043F\u0440\u043E\u0432\u0435\u0440\u0438\u0442\u044C \u00AB\u0432\u0438\u0434\u0435\u043D\u0438\u0435\u00BB \u0441\u0430\u0439\u0442\u0430 \u0440\u043E\u0431\u043E\u0442\u043E\u043C. IP-\u0430\u0434\u0440\u0435\u0441\u0430 \u0431\u043E\u0442\u0430 \u0432\u0440\u0435\u043C\u044F \u043E\u0442 \u0432\u0440\u0435\u043C\u0435\u043D\u0438 \u043C\u0435\u043D\u044F\u044E\u0442\u0441\u044F.\u041F\u0440\u0438 \u0441\u043A\u0430\u043D\u0438\u0440\u043E\u0432\u0430\u043D\u0438\u0438 Googlebot \u0443\u043A\u0430\u0437\u044B\u0432\u0430\u0435\u0442 user-agent: \n* Googlebot/2.1 (+http://www.googlebot.com/bot.html) \n* Googlebot/2.1 (+http://www.google.com/bot.html) \n* Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) \n* Googlebot-Image/1.0 (\u043F\u0430\u0443\u043A, \u0441\u043A\u0430\u043D\u0438\u0440\u0443\u044E\u0449\u0438\u0439 \u0438\u0437\u043E\u0431\u0440\u0430\u0436\u0435\u043D\u0438\u044F)"@ru . "Googlebot"@sv . . . "Googlebot"@en . "Googlebot \u0446\u0435 \u043F\u043E\u0448\u0443\u043A\u043E\u0432\u0438\u0439 \u0440\u043E\u0431\u043E\u0442, \u0441\u0442\u0432\u043E\u0440\u0435\u043D\u0438\u0439 Google, \u044F\u043A\u0438\u0439 \u0437\u0431\u0438\u0440\u0430\u0454 \u0434\u043E\u043A\u0443\u043C\u0435\u043D\u0442\u0438 \u0437 \u0456\u043D\u0442\u0435\u0440\u043D\u0435\u0442\u0443 \u0434\u043B\u044F \u0431\u0443\u0434\u0456\u0432\u043D\u0438\u0446\u0442\u0432\u0430 \u043F\u043E\u0448\u0443\u043A\u043E\u0432\u043E\u0433\u043E \u0456\u043D\u0434\u0435\u043A\u0441\u0443 \u043F\u043E\u0448\u0443\u043A\u043E\u0432\u0438\u043A\u0430 Google. \u042F\u043A\u0449\u043E \u0432\u0435\u0431\u043C\u0430\u0439\u0441\u0442\u0435\u0440 \u0431\u0430\u0436\u0430\u0454 \u043E\u0431\u043C\u0435\u0436\u0438\u0442\u0438 \u0456\u043D\u0444\u043E\u0440\u043C\u0430\u0446\u0456\u044E \u043D\u0430 \u0441\u0432\u043E\u0454\u043C\u0443 \u0441\u0430\u0439\u0442\u0456 \u0434\u043B\u044F Googlebot, \u0432\u0456\u043D \u043C\u043E\u0436\u0435 \u0432\u0438\u043A\u043E\u0440\u0438\u0441\u0442\u0430\u0442\u0438 \u0444\u0430\u0439\u043B robots.txt \u0430\u0431\u043E \u043C\u0435\u0442\u0430-\u0442\u0435\u0433 noindex. Googlebot \u0456\u0434\u0435\u043D\u0442\u0438\u0444\u0456\u043A\u0443\u0454 \u0441\u0435\u0431\u0435 \u0447\u0435\u0440\u0435\u0437 \u0441\u0442\u0440\u0456\u0447\u043A\u0443 user agent, \u0432 \u044F\u043A\u0456\u0439 \u043C\u0456\u0441\u0442\u0438\u0442\u044C\u0441\u044F \u0441\u043B\u043E\u0432\u043E googlebot."@uk . . . . . "\u062C\u0648\u062C\u0644 \u0628\u0648\u062A"@ar . . "GoogleBot es el robot de b\u00FAsqueda usado por la empresa Google que colecciona documentos desde la web con el fin de construir una base de datos para el motor de b\u00FAsqueda Google."@es . "Googlebot"@ru . . . "Googlebot \u00E4r en s\u00F6krobot som anv\u00E4nds av s\u00F6kmotorn Google. Den h\u00E4mtar dokument fr\u00E5n Internet, fr\u00E4mst World Wide Web, f\u00F6r att bygga upp en databas f\u00F6r Googles s\u00F6kmotor. Googlebot finns i tv\u00E5 olika versioner, Freshbot och Deepbot. Deepbot f\u00F6ljer alla l\u00E4nkar den hittar och laddar ner s\u00E5 mycket den kan till Googles indexerare. En runda f\u00F6r Deepbot tar cirka en m\u00E5nad att slutf\u00F6ra. Freshbot far runt p\u00E5 Internet och letar efter nytt material. Den bes\u00F6ker enskilda webbsidor beroende p\u00E5 hur mycket de \u00E4ndras. Den kan bes\u00F6ka en del webbplatser s\u00E5som utpr\u00E4glade nyhetssajter upp till sju g\u00E5nger om dagen. Alla Googles s\u00F6krobotar har v\u00E4rdmasken crawl-xxx-xxx-xxx-xxx.googlebot.com, d\u00E4r alla xxx representerar det IP-nummer s\u00F6kroboten har. Googlebot uppt\u00E4cker webbsidor genom att sk\u00F6rda alla l\u00E4nkar p\u00E5 alla sidor den hittar. Sedan f\u00F6ljer den l\u00E4nkarna vidare till andra sidor. Nya webbsidor m\u00E5ste vara l\u00E4nkade fr\u00E5n en som redan \u00E4r k\u00E4nd av Googlebot. Googlebot \u00E4r ett problem f\u00F6r en del webbmasters som hyr webbutrymme och har en gr\u00E4ns p\u00E5 hur mycket trafik de f\u00E5r utnyttja. M\u00E5nga webbmasters har klagat p\u00E5 att Googlebot anv\u00E4nder f\u00F6r mycket av deras bandbredd och att deras webbplatser tempor\u00E4rt tagits ned f\u00F6r att de har g\u00E5tt \u00F6ver bandbreddsgr\u00E4nsen. Detta \u00E4r ett stort problem f\u00F6r alla webbspeglar (website mirrors) som ofta lagrar flera gigabyte data. Om en webbmaster registrerar sin webbplats hos , f\u00E5r Googlebot en liten ledtr\u00E5d om vilken sida den ska s\u00F6ka igenom. Man kan \u00E4ven konfigurera att boten ska begr\u00E4nsa sina genoms\u00F6kningar av en viss webbsida/webbplats. Det kan g\u00F6ras genom att man registrerar ett Googlekonto."@sv . "Googlebot is the web crawler software used by Google, which collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler (to simulate desktop users) and a mobile crawler (to simulate a mobile user)."@en . "Les googlebots sont des robots d'indexation utilis\u00E9s par le moteur de recherche Google afin de recenser et indexer les pages web. Depuis 2019, Google offre une version \"evergreen\" de Google Bot bas\u00E9 sur Chromium. Google Bot va donc \u00EAtre mis \u00E0 jour beaucoup plus r\u00E9guli\u00E8rement. La nouvelle version de Google Bot apporte plus d'un millier de nouvelles fonctionnalit\u00E9s."@fr . . . "Googlebot\uFF08\u30B0\u30FC\u30B0\u30EB\u30DC\u30C3\u30C8\uFF09\u306F\u3001Google\u304C\u63A1\u7528\u3057\u3066\u3044\u308B\u30AF\u30ED\u30FC\u30E9\uFF08Web\u30ED\u30DC\u30C3\u30C8\uFF09\u30BD\u30D5\u30C8\u30A6\u30A7\u30A2\u3067\u3042\u308B\u3002"@ja . . . "GoogleBot es el robot de b\u00FAsqueda usado por la empresa Google que colecciona documentos desde la web con el fin de construir una base de datos para el motor de b\u00FAsqueda Google."@es . . . . . "Googlebot \u00E8 lo spider web (italianizzato in \"agente mobile\") di Google. Googlebot effettua due tipi di scansione del web: la Deep-crawl (fatta una volta al mese circa) e la Fresh-crawl (quasi tutti i giorni).La prima come detto viene fatta una volta al mese e viene scansionato tutto il web pagina per pagina, aggiornando indici, pagerank e cache. Dopo una Deep-crawl Google impiega circa 6-8 giorni per aggiornare completamente i suoi indici e propagarli in tutti i datacenter. In questo periodo di tempo si parla della cosiddetta \"Google dance\" perch\u00E9 i risultati che escono possono essere diversi di volta in volta. Dopo qualche giorno invece si stabilizzano. La seconda invece viene fatta quasi tutti i giorni e in pratica aggiorna le pagine che gi\u00E0 sono presenti nell'indice e aggiunge quelle eventualmente create dopo l'ultima Deep-crawl. Esistono altri strumenti e procedure che Googlebot utilizza per indirizzare (e limitare) le sue scansioni. Uno tra i pi\u00F9 importanti \u00E8 il file \"robots.txt\", che dovrebbe essere sempre presente sui web server attestati su internet, in quanto naturale oggetto di scansione da parte degli spider."@it . . . . . "Googlebot"@en . "Googlebot ist ein Webcrawler des US-amerikanischen Unternehmens Google LLC. Das Computerprogramm l\u00E4dt World-Wide-Web-Inhalte eigenst\u00E4ndig herunter und f\u00FChrt diese der herstellereigenen Suchmaschine Google zu."@de . "969533295"^^ . "Google 2015 logo.svg"@en . . . . "Googlebot"@es . . . . . . . "Googlebot \u00E8 lo spider web (italianizzato in \"agente mobile\") di Google. Googlebot effettua due tipi di scansione del web: la Deep-crawl (fatta una volta al mese circa) e la Fresh-crawl (quasi tutti i giorni).La prima come detto viene fatta una volta al mese e viene scansionato tutto il web pagina per pagina, aggiornando indici, pagerank e cache. La seconda invece viene fatta quasi tutti i giorni e in pratica aggiorna le pagine che gi\u00E0 sono presenti nell'indice e aggiunge quelle eventualmente create dopo l'ultima Deep-crawl."@it . . . . .