Robots.txtÊÇ·ÅÒ»Ð©ÍøÕ¾Éϲ»ÏëÈÃËÑË÷ÒýÇæ×¥È¡ÊÕ¼µÄÎļþ£¬ÐèÒªÎÒÃÇ×Ô¼ºÖªµÀÍøÕ¾ÄÄЩÎļþÊDz»Ïë±»ÊÕ¼µÄ£¬±ÈÈçÒ»°ãÍøÕ¾µÄºǫ́¾ÍûÓбØÒªÊÕ¼ÒÔ¼°»áÔ±ÐÅÏ¢Ò³ÃæµÈ¡£¶«Ý¸ÍøÕ¾ÓÅ»¯-ÐÂÆðµãseoͨ¹ýÒÔÏÂÎåµãÀ´ºÍ´ó¼Ò·ÖÏíÊǵÄRobotsËÑË÷ÒýÇæÐÒéд·¨ºÍ×¢ÒâÊÂÏî¡£
Ò»£¬Ê²Ã´ÊÇRobotsÐÒ飿
1>ÍøÕ¾ºÍËÑË÷ÒýÇæµÄÐÒéÎļþ
2>Ö©ÖëÅÀÐÐÍøÕ¾µÄµÚÒ»¸öÒª·ÃÎʵÄÎļþ
¶þ.RobotsÓÐʲô×÷Óã¿
1>ÆÁ±ÎÍøÕ¾ÄÚµÄËÀÁ´½Ó¡£
3>ÆÁ±ÎËÑË÷ÒýÇæÖ©ÖëץȡվµãÄÚÖØ¸´ÄÚÈݺÍÒ³Ãæ¡£
3>×èÖ¹ËÑË÷ÒýÇæË÷ÒýÍøÕ¾Òþ˽ÐÔµÄÄÚÈÝ
Èý.RobotsÓï·¨:
1>User-agent¶¨ÒåËÑË÷ÒýÇæÀàÐÍ:
(1)°Ù¶ÈÖ©Ö룺Baiduspider
(2)¹È¸èÖ©Ö룺Googelebot
(3)360Ö©Ö룺360Spider
(4)SosoÖ©Ö룺Sosospider
(5)ÓеÀÖ©Ö룺YoudaoBot
(6)Ëѹ·Ö©Ö룺SohpiNews Spider µÈ
2>Disallow½ûֹץȡÊÕ¼µØÖ·£º
(1)½ûÖ¹ÕûÕ¾£ºÐ´·¨ User-agent: Baiduspider Disallow: /
(2)½ûÖ¹Ò»¸öÎļþ¼Ð:д·¨ User-agent: Baiduspider Disallow: /data
(3)½ûÖ¹Îļþ:д·¨ User-agent: Baiduspider Disallow: /jianzhan/list_1.html
(4)½ûÖ¹¶¯Ì¬Â·¾¶: ±¾À´ÊǶ¯Ì¬Â·¾¶£¬ºóÃæÓÖÐ޸ijɾ²Ì¬Â·¾¶£¬µ¼ÖÂÁ½¸ö²»Í¬Â·¾¶Í¬Ê±ÄÜ·ÃÎÊÒ»¸öÒ³Ãæ£¬ÎÒÃÇ¿ÉÒ԰Ѷ¯Ì¬Â·¾¶½ûÖ¹,¶¯Ì¬Á´½ÓµÄÒ»¸ö±ê×¼·ûºÅ¾ÍÊÇ“?”Òò´Ë½û×¥¶¯Ì¬UrlÃüÁîд×÷£ºDisallow: /*?*
3>Allow¶¨ÒåץȡÊÕ¼µØÖ·:д·¨ Allow: /data/common.inc.php
4>Óï·¨·ûºÏ:
(1)/µ¥¸ö·ûºÏËÑË÷ÒýÇæÈÏΪÊǸùĿ¼:д·¨£ºDisallow: /*.php$
(2)*ÄÜÆ¥Åä0»òÕßËùÓÐÓ¢ÎÄ×Ö·û:×ÔÓÉÖ÷Ìâ
(3)$½áÊø·û
ËÄ.ʲôÇé¿öÏÂÓÃRobots
1>ͳһ·¾¶£¬¶¯Ì¬ºÍ¾²Ì¬ www.wglseo.com/
2>ËÑË÷Á´½Ó
3>±êÇ©Á´½Ó
4>ÖÐÎÄÁ´½Ó
5>¸ù¾ÝÇé¿ö£¬²»ÏëÈÃÍøÕ¾ÊÕ¼µÄÎļþ£¬Òþ˽Îļþ£¬»áÔ±×¢²áµÈ
Îå.ʹÓÃRobots×¢ÒâÊÂÏî
1>·ûºÏʹÓõÄϸ½Ú[/ £º ¿Õ¸ñ ´óСд]
2>ÉúЧʱ¼ä[¼¸Ìì»òÕßÁ½¸öÔÂÒÔÄÚ]
3>²»ÒªËæ±ãÆÁ±ÎÕûÕ¾
ÒÔÉÏÄÚÈÝÓɶ«Ý¸ÍøÕ¾ÓÅ»¯-ÐÂÆðµãseoÕûÀí·¢²¼