Here’s a way to have little fun with our robot friends.

Edit the wp-includes/functions.php for the following function:

function do_robots() {

    $rss_bot_ip_list = array('74.125.44.136', '74.125.158', '61.4.177.'); 
    //IP for feedburner and feedsky
    $robots_allow = 0;

    for ($j = 0; $j < count($rss_bot_ip_list); $j++) {
        if (is_numeric(strpos($_SERVER["REMOTE_ADDR"], $rss_bot_ip_list[$j]))) {
            $robots_allow = 1;
        }
    }


    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    if ( '0' == $robots_allow ) {
        echo "User-agent: *\n";
        echo "Disallow: /\n";
    } else {
        echo "User-agent: FeedBurner/1.0 (http://www.FeedBurner.com)\n";
        echo "Allow /feed\n";
        echo "Disallow: /\n";
        echo "\n";
        echo "User-agent: Mozilla 5.0 (compatible; Feedsky\n";
        echo "Allow /feed\n";
        echo "Disallow: /\n";
        echo "\n";
        echo "User-agent: *\n";
    }
}

This allows customized robots.txt for feed crawlers exclusively.