Determine what pages on your site might be seen as authoritative by search engines by computing your Internal PageRank.
How to crawl a website using Rcrawler package
By: François JOLY
Perfect to crawl a few of URLs, easy to use css style or XPath selectors
by Hadley Wickham
Good to crawl a website and extract metadata
By Salim Khalil