Dignity nobility high-quality point out highness standing course caste position superiority precedence trend distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robotic concerning which internet pages usually are not to generally be crawled. Like a online search engine https://www.youtube.com/watch?v=imiGL2YS2uo