What is robots.txt?

It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage.