.htaccess is a per-directory configuration file used on the Linux Apache server. It has an extensive number of ways it can be used, but for SEO, it is often used for redirections or making search-friendly URL changes. The effects of an .htaccess change are applied before the user or search engine fetches the page, so everything you do there will affect the pages that the search engines see. This is the power for SEO. One important way .htaccess is used is to reduce the complexity of URLs that are being created run-time by a database. Another reason to use it is to forward folders or directories to a new site design if your file system structure has changed. SEOs can use the .htaccess file in such a way to sculpt page rank, by guiding Google in what parts of the site should be considered in relevancy algorithms versus less useful pages that can dilute the effect.

Robots.txt is a file which is placed on a website to guide indexing algorithms (crawlers.) Its syntax is defined as a standard (Robots Exclusion Standard) and pretty much every search engine crawler obeys it. The file may be used to entirely certain crawlers from seeing all or part of the site, making it a useful tool for keeping certain parts of your site out of the search engines that could be considered duplicate content or low-value content.

A third file to consider is the sitemaps file. This XML (Extended Markup Language) file complements the robots standard, but is not as well supported/standardized.