Robots.txt Files Need to be Smaller than 500 KB, says Google

According to John Mueller from Google, the leading search engine can only process up to 500 KB of
the robots.txt file. This is so because having a heavy robots.txt file
can confuse GoogleBot and can cause serious problems with your
website’s search engine optimisation (SEO) performance in the Google search results.