Tests that fail intermittently are the biggest obstacle to an automated testing process. As we deal with them all the time, we even have a nickname for them. We call them 'flaky' tests.
The problem with having flaky tests is that they remove the element of reproducibility from our testing process. If our tests pass, we don't know whether all the bugs we're testing against have actually been fixed, or whether they just happened not to manifest themselves in that testing run. If our tests fail, this could be due to either production errors or testing errors, or both.
If we can't be sure of our test results, we can't be sure of our software.
Furthermore, even if we are fairly confident that the "flakes" are due to testing errors ...