Do not test each and every file in ABI dirs

Once we know that the directory fulfills the search conditions, stop
running file on the files. Saves about 30 seconds on a full configure
run for me.