Mathematical formulas in vector calculus often have direct visual representations, which in form of illustrations are used extensively during teaching and when assessing students’ levels of understanding. However, there is very little, if any, empirical evidence of how the illustrations are utilized during problem solving and whether they are beneficial to comprehension. In this paper we collect eye movements and performance scores (true or false answers) from students while solving eight problems in vector calculus; 20 students solve illustrated problems whereas 16 students solve the same problems, but without the illustrations. Results show no overall performance benefit for illustrated problems even though they are clearly visually attended. Surprisingly, we found a significant effect of whether the answer to the problem was true of false; students were more likely to answer that the question was true given an illustrated problem. We interpret this finding as if the illustrations persuade the students that the answer is true, irrespective of whether or not it in fact is. These results may question the tacit consensus among teachers of vector calculus that illustrations are generally beneficial for comprehending a problem.