Optically trapped nanoparticles can be used as efficient mobile probes of nanoscopic forces and temperatures. However, it is crucial that the trapped particle has minimal influence on the system under study while providing a strong enough optical response to actually allow for optical manipulation. This puts severe constraints on the particle size and thermal properties. In particular, strong optical responses associated with plasmon resonances in noble metal nanoparticles and Mie resonances in high index dielectric particles can significantly affect trap stability through enhanced radiation pressure forces and photoinduced heating. Using Mie theory and hot Brownian motion analysis, we calculate trap stability and photothermal properties for nanospheres composed of the best (Ag) and most widely used (Au) plasmonic materials as well as for crystalline and amorphous Si, the prototypic high-index dielectric, using polystyrene as a low-index reference material. We calculate trap stability properties for optical tweezers based on high numerical aperture optics (NA = 1.2) for three of the most widely used laser wavelengths (532, 785, and 1064 nm) and for the case of trapping in water. The results reveal the specific particle size ranges for which optical tweezing is possible in two and three dimensions and indicate that crystalline Si nanoparticles trapped using near-infrared laser beams are the optimal choice for temperature-sensitive optical manipulation applications with small particles.