The atomic structure of materials can be characterized by transmission electron microscopy (TEM) and scanning TEM (STEM). However, determining the positions of atoms in three dimensions from two-dimensional images is non-trivial. In this work, we use atomistic modeling with first principles density functional theory (DFT) or empirical potentials, in conjunction with machine learning, to tackle the S/TEM image inversion problem. We discuss the use of single and multi-objective evolutionary and basin-hopping approaches for S/TEM-guided atomistic structure determination, incorporating comparison of simulated and experimental S/TEM images using computer vision approaches. We show that the combined use of energetic and experimental information is effective in arriving at physical solutions.