Proteins are often engineered for greater affinity and/or specificity to achieve therapeutic benefit. For example, as antibodies and T cell receptors (TCRs) have emerged as a new class of therapeutics, phage and yeast display libraries have been used to affinity mature these molecules for specific targets. Although effective, these techniques offer limited structural control of the engineered sites under given selection pressures, and are often termed "irrational design." In contrast, in silico methods use structure-based physical models to estimate interaction energies, allowing for "rational" design and more targeted interactions. The role of the TCR in identifying immunological targets and signaling appropriate responses has resulted in exciting translational opportunities. Yet TCRs mediate one of the most complex protein-protein interactions in biology, with intricate biochemical/biomolecular mechanisms intersecting structure, affinity, and specificity. Here, we address these hallmarks of TCR recognition, emphasizing the use of computational models and structure guided design to examine the fine details of TCR interfaces. With broad implications to T cell based immunotherapies, we highlight how the 'imperfect' interfaces of TCRs bound to their ligands influences aspects of TCR design, T cell specificity, and epitope recognition of the immune system.