The performance scaling trends associated with Moore's Law have motivated device feature size scaling and development of conventional complementary metal-oxide-semiconductor (CMOS) technology over the past several decades. However in recent years the end of Dennard scaling and the Dark Silicon problem, [1, 2] as well as therapid growth of machine learning (ML) and Internet of Things (IoT) are challenging the capabilities of both existing technologies and conventional von Neumann computing paradigms and architectures. On the one hand, the increasing subthreshold leakage current of CMOS as the device size reduces to sub-μm scale becomes a major concern in integrated circuit (IC) designs, resulting in significant power consumption; on the other hand, in the era of Artificial Intelligent (AI), ML, and the IoT, many applications and systems place new demands on computing paradigms and architectures for massive data transfer, extremely low power consumption, high-speed performance and nonvolatility in energy constrained environments. Conventional von Neumann architectures alone are insufficient to support future applications and systems, thus revolutionary advancements exploiting new computing paradigms and architectures, as well as emerging technologies are needed. This dissertation studies novel circuit and architecture designs by exploiting the unique characteristics of emerging devices. A number of emerging device have been explored to develop novel circuit designs that can address the processor-memory bottleneck caused by massive data transfer and achieve better energy efficiency and performance than conventional CMOS designs. Among the emerging devices being studied, ferroelectric field-effect transistors (FeFETs) stand out with unique hysteretic Ids-Vgs characteristics. We proposed novel logic-in-memory (LiM) circuits to address the processor-memory bottleneck in different computing platforms by exploiting the hysteretic characteristic of FeFETs, which allows the devices to function as both switches and a non-volatile (NV) storage element. Moreover, FeFETs offer advantages over other emerging technologies, including its three-terminal structure, voltage-driven write mechanism, high ON current, high on-off ratios, etc. Our proposed circuits are evaluated to be superior to equivalent LiM approaches based on CMOS and other emerging technologies in terms of area, energy and performance. Another class of emerging devices, i.e., symmetric FET (SymFET), exhibit bell-shaped I-V characteristics. By exploiting this unique characteristic, we proposed SymFET-based sequential circuits that are more compact and energy efficient than conventional equivalents. We also proposed a compact SymFET-based voltage protector that improves the hardware security aspect of protected circuits against the side channel attack. Our work paved the way to building pure emerging device based systems and emerging technology based hardware security primitives.This dissertation also considers novel hardware design based on emerging computational paradigms for hard optimization problems in the context of ML and IoT applications. Boolean Satisfiability (SAT), the first problem proven to be NP-complete, is intractable on digital computers based on the von Neumann architecture. An efficient SAT solver can benefit many applications such as artificial intelligence, electronic design automation, as well as scheduling and functional verification. We propose a novel analog hardware SAT solver, AC-SAT, implementing an analog computing algorithm called continuous-time dynamical system (CTDS). This algorithm shows polynomial analog time-complexity on the hardest k-SAT (k≥3) problem instances, but at an energy cost dependent on exponentially growing auxiliary variables. AC-SAT is intended to be used as a co-processor and is programmable for handling different problem specifications. It is especially effective for solving the hard k-SAT problem instances that are challenging for algorithms running on digital machines. Furthermore, with its modular design, AC-SAT can be readily extended to solve larger size problems. SPICE simulation results show that AC-SAT can indeed solve the SAT problems, and it has speedup factors of ∼10^4 on even the hardest 3-SAT problems, when compared with a state-of-the-art SAT solver on digital computers.