Optical Neural Networks: The 3D connection

We motivate a new canonical strategy for integrating photonic neural networks (NNs) by leveraging 3D printing. Our believe is that a NN's parallel and dense connectivity is not scalable without 3D integration. 3D additive fabrication complemented with photonic signal transduction can dramatically augment the current capabilities of 2D CMOS and integrated photonics. Here we review some of our recent advances made towards such a breakthrough architecture.


I. INTRODUCTION
Several decades passed between the introduction and the large-scale exploration of neural networks (NN). Since the proposal of simple NNs in 1943 1 , the field has gone through multiple cycles of euphoria and challenges until reaching todays large-scale interest and exploitation 2 . Readily available high-performance computing systems now allow emulating powerful (deep) NN architectures whose connections are optimized based on computationally expensive learning concepts such as gradient back-propagation. As a consequence, NNs currently excel on previously unseen scales, but at the same time the constraints of todays CMOS-based computing threatens to limit the reach of this revolution.
As illustrated by their name, the initial objective of NN, cf. Fig. 1(A), was providing a logical calculus of the ideas immanent in nervous activity 1 , and as such their composition mirrors a most rudimentary aspect of the mammalian neo-cortex: nodes are densely linked into a network with connections much like synapses, dendrites and axons connecting biological neurons. However, this is only possible in the context of a global structural property of the neocortex in which neurons, and even more so connections, are distributed 2 Optical Neural Networks: The 3D connection across a 3D volume, cf. Fig. 1(B). The majority of cortical neurons are arranged in planes located inside the grey matter that wraps around the brain, and stacks of neurons form short-range connections (labelled a in Fig. 1(B)) which travers the grey matters volume.
Crucially, grey matter encloses white matter, and inside this volume the brains long-range (labelled b and c in Fig. 1(B)) connections are located. 3D connections are therefore a canonical feature of brain architecture. The scale and connectivity of the human brains network would otherwise simply not fit inside the human skull. The brain therefore provides a very good primer for exploiting 3D circuit topology. Even though the 3D topology of brains emerged from evolutionary development, science and engineering can deliberately combine advantageous strategies and concepts. Combining the 3D network topology of biological brains with photonic signal transduction is a highly appealing strategy for next generation NN computing. In this paper, we elaborate the potential of 3D printing technology for integrated photonic NN chips. Such additive fabrication enables true 3D integration and naturally complements the mostly 2D lithography that struggles to implement parallel NN connections with a scalable strategy. Photonics offers fundamental energy, speed and latency advantages when establishing the communication between NN neurons along the staggering amount of network connections. 3D printing is a potential path for 3D integration of optically interconnected Si or other electro-optic chips.

II. CANONICAL 3D PHOTONIC NEURAL NETWORK ARCHITECTURE
Physically realizing dense connections for the large number of neurons (typically ¿1000 units) contained in each NN layer results in a formidable challenge. A parallel NN processor needs to provide a dedicated physical link for each connection, which is difficult since the amount of possible connections scales quadratically with the number of neurons. A connections defining property is its strength, and its physical implementation for example by memristors, micro-rings or holographic memory always occupies some basic unit of area i.e. volume. Integration in 2D results in a quadratic scaling of the circuits area with a networks size 3 , cf. Fig. 2(A). In a 3D implementation weights can be stacked, for example, in planes, and for the simplest organization 3 , both, the number of required planes and memory-elements per plane scale linearly with the number of neurons. This mitigates the size-scalability roadblock and 3D routing may well be a fundamental prerequisite for scalable 3 Optical Neural Networks: The 3D connection and parallel NN chips. Realizing such 3D circuits electronically is challenging due to the capacitive coupling and the associated energy dissipation when sending information along signalling wires.
In order to overcome these challenges we investigated a new, canonical photonic NN architecture where neurons in the form of nonlinear components are arranged in 2D sheets, while connections are integrated in 3D printed photonic circuits, cf. Fig. 2(B). We do not constrain the nature of photonic neurons or the 3D routing strategy. All-optical as well as electro-optical components acting as neurons are possible, and the 3D photonic interconnect can be realized by refractive index modifications in a 3D medium, multiple stacks of diffractive-optics planes 4 as well as complex 3D circuitry of photonic waveguides 3 .

III. 3D NANO-PRINTING TECHNOLOGY
Additive manufacturing (AM) has been a popular method for prototyping ever since it was developed in the 1980s as it does not require special tooling or molds. However, its true advantage over most conventional manufacturing methods is AMs ability to produce 3D parts of great complexity, which is unfeasible or even impossible with subtractive or 2D lithographic methods. Among various AM techniques, two-photon polymerization (TPP) is 4 Optical Neural Networks: The 3D connection Different connection topologies were demonstrated. Arranging 1 to 81 splitters in an 15x15 input waveguide array, c.f. Fig. 4(A), demonstrated a 3D printed dense interconnect for 225 neurons in an area of only 300x300 m2. Inspired by convolutional NNs, the same authors realized Boolean Haar filters arranged in a 7x7 array, see Fig. 4(B). Such arrays can filter images containing 21x21 pixels in parallel, which in principle is sufficient for realizing a convolutional layer applied to the MNIST handwritten digit dataset. Most importantly the area of both 3D interconnects scales linearly with the number of inputs.

V. GRADIENT INDEX CONTINUOUS INTERCONNECTS
Multilayered diffractive optical elements, c.f. Fig. 5, can also perform interconnection tasks utilizing the 3D via optical volume elements (OVEs). OVEs can be designed by utilizing a nonlinear optimization scheme, learning tomography (LT), which calculates the topography of either multilayered or GRIN volume elements to approximate desired mappings. Figure 5(A,B) shows an demonstration by Dinc et al., which acts as an angular multiplexer 7 Optical Neural Networks: The 3D connection (lantern) that maps plane waves with different incidence angles to linearly polarized multimode fiber modes 4 . It provides an interconnect between single mode fibers stacked with different angles and a multimode fiber to map each single mode fiber input/output to a specific mode of multimode fiber, hence performs mode-division multiplexing. Another example of LT computed OVEs realizing Haar filters such as demonstrated in 3 are shown in Fig. 5(C,D).

VI. POSSIBILITIES FOR PHOTONIC NEURONS
The function of a NN neuron is the summation of its inputs followed by a nonlinear transformation. Summation of the individual fields impinging on a neuron can be realized in photonics by the superposition of optical fields. Unfortunately, nonlinearity is since many years the Achilles-heel of photonics compared to electronics. However, modern photonic devices have significantly lowered the energy consumption which can now be below 100 fJ per nonlinear transformation 8 . Many standard nonlinear photonic components have potentially high modulation bandwidths, fast response times and can directly be interfaced with fully parallel as well as dense 3D photonic interconnect. Photonic neurons combined with our 2D/3D canonical NN architecture therefore offer new concepts for addressing the long-standing challenges of parallelism and connection density for high-speed NN computers.
In order to make most efficient use of the footprint and circuit volume, photonic neurons need to be arranged in a 2D array. Furthermore, neurons that accept multi-mode fields as their input could potentially be beneficial as this relaxes design constraints and allows for high-density integration of 3D photonic waveguides without a cladding. Finally, any optical transformation is associated with losses and the 3D photonic interconnect is no exception; neurons including optical amplification would mitigate such losses. At this stage, we can imagine all-optical, electro-optical as well as plasmonic neurons, and the most promising concept will certainly have to strike a balance between speed, efficiency, flexibility and potentially amplification.
8 Optical Neural Networks: The 3D connection

VII. OUTLOOK
The viability of integrating photonic circuits suited for NN interconnects in 3D has recently been demonstrated in principle 3,4 . Ultimately, scalability is key for computing hardware, which implies that stacking 2D neurons and 3D interconnects into deep photonic NNs requires optical losses to be counterbalanced by amplification without resulting in an unsustainable thermal energy deposition inside the integrated photonic circuit. However, the computational power of a NN relies on more than simply establishing specific connections in parallel. The nonlinearity of its neurons is a fundamental requirement for solving complex tasks, and here significant room for improvement exists. Another defining feature of NN is the optimization of their connections during training. New, ideally in-situ optimization strategies are in urgent demand. In combination with plasticity such as non-volatile memristive effects, these concepts would significantly reduce the complexity of potential auxiliary support circuits as well as of the 3D interconnect itself.