The Network Forward Pass

A polynomial-activation MLP with inputs $(x,y,z) \in \mathbb{R}^3$ computes, for each hidden layer $l$ and neuron $j$:

$$h^{(l)}_j = \Bigl(\sum_i W^{(l)}_{ji}\,h^{(l-1)}_i + b^{(l)}_j\Bigr)^{d_l}$$

where $d_l \in \mathbb{N}$ is the monomial activation degree of layer $l$. The final output is a linear readout:

$$f_\theta = \sum_j W^{\mathrm{out}}_j\,h^{(L)}_j + b^{\mathrm{out}}$$

Because each activation is a monomial $\sigma(t)=t^{d_l}$, the output $f_\theta \in \mathbb{R}[x,y,z]$ is a polynomial of total degree bounded by the degree product:

$$D = \prod_{l=1}^{L} d_l$$

The Realization Family $\mathcal{F}_A$

For a fixed architecture $A$, define the realization map:

$$\Phi_A : \theta \in \mathbb{R}^P \;\longrightarrow\; f_\theta \in \mathbb{R}[x,y,z]_{\leq D}$$

The realization family is the image $\mathcal{F}_A = \mathrm{Im}(\Phi_A)$, a semi-algebraic subset of the vector space of degree-$\leq D$ polynomials. We ask three questions:

1. Family question: What is $\mathcal{F}_A \subseteq \mathbb{R}[x,y,z]_{\leq D}$?
2. Exact representability: Does $p \in \mathcal{F}_A$?
3. Training recovery: Does gradient descent find $\theta^*$ with $f_{\theta^*} = p$?

---

Target Polynomials

We study three targets $p : \mathbb{R}^3 \to \mathbb{R}$:

• $p_3(x,y,z) = z^3 - xy$  (degree 3)
• $p_4(x,y,z) = z^4 - xy$  (degree 4)
• $p_5(x,y,z) = z^5 - xy$  (degree 5, prime)

The $z^4-xy$ case admits an explicit construction via the double-squaring identity: $(a+b)^2-(a-b)^2=4ab$ applied twice through two layers of degree-2 activations.