Implicit Layers and Deep Equilibrium Models

By Quant Club, IIT Kharagpur — Jan 31, 2026 · 11 min read

What if a neural-network layer didn't follow a set of instructions you decide to give it, but found the answer on its own? In traditional deep learning, we stack layers like building blocks — each layer implements a predefined forward computation (a recipe followed step-by-step). Implicit layers, by contrast, do not give an explicit forward formula for the output. Instead, the layer's output is defined implicitly as the solution to an equation or condition (e.g., a fixed-point equation).

The output y* is whatever value satisfies:

f(y*, x, θ) = y*

We solve for y* using iterative methods, and train using gradients via:

• Backpropagating through the unrolled solver, or
• Implicit differentiation to get ∂L/∂θ without storing all solver iterations.

1. Traditional Neural Networks

In a traditional network, each layer has a fixed role (convolution, linear, recurrent), and we stack them to shape the final model — like Lego blocks.

For a single Dense layer:

• Pre-activation: z = Wx + b
• Activation: y = σ(z) (e.g., ReLU, Sigmoid, Tanh)
• Loss (regression): L = ½ ||y - y_true||²
• Gradient: ∂L/∂W = ∂L/∂y · ∂y/∂z · ∂z/∂W
• Update: W ← W - η · ∂L/∂W

import torch
import torch.nn as nn
import torch.optim as optim

class ExplicitNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.lay1 = nn.Linear(3, 4)
        self.act1 = nn.ReLU()
        self.lay2 = nn.Linear(4, 1)

    def forward(self, x):
        x = self.lay1(x)
        x = self.act1(x)
        x = self.lay2(x)
        return x

x = torch.randn(5, 3)
y_true = torch.randn(5, 1)
model = ExplicitNN()
optimizer = optim.Adam(model.parameters(), lr=0.01)
criterion = nn.MSELoss()

for epoch in range(10):
    y_pred = model(x)
    loss = criterion(y_pred, y_true)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}: Loss = {loss.item():.6f}")

Over 10 epochs, loss decreases from 0.827 → 0.656, showing gradual learning.

Limitations: Modeling equilibrium states or constraints requires stacking many layers — consuming memory, slowing training, and still failing to express the system's underlying rules.

2. Implicit Layers

An implicit layer defines the output as the solution to an equation:

y* = f(y*, x, θ)

The layer iteratively searches for a y* satisfying this fixed-point condition.

class ImplicitLayer(nn.Module):
    def __init__(self, hidden_dim=4):
        super().__init__()
        self.W = nn.Linear(hidden_dim, hidden_dim)
        self.U = nn.Linear(3, hidden_dim)
        self.activation = nn.Tanh()

    def forward(self, x, tol=1e-4, max_iter=50):
        y = torch.zeros(x.size(0), self.W.out_features)
        norm_diffs = []

        for _ in range(max_iter):
            y_next = self.activation(self.W(y) + self.U(x))
            norm = torch.norm(y_next - y).item()
            norm_diffs.append(norm)
            if norm < tol:
                break
            y = y_next.detach()

        return y, norm_diffs

Why bother?

3. Deep Equilibrium Models (DEQs)

DEQs take implicit layers further: instead of stacking identical residual layers repeatedly, they jump directly to the equilibrium point — the stable state of an infinitely deep network.

Setup

Consider a ResNet with weight-tying and input injection:

z_{k+1} = f(z_k, x; θ)  =  σ(Wz_k + Ux + b)

The DEQ directly finds the fixed point z* satisfying:

z* = f(z*, x; θ)

This models an infinitely deep network using only the parameters θ = {W, U, b} of a single function f.

Forward Pass: Anderson Acceleration

Finding z* uses Anderson Acceleration (AA) — a quasi-Newton method that computes the next iterate as a linear combination of previous iterates and function evaluations.

The forward solve runs inside torch.no_grad() to avoid storing the computational graph:

class DEQFixedPoint(nn.Module):
    def forward(self, x):
        with torch.no_grad():
            z_star, self.forward_res = self.solver(
                lambda z: self.f(z, x),
                torch.zeros_like(x)
            )

        z_out = self.f(z_star.clone().detach().requires_grad_(), x)
        return z_out

4. The Backward Pass: Implicit Differentiation

DEQs compute ∂L/∂θ without backpropagating through the unrolled solver, using the Implicit Function Theorem (IFT).

The Implicit Function Theorem (IFT)

Starting from the fixed-point identity z* = f(z*, x, θ), differentiating implicitly w.r.t. θ gives:

∂z*/∂θ = (I - ∂f/∂z*)⁻¹ · ∂f/∂θ

This depends only on f and its derivatives at the equilibrium point z*.

Step 1: Solve for gradient vector g

Solve this linear fixed-point problem iteratively (again with AA):

g = (∂L/∂z*) · (I - ∂f/∂z*)⁻¹

No intermediate forward-pass variables need to be stored — major memory savings.

Step 2: Compute final gradient

Once g is found, the final gradient is a single VJP (vector-Jacobian product):

∂L/∂θ = g · ∂f/∂θ |_{z*}

This is implemented via a backward hook on the output z.

5. Implicit Layers in Quant Finance: Differentiable Optimization

Implicit layers extend to Differentiable Optimization (DiffOpt), where output is defined as the solution to a minimization:

z* = argmin_z  h(z, x; θ)   subject to constraints

The Quant Advantage

• Hard Constraints: DiffOpt layers guarantee constraint satisfaction (e.g., ∑zᵢ = 1 for portfolio weights), unlike explicit layers that only approximate them.
• KKT Conditions: For convex problems (e.g., Quadratic Programs), the optimal solution is characterized by the Karush-Kuhn-Tucker (KKT) conditions — a system of equations treated as an implicit equation G(z*, λ*, ν*, x, θ) = 0.
• Differentiability via IFT: Applying IFT to the KKT conditions yields exact gradients.

Differentiable Quadratic Program (DQP) — Mean-Variance Portfolio Optimization

z* = argmin_z  ½ zᵀPz - qᵀz
     subject to: Az = b,  z ≥ 0

Where risk P and return q are parameterized by a neural network with weights θ. This enables end-to-end learning — the system learns θ such that the final constrained portfolio z* minimizes a top-level trading loss.

Conclusion

Implicit layers mark a significant shift in deep learning — from telling networks what to compute, to asking them to find solutions satisfying given conditions.

The future of deep learning may no longer be about building deeper towers — but about finding deeper understanding within a single, self-consistent layer.

Tags: Quantitative Finance · Deep Learning