OpenLIME

Class: ShaderNeural

ShaderNeural

ShaderNeural implements a WebGL-based neural network for real-time image relighting. Used in conjunction with LayerNeuralRTI for Neural Reflectance Transformation Imaging.

Features:

  • Three-layer neural network architecture
  • Real-time image relighting
  • Multiple texture plane support
  • Configurable network parameters
  • ELU activation function
  • WebGL acceleration
  • Automatic color space conversion

Technical Implementation:

  • Forward pass computation in fragment shader
  • Vectorized operations for performance
  • Dynamic shader generation based on network size
  • Multi-texture sampling
  • Weight matrix management
  • Dequantization support

/** Neural Network Architecture Details

The network consists of three layers:

  1. Input Layer:

    • Accepts coefficient planes and light direction
    • Applies dequantization and normalization

  2. Hidden Layers:

    • Two fully connected layers
    • ELU activation function
    • Vectorized operations for efficiency

  3. Output Layer:

    • Produces final RGB/XYZ color
    • Linear activation

Implementation Notes:

  • All matrices are packed into vec4 for efficient GPU processing
  • Network dimensions are padded to multiples of 4
  • Uses texture sampling for coefficient input
  • Implements forward pass only

Example usage with LayerNeuralRTI:

// Create neural shader
const shader = new ShaderNeural({
    mode: 'light',
    nplanes: 9
});

// Configure network
shader.setShaderInfo(samples, 9, 52, 12, 'rgb');

// Update weights
shader.setUniform('layer1_weights', weights1);
shader.setUniform('layer1_biases', biases1);
// ... set other layers

// Set light direction
shader.setLight([0.5, 0.3]);

Fragment Shader Implementation

Key Components:

  1. Input Processing:

    • Texture sampling
    • Dequantization
    • Light direction incorporation

  2. Network Computation:

    • Vectorized matrix multiplication
    • ELU activation function
    • Layer-wise processing

  3. Output Processing:

    • Color space conversion
    • Final color computation

Uniforms:

  • {sampler2D} u_texture_[1-3] - Coefficient plane textures
  • {vec2} lights - Light direction vector
  • {vec4[]} layer[1-3]_weights - Layer weight matrices
  • {vec4[]} layer[1-3]_biases - Layer bias vectors
  • {vec3} min - Minimum values for dequantization
  • {vec3} max - Maximum values for dequantization

new ShaderNeural( [options])

Creates a new neural network shader

Parameters:
Name Type Argument Description
options Object <optional>

Configuration options

Properties
Name Type Argument Default Description
modes Array.<string> <optional>
 ['light']

Available modes
mode string <optional>
 'light'

Initial mode
nplanes number <optional>
 null

Number of coefficient planes
scale number <optional>
 null

Dequantization scale factor
bias number <optional>
 null

Dequantization bias
Source:

Extends

Methods

addFilter(filter)

Adds a filter to the shader pipeline.

Parameters:
Name Type Description
filter Object

Filter to add
Inherited From:
Overrides:
Source:
Fires:

allUniforms()

Returns all uniform variables associated with the shader and its filters. Combines uniforms from both the base shader and all active filters into a single object.

Inherited From:
Overrides:
Source:
Returns:

Combined uniform variables

Type
Object.<string, Object>

clearFilters()

Clears all filters from the shader pipeline.

Inherited From:
Overrides:
Source:
Fires:

getUniform(name)

Gets a uniform variable by name. Searches both shader uniforms and filter uniforms.

Parameters:
Name Type Description
name string

Uniform variable name
Inherited From:
Overrides:
Source:
Returns:

Uniform object if found

Type
Object | undefined

init()

Initializes default weights

Source:

removeFilter(name)

Removes a filter from the pipeline by name.

Parameters:
Name Type Description
name string

Name of filter to remove
Inherited From:
Overrides:
Source:
Fires:

setLight(light)

Sets the light direction for relighting

Parameters:
Name Type Description
light Array.<number>

Light direction vector [x, y]
Source:

setMode(mode)

Sets the current shader mode.

Parameters:
Name Type Description
mode string

Mode identifier (must be in options.modes)
Inherited From:
Overrides:
Source:
Throws:

If mode is not recognized

Type
Error

setShaderInfo(samples, planes, n, c, colorspace)

Configures shader for specific network architecture

Parameters:
Name Type Description
samples Array.<number>

Input samples
planes number

Number of coefficient planes
n number

Neurons per layer
c number

Input channels
colorspace string

Color space for processing
Source:

setTileSize(size)

Sets tile dimensions for shader calculations.

Parameters:
Name Type Description
size Array.<number>

[width, height] of tile in pixels
Inherited From:
Overrides:
Source:
Fires:

setUniform(name, value)

Sets a uniform variable value.

Parameters:
Name Type Description
name string

Uniform variable name
value number | boolean | Array

Value to set
Inherited From:
Overrides:
Source:
Fires:
Throws:

If uniform doesn't exist

Type
Error

Type Definitions

NetworkConfig

Configuration for neural network weights and parameters

Type:
  • Object
Properties:
Name Type Description
n number

Number of neurons per layer (padded to multiple of 4)
c number

Number of input channels (padded to multiple of 4)
colorspace string

Color space for processing ('rgb'|'xyz'|etc)
nplanes number

Number of coefficient planes
scale number

Dequantization scale factor
bias number

Dequantization bias
Source:

Events

update

Fired when shader state changes (uniforms, filters, etc).

Inherited From:
Overrides:
Source: