pyNgSpice/INCREMENTAL_DATA_SPEC.md

Incremental Simulation Data Feature - Implementation Spec

Goal

Enable live waveform plotting in pyTesla by exposing ngspice's SendData callback data to Python. During a transient simulation, pyTesla needs to periodically read partial results and update the plot while the simulation is still running.

Current State

The send_data_callback is already registered with ngspice in simulator.cpp line 71, but the handler handle_data() is empty - it discards the data. The pvecvaluesall struct that ngspice passes to this callback contains ALL vector values at each simulation timestep.

The relevant data structures from sharedspice.h:

typedef struct vecvalues {
    char* name;        // Vector name (e.g., "v(top)", "i(l1)")
    double creal;      // Real value at this timestep
    double cimag;      // Imaginary value
    NG_BOOL is_scale;  // True if this is the sweep/time variable
    NG_BOOL is_complex;
} vecvalues, *pvecvalues;

typedef struct vecvaluesall {
    int veccount;      // Number of vectors
    int vecindex;      // Current step index
    pvecvalues *vecsa; // Array of vector value pointers
} vecvaluesall, *pvecvaluesall;

Required Changes

1. Add incremental data buffer to Simulator class (simulator.h)

Add these members to the Simulator class:

private:
    // Incremental data buffer - written by SendData callback, read by Python
    struct IncrementalBuffer {
        std::vector<std::string> vector_names;  // Set once by SendInitData
        std::vector<std::vector<double>> data;  // [vector_index][step] = value
        size_t read_cursor = 0;                 // How far Python has read
        bool initialized = false;               // True after SendInitData sets names
        std::mutex mutex;                        // Separate mutex (callback is hot path)
    };
    IncrementalBuffer incr_buffer_;

2. Implement handle_init_data() (simulator.cpp)

When ngspice calls SendInitData before simulation starts, capture the vector names:

void Simulator::handle_init_data(pvecinfoall data) {
    std::lock_guard<std::mutex> lock(incr_buffer_.mutex);
    incr_buffer_.vector_names.clear();
    incr_buffer_.data.clear();
    incr_buffer_.read_cursor = 0;
    incr_buffer_.initialized = false;

    if (data && data->veccount > 0) {
        for (int i = 0; i < data->veccount; i++) {
            if (data->vecs[i] && data->vecs[i]->vecname) {
                incr_buffer_.vector_names.push_back(data->vecs[i]->vecname);
            }
        }
        incr_buffer_.data.resize(incr_buffer_.vector_names.size());
        incr_buffer_.initialized = true;
    }
}

3. Implement handle_data() (simulator.cpp)

When ngspice calls SendData during simulation, append the values:

void Simulator::handle_data(pvecvaluesall data, int count) {
    if (!data || !incr_buffer_.initialized) return;

    std::lock_guard<std::mutex> lock(incr_buffer_.mutex);

    for (int i = 0; i < data->veccount && i < (int)incr_buffer_.data.size(); i++) {
        if (data->vecsa[i]) {
            incr_buffer_.data[i].push_back(data->vecsa[i]->creal);
        }
    }
}

4. Add Python-accessible methods to Simulator (simulator.h / simulator.cpp)

// Returns vector names from the incremental buffer
std::vector<std::string> get_incremental_vector_names() const;

// Returns new data since last call.
// Returns map: vector_name -> vector of new values since last read.
// Advances the read cursor.
std::map<std::string, std::vector<double>> get_incremental_data();

// Returns total number of data points buffered so far
size_t get_incremental_count() const;

// Clears the incremental buffer (call before starting a new simulation)
void clear_incremental_buffer();

Implementation of get_incremental_data():

std::map<std::string, std::vector<double>> Simulator::get_incremental_data() {
    std::lock_guard<std::mutex> lock(incr_buffer_.mutex);
    std::map<std::string, std::vector<double>> result;

    if (!incr_buffer_.initialized || incr_buffer_.data.empty()) {
        return result;
    }

    size_t total = incr_buffer_.data[0].size();
    if (incr_buffer_.read_cursor >= total) {
        return result;  // No new data
    }

    for (size_t i = 0; i < incr_buffer_.vector_names.size(); i++) {
        auto begin = incr_buffer_.data[i].begin() + incr_buffer_.read_cursor;
        auto end = incr_buffer_.data[i].end();
        result[incr_buffer_.vector_names[i]] = std::vector<double>(begin, end);
    }

    incr_buffer_.read_cursor = total;
    return result;
}

5. Expose in pybind11 bindings (module.cpp)

Add to the Simulator class bindings:

.def("get_incremental_vector_names", &ngspice::Simulator::get_incremental_vector_names,
    "Get vector names available in the incremental buffer")

.def("get_incremental_data", &ngspice::Simulator::get_incremental_data,
    "Get new simulation data since last call. Returns dict of vector_name -> list of new values.")

.def("get_incremental_count", &ngspice::Simulator::get_incremental_count,
    "Get total number of timesteps buffered so far")

.def("clear_incremental_buffer", &ngspice::Simulator::clear_incremental_buffer,
    "Clear the incremental data buffer")

6. Clear buffer on reset (simulator.cpp)

In the existing reset() method, add:

void Simulator::reset() {
    // ... existing reset code ...
    clear_incremental_buffer();
}

Usage Pattern (Python side - for reference only, don't implement)

sim = Simulator()
sim.initialize()
sim.load_netlist("circuit.net")
sim.command("set filetype=binary")
sim.clear_incremental_buffer()
sim.run_async()  # Non-blocking

while sim.is_running():
    new_data = sim.get_incremental_data()
    if new_data:
        # new_data is dict: {"time": [0.1, 0.2, ...], "v(top)": [1.5, 2.3, ...], ...}
        update_plot(new_data)
    time.sleep(0.1)

# Final read to get any remaining data
final_data = sim.get_incremental_data()

Important Notes

• The SendData callback fires on ngspice's simulation thread, so the buffer mutex must be lightweight (no Python GIL interaction)
• get_incremental_data() is called from the Python main thread via pybind11, which handles GIL automatically
• The read cursor pattern avoids copying the entire buffer each poll - only new data is returned
• Don't forget to release the GIL when calling run_async() so the background thread can actually run (use py::call_guard<py::gil_scoped_release>() in the binding if not already done)
• After implementation, rebuild with: pip install -e . from the ngspice repo root

Files to Modify

1. C:\git\ngspice\src\cpp\simulator.h - Add IncrementalBuffer struct and new methods
2. C:\git\ngspice\src\cpp\simulator.cpp - Implement handle_data(), handle_init_data(), and new methods
3. C:\git\ngspice\src\bindings\module.cpp - Add pybind11 bindings for new methods

Testing

After building, verify with:

from pyngspice._pyngspice import Simulator
sim = Simulator()
sim.initialize()
# Load a simple transient netlist
sim.load_netlist("test.net")
sim.clear_incremental_buffer()
sim.run_async()
import time
while sim.is_running():
    data = sim.get_incremental_data()
    if data:
        for name, values in data.items():
            print(f"  {name}: {len(values)} new points")
    time.sleep(0.1)
# Final read
data = sim.get_incremental_data()
print(f"Final: {sum(len(v) for v in data.values()) // max(len(data),1)} total points per vector")