# Incremental Simulation Data Feature - Implementation Spec

## Goal

Enable live waveform plotting in pyTesla by exposing ngspice's `SendData` callback data to Python. During a transient simulation, pyTesla needs to periodically read partial results and update the plot while the simulation is still running.

## Current State

The `send_data_callback` is already registered with ngspice in `simulator.cpp` line 71, but the handler `handle_data()` is empty - it discards the data. The `pvecvaluesall` struct that ngspice passes to this callback contains ALL vector values at each simulation timestep.

The relevant data structures from `sharedspice.h`:

```cpp
typedef struct vecvalues {
    char* name;        // Vector name (e.g., "v(top)", "i(l1)")
    double creal;      // Real value at this timestep
    double cimag;      // Imaginary value
    NG_BOOL is_scale;  // True if this is the sweep/time variable
    NG_BOOL is_complex;
} vecvalues, *pvecvalues;

typedef struct vecvaluesall {
    int veccount;      // Number of vectors
    int vecindex;      // Current step index
    pvecvalues *vecsa; // Array of vector value pointers
} vecvaluesall, *pvecvaluesall;
```

## Required Changes

### 1. Add incremental data buffer to Simulator class (`simulator.h`)

Add these members to the `Simulator` class:

```cpp
private:
    // Incremental data buffer - written by SendData callback, read by Python
    struct IncrementalBuffer {
        std::vector<std::string> vector_names;  // Set once by SendInitData
        std::vector<std::vector<double>> data;  // [vector_index][step] = value
        size_t read_cursor = 0;                 // How far Python has read
        bool initialized = false;               // True after SendInitData sets names
        std::mutex mutex;                        // Separate mutex (callback is hot path)
    };
    IncrementalBuffer incr_buffer_;
```

### 2. Implement handle_init_data() (`simulator.cpp`)

When ngspice calls `SendInitData` before simulation starts, capture the vector names:

```cpp
void Simulator::handle_init_data(pvecinfoall data) {
    std::lock_guard<std::mutex> lock(incr_buffer_.mutex);
    incr_buffer_.vector_names.clear();
    incr_buffer_.data.clear();
    incr_buffer_.read_cursor = 0;
    incr_buffer_.initialized = false;

    if (data && data->veccount > 0) {
        for (int i = 0; i < data->veccount; i++) {
            if (data->vecs[i] && data->vecs[i]->vecname) {
                incr_buffer_.vector_names.push_back(data->vecs[i]->vecname);
            }
        }
        incr_buffer_.data.resize(incr_buffer_.vector_names.size());
        incr_buffer_.initialized = true;
    }
}
```

### 3. Implement handle_data() (`simulator.cpp`)

When ngspice calls `SendData` during simulation, append the values:

```cpp
void Simulator::handle_data(pvecvaluesall data, int count) {
    if (!data || !incr_buffer_.initialized) return;

    std::lock_guard<std::mutex> lock(incr_buffer_.mutex);

    for (int i = 0; i < data->veccount && i < (int)incr_buffer_.data.size(); i++) {
        if (data->vecsa[i]) {
            incr_buffer_.data[i].push_back(data->vecsa[i]->creal);
        }
    }
}
```

### 4. Add Python-accessible methods to Simulator (`simulator.h` / `simulator.cpp`)

```cpp
// Returns vector names from the incremental buffer
std::vector<std::string> get_incremental_vector_names() const;

// Returns new data since last call.
// Returns map: vector_name -> vector of new values since last read.
// Advances the read cursor.
std::map<std::string, std::vector<double>> get_incremental_data();

// Returns total number of data points buffered so far
size_t get_incremental_count() const;

// Clears the incremental buffer (call before starting a new simulation)
void clear_incremental_buffer();
```

Implementation of `get_incremental_data()`:

```cpp
std::map<std::string, std::vector<double>> Simulator::get_incremental_data() {
    std::lock_guard<std::mutex> lock(incr_buffer_.mutex);
    std::map<std::string, std::vector<double>> result;

    if (!incr_buffer_.initialized || incr_buffer_.data.empty()) {
        return result;
    }

    size_t total = incr_buffer_.data[0].size();
    if (incr_buffer_.read_cursor >= total) {
        return result;  // No new data
    }

    for (size_t i = 0; i < incr_buffer_.vector_names.size(); i++) {
        auto begin = incr_buffer_.data[i].begin() + incr_buffer_.read_cursor;
        auto end = incr_buffer_.data[i].end();
        result[incr_buffer_.vector_names[i]] = std::vector<double>(begin, end);
    }

    incr_buffer_.read_cursor = total;
    return result;
}
```

### 5. Expose in pybind11 bindings (`module.cpp`)

Add to the Simulator class bindings:

```cpp
.def("get_incremental_vector_names", &ngspice::Simulator::get_incremental_vector_names,
    "Get vector names available in the incremental buffer")

.def("get_incremental_data", &ngspice::Simulator::get_incremental_data,
    "Get new simulation data since last call. Returns dict of vector_name -> list of new values.")

.def("get_incremental_count", &ngspice::Simulator::get_incremental_count,
    "Get total number of timesteps buffered so far")

.def("clear_incremental_buffer", &ngspice::Simulator::clear_incremental_buffer,
    "Clear the incremental data buffer")
```

### 6. Clear buffer on reset (`simulator.cpp`)

In the existing `reset()` method, add:

```cpp
void Simulator::reset() {
    // ... existing reset code ...
    clear_incremental_buffer();
}
```

## Usage Pattern (Python side - for reference only, don't implement)

```python
sim = Simulator()
sim.initialize()
sim.load_netlist("circuit.net")
sim.command("set filetype=binary")
sim.clear_incremental_buffer()
sim.run_async()  # Non-blocking

while sim.is_running():
    new_data = sim.get_incremental_data()
    if new_data:
        # new_data is dict: {"time": [0.1, 0.2, ...], "v(top)": [1.5, 2.3, ...], ...}
        update_plot(new_data)
    time.sleep(0.1)

# Final read to get any remaining data
final_data = sim.get_incremental_data()
```

## Important Notes

- The `SendData` callback fires on ngspice's simulation thread, so the buffer mutex must be lightweight (no Python GIL interaction)
- `get_incremental_data()` is called from the Python main thread via pybind11, which handles GIL automatically
- The read cursor pattern avoids copying the entire buffer each poll - only new data is returned
- Don't forget to release the GIL when calling `run_async()` so the background thread can actually run (use `py::call_guard<py::gil_scoped_release>()` in the binding if not already done)
- After implementation, rebuild with: `pip install -e .` from the ngspice repo root

## Files to Modify

1. `C:\git\ngspice\src\cpp\simulator.h` - Add IncrementalBuffer struct and new methods
2. `C:\git\ngspice\src\cpp\simulator.cpp` - Implement handle_data(), handle_init_data(), and new methods
3. `C:\git\ngspice\src\bindings\module.cpp` - Add pybind11 bindings for new methods

## Testing

After building, verify with:

```python
from pyngspice._pyngspice import Simulator
sim = Simulator()
sim.initialize()
# Load a simple transient netlist
sim.load_netlist("test.net")
sim.clear_incremental_buffer()
sim.run_async()
import time
while sim.is_running():
    data = sim.get_incremental_data()
    if data:
        for name, values in data.items():
            print(f"  {name}: {len(values)} new points")
    time.sleep(0.1)
# Final read
data = sim.get_incremental_data()
print(f"Final: {sum(len(v) for v in data.values()) // max(len(data),1)} total points per vector")
```