Slow and strange behaviour when using KVM model; possibly due to `compute_ancillaries`

[PinkShnack]

Reporting some strange behaviour with Pyjibe when using the (extension) KVM model.

Issues:

• First, the calculation is slow, and gets progressively slower as Pyjibe goes through the folder after clicking "Apply Model and Fit All". However, going to a new folder speeds up again and the slowing down "starts again"
• After running "Apply Model and Fit All", the displayed rating sometimes changes when the user clicks on a single dataset in the file list.

Questions for understanding:

1. Is the rating system deterministic?
2. Is it correct that the ancillary parameter calculation get progressively slower as pyjibe runs through the files? Is this due to the curves needing the previous curve calculations because they are one dataset? Or is it just a bug?
3. The ancillary parameters seems to bounce between very small values and the values they land on, presumably because they are being fit between an interval.

Possible reasons/solutions

I added caching to nanite/model/core/compute_ancillaries, which makes the calculation fast. However, max_indent ends up being nan which of course changes the expected ancillary param values. You even wrote a TODO note in this function for caching, but I am not yet familiar enough to implement it. Any ideas would be appreciated:

    def compute_ancillaries(self, fd):
        # TODO:
        # - ancillaries are not cached yet (some ancillaries might depend on
        #   fitting interval or other initial parameters - take that into
        #   account)
        # - "max_indent" actually belongs to "common_ancillaries" (see fit.py)"

[PinkShnack]

Here is an example output from nanite/model/core/compute_ancillaries:

• Current version (main)

 Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.44.55.309.jpk-force'[0], 1
no mod ancill: OrderedDict({'max_indent': nan})
no mod ancill: OrderedDict({'max_indent': 6.345093970550771e-07})
no mod ancill: OrderedDict({'max_indent': 6.250291057246192e-07})
no mod ancill: OrderedDict({'max_indent': 6.250291057246192e-07})
no mod ancill: OrderedDict({'max_indent': 6.345093970550771e-07})
no mod ancill: OrderedDict({'max_indent': 6.250291057246192e-07})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.235727936948198e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.235727936948198e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})

 Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.45.26.631.jpk-force'[0], 2
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})

• Version with Caching

Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.44.55.309.jpk-force'[0], 1
no mod ancill: OrderedDict({'max_indent': nan})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 7.940933880509066e-23, 'eta': 4.496924133483972e-13, 'time_ind': 0.11864952963088483})

 Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.45.26.631.jpk-force'[0], 2
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})

[paulmueller]

Overall, this sounds like a bug. Fitting should not get slower with number of curves analyzed (A sanity check would be to change the order of curves, but chances are low this is data-related).

It could also be that this is a UI issue (widgets being created and possibly redrawn after every fit).

To answer your questions:

1. Is the rating system deterministic?

Yes. It's based on decision trees.

2. Is it correct that the ancillary parameter calculation get progressively slower as pyjibe runs through the files? Is this due to the curves needing the previous curve calculations because they are one dataset? Or is it just a bug?

This sounds like a bug. Maybe there is a callback or somesuch that triggers computations with previous curves. Did you look into memory usage?

3. The ancillary parameters seems to bounce between very small values and the values they land on, presumably because they are being fit between an interval.

This sounds like mechanics of a specific fitting algorithm. Did you try a different fitting algorithm? It could also be that there are function calls that should not happen (I assume you print-debugged the model function).

Regarding the Caching TODO

Yes, be careful here. Ancillary parameter calculation can involve many parameters. The task is to take into account any possible parameter in the caching key. An alternative strategy would be to do caching in the KVM model (not sure whether that's what you tried), where you have full control over what is relevant.