[DataGrid] Refactor row state propagation

[lauri865]

• I have followed (at least) the PR section of the contributing guide.

After some further thoughts on #15616, probably better to try and fix the root cause of the row existence checks instead.

@romgrk, what if anything am I breaking here? Fixes the need for try/catch and out of order state propagation (currently on removing a row, first all the cells re-render to null, then the rows unmount. After this change, a row would be responsible for cells being rendered or not).

Fixes #16234
Before: https://codesandbox.io/p/sandbox/xenodochial-hooks-t2dymf
After: https://codesandbox.io/p/sandbox/unruffled-bas-pztmw9

Fixes #16181
Fixes #16170
Fixes #16225

Fixes race conditions in updating rows / columns
Fixes row editing state re-rendering all rows, and extracted actual editing state into a selector to avoid it working due to side effects of other selectors
Fixes columns always being rehydrated on resize (only needed for flex columns)
Fixes pinned columns propagation (happens in one render pass, rather than two)
Fixes rowSpanning with pagination being correctly processed on initialization
Fixes serverSide lazy loader incorrectly resetting state / scroll on mount (and a flaky test)
Fixes flaky test with server side aggregation (trying to test for aggregation before data has been loaded)
Fixes autoresize (=expand columns) with pinned columns (previously didn't expand to full width due to wrong availableWidth / 'viewPortInnerSize.width' excluding pinned columns)

Changelog

Breaking changes

• viewportInnerSize.width now includes pinned columns' widths (fixes recursive loops in updating dimensions <-> columns) (Edit: probably should be reclassified as a bugfix, ref: [DataGrid] Refactor row state propagation #15627 (comment))

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[mui-bot]

Deploy preview: https://deploy-preview-15627--material-ui-x.netlify.app/

Generated by 🚫 dangerJS against 581119f

[lauri865]

All tests should pass now. To me this approach makes much more sense:

• Baseline row state is owned and provided by rows that render the cells.
• We sprinkle in reactivity to cells based on more atomic selectors, which makes state flow more understandable and debuggable. Currently, all the valueGetters, formatters, etc. have to re-evaluate on each and every state change, which is wasteful.

I was slightly confused why the master detail test started failing, but after looking into it, I'm more worried that it worked the way it did – a breeding ground for race conditions.

The only reason I can think of why the other way around may have some merit is if the cells would be atomically updated based on changing row values. But since all the APIs expose the row to all the cells anyways, that cannot be the case.

Also squeezed in a tiny change that fixes every row selection checkbox re-rendering on every selection model change. Was such a tiny change that I didn't bother making it a new.

[romgrk]

Haven't looked in details, but at first glance this looks wrong to me. This function does state reads, and those should always be wrapped in useGridSelector to enable reactive updates when the state changes. Is there a reason for which this doesn't need to be wrapped? If yes, then we probably need a comment to explain why we don't wrap with useGridSelector.

[lauri865]

That's actually one of the core changes that is a step in the right direction imho, tried to explain it briefly here: #15627 (comment)

The current behaviour casts an incredibly wide net to cell-level reactivity that opens it up to race conditions, and prevents from carefully managing state propagation.

Should cell valueGetter and valueFormatter of each cell in the datagrid (renderecontext) re-evaluate whenever any part of grid state changes? (focus, tabindex, rendercontext, etc.?). They currently do, but I would argue they shouldn't. They should only be re-evaluated if the row changes (through edits or row updates).

Imagine if you define a valueFormatter that formats the cell value based on external state. Changing that external state wouldn't update the rendered cell value, but changing the focus or scrolling a bit would suddenly trigger the cell to re-render with a new value. What?!

Want to understand why the cell re-rendered? Most of the cases, the answer is that getCellParams changed. But why did they change? Good luck understanding that.

So, ideally:

• Row data is provided by row (or editing api)
• Rest of the conditions why the cell should re-render independent of the row should be carefully managed based on atomic selectors. As you can see, the scope of selectors that should trigger cell-level reactivity currently is not that wide in order to make the tests pass. This should result in both more optimised code as well as more understandable state flow within the grid.

[cherniavskii]

Okay, this explains why tabIndex and hasFocus are overridden with the values using useGridSelector.

@romgrk Does this make sense to you?

[romgrk]

Imagine if you define a valueFormatter that formats the cell value based on external state. Changing that external state wouldn't update the rendered cell value, but changing the focus or scrolling a bit would suddenly trigger the cell to re-render with a new value. What?!

The solution to that would be the approach that we've been introducing of adding some props to the grid's store reactivity model. In this case, columns[.valueFormatter] changes should trigger a store update, though I'm a bit surprised it doesn't.

@romgrk Does this make sense to you?

No, it still doesn't. AFAICT, we would be losing reactivity for the cellParams values other than the ones overriden with useGridSelector values. And I see some of them for which it would be problematic, e.g. cellMode or isEditable.

Want to understand why the cell re-rendered? Most of the cases, the answer is that getCellParams changed. But why did they change? Good luck understanding that.

getCellParams has been problematic for a long time (perf-wise and logic-wise), I haven't been able to get rid of it because it would be a breaking change, some of the API uses GridCellParams. I think if we want to fix everything cleanly, we need to decompose that function into more logical parts than "here's every bit of cell data". But that would prevent cherry-picking this PR in v7.

Thoughts?

[lauri865]

I don't think that's quite true that reactivity is lost. A bunch of tests would be failing if that were the case, and cell editing wouldn't work. Any colDef update propagates through the row as it is already. Whereas tabIndex and focus live in unrelated store to columns/rows.

Complete decomposition and granular cell-level reactivity would be a much bigger rewrite. My quick assessment currently is that it's not worth it and would needlessly complicate the code, as there's too many interrelated layers. One would need to ask what are the specific use cases where this would bring meaningful performance gains – I'm personally not seeing it. Maybe editing in theory, but since editing apis expose the whole row for various reasons, that's also not really the case either.

[lauri865]

I agree with the rationale that it shouldn't work due to side-effects. However, isEditable is not the case – every input to isEditable already triggers a re-render through the row->cell tree. It's not a side-effect by any means, but logical flow of state. Adding a selector as well just introduces extraneous work for no benefit. It will simply trigger a re-render from both the Cell within and Row as well.

On a second look, the only thing I actually think needs extracting out is cellMode, which does probably work due to side-effects.

The mental model to me is:

1. Everything in the row -> cell tree that depends directly on the row / cell store shouldn't need additional selectors, as the mere existence of a rendered cell means it's already reactive to these slices of the store.
2. Additional global state that affects how the particular cell is rendered (tabIndex, focus, cellMode) needs selectors.

Does that make sense to you or am I thinking completely wrong here?

[lauri865]

Or actually, cellMode is not even a side-effect – it depends on the same slice of the state that is already selected in the GridRow:

mui-x/packages/x-data-grid/src/components/GridRow.tsx

Line 346 in 2700d7e

editCellState={editCellState}

editCellState prop:

mui-x/packages/x-data-grid/src/components/GridRow.tsx

Line 322 in 2700d7e

const editCellState = editRowsState[rowId]?.[column.field] ?? null;

getCellMode prop depends on the same slice of the state:

mui-x/packages/x-data-grid/src/hooks/features/editing/useGridCellEditing.ts

Lines 254 to 261 in 2700d7e

	const getCellMode = React.useCallback<GridCellEditingApi['getCellMode']>(
	(id, field) => {
	const editingState = gridEditRowsStateSelector(apiRef.current.state);
	const isEditing = editingState[id] && editingState[id][field];
	return isEditing ? GridCellModes.Edit : GridCellModes.View;
	},
	[apiRef],
	);

So, there are two options:

1. cellMode should be a prop of a cell
2. editRowsState selector should be removed from the row, and there should be a editCellState selector withing GridCell. On first look, I don't see why the whole row needs to re-render in response to that, since it's only used to pass a prop to cells, so it could be handled within cells themselves. Should work the same, regardless of whether the whole row is editable or only individual cells.

[romgrk]

As long as it's an explicit reactive binding (through a prop or through a selector) I'm happy with that, but option 2 sounds better. I didn't touch the editing code when I did the reactivity refactor so there's leftover stuff from the old "reactivity" architecture (re-render everything anytime anything changes).

Once the editing state is split into an editCellState selector, all that is left to do would be to split value and formattedValue logic, and we could get rid of getCellParams in GridCell. The other values (id, field, row, rowNode, colDef) are all available through props.

If you want to go ahead with that please do, otherwise I'll see if I can complete this tomorrow. Btw we appreciate all the PRs you submitted, we're a bit short on time with v8 preparation and we've been slow to review & merge them, but they're all great.

[lauri865]

Fixed it up now. Seems to work well. There was one thing that seems to have worked due to side-effects (already before this PR) in the GridRow, moved it to a discrete selector:
Before:

mui-x/packages/x-data-grid/src/components/GridRow.tsx

Line 113 in 878a538

const editing = apiRef.current.getRowMode(rowId) === GridRowModes.Edit;

After:

mui-x/packages/x-data-grid/src/components/GridRow.tsx

Line 127 in 33cc4c9

const editing = useGridSelector(apiRef, gridRowIsEditingSelector, rowId);

There are some additional re-renderings that can be fixed when stopping editing that should speed up editing in large grids, but that should be done when other PRs have been merged around memoization + GridRoot + forwardRef patch.

I'm a bit hesitant to drop getCellParams altogether in GridCell as it opens up an easy divergence between the public API and internal logic. I refactored it a bit to make sure we don't do any unnecessary operations there though, but still keep a solid contract. Unless we need to extract value and formattedValue logic into selector hooks, I don't really see the point either. And I don't currently see a reason why we would need to use hooks there. If we do, we should come up with a test case that it solves, otherwise it's probably just redundant operations.

[romgrk]

I'm a bit hesitant to drop getCellParams altogether in GridCell as it opens up an easy divergence between the public API and internal logic.

Not as long as we use the GridCellParams type.

I still think this section could be improved a bit, but I'm happy enough to merge it as it is.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[cherniavskii]

calling getRowIndexRelativeToVisibleRows no longer works with incorrect type (e.g. if ids are strings, you can't access it with an equivalent number). Not sure if not supporting javascript oddities would be considered a breaking change or not (personally, I wouldn't), but worth noting at least.

Hmm, I did not foresee this, but I think this is the correct behavior because nothing prevents you from having both "4" and 4 as row IDs in the same dataset. That wouldn't work correctly with an object.
The only potential inconvenience with this is trying to use row IDs extracted from DOM data attributes (always strings), while the actual IDs are numbers. But I still prefer the row ID type to be strict, without implicit typecasting.

[lauri865]

calling getRowIndexRelativeToVisibleRows no longer works with incorrect type (e.g. if ids are strings, you can't access it with an equivalent number). Not sure if not supporting javascript oddities would be considered a breaking change or not (personally, I wouldn't), but worth noting at least.

Hmm, I did not foresee this, but I think this is the correct behavior because nothing prevents you from having both "4" and 4 as row IDs in the same dataset. That wouldn't work correctly with an object. The only potential inconvenience with this is trying to use row IDs extracted from DOM data attributes (always strings), while the actual IDs are numbers. But I still prefer the row ID type to be strict, without implicit typecasting.

I agree with that. I've been only bitten by this implicit behaviour.

Regarding the viewportInnerSize.width change, I would definitely categorise it as a bug fix, rather than a breaking change. There's many calculations that are wrong due to that:

1. https://stackblitz.com/edit/react-b1esajbm?file=package.json,Demo.tsx - try autosizing with expand and pinned columns – doesn't fill the width.
2. This is wrong:

   mui-x/packages/x-data-grid/src/components/GridScrollArea.tsx

   Line 87 in b286684

   const maxScrollLeft = columnsTotalWidth - dimensions.viewportInnerSize.width;

3. Plus the discrepancy between autoHeight (correct) and regular values (been struggling with that in custom plugins)

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[romgrk]

LGTM, and I've added a few refactoring commit.

[github-actions]

Cherry-pick PRs will be created targeting branches: v7.x