docs/nodes-grouping #4519

Huongg · 2025-02-25T20:46:16Z

Description

The first draft for nodes grouping documentation. Content is TBC by the team.
closes #4441

Development notes

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Huong Nguyen <[email protected]>

DimedS · 2025-02-26T10:53:27Z

Thanks for the PR, @Huongg - great job! Here are a few initial comments:

The table is a bit hard to read, possibly due to short line spacing.
The images are too small and not clickable, making it difficult to see details.
Some table entries contain too much text. We might consider shortening them by keeping only the key points where necessary.

merelcht

The general content looks good to me, but I'm not sure the table is the right format. It's quite small and the bullet points aren't formatted nicely inside the columns. Perhaps you can just write the content inside paragraphs with clear headings?

I've also left some comments on the phrasing, I find the tone of the text a bit too marketing style, like we're trying to sell this feature instead of explaining it. I'd suggest changing the tone to make it more informative and explain the why more than the what.

docs/source/deployment/index.md

docs/source/deployment/nodes_grouping.md

Signed-off-by: Huong Nguyen <[email protected]>

Huongg · 2025-02-27T19:23:41Z

hey @DmitrySorokinQB and @merelcht thanks for reviewing this PR! I agree that the table format was quite hard to read in this form. I've updated it to use paragraphs instead, but I've also included a summary table at the end to highlight the key points and make it easier for users.

astrojuanlu

Thanks a lot @Huongg and team! Left a few comments.

More broadly, the "what works in Kedro - what doesn't work in Kedro - how to use" section titles feel a bit repetitive, but the "best used when - not to use when" are useful. Maybe when we're settled on the content we can turn some of these bullet lists into prose?

astrojuanlu · 2025-02-28T07:17:16Z

docs/source/deployment/index.md

@@ -36,6 +36,11 @@ This following pages provide information for deployment to, or integration with,
 * [Prefect](prefect.md)
 * [Vertex AI](vertexai.md)

+
+# Effective node grouping for deployment


Suggested change

# Effective node grouping for deployment

## Effective node grouping for deployment

There's something funky happening with the navigation bar with these pages 🤔

Compare https://docs.kedro.org/en/nightly/

with this PR

maybe turning this into an H2 will fix it

astrojuanlu · 2025-02-28T07:17:42Z

docs/source/deployment/index.md

 ``` {warning}
 We also have legacy documentation pages for the following deployment targets, but these have not been tested against recent Kedro releases and we cannot guarantee them:


We might want to move this warning before the "effective node grouping" section

astrojuanlu · 2025-02-28T08:17:21Z

docs/source/deployment/nodes_grouping.md

+![Switching between different pipelines in Kedro Viz](../meta/images/kedro_viz_switching_pipeline.gif)
+
+### What doesn't work with Kedro
+- If you need to create custom node groupings that are different with your existing pipelines, creating new pipelines for them may not be convenient. Instead, you can use alternative grouping methods such as tags or namespaces.


I don't quite understand this sentence 🤔

docs/source/deployment/nodes_grouping.md

astrojuanlu · 2025-02-28T08:18:10Z

docs/source/deployment/nodes_grouping.md

+### What doesn't work with Kedro
+- If you need to create custom node groupings that are different with your existing pipelines, creating new pipelines for them may not be convenient. Instead, you can use alternative grouping methods such as tags or namespaces.
+- You cannot execute more than one pipeline in a single step because the kedro run --pipeline command allows running one pipeline at a time.
+- You can switch between different pipelines in Kedro Viz, but the visualisation does not support collapsing or expanding pipelines within Kedro Viz.


Suggested change

- You can switch between different pipelines in Kedro Viz, but the visualisation does not support collapsing or expanding pipelines within Kedro Viz.

- You can switch between different pipelines in Kedro Viz, but the flowchart view does not support collapsing or expanding pipelines.

?

astrojuanlu · 2025-02-28T08:20:30Z

docs/source/deployment/nodes_grouping.md

+
+### What doesn't work with Kedro
+- Nodes with the same tag can exist in different pipelines, making debugging and maintaining the codebase more challenging.
+- Executing more than one tag in a single step isn't always possible, and sometimes it will fail to run if a part of pipeline is not populated before at the time of execution.


Isn't this part of every method though? For example, if I run kedro run --pipeline data_science before data_processing, it will fail, would it not?

astrojuanlu · 2025-02-28T08:21:21Z

docs/source/deployment/nodes_grouping.md

+- Tags are not hierarchical, so tracking groups of nodes can become difficult.
+- Tags do not enforce structure like pipelines or namespaces.


These two are they key ones I think 💯

astrojuanlu · 2025-02-28T08:22:06Z

docs/source/deployment/nodes_grouping.md

+### What works with Kedro
+- Namespaces allow you to group nodes, ensuring clear dependencies and separation within a pipeline while maintaining a consistent structure.
+- As the same as pipelines or tags, you can enable selective execution using namespaces.
+- Namespaces improve visualisation in Kedro-Viz.


Maybe "Kedro Viz supports expand and collapsing namespaces as nested pipelines" or sth like that?

docs/source/deployment/nodes_grouping.md

astrojuanlu · 2025-02-28T08:23:24Z

docs/source/deployment/nodes_grouping.md

+- **Defining namespace at Pipeline-level:** When applying a namespace at the pipeline level, Kedro automatically renames all inputs, outputs, and parameters within that pipeline. This can introduce naming conflicts.
+
+### Best used when
+- You want to organise nodes logically within a pipeline while keeping a structured execution flow.


Maybe we could mention that namespaces allow nesting?

astrojuanlu · 2025-02-28T08:25:26Z

Rendered result https://kedro--4519.org.readthedocs.build/en/4519/deployment/nodes_grouping.html

docs/source/deployment/nodes_grouping.md

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

template for nodes grouping

f4d1a73

Signed-off-by: Huong Nguyen <[email protected]>

Huongg requested review from DimedS, iamelijahko, rashidakanchwala and merelcht February 25, 2025 20:46

merelcht reviewed Feb 26, 2025

View reviewed changes

Huong Nguyen added 3 commits February 26, 2025 19:00

change table layout

c1d924d

Signed-off-by: Huong Nguyen <[email protected]>

update content and structure, include summary table at the end

bbafbf5

Signed-off-by: Huong Nguyen <[email protected]>

Adding break instead of bullet point

53dae3c

Signed-off-by: Huong Nguyen <[email protected]>

Huongg force-pushed the docs/nodes-grouping branch from 7fae928 to 53dae3c Compare February 27, 2025 14:58

Huong Nguyen added 4 commits February 27, 2025 15:13

update vale comments

1c1218f

Signed-off-by: Huong Nguyen <[email protected]>

remove unused photo

14b5f0a

Signed-off-by: Huong Nguyen <[email protected]>

update copy

0604149

Signed-off-by: Huong Nguyen <[email protected]>

include nodes grouping in toctree

7c3c9a9

Signed-off-by: Huong Nguyen <[email protected]>

Huongg marked this pull request as ready for review February 27, 2025 19:23

Huongg requested review from yetudada and astrojuanlu as code owners February 27, 2025 19:23

Huongg changed the title ~~[DRAFT] docs/nodes-grouping~~ docs/nodes-grouping Feb 27, 2025

astrojuanlu reviewed Feb 28, 2025

View reviewed changes

merelcht reviewed Feb 28, 2025

View reviewed changes

docs/source/deployment/nodes_grouping.md Outdated Show resolved Hide resolved

docs/source/deployment/nodes_grouping.md Outdated Show resolved Hide resolved

docs/source/deployment/nodes_grouping.md Outdated Show resolved Hide resolved

Huongg and others added 5 commits February 28, 2025 16:04

update copy

0856b68

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

update copy

03ef6d4

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

update title

f3f45b4

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

update title

6c9b7fa

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

update title

dd362a0

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Huong Nguyen <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs/nodes-grouping #4519

docs/nodes-grouping #4519

Huongg commented Feb 25, 2025 •

edited by DimedS

Loading

DimedS commented Feb 26, 2025

merelcht left a comment

Huongg commented Feb 27, 2025

astrojuanlu left a comment

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu Feb 28, 2025

astrojuanlu commented Feb 28, 2025

	# Effective node grouping for deployment
	## Effective node grouping for deployment

		``` {warning}
		We also have legacy documentation pages for the following deployment targets, but these have not been tested against recent Kedro releases and we cannot guarantee them:

	- You can switch between different pipelines in Kedro Viz, but the visualisation does not support collapsing or expanding pipelines within Kedro Viz.
	- You can switch between different pipelines in Kedro Viz, but the flowchart view does not support collapsing or expanding pipelines.

		- Tags are not hierarchical, so tracking groups of nodes can become difficult.
		- Tags do not enforce structure like pipelines or namespaces.

docs/nodes-grouping #4519

Are you sure you want to change the base?

docs/nodes-grouping #4519

Conversation

Huongg commented Feb 25, 2025 • edited by DimedS Loading

Description

Development notes

Developer Certificate of Origin

Checklist

DimedS commented Feb 26, 2025

merelcht left a comment

Choose a reason for hiding this comment

Huongg commented Feb 27, 2025

astrojuanlu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astrojuanlu commented Feb 28, 2025

Huongg commented Feb 25, 2025 •

edited by DimedS

Loading