Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add getLastStateChange, getLastStateUpdate, and getLastState to GenericItem #4351

Merged
merged 2 commits into from
Feb 15, 2025

Conversation

jimtng
Copy link
Contributor

@jimtng jimtng commented Aug 17, 2024

These methods are handy in many situations and isn't possible with mapdb persistence, so previously it would require something like influxdb.

It also solves the issue of needing to wait until persistence write operation is finished (if it was at all set up to persist on change), so this gives us a cheap and reliable / dependable way of getting the last change / last update time.

See discussions in
https://community.openhab.org/t/persistence-on-off-transitions-with-influx/156271/
https://community.openhab.org/t/blockly-persistence-last-changed-returns-no-result-quite-often/157842

@jimtng jimtng requested a review from a team as a code owner August 17, 2024 02:04
@jimtng
Copy link
Contributor Author

jimtng commented Aug 17, 2024

ping @mherwege @rkoshak

@rkoshak
Copy link

rkoshak commented Aug 17, 2024

I like the idea of this information being part of the Item. Of course it won't go back beyond OH startup but something is better than nothing. It certainly handles this use car better than persistence I think given all the different ways perspective can be configured.

How does this interact with restoreOnStartup? If an item is restored and not changed does this return the restored time or null? What makes the most sense? I'm not sure I know (it's close to bed time here so maybe I'll sleep on it).

@jimtng
Copy link
Contributor Author

jimtng commented Aug 17, 2024

How does this interact with restoreOnStartup? If an item is restored and not changed does this return the restored time or null?

It is not aware of persistence restoration, so I guess once persistence restores the state, it will return the time the state was restored.

@spacemanspiff2007
Copy link
Contributor

Imho it should be possible to query these values from the db so restore on startup should work as expected.
Will these fields also be part of the RestAPI / Websockets response?

@jimtng
Copy link
Contributor Author

jimtng commented Aug 17, 2024

Imho it should be possible to query these values from the db so restore on startup should work as expected.

@spacemanspiff2007 would you mind elaborating this please?

Will these fields also be part of the RestAPI / Websockets response?

Good point. I will add them there too.

@spacemanspiff2007
Copy link
Contributor

It's possible to query the last persisted state from persistence and subsequently restore this state as the current state on startup. It's also possible to query the persisted state before that.
Subsequently it should be possible (imho) to get all the values from the persistence service so restore on startup should also work for these fields as expected.

@jimtng
Copy link
Contributor Author

jimtng commented Aug 17, 2024

It's possible to query the last persisted state from persistence and subsequently restore this state as the current state on startup.

Yes, isn't this what restoreOnStartup does?

It's also possible to query the persisted state before that.
Subsequently it should be possible (imho) to get all the values from the persistence service so restore on startup should also work for these fields as expected.

I don't quite understand the above statements.

This PR doesn't interfere with persistence's restoreOnStartup. If I understand @rkoshak's question correctly, it is a question of whether getPreviousState should return (java) null as is the case when there was no previous state being set on the item, or whether it should return UnDefType.NULL because that's the item's state before restoration. Currently it's the latter.

@spacemanspiff2007
Copy link
Contributor

I think it's a misunderstanding.
All I am saying is that it's technically possible to restore these new fields from persistence so imho it makes sense to do so during restoreOnStartup.

@jimtng
Copy link
Contributor Author

jimtng commented Aug 17, 2024

@spacemanspiff2007 I've added it to the REST response

{
  "link": "http://192.168.1.10:8484/rest/items/TestSwitch1",
  "state": "ON",
  "previousState": "NULL",
  "lastUpdate": 1723872081231,
  "lastChange": 1723871965049,
  "editable": false,
  "type": "Switch",
  "name": "TestSwitch1",
  "tags": [],
  "groupNames": [
    "TestSwitches"
  ]
}

@mherwege
Copy link
Contributor

All I am saying is that it's technically possible to restore these new fields from persistence so imho it makes sense to do so during restoreOnStartup.

That depends on the persistence service. And it is not the case for mapDB, the most used persistence service for restoreOnStartup.

I would argue that, when the item state is updated as a consequence of restoreOnStartup, it should not update these lastChange and lastUpdate fields because now is probably the wrong time. But I am not sure this is feasible. I doubt the item knows anything about the source of the update/change.

In general, I think having the extra fields is a good idea. I just feel like we are still not there to give an answer to a simple question: when did my item last change?
If I use this new function (lastChange):

  1. Item did changed since startup -> OK, correct answer, and better then what I could get from persistence as persistence could miss the everyChange strategy or could have compressed
  2. Item did not change since start -> now, which is wrong
  3. If I can figure out the answer is wrong and I have a persistence service with everyChange: call lastChange on the service ->
    a. Current state equal to last state in DB -> correct answer from persistence (ignoring for now impact of compression)
    b. Current state different from last state in DB -> no answer (null)
    c. If no answer, you could return now in the calling rule, although in this case the startup time might be a more appropriate approximation

In an ideal world, mapDB would just store the lastChange time of an item with the state and we could restore that together with the state. I am just not sure it is easily feasible.
Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup, would it make sense to remove restoreOnStartup from persistence all together and make it part of the core infrastructure, just to take care of this? I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

We can keep adding layers to this, but there are always issues and exceptions. I like the idea of keeping lastChange and lastUpdate with the item. But I think the real simplification comes when we have a simpler restore mechanism on startup. Because than, you always get the right answer. Other persistence services then have the focus on data for graphing and calculations, which makes sense in my view.

@jimtng
Copy link
Contributor Author

jimtng commented Aug 17, 2024

OK, I think I now understand what @spacemanspiff2007 was alluding to, thanks to @mherwege's explanation.

Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup, would it make sense to remove restoreOnStartup from persistence all together and make it part of the core infrastructure, just to take care of this? I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

Not a bad idea.

@rkoshak
Copy link

rkoshak commented Aug 17, 2024

In an ideal world, mapDB would just store the lastChange time of an item with the state and we could restore that together with the state. I am just not sure it is easily feasible.

Why would this not be feasible? I beleive all the apis already exists in persistence to get the time from persistence. Once you have that it's just a matter of adding the last change and last update time as a parameter on the Item.

Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup,

I'm not sure we can make that case that's it's always only used for restoreOnStartup. It's useable if you only care about the most recent change to the Item in a rule (given it's current default strategies). One could change the strategies and it would work for most recent update.

But if these properties are added to the Item and we go down a path where those are made available and accurate on the the Item this use case is handled.

I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

Be careful here.

Not all uses want all Items restored.

Not all users want all Items restored to their most recent change or update (e.g they use a rule with .persist to control when the state gets saved).

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

@spacemanspiff2007
Copy link
Contributor

That depends on the persistence service. And it is not the case for mapDB, the most used persistence service for restoreOnStartup.

It would be trivial to modify mapDB persistence service to save the additional three values. A quick glance at the docs shows it should be possible. If not it's always possible to name mangle additional values with a separator that's not a valid item name (e.g. MyItem#lastUpdate)
The question is - as you correctly pointed out - if that's something that results in a meaningful behavior.

2. Item did not change since start -> now, which is wrong

No - it should return null because it's undefined.

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

I strongly agree. For many devices that report the state themselves I explicitly do not save the item state so I don't have the wrong state after startup. If the restore makes sense depends strongly on the device, the user and how the rules are written.
I for once was struggling with a rule and strange startup behavior during testing and the root cause was rrd4j restoring states during startup which was automatically configured (rrd4j).


Just a hint:
HABApp already provides the last_update and last_change datetimes. I deliberately chose not to make them null to simplify the rules so the users don't have to always check for that. Instead I chose to initialize them with the timestamp when the item gets created.

That would also result in a proper behavior when restoring from persistence.

Time 0 - item creation:
  state: NULL
  previousState: NULL
  lastUpdate: 0,
  lastChange: 0,
...
Time 10 - persistence restore:
  state: "asdf"
  previousState: NULL
  lastUpdate: 10,
  lastChange: 10,

So I am basically suggesting the opposite of what I wrote above 🙈 🤣

@jimtng
Copy link
Contributor Author

jimtng commented Aug 19, 2024

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

Each item can have a special metadata restoreOnStartup=true. When this is on a group item, persist all its direct members.

@mherwege
Copy link
Contributor

Each item can have a special metadata restoreOnStartup=true. When this is on a group item, persist all its direct members.

I think, what @rkoshak also implied is that one may want to use different rules for persisting values, not just on every change, but e.g. persist it once and always use that same value at startup. And that would mean we also need to support that part of the configuration in core. And then we are back to doing it the way it is done now anyway.

Thinking further about it:

  • Changing the restoreOnStartup logic to also restore lastChange, lastUpdate and previousState at startup is fairly easy to do. The method to change is . You may need some extra methods (or an extended setState method that has extra parameters) in GenericItem to also set these fields, although it should probably only be used for restoreOnStartup.
  • Many persistence services can already provide the extra info required. MapDB cannot and should be extended to store and retrieve the extra information. I don't know how easy that would be to do. We also seem to be using a very old version of MapDB, so I don't know how much impact that has on feasability.

With the above logic in place, it also would make sense to revisit the persistence extension actions to not return null, but return what is actually in the database, and document it as a breaking change, advising users to use the Item methods for this use case.

@mherwege
Copy link
Contributor

  • Many persistence services can already provide the extra info required. MapDB cannot and should be extended to store and retrieve the extra information. I don't know how easy that would be to do. We also seem to be using a very old version of MapDB, so I don't know how much impact that has on feasability.

Looking at MapDB code, it is actually not that difficult. The item state and time are already stored as a JSON string. Adding extra fields should not be a problem.

@rkoshak
Copy link

rkoshak commented Aug 19, 2024

I think, what @rkoshak also implied is that one may want to use different rules for persisting values, not just on every change, but e.g. persist it once and always use that same value at startup. And that would mean we also need to support that part of the configuration in core. And then we are back to doing it the way it is done now anyway.

Since a user needs a rule for calling .persist, if there's also way to turn off the restoreOnStartup for that Item in this new core feature, maybe that would be sufficient as they could create a system started rule to restore the Item's state manually from whatever persistence they are using instead of this new core one. Or maybe we can add a persistence extension to restore the state with the most recent value in the database that could be called from a rule in place of the restoreOnStartup strategy that would be removed.

This new core service doesn't need to do all the use cases. We just need to ensure that all the use cases are covered somehow.

With #4324 I think we will reliably be able to count on persistence being there by runlevel 100. So maybe a system started rule with previousState would be sufficient. I don't know though, I haven't fully explored all the edge cases that can crop up due to timing and such.

  1. Item did not change since start -> now, which is wrong

No - it should return null because it's undefined.

Maybe we are thinking too hard about this.

What do we intend these new properties to represent? I see two options:

  1. reflect the changes/updates made to the Item
  2. refelct the changes/updates to the device

If it's just 1, we don't really need to worry about most of these complexities. If an Item doesn't change after OH startup, getLastChange will be the time when the Item was created and set from restoreOnStartup and getPreviousState would be NULL. If you want the actual device information and not just information about the Item during this run session of OH, then you need to use the existing persistence capabilities. But this also means we don't need a new restoreOnStartup service in core or any of that complexity and the implementation becomes pretty simple.

If we want to make 2 happen then it gets much more complicated, as we can see and we probably need something built into core.

But do we really need to solve 2 here? Maybe 1 is sufficient? I think users who prefer not to use persistence at all would love to have 1.

We also need to thing of second tier impacts like the increase in writes for those running on SD cards and the like if this core restoreOnStartup feature becomes the default behavior.

@jimtng
Copy link
Contributor Author

jimtng commented Aug 28, 2024

So what's the next step?

If we are going to proceed in this direction, I would suggest:

  • First merge this PR without the extra setState / setters for lastChange, lastUpdate etc.
  • If and when the integration with the persistence extensions is added, then add the setters as you suggested. This way, it's clearer how the setters are implemented according to how it's actually used.

Copy link
Contributor

@mherwege mherwege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mherwege
Copy link
Contributor

If we are going to proceed in this direction, I would suggest:

  • First merge this PR without the extra setState / setters for lastChange, lastUpdate etc.
  • If and when the integration with the persistence extensions is added, then add the setters as you suggested. This way, it's clearer how the setters are implemented according to how it's actually used.

I am fine with this.

@spacemanspiff2007
Copy link
Contributor

I think this is the best way, too.
Maybe it makes sense to make persistence of lastChange etc. configurable. This could and should be discussed in the according PR.
Do the the new values also get reported in the corresponding item events?

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/lastupdate-and-null-values/158134/6

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-monitor-if-item-is-frequently-updated/158260/6

@jimtng
Copy link
Contributor Author

jimtng commented Sep 9, 2024

how do we resolve the access to these methods vs the persistence methods with the same name within rulesDSL?

@mherwege
Copy link
Contributor

mherwege commented Sep 9, 2024

how do we resolve the access to these methods vs the persistence methods with the same name within rulesDSL?

Maybe I miss something here, but what gets called would be determined by the imports I would think. As both items and persistence are imported by default, it indeed leads to a conflict which will force changing the rules to have a fully qualified name.

The options then become:

  1. Give these new methods another name: this is fully backward compatible.
  2. Change the name of the existing persistence actions to getLastPersistedState, getLastPersistedUpdate and getPreviousPersistedState.

In both cases, rules would work without being changed, but 2 would have a slight change in behaviour. I would still prefer 2 because it is more natural not to call persistence if not needed. If one absolutely wants the value from persistence the rule would need to be changed, but it would be very clear what is expected. At the same time, the persistence actions could be modified again to not return null, but the last effective persisted value.
2 also requires adapting the scripting languages to call these changed methods.

@jimtng
Copy link
Contributor Author

jimtng commented Nov 28, 2024

So do you agree that renaming the new methods is a better idea?

@mherwege
Copy link
Contributor

So do you agree that renaming the new methods is a better idea?

Yes, I do in the end.

@jimtng jimtng changed the title Add getLastChange, getLastUpdate, and getPreviousState to GenericItem Add getLastStateChange, getLastStateUpdate, and getLastState to GenericItem Nov 28, 2024
@jimtng
Copy link
Contributor Author

jimtng commented Nov 28, 2024

@mherwege Perhaps with no naming changes in persistence, at least now instead of returning null, the persistence method can call the Item's method to fulfil the request when such data is not available from the persistence service.

Or another strategy is to try Item's method first, and if it's null (when it was first loaded and still UNDEF), then it can query the persistence service.

This will solve the issues linked in the first post above, without requiring users to change their existing scripts!

@mherwege
Copy link
Contributor

@mherwege Perhaps with no naming changes in persistence, at least now instead of returning null, the persistence method can call the Item's method to fulfil the request when such data is not available from the persistence service.

Or another strategy is to try Item's method first, and if it's null (when it was first loaded and still UNDEF), then it can query the persistence service.

This will solve the issues linked in the first post above, without requiring users to change their existing scripts!

Agreed. That makes sense.

@jimtng
Copy link
Contributor Author

jimtng commented Nov 28, 2024

I've renamed the methods to getLastStateChange, getLastStateUpdate, and getLastState to avoid conflicts with the PersistenceExtensions methods. There should be no issues going forward now, I hope.

I don't think these values need to be "restored on startup" like persistence. If people need that restoration feature, they can call the equivalent PersistenceExtensions method.

I am still not 100% satisfied with the naming, but I can't think of any better names.

@andrewfg
Copy link
Contributor

not 100% satisfied with the naming, but I can't think of any better names

You could simply call the original ones 'xyz' and the new ones 'xyz2' ...

@mherwege
Copy link
Contributor

I don't think these values need to be "restored on startup" like persistence. If people need that restoration feature, they can call the equivalent PersistenceExtensions method.

Not required, true. But that means a user has to explicitely create startup rules if they want to do this, or have a check in every rule if the values are there to retrieve from persistence if not. So at some point I would like to have it.

I am still not 100% satisfied with the naming, but I can't think of any better names.

Same for me. That's part of the reason why renaming in the PersistenceExtensions class was attractive. It then clearly conveyed the purpose. Putting '2' behind the name does not solve that either. By lack of better idea, I think what you propose is fine.

@mherwege
Copy link
Contributor

I have created:

@holgerfriedrich holgerfriedrich added the enhancement An enhancement or new feature of the Core label Dec 4, 2024
@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ideas-and-discussion-what-features-do-you-want-in-openhab-5-0/160573/161

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ideas-and-discussion-what-features-do-you-want-in-openhab-5-0/160573/464

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/why-is-persistence-lastupdate-not-always-working/161949/3

Copy link
Member

@kaikreuzer kaikreuzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jimtng for this useful feature that nicely integrates into the existing code and thanks to all others for the in-depth discussion. Let's merge this!

@kaikreuzer kaikreuzer merged commit 3b3fd3a into openhab:main Feb 15, 2025
5 checks passed
jimtng added a commit to jimtng/openhab-jruby that referenced this pull request Feb 15, 2025
@jimtng jimtng deleted the item-last-change branch February 15, 2025 23:48
@spacemanspiff2007
Copy link
Contributor

spacemanspiff2007 commented Feb 16, 2025

@jimtng Have you already included the new timestamps in the ItemStateUpdatedEvent and ItemStateChangedEvent?

@jimtng
Copy link
Contributor Author

jimtng commented Feb 16, 2025

@jimtng Have you already included the new timestamps in the ItemStateEvent and ItemUpdatedEvent?

no, but it sounds like it might be useful

@spacemanspiff2007
Copy link
Contributor

Should I create an issue as a reminder?

@mherwege
Copy link
Contributor

@jimtng Do you have plans to extend documentation for this?
I guess the scripting addons and documentation may also require updating.

@jimtng
Copy link
Contributor Author

jimtng commented Feb 22, 2025

There's still a bit of a debate over whether we need to convert this from ZDT to Instant.
In addition to the javadoc, where else should this be mentioned?

As for the scripting addons, I've got a draft for jruby.

@mherwege
Copy link
Contributor

mherwege commented Feb 23, 2025

For documentation, what about:

I will keep jsscripting and Blockly in mind when the dust settles on Instant vs. ZonedDatetime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An enhancement or new feature of the Core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants