Storing values / Node-Red context

[Sineos]

Is there a way to write values to the flash from within Node-Red? Something like the Context Stores in Node-Red?
"Just" extending the context stores by a flash option would be really cool, e.g.

contextStorage: {
   default: "memoryOnly",
   memoryOnly: { module: 'memory' },
   flash: { module: 'node-mcu-flash' }
}

[phoddie]

Interesting question. I considered persisting values to flash early on, which you can still see in the note about preferences here. In the Moddable SDK, preferences are small pieces of data that persist across runs.

In Node-RED MCU Edition, the file system node is supported, so that does provide an option for storing state. However, as you note, in Node-RED that isn't the usual way to manage the state of flows, but that is usually done with a file based Context Stores.

Node-RED MCU Edition does support in-memory Context Stores. I did a quick review of the Node-RED Editor's Context Stores support. At first glance, it looks possible. Creating a new Context Store type seems unnecessary. We could just re-interpret the localfilesystem already present in Node-RED to mean "use preferences" on the MCU. It would be more limited, but it would work and provide some compatibility with Node-RED running on computers.

The current implementation of [Context](class Context) on MCUs is a light subclass of Map. An implementation would need to do two things:

• Update the Context class implementation to route requests for the file system store to preferences
• Update the nodered2mcu translator to recognize access to the file system store and generate the correct calls to the context

I have a couple of concerns, mostly because of my limited Node-RED experience:

• Node-RED appears to allow each context value to exist both in the local file system and in-memory. It appears that those two values are independent. Setting one does not change the other. That's a little strange. Am I misunderstanding or is that the design intent?
• Persistent settings eventually lead to the problem of needing to be reset. That's a higher level problem. Does Node-RED have a "reset stored context" option? One option is always to erase flash and start again. ;)

[Sineos]

Node-RED appears to allow each context value to exist both in the local file system and in-memory. It appears that those two values are independent. Setting one does not change the other. That's a little strange. Am I misunderstanding or is that the design intent?

I cannot really answer this qualified. I use both in my flows pretty extensively and according to my experience they are independent. My design philosophy is to use file-based context only for values that shall survive a restart and memory for all the rest.

Persistent settings eventually lead to the problem of needing to be reset. That's a higher level problem. Does Node-RED have a "reset stored context" option? One option is always to erase flash and start again. ;)

I know two ways to accomplish a delete:

Change node:

Function node:
flow.set("my_key", undefined, "file");

In Node-RED MCU Edition, the file system node is supported, so that does provide an option for storing state. However, as you note, in Node-RED that isn't the usual way to manage the state of flows, but that is usually done with a file based Context Stores.

Actually didn't really think about the file nodes, since it just felt "normal" to use contexts. For me perfectly fine. When working with a MCU not everything must be the same.

I quickly tried the file stuff but again hit some road blocks

flow.json

[
    {
        "id": "aa7ba678a2b6224d",
        "type": "tab",
        "label": "Flow 5",
        "disabled": false,
        "info": "",
        "env": [],
        "_mcu": {
            "mcu": true
        }
    },
    {
        "id": "9fd12e0c6f71cff1",
        "type": "ui_group",
        "name": "mcu_dash",
        "tab": "24f790420c419ffb",
        "order": 3,
        "disp": true,
        "width": "6",
        "collapse": false,
        "className": "",
        "_mcu": {
            "mcu": false
        }
    },
    {
        "id": "097119e1f1912126",
        "type": "ui_base",
        "theme": {
            "name": "theme-light",
            "lightTheme": {
                "default": "#0094CE",
                "baseColor": "#0094CE",
                "baseFont": "-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif",
                "edited": true,
                "reset": false
            },
            "darkTheme": {
                "default": "#097479",
                "baseColor": "#097479",
                "baseFont": "-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif",
                "edited": false
            },
            "customTheme": {
                "name": "Untitled Theme 1",
                "default": "#4B7930",
                "baseColor": "#4B7930",
                "baseFont": "-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"
            },
            "themeState": {
                "base-color": {
                    "default": "#0094CE",
                    "value": "#0094CE",
                    "edited": false
                },
                "page-titlebar-backgroundColor": {
                    "value": "#0094CE",
                    "edited": false
                },
                "page-backgroundColor": {
                    "value": "#fafafa",
                    "edited": false
                },
                "page-sidebar-backgroundColor": {
                    "value": "#ffffff",
                    "edited": false
                },
                "group-textColor": {
                    "value": "#1bbfff",
                    "edited": false
                },
                "group-borderColor": {
                    "value": "#ffffff",
                    "edited": false
                },
                "group-backgroundColor": {
                    "value": "#ffffff",
                    "edited": false
                },
                "widget-textColor": {
                    "value": "#111111",
                    "edited": false
                },
                "widget-backgroundColor": {
                    "value": "#0094ce",
                    "edited": false
                },
                "widget-borderColor": {
                    "value": "#ffffff",
                    "edited": false
                },
                "base-font": {
                    "value": "-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"
                }
            },
            "angularTheme": {
                "primary": "indigo",
                "accents": "blue",
                "warn": "red",
                "background": "grey",
                "palette": "light"
            }
        },
        "site": {
            "name": "Node-RED Dashboard",
            "hideToolbar": "false",
            "allowSwipe": "false",
            "lockMenu": "false",
            "allowTempTheme": "true",
            "dateFormat": "DD/MM/YYYY",
            "sizes": {
                "sx": 48,
                "sy": 48,
                "gx": 6,
                "gy": 6,
                "cx": 6,
                "cy": 6,
                "px": 0,
                "py": 0
            }
        }
    },
    {
        "id": "fad211c337a9b81b",
        "type": "ui_group",
        "name": "mcu_dash",
        "tab": "7dea3d93aa7eae5f",
        "order": 1,
        "disp": true,
        "width": "6",
        "collapse": false,
        "className": "",
        "_mcu": {
            "mcu": false
        }
    },
    {
        "id": "24f790420c419ffb",
        "type": "ui_tab",
        "name": "Home",
        "icon": "dashboard",
        "disabled": false,
        "hidden": false,
        "_mcu": {
            "mcu": false
        }
    },
    {
        "id": "7dea3d93aa7eae5f",
        "type": "ui_tab",
        "name": "Settings",
        "icon": "dashboard",
        "disabled": false,
        "hidden": false,
        "_mcu": {
            "mcu": false
        }
    },
    {
        "id": "3ae8446aecf9c1c3",
        "type": "debug",
        "z": "aa7ba678a2b6224d",
        "name": "debug 4",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "_mcu": {
            "mcu": true
        },
        "x": 840,
        "y": 320,
        "wires": []
    },
    {
        "id": "043af315965592aa",
        "type": "ui_text",
        "z": "aa7ba678a2b6224d",
        "group": "9fd12e0c6f71cff1",
        "order": 0,
        "width": 0,
        "height": 0,
        "name": "",
        "label": "text",
        "format": "test",
        "layout": "row-spread",
        "className": "",
        "_mcu": {
            "mcu": true
        },
        "x": 330,
        "y": 280,
        "wires": []
    },
    {
        "id": "fd1f8d63fadb86b6",
        "type": "ui_button",
        "z": "aa7ba678a2b6224d",
        "name": "write",
        "group": "9fd12e0c6f71cff1",
        "order": 1,
        "width": 0,
        "height": 0,
        "passthru": false,
        "label": "write",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "{\"bla\":0,\"test\":{\"bla2\":1}}",
        "payloadType": "json",
        "topic": "topic",
        "topicType": "msg",
        "_mcu": {
            "mcu": true
        },
        "x": 330,
        "y": 320,
        "wires": [
            [
                "fc615b4e02d96ebb"
            ]
        ]
    },
    {
        "id": "32c2ea02d393f84f",
        "type": "file",
        "z": "aa7ba678a2b6224d",
        "name": "",
        "filename": "my_settings.json",
        "filenameType": "str",
        "appendNewline": false,
        "createDir": false,
        "overwriteFile": "true",
        "encoding": "utf8",
        "_mcu": {
            "mcu": true
        },
        "x": 630,
        "y": 320,
        "wires": [
            [
                "3ae8446aecf9c1c3"
            ]
        ]
    },
    {
        "id": "ee74b210e8b5abc8",
        "type": "file in",
        "z": "aa7ba678a2b6224d",
        "name": "",
        "filename": "my_settings.json",
        "filenameType": "str",
        "format": "utf8",
        "chunk": false,
        "sendError": false,
        "encoding": "utf8",
        "allProps": false,
        "_mcu": {
            "mcu": true
        },
        "x": 510,
        "y": 360,
        "wires": [
            [
                "513d98fbbcab7d09"
            ]
        ]
    },
    {
        "id": "45d9b0548696232b",
        "type": "ui_button",
        "z": "aa7ba678a2b6224d",
        "name": "read",
        "group": "9fd12e0c6f71cff1",
        "order": 1,
        "width": 0,
        "height": 0,
        "passthru": false,
        "label": "read",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "bla",
        "payloadType": "str",
        "topic": "topic",
        "topicType": "msg",
        "_mcu": {
            "mcu": true
        },
        "x": 330,
        "y": 360,
        "wires": [
            [
                "ee74b210e8b5abc8"
            ]
        ]
    },
    {
        "id": "8b9384db98c5f1f9",
        "type": "debug",
        "z": "aa7ba678a2b6224d",
        "name": "debug 5",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "_mcu": {
            "mcu": true
        },
        "x": 840,
        "y": 360,
        "wires": []
    },
    {
        "id": "513d98fbbcab7d09",
        "type": "json",
        "z": "aa7ba678a2b6224d",
        "name": "",
        "property": "payload",
        "action": "obj",
        "pretty": false,
        "_mcu": {
            "mcu": true
        },
        "x": 690,
        "y": 360,
        "wires": [
            [
                "8b9384db98c5f1f9"
            ]
        ]
    },
    {
        "id": "fc615b4e02d96ebb",
        "type": "json",
        "z": "aa7ba678a2b6224d",
        "name": "",
        "property": "payload",
        "action": "str",
        "pretty": false,
        "_mcu": {
            "mcu": true
        },
        "x": 470,
        "y": 320,
        "wires": [
            [
                "32c2ea02d393f84f"
            ]
        ]
    }
]

This works in normal Node-Red but fails in the MCU:

IP address 192.168.178.165
{"payload":"{\"bla\":0,\"test\":{\"bla2\":1}}","_msgid":"5024459facdd6c74","source":{"id":"3ae8446aecf9c1c3","type":"debug","name":"debug 4"}}
C:\Users\souls\.node-red\node_modules\@ralphwetzel\node-red-mcu-plugin\node-red-mcu\nodered.js (984) # Break: parse: 1: invalid value!

[phoddie]

I cannot really answer this qualified. I use both in my flows pretty extensively and according to my experience they are independent. My design philosophy is to use file-based context only for values that shall survive a restart and memory for all the rest.

That's very practical. But to implement support in a compatible way, it is necessary to understand the behavior in edge cases like this.

I know two ways to accomplish a delete:

That deletes a single key. The problem is that each time you switch to running a different flow, the stored state from the previous flow remains. That builds up over time. That's not a problem on for Node-RED on computers, as the file system is huge and memory is nearly infinite. But.... on a microcontroller, the space reserved in flash for storing such preferences is limited. It needs to be completely reset at some point.

This works in normal Node-Red but fails in the MCU...

Thanks for trying. This is a separate topic though. It should go in a separate issue.

[phoddie]

I had a chance to try your file system flow. It is interesting. You have one node writing and another reading the same file. The Node-RED File node is defined to keep the write file open (for performance reasons). That was getting in the way of the read having access to the data written. I made a small patch to the Moddable SDK's LittleFS support and your flow works. Can you give that a try too? The change is after these lines to add a call to sync:

#else
		result = lfs_file_write(&gLFS->lfs, file, src, srcLen);
		if (result != (int)srcLen)
			lfs_error(the, result);
#endif
		lfs_file_sync(&gLFS->lfs, file);

I don't think this fix is optimal (flushing on every write is unnecessary expensive), so I don't want to commit it as-is.

[Sineos]

Works 👍

IP address 192.168.178.165
{"payload":"{\"bla\":0,\"test\":{\"bla2\":1}}","_msgid":"4b091583e9186ee2","source":{"id":"3ae8446aecf9c1c3","type":"debug","name":"debug 4"}}
{"payload":{"bla":0,"test":{"bla2":1}},"_msgid":"455da54929f047ee","source":{"id":"8b9384db98c5f1f9","type":"debug","name":"debug 5"}}

[phoddie]

Actually didn't really think about the file nodes, since it just felt "normal" to use contexts. For me perfectly fine. When working with a MCU not everything must be the same.

I can see both sides on this one. The MCU is different enough that there are going to be some differences in the nodes & flows. On the other hand, to the degree that things can be the same, it is easier to remember and work with.

I like the idea of mapping the Node-RED file context to the Moddable SDK preferences on an MCU. It is natural for Node-RED developers and lighter than using a file for the equivalent operation. As outlined above, it looks possible to do.

[Sineos]

Ah, bummer. I just opened a new discussion. Anyway thanks for looking into it.
What I wanted to "simulate" is basically a context file, i.e. some nodes adding to the context while others are reading it.

Will test it.

[phoddie]

Support for File Context Stores is now in place. The backing store is the Preferences module. See the readme for details.

The implementation is fresh so there are likely still some rough edges, though it is working nicely across microcontroller families and the simulator.

Because Node-RED doesn't include the default context store in the exported flow, the implementation assumes that the default Context Store is in-memory.

[Sineos]

This is strange. Updated all and still my flow breaks with above error message.

FWIW, here's a working flow that might be useful as a reference point.

This one works. But as soon as I slightly modify it to write / read a simple JSON, then it fails with the error above

modified flow

[
    {
        "id": "96c29fd0c49e316e",
        "type": "tab",
        "label": "flow.set",
        "disabled": false,
        "info": "",
        "env": [],
        "_mcu": {
            "mcu": true
        }
    },
    {
        "id": "1f36fad2e4196027",
        "type": "inject",
        "z": "96c29fd0c49e316e",
        "name": "start",
        "props": [],
        "repeat": "",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "_mcu": {
            "mcu": true
        },
        "x": 110,
        "y": 240,
        "wires": [
            [
                "80ee12d717d9fa76"
            ]
        ]
    },
    {
        "id": "80ee12d717d9fa76",
        "type": "change",
        "z": "96c29fd0c49e316e",
        "name": "set numbers",
        "rules": [
            {
                "t": "set",
                "p": "memory_number",
                "pt": "flow",
                "to": "1",
                "tot": "num"
            },
            {
                "t": "set",
                "p": "#:(file)::file_number",
                "pt": "flow",
                "to": "2",
                "tot": "num"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "_mcu": {
            "mcu": true
        },
        "x": 270,
        "y": 240,
        "wires": [
            [
                "8bf56790cf23d67c"
            ]
        ]
    },
    {
        "id": "8bf56790cf23d67c",
        "type": "function",
        "z": "96c29fd0c49e316e",
        "name": "set strings",
        "func": "const myFile = {\n    bla: 1,\n    blubb: 2\n}\nflow.set(\"memory_string\", \"MEMORY\");\nflow.set(\"file_string\", myFile, \"file\");\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "_mcu": {
            "mcu": true
        },
        "x": 450,
        "y": 240,
        "wires": [
            [
                "02f0a8f5d4f1f9c7"
            ]
        ]
    },
    {
        "id": "02f0a8f5d4f1f9c7",
        "type": "change",
        "z": "96c29fd0c49e316e",
        "name": "get all",
        "rules": [
            {
                "t": "set",
                "p": "file_string_bla",
                "pt": "msg",
                "to": "#:(file)::file_string.bla",
                "tot": "flow"
            },
            {
                "t": "set",
                "p": "file_string_blubb",
                "pt": "msg",
                "to": "#:(file)::file_string.blubb",
                "tot": "flow"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "_mcu": {
            "mcu": true
        },
        "x": 610,
        "y": 240,
        "wires": [
            [
                "96c2d7e7293b8aea"
            ]
        ]
    },
    {
        "id": "96c2d7e7293b8aea",
        "type": "debug",
        "z": "96c29fd0c49e316e",
        "name": "output",
        "active": true,
        "tosidebar": true,
        "console": true,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "_mcu": {
            "mcu": true
        },
        "x": 760,
        "y": 240,
        "wires": []
    }
]

Here it breaks:

It outputs the followng:

{
  "payload": 1677405404001,
  "topic": "",
  "_msgid": "34f2d440898955c3",
  "seg1": 1.5,
  "seg2": 2.5,
  "source": {
    "id": "264d2710e1eaecb6",
    "type": "debug",
    "name": "debug 1"
  }
}

This is exactly what I would have expected.

What would I need to do to erase the complete flash of the Moddable Two?

[Sineos]

Ah interesting. Doing a
npm install https://github.com/ralphwetzel/node-red-mcu-plugin
was not sufficient. Needed a
git pull in \.node-red\node_modules\@ralphwetzel\node-red-mcu-plugin\node-red-mcu.

Working now with your latest fixes. Again my thanks.

[phoddie]

Working now with your latest fixes

Cool. Glad you sorted that out. I just tried your example above and it works here too.

Can we mark this one as answered? ;)

What would I need to do to erase the complete flash of the Moddable Two?

This will do it:

esptool.py erase_flash

If esptool.py isn't in your path already, first do this:

source $IDF_PATH/export.sh

[Sineos]

Can we mark this one as answered? ;)

Would have done so already but the "General" category does not seem to offer this

This will do it:

esptool.py erase_flash

If esptool.py isn't in your path already, first do this:

source $IDF_PATH/export.sh

Thanks

[phoddie]

Would have done so already but the "General" category does not seem to offer this

Strange. So much still to learn about GitHub...