Tags:At, the data-services team has been tasked with migrating vertical-slices of data from one Tenant in one VPN (Virtual Private Network) to another Tenant in another VPN. To do this, we're gathering the database records (MySQL and MongoDB in this case), dumping those records so JSON (JavaScript Object Notation) files, and then importing those JSON files into the target system. This proof-of-concept-cum-production-code workflow has been working fairly well until recently - in the last migration, the size of the data exceeded V8's ability to execute JSON.stringify. As such, I wanted to take a quick look at how JSON can be parsed and generated incrementally using Node.js Transform streams.I am not sure, off-hand, which version of Node.js was running on the machine performing the migration; so, V8's ability to parse and stringify JSON may very well be version-dependent. That said, when we went to execute JSON.stringify on a massive record-set, we were getting the JavaScript error:RangeError: Invalid string lengthA quick Google search will reveal that this error actually means that the V8 process ran 'out of memory' while performing the serialization operation. So, instead of trying to serialize the entire record-set in one operation, we need to break the record-set apart and serialize the individual records.To do this, I am experimenting with the npm module.
JSONStream presents.parse and.stringify methods that provide Node.js Transform streams (aka 'through' streams) through which large operations can be broken down into smaller, resource-friendly operations. The nice thing about this module is that the final deliverables are the same - large JSON files; the only difference is that they are being generated and consumed incrementally instead of using a single, resource-exhausting operation.To test out this module, I'm going to take an in-memory record-set and stream it disk as JSON; then, when the JSON output file has been generated, I'm going to stream it back into memory and log the data to the terminal.JSONStream provides two methods for serialization:.stringify and.stringifyObject.
Node Js Php Serialize Objects Pdf
As you can probably guess from the names, stringify deals with Arrays and stringifyObject deals with Objects. I am using.stringify for my demo; but, in either case, it's important to understand that you are not passing a top-level entity to these methods. Instead, you are letting JSONStream generate the top-level entity from the sub-items that you pass to it. Thats why, in the following code, I'm passing individual records to the.stringify method, rather than trying to pass it the entire record-set.