I'm wondering if there's some proper way (internet protocol, browser API, etc.), for a web app in a modern browser, to get a stream of large data from a Web Server (in my case, nodejs, but the question is focused on client side), and to save it to one big file in the client's file system. The assumption is that the amount of data is too large to be entirely buffered in the App's assigned memory, so I guess somehow the process should append chunks of data from the stream directly to a file.
The data is basically tens of thousands of db records, that should be written to a format such as csv.
Hope I'm making any sense here.
I'm wondering if there's some proper way (internet protocol, browser API, etc.), for a web app in a modern browser, to get a stream of large data from a Web Server (in my case, nodejs, but the question is focused on client side), and to save it to one big file in the client's file system. The assumption is that the amount of data is too large to be entirely buffered in the App's assigned memory, so I guess somehow the process should append chunks of data from the stream directly to a file.
The data is basically tens of thousands of db records, that should be written to a format such as csv.
Hope I'm making any sense here.
Share Improve this question edited Nov 14, 2019 at 16:46 Luke Girvin 13.5k10 gold badges67 silver badges85 bronze badges asked Mar 22, 2017 at 14:29 IsraelIsrael 1,2242 gold badges14 silver badges26 bronze badges 4- You mean like a normal file download? Aka, generating the file on the server and then sending that file to the client. – George Commented Mar 22, 2017 at 14:31
- 1 @George no, the server should stream data from the db and the client should process and build the file – Israel Commented Mar 22, 2017 at 14:34
-
You will want to use
pipe(). nodejs/dist/latest-v6.x/docs/api/… – Diego ZoracKy Commented Mar 22, 2017 at 15:04 - 1 @DiegoZoracKy thanks, but this is an approach for the server side, while I'm asking specifically about the client side – Israel Commented Mar 22, 2017 at 15:43
1 Answer
Reset to default 4You asked if there was a proper way. No, I'm afraid the technology that does it is badly supported and brand new. However, if you are brave enough to try anyway this is what you have.
1. Getting a stream of data from Web Server
If you don't have to support IE then you can use the Fetch API. See this question. If you do have to support IE then you could use a websocket to push down little bits at a time.
2. Writing a stream of data to the disk
StreamSaver.js Is a project that tries to acplish this using the ReadableStream API, however it currently only works in Opera and Chrome.