javascript - Process node growing until eating all memory -
this question same previous one, since became huge repeated edit, prefer ask again properly.
i m working on raspberry pi (raspbian, 512m ram), , doing javascript app in node.js (v0.10.2).
in it, need download video, wich chunk this:
function download (file_url, callback){ var option={host:url.parse(file_url).host, port:80,path:url.parse(file_url).pathname}; var file_name=url.parse(file_url).pathname.split('/').pop(); var file=fs.createwritestream(download_dir+file_name); http.get(options, function(res){ res.on('data', function(data){ file.write(data); }).on('end'), function(){ file.end(); callback(download_dir+file_name); }); }); };
wich believe similar download large file node.js avoiding high memory consumption
the problem is, rss of node process don t stop increasing, s start around 100m, , killed os when around 400m (wich maximal ram).
since m new memory leak tracing , running app outside of browser, either can t use or can t understand how use memory tool, ve rss util library.
my question is: why rss keep increasing when m writing chunk of data on fly, and/or search solution?
edit: on second @ link above, noticed use of flags on createwritestream, can t find list nor effect of them, there here know way access such documentation, may related
edit2: m blindly following code of corlosdp, adding flag (appends?) make rss drop of 2m @ random intervals, , make app last longer, still end killed
edit3:i have discovered interesting
i edited download function see in file:
function download (file_url, callback){ var i=0; var option={host:url.parse(file_url).host, port:80,path:url.parse(file_url).pathname}; var file_name=url.parse(file_url).pathname.split('/').pop(); var file=fs.createwritestream(download_dir+file_name, {flags:'a', encoding:'binary'); setinterval(function(){ console.log(i+"o downloaded"); console.log(util.inspect(process.memoryusage())); },1000); http.get(options, function(res){ res.on('data', function(data){ i=i+data.length; file.write(data); }).on('end'), function(){ file.end(); callback(download_dir+file_name); }); }); };
and log looked this:
[downloading config file] 0o downloaded 0o downloaded 0o downloaded [logs of config file getting read, , first file start downloaded] 986o downloaded {rss: ...} 6713830o downloaded {rss:...} 986o downloaded {rss:...} 14421142o downloaded {rss: ...} 986o downloaded {rss:...} 29530702o downloaded {rss:...} ...
it seems there 2 download in parrallel, keep getting stranger when second file start (in callback of first file download), since there s 3 different download: 986 constant, first file keep appearing, , second file!
doesn t function supposed end when callback reached?
edit5: silly me, forgot clear interval, wich why got this.
i still don t know why code didn t worked, irc user, tried replace
res.on('data', function(data){ ... });
by
res.pipe(file);
it work, , don t go on 60m of ram, while struggling don t go on 400m, don t know why neither he, problem solved.
Comments
Post a Comment