0

I'm trying to read a stream of JSON objects line-by-line from stdin and extract the "value" key from them as a string:

use rayon::prelude::*; // 1.0.3
use serde_json::Value; // 1.0.37
use std::io::{self, BufRead};
use std::sync::mpsc::channel;

fn main() {
    let stdin = io::stdin();

    let rx = {
        let (tx, rx) = channel();

        for line in stdin.lock().lines() {
            tx.send(line.unwrap()).unwrap();
        }

        rx
    };

    let it = rx.into_iter().par_bridge().map(|line| -> String {
        let v: Value = serde_json::from_str(&line.clone()).unwrap();
        let ret = v["value"].as_str().unwrap().into();
        println!("{}", ret);
        ret
    });

    eprintln!("Starting actual work.");
    it.for_each(|x| {
        println!("{}", x);
    });
}

eprintln!() is never called and I run out of memory because I use a huge file as input. Why doesn't map() just return an iterator I could use but instead waits for its input to end?

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
d33tah
  • 10,999
  • 13
  • 68
  • 158
  • *because I use a huge file as input* — define "huge". Running this code with 1094944 lines of input via `cargo run --release` takes 6.5s and seems to take 70MiB. It's printing lines of output the whole time. *`eprintln!()` is never called* — it's the very first thing printed for me. – Shepmaster Jan 30 '19 at 16:20
  • @Shepmaster we have completely different experiences then. How about 100 times as big a file? Perhaps it does load into memory as well in your case. Somebody on #rust-beginners told me that this closure does look like this, but I have no idea how to change it. – d33tah Jan 31 '19 at 09:44

0 Answers0