Streams are often described as 'async iterators'. I am more used to C++ iterators which encourage you to keep the intermediate states of the iterators for re-use in algorithms. It can be difficult to see how you neatly achieve more complex iterator patterns in Rust which doesn't seem to encourage this. So how would you idiomatically express the below?
In the example the code attempts to group items with a matching timestamp, where one stream is providing the timestamps of each group, and the other the data that matches against it.
It is reasonably neat but doesn't work because take_while will consume the first value that it chooses NOT to take, meaning that is then omitted from the next group.
Is there a nice way to do this without having to loop and continually 'peek' at the next item in the stream to know when to stop?
use async_stream::stream;
use futures::{Stream, StreamExt, pin_mut};
use futures::future::{ready};
use futures::stream::iter;
#[derive(Debug)]
struct DataItem {
timestamp: usize,
source_id: i64,
value: f32
}
fn group_stream_items<DataStream: Stream<Item=DataItem>, MarkerStream: Stream<Item=usize>>(data: DataStream, markers: MarkerStream) -> impl Stream<Item=Vec<DataItem>> {
stream! {
pin_mut!(data);
pin_mut!(markers);
while let Some(marker) = markers.next().await {
let items_at_marked_time = data.as_mut()
.skip_while(|item| ready(item.timestamp < marker))
.take_while(|item| ready(item.timestamp == marker))
.collect::<Vec<_>>().await;
if items_at_marked_time.len() > 0 {
yield items_at_marked_time;
}
}
}
}
#[tokio::main]
async fn main() {
let data = [
DataItem {timestamp: 100, source_id: 1, value: 1_f32}, // Group: 0
DataItem {timestamp: 100, source_id: 2, value: 2_f32}, // Group: 0
DataItem {timestamp: 200, source_id: 1, value: 3_f32}, // Group: 1
DataItem {timestamp: 200, source_id: 2, value: 4_f32}, // Group: 1
DataItem {timestamp: 300, source_id: 1, value: 5_f32}, // Group: 2
DataItem {timestamp: 400, source_id: 2, value: 6_f32}, // Group: 3
];
let markers = [100, 200, 300, 400];
let groups = group_stream_items(iter(data), iter(markers))
.collect::<Vec<_>>().await;
println!("Found groups: {:#?}", groups);
}
[package]
name = "rust-stream-example"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
tokio = { version = "1", features = ["full"] }
async-stream = "0.3"
futures = "0.3"