Removing a HashMap entry inside a match arm that borrowed a value from the map

Question

I have a HashMap<String, Vec<String>>. After removing an element from a vector, I want to remove this vector's entry if it is empty. I've tried to do this:

use std::collections::HashMap;

fn main() {
    let mut map: HashMap<String, Vec<String>> = HashMap::new();

    map.insert(String::from("key"), vec![String::from("value")]);

    let a_key = String::from("key");
    let del_from_vec = String::from("value");

    match map.get_mut(&a_key) {
        Some(vect) => match vect.binary_search(&del_from_vec.to_string()) {
            Ok(index) => {
                vect.remove(index);
                println!("value removed from vector.");
                if vect.len() == 0 {
                    map.remove(&a_key);
                    println!("entry removed from map.");
                }
            }
            Err(_) => println!("not found in vec."),
        },
        None => println!("no such key in map."),
    }
}

The compiler complains about this:

error[E0499]: cannot borrow `map` as mutable more than once at a time
  --> src/main.rs:17:21
   |
11 |     match map.get_mut(&a_key) {
   |           --- first mutable borrow occurs here
...
17 |                     map.remove(&a_key);
   |                     ^^^ second mutable borrow occurs here
...
24 |     }
   |     - first borrow ends here

I fully understand the error. I can overcome it by doing this:

use std::collections::HashMap;

fn main() {
    let mut map: HashMap<String, Vec<String>> = HashMap::new();

    map.insert(String::from("key"), vec![String::from("value")]);

    let a_key = String::from("key");
    let del_from_vec = String::from("value");

    if map.contains_key(&a_key) {
        match map[&a_key].binary_search(&del_from_vec.to_string()) {
            Ok(index) => {
                map.get_mut(&a_key).unwrap().remove(index);
                println!("value removed from vector.");
                if map[&a_key].len() == 0 {
                    map.remove(&a_key);
                    println!("entry removed from map.");
                }
            }
            Err(_) => println!("not found in vec."),
        }
    }
}

I find this code kind of ugly because it uses contains_key() and unwrap(). How else could I write this?