I've been looking for the protocol that Bluetooth (EDR and LE), use to encrypt the data. I've seen the official documentation but I don't get clear if its Diffie Hellman or E0.
PD: if I'm not wrong there are different situations in where the protocol implements one or another, i need when two devices share the 6-digits simple passkey, for example, when we pair our smartphone to a headset or to a speaker.
First of all, this question is not suitable to be asked on SO, as it does not involve any coding problem at all, but refers to specifications. Nevertheless, I will give you an answer.
The Diffie–Hellman key exchange (or Elliptic-Curve Diffie–Hellman (ECDH) if operating on elliptic curves) is - as its name states - a protocol for exchanging / generating cryptographic key material. It is not about the actual data encryption nor about the actual data integrity.
E0 on the other hand is an encryption algorithm used for actually encrypting data - but also not for authentication (integrity) of encrypted data.
Regarding BR/EDR key-exchange, authentication and encryption, you can find a nice overview in Table 1.1: Security algorithms in the [BR/EDR Controller] Security Specification (Bluetooth Core Specification Version 5.2 [Vol 2] Part H [Section 1]).
To summarize it for you, the Security Mechanisms Legacy and Secure Simple Pairing both use E0 for encryption, while Secure Connections uses AES-CCM.
Note, that Secure Simple Pairing and Secure Connections both use ECDH for key generation, while Legacy uses SAFER+.
Encryption on the LE Link Layer is performed using AES-CCM, as described in [Low Energy Controller] Low Energy Link Layer Security (Bluetooth Core Specification Version 5.2 [Vol 6] Part E).
The actual encryption on the Link Layer needs some cryptographic keys, which are derived / calculated during pairing using the methods described in the [Host] Security Manager Specification (Bluetooth Core Specification Version 5.2 [Vol 3] Part H).
To summarize it for you, there are LE legacy pairing (Short Term Key (STK) Generation) and LE Secure Connections (Long Term Key (LTK) Generation).
Note that both are about key-generation / exchange, while LE legacy pairing uses (somehow) AES and LE Secure Connections uses ECDH.
