2

I installed the transformers in the Macbook Pro M1 Max

Following this, I installed the tokenizers with

pip install tokenizers

It showed

Collecting tokenizers
Using cached tokenizers-0.12.1-cp39-cp39-macosx_12_0_arm64.whl
Successfully installed tokenizers-0.12.1

It seems to use the correct architecture for the whl file

When I import it I get

'/Users/myname/miniforge3/envs/tf/lib/python3.9/site-packages/tokenizers/tokenizers.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))

I see that this problem used to happen to others before. Any thoughts on how to fix this?

trialcritic
  • 1,225
  • 1
  • 10
  • 14

5 Answers5

5

James Briggs method works but produces the following error

note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

The Issue

After installing Rust and Cargo, we must source the environment file. This is the missing step in the previous answer.

The Solution

The workaround to solving this is to type the following in the terminal, right after installing Rust:

source "$HOME/.cargo/env"

Then, you can install transformers with the following code snippet:

pip install transformers
Mukilan Krishnakumar
  • 51
  • 4
3

If using Anaconda we switch to a terminal window and create a new ARM environment like so:

CONDA_SUBDIR=osx-arm64 conda create -n ml python=3.9 -c conda-forge

now get in to ml envoriment

conda activate ml

run inside the env

conda env config vars set CONDA_SUBDIR=osx-arm64

needs to restart env

conda deactivate

get into to env

conda activate ml

PyTorch Installation To get started we need to install PyTorch v1.12. For now, this is only available as a nightly release.

pip3 install -U --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Side note: The transformers library uses tokenizers built in Rust (it makes them faster). Because we are using this new ARM64 environment we may get ERROR: Failed building wheel for tokenizers. If so, we install Rust (in the same environment) with:

curl — proto ‘=https’ — tlsv1.2 -sSf https://sh.rustup.rs | sh

restart your env

conda deactivate 
conda activate ml

than you can install transformer comes with tokenizers or only install tokenizers

pip install tokenizers or pip install transformer

thanks to James Briggs

Emerson Pedroso
  • 181
  • 3
  • 13
1

You can try

conda install -c huggingface transformers
ansen
  • 11
  • 1
0

I got this error too. Solved it after a lot of trial & error.

The Problem: my brew was still running on Rosetta. Fixed that by uninstalling, cleaning and reinstalling. So everything seemed to run fine. Except this problem still kept cropping up

Until I discovered that pip is quite agressive in caching. So it caches the build even if the architecture changed. Solution: pip cache purge. Or remove the whole cache directory which you find with pip cache info

Touwer
  • 301
  • 2
  • 5
0

After testing most of the solutions provided I finally got it working by doing

brew install ffmpeg

sudo pip install tokenizers

️

Mathias Asberg
  • 3,562
  • 6
  • 30
  • 46