I assume you built your perl on a 32 bit machine, so during the build process, Configure didn't include any of the 'make this 32 bit' compiler switches. If you build on a 64 bit machine now, the build process will use exactly the same switches, so you get a 64 bit binary that cant't be loaded from 32 bit perl - not even on the 64 bit machines, beacuse the 32 bit perl binary you're running there can't load a 64 bit shared library either.
You might try building your shared perl on a 64 bit machine, explicitly stating you want a 32 bit perl. There should be some configure parameters for this. That way, you have a perl that sets the "use 32 bit" compiler flag when building modules. Then, you can use that version of perl on each of the machines to build the module. The modules won't be identical, but each of them will run on its bit size, and your software distribution process could pull the correct module when distributing to a specific machine.
However, the real problem is somewhat behind. I assume someone in your company at some point said "We don't want to be dependent on what the distributions provide, let's build our own perl that we can copy everywhere". This sounds like a good idea, but it is NOT. Different Linux versions use different versions of shared libraries, default directories for configuration files, default path variables etc. The configure process takes care of that and creates a perl binary for exactly your machine. If you copy this to a different machine, it might not find symbols in other versions of shared libraries. It might try to read lib from directories that don't exist there. It might not include a workaround for some bug that was corrected on the machine where you built it, but need the workaround on the older system you copied it to. Or, it might provide a workaround for something that has long been fixed on the newer system, thus wasting CPU time.
So, essentially, creating one perl to copy everywhere will ONLY work well if you build a static perl that includes everything and doesn't need any shared libraries. The standard, shared-library-using perl you compile on one machine, does NOT meet the "behaves the same everywhere i copy it to" request you probably had, because it depends way too much on stuff "around" it.