P4python how do I mark a list of files for add/edit?

Question

I have a TON of files (~260,000) sorted into files with lists of files by type. (binary, binary+Dx, text, text+x, etc...)

EI -

binary:

D:/Workspace/folder1/file.dll
D:/Workspace/folder2/file.res
D:/Workspace/folder3/file.pyd

text:

D:/Workspace/folder1/file.txt
D:/Workspace/folder2/file.txt
D:/Workspace/folder3/file.doc

etc, etc....

Using p4python, is there a way to add a list of files to a changelist? I have managed to find a way to add these line-by-line, but that will take FOREVER; I think I calculated it at like 3 days.... This method takes about 1 sec per line:

for lines in fileList:
    p4.run_edit(f"-c{changeNum}", f"-t{fileName}", lines )

The command I would try to mimic is like:

p4 add -t text -c 1214 D:/Workspace/folder1/file.txt, D:/Workspace/folder2/file.txt, D:/Workspace/folder3/file.doc

The documentation for p4python suggests that there is a way to use files that are already marked for add (in the default CL?), but that still requires these files to be added.

I have also found the "-x" option for add Here, but p4python doesn't seem to like this option. At least I can't get it to accept it...

In a nut-shell: I am just looking for a relatively time-efficient way to get the files from my disk to the depot, 100% in-code. My knowledge of Python is still pretty limited, so please forgive me if there is a super-obvious way to do this that I don't know about.

Here is what I have so far....

def generate_changelist( fileLoc, fileType ):
    fileList = []
    ## starts 'New' changelist
    print("Generating Changelist...")
    change = p4.fetch_change()
    change._description = f"P4 Script; Uploading files of type: {fileType}"
    
    ## give it a Number
    CL = p4.save_change( change ) # returns a list with the string "Change ### created"
    print(CL)
    for item in CL: changeNum = item.split()[1]
    
    print(f"Attaching Files to CL {changeNum}")
    with open(fileLoc, 'r') as f:
        for line in f:
            line = line[:-1]    # trims the '\n' off the end...
            fileList.append(line) # p4 command would go here....
    ## tinkering with these right now....
    p4.run_edit(f"-c{changeNum}", f"-t{fileType}", f"{fileList}")
    #p4.run("-s", f"-x {fileList}", "add" )
    
    ## works....but painfully slow....
    ## I know I can add this to the for loop above...
    #for lines in fileList:
    #    p4.run_edit(f"-c{changeNum}", f"-t{fileType}", f"{lines}")
    return changeNum

EDIT / Solution-ish: SO i was able to get this to work, kinda:

print(f"Attaching Files to CL {changeNum}")
with open(fileLoc, 'r') as f:
    for line in f:
        line = line[:-1]    # trims the '\n' off the end...
        if (DEBUG): print(line)
        fileList.append(line)
p4.run_edit(f"-c{changeNum}", f"-t{fileName}", fileList)

There seems to be an issue that P4Python is not throwing the error:

So, I think it might be that the list is just too large. Might need to have a counter to break up the fileList into chunks... 5k per?

Thanks all for the help!!

The loop can be replaced with `fileList = [line.strip() for line in f]` — Barmar, Aug 24 '23 at 23:46
Are you sure you need to specify the filetype of each file individually? The type should have been auto-detected (or set based on the `typemap`) when the files were originally added; you don't need to re-specify it each time you edit them. If you eliminate that complication you can simply do `p4 edit //...` to open everything in your workspace for edit, and it will be *much* faster than opening each one for edit individually. — Samwise, Aug 25 '23 at 00:38
@Samwise - In our case, yes. the typemap is not picking up some types properly (utf16 for example). Upper management wants "Exact copies". :shrug: — Xolin_, Aug 25 '23 at 16:27
The typemap isn't the same as the auto-detection; you configure it based on file path (usually file extension) and it's applied exactly as defined, every time. There's no way for it to "not pick up" a type. https://www.perforce.com/blog/vcs/perforce-p4-typemap But note that it's applied *on add*, not on subsequent edits (so that if you manually override the typemap for a specific file, it doesn't get set back the next time someone edits it). — Samwise, Aug 25 '23 at 16:55

score 2 · Accepted Answer · answered Aug 24 '23 at 23:45

2

Don't put filelist in a single argument, they should be separate arguments.

p4.run_edit(f"-c{changeNum}", f"-t{fileType}", *fileList)

answered Aug 24 '23 at 23:45

Barmar

741,623
53
500
612

Removing the F string did the trick..... I forgot that that's for string references, not lists. Stoopid n00b error. TY for the help! – Xolin_ Aug 25 '23 at 16:29
I think you still need `*` to spread it, unless `run_edit()` does that automatically. – Barmar Aug 25 '23 at 16:30
looks like `run_edit()` does it for me. – Xolin_ Aug 25 '23 at 17:03

score 0 · Answer 2 · answered Aug 25 '23 at 04:21

Focusing on the "in a nutshell" part:

I am just looking for a relatively time-efficient way to get the files from my disk to the depot, 100% in-code.

Generating big lists of files and then using them to drive a script is certainly possible, but it's significantly less efficient than doing it the conventional way, and smacks of "XY problem".

Getting files from your disk to the depot in a Python script should usually be as simple as:

p4.run_reconcile()
p4.run_submit("-d", "P4 script uploading files")

If you want to set custom filetypes based on extension, this is best done via the p4 typemap command; the typemap is persistently stored on the server and is automatically applied when you add a file. A file's type "sticks" with it unless explicitly changed, so it's not necessary to re-specify it every time you make an edit.

P4python how do I mark a list of files for add/edit?

2 Answers2