Maybe not the right place to post this but I have a quick question:
I have a large amount of experiments I would like to run on the QPU. This will be between 1000-10000 jobs to submit. I have an academic licence which allows me to submit up to 5 concurrent jobs.
So my question is how to best run/submit this amount of jobs. I am currently running some tests locally which works pretty well, but I’d need some syntax that basically submits 5 jobs, waits until the first is done and then submits another one etc, etc.
Is it even possible to submit so many jobs? Time is (in principle) not a problem, happy to run the script for a week but just want to make sure it is all streamlined and doesn’t crash halfway through
In principle, the best way to run many jobs is to use a JobGroup (JobGroup — perceval v0.13 documentation) that allows you to save your jobs on your computer and run them easily with our provided methods.
However, JobGroups don’t give the possibility to choose how many concurrent jobs to run at once for now, they can only do all at once (using run_parallel) or one by one (using run_sequential).
I think the best ways would be, depending on the structure of your jobs, to:
create 5 job groups in 5 different threads (or 5 python instances run in parallel) and launch all of them with run_sequential. This has the advantage of always having 5 jobs running if the time taken for one job is constant (else a job group could finish before the others).
or create many job groups of 5 jobs and launch them sequentially using run_parallel.
You are not guaranteed that there will be no crash but you will easily have the possibility to rerun failed jobs and your jobs will be saved on your computer.
However, if you plan to run these on a QPU (and not a simulator), it is useless to try to run them in parallel since there is only one QPU, you can use only one JobGroup and run it sequentially.
