web-dev-qa-db-ja.com

シェルプロセスの並列実行

Windowsバッチファイルで複数のプロセスを並行して実行できるツールはありますか？ Linux用の興味深いツール（ parallel および [〜＃〜] ppss [〜＃〜] ）を見つけましたが、Windowsプラットフォーム用のツールが必要です。

おまけ： PsExec でプロセスをリモートで実行し、複数のマシン間でプロセスを簡単な方法で分散できるツールがあれば素晴らしいでしょう。

例：次のforループでそれが欲しい

for %F in (*.*) do processFile.exe %F

processCore.exeの限られた量のインスタンスが並行して実行され、マルチコアCPUを利用しています。

windowsbatch-fileshellparallel-processingmulticore
43
Dirk Vollmar

LinuxのGNU xargsには、「-P n」スイッチがあり、「n」プロセスを並行して起動します。

Xargsのcygwin/mingwビルドもこれをサポートしているのでしょうか？

次に使用できます：

xargs -P 4 processFile < fileList

ただし、派手なマルチノードプロセスの生成はありません。

5
ADEpt

Edit-各プロセスの出力をオプションで表示するようにスクリプトを変更しました

以下は、コマンドのリストを確実に並列に実行し、一度にnプロセスを超えて起動しないネイティブバッチソリューションです。 。

PSEXECを介して特定のCPUまたはリモートマシンにプロセスを配布するメカニズムも組み込まれていますが、その機能はテストしていません。

これを機能させるコツは、stdoutまたは未定義のハンドルをロックファイルにリダイレクトするCMDプロセスを介して各コマンドを開始することです。プロセスは、終了するまでファイルの排他ロックを維持します。プロセスの終了方法（通常の終了、クラッシュ、プロセスの強制終了）は関係ありません。ロックはすぐに解除されます。

マスタースクリプトは、同じロックファイルにリダイレクトすることにより、プロセスがまだアクティブかどうかをテストできます。プロセスがまだアクティブな場合、リダイレクトは失敗し、終了した場合は成功します。

デフォルトでは、スクリプトは各プロセスの出力を無視します。最初のパラメーターとして/Oオプションで開始した場合、インターリーブせずに各プロセスの出力が表示されます。

私のデモでは、プロセス制限を4に設定し、さまざまな長さの一連のPINGコマンドを実行するだけです。

これをXP、Vista、およびWindows 7でテストしました。

@echo off
setlocal enableDelayedExpansion

:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
  set "lockHandle=1"
  set "showOutput=1"
) else (
  set "lockHandle=1^>nul 9"
  set "showOutput="
)

:: The list of commands could come from anywhere such as another file
:: or the output of another command. For this demo I will list the
:: commands within this script - Each command is prefixed with :::
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1

:: Define the maximum number of parallel processes to run.
:: Each process number can optionally be assigned to a particular server
:: and/or cpu via psexec specs (untested).
set "maxProc=4"

:: Optional - Define CPU targets in terms of PSEXEC specs
::           (everything but the command)
::
:: If a CPU is not defined for a proc, then it will be run on the local machine.
:: I haven't tested this feature, but it seems like it should work.
::
:: set cpu1=psexec \\server1 ...
:: set cpu2=psexec \\server1 ...
:: set cpu3=psexec \\server2 ...
:: etc.

:: For this demo force all CPU specs to undefined (local machine)
for /l %%N in (1 1 %maxProc%) do set "cpu%%N="

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%\lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
:: Modify the IN () clause as needed to retrieve the list of commands
  set launch=1
  for /f "tokens=* delims=:" %%A in ('findstr /b ":::" "%~f0"') do (
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set cmd!nextProc!=%%A
    if defined showOutput echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting %%A
    2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! %%A
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do 2>nul (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      if defined showOutput echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      if defined showOutput type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    )
  )
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Thats all folks^^!

プロセス出力を無視するサンプル実行からの出力はここにあります

12:24:07.52 - proc1: starting  ping /n 05 ::1
12:24:07.52 - proc2: starting  ping /n 20 ::1
12:24:07.53 - proc3: starting  ping /n 10 ::1
12:24:07.54 - proc4: starting  ping /n 15 ::1
12:24:11.60 - proc1: finished  ping /n 05 ::1
12:24:11.60 - proc1: starting  ping /n 07 ::1
12:24:16.66 - proc3: finished  ping /n 10 ::1
12:24:16.66 - proc3: starting  ping /n 05 ::1
12:24:17.68 - proc1: finished  ping /n 07 ::1
12:24:17.68 - proc1: starting  ping /n 20 ::1
12:24:20.72 - proc3: finished  ping /n 05 ::1
12:24:20.72 - proc3: starting  ping /n 10 ::1
12:24:21.75 - proc4: finished  ping /n 15 ::1
12:24:21.75 - proc4: starting  ping /n 15 ::1
12:24:26.82 - proc2: finished  ping /n 20 ::1
12:24:26.82 - proc2: starting  ping /n 07 ::1
12:24:29.86 - proc3: finished  ping /n 10 ::1
12:24:32.89 - proc2: finished  ping /n 07 ::1
12:24:35.92 - proc4: finished  ping /n 15 ::1
12:24:36.93 - proc1: finished  ping /n 20 ::1
Thats all folks!

プロセス出力を示す/Oオプションを指定して実行した場合の出力は次のとおりです。

-------------------------------------------------------------------------------
12:24:51.02 - proc1: starting  ping /n 05 ::1
-------------------------------------------------------------------------------
12:24:51.02 - proc2: starting  ping /n 20 ::1
-------------------------------------------------------------------------------
12:24:51.03 - proc3: starting  ping /n 10 ::1
-------------------------------------------------------------------------------
12:24:51.04 - proc4: starting  ping /n 15 ::1
===============================================================================
12:24:55.10 - proc1: finished  ping /n 05 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:24:55.10 - proc1: starting  ping /n 07 ::1
===============================================================================
12:25:00.17 - proc3: finished  ping /n 10 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:00.19 - proc3: starting  ping /n 05 ::1
===============================================================================
12:25:01.22 - proc1: finished  ping /n 07 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:01.23 - proc1: starting  ping /n 20 ::1
===============================================================================
12:25:04.27 - proc3: finished  ping /n 05 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:04.28 - proc3: starting  ping /n 10 ::1
===============================================================================
12:25:05.30 - proc4: finished  ping /n 15 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:05.32 - proc4: starting  ping /n 15 ::1
===============================================================================
12:25:10.38 - proc2: finished  ping /n 20 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:10.40 - proc2: starting  ping /n 07 ::1
===============================================================================
12:25:13.44 - proc3: finished  ping /n 10 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
12:25:16.48 - proc2: finished  ping /n 07 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
12:25:19.52 - proc4: finished  ping /n 15 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
12:25:20.54 - proc1: finished  ping /n 20 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
Thats all folks!
51
dbenham

startを試してください：

start "title of the process" "P:\ath\to.exe"

指定されたタイトルで新しいウィンドウを開き、BAT、CMD、またはEXEファイルを実行します。優先度の設定、同じ環境の設定などもできます。

実行できないファイルは、関連するプログラムで開かれます。

さらに読む：スタート->実行

cmd /k start /?

Startは少なくともWinME以降で利用可能です。

幸運を！

23
guerda

Powershell 2を使用したいように聞こえますが、cmdを使用して新しいstartウィンドウ（または他のプロセス）を生成できます。 this answerも参照してください。 「プロセスプール」のようなものを作成するには、おそらく他のツールと少しの策略を使用する必要があります（最大（nインスタンスのみを実行するには）一度に）。後者を達成するには、tasklist /imを使用し、すでに存在する数（forループまたは該当する場合はwc）を数えて、単に待機（ping -n 2 ::1 >nul 2>&1）してから-新しいプロセスを生成できるかどうかもう一度確認します。

このための小さなテストバッチをまとめました。

@echo off
for /l %%i in (1,1,20) do call :loop %%i
goto :eof

:loop
call :checkinstances
if %INSTANCES% LSS 5 (
    rem just a dummy program that waits instead of doing useful stuff
    rem but suffices for now
    echo Starting processing instance for %1
    start /min wait.exe 5 sec
    goto :eof
)
rem wait a second, can be adjusted with -w (-n 2 because the first ping returns immediately;
rem otherwise just use an address that's unused and -n 1)
echo Waiting for instances to close ...
ping -n 2 ::1 >nul 2>&1
rem jump back to see whether we can spawn a new process now
goto loop
goto :eof

:checkinstances
rem this could probably be done better. But INSTANCES should contain the number of running instances afterwards.
for /f "usebackq" %%t in (`tasklist /fo csv /fi "imagename eq wait.exe"^|find /c /v ""`) do set INSTANCES=%%t
goto :eof

並行して実行され、最小化される最大4つの新しいプロセスが生成されます。各プロセスの処理量と実行時間に応じて、おそらく待機時間を調整する必要があります。他のことをしている場合は、おそらくタスクリストが探しているプロセス名を調整する必要があります。

ただし、このバッチによって生成されるプロセスを適切にカウントする方法はありません。 1つの方法は、バッチの開始時に乱数（%RANDOM%）を作成し、処理を実行する（または処理プログラムを生成する）が、ウィンドウタイトルをパラメーターに設定できるヘルパーバッチを作成することです。

@echo off
title %1
"%2" "%3"

これは、タイトルを最初のパラメーターに設定し、3番目を引数として2番目のパラメーターを実行する単純なバッチになります。その後、指定されたウィンドウタイトル（tasklist /fi "windowtitle eq ..."）を持つプロセスのみを選択することにより、タスクリストでフィルターできます。これはかなり信頼性の高い動作をするはずで、誤検知が多すぎることを防ぎます。いくつかのインスタンスがまだ実行されている場合、cmd.exeを検索するのは悪い考えです。ワーカープロセスのプールが制限されるためです。

%NUMBER_OF_PROCESSORS%を使用して、生成するインスタンスの数の適切なデフォルトを作成できます。

psexecを使用してプロセスをリモートで生成するように簡単に調整することもできます（ただし、バッチでパスワードを提供するだけでなく、他のマシンの管理者権限を持っている必要があるため、あまり実行できません）。ただし、フィルタリングにはプロセス名を使用する必要があります。

19
Joey

http://www.pirosa.co.uk/demo/wxargs/wxargs.html で-P並列処理オプションをサポートする基本的なWindows xargs-like-cloneがあります

6
PP.