Do many things, like pings

From redspid.blog.163.com
2007年 06月06日 19:42

Perl 多进程的例子，收藏先。

作 者：Randal L. Schwartz

As a Unix system administrator, I'm often faced with those little mundane tasks that seem so trivial to me but so important to the community I'm supporting. Little things like ``hey, is that host up and responding to pings?''. Such tasks generally have a very repetitive nature to them, and scripting them seems to be the only way to have time to concentrate on the tasks that really need my attention.

作为一个 Unix 系统管理员，我经常会面对一些单调乏味的任务，虽然这对我来说是小菜一碟，但是对于我所支持的社区来说却显得比较重要了（比如很简单的事情"嘿！那台主机在工作吗？回应我的 ping 请求吗？"）。这类任务通常都具有重复性，所以为了让我们能把精力集中在该集中的地方，写个脚本搞定这些琐事似乎就是唯一的方法了。

Let's look at the specific task of pinging a number of hosts on a subnet. Now, there are tools to do this quickly (like nmap), and there are even Perl modules to perform the ping (as in Net::Ping), but I wanted to focus on something familar that can be launched from Perl as an external process, and the system ping command seems mighty appropriate for that.

让我们看看这个在子网内 ping 大量主机的典型任务。现在有一些工具（像 nmap）可以很快地完成这个任务，而且甚至还有 Perl 模块来实现 ping（比如 Net::Ping),但我还是想把目光集中在一些能在 Perl 中作为额外进程运行的我们所熟悉的工具上,这样看来，系统所带的 ping 命令似乎比较适合。

First, let's look at how to ping one host, on my BSD-ish system:

首先，让我们看看如何 ping 一台主机，在我的 BSD 类系统中：

  sub ping_a_host {

    my $host = shift;

    `ping -i 1 -c 1 $host 2>/dev/null` =~ /0 packets rec/ ? 0 : 1;

  }

Here, I'm firing up a subshell to execute the ping -i 1 -c 1 command, which on my system requests ping have a 1-second timeout, and selects (as Sean Connery's character said in The Hunt for Red October so eloquently) ``one ping only''. Your ping parameters may vary: check your manpage.

这里，我们创建了一个子进程来执行 ping -i 1 -c 1 命令，这个命令只会 ping 目标主机一次，而且设置了超时时间为1秒。并选择（像肖恩-康纳利在《追击红色十月》里演的那个角色一样非常意味深长地说）"只 ping 一次"。你系统上的 ping 所需的参数可能有不同，请查看你的 man 说明。

The output is scanned for the string 0 packets rec, which if absent means we got a good ping. So if the match is found, we return 0 (the ping was bad), otherwise we'll return 1. The ping command spits out some diagnostics on standard error, which we'll toss using Bourne-shell syntax.

程序会在命令执行完所得到的输出中搜索 "0 packets rec" 这个字串，如果没有找到这个字串，说明 ping 成功了（目标主机有回应）。所以，如果我们找到了这个字串，那么返回 0（ping 失败了），否则我们返回 1。由于 ping 命令会向标准错误端输出一些诊断信息，所以我们把这些信息丢到 /dev/null 中去。

Note that the value of $host is not checked here for sanity. We certainly wouldn't want to accept a random command-line parameter or (gasp) a web form value here without some serious validation. However, as we use this in our program, all of the values will be internally generated, so we've got some degree of safety.

注意，这里没有对 $host 的值进行正确性检查。我们可不想接收一些事先没有经过验证的任意命令行参数，或者是通过网络表单传递来的值。不过，程序里所有值我们都将在程序内部生成，所以这已经是一层安全保障了。

So to scan a particular subnet, looking for hosts that are alive, we would add to that subroutine something like:

为了 ping 一个特定的子网看看有哪些主机存活，我们可以为上面的子程序添加如下的代码：

  print "ping $_ is ", ping_a_host($_), "\n"

    for map "10.0.1.$_", 1..254;

Now, this routine completes very quickly for hosts that are alive, but is slow-as-molasses for hosts that aren't present, because the TCP protocol demands that the host have a chance to respond.

现在，当主机存活时， ping 这一步很快就会完成，但如果主机不在（或者没有回应），那这一步就会非常慢了（由于 TCP 协议要求主机可以有一定的响应延迟的原因）。

So how can we speed that up? This is not a CPU-intensive loop: practically the entire time is waiting for some remote host to respond. We'll leave the ping_a_host subroutine alone, because that's not where we have a problem: it's doing its job as fast as it can. What we need to do is to do more of them at a time.

那么我们如何让程序加速呢？这并不是一个占用 CPU 很厉害的循环：实际上整个时间都花费在等待远程主机的回应上了。我们不管这个 ping_a_host 子程序，因为这并不是问题的所在：他正全速工作呢。我们需要的是在同一时间能做更多这样的任务。

One first approach is to fork a separate process for each host we want to ping. We'll then sit back in a wait loop. As each child process completes, we'll note its exit status, and when there are no more kids, we'll spit out a report.

其中一个首要的途径是为我们想 ping 的每一个主机 fork 出一个独立的进程。然后我们将返回到一个等待的循环中。当所有的子进程都完成的时候，我们会留意他们的退出状态，并且当子进程都结束后，再给出一个结果报告。

So, first, we'll define the host list for the task:

那么，首先，我们得先定义所需的主机列表：

  my @hosts = map "10.0.1.$_", "001".."010";

The numbers here are padded to three digits so that they sort as strings in a numeric sequence, a cheap but effective trick. Note also that I'm only selecting the first 10 hosts this time. I'll explain that shortly.

这些主机的最后部分被填充成三位，这样就能让这组字符串按照数字顺序来排序，简单而有效的技巧。注意我只选了子网中的前10台主机，后面我会解释原因。

Next, we'll want a hash to keep track of the kids:

下一步，我们将利用一个散列（hash)来跟踪子进程：

  my %pid_to_host;

The keys of this hash will be the child process ID (PID), and the value will be the corresponding host that the child is processing. Next, we'll want to loop over the host list, firing up a child for each:

子进程的进程号（PID）将成为这个散列的关键字，而其键值就是该子进程正在处理的相对应的主机地址。下一步，我们将遍历整个主机列表，为其中的每一个主机创建一个子进程：

  for (@hosts) {

    if (my $pid = fork) {

    ## parent does...

    $pid_to_host{$pid} = $_;

    warn "$pid is processing $_\n";

    } else { # child does

    ## child does...

    exit !ping_a_host($_);

    }

As each host is placed into $_, we'll fork. The result of fork is a child process running in parallel with the parent process. These processes are distinguished only by the return value of fork, which is 0 in the child, but the child's PID in the parent. So, if we get back a non-zero value, we're the parent, and we'll store the PID into the hash, along with the host that particular child is processing. If we're the child, then we'll call the ping_a_host routine, and arrange for our exit status to be good (0) if that routine gives a thumbs up.

在上面的代码中，所有的主机都会在 for 循环中依次传递给变量 $_ ,然后我们将进行 fork。调用 fork 函数后，我们将得到一个和父进程并行运行的子进程。它们之间的区别仅仅在于 fork 的返回值（fork 会返回两个值），如果返回值为 0,那说明在子进程中，父进程中 fork 的返回值就是子进程的 PID(进程号）。所以，如果我们得到一个非零的返回值，那么我们就在父进程中，在该进程中我们将会把子进程的 PID 及与工作中的该进程相应的主机 ip 保存到散列中。假如我们在子进程中，那么我们将会调用 ping_a_host 程序，并且当这个程序正常退出的时候，我们把退出状态码置为 0 (表示工作正常的）。

The warn in the loop is merely for diagnostic purposes so that you can see what's happening. In a production program, I'd certainly remove that.

这个循环中的 warn 仅仅用来输出诊断信息，这样你就可以知道程序正在干什么。当程序作为成品的时候，我会把这行去掉。

At the end of this loop, we'll have a number of processes. Far too many, in fact. For each host to check, we'll have two processes running: the shell forked by the backquotes, and the ping process itself. Perl has to fork a shell because I needed that child to have its standard error output redirected. If I could have gotten the redirection out of those backquotes somehow, we'd have only one child process per host, not two.

在这次循环的最后，我们将会有很多个进程。事实上太多了。对于每个要检查的主机，我们将耗费两个进程：一个是反引号调用系统命令分出的 shell 进程，一个是 ping 这个进程本身。Perl 之所以得分出一个 shell 进程是因为我们需要把这个子进程的标准错误输出转向（不会在屏幕上显示出来）。假如我们能用某种方式避免使用反引号调用的输出转向，那么我们就能只分出一个进程而不是两个了。

Launching 20 processes to check 10 hosts will start pushing us up against the typical per-user process limit. And now you can see why I didn't do all 254 hosts at once!

创建20个进程去检查10台主机将使我们面临经典的每用户进程数限制的问题（校者注：unix 下可以通过 ulimit -u 命令查到这个进程限制）。现在你应该明白我为什么先前只选前10台而不是一次性把254台主机都放进来的原因（会产生 508 个子进程！）。

Now it's time to wait for the results. A simple ``wait'' loop will reap the children as fast as they complete their task. First, we'll declare a hash to hold the results:

现在是时候等待返回结果了。用一个简单的 "wait" 循环就能在子进程完成任务之后尽快地收割它们。首先，我们得声明一个散列来保存结果：

  my %host_result;

The key will be the host, and the value will be 1 if the child said it was pingable, otherwise 0.

主机地址将成为该散列的关键字，而其对应的键值将是 1（当子进程说那台主机可以 ping 到）或者 0（ping 不到）

  while (keys %pid_to_host) {

    my $pid = wait;

    last if $pid < 0;

    my $host = delete $pid_to_host{$pid}

    or warn("Why did I see $pid ($?)\n"), next;

    warn "reaping $pid for $host\n";

    $host_result{$host} = $? ? 0 : 1;

  }

As long as we've got kids (indicated by the ever decreasing size of the %pid_to_host hash), we'll wait for them. The child process ID comes back from wait, which we'll stick into $pid. At this point, the exit status of that particular child is in $?. If the return value of wait is negative, then we don't have any more kids. This is an unexpected result, which we could check later by noticing that %pid_to_host is not yet empty, or we could have simply died here.

当有子进程存在的时候（由大小递减的散列 %pid_to_host 所指示），我们将会等待它们直到它们完结。wait 函数会返回子进程的进程号，我们把它保存至 $pid。此时，指定的子进程的退出状态保存在特殊变量 $? 中。假如 wait 返回值是负数，那说明我们没有子进程了。这不是我们所期待的结果，我们可以通过检查散列 %pid_to_host 是否非空来确认，或者简单的就在此处结束。

Next, we'll use the %pid_to_host hash to map the PID into the host for which it was processing. Again, we might have accidentally reaped a completed child which wasn't one of ours, so defensive programming requires checking for that. This won't happen unless other parts of this program are also forking children somehow, but I'm a cautious programmer most of the time.

接下来，我们将利用散列 %pid_to_host 将进程号映射到正在处理的主机。也许我们会偶然地将一个已完成的不属于我们的子进程收割掉，所以我们要对其进行检查。当我们没有在这个程序的其他地方以某种方式分出子进程的时候，就不会产生这种情况，虽然我们这个程序满足这个要求，但大多数时候我是个谨慎的程序员（所以我在程序中加入了对这种情况进行检查的代码）。

Finally, we'll take the exit status in $?, and map it into the appropriate good/bad value for the result hash.

最后，我们将参照特殊变量 $? 中的退出状态码把 ping 结果的 好/坏 值保存至结果散列。

When this loop completes, we have no more kids performing tasks, and it's time to show the result:

当这个循环结束，就没有子进程在处理任务了，现在是时候显示结果了：

  for (sort keys %host_result) {

    print "$_ is ", ($host_result{$_} ? "good" : "bad"), "\n";

  }

For each key of the result table, we'll say whether the result was good or bad.

以上的代码打印出每台主机 ping 的结果。

Putting this all together makes a nice little demo of forking 20 kids to check 10 hosts, but it won't scale to 254 hosts, because that would require more process slots than we typically have (or want to use, actually). What we need to do is perform the forking gradually, so that we never have more than 20 kids at a time. One naive approach is to chunk the data into bite-size bits:

把上面所有的代码凑到一起就成了一个分出20个子进程来检查10台主机的漂亮的演示程序，但是它却不适合有254台主机的情况，还记得前面提到的每用户进程限制吗？那怎么办呢？我们可以循序渐进地分出子进程的嘛，这样我们就可以保证同时不会有超过20个子进程存在。一个笨办法就是把数据（这里指主机列表）分割成几个部分：

  my @all_hosts = ...;

  my %host_results;

  while (my @hosts = splice @all_hosts, 0, 10) {

    ... process @hosts, adding into %host_results ...

  }

  ... show results ...

Here, most of the code above gets wrapped into an outer loop which hands 10 hosts at a time to be processed, using splice to peel them off of the master list. While this strategy certainly solves the ``no more than 10 at a time'' condition, each batch of 10 has to wait for the slowest of the 10 to complete.

以上的代码在用 splice 函数将 @all_hosts 每次分成10份付给 @hosts，然后针对 @hosts 来进行多进程的处理，这样，用一个外层的 while 循环（相对于那个多进程处理的循环）来保证同时只处理主机总列表中的10台主机。这种策略的确能解决"同时不多于10台主机"的问题，但是实际上每一批 10台主机都要等前面的10台主机处理完后才能继续，这样显然太慢了。

A better way would be to fork until we hit the limit of active children, then wait for one child to finish before we need to fork again. First, we'll need to factor out ``waiting for a kid'' into a subroutine so we can call it in two different places: while forking a new task, and at the end to reap all the remaining children:

一种更好的方法就是如果正在运行的子进程数量没有达到设定的极限值，就继续 fork，否则就等待某个子进程完成然后再 fork。首先，我们需要把"等待某个子进程"写成一个子程序以方便我们在两个不同的地方调用：当为新任务创建子进程以及在最后收割所有仍然存在的子进程的时候。

  sub wait_for_a_kid {

    my $pid = wait;

    return 0 if $pid < 0;

    my $host = delete $pid_to_host{$pid}

    or warn("Why did I see $pid ($?)\n"), next;

    warn "reaping $pid for $host\n";

    $host_result{$host} = $? ? 0 : 1;

    1;

  }

Note that we're accessing %pid_to_host and %host_result directly here, so those variables must be in scope before the subroutine definition. The subroutine now returns 1 if a kid was reaped, and 0 otherwise. The final reap loop now becomes:

注意我们这里是直接的访问散列 %pid_to_host 和 %host_result 的。所以它们必须已经在这个子程序定义之前定义了。当一个子进程被顺利收割时，该子程序返回1,否则返回0。程序最后的收割循环（译者注：这个收割循环会持续到所有的子进程全部被收割为止，这样是为了避免僵进程导致进程泄漏，关于僵进程和进程泄漏的概念，请自行查找有关资料）现在变成了：

  ## final reap:

  1 while wait_for_a_kid();

At this point, the program functions identically to the prior one, except that I've refactored the kid reaping. The magic happens next. We'll put wait_for_a_kid in the middle of the forking loop as well, just before we're about to fork, conditionally if the number of kids is already at the maximum we chose:

到这步，除了我重新写的这个子进程收割程序，这个程序的功能就和先前的那个一样了。接下来将发生一些魔术般的事情,呵呵。我们将把 wait_for_a_kid 子程序放到创建子进程的那个循环中，调用 fork 之前,在子进程数达到设定的极限值时调用该子程序。

  for (@hosts) {

    wait_for_a_kid() if keys %pid_to_host >= 10;

    ...

Ahh. That does it. We can now crank @hosts back up to our 254 items. As we fire off the first 10, this new statement has no effect. But when it comes time for the 11th, we'll wait until at least one of the other 10 to complete first. So, at no time do we have more than 10 hosts active (using 20 child processes for reasons explained earlier). The entire program is given here in case you want to see it all in context:

啊哈！现在我们可以把 @hosts 中主机的数量增加到254台了。当在为前10台主机创建子进程的时候，这条新加的表达式并不起什么作用，但是，当为第11台主机创建子进程的时候，它就会一直等待直到至少有一台主机已经处理完毕才继续。这样，我们就不会在同一时间处理超过10台主机了（处理10台机会使用20个子进程，原因前面说过了）。下面就是完整的程序：

  sub ping_a_host {

    my $host = shift;

    `ping -i 1 -c 1 $host 2>/dev/null` =~ /0 packets rec/ ? 0 : 1;

  }

  my %pid_to_host;

  my %host_result;

  sub wait_for_a_kid {

    my $pid = wait;

    return 0 if $pid < 0;

    my $host = delete $pid_to_host{$pid}

    or warn("Why did I see $pid ($?)\n"), next;

    warn "reaping $pid for $host\n";

    $host_result{$host} = $? ? 0 : 1;

    1;

  }

  my @hosts = map "10.0.1.$_", "001".."254";

  for (@hosts) {

    wait_for_a_kid() if keys %pid_to_host > 10;

    if (my $pid = fork) {

    ## parent does...

    $pid_to_host{$pid} = $_;

    warn "$pid is processing $_\n";

    } else { # child does

    ## child does...

    exit !ping_a_host($_);

    }

  }

  ## final reap:

  1 while wait_for_a_kid();

  for (sort keys %host_result) {

    print "$_ is ", ($host_result{$_} ? "good" : "bad"), "\n";

  }

As a working program, this does pretty good, although it could be made a bit more robust, and is very specific to the particular ping program on my machine. If you don't want to write this pattern of code into each program that wants to do parallel things, look at Parallel::Fork Manager in the CPAN, which does pretty much the same thing with a friendly interface.

虽然这个程序还可以进一步的完善以使它更强健，但现在他工作得很好，比起单单用我系统上的 ping 命令，用这个程序来完成任务就显得有效多了。如果你不想每次一涉及并行处理的事情就套用上面的代码，那么可以试试 CPAN 上的 Parallel::Fork Manager 模块，它也能够漂亮的实现到相同的效果，不过更妙的是它拥有一个更友好的接口。

One improvement to this program might be to pre-fork and re-use the children, using some sort of IPC (pipes or sockets) to communicate additional tasks to perform as each task completes, but I've run out of space to talk about that here. Until next time, enjoy!

预创建进程以及子进程重用（在每个任务完成的时候利用一些进程间通信（IPC）的方式（管道或者套接字）来与接下来的任务进行通讯）可以作为该程序的一个改进，但是这已经超出了本文的范围。okay, 希望你能喜欢这篇文章，下次再见！

―――――――――――――-

在 Parallel::Fork Manager 模块文档里摘一个实例给懒惰的 Perl 程序员 :) 这个例子会创建 30 个子进程来并行下载不同的链接。这要比亲自动手调用 fork 简单多了。

use LWP::Simple;

use Parallel::ForkManager;

...

@links=(

  ["http://www.foo.bar/rulez.data","rulez_data.txt"],

  ["http://new.host/more_data.doc","more_data.doc"],

  ...

);

...

# 同时使用 30 个进程

my $pm = new Parallel::ForkManager(30);

foreach my $linkarray (@links) {

  $pm->start and next; # 开始 fork

  my ($link,$fn) = @$linkarray;

  warn "Cannot get $fn from $link"

    if getstore($link,$fn) != RC_OK;

  $pm->finish; # do the exit in the child process

}

$pm->wait_all_children;

First you need to instantiate the Fork Manager with the "new" constructor. You must specify the maximum number of processes to be created. If you specify 0, then NO fork will be done; this is good for debugging purposes.

首先通过 new 来初始化一个 Fork Manager 对象，同时必须标明最大进程数。如果使用 0 的话就可以避免 fork 来达到调试程序的作用。

Next, use $pm->start to do the fork. $pm returns 0 for the child process, and child pid for the parent process (see also "fork()" in perlfunc(1p)). The "and next" skips the internal loop in the parent process. NOTE: $pm->start dies if the fork fails.

然后使用 $pm->start 来开始 fork。 $pm 在子进程时返回 0 ,父进程时返回子进程的进程号（具体请参阅 Perl 的 fork 文档）。"and next" 用来跳过父进程。注意：如果 fork 失败的话， $pm->start 就提示错误，程序结束。

$pm->finish terminates the child process (assuming a fork was done in the "start").

$pm->finish 结束子程序（假设我们一开始使用了 start 来 fork）

NOTE: You cannot use $pm->start if you are already in the child process. If you want to manage another set of subprocesses in the child process, you must instantiate another Parallel::Fork Manager object!

注意：在子进程中不能使用 $pm->start。 如果你想在子进程中使用另外一组子进程，你必须再初始化一个 Parallel::Fork Manager 对象。