regex - 将全角字符转换为半角字符

Question

我有一个将全角字符转换为半角的程序。它工作正常，除了数字零。全角零不会转换为半角零。

Perl

use strict;
use warnings;
use warnings qw(FATAL utf8);
use utf8;
use feature qw(unicode_strings);
use open qw(:std :utf8);

unless ( @ARGV == 2 ) { 
    print "Usage: script.pl input_file output_file\n";
    exit;
}

my %fwhw = (
        '０' => '0', '１' => '1', '２' => '2', '３' => '3', '４' => '4',
        '５' => '5', '６' => '6', '７' => '7', '８' => '8', '９' => '9',
        'Ａ' => 'A', 'Ｂ' => 'B', 'Ｃ' => 'C', 'Ｄ' => 'D', 'Ｅ' => 'E',
        'Ｆ' => 'F', 'Ｇ' => 'G', 'Ｈ' => 'H', 'Ｉ' => 'I', 'Ｊ' => 'J',
        'Ｋ' => 'K', 'Ｌ' => 'L', 'Ｍ' => 'M', 'Ｎ' => 'N', 'Ｏ' => 'O',
        'Ｐ' => 'P', 'Ｑ' => 'Q', 'Ｒ' => 'R', 'Ｓ' => 'S', 'Ｔ' => 'T',
        'Ｕ' => 'U', 'Ｖ' => 'V', 'Ｗ' => 'W', 'Ｘ' => 'X', 'Ｙ' => 'Y',
        'Ｚ' => 'Z', 'ａ' => 'a', 'ｂ' => 'b', 'ｃ' => 'c', 'ｄ' => 'd',
        'ｅ' => 'e', 'ｆ' => 'f', 'ｇ' => 'g', 'ｈ' => 'h', 'ｉ' => 'i',
        'ｊ' => 'j', 'ｋ' => 'k', 'ｌ' => 'l', 'ｍ' => 'm', 'ｎ' => 'n',
        'ｏ' => 'o', 'ｐ' => 'p', 'ｑ' => 'q', 'ｒ' => 'r', 'ｓ' => 's',
        'ｔ' => 't', 'ｕ' => 'u', 'ｖ' => 'v', 'ｗ' => 'w', 'ｘ' => 'x',
        'ｙ' => 'y', 'ｚ' => 'z', '－' => '-', '、' => ', ', '　' => ' ',
         '／' => '/',);

sub slurp {
    my $file = shift;
    open my $fh_read, '<', $file or die "Could not open file: $!";
    return do {local $/; <$fh_read>};
}

sub convert {
    my $sub_string = shift;
    $sub_string =~ s/(.)/$fwhw{$1}?$fwhw{$1}:$1/seg;
    return $sub_string;
}

my $string = slurp($ARGV[0]);

$string =~ s/<target>\s*<g id="\d+">\K(.*?)(?=<\/g>\s*<\/target>)/convert($1)/seg;

open my $fh_write, ">", $ARGV[1] or die "Could not open file: $!";

print $fh_write $string;

close $fh_write;

这是我尝试过的

• 通过检查它们的代码点，我确保数字 0（零）和字母 O（哦）确实不同。全宽 0 是\x{ff10}. 全角字母 O 是\x{ff2f}. 我使用以下代码检查了这一点：

    use Encode;
  
    sub codepoint_hex {
        sprintf "%04x", ord Encode::decode("UTF-8", shift);
    }
  
    my $codepoint = codepoint_hex('０');
    print $codepoint, "\n";
  

• 我已经检查过哈希确实正确加载了所有键和值。

我还没有尝试过的：

• 我还没有尝试在 Linux 上复制这种奇怪的东西。我在 Windows 10 上使用 ActiveState Perl 5.24。

如果有人有任何建议或看到我的错误，我将非常感谢您的指导。

标签： regexperlcharacterdata-conversioncjk

Answer 1

由于$fwhw{'０'}return0和 since0为 false，因此不会发生替换。代替

$sub_string =~ s/(.)/$fwhw{$1}?$fwhw{$1}:$1/seg;

和

$sub_string =~ s/(.)/exists($fwhw{$1})?$fwhw{$1}:$1/seg;

---

如果这仍然不起作用，请使用sprintf "%vX", $str以查看您真正拥有的东西。

---

顺便一提，

sub convert {
    my $sub_string = shift;
    $sub_string =~ s/(.)/exists($fwhw{$1})?$fwhw{$1}:$1/seg;
    return $sub_string;
}

如果替换为会快得多

sub convert {
    state $chars = join '', keys(%fwhw);
    state $re = qr/([\Q$chars\E])/;
    return $_[0] =~ s/$re/$fwhw{$1}/gr;
}

再快一点，

sub convert {
    state $s = join '', keys(%fwhw);
    state $r = join '', values(%fwhw);
    state $tr = eval("sub { $_[0] =~ tr/\Q$s\E/\Q$r\E/r }");
    return $tr->($_[0]);
}