StdoutをCSVとJSONの組み合わせとして解析する方法は？

Question

私は現在、コードを自動採点者に送信してから結果を返すクラスに取り組んでいます。返される形式は視覚的に解析するのが難しいので、パイプで使用して読みやすくするためのスクリプトを作成したいと思います。

オートグレーダーの出力は次のとおりです。

Problem,Correct?,Correct Answer,Agent's Answer
"Challenge Problem B-04",0,4,-1
"Basic Problem B-12",0,1,-1
"Challenge Problem B-05",0,6,-1
"Challenge Problem B-07",0,6,-1
"Challenge Problem B-06",0,3,-1
"Basic Problem B-11",0,1,-1
"Basic Problem B-10",0,3,-1
"Challenge Problem B-03",0,3,-1
"Challenge Problem B-02",0,1,-1
"Challenge Problem B-01",0,6,-1
"Challenge Problem B-09",0,4,-1
"Challenge Problem B-08",0,4,-1
"Basic Problem B-08",0,6,-1
"Basic Problem B-09",0,5,-1
"Basic Problem B-04",0,3,-1
"Basic Problem B-05",0,4,-1
"Basic Problem B-06",0,5,-1
"Basic Problem B-07",0,6,-1
"Basic Problem B-01",0,2,-1
"Basic Problem B-02",0,5,-1
"Basic Problem B-03",0,1,-1
"Challenge Problem B-10",0,4,-1
"Challenge Problem B-11",0,5,-1
"Challenge Problem B-12",0,1,-1
{
    "Basic Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Basic Problems B"
    },
    "Challenge Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Challenge Problems B"
    }
}

これは、コンマ区切りの値とJSONを組み合わせたものです。私が読むことができるきれいなテーブルにこれをすべて置くのは素晴らしいことです。

現在、私は次のようなものを持っています

python submit.py --provider gt --assignment error-check | column -t -s, | less -S

どの出力：

{
    "Basic Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Basic Problems B"
    },
    "Challenge Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Challenge Problems B"
    }
}
Problem                   Correct?  Correct Answer  Agent's Answer
"Challenge Problem B-04"  0         4               -1
"Basic Problem B-12"      0         1               -1
"Challenge Problem B-05"  0         6               -1
"Challenge Problem B-07"  0         6               -1
"Challenge Problem B-06"  0         3               -1
"Basic Problem B-11"      0         1               -1
"Basic Problem B-10"      0         3               -1
"Challenge Problem B-03"  0         3               -1
"Challenge Problem B-02"  0         1               -1
"Challenge Problem B-01"  0         6               -1
"Challenge Problem B-09"  0         4               -1
"Challenge Problem B-08"  0         4               -1
"Basic Problem B-08"      0         6               -1
"Basic Problem B-09"      0         5               -1
"Basic Problem B-04"      0         3               -1
"Basic Problem B-05"      0         4               -1
"Basic Problem B-06"      0         5               -1
"Basic Problem B-07"      0         6               -1
"Basic Problem B-01"      0         2               -1
"Basic Problem B-02"      0         5               -1
"Basic Problem B-03"      0         1               -1
"Challenge Problem B-10"  0         4               -1
"Challenge Problem B-11"  0         5               -1
"Challenge Problem B-12"  0         1               -1

これは私をそこへの道のほとんどに連れて行きます。 JSONを処理する方法があるかどうか疑問に思っていますか？

特定の行番号で出力を分割することに頼ることはできませんが、最初に{が見つかったときに出力をセグメント化できると考えています。

クラスメートと共有できるように、これは可能な限り最小限に抑えたいと思います。したがって、依存関係が少ないほど良いです。

外部コードの使用を提案する他のJSON解析投稿を見てきました。

理想的な出力は次のようになります。

Problem                   Correct?  Correct Answer  Agent's Answer
"Challenge Problem B-04"  0         4               -1
"Basic Problem B-12"      0         1               -1
"Challenge Problem B-05"  0         6               -1
"Challenge Problem B-07"  0         6               -1
"Challenge Problem B-06"  0         3               -1
"Basic Problem B-11"      0         1               -1
"Basic Problem B-10"      0         3               -1
"Challenge Problem B-03"  0         3               -1
"Challenge Problem B-02"  0         1               -1
"Challenge Problem B-01"  0         6               -1
"Challenge Problem B-09"  0         4               -1
"Challenge Problem B-08"  0         4               -1
"Basic Problem B-08"      0         6               -1
"Basic Problem B-09"      0         5               -1
"Basic Problem B-04"      0         3               -1
"Basic Problem B-05"      0         4               -1
"Basic Problem B-06"      0         5               -1
"Basic Problem B-07"      0         6               -1
"Basic Problem B-01"      0         2               -1
"Basic Problem B-02"      0         5               -1
"Basic Problem B-03"      0         1               -1
"Challenge Problem B-10"  0         4               -1
"Challenge Problem B-11"  0         5               -1
"Challenge Problem B-12"  0         1               -1

Set                   Incorrect Skipped Correct
Basic Problems B      0         12      0
Challenge Problems B  0         12      0

Answer 1

JSONを残りの部分から分離するのは非常に簡単です。これにより、JSON以外のみが提供されます。

python submit.py --provider gt --assignment error-check | sed '/{/,$d' 

そしてこれは、JSONのみです。

python submit.py --provider gt --assignment error-check | sed -n '/{/,$p' 

説明のために、入力例をfileとして保存しました。

$ sed '/{/,$d' file
Problem,Correct?,Correct Answer,Agent's Answer
"Challenge Problem B-04",0,4,-1
"Basic Problem B-12",0,1,-1
"Challenge Problem B-05",0,6,-1
"Challenge Problem B-07",0,6,-1
"Challenge Problem B-06",0,3,-1
"Basic Problem B-11",0,1,-1
"Basic Problem B-10",0,3,-1
"Challenge Problem B-03",0,3,-1
"Challenge Problem B-02",0,1,-1
"Challenge Problem B-01",0,6,-1
"Challenge Problem B-09",0,4,-1
"Challenge Problem B-08",0,4,-1
"Basic Problem B-08",0,6,-1
"Basic Problem B-09",0,5,-1
"Basic Problem B-04",0,3,-1
"Basic Problem B-05",0,4,-1
"Basic Problem B-06",0,5,-1
"Basic Problem B-07",0,6,-1
"Basic Problem B-01",0,2,-1
"Basic Problem B-02",0,5,-1
"Basic Problem B-03",0,1,-1
"Challenge Problem B-10",0,4,-1
"Challenge Problem B-11",0,5,-1
"Challenge Problem B-12",0,1,-1

そして

$ sed -n '/{/,$p' file
{
    "Basic Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Basic Problems B"
    },
    "Challenge Problems B": {
        "Incorrect": "0",
        "Skipped": "12",
        "Correct": "0",
        "Set": "Challenge Problems B"
    }
}

さて、あなたはすでに非JSONを完全にうまく扱っているので、私はそれを変更しません。理想的には、JSONデータはjqのようなJSONパーサーを使用して解析する必要があります。悲しいことに、これを適切に行うのに十分なjqがわからないので、私が思いつくことができる最善の方法は、この、かなりエレガントでないソリューションです。少なくともそれはあなたが望むことをします（cat fileをあなたのpython submit.py --provider gt --assignment error-checkコマンドに置き換えてください：

$ cat file | sed -n 's/[,"]//g; s/^ *//; /{/,$p'  | tac | awk -F': ' 'BEGIN{printf "%-30s%-10s%-10s%-10s\n", "Set", "Incorrect", "Skipped", "Correct"} NF==2 && !/\{/{if($1=="Set"){set=$2;data[set]["Incorrect"] = 0;data[set]["Skipped"] = 0;data[set]["Correct"] = 0;} data[set][$1]=$2}END{for(set in data){printf "%-30s%-10s%-10s%-10s\n", set,data[set]["Incorrect"],data[set]["Skipped"],data[set]["Correct"]}}' 
Set                           Incorrect Skipped   Correct   
Challenge Problems B          0         12        0         
Basic Problems B              0         12        0      

これらすべてをシェルスクリプトにまとめると、次のようになります。

#!/bin/bash

tmpFile=$(mktemp)
python submit.py --provider gt --assignment error-check > "$tmpFile";

sed '/{/,$d' "$tmpFile" | column -t -s, 
sed -n 's/[,"]//g; s/^ *//; /{/,$p' "$tmpFile" |
  tac |
  awk -F': ' '
    BEGIN{
      printf "%-30s%-10s%-10s%-10s\n", "Set", "Incorrect", "Skipped", "Correct"
    }
    NF==2 && !/\{/{
      if($1=="Set"){
         set=$2;
         data[set]["Incorrect"] = 0;
         data[set]["Skipped"] = 0;
         data[set]["Correct"] = 0;
      } 
      data[set][$1]=$2
    }
    END{
       for(set in data){
         printf "%-30s%-10s%-10s%-10s\n", set, 
                                     data[set]["Incorrect"], 
                                     data[set]["Skipped"], 
                                     data[set]["Correct"]}
    }' 
rm "$tmpFile"

これにより、次の出力が生成されます。

$ foo.sh
Problem                   Correct?  Correct Answer  Agent's Answer
"Challenge Problem B-04"  0         4               -1
"Basic Problem B-12"      0         1               -1
"Challenge Problem B-05"  0         6               -1
"Challenge Problem B-07"  0         6               -1
"Challenge Problem B-06"  0         3               -1
"Basic Problem B-11"      0         1               -1
"Basic Problem B-10"      0         3               -1
"Challenge Problem B-03"  0         3               -1
"Challenge Problem B-02"  0         1               -1
"Challenge Problem B-01"  0         6               -1
"Challenge Problem B-09"  0         4               -1
"Challenge Problem B-08"  0         4               -1
"Basic Problem B-08"      0         6               -1
"Basic Problem B-09"      0         5               -1
"Basic Problem B-04"      0         3               -1
"Basic Problem B-05"      0         4               -1
"Basic Problem B-06"      0         5               -1
"Basic Problem B-07"      0         6               -1
"Basic Problem B-01"      0         2               -1
"Basic Problem B-02"      0         5               -1
"Basic Problem B-03"      0         1               -1
"Challenge Problem B-10"  0         4               -1
"Challenge Problem B-11"  0         5               -1
"Challenge Problem B-12"  0         1               -1
Set                           Incorrect Skipped   Correct   
Challenge Problems B          0         12        0         
Basic Problems B              0         12        0         

しかし、それはハッキーな感じがします。誰かが専用のJSONパーサーを使ったよりクリーンなソリューションを考え出すことができるといいのですが。

---

Steeldriver は、コメントで適切なjqソリューションを提供するのに十分良かったので、それを組み込むと、はるかに単純（かつ安全）になります。

#!/bin/bash

tmpFile=$(mktemp)
python submit.py --provider gt --assignment error-check > "$tmpFile";

sed '/{/,$d' "$tmpFile" | column -t -s, 
sed -n '/{/,$p' "$tmpFile" | 
  jq -r '["Set","Incorrect","Skipped","Correct"], (.[] | [.Set,.Incorrect,.Skipped,.Correct]) | @tsv'
 rm "$tmpFile"

Answer 2

$ cat tst.awk
BEGIN { FS=","; OFS="\t" }
/{/ { FS="(^|\":)[[:space:]]+\"|\",?" }
FS == "," { $1=$1; print; next }
{ f[$2] = $3 }
/}/ {
    if ( !doneHdr++ ) {
        print "Set", "Incorrect", "Skipped", "Correct"
    }
    print f["Set"], f["Incorrect"], f["Skipped"], f["Correct"]
}

。

$ awk -f tst.awk file | column -s$'\t' -t
Problem                   Correct?   Correct Answer  Agent's Answer
"Challenge Problem B-04"  0          4               -1
"Basic Problem B-12"      0          1               -1
"Challenge Problem B-05"  0          6               -1
"Challenge Problem B-07"  0          6               -1
"Challenge Problem B-06"  0          3               -1
"Basic Problem B-11"      0          1               -1
"Basic Problem B-10"      0          3               -1
"Challenge Problem B-03"  0          3               -1
"Challenge Problem B-02"  0          1               -1
"Challenge Problem B-01"  0          6               -1
"Challenge Problem B-09"  0          4               -1
"Challenge Problem B-08"  0          4               -1
"Basic Problem B-08"      0          6               -1
"Basic Problem B-09"      0          5               -1
"Basic Problem B-04"      0          3               -1
"Basic Problem B-05"      0          4               -1
"Basic Problem B-06"      0          5               -1
"Basic Problem B-07"      0          6               -1
"Basic Problem B-01"      0          2               -1
"Basic Problem B-02"      0          5               -1
"Basic Problem B-03"      0          1               -1
"Challenge Problem B-10"  0          4               -1
"Challenge Problem B-11"  0          5               -1
"Challenge Problem B-12"  0          1               -1
Set                       Incorrect  Skipped         Correct
Basic Problems B          0          12              0
Challenge Problems B      0          12              0
Challenge Problems B      0          12              0

Answer 3

Miller（ https://github.com/johnkerl/miller ）を使用して実行している

# get the CSV and transform it into a pretty print table
<input grep -P '^("|\w)' | mlr --c2p cat >out
# add a carriage return
echo "" >> out
# convert the json into a pretty print table and add it to the output
<input grep -vP '^("|\w)'  | mlr --j2p cat -n then reshape -r "(Basi|Chal)" -o i,v \
then nest --explode --values --across-fields --nested-fs ":" -f i \
then reshape -s i_2,v \
then cut -x -f i_1,n \
then reorder -f Set >>out

あなたが持っているでしょう

Problem                Correct? Correct Answer Agent's Answer
Challenge Problem B-04 0        4              -1
Basic Problem B-12     0        1              -1
Challenge Problem B-05 0        6              -1
Challenge Problem B-07 0        6              -1
Challenge Problem B-06 0        3              -1
Basic Problem B-11     0        1              -1
Basic Problem B-10     0        3              -1
Challenge Problem B-03 0        3              -1
Challenge Problem B-02 0        1              -1
Challenge Problem B-01 0        6              -1
Challenge Problem B-09 0        4              -1
Challenge Problem B-08 0        4              -1
Basic Problem B-08     0        6              -1
Basic Problem B-09     0        5              -1
Basic Problem B-04     0        3              -1
Basic Problem B-05     0        4              -1
Basic Problem B-06     0        5              -1
Basic Problem B-07     0        6              -1
Basic Problem B-01     0        2              -1
Basic Problem B-02     0        5              -1
Basic Problem B-03     0        1              -1
Challenge Problem B-10 0        4              -1
Challenge Problem B-11 0        5              -1
Challenge Problem B-12 0        1              -1

Set                  Incorrect Skipped Correct
Basic Problems B     0         12      0
Challenge Problems B 0         12      0