---

??新智元報道??

test_dataset

Dataset({    features: ['name', 'eventrefnumber', 'text', 'StartDate', 'eventtype', 'province', 'citydistrict', 'village', 'targetgroup', 'commander', 'position', 'minkilled', 'mincaptured', 'capturedcharacterisation', 'killedcharacterisation', 'killq', 'captureq', 'killcaptureraid', 'airstrike', 'noshotsfired', 'dataprocessed', 'flagged', 'glossarymeta', 'minleaderskilled', 'minfacilitatorskilled', 'minleaderscaptured', 'minfacilitatorscaptured', 'leaderq'],    num_rows: 724})

[    IsafEvent(        name='5',        text='2013-01-S-025\n\nKABUL, Afghanistan (Jan. 25, 2013)\nDuring a security operation in Andar district, Ghazni province, yesterday, an Afghan and coalition force killed the Taliban leader, Alaudin. Alaudin oversaw a group of insurgents responsible for conducting remote-controlled improvised explosive device and small-arms fire attacks against Afghan and coalition forces. Prior to his death, Alaudin was planning attacks against Afghan National Police in Ghazni province.',        start_date=datetime.date(2013, 1, 24),        event_type={'insurgentskilled'},        province={'ghazni'},        target_group={'taliban'},        min_killed=1,        min_captured=0,        killq=True,        captureq=False,        killcaptureraid=False,        airstrike=False,        noshotsfired=False,        min_leaders_killed=1,        min_leaders_captured=0,        predictions={}    ),    IsafEvent(        name='2',        text='2011-11-S-034\nISAF Joint Command - Afghanistan\nFor Immediate Release\n\nKABUL, Afghanistan (Nov. 20, 2011)\nA coalition security force detained numerous suspected insurgents during an operation in Marjeh district, Helmand province, yesterday.  The force conducted the operation after receiving information that a group of insurgents were at a compound in the area.  After calling for the men inside to come out peacefully, the insurgents emerged and were detained without incident.',        start_date=datetime.date(2011, 11, 19),        event_type={'detention'},        province={'helmand'},        target_group={''},        min_killed=0,        min_captured=4,        killq=False,        captureq=True,        killcaptureraid=True,        airstrike=False,        noshotsfired=False,        min_leaders_killed=0,        min_leaders_captured=0,        predictions={}     )]

json_str = events[0].model_dump_json(exclude={"text", "predictions"})print(json_str)

{"name":"5","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":false,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}

from openai import OpenAIfrom rich import printimport jsonimport os

def query_openai(article_text: str, model: str) -> str:    query = (        f"The following is a press release issued by ISAF (formerly operating in Afghanistan):\n{article_text}\n\n"        "## Extraction request\n"        "Please extract the following information from the press release:\n"        "- The name of the event (summarising the event / text as a headline)\n"        "- The start date of the event\n"        "- The event type(s)\n"        "- The province(s) in which the event occurred\n"        "- The target group(s) of the event\n"        "- The minimum number of people killed during the event\n"        "- The minimum number of people captured during the event\n"        "- Whether someone was killed or not during the event\n"        "- Whether someone was captured or not during the event\n"        "- Whether the event was a so-called 'kill-capture raid'\n"        "- Whether an airstrike was used during the event\n"        "- Whether no shots were fired during the event\n"        "- The minimum number of leaders killed during the event\n"        "- The minimum number of leaders captured during the event\n\n"        "## Annotation notes:\n"        "- A 'faciliator' is not a leader.\n"        "- If a press release states that 'insurgents' were detained without further "        "details, assign a minimum number of two detained. Interpret 'a couple' as "        "two. Interpret 'several' as at least three, even though it may sometimes "        "refer to seven or eight. Classify the terms 'a few', 'some', 'a group', 'a "        "small group', and 'multiple' as denoting at least three, even if they "        "sometimes refer to larger numbers. Choose the smaller number if no other "        "information is available in the press release to come up with a minimally "        "acceptable figure. Interpret 'numerous' and 'a handful' as at least four, "        "and 'a large number' as at least five.\n\n"        "## Example:\n"        "Article text: 'ISAF Joint Command Evening Operational Update Feb. 19, 2011\nISAF Joint Command - "        "Afghanistan\u20282011-02-S-143\u2028For Immediate Release \u2028\u2028KABUL, Afghanistan (Feb. 19)\u2028\u2028ISAF "        "service members at a compound in Sangin district, Helmand province observed numerous insurgents north and south of "        "their position talking on radios today. After gaining positive identification of the insurgent positions, the "        "coalition troops engaged, killing several insurgents. Later, the ISAF troops observed more insurgents positioning "        "in the area with weapons. After positive identification, coalition forces continued firing on the various insurgent "        "positions, resulting in several more insurgents being killed.'\n\n"        'Output: `{"name":"Several insurgents killed in '        'Helmand","start_date":"2011-02-18","event_type":["insurgentskilled"],"province":["helmand"],"target_group":[""],"mi'        'n_killed":6,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":false,"airstrike":false,"noshotsfired"'        ':false,"min_leaders_killed":0,"min_leaders_captured":0}`'    )
    # set up the prediction harness    client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    response = client.chat.completions.create(        model=model,        response_format={"type": "json_object"},        messages=[            {                "role": "system",                "content": "You are an expert at identifying events in a press release. You are precise "                "and always make sure you are correct, drawing inference from the text of the "                "press release.\n\n You always return a JSON string with the following schema: "                "## JSON Schema details\n"                "Here is some of the schema for the JSON output string you "                "should make use of: event_types = ['airstrike', 'detention', "                "'captureandkill', 'insurgentskilled', 'exchangeoffire', 'civiliancasualty'], "                "provinces = ['badakhshan', 'badghis', 'baghlan', 'balkh', 'bamyan', "                "'day_kundi', 'farah', 'faryab', 'ghazni', 'ghor', 'helmand', 'herat', "                "'jowzjan', 'kabul', 'kandahar', 'kapisa', 'khost', 'kunar', 'kunduz', "                "'laghman', 'logar', 'nangarhar', 'nimroz', 'nuristan', 'paktya', 'paktika', "                "'panjshir', 'parwan', 'samangan', 'sar_e_pul', 'takhar', 'uruzgan', "                "'wardak', 'zabul'], target_groups = ['taliban', 'haqqani', 'criminals', "                "'aq', 'hig', 'let', 'imu', 'judq', 'iju', 'hik', 'ttp', 'other']\n\n",            },            {"role": "user", "content": query},        ],        temperature=1,    )
    return response.choices[0].message.content

json_str = query_openai(events[0].text, "gpt-4o")print(json.loads(json_str))

{    'name': 'Taliban leader Alaudin killed in Ghazni',    'start_date': '2013-01-24',    'event_type': ['insurgentskilled'],    'province': ['ghazni'],    'target_group': ['taliban'],    'min_killed': 1,    'min_captured': 0,    'killq': True,    'captureq': False,    'killcaptureraid': True,    'airstrike': False,    'noshotsfired': False,    'min_leaders_killed': 1,    'min_leaders_captured': 0}

print(events[0])

IsafEvent(    name='5',    text='2013-01-S-025\n\nKABUL, Afghanistan (Jan. 25, 2013)\nDuring a security operation in Andar district, Ghazni province, yesterday, an Afghan and coalition force killed the Taliban leader, Alaudin. Alaudin oversaw a group of insurgents responsible for conducting remote-controlled improvised explosive device and small-arms fire attacks against Afghan and coalition forces. Prior to his death, Alaudin was planning attacks against Afghan National Police in Ghazni province.',    start_date=datetime.date(2013, 1, 24),    event_type={'insurgentskilled'},    province={'ghazni'},    target_group={'taliban'},    min_killed=1,    min_captured=0,    killq=True,    captureq=False,    killcaptureraid=False,    airstrike=False,    noshotsfired=False,    min_leaders_killed=1,    min_leaders_captured=0,    predictions={'gpt-4o': '{\n  "name": "Taliban leader Alaudin killed in Ghazni",\n  "start_date": "2013-01-24",\n  "event_type": ["insurgentskilled", "captureandkill"],\n  "province": ["ghazni"],\n  "target_group": ["taliban"],\n "min_killed": 1,\n  "min_captured": 0,\n  "killq": true,\n  "captureq": false,\n  "killcaptureraid": true,\n  "airstrike": false,\n  "noshotsfired": false,\n  "min_leaders_killed": 1,\n  "min_leaders_captured": 0\n}',        'gpt-4-turbo': '{\n    "name": "Taliban leader Alaudin killed in Ghazni",\n    "start_date": "2013-01-24",\n    "event_type": ["captureandkill"],\n    "province": ["ghazni"],\n    "target_group": ["taliban"],\n    "min_killed": 1,\n    "min_captured": 0,\n    "killq": true,\n    "captureq": false,\n    "killcaptureraid": true,\n    "airstrike": false,\n    "noshotsfired": false,\n    "min_leaders_killed": 1,\n    "min_leaders_captured": 0\n}',        'gpt-3.5-turbo': '{\n    "name": "Taliban leader Alaudin killed in Ghazni province",\n    "start_date": "2013-01-24",\n    "event_type": ["captureandkill"],\n    "province": ["ghazni"],\n    "target_group": ["taliban"],\n    "min_killed": 1,\n    "min_captured": 0,\n    "killq": true,\n    "captureq": false,\n    "killcaptureraid": false,\n    "airstrike": false,\n    "noshotsfired": false,\n    "min_leaders_killed": 1,\n    "min_leaders_captured": 0\n}'    })

def convert_to_dataset(data: List[BaseModel]) -> Dataset:    dataset_dict = {}
    for field_name, field_value in data[0].__fields__.items():        field_type = field_value.outer_type_        if field_type in [str, int, float, bool, date]:            dataset_dict[field_name] = [getattr(item, field_name) for item in data]        elif field_type == set:            dataset_dict[field_name] = [list(getattr(item, field_name)) for item in data]        elif issubclass(field_type, BaseModel):            dataset_dict[field_name] = [getattr(item, field_name).dict() for item in data]        else:            dataset_dict[field_name] = [getattr(item, field_name) for item in data]
    dataset = Dataset.from_dict(dataset_dict)    return dataset

convert_and_push_dataset(events, "isafpressreleases_with_preds", split_name="test")

重新加載預(yù)測數(shù)據(jù)集

from datasets import load_dataset
preds_test_data = load_dataset("strickvl/isafpressreleases_with_preds")[    "test"].to_list()

微調(diào)TinyLlama的預(yù)測

from rich import print
print(preds_test_data[0])

{    'name': '5',    'text': '2013-01-S-025\n\nKABUL, Afghanistan (Jan. 25, 2013)\nDuring a security operation in Andar district, Ghazni province, yesterday, an Afghan and coalition force killed the Taliban leader, Alaudin. Alaudin oversaw a group of insurgents responsible for conducting remote-controlled improvised explosive device and small-arms fire attacks against Afghan and coalition forces. Prior to his death, Alaudin was planning attacks against Afghan National Police in Ghazni province.',    'predictions': {'gpt-3.5-turbo': '{\n    "name": "Taliban leader Alaudin killed in Ghazni province",\n    "start_date": "2013-01-24",\n    "event_type": ["captureandkill"],\n    "province": ["ghazni"],\n    "target_group": ["taliban"],\n    "min_killed": 1,\n    "min_captured": 0,\n    "killq": true,\n    "captureq": false,\n    "killcaptureraid": false,\n    "airstrike": false,\n    "noshotsfired": false,\n    "min_leaders_killed": 1,\n    "min_leaders_captured": 0\n}',        'gpt-4-turbo': '{\n    "name": "Taliban leader Alaudin killed in Ghazni",\n    "start_date": "2013-01-24",\n    "event_type": ["captureandkill"],\n    "province": ["ghazni"],\n    "target_group": ["taliban"],\n    "min_killed": 1,\n    "min_captured": 0,\n    "killq": true,\n    "captureq": false,\n    "killcaptureraid": true,\n    "airstrike": false,\n    "noshotsfired": false,\n    "min_leaders_killed": 1,\n    "min_leaders_captured": 0\n}',        'gpt-4o': '{\n  "name": "Taliban leader Alaudin killed in Ghazni",\n  "start_date": "2013-01-24",\n  "event_type": ["insurgentskilled", "captureandkill"],\n  "province": ["ghazni"],\n  "target_group": ["taliban"],\n "min_killed": 1,\n  "min_captured": 0,\n  "killq": true,\n  "captureq": false,\n  "killcaptureraid": true,\n  "airstrike": false,\n  "noshotsfired": false,\n  "min_leaders_killed": 1,\n  "min_leaders_captured": 0\n}',        'tinyllama-templatefree': '\n{"name":"Taliban leader killed in Ghazni","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":false,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}',        'tinyllama-sharegpt': '{"name":"2","start_date":"2013-01-24","event_type":["airstrike"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":false,"airstrike":true,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}'    },    'start_date': datetime.date(2013, 1, 24),    'province': ['ghazni'],    'target_group': ['taliban'],    'event_type': ['insurgentskilled'],    'min_killed': 1,    'min_captured': 0,    'killq': True,    'captureq': False,    'killcaptureraid': False,    'airstrike': False,    'noshotsfired': False,    'min_leaders_killed': 1,    'min_leaders_captured': 0}

微調(diào)Mistral的預(yù)測

微調(diào)OpenAI的預(yù)測

微調(diào)Mistral模型（通過OpenPipe）

微調(diào)Solar LLM（通過Predibase）

微調(diào)Llama3的預(yù)測（通過OpenPipe）

from rich import print
print(preds_test_data[0])

{    'name': '5',    'text': '2013-01-S-025\n\nKABUL, Afghanistan (Jan. 25, 2013)\nDuring a security operation in Andar district, Ghazni province, yesterday, an Afghan and coalition force killed the Taliban leader, Alaudin. Alaudin oversaw a group of insurgents responsible for conducting remote-controlled improvised explosive device and small-arms fire attacks against Afghan and coalition forces. Prior to his death, Alaudin was planning attacks against Afghan National Police in Ghazni province.',    'predictions': {'finetuned-llama3-7b-32k-openpipe': '{"name":"1","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":true,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}',        'finetuned-mistral-7b-optimised-openpipe': '{"name":"1","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":true,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}',        'finetuned-openai-gpt-3.5-turbo-1106': '{"name":"4","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":true,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}',        'gpt-3.5-turbo': '{\n    "name": "Taliban leader Alaudin killed in Ghazni province",\n    "start_date": "2013-01-24",\n    "event_type": ["captureandkill"],\n    "province": ["ghazni"],\n    "target_group": ["taliban"],\n    "min_killed": 1,\n    "min_captured": 0,\n    "killq": true,\n    "captureq": false,\n    "killcaptureraid": false,\n    "airstrike": false,\n    "noshotsfired": false,\n    "min_leaders_killed": 1,\n    "min_leaders_captured": 0\n}',        'gpt-4-turbo': '{\n    "name": "Taliban leader Alaudin killed in Ghazni",\n    "start_date": "2013-01-24",\n    "event_type": ["captureandkill"],\n    "province": ["ghazni"],\n    "target_group": ["taliban"],\n    "min_killed": 1,\n    "min_captured": 0,\n    "killq": true,\n    "captureq": false,\n    "killcaptureraid": true,\n    "airstrike": false,\n    "noshotsfired": false,\n    "min_leaders_killed": 1,\n    "min_leaders_captured": 0\n}',        'gpt-4o': '{\n  "name": "Taliban leader Alaudin killed in Ghazni",\n  "start_date": "2013-01-24",\n  "event_type": ["insurgentskilled", "captureandkill"],\n  "province": ["ghazni"],\n  "target_group": ["taliban"],\n "min_killed": 1,\n  "min_captured": 0,\n  "killq": true,\n  "captureq": false,\n  "killcaptureraid": true,\n  "airstrike": false,\n  "noshotsfired": false,\n  "min_leaders_killed": 1,\n  "min_leaders_captured": 0\n}',        'mistral-lora-templatefree': '1',        'tinyllama-sharegpt': '{"name":"2","start_date":"2013-01-24","event_type":["airstrike"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":false,"airstrike":true,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}',        'tinyllama-templatefree': '\n{"name":"Taliban leader killed in Ghazni","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":false,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}',        'ft-solar-1-mini-chat-240612-predibase': '\n\n{"name":"2","start_date":"2013-01-24","event_type":["insurgentskilled"],"province":["ghazni"],"target_group":["taliban"],"min_killed":1,"min_captured":0,"killq":true,"captureq":false,"killcaptureraid":true,"airstrike":false,"noshotsfired":false,"min_leaders_killed":1,"min_leaders_captured":0}'    },    'start_date': datetime.date(2013, 1, 24),    'province': ['ghazni'],    'target_group': ['taliban'],    'event_type': ['insurgentskilled'],    'min_killed': 1,    'min_captured': 0,    'killq': True,    'captureq': False,    'killcaptureraid': False,    'airstrike': False,    'noshotsfired': False,    'min_leaders_killed': 1,    'min_leaders_captured': 0}

JSON有效性測試

from datasets import load_dataset
dataset_with_preds = load_dataset("strickvl/isafpressreleases_test_predictions")[    "train"].to_list()

# find out how many of the predictions are None values or empty stringsmissing_values = {    "gpt-4o": 0,    "gpt-4-turbo": 0,    "gpt-3.5-turbo": 0,    "tinyllama-templatefree": 0,    "tinyllama-sharegpt": 0,    "finetuned-openai-gpt-3.5-turbo-1106": 0,    "finetuned-llama3-7b-32k-openpipe": 0,    "mistral-lora-templatefree": 0,    "finetuned-mistral-7b-optimised-openpipe": 0,    "ft-solar-1-mini-chat-240612-predibase": 0,}
for row in dataset_with_preds:    for model in row["predictions"]:        if row["predictions"][model] is None or row["predictions"][model] == "":            missing_values[model] += 1
print(missing_values)

{    'gpt-4o': 0,    'gpt-4-turbo': 0,    'gpt-3.5-turbo': 0,    'tinyllama-templatefree': 0,    'tinyllama-sharegpt': 38,    'finetuned-openai-gpt-3.5-turbo-1106': 0,    'finetuned-llama3-7b-32k-openpipe': 0,    'mistral-lora-templatefree': 0,    'finetuned-mistral-7b-optimised-openpipe': 0,    'ft-solar-1-mini-chat-240612-predibase': 0}

• start_date

• province

• target_group

• event_type

• min_killed

• min_captured

• killq

• captureq

• killcaptureraid

• airstrike

• noshotsfired

• min_leaders_killed

• min_leaders_captured

開始日期準(zhǔn)確性

{    'gpt-4o': 527,    'gpt-4-turbo': 522,    'gpt-3.5-turbo': 492,    'tinyllama-templatefree': 231,    'tinyllama-sharegpt': 479,    'finetuned-openai-gpt-3.5-turbo-1106': 646,    'finetuned-llama3-7b-32k-openpipe': 585,    'mistral-lora-templatefree': 0,    'finetuned-mistral-7b-optimised-openpipe': 636,    'ft-solar-1-mini-chat-240612-predibase': 649}

省份準(zhǔn)確性

{    'gpt-4o': 649,    'gpt-4-turbo': 645,    'gpt-3.5-turbo': 595,    'tinyllama-templatefree': 335,    'tinyllama-sharegpt': 660,    'finetuned-openai-gpt-3.5-turbo-1106': 704,    'finetuned-llama3-7b-32k-openpipe': 707,    'mistral-lora-templatefree': 0,    'finetuned-mistral-7b-optimised-openpipe': 711,    'ft-solar-1-mini-chat-240612-predibase': 704}

目標(biāo)群體準(zhǔn)確性

事件類型準(zhǔn)確性

min_killed準(zhǔn)確性

標(biāo)注說明：‘facilitator’ 不是領(lǐng)導(dǎo)者。如果新聞稿中提到‘叛亂分子’被拘留而沒有進(jìn)一步細(xì)節(jié)，則分配至少兩名被拘留者。將‘a(chǎn) couple’解釋為兩人。將‘several’解釋為至少三人，盡管有時可能指七或八人。將‘a(chǎn) few’、‘some’、‘a(chǎn) group’、‘a(chǎn) small group’和‘multiple’解釋為至少三人，即使有時它們指代更多人數(shù)。如果新聞稿中沒有其他信息來提供一個最低可接受的數(shù)字，請選擇較小的數(shù)字。將‘numerous’和‘a(chǎn) handful’解釋為至少四人，而‘a(chǎn) large number’解釋為至少五人。

min_captured準(zhǔn)確性

killq準(zhǔn)確性

captureq準(zhǔn)確性

killcaptureraid準(zhǔn)確性

airstrike準(zhǔn)確性

noshotsfired準(zhǔn)確性

min_leaders_killed準(zhǔn)確性

min_leaders_captured準(zhǔn)確性

• 數(shù)據(jù)隱私（不會將你的信息發(fā)送給 OpenAI）
• 更小的模型很可能意味著更好的性能（盡管我仍需測試和證明這一點）
• 整體上更多的控制
• 成本改進(jìn)

評估過程很痛苦

但可以了解是否在進(jìn)步