Introduction of Evaluation Functions

  1. URL exact match

If the webpage URL and the answer are exactly the same, you can get 1 point, otherwise 0 points

Because the main URL may not be https://store.steampowered.com/, it could be the non-http version store.steampowered.com, or store.steampowered.us with country domains like us, cn, jp, etc(of course, this URL may not exist). Therefore, when comparing the entire URL, try not to use URL exact match.

  1. URL exact match (param)

First, parse the url according to param (for example https://www.google.com/search?q=steam&newwindow=1 , when param=q, return the value of param steam).

Then compare the analyzed Steam with the answer, and get one point if they are completely consistent, otherwise 0 points

  1. URL include match

If the webpage URL contains the answer, you can get 1 point, otherwise 0 points.

For example: https://store.steampowered.com/app/570/Dota_2/ , the answer is app/570/Dota_2/ , the url contains the answer so it gets one point

  1. URL include match (param)

Same as (2), compare using the include function after parsing the URL

  1. URL semantic match

To semantically match the URL, you can enter a matching command, which is a language description used to describe how it can be considered semantic matching compared to the answer.Explanation and examples:

  • The semantics of "red clothes" and "green clothes" are similar when the task is "search for a piece of clothing", but different when the task is "search for a piece of red clothing". Therefore, when the task is "search for a piece of red clothing", the matching instruction input here needs to clearly describe all key features of the answer: the information that needs to be filled here is the red clothing.

  • If when searching for "a piece of clothing", the matching command can be "Decide whether are searching for clothes", so that both red and green clothes can get scores.

  • If searching for "a red dress", the matching command can be "Decide whether are searching for red clothes", so that only searching for red clothes can score points.

  • If when searching for "looking for coffee shops in New York", the destination content of the URL is coffee bar, and the matching command is "Decide wether the destination is coffee shop", and then hand it over to AI for comparison, even if the search content is coffee bar, it can still score points

  • Generally speaking, the following sentence structure can be used: 'Decide wether are searching for xxx' or'Decide whether the name/destination/goal is xxx '.

  1. URL semantic match (param)

After parsing the URL, perform semantic matching. You can enter a matching command , and the specific method is described in (5).

  1. Element path exact match

Extract the selector path of the element, locate the element, and score one point if they are exactly the same.

  1. Element value exact match

Extract the text content of the target element, then match it with the answer, and get one point for complete consistency.

  1. Element value include match

Same as (8), change the evaluation method to include.

  1. Element value semantic match

Same as (8), the evaluation method is changed to semantic matching. See (5) for specific description of the method.

PreviousHow to Define Key Nodes & Evaluation FunctionsNext3. Evaluation

Last updated