getName.ai Benchmark 11.2026. ETIM classification with an F1 score of 93.8%

Customers almost always ask us the same thing: how accurate is getName.ai and how good are the attribute values it returns? The honest answer is not simple and, as usual in such cases, it starts with "it depends". On what the product description contains, on the classification, on the kind of attribute.

We decided to take that question on. We built a test that measures quality as objectively as we could, and we open it to anyone who wants to check it. The descriptions, the ideal answer, every scored value and the full statistics are available for download at the end of this article, so anyone can verify the result or repeat the test. This is the first edition of the getName.ai Quality Benchmark. We will repeat it and compare future editions with this one.

In the ETIM classification, getName.ai fills in attributes from the product description with a quality of 93.8%. In Amazon the quality was 93.1%, and in Kaufland 91.7%. That is the result of a test on 100 products from 17 categories, with descriptions in Polish, German and English.

We measure quality with the F1 score, the standard measure for evaluating information extraction from text. It combines precision, i.e. how many of the values the API entered are true, with recall, i.e. how many of the values that follow from the description the API entered. An empty answer scores 0%, an ideal one 100%.

A quality of 93.8% does not mean that 93.8% of product records are ready without corrections. It means that the API's answers match the ideal answer to that degree, counting omitted and wrongly entered values together. The test does not measure time or cost of work, and it does not compare the API with a human.

Below we explain why such a measurement is difficult, how we calculated the result, where the API makes mistakes and what quality depends on.

• Quality by classification: ETIM 93.8%, Amazon 93.1%, Kaufland 91.7%.
• Precision and recall are close: in ETIM 93.9% and 93.7%, in Amazon 92.6% and 93.7%, in Kaufland 92.0% and 91.3%. In ETIM and Kaufland the API omits a value more often than it enters a wrong one, in Amazon the other way round.
• Values stated explicitly in the description are entered with a recall from 96.0% in Kaufland to 98.0% in ETIM. Values that have to be inferred from the description are entered less often (79.7%). The API most often omits inferred yes/no features, e.g. that a kettle is cordless.
• The description language matters little: from 92.6% for Polish to 93.8% for English.
• Median quality of a single answer: ETIM 95.2%, Amazon 94.0%, Kaufland 92.3%.

The result depends on the description, not only on the API. The API can only enter what is in the description. A short three-sentence offer gives fewer features than a technical data sheet, so the API can enter fewer values from it. If the test counted how many attributes of a class were filled in, it would measure the description, not the API. That is why we prepared a separate ideal answer, the reference, for every description: the complete set of attribute values that follow from that description. Across all descriptions there were 7,994 of them, and we compared the API's answers with them. We call them required values: since the description provides them, the API should enter them. "Required" here does not mean mandatory in the classification or on the marketplace.

Not everything is stated explicitly. A kettle that "switches off when lifted from the base" is cordless, even though the word does not appear in the description. A product sold individually has a quantity of 1. But dimensions "410 × 230 × 195 mm" without axis labels do not say which of them is the width. We drew the line between a certain inference and a guess separately for each such value.

The same field means different things in different classifications. "Floor area" on the Polish and German Amazon is the area the product occupies; on the British Amazon it is the size of the room an air purifier serves. Scoring must know what a field means, not only its name.

The reference is an ideal, not a human. We prepared the ideal answer without time pressure and checked it many times. Nobody fills in product records like that day to day. The proper benchmark would be a human doing the same work under real conditions: after several hours of repetitive work, with the fatigue and boredom that come with it, and often without deep knowledge of the industry the product comes from. Nobody has claimed they would fill in attributes with 100% quality. We do not have such a measurement yet, so the result in this article says how far the API is from the ideal, not from a human.

getName.ai receives a product description and a class in the selected classification. It returns the values of that class's attributes in the classification's format: picklist values, numbers in the field's unit, yes/no, ranges and text. The test covered exactly this step. We provided the class up front, so this test does not measure class recognition.

The description and the classification were always in the same language:

Kaufland has no English version, so the English descriptions went only to ETIM and Amazon. Each request is one description and one class. The test comprised 672 API requests:

The number of products differs because we scored only products that the class certainly fits (more in the reference rules below). The API's answers produced 10,703 scored items, i.e. values from the reference and values returned by the API. The median time of a single call was 3.2 seconds, and 90% of calls took less than 6.5 seconds.

We sent the ETIM and Kaufland requests on 24 October 2026, and the Amazon requests on 6 November.

A bullet-point description (Polish original, translated):

Bosch DesignLine TWK3P420 electric kettle – stainless steel / black, 1.7 l, 2400 W

• Fast boiling thanks to 2400 W and a concealed flat heating element – the interior is easy to keep clean
• Capacity 1.7 l (minimum fill 0.5 l), clearly visible water level indicator on the outside of the housing
• Brushed stainless steel housing with an anti-fingerprint coating, black handle and base
• Removable limescale filter in the spout – no limescale particles in your cup
• Safety: automatic switch-off after boiling, switch-off when lifted from the base, protection against switching on without water
• Lid opened with a button, wide opening for filling and cleaning, power indicator light
• No temperature control and no keep-warm function

Technical data: voltage 220–240 V, 50/60 Hz
• cord length 75 cm
• dimensions (H × W × D) 27.5 × 21.6 × 15 cm
• weight 1.2 kg

The ideal answer for the kettle class in ETIM has 15 values. This is how it compared with the API's answer to the Polish description (ETIM attribute names and values in English):

From this comparison we calculate three measures:

• hits: 13, i.e. correct values;
• precision = hits / values returned = 13 / 14 = 92.9%. The API entered 14 values, and one of them, the reservoir material, is a guess;
• recall = hits / required reference values = 13 / 15 = 86.7%. The API omitted two values that follow from the description;
• quality (F1) = 2 × hits / (values returned + required reference values) = 2 × 13 / (14 + 15) = 89.7%.

The same description in the other classifications (attribute names in English):

For this product, answer quality was 93.9% in Amazon and 86.7% in Kaufland. In English, ETIM scored 93.3% and Amazon 93.5%.

Scoring categories

In a required field, a wrong value or a guess counts twice: as a returned value without a hit and as a reference value without a hit. An optional value, or a value outside the reference, entered correctly counts neither for nor against. We count a guess the same as a wrong value. It may be true, but in a catalogue it looks like a fact, and nobody has checked it.

A list gets the share of a hit to which it matches the reference. Example: the reference has material [aluminium], and the API returned [aluminium, titanium]. Half of the returned list is correct and the whole reference is contained in it, so the hit is 0.67.

Measures

• Precision = hits / values returned by the API. It says how much of what the API entered is true.
• Recall = hits / required reference values. It says how much of the ideal answer the API delivered.
• Quality = 2 × hits / (values returned + required reference values). This is the F1 score, i.e. the harmonic mean of precision and recall.

F1 is the standard measure for evaluating information extraction from text and in language model benchmarks. It is used, for example, by SimpleQA, OpenAI's benchmark of factual knowledge in language models. Only an ideal answer scores 100%. An empty answer and a completely wrong answer score 0%, and the result never drops below zero. Guessing does not pay off: a value the description does not provide lowers precision and does not raise recall.

We count rounding a value to fewer significant digits as correct, e.g. 0.353 m written as 0.35 m. getName.ai writes numbers this way on purpose. Across the whole test this concerned 16 values.

Who scores

• Structured fields are scored by a script. It compares numbers (with a tolerance of 0.5%), yes/no values, ranges and picklist values. Here the result is unambiguous: the value matches the reference or it does not.
• Text fields and values outside the reference are scored by a judge, the Claude Opus 5.5 language model by Anthropic. The judge sees the description and compares meaning, not wording. "Stal szlachetna, czarny" means the same as "stainless steel / black". It must not use knowledge about the specific product, and general knowledge only when it is true for every product of that kind. In Amazon the judge also receives Amazon's instructions for sellers, because they, not the name alone, define what a field means. The judge's full prompt is in the downloads.

Each kind of error with an example from the test (Polish descriptions, quotes translated, attribute names in English).

Missing required value

The description provides the value, and the API did not enter it.

Missing required value, inferred

The description does not state the value explicitly, but it follows from it with certainty, and the API did not enter it.

Wrong value

The API entered a value that is false or does not fit the meaning of the attribute.

The description gives safety features, while the attribute is about instructions. The value is true, but it went into an attribute with a different meaning.

Guess

The API entered a possible value that does not, however, follow from the description with certainty.

Wrong list element

The attribute takes several picklist values, and the API added a wrong value to the correct ones.

The description mentions titanium only for the coating, not as the material of the pan.

Incomplete value

The API entered a value that is true but clearly incomplete.

The API gave the battery chemistry, but without the voltage and capacity.

How often each kind of error occurs

This is how the gap to 100% splits between the kinds of errors across all classifications:

The number of values and the share of the gap do not go hand in hand, because the kinds of errors weigh differently. A wrong value in a required field counts twice: as a returned value without a hit and as a missing reference value. An incomplete value counts only in a required field (81 of 133) and gets half a hit, and a list loses only the part in which it differs from the reference.

Omitted required values make up the largest part of the gap. Three quarters of them (320 of 440) are inferred values that the description does not state explicitly, most often in yes/no fields, such as stepless power control in a mixer with four speeds. The API almost always enters a quantity of 1 for a product sold individually: of 849 such fields in Amazon (number of items, units, packs and package quantity) it filled in 840 correctly.

The API enters guesses and wrong values mainly in fields the description says nothing about or that we do not require: that is the case in 477 of 506 instances. Among the 7,994 required values the API entered a wrong value or a guess in 81 cases, including 52 lists with a wrong or guessed element.

Among wrong values the most common is a true value entered in a field with a different meaning, as in the angle grinder example: that applies to 141 of 212 (we assigned wrong values to patterns with the help of the Claude model). Other examples: a capacity of 75 ml in a dimensions field, the green colour of a laser beam as the product colour, a screen diagonal in a field that on the British Amazon means supported colours.

The same description yields a different set of fields in each classification. Classifications differ in the number of fields, their type and how often a value has to be inferred from the description, and Kaufland did not receive English descriptions. The differences in quality should therefore not be read as a comparison under the same conditions: each classification sets the API a different task.

• ETIM is a technical classification. Every field has a precisely defined meaning, unit or picklist, and there are no text fields at all. It has the fewest inferred values. ETIM loses most on yes/no fields, where the API omits features that follow from the description, and on picklists, where it sometimes picks a typical value the description does not give.
• Amazon has the most fields per product, and more than a quarter of the values in its reference are inferred, most often the quantity. The API enters such values most often here: recall of inferred values is 86.9% in Amazon, 61.4% in Kaufland and 58.8% in ETIM. Field names are often generic, and their meaning is defined only by Amazon's instructions for sellers. Omitted required values and guesses each account for about a third of the gap to 100% in Amazon.
• Kaufland has the fewest fields per product, but one in three is a text field. Text fields do worse there (88.7%) than structured ones (93.2%). Wrong values account for a larger part of the gap to 100% here than in the other classifications, and 46 of 55 of them are values entered in a field with a different meaning, such as "Safety instructions" filled with safety features.

The ETIM, Amazon and Kaufland columns show quality. A dash means the classification has no fields of that type. The last three columns cover all classifications combined.

The field type itself changes little. The API enters values stated explicitly in the description with a recall from 93.8% (text) to 100% (range) for every type. The differences between types come from how often a value of that type has to be inferred from the description and how hard that inference is. Descriptions almost always give dimensions, power and capacity as numbers, so decimal numbers do well in every classification. Integers are mostly quantities, which descriptions usually leave out because they are obvious. Almost four in five of them have to be inferred, but the inference is simple: for a product sold individually the quantity is 1. The API almost always enters it, so integers have the highest quality (98.0%). Yes/no fields do worst (86.1%): "no temperature control" is a value stated explicitly, but "cordless" has to be inferred from the description of the base, and the API draws such inferences in only 60.7% of cases.

Kinds of errors by field type:

The shares in the columns from "correct" to "incomplete" are calculated from values returned, and the missing required value from required reference values. The table covers all classifications combined; a table for each classification separately is in the statistics file. A multi-select list counts here as one value: of 51 wrong lists, 36 have one bad element next to correct ones.

• Numbers returned by the API are the most reliable: about 97% correct, few guesses and errors.
• Yes/no fields are most often left empty: the API omits 17.7% of required values, compared with 1.6% to 7.2% in the other types.
• Single-value picklists are rarely wrong (1.5%) but more often guessed (8.5%). The API picks a typical value the description does not give.
• Multi-select lists are least often entirely correct (84.7%), because one wrong or missing element is enough. A partially correct list does get part of a hit, though, so in quality they do well: 92.2% in Amazon and 94.7% in Kaufland.
• In text, the most common error is an incomplete value (5.3%), and text has the most wrong values in absolute terms (89).

By language

The differences between languages are small: up to 0.9 percentage points in structured fields and up to 1.8 points in text fields. Text fields in Polish Kaufland do worst (87.2%). The comparison is not entirely even, because English descriptions did not go to Kaufland.

By product category

Tools (95.0%), garden, and home and kitchen (94.5% each) score highest. Small kitchen appliances (92.2%), toys (91.1%) and beauty (90.0%) score lower. Some categories have only 1-4 products in the test, so we treat their results as indicative. The full table is in the statistics file.

Distribution of single-answer results

• Fridge, chaotic shop export. The description gives three capacities: fridge 260 l, freezer 103 l, total 363 l. The API entered each into the right field in all three classifications. It converted the dimensions 203 × 60 × 66.5 cm to millimetres for ETIM and to metres for Kaufland, and arranged them in the order D × W × H for Amazon. It did not use the package dimensions and weight that stood right next to them.
• Drill driver, short three-sentence offer. Two gears of 0-500 and 0-1900 rpm went into the first and second gear fields. With two 3.0 Ah batteries the API gave the capacity of one battery (3.0 Ah), not the sum, and their number (2) separately. The 1.5-13 mm chuck range did not end up in the drilling diameter fields for steel and wood. It missed one trap: in ETIM it entered 50 Nm in the torque adjustment range field, although the description gives only the maximum torque and 21 clutch settings.
• TV. Energy consumption given in kWh per 1000 hours was not entered as annual consumption. The API converted the 55-inch diagonal to 139 cm.

The API reads numbers, units and explicitly stated features very accurately: recall of explicitly stated values is from 96.0% in Kaufland to 98.0% in ETIM. The gap to 100% comes mainly from three things, and none of them is a misreading of what the description states explicitly:

• values that follow from the description although they are not written in it, e.g. that a kettle is cordless. The API omits them more often than it enters them wrongly;
• answers in fields the description says nothing about, most often values typical for that kind of product;
• confusing the meaning of a field, e.g. safety instructions with safety features, where the attribute name alone does not make clear what the field is about.

We are working on all three.

What does this mean in practice? Below are our conclusions from the numbers in this article, not the result of a separate measurement:

• the API enters numbers given in the description most reliably, including unit conversion. About 97% of the numbers it returned are correct;
• yes/no fields and single-value picklists, where the API guesses more often, need the most attention, as do text fields whose meaning cannot be told from the name alone;
• features obvious to a human, such as that a kettle is cordless, are worth writing into the description. The API enters explicitly stated values with a recall from 96.0% to 98.0%, and inferred ones less often (79.7%).

All test files are available for download, so anyone can carry out the comparison with a human that we mention at the beginning. If you do, write to us.

100 real products (a specific brand and model) from 17 categories of a broad marketplace. They include electrical installation and lighting, tools, small and large household appliances, electronics, garden, plumbing and heating, home and kitchen, fashion, beauty, toys, sports, baby products, pets, automotive and food. The technical data comes from manufacturer data sheets and large shops.

5 description styles, 20 products each, because descriptions in real catalogues look very different:

3 languages: Polish, German and English. Each description is written the way a local seller would write it, not as a translation. The facts are the same in all languages. The average description length is about 750-800 characters.

The descriptions were prepared by AI agents following written instructions, based on the source data of each product. The same agents compiled the ideal answer (more on it below).

409 traps. We deliberately placed spots in the descriptions where it is easy to make a mistake:

• unit conversions, e.g. dimensions in centimetres when ETIM expects millimetres and Kaufland metres;
• negations, e.g. "no temperature control" or "not dishwasher safe";
• several similar numbers at once, e.g. three fridge capacities or two drill gears;
• decoy numbers, e.g. package dimensions and weight, warranty period or delivery time;
• a range where the field takes a single number;
• mentions of compatible products that are not a feature of the product itself.

For each product and each classification we prepared a full list of attribute values that follow from the description. That is what we compared the API's answers with.

Rules

• The reference contains only what follows from the description. Knowledge about the specific product from outside the description does not count. If the description does not give the colour, the reference has no colour, even if we know what it is.
• A certain value belongs in the reference, a guess does not. A value does not have to be written in the description explicitly. It also belongs in the reference when it follows from the description with certainty: from a conversion, from the kind of product (a straightener gives straight hair, a power strip with type F sockets has a type F plug), from a feature true for every product of that kind (a cosmetic has an expiry date) or from the fact that the product is sold individually (quantity 1). There is no general rule here, so we assessed each such value separately. A value that is merely typical or likely, e.g. "lightweight", "classic style" or taking one of the unlabelled dimensions as the width, is a guess.
• Required and optional values. Required values are those the description gives or that follow from it with certainty. "Required" means only that, since the description provides the value, the API should enter it. It is not about attributes that are mandatory in the classification. An attribute the description says nothing about has no value in the reference and does not affect the result, as long as the API does not fill it in. Optional are only generic fields (e.g. style or a descriptive size), fields where the description allows several readings (e.g. which code denotes the product version), and approximate equivalents in a picklist. A missing optional value is not an error.
• Alternative values. When there are several correct answers (e.g. 230 V or a range of 220-240 V), the reference accepts each of them.
• Certain classes only. We scored attributes only when the class certainly fits the product. ETIM does not cover clothing, toys or food, so we scored 66 products in ETIM, 93 in Kaufland and 96 in Amazon.
• The test measures reading the description, not the quality of the classifications themselves. Where the classification's definition made a correct answer difficult (e.g. a picklist without a matching value or an unclear label), the field was optional.
• No search-optimisation fields. Amazon's "Generic Keyword" and "Subject Keyword" are phrases customers search for, not features of the product. The test does not score them.

How it was created

The reference was prepared by AI agents following written instructions. Each of 10 author agents worked on 10 products: descriptions in three languages and the ideal answer for each classification. Then 6 other agents, who were not authors, reviewed everything and made 193 corrections. Rules and disputed cases, e.g. the line between a certain value and a guess, were decided by a human.

After the first scoring the reference went through four more rounds of checks:

• Gaps. When the API returned a value the reference did not have, and the judge found that it follows from the description, the value was added to the reference. After the first scoring there were 675 such values, most of them optional. We repeated the same check after subsequent Amazon runs, with the exception described below.
• Consistency across languages. The same field of the same product must get the same verdict in every language. We resolved every discrepancy in the reference rather than leaving it to the judge.
• Field meaning according to Amazon. Amazon's instructions for sellers showed that some fields mean something other than the name suggests. "Sub Brand" also covers a licence, e.g. Minecraft on a puzzle. We reviewed the Amazon reference field by field according to these instructions and to the Amazon picklists in getName.ai as of 27 October
• Together with the gaps we changed or added 393 fields in 95 products, 246 of them new. In each marketplace a field has the meaning given in that marketplace's instructions.
• Inferred values. Values that the description does not state explicitly but that follow from it were initially optional. We checked each of them, 674 in total, separately for whether it follows from the description with certainty. 568 are now required, 84 we classed as guesses, and 22 remained optional (generic fields, approximations in a picklist and lists where only some elements are certain). The check was done by the Claude Opus 5.5 model according to rules set by a human.

• The sample is enough to compare classifications and field types, but not narrower breakdowns. 100 products and 7,994 required values are enough to compare classifications and attribute types. A single product category has between 1 and 14 products in the test, so we treat category results as indicative.
• The class was given up front. The test does not measure class recognition, and in practice a wrong class means wrong attributes.
• The descriptions were written for the test based on real product data. They are meant to reflect the variety of real catalogues, but real descriptions are sometimes even shorter and poorer.
• The descriptions and the reference were prepared by AI agents under human control, and text fields are scored by an LLM judge. The reference went through an independent review and a consistency check, but it may contain individual errors. That is why we publish it in full.
• We added to the reference after seeing the API's answers. In the gaps round we added values that the API returned and the judge found to follow from the description (more in "How it was created"). We do not have a separate sample for which the reference was created without insight into the API's answers. This round worked both ways: an attribute added as required lowers the result in the languages where the API did not fill it in. We checked inferred values without the API's answers.
• An LLM judge is also a model. When re-scoring the same answers (after its instructions were refined) it changed about a dozen of roughly 3,000 text field verdicts, mainly between "incomplete" and "correct".
• One run. The main test is a single run. We repeated Amazon two more times, and across the three runs quality was from 93.0% to 93.1% in PL, from 92.7% to 92.9% in DE and from 93.4% to 93.7% in UK. We did not repeat ETIM and Kaufland.
• Three classifications and three languages out of the many that getName.ai supports. On other classifications the result may differ.

Everything in one archive: pakiet_testowy_getname.zip (1.8 MB). Or separately (labels and column values inside the files are partly in Polish):

Columns of wyniki_atrybuty.csv (separator: semicolon):

The quality of any group of rows is 2 × sum of hit / (sum of gold_count + sum of returned_count).

How to repeat the test

• Take a description from opisy_<language>.csv and the class from wzorzec.xlsx for the same classification (column "klasa", only rows with "klasa pewna" = "tak", i.e. certain class = yes).
• Send them to getName.ai and save the returned attributes.
• Compare them with the reference value, the alternative values and the "required" field according to the rules described above. Score text fields and values outside the reference with the help of prompt_sedziego.txt, for Amazon together with Amazon's instructions for sellers for the given attribute. We do not include those instructions, because they are Amazon content made available to sellers.
• Calculate precision, recall and quality using the formulas in "Measures".

If you find an error in the reference, write to us. We will fix it and recalculate the result.