Apparently there are items and plugins out there that still use legacy
color codes within text components, and which thereby break this
heuristic.
Our remaining approach to differentiate between legacy (plain) and
modern (JSON-based) text is to check if a particular text can be parsed
as JSON-based text. This approach is not perfect either as there are
ambiguous cases that it cannot resolve correctly. However, these cases
are hopefully rare enough in practice that this approach remains
suitable.
For example, this affects empty Strings, and Strings that consist only
of whitespace. In lenient mode this would also affect text that contains
comments, but that is not the case currently.
Minecraft's component parser (i.e. Gson) produces null for these texts
instead of throwing a parse exception. By interpreting this text as
legacy text we correctly create text components for them that contain
their contents.
Issues resolved by this:
* SPIGOT-5063: Internal text representation of ItemStacks changes during ItemStack serialization. This issue was initially primarily concerned with the conversion between color text attributes to legacy color codes.
* SPIGOT-5304: Internal text representation of ItemStacks changes when opening the inventory (in creative mode). In particularly, this issue is also concerned with the conversion between plain text representations to non-plain ones.
* SPIGOT-5656: Internal text representation of ItemStacks changes during ItemStack serialization. This issue is particularly concerned with reordering of text attributes in the text's Json representation.
* SPIGOT-3206: Internal text representation of book pages changes during ItemStack serialization.
* SPIGOT-5350: Any non-plain text features are stripped from books during ItemStack serialization.
* SPIGOT-5980: Written books are marked as 'resolved' during ItemStack serialization and on various inventory interactions, even though they aren't, and thereby breaking any non-resolved page contents.
* SPIGOT-4672: Since item display names are serialized in their internal Json representation, any translatable components get properly persisted as well now.
---------
Minecraft uses text components to represent text. Internally Minecraft stores these components as Json formatted Strings and dynamically parses the text components from this Json representation whenever required.
In some cases Minecraft will create the text components and then convert them to Json itself for the internal storage. In other cases the Json representation is specified by users (eg. in Minecraft give commands, loot tables, mob equipment specified via Minecraft's summon commands, etc.).
There are many different ways in which the same text components can be represented in Json. When Minecraft compares objects which store this textual information, it takes the exact Json representation into account to determine whether the objects are considered equal. For example, ItemStacks will not match (and therefore not stack) if there is a difference in this internal Json representation for at least one if the item's text attributes (such as display name, lore, book pages, etc.). And when specifying nbt data in command selectors (eg. to only match entities/players which hold an item with specific name), the selector compares the raw Json representation as well.
As long as the Json representation is valid and can be parsed, Minecraft will not modify or normalize it.
However, under various circumstances Spigot converts this text information from the internal Json representation to text components (and in some cases even to plain text with legacy color codes) and then later tries to convert the text from these representations back to text components in the Json representation. Because this backwards conversion is in many cases not able to reproduce the original Json representation, the internal data of some affected Minecraft objects (ItemStacks, TileEntities, Entities, etc.) will in some cases get modified.
One especially notable situation in which this issue can come up is Bukkit's configuration serialization of ItemStacks: When a plugin serializes and later deserializes ItemStacks with display name, localized name, lore, or book pages of signed books, Spigot would convert these textual ItemStack attributes to plain text with legacy color codes and later try to convert those back to chat components in the Json representation. If the reconstructed Json representation does not match the original representation, the deserialized ItemStacks would no longer match nor stack with any original ItemStacks.
This case is particularly common if the original ItemStacks are created by users via some vanilla Minecraft mechanism (eg. Minecraft's give command, loot tables, mob equipment specified via Minecraft's summon command, etc.) and the used internal text representation for the created ItemStacks does not match the text representation produced by Spigot. This is also quite likely to be case, because the internal text representation produced by Spigot can sometimes be slightly verbose and, until recently, contained legacy color codes which cannot be used in Minecraft commands in-game.
However, this issue is not limited to items created by users, but affects items created by Minecraft itself as well.
Other cases in which Spigot itself (without any plugins involved) will convert between these text representations include dragging items around inside the inventory or opening the inventory while in creative mode. In these cases Spigot creates Bukkit representations of the affected items for use in Bukkit events and then, after the events have been handled, converts these Bukkit representations back to Minecraft items. See for example SPIGOT-5656 and SPIGOT-5304.
The idea of these changes is to avoid this back and forth conversion between the internal Json representation and the text component or plain text representations in various situations in which it is not actually required:
* CraftMetaItem stores the raw original Json representation for the display name, localized name, lore and pages of signed books now. As long as no plugin modifies these text attributes via the API, they can be reapplied in their original form to an ItemStack.
* The configuration serialization will serialize the original Json representation for these text attributes now so that it can also be restored during deserialization.
* However, in order to still be able to deserialize previously serialized items, and in order to allow users to specify text in the more simple plain representation in configuration files, we also still accept plain text during deserialization. Our approach is to check if the serialized text contains legacy color codes, in which case we convert it to chat components using our own converter and then to Json. Otherwise we try to parse it via Minecraft's Json parser. If the parsing fails due to the text not being valid Json, we interpret the text as plain text and convert it via our own converter as well.
* Various duplicated code has been removed from CraftMetaBookSigned and instead the base CraftMetaBook class allows sub classes to override the relevant aspects of how pages are parsed, serialized and deserialized.
* The BlockStates for command blocks and signs preemptively retrieved the custom name and sign line components, converted them to plain text and later converted them back to text components when applying the BlockState. We now only perform this conversion if a plugin has explicitly modified these texts.
Other changes:
* Minor: We also retrieve, convert and update a few other BlockState attributes directly from the underlying snapshot and only when requested by plugins now.
* SPIGOT-5980: Written books did not properly persist their 'resolved' attribute, resulting in unresolved book pages not getting resolved.
* There are methods to get and set the resolved value for books. However, these are not yet exposed in Bukkit.
* Minor fix: CraftMetaBook#isBookEmpty did not check some of the book attributes. This is probably a minor issue, but for consistency reasons there are checks for the missing attribute(s) now.
----
Covered cases
---
* By remembering the raw original String data, we can persist the exact text representation (eg. the ordering of elements within the Json text object (SPIGOT-5656), used style of escaping quotes (single quotes, escaped double quotes, etc.), use of plain texts (SPIGOT-5304), used boolean style, modern text component features such as translatable texts (SPIGOT-4672), etc.). All of these differences would otherwise cause the ItemStack to no longer be considered equal to its original.
* An empty String in the serialized config data results in no display name rather than an empty display name, like before. An item with explicitly empty display name (`{display: {Name: '""'}}`) is saved as `'""'` and can also be loaded from that representation again.
* Any plain texts, with or without color codes, which don't parse as Json (eg. `display-name: 'Bla'`) are still getting run through Spigot's text to components converter, like before.
* We can now also persist empty but explicitly present lore (`{display:{Lore:[]}}`). Previously this would get removed when the ItemMeta gets reapplied to the item. And ItemMeta#equals would return true for items with and without this empty lore data, even though Minecraft considers them to be different. For plugins using the API there should be no change: #hasLore still checks whether the lore is both present and not empty, and #getLore returns an empty list instead of null in this case (however, it previously already returned an empty list in this case). And setting the lore to an empty list via #setLore will still result in an item with no lore.
* Similarly, we can also persist explicitly specified but empty lists of book pages now.
----
Cases that are not covered (i.e. which may lead to changes in items), but were already not covered previously:
----
* NBT data for text that is not actually of type String.
* Empty or unexpected entries within the display compound.
* Variations in the NBT data representation in item features other than the above mentioned ones.
* Texts containing color codes. During deserialization these texts get interpreted as plain text and converted to a text component representation. This will break the serialization of any ItemStacks which actually use a text component representation with embedded color codes for some reason. Usually the likelihood for encountering such items in practice would probably be small. However, in the past (pre MC 1.16) Spigot would actually produce such items during ItemStack deserialization or when plugins created ItemStack via the Bukkit API. However, Spigot has changed the text representation it produces in MC 1.16, so any previously created and still existing items with this text representation are already problematic anyways now. See SPIGOT-5964. A fix for this linked issue (eg. the automatic conversion of these items) would probably resolve this deficit here as well.
* Spigot's String to text components converter produces quite verbose components since 1.16. See SPIGOT-5964 as well. However, this applies regardless of the changes of this PR.
* Book ItemStacks with more pages than 100 pages or oversized pages are truncated (like before) and may therefore change.
hange.
While cfeef75cd97 might of semi helped being able to save black text
lore, it actually took a fundamental problem with the legacy serialization
code and expanded it to break even more aspects of the server when dealing
with Component to Legacy conversion.
This is causing data loss in Spigot with cases such as setting an item name
to white gets stripped resulting in it being italic.
Additionally, things such as book pages have been returning black formatting
codes for the end of the line even when the user doesn't have colors in the book.
The root issue is that the "Default Color" system is fundamentally wrong.
Components do not and should not care about what element of the game they
are being used by, and that's what the default color system did.
It results in components that if obtained from 1 source such as a Book
where the default / rendered color is black, is then copied to another
source such as an Entity name, the black is carried forward and shown
in the Entity name, when in reality it should have been white.
This commit reverts cfeef75cd97 and fixes the underlying serialization
issues when it comes to Legacy to and From conversions.
There was quite a number of issues with this code overall, in how
it handles inserting color codes, new line parsing and such.
Books was using mojangs own "getLegacyString" which doesn't match behavior.
We also do not want to use Mojangs method as there is no guarantee they don't
remove that in future.
Plus, everything about books uses the CB implementation anyways, and it should
be consistent (this was mandatory to avoid serialization format changes on old vs new)
These changes as is results in Item Stacks already serialized will not
change contents when they go to component and back, so this won't impact
any existing data.
Newly created books though for example will change behavior in that they
will no longer insert black color codes in the serialized data and will
only represent intentional color changes by the creator of the book.
This will result in cleaner data on them, and books are the only thing
I'm aware of that has a behavioral shift due to the likelyhood of the
default color system kicking in on other parts of the string.
A unit test has been added to verify integrity of serialization to
ensure that any legacy string that is converted into Components will
always re-encode back in the same way when going back to Legacy.
