fiction nlp

(blog still being drafted)

so near the start of the year i got a new job as a software / systems engineer, where i became responsible for the modernisation and a general move to AI tooling and infrastructure for a firm. we work on very lengthy and complex documents for our clients, and the climate within the firm is tending towards finding ways to streamline rote tasks; relatively easy, but usually extremely time consuming for employees. an example task that someone in the firm might have to do is renaming files within a client’s case folder, of which it can have hundreds of scattered communiques in the form of emails, recorded correspondence, to medical reports, to legal affidavits, where these files can also reach hundreds if not thousands of pages. when we receive these files from our clients, we expect to store them in a structured manner. dates, proper filenames, folder structure, etc. yet, as it usually goes with external clients, that expectation generally lies on us to fulfill, and not the clients themselves. anyway, the natural conclusion of all of this rote work is that the firm wants to create infrastructure that automates a lot of these tasks, which includes the use of AI tooling.

one area that we proposed researching is a sort of “smarter” document preprocessing. we rely heavily on a tool called docling, which is a document processor that can parse documents with complex structure. while it is generally very good, we still run into issues with output consistency and accuracy. this led me down a rabbit hole for text processing, which includes the likes of NLP (natural language processing) and NER (named entity recognition). both are something that i was interested in looking into in my personal time, which leads me to this project.

one of my pet writing projects, a touhou inspired world that ive been writing on and off for a few years, has gotten to a point where i wanted to model / visualise the narrative connections within its stories. things like cataloging its characters, events, timelines and what not. which led me to researching ontologies for fiction, and settled with trying to model my story within the bounds of an ontology called GOLEM. although, i specifically wanted this modeling of my world to be automated. or at least to start with, the extraction of useful narrative data out of my stories to be automated. which naturally led me to NLP and NER.

i quickly found out that i was in over my head for this idea, because from what i read i am quite sure that NLP on fiction - narratives famously containing endlessly made up bullshit - may be quite difficult due to that nature. nevertheless, it was interesting enough to try (and maybe also burn some tokens on, considering this is transformer AI work). one NLP pipeline that i found which i tried is BookNLP. its an NLP pipeline trained on, you guessed it, fiction. most NLP pipelines out there are mostly trained for news and non-fiction so generally they wont work on fiction. when i tried booknlp i felt like it wasnt catered to what i wanted. i wanted something more fundamental in terms of tagging entities; since the goal is to model the world based on GOLEM, BookNLP was already making assumptions with categorising entities which might conflict with GOLEM.

anyway, the reason this matters is because to actually catalog things from a story, the first step is actually to extract them out of the text. “entities”, as a lot of NLP pipelines call them. and because my long term goal is to use GOLEM’s model, i started with a more fundamental question for extraction and detection of entities within text: what in the context warrants treating this passage as referring to something distinguishable, before deciding what that something is? if, lets say, you had some text that described a something:

“something appeared outside. it moved across the window.”

nothing about this passage tells you anything about what the “something” is, but you know that it is something, that its an entity. therefore, what and which features of this text warrants this passage as referring to that “something”? which leads me to a conceptual tool called discourse representation theory. something from formal linguistics which is way out of my field of study, but basically, it gives a way of representing what the text is talking about without assigning it any categories.

DRT builds a representation of discourse with two parts: discourse referents, like x, and conditions that describe them. for “something appeared. it disappeared,” the representation could be:

[x : appeared(x), disappeared(x)]

looks familiar to conceptualising something programatically or mathematically, right? so i used this as the basis for the start of my own NLP pipeline. so which stretches of text should be extracted or preserved, and what evidence about each occurrence could help decide what it refers to? which leads to the introduction of two things, mentions, and referents. a mention is a textual occurrence, and a referent is what the occurrence points to in the story or discussion (discourse).

but, when is an occurrence in text a mention? this is a particularly slippery question. i think it consists of two separate questions when you boil it down to an implementation of the idea of DRT.

the first question is: does this occurrence (string of text at some location) have a candidate of a mention? meaning, are there rules or features in text, such as capitalisation, title patterns (character honorifics), stopwords, parts of speech, and name-like runs of text, that point to evidence of the text describing a mention?

the second question, arguably more difficult, is: after those occurrences are identified, and candidates are proposed from those occurrences, do those candidates actually count as mentions? as in, if you read the candidate in its occurrence context, does it introduce, pick up, or identify something specific that the text treats as a thing to talk about in its own right?

the current experimental implementation of this NLP only asks and tries to answer these two questions within an extremely limited scope.

to answer the first question, consider an input text, lets call it a manuscript. this manuscript is first normalised into plain text. markup characters are removed, then structural and formatting information such as headings, italics, verse lines, and front matter is preserved in offset annotations. sentence segmentation and quotes identify sentence and quotation boundaries from the text; the annotations provide line and formatting cues. the text is then tokenised by sentence and annotated with part of speech tags, capitalisation and ambiguity cues, stopword and title information, and quote, zone, and markup features. lets call this step the preprocessing step.

the result of this preprocessing feeds candidate generation, which creates the first layer of this NLP work, and gives answers to the first question. candidate spans are generated from the tokens and their surrounding context, either with rules based cascade (using features like capitalisation within a quote, which could mean that the text refers to a proper noun which could potentially be a mention). this proposes eligible tokens as individual spans, then join some adjacent tokens into longer spans. an example passage from one of my stories:

⟦2: ⟨1: Even⟩ ⟨3: Kohaku⟩⟧ approved, her holographic voice ringing in ⟨4: Yō⟩’s head. “⟨5: Vacation⟩ compliance correlates to a ninety-seven percent reduction in cortisol markers from high-stress engagements,” she pronounced, clinically smug. “⟨6: Strongly⟩ recommended.“

3: Kohaku and 2: Even Kohaku are proposed, and a later possessive rule turns 4: Yō’s into 4: Yō. these are candidates for review, not decisions that the words refer to specific things. for example, 2: Even Kohaku includes an adverb in the span, which is not a valid mention.

the generation of these candidates is done by:

  1. span-generation rules that use capitalisation, position, titles, possessives, connectors, numbers, abbreviations, and other patterns to generate 1: Even, 2: Even Kohaku, 3: Kohaku, 4: Yō, 5: Vacation, and 6: Strongly. these rules use the annotations and tokens done by the preprocessing step.
  2. a GLiNER2.5 model pass reads the document text and proposes spans. i gave gliner label descriptions such as “proper name”, “named concept”, and “title or role”, then applied a score threshold and a coined-word filter to its proposed spans. this pass generates 3: Kohaku and 4: Yō as proper names.
  3. document-wide evidence tracks how these forms appear elsewhere, groups related forms, and admits some forms for a recovery pass. recovery can add occurrences that the first generation pass missed. the resulting candidates retain their derivation and overlap evidence.

i also painstakingly annotated a gold set of entities beforehand, from books that i know pretty intimately. specifically, my own stories, and various sci-fi series like arkady martine’s teixcalaan duology. i also used open source annotated libraries like litbank, which consists of classics that have been annotated by real humans. i also experimented with running a bunch of agents on an annotating task with other fantasy series’ that i havent read, but with requirements based on my own handmade gold set. this gold set is used to score the accuracy of the generated candidates as a reference point.

from these candidate proposals, we can then answer the second question. so, what of these proposal candidates count as mentions? or, in DRT terms. does this occurrence refer to a specific referent that the text establishes or picks up?, something later text could go on talking about as “that one” or “those”? we can answer this with three outcomes. specific, not specific, or unresolved. if the outcome is specific, then yes, this candidate is a mention. but “specific” doesnt mean real, named, repeated, or human. an unnamed man can count, like my first example a few paragraphs back. so can an imagined or hypothetical ghost, if the text picks out one, or even a group can count too.

so this makes up the second layer of this nlp work. the rules that i came up with for putting candidates into the buckets of specificity goes like this (occurrence in text in brackets):

  1. if the occurrence refers over many instances (any member of a category, or the category itself), it is about a kind, so it gets not_specific. a reason is annotated along with it, category. if the story has only the one thing, every occurrence is about that one: specific. for example:
  • “[The Stationers] have a saying” -> not specific (talks about the people in general)
  • “[The Stationers] closed the door” -> specific (talks about the particular people in this scene)
  1. a role whoever fills it (an office, rank, or title) is not_specific, reason role. but, a title used to address someone picks out the person addressed is specific. if the occurrence is by chance a single token, like a title inside a name (“Mrs” in “Mrs. Yō”) stays not_specific with reason role. for example:
  • “Nineteen Adze is the [Emperor of All Teixcalaan].” -> not_specific (the referent here is nineteen adze, but the candidate is the position of emperor. so nineteen adze is specific, but the position of emperor is not)
  • “[The Emperor of All Teixcalaan] smiled at her.” -> specific (a specific emperor is smiling at someone)
  1. an expression that says something about a referent, rather than picking one out, is not_specific, for reason predicative. for example:
  • “Elara became [a queen]” -> not_specific (predicative)
  • “[A queen] entered” -> specific
  1. a word cited as a word, a greeting, a figure of speech, or an oath stands for nothing. it is not_specific for reason non-referential. similarly, a heading or label that only names the function of a page or block stands for nothing is also not_specific for the same reason. but, a heading that names a document or work in the story, or a named part of the book, refers to that thing, which means its specific. a chapter or part title names that chapter or part whatever its grammar, so the whole title is specific, in the heading and in the contents (“The Lion Becomes the King of Beasts” as a contents entry). its constituent pieces however are not_specific for reason non-referential. a speaker label at the start of a line in a play or dialogue format names who speaks, so its specific
  • “the name [Yō]” -> not_specific (non-referential to yo, anyone can be named yo)
  • “[Welcome] to the Weight for the Wheel” -> not_specific (non-referential)
  • “Jonathan Harker’s Journal” -> specific (as a section heading)
  • “II.” or “VII” alone as a section number -> not_specific (non-referential)
  • “[Five Minutes] Later” as a time-skip heading not_specific (value)
  • “Entering [Safe Room].” -> specific (referent is the safe room, where something is entering)
  1. find the full phrase that refers to something. a candidate can leave off a leading determiner such as “the” or “her” and still stand for the same referent. it can also leave off a leading title. this applies in prose, headings, and contents entries. if another candidate covers the full referring phrase, a shorter piece of that phrase gets not_specific, reason non-referential. if no candidate covers the full phrase, a cut-off piece that points towards its referent gets unresolved, reason boundary. a complete name inside a titled phrase can still refer on its own. a title or determiner alone is judged as a part of the phrase.
  • “the [Triumvirate] watches” -> specific (triumvirate refers even though its candidate omits “the”)
  • “[Polar]” in “the Polar North”, when “the Polar North” is also another candidate -> not_specific (non-referential)
  • “[The Norre]” in “the Norre Institute of Magics”, when no other candidate covers the full name -> unresolved (boundary)
  • “[Mrs]” in “Mrs. Yō” -> not_specific (role); “Mrs. [Yō]” -> specific
  • “the Emperor of All [Teixcalaan]” -> specific (Teixcalaan is a different referent inside the longer phrase, next section)
  • “[Nilam on the Polar North]” -> not_specific (non-referential) (this candidate joins two referent phrases into one span)
  1. a word inside a larger phrase can refer to something of its own. the word is judged by what it stands for, not by what the whole phrase stands for. a possessor can pick out its owner, and a modifier can name a particular thing, describe a kind, or leave the distinction unclear.
  • “the [Estuary] families” -> specific (Estuary names one ship)
  • “[the Estuary]” in “the Estuary families” -> not_specific (non-referential) (“the” belongs to “families”, so this candidate doesnt form the ship’s name)
  • “the [Prologue]’s analysis stack” -> specific (Prologue picks out the ship that owns the stack)
  • “a [Teixcalaanli]-style smile” -> not_specific (category) (the word describes a manner of smiles)
  • “a [Teixcalaanli] ship” -> specific, not_specific, or unresolved (depending on whether the text means the polity’s ship, a kind of ship, or leaves that open)
  1. a date, time, quantity, measure, or coordinate gets not_specific, for reason value. capitalisation and uniqueness doesnt change that answer, and a named event that happened once can refer to that event and gets specific.
  • “in [May]” -> not_specific (value)
  • “[That [Christmas] changed everything]” as an event -> specific
  • “[Five Minutes Later]” as a time-skip heading -> not_specific (value)
  1. if a candidate points towards a specific referent but stops short of the full referring phrase, check whether another candidate covers that phrase. if none does, give the cut-off candidate unresolved, for reason boundary, and record the full span it needed. a missing leading determiner alone doesnt count as a wrong edge.
  • “[The Norre]” in “the Norre Institute of Magics”, with no other candidate for the full name -> unresolved (boundary)
  • “[Triumvirate]” in “the Triumvirate watches” -> specific (the missing “the” does not change the referent)

i feel like theres still some contradictions within these rules, which is only natural, honestly. from what i read of people who did litbank annotations (and other annotations for that matter), being consistent in decision-making against the criteria is actually quite difficult, which through my own annotation and auditing work, i have to say i agree.

but anyway, with all these rules, how does a computer actually decide with these rules? unlike the first layer, where all i was doing was loosely grabbing shit out of text, this time i actually have to ask my computer “tell me if this thing is actually a thing,” which is, pretty dang hard without a statistical model.

initially i tried to do this decision of acceptance into buckets without a statistical model, based on the rules that were easiest to algorithmically decide. values most of the time are not specific, headings are usually not specific, a noun after a predicate usually isnt specific, and other algorithms broken down from the 8 initial rules above.

while those rules could settle clear cases, many occurrences couldnt be matched by them at all. i then tried a gradient boosted tree model on whatever layer 1 had given me, which was how a candidate was generated, all of its clues from grammar, nearby words, overlaps, and other uses elsewhere in the document. so basically, instead of my choosing how much each clue mattered, the model learned combinations of clues from labelled examples and returned a probability for each acceptance outcome. this was a first try into statistical models.

the thing that worked the best was fine tuning a BERT model, specifically, a fine tuned DeBERTa which is an encoder, to read each candidate in its sentence. i marked the exact span under judgement, so the model saw:

Even <candidate> Kohaku </candidate> approved... when judging Kohaku.

this let it learn from the surrounding language, while the tree model used the explicit evidence about competing spans and document patterns. then, another boosted tree then combined both models’ probabilities, the overlap evidence, and the rule modules’ decisions to make the current final judgement.

basically, the main result is that fine tuning deberta improved decision making for accepting candidates so much more than adding hand written algorithmic rules. which i mean, considering NLP generally implies the use of a statistical model, this makes a lot of sense.