Access document embedding properties
You can now access the model name and embedding vector length of a document
embedding model by using the ModelName and
EmbeddingLength properties of a documentEmbedding
New and updated examples and topics
Use these new examples and topics to progress with text analytics workflows:
Text Chunking: Split large documents into chunks for retrieval-augmented generation (RAG)
You can improve the quality of AI generated text by providing a large language model (LLM) with additional information from a set of documents. This process is known as retrieval-augmented generation (RAG). Use text chunking to split a large document into smaller documents, then choose only the relevant document chunks to give to the LLM.
To get started, use the splitTextChunks function to recursively split the text into
paragraphs, sentences, tokens, and characters until each chunk is smaller than the
target length. For more information on text chunking, see Text Chunks. For more information on RAG, see Retrieval-Augmented Generation (RAG).
For a more customized workflow, use any of these additional functions.
Split your document into sections and preserve the section metadata using one of these functions:
splitHTMLSectionsSplit an HTML-formatted document into HTML sections according to the section tags
<h1>...</h1>,<h2>...</h2>, …,<h6>...</h6>.splitMarkdownSectionsSplit a Markdown-formatted document into Markdown sections, for example according to ATX section tags #,##, …,######.splitCustomSectionsSplit a document into custom sections according to custom section delimiters. Split your documents or your chunks recursively into paragraphs, sentences, and tokens using the
splitTextChunksfunction.To avoid redundancy, join similar adjacent chunks using the
joinSimilarTextChunksfunction.Add overlap between adjacent text chunks using the
addTextChunkOverlapfunction.After you have identified relevant text chunks, use the
findTextChunkContextfunction to identify other text chunks in the same section, subsection, etc., that you can add to the relevant text chunk to provide additional context.Finally, use the
formatTextChunksfunction to generate a Markdown-formatted prompt to provide to an LLM.
For an example showing this workflow, see Split Document Into Semantically Meaningful Text Chunks.
Use an LLM to generate text based on the retrieved text chunks. To connect to LLM APIs using MATLAB®, use the Large Language Models (LLMs) with MATLAB add-on. You can download Large Language Models (LLMs) with MATLAB from the Add-On Explorer.
Text Preprocessing: eraseURLs supports
tokenizedDocument
The eraseURLs function now supports tokenizedDocument inputs.
Tokenizers: Access vocabulary size
You can now access the vocabulary size of bpeTokenizer and bertTokenizer objects using the new NumTokens
property.
For example, you can use the vocabulary size to specify the number of words when
you create a wordEmbeddingLayer object.
Quality and stability improvements
R2025b delivers quality and stability improvements, building on the new features introduced in R2025a.
LLMs with MATLAB (GitHub): Ensure output format
You can now use structured outputs to generate output using openAIChat (GitHub®), azureChat (GitHub), and ollamaChat (GitHub) objects. Using structured outputs enforces the structure of the
generated output from supported large language models for postprocessing and data
analysis.
You can specify the structure of the output in two ways:
Specify a valid JSON Schema directly.
Specify an example structure array that adheres to the required output format. The software automatically generates the corresponding JSON Schema and provides it to the LLM. Then, the software automatically converts the output of the LLM back into a structure array.
To specify the structure of the output, set the
ResponseFormat name-value argument of
openAIChat, azureChat,
ollamaChat, or generate (GitHub) to one of these options:
A string scalar containing a valid JSON Schema
A structure array containing an example that adheres to the required format, for example:
ResponseFormat=struct("Name","Rudolph","NoseColor",[255 0 0])
For more information on structured outputs, see https://platform.openai.com/docs/guides/structured-outputs.
You can download Large Language Models (LLMs) with MATLAB from the Add-On Explorer. You can also download it from https://github.com/matlab-deep-learning/llms-with-matlab/.
LLMs with MATLAB (GitHub): Generate text with OpenAI o1 models
You can now use the OpenAI® models o1-mini and o1-preview to generate text from MATLAB. When you create an openAIChat (GitHub) object, set the ModelName name-value argument to
"o1-mini" or "o1-preview".
You can download Large Language Models (LLMs) with MATLAB from the Add-On Explorer. You can also download it from https://github.com/matlab-deep-learning/llms-with-matlab/.
LLMs with MATLAB (GitHub): Temporarily override model parameters when generating text
You can now set model parameters such as MaxNumToken and
ResponseFormat for a single API call by using the
corresponding name-value arguments of the generate (GitHub) function. The generate function then uses the
specified parameter for text generation instead of the corresponding model parameter
of the openAIChat (GitHub), azureChat (GitHub), or ollamaChat (GitHub) input.
For a full list of supported model parameters, see generate (GitHub).
You can download Large Language Models (LLMs) with MATLAB from the Add-On Explorer. You can also download it from https://github.com/matlab-deep-learning/llms-with-matlab/.
correctSpelling: New and updated extension
dictionaries
By default, the correctSpelling function now recognizes words from MathWorks® default extension dictionaries as correct. For example, the function
recognizes the word "MATLAB" and function names such as
"correctSpelling". To disable this feature and reproduce the
default behavior from previous releases, set ExtensionDictionary to [].
Functionality Being Removed or Changed
FontSmoothing property of TextScatter
objects has no effect
Behavior change
The FontSmoothing property of
TextScatter objects has no effect. Font smoothing is
enabled regardless of the value of the property. For more information about
TextScatter properties, see TextScatter Properties.
This property will be removed in a future release.
LLMs with MATLAB (GitHub): Connect to local large language models using Ollama
You can now generate text from local large language models in MATLAB. Connect to a local Ollama™ server and generate text using the ollamaChat
function.
To access the ollamaChat function, clone the GitHub repository at https://github.com/matlab-deep-learning/llms-with-matlab/tree/v3.1.0. To open all the files in MATLAB
Online™, from the GitHub repository, click Open in MATLAB
Online.
For examples and more information, see https://github.com/matlab-deep-learning/llms-with-matlab/blob/v3.1.0/doc/Ollama.md.
LLMs with MATLAB (GitHub): Connect to Azure OpenAI Service
You can now generate text from deployed large language models hosted on
Azure®
OpenAI Service using the azureChat function.
To access the azureChat function, clone the GitHub repository at https://github.com/matlab-deep-learning/llms-with-matlab/tree/v3.1.0. To open all the files in MATLAB
Online, from the GitHub repository, click Open in MATLAB
Online.
For examples and more information, see https://github.com/matlab-deep-learning/llms-with-matlab/blob/v3.1.0/doc/Azure.md.
Text Analytics Workflows: New and updated examples and topics
Use these new examples and topics to progress with text analytics workflows:
Document Embeddings: Convert documents to numeric vectors using pretrained embedding models
Load a pretrained document embedding model using the documentEmbedding function.
You can load these document embedding models using the Model
name-value argument:
"all-MiniLM-L6-v2"— Sentence transformer model with six self-attention layers. This option requires the Text Analytics Toolbox™ Model for all-MiniLM-L6-v2 Network support package."all-MiniLM-L12-v2"— Sentence transformer model with twelve self-attention layers. This option requires the Text Analytics Toolbox Model for all-MiniLM-L12-v2 Network support package.
For an example, see Classify Documents Using Document Embeddings.
You can download the support packages from the Add-On Explorer. For more information, see Get and Manage Add-Ons.
This feature requires a Deep Learning Toolbox™ license.
Transformer Neural Networks: Tokenize text using BPE tokenization
Byte pair encoding (BPE) is a tokenization algorithm that allows transformer networks to handle a wide range of vocabulary without assigning individual tokens for every possible word. During tokenization, the algorithm replaces out-of-vocabulary (OOV) words with subword counterparts, which allows models to handle unseen words more effectively. This process creates a set of subword tokens that can better represent common and rare words.
To tokenize text using byte pair encoding (BPE), use a bpeTokenizer object. You can use these functions with a
bpeTokenizer object:
Transformer Neural Networks: Pretrained BERT large language model
Load a pretrained Bidirectional Encoder Representations from Transformers (BERT)
large language model using the bert function.
You can load these variants of the BERT model using the Model
name-value argument:
"base"— BERT base-sized model (108.8 million parameters). This option requires the Text Analytics Toolbox Model for BERT-Base Network support package."tiny"— BERT tiny-sized model (4.3 million parameters). This option requires the Text Analytics Toolbox Model for BERT-Tiny Network support package."mini"— BERT mini-sized model (11.1 million parameters). This option requires the Text Analytics Toolbox Model for BERT-Mini Network support package."small"— BERT small-sized model (28.5 million parameters). This option requires the Text Analytics Toolbox Model for BERT-Small Network support package."large"— BERT large model (334 million parameters). This option requires the Text Analytics Toolbox Model for BERT-Large Network support package."multilingual"— BERT multilingual model (117.2 million parameters). This option requires the Text Analytics Toolbox Model for BERT-Base Multilingual Cased Network support package.
You can download the support packages from the Add-On Explorer. For more information, see Get and Manage Add-Ons.
This feature requires a Deep Learning Toolbox license.
Transformer Neural Networks: Train BERT document classification model
Create and train a BERT document classifier using the trainBERTDocumentClassifier function. The function outputs a bertDocumentClassifier object that you can use for document
classification and a TrainingInfo (Deep Learning Toolbox) object that contains information
about the training process.
To classify models using a trained BERT document classifier, use the classify function.
For an example, see Train BERT Document Classifier.
This feature requires a Deep Learning Toolbox license.
Transformer Neural Networks: Create tokenizer for BERT models
Create a tokenizer for BERT models using the bertTokenizer function.
You can interact with the tokenizer using these object functions:
Word Embedding Layer: Specify out-of-vocabulary word handling mode
For wordEmbeddingLayer objects, specify the out-of-vocabulary word
handling mode using the OOVMode property. You can specify that the layer maps
out-of-vocabulary words to the same embedding vector (the last vector in the
weights) or throws an error using the "map-to-last" and
"error" values, respectively. This property enables support
for models that already have an out-of-vocabulary token in its vocabulary, such as
BERT.
Text Analytics Workflows: New and updated examples and topics
Use these new examples and topic to progress with text analytics workflows:
Functionality Being Removed or Changed
ismember function for wordEmbedding
objects has been removed
Errors
To update your code for wordEmbedding object input, replace ismember with isVocabularyWord. You do not need to change the arguments. The
syntaxes are equivalent.
Text Preprocessing Live Editor Task: Preprocess text data in Live Editor
Interactively explore text preprocessing steps using the Preprocess Text Data Live Editor task. The task helps you explore and visualize different preprocessing steps for your workflows. You can clean up HTML, tokenize, add token details, change and remove words, and display results. For an example that explores different preprocessing options, see Preprocess Text Data in Live Editor.
The Preprocess Text Data Live Editor task automatically generates MATLAB code for your live script. For an example that shows how to use the Preprocess Text Data task to create a preprocessing function that you can reuse in your workflows, see Create Simple Preprocessing Function.
PDF File Information: Extract file information from PDF files
Extract information from PDF files using the pdfinfo function. The function returns a structure containing
information about:
The number and sizes of the pages
The PDF version
The title, language, and keywords
The author, creator, and producer
The dates and times when the file was created and modified
If the file is encrypted and allows text extraction
The PDF filename
Text Preprocessing: Split text into paragraphs
Split strings and documents into paragraphs using the splitParagraphs function.
Named Entity Recognition: Train custom NER model
Train a custom model for named entity recognition (NER) using the trainHMMEntityModel. The function returns an hmmEntityModel object.
Add entity details to tokenized documents using the addEntityDetails function. To specify a custom NER model, use the
Model name-value argument. To view the details of document tokens,
use the tokenDetails function.
For an example that shows how to train a custom NER model, see Train Custom Named Entity Recognition Model.
Grammatical Dependency Parsing: Visualize grammatical dependency parse trees
Visualize the grammatical dependency parse tree of a sentence using the sentenceChart function.
You can customize the resulting DependencyChart object by its
modifying properties. For a list of properties, see DependencyChart Properties.
For string input, this feature requires Deep Learning Toolbox and the Text Analytics Toolbox Model for UDify Data support package. If this support package is not installed, then the function provides a download link.
This feature supports English, German, Japanese, and Korean text.
For an example showing how to visualize and extract information from a sentence, see Analyze Sentence Structure Using Grammatical Dependency Parsing.
Grammatical Dependency Parsing: Add grammatical dependency details to documents
To add grammatical dependency details to documents, use the addDependencyDetails function. To view the token details, use the
tokenDetails function. The grammatical dependency details are in the
Head and Dependency columns, where
Head contains the indices of the head for each token, and
Dependency contains the grammatical dependency
categories.
This feature requires the Text Analytics Toolbox Model for UDify Data support package. If this support package is not installed, then the function provides a download link.
This feature supports English, German, Japanese, and Korean text.
For an example showing how to visualize and extract information from a sentence, see Analyze Sentence Structure Using Grammatical Dependency Parsing.
Keyword Extraction: Extract case-insensitive keywords and use case-sensitive delimiters using RAKE algorithm
Extract case-insensitive keywords using the rapid automatic keyword extraction
(RAKE) algorithm by setting the IgnoreKeywordCase option of the rakeKeywords function to 1
(true). Use this option when you expect the same keywords to
appear with variations in letter case and want to treat them as the same keyword,
for example, the words "analytics", "Analytics", and "ANALYTICS".
Use case-sensitive delimiters for RAKE by setting the IgnoreDelimiterCase option of the rakeKeywords function to 0
(false). Use this option when you expect keywords and
delimiters to differ only by case, for example, the delimiter "and" and the acronym
"AND".
Document Similarity: Ignore case in BLEU score evaluation
Ignore case when you evaluate the bilingual evaluation understudy (BLEU) scores of
a candidate document against a set of reference documents by setting the IgnoreCase option of the bleuEvaluationScore function to true.
Document Preprocessing: Check for substrings, words, and n-grams in documents
Check for substrings, words, and n-grams in tokenizedDocument objects using the contains, containsWords, and containsNgrams functions, respectively.
Document Preprocessing: Split document into sentences
Split a single tokenizedDocument object into sentences using the splitSentences function.
Unicode Representation: Convert text to more normalized forms
Strings that look identical can have different underlying representations. The Unicode composed normalized form (NFC), Unicode compatibility composed normalized form (NFKC), and Unicode compatibility decomposed normalized form (NFKD) ensure that equivalent strings have unique binary representations. Use these functions to help compare strings that contain accented characters, which can have several representations.
To normalize text using NFC, use the
textanalytics.unicode.nfcfunction.To normalize text using NFKC, use the
textanalytics.unicode.nfkcfunction.To normalize text using NFKD, use the
textanalytics.unicode.nfkdfunction.
Text Analytics Examples: Explore text analytics workflows
The Classify Documents Using Document Embeddings example shows how to train a document classifier by converting documents to feature vectors using word embeddings.
Functionality Being Removed or Changed
FontSmoothing property of TextScatter
objects will have no effect in a future release
Behavior change in future release
The FontSmoothing property of
TextScatter objects will have no effect in a future
release. Font smoothing will be enabled regardless of the value of the
property.
tokenizedDocument does not split tokens containing
digits and some special characters
Behavior change
Starting in R2022a, tokenizedDocument does not split some tokens where digits appear
next to some special characters such as periods, hyphens, colons, slashes, and
scientific notation. This behavior can produce better results when tokenizing
text containing numbers, dates, and times.
In previous versions, tokenizedDocument might split at these
characters. To reproduce the behavior, tokenize the text manually or insert
whitespace characters around special characters before using
tokenizedDocument.
Topic Modeling: Explore Visualization Examples
New examples help you visualize data using topic models:
For more information about topic modeling, see Analyze Text Data Using Topic Models.
Deep Learning: Language Translation Example
For an example showing how to construct and train a neural network for language translation, see Language Translation Using Deep Learning.